FDE, Part 3: Retrieval, Tool Use and Evaluation in Enterprise AI
Connect document processing, RAG, access controls, tool calls and evaluation into a traceable, maintainable AI system.
Updated
An enterprise knowledge assistant usually finds information before answering. That involves parsing, retrieval, permissions, models and business APIs. Define each step’s inputs, outputs and failure handling so problems can be located, rather than hiding everything inside one model call.
Preserve the context of documents
Keep sources, sections, versions and applicability. Table column relationships and numbered repair steps may change the meaning. Sample parsed documents to ensure essential structure survived.
Split by section or semantic boundary. Chunks that are too short lose conditions; chunks that are too long mix in unrelated material. Validate the choice with task samples. New publications and withdrawn documents must also update indexes and caches.
Evaluate retrieval and generation separately
RAG means retrieval-augmented generation: obtain relevant external material, then let the model organize an answer. It enables enterprise knowledge use, but correctness still needs verification.
Retrieval checks ask whether required evidence appears and matches version and permission requirements. Generation checks ask whether the answer is faithful to evidence, omits conditions or makes unsupported judgments when evidence is insufficient. Separating them shows which stage needs improvement.
Model numbers and error codes often require exact matching; natural-language descriptions can benefit from semantic retrieval. Adopt hybrid retrieval or reranking based on tests, considering added latency and maintenance.
Implement citations and permissions together
Citations should locate the supporting passage and show relevant version information. A merely related link that doesn’t expose the specific evidence increases the user’s checking effort. With insufficient evidence, ask questions, limit the answer or hand over to a human.
Enforce permissions in retrieval and API layers. Answer summaries must not leak unauthorized material, and source links need identity checks. Confidentiality instructions can guide a model but cannot replace access control.
Set clear boundaries for tool operations
Reading a ticket and changing it have different consequences. After the model proposes parameters, the server should validate identity, permissions and business rules. Significant writes should show their exact content and require confirmation. Retries must avoid duplicate writes.
An agent can select steps and tools, but when the path is already known, a fixed workflow is often easier to test. Either approach needs limits on time, call count and cost, plus stopping conditions for API failures. External documents are data, not authority to change permissions.
Use evaluation to support updates
Include real tasks, difficult examples and abnormal inputs. Check citations, essential conditions, permission isolation, API results and complete task time. Model-based grading can help analysis, but specialists should review professional conclusions.
After changing a prompt, model, index or parser, compare relevant samples. Check regressions in critical tasks, not only aggregate performance. Link release records to these configurations to reproduce problems.
Provide fallback paths. If the model fails, retain document search; if a business API fails, return the operation to a person. A clear status is more useful than another conclusion based on incomplete information.