Picture a support chatbot that invents a pricing tier, and a customer who quotes it back to your sales team. That is the failure mode everyone fears, and it is entirely preventable.
1. Ground every answer in your own content
A model answering from its training weights will confabulate about your product, because it does not know your product. Retrieval-augmented generation fixes this: retrieve the relevant passages from your documentation, changelog and resolved tickets, and instruct the model to answer only from those passages.
The retrieval quality matters more than the model choice. In our benchmarks, moving from naive vector search to hybrid search — vector plus BM25 keyword, with a reranking pass — improved answer accuracy more than upgrading to a larger model, at a fraction of the cost.
2. Require citations
Every answer cites the source passages it used, rendered as links. This does three things: the customer can verify, your support team can spot a wrong answer instantly, and the model behaves more conservatively when it knows it has to point at a source.
If the model cannot cite, it should not answer.
3. Train refusal as a correct answer
The single most useful thing we do is include deliberately unanswerable questions in the evaluation set, where "I do not have information about that — let me connect you with someone who does" is scored as fully correct.
Most teams build evaluation sets entirely from questions with good answers. That optimises for answering, which is exactly the wrong incentive.
4. Build the evaluation set before choosing a model
This is where most projects go wrong in sequence. They pick a model, build a prototype, demo it, and then try to work out whether it is good.
Build the golden set first — 200 to 300 questions with expected answers, drawn from real tickets, including edge cases, ambiguous phrasings and unanswerable questions. Then benchmark configurations against it. A golden set often reveals that a smaller, cheaper model with good retrieval beats a frontier model with naive retrieval — at a fraction of the cost. You only find that out by measuring.
The evaluation set is also what protects you afterwards. When a model version updates, you re-run the set. Silent degradation after a provider update is a real phenomenon, and without an evaluation harness you find out from a customer.
5. Confidence-gated handoff
Score every response. Below the threshold, hand to a human with the conversation and the retrieved context attached, so the agent does not start from nothing.
Set the threshold conservatively at launch and relax it as the accuracy data comes in. Over-deflecting damages trust much faster than under-deflecting.
6. Guardrails on inputs and outputs
Prompt injection is real: users will paste text designed to override your instructions. Treat retrieved content and user input as untrusted, keep system instructions outside the user-controlled context where the API allows it, and validate output shape before rendering.
Redact PII before it reaches the model. Enforce a per-request token ceiling so a pathological input cannot generate a pathological invoice.
What good looks like
The deployment described above runs at 94.2% accuracy on its evaluation set, 0.4% hallucination rate, 68% deflection, and about $340 a month in inference for roughly 11,000 conversations.
None of those numbers came from a better model. They came from retrieval quality, refusal training, and measuring properly.