It wasn't hallucinating

The assistant was saying things nobody had told it to say. Small things, delivered confidently, that didn’t match the clinical guidance we’d given it. In a health product that’s the bug you fear most, and it has a familiar name: hallucination. The usual fixes are familiar too: tighten the wording, add a guardrail, add a judge that catches it.

We almost did exactly that.

Reading the prompt like the model does

Before changing anything, we went back through the prompt line by line, reading it the way the model does: all at once, every instruction with equal weight.

The clinical guidance disagreed with itself. It had grown over months, added by different people for different cases, and it lived in several places. One section said one thing, another section said something slightly different for an overlapping situation. Neither was wrong on its own. Together they had no single right answer.

So the model picked one, or blended them. From the outside that looked like invention. It was the model doing its best with instructions that couldn’t all be followed at once.

What changed

Judges that trace, not just flag. Our evaluation judges had been asking “is this hallucinated?” That question couldn’t tell invention apart from a model resolving a conflict we’d created. We retuned them to point at where an answer came from, so a failure now names the instruction it traces back to, not just that something went wrong.

One home for clinical guidance. The clinical parts of the prompt moved into one place, with clear ownership. Fixing or extending them became a contained change instead of a hunt through the whole prompt. But putting two rules side by side only helps if someone reads them side by side, and on a busy team nobody reliably does.

A conflict check on every prompt change. So the real fix moved upstream. Every change to the prompt now goes through a review skill: an agent that reads the change against the guidance already there and flags anything that contradicts or overlaps it, before the change merges. Catching a contradiction stopped depending on a reviewer happening to remember the other rule.

The lesson

“The model hallucinated” is a diagnosis that ends the investigation too early. Before blaming the model, check whether you’ve given it an impossible job. Prompts are code, and code that grows without structure ends up contradicting itself. So treat them like code: one owner per rule, and an automated check on every change.