Notes on abstention
The most underrated capability in high-stakes AI is the ability to say "I don't know — this needs a human." A system that always answers is easy to build and hard to trust.
A system that always produces an answer is easy to build. A system that knows when not to is the harder and more valuable thing, and in high-stakes work it's the difference between a tool a professional will rely on and one they'll quietly stop using after it burns them once.
Abstention isn't the system failing. It's the system respecting the cost of being wrong.
The mechanics are less mysterious than they sound. In a grounded retrieval system, two signals do most of the work:
- Coverage. Do the retrieved passages actually support the claim the system wants to make? If the best evidence is weak, the answer is weak, no matter how fluent it reads.
- Margin. Is the top result clearly ahead of the next one, or is it a coin flip between two passages that say different things? A thin margin is the system telling you it isn't sure.
When either drops below a threshold, the right move is not to answer more carefully — it's to route the question to a person, with the partial evidence attached so they start ahead of zero. You can see this directly in the retrieval visualizer: push it into a case with poor coverage and it drops to a "needs review" state instead of manufacturing confidence.
Which suggests a metric most demos never report: not just how often the system is right when it answers, but how well it declines when it shouldn't. Precision on the questions it chooses to take on is worth more than coverage of every question you can throw at it. I'd rather ship a system that answers 70% of cases well and hands off the rest cleanly than one that answers everything and is quietly wrong a tenth of the time — because in this domain, you don't find out about the tenth until it's expensive.