AI judgment should know how likely it is to be right, and hold back where the evidence runs out.
We build models for AI judgment and evaluations. You state the criterion in plain words and set the accuracy. They answer where the evidence meets it, and abstain everywhere else.
The calls
AI systems make judgment calls all day.
- Whether a reply meets the standard.
- Which queue a ticket belongs in.
- Whether an agent finished its task.
- Whether an answer is right.
- Whether a listing breaks the rules.
Each is a judgment: a criterion, a thing to hold against it, and a verdict. A careful person making these calls knows which ones are beyond them, and says so. That is what makes their verdicts worth acting on.
The two ways today
There are two ways to automate a judgment. Each costs something.
A trained classifier
It is consistent, and its accuracy is measured before it goes live. But every criterion needs its own labeled data, and someone to train it.
A generative model
It takes any criterion in plain words. But its verdict is sampled: written the way it writes any text. How likely it is to be right is yours to measure, with the same labeled data the classifier needed. Ask again, and the verdict can change.
One knows its accuracy and learns each criterion from scratch. The other takes any criterion, and leaves its accuracy to you to measure.
What we build · A8 (Touchstone), available for preview
A8 is a model that takes the criterion in plain words, and answers only where it can carry the accuracy you set.
You call it through the OpenAI or Anthropic SDK. The system message is the criterion. The user message is the thing to judge.
It answers at the accuracy you set, and abstains below it.
You set it on each call, as a number from 0 to 99. At 95, at least 95 of every 100 verdicts A8 gives are correct, and a case the evidence can’t carry that far comes back as an abstention. The same request returns the same verdict.
It learns from every correction, and only for you.
Teaching a generative model a new criterion means fine-tuning it. Teaching A8 is one sentence to your assistant: what it got wrong, and what the answer should have been.
| Fine-tuning a generative model | Teaching A8 |
|---|---|
| A labeled dataset, gathered and uploaded | The answer you wanted, taught from any MCP host |
| A training run to start and watch | A fitting round in the background, a minute after your workspace goes quiet |
| A copy of the model to host | A new version, served for you |
What you teach is kept the moment you approve it, and A8 keeps answering while it fits. It stays in your workspace and trains its model alone.
Automate the calls A8 answers. Send the ones it abstains on to a person.
How we know
We test the approach against trained human raters, in the open.
In qed-bench, our open benchmark, we scored essays with the approach A8 is built on, and compared the ranking with the one trained human raters gave. A panel of eight generative models judged the same essays.
- Our approach
- 0.815
- The strongest generative judge on the panel
- 0.813
Rank correlation with trained human raters, essay quality. At 1, the two rankings match exactly. The notebooks are public. read qed-bench
When a judgment is automated, its accuracy should be a number you set before it answers.
Tell us where AI judgment falls short in your work.
We are asking the people who build, run and sign off on automated systems how they check them and decide when to trust them.
The AI Oversight Study is open until December 31, 2026. About five minutes, with questions for your role.
take the survey