available for preview

A8 Touchstone

Any criterion, in plain words. A verdict that clears the accuracy you set, or an abstention.

A touchstone. The sample’s streak is read against the streaks of needles of known fineness, in parts per thousand.

the stone

Gold had to prove its grade.

Theophrastus described the test around 300 BC. An assayer drew the gold across a black stone and left a streak. Beside it he drew needles of known fineness, one streak each, and read the gold’s color against theirs.

The needle it matched gave the grade. An unmatched streak was set aside.

The sample reads between 916 and 750.

the same five parts

A8 reads a subject the way an assayer read a streak.

The gold: The subject, sent as the user message.
The test: The criterion, in plain words, sent as the system message.
The needles: The examples you teach it.
The grade: The minimum accuracy, from 0 to 99, set on each call.
An unmatched streak: An abstention.

the grade

You set the grade. Below it, A8 abstains.

Every call carries the accuracy its verdict must meet, from 0 to 99. At 95, at least 95 answered verdicts in a hundred are correct. A case whose evidence falls short of 95 comes back as an abstention.

The verdict is scored from the evidence, so the same request returns the same verdict.

a call

Call it like a chat model. Set the grade on every call.

1
The system message is the criterion: what you are judging.
2
The user message is the subject: the thing under judgment.
3
The schema is the verdict’s shape. A8 answers in JSON that matches it.
4
The model’s tag sets the grade: a8:95 is 95, and 90 is the default. At 95, at least 95 answered verdicts in a hundred are correct. Below it, A8 abstains.
5
To correct it, tell your assistant. Over MCP it teaches A8 the answer you wanted, once you approve.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.u22a8.ai/eval/v1",
    api_key="<your key>",
)

criterion = "Which queue does the ticket belong in?"  # 1
subject = "It arrived crushed. I want a refund."  # 2
messages = [
    {"role": "system", "content": criterion},
    {"role": "user", "content": subject},
]
queues = {"enum": ["billing", "shipping", "account"]}
route = {"type": "json_schema", "json_schema": {"name": "route",
    "schema": {"properties": {"queue": queues}}}}  # 3

resp = client.chat.completions.create(
    model="a8:95",  # 4: answer only at 95 or above
    messages=messages, response_format=route,
)
resp.choices[0].message.content  # '{"queue": "shipping"}'

# 5: it belonged in billing. Tell your assistant, over MCP:
#    "That ticket belongs in billing. Teach A8."

the needles

Teach it in one sentence.

a generative model

Fine-tuning

A labeled dataset, collected first
A training run to schedule
A copy of the model to host

One sentence

Tell your assistant what A8 got wrong
Approve the teach it sends over MCP
A new version fits in the background

A teaching is kept the moment you approve it. About a minute after your workspace goes quiet, A8 fits a new version in the background, and the next verdicts come from it. Until then, the current version answers.

Ask why it answered, and it cites the examples you taught that decided it. What you teach stays in your workspace and trains its model alone.

the method, assayed

We tested the method against trained human raters.

On essay quality, we compared the scoring approach A8 is built on with trained human raters and with a panel of eight LLM judges. Its rank correlation with the raters was 0.815. The strongest judge on the panel reached 0.813.

read qed-bench

u22a8 metric0.815
claude opus 4.70.813
claude sonnet 4.60.724
deepseek v3.20.708
llama 4 maverick0.687
gemma 3 27b0.659
mistral large 30.658
qwen3 32b0.653
claude haiku 4.50.637

Spearman rank correlation with trained human raters, essay quality (ASAP 2.0). Streaks start at zero.

at a glance

interfaces: OpenAI and Anthropic APIs, and MCP
minimum accuracy: 0–99, set per call
determinism: same request, same verdict
verdicts: scored against the evidence
learning: per workspace, from what you teach
explanation: the taught examples behind a verdict
access: preview

read the model card

The stone is ours. The needles are yours.

State the criterion and the grade it must clear. Correct what A8 gets wrong, and your workspace’s model learns from it. A8 is available for preview.

To build A8 into your stack or a tool, write to .