Chat Completions

A8 answers the OpenAI Chat Completions API at POST https://api.u22a8.ai/eval/v1/chat/completions. Any client that takes a custom base URL calls it this way: the OpenAI SDKs, LangChain, LlamaIndex, the Vercel AI SDK, Pydantic AI and LiteLLM among them.

§1Request

A8 reads the fields below. The system message states the criterion, the user messages carry the subject, and the schema gives the verdict its shape. A8 answers the conversation’s latest user message:

The subject can carry a context, a reference or a question beside the output it holds, and one schema can carry several criteria. Both work the same way in every interface. See API overview.

To follow up on a verdict, send the conversation again with A8’s response as an assistant message and a new user message after it. See Continue a conversation.

§1.1Prompts written for other judges

Eval frameworks often put the criterion and the subject in one message and ask for a format such as GRADE: C or a trailing Y/N. With reasoning on, A8 answers that prompt too: it finds the criterion, the subject and the format, and answers in the format the prompt asks for.

A request that is only conversation, such as small talk, comes back as an abstention of kind no_question.

§2What is refused

Each of these returns 400, with an error that names the field and the value that passes:

§3Minimum accuracy

The minimum accuracy is the accuracy A8 promises on the verdicts it returns. At 95, at least 95% of answered verdicts are correct. The model’s tag sets it: a8:95 asks for 95, and a pinned identifier carries it too, as a8-1-1:95@20260715t140322z.

• The tag is a whole number from 0 to 99. A higher level answers less often and is right more often.
• By default, the minimum accuracy is 90.
• The response’s model names the level that answered, so a stored response says which promise it was answered under.

The tag alone sets the minimum accuracy. “Only answer if you are sure” in a prompt counts as part of the criterion. The identifier’s parts are on Model identifiers, and what the minimum accuracy promises, with the levels a criterion earns, is on Accuracy and abstention.

§4Reasoning

reasoning_effort turns A8’s reasoning on and off. Reasoning is on unless the request sends none.

• Reasoning on reads the request however it is laid out: a framework’s grader template, a follow-up, or a request in plain words. When the layout already says what each part is, A8 answers as fast as with reasoning off.
• Reasoning off (reasoning_effort: "none") reads the layout as written, which is faster:

A framework’s grader template sent with reasoning off is read as written too: its system message, when it has one, is the criterion, and the template in the user message is the subject, so an untaught criterion abstains. A template sent as one user message alone abstains with kind no_question, since the criterion travels in a system message.

The reading is reported as reasoning tokens, in usage.completion_tokens_details.reasoning_tokens and inside usage.completion_tokens. A repeat of the request reads from the cache and reports the same count as cached input, in usage.prompt_tokens_details.cached_tokens and inside usage.prompt_tokens, as this API reports a prompt it had cached. A layout read as written reports 0.

§5Response

A8 answers with a standard chat.completion. The verdict is the message content: a label, a number or JSON that matches the schema.

{
  "id": "eval-9f2c1d84ab30e5f7",
  "object": "chat.completion",
  "model": "a8-1-1:95@20260715t140322z", // the version and level that answered
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "{\"queue\": \"shipping\"}", // the verdict
        "refusal": null
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 812,
    "completion_tokens": 0,
    "total_tokens": 812,
    "prompt_tokens_details": {"cached_tokens": 0},
    "completion_tokens_details": {"reasoning_tokens": 0}
  },
  "system_fingerprint": "fit-1a2b3c4d5e6f7890.9e2a1b3c",
  "min_accuracy": 95 // the level this verdict is promised at
}

• model names the exact version that answered, with the minimum accuracy in its tag. Send it back to get the same verdict. See Horizons.
• usage.completion_tokens holds reasoning tokens alone, since the verdict is scored. They are 0 here, as the layout is read as written.
• With n above 1, A8 computes each choice on its own, and usage.prompt_tokens counts the input once per choice. The same request gets the same verdict, so every choice holds it: comparing them checks determinism.
• min_accuracy, abstention, interval and precedents are A8’s fields beside the standard ones. Standard clients carry them through. Each is defined on API overview.

A forced tool call is answered as that tool call, with the verdict as its arguments and finish_reason set to tool_calls.

The OpenAI SDKs keep A8’s fields on the response object:

resp.model_extra.get("min_accuracy")  # 95, or None on an abstention
resp.model_extra.get("abstention")  # set on an abstention

type A8Fields = { min_accuracy?: number; abstention?: { kind: string } };

const { min_accuracy, abstention } = resp as typeof resp & A8Fields;

§6An abstention

An abstention succeeds with 200. The message carries a refusal and its content is null, so a typed client reads it as it reads any refusal. The top-level abstention says which check the evidence failed, and at what level the same request is answered:

{
  "object": "chat.completion",
  "model": "a8-1-1:95@20260715t140322z",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null,
        "refusal": "Too close to call at min_accuracy 95; answered at 80."
      },
      "finish_reason": "stop"
    }
  ],
  "abstention": {
    "kind": "too_close",
    "asked": 95,
    "answerable_at": 80,
    "earned": 97,
    "missing": null
  }
}

Recorded from a8-1-1@20260715t140322z on 2026-10-22.

The refusal text is for people to read. Branch on abstention.kind. The kinds, and what clears each, are on Accuracy and abstention.

§7A filtered request

With reasoning on, A8 filters some requests whose text is harmful, such as hate speech, violence or self-harm. A filtered request gets an outcome of its own, apart from verdicts and abstentions. It succeeds with 200, as this API reports content its filters omit: finish_reason is "content_filter", and content and refusal are both null:

{
  "object": "chat.completion",
  "model": "a8-1-1:95@20260715t140322z",
  "choices": [
    {
      "index": 0,
      "message": {"role": "assistant", "content": null, "refusal": null},
      "finish_reason": "content_filter"
    }
  ],
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0,
    "prompt_tokens_details": {"cached_tokens": 0},
    "completion_tokens_details": {"reasoning_tokens": 0}
  }
}

A filtered request is free, and the same request is filtered every time it is sent. With n above 1, every choice is filtered.

§8Precedents and recall

Ask for precedents in the prompt (“quote precedents”), and the response carries a top-level precedents object: the taught examples behind the verdict. Reserve a precedents property in your schema, and they arrive inside the verdict, where a typed client reads them with the rest. Either way the key needs the models:explain permission. See API keys.

A question about what you taught is answered with recall, your taught examples:

{
  "model": "a8",
  "messages": [
    {"role": "user", "content": "What have I taught you about refunds?"}
  ]
}

The message content is the precedents object as JSON, and the same object sits at the top level of the response.

§9Streaming

With stream: true, the answer arrives as chat.completion.chunk events. The verdict comes whole, in one delta, and each choice’s finish chunk carries A8’s fields. With stream_options.include_usage, a usage chunk follows them. With n above 1, each choice arrives under its own index.

§10Example

A ticket router at a minimum accuracy of 95. The Python and TypeScript samples read the verdict into a typed object:

curl https://api.u22a8.ai/eval/v1/chat/completions \
  -H "Authorization: Bearer $U22A8_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "a8:95",
    "messages": [
      {"role": "system", "content": "Which queue does the ticket belong in?"},
      {"role": "user", "content": "It arrived crushed. I want a refund."}
    ],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "route",
        "schema": {
          "type": "object",
          "properties": {
            "queue": {"enum": ["billing", "shipping", "account"]}
          }
        }
      }
    }
  }'

import os
from typing import Literal

from openai import OpenAI
from pydantic import BaseModel

client = OpenAI(
    base_url="https://api.u22a8.ai/eval/v1",
    api_key=os.environ["U22A8_API_KEY"],
)


class Route(BaseModel):
    queue: Literal["billing", "shipping", "account"]


resp = client.chat.completions.parse(
    model="a8:95",  # minimum accuracy 95
    messages=[
        {"role": "system", "content": "Which queue does the ticket belong in?"},
        {"role": "user", "content": "It arrived crushed. I want a refund."},
    ],
    response_format=Route,
)
message = resp.choices[0].message
print(message.parsed)  # Route(queue='shipping'), or None on an abstention
print(message.refusal)  # None, or the abstention in words
print(resp.model)  # the version that answered

import OpenAI from "openai";
import { zodResponseFormat } from "openai/helpers/zod";
import { z } from "zod";

const client = new OpenAI({
  baseURL: "https://api.u22a8.ai/eval/v1",
  apiKey: process.env.U22A8_API_KEY,
});

const Route = z.object({ queue: z.enum(["billing", "shipping", "account"]) });

const resp = await client.chat.completions.parse({
  model: "a8:95", // minimum accuracy 95
  messages: [
    { role: "system", content: "Which queue does the ticket belong in?" },
    { role: "user", content: "It arrived crushed. I want a refund." },
  ],
  response_format: zodResponseFormat(Route, "route"),
});
const message = resp.choices[0].message;
console.log(message.parsed); // { queue: "shipping" }, or null on an abstention
console.log(resp.model); // the version that answered

{
  "id": "eval-9f2c1d84ab30e5f7",
  "object": "chat.completion",
  "created": 0,
  "model": "a8-1-1:95@20260715t140322z",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "{\"queue\": \"shipping\"}",
        "refusal": null
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 812,
    "completion_tokens": 0,
    "total_tokens": 812,
    "prompt_tokens_details": {"cached_tokens": 0},
    "completion_tokens_details": {"reasoning_tokens": 0}
  },
  "system_fingerprint": "fit-1a2b3c4d5e6f7890.9e2a1b3c",
  "min_accuracy": 95
}

Recorded from a8-1-1@20260715t140322z on 2026-10-22.

The minimum accuracy travels in model, which every SDK and framework passes through, so it is set wherever the model is.