Integrations

A8 in Inspect

Run A8 as an Inspect model, score each abstention as unanswered, and move a model_graded_qa grader to A8.

Inspect runs evaluation tasks: a dataset of samples, a solver that calls a model and a scorer that grades each answer. Its openai-api provider calls any service that answers the Chat Completions API, A8 among them.

§1Before you start

Create an API key with the models:infer permission, as API keys shows. The next section sets it as U22A8_API_KEY.

Shell

pip install "inspect-ai==0.3.283" openai

§2Configure the model

The openai-api provider takes a model name of the form openai-api/<service>/<model>. It reads the service’s key and base URL from environment variables named after the service. With u22a8 as the service:

Shell

export U22A8_API_KEY="<your key>"
export U22A8_BASE_URL="https://api.u22a8.ai/eval/v1"

The model is then openai-api/u22a8/ followed by an A8 identifier, with the minimum accuracy as its tag. Pin the version you test, so a re-run tests the same one.

Inspect setting	What it does for A8
`--model openai-api/u22a8/<identifier>`	Calls A8, at the version the identifier names and the minimum accuracy its tag names: `:95` is 95, none is 90
`GenerateConfig(reasoning_effort=...)`	The reasoning switch: `"none"` reads the layout as written
`system_message(...)`	The criterion
`Sample(input=...)`	The subject

Table 1. How Inspect’s settings reach A8. Inspect sends reasoning_effort to every openai-api model, so "none" turns reasoning off for every request of a task.

§3Judge one subject

A task with one sample and one solver:

Python

from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.scorer import match
from inspect_ai.solver import generate, system_message


@task
def refund():
    return Task(
        dataset=[Sample(input="We will refund you in full by Friday.",
                        target="yes")],
        solver=[
            system_message("Does the reply promise a refund? Yes or no."),
            generate(),
        ],
        scorer=match(),
    )

Shell

inspect eval refund.py --model openai-api/u22a8/a8-1-1@20260715t140322z

The sample’s output is the verdict, yes, in the format the criterion asks for.

§4Score an abstention as unanswered

An abstention reaches Inspect as a refusal: the first choice’s stop_details.type is "refusal", and its text is the refusal. Score it NOANSWER, so it lowers coverage and leaves accuracy as it is. A metric reads each score as a number, and NOANSWER and INCORRECT are both 0 there, so the scorer marks an abstention in the score’s metadata. Two metrics read the mark and report accuracy and coverage apart:

Python

from inspect_ai.scorer import (
    CORRECT,
    INCORRECT,
    NOANSWER,
    SampleScore,
    Score,
    Target,
    metric,
    scorer,
)
from inspect_ai.solver import TaskState


def unanswered(sample: SampleScore) -> bool:
    said = sample.score.metadata or {}
    return said.get("abstained", False) or said.get("filtered", False)


@metric
def answered_accuracy():
    def compute(scores: list[SampleScore]) -> float:
        answered = [s for s in scores if not unanswered(s)]
        return sum(s.score.as_float() for s in answered) / max(len(answered), 1)

    return compute


@metric
def coverage():
    def compute(scores: list[SampleScore]) -> float:
        answered = [s for s in scores if not unanswered(s)]
        return len(answered) / max(len(scores), 1)

    return compute


def refused(state: TaskState) -> bool:
    details = state.output.choices[0].stop_details
    return details is not None and details.type == "refusal"


@scorer(metrics=[answered_accuracy(), coverage()])
def a8_verdict():
    async def score(state: TaskState, target: Target) -> Score:
        if state.output.stop_reason == "content_filter":
            return Score(
                value=NOANSWER,
                explanation="filtered",
                metadata={"filtered": True},
            )
        if refused(state):
            return Score(
                value=NOANSWER,
                explanation="abstained",
                metadata={"abstained": True},
            )
        answer = state.output.completion.strip().lower()
        return Score(value=CORRECT if answer == target.text else INCORRECT)

    return score

answered_accuracy is the share of answered samples that came out right. coverage is the share of samples answered. A request A8 filtered has the stop_reason content_filter, and the scorer marks it filtered, unanswered as an abstention is. See Testing for why they stay apart.

§5Move a `model_graded_qa` grader

model_graded_qa grades each answer with a grader model. Pass A8 as that model, with its minimum accuracy as the identifier’s tag, and keep the template:

Python

from inspect_ai.model import get_model
from inspect_ai.scorer import model_graded_qa

# The tag asks for a minimum accuracy of 95.
grader = get_model("openai-api/u22a8/a8-1-1:95@20260715t140322z")
scorer = model_graded_qa(model=grader)

A8 answers in the template’s GRADE: C, GRADE: P and GRADE: I format.

On an abstention, the grader’s output is the refusal text, and Inspect records the sample as unscored with the reason grader_failed. Read an unscored sample as an abstention, or grade with the a8_verdict scorer to score abstentions directly.

§6A complete example

Python

# refund.py
from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.scorer import (
    CORRECT,
    INCORRECT,
    NOANSWER,
    SampleScore,
    Score,
    Target,
    metric,
    scorer,
)
from inspect_ai.solver import TaskState, generate, system_message

CRITERION = "Does the reply promise a refund? Yes or no."


def unanswered(sample: SampleScore) -> bool:
    said = sample.score.metadata or {}
    return said.get("abstained", False) or said.get("filtered", False)


@metric
def answered_accuracy():
    def compute(scores: list[SampleScore]) -> float:
        answered = [s for s in scores if not unanswered(s)]
        return sum(s.score.as_float() for s in answered) / max(len(answered), 1)

    return compute


@metric
def coverage():
    def compute(scores: list[SampleScore]) -> float:
        answered = [s for s in scores if not unanswered(s)]
        return len(answered) / max(len(scores), 1)

    return compute


@scorer(metrics=[answered_accuracy(), coverage()])
def a8_verdict():
    async def score(state: TaskState, target: Target) -> Score:
        if state.output.stop_reason == "content_filter":
            return Score(
                value=NOANSWER,
                explanation="filtered",
                metadata={"filtered": True},
            )
        details = state.output.choices[0].stop_details
        if details is not None and details.type == "refusal":
            return Score(
                value=NOANSWER,
                explanation="abstained",
                metadata={"abstained": True},
            )
        answer = state.output.completion.strip().lower()
        return Score(value=CORRECT if answer == target.text else INCORRECT)

    return score


@task
def refund():
    return Task(
        dataset=[
            Sample(input="We will refund you in full by Friday.", target="yes"),
            Sample(input="We will sort it out on Monday.", target="no"),
        ],
        solver=[system_message(CRITERION), generate()],
        scorer=a8_verdict(),
    )

Shell

inspect eval refund.py --model openai-api/u22a8/a8-1-1:95@20260715t140322z

Keep the samples apart from what you teach A8, so a passing run measures what it learned. See Testing.

Next steps

Testing A8Test what A8 learned the way you would test a student, with new questions, known answers, accuracy and coverage, on a pinned version.A8 in promptfooRun A8 as a promptfoo provider, score each abstention as unanswered, and move an llm-rubric grader to A8.Accuracy and abstentionHow a request sets the minimum accuracy a verdict must meet, what that level promises and the five kinds of abstention.

§1Before you start

§2Configure the model

§3Judge one subject

§4Score an abstention as unanswered

§5Move a model_graded_qa grader

§6A complete example

Next steps

§5Move a `model_graded_qa` grader