Gate a release on a pinned horizon

Your system moves to a new version of A8 when your tests say so. Pin the version you tested, run the suite on each new version as it is fitted, and move the pin when it passes.

§1Motivation

Your organization’s model moves forward with every fit, and with each reference model we publish. The a8 alias follows it. That suits a system that improves with every teach. A system whose verdicts reach users or a published result moves on a test instead: it serves a pinned horizon, and a new version reaches it through a gate.

The gate keeps two promises at once. A pinned version answers the same request with the same verdict while its system fingerprint holds, and the version that replaces it has passed the same suite.

§2Example

A content team checks support replies against its tone policy before they go out. The checker serves a pinned horizon from its configuration. Each night a job tests the newest ready version and moves the pin when it passes.

§2.1Step 1: serve the pin

Keep the identifier in configuration, and call A8 with it:

import os

from openai import OpenAI

client = OpenAI(
    base_url="https://api.u22a8.ai/eval/v1", api_key=os.environ["U22A8_API_KEY"]
)
PINNED = os.environ["A8_MODEL"]  # for example a8-1-1:95@20260715t140322z
reply = "Thanks for waiting. Your refund went out today."

resp = client.chat.completions.create(
    model=PINNED,
    messages=[
        {"role": "system", "content": "Is the reply polite?"},
        {"role": "user", "content": reply},
    ],
)

§2.2Step 2: find the version to test

GET /models lists every horizon of your organization’s model, newest first. status is ready for a version you can pin, and active marks the one a8 serves:

models = client.models.list()
ready = [
    m.id
    for m in models
    if (m.model_extra or {}).get("status") == "ready"
]
candidate = ready[0]  # the newest ready horizon

§2.3Step 3: run the suite and move the pin

Run your test on the candidate. Move the pin when, at every served level, the share of right answers reaches the level and coverage stays at or above your floor:

def passes(result) -> bool:
    accuracy_holds = all(
        result.right[level] / result.answered[level] >= level / 100
        for level in result.answered
    )
    return accuracy_holds and result.coverage >= COVERAGE_FLOOR


if candidate != PINNED and passes(run_suite(candidate)):
    promote(candidate)  # writes A8_MODEL in your configuration

run_suite, promote and COVERAGE_FLOOR are your own. Record the identifier with each result, so a published number names the version that produced it.

§3How the gate was chosen

• Accuracy per served level. The promise is per level, so the suite checks it per level. Give each level enough answers: on 30 answers, a 95 promise can show 93 and still hold.
• A coverage floor. A new version that answers less often holds its promise and still sends more work to people. Set the floor from today’s coverage on the same suite.
• New questions in the suite. A suite built from taught examples tests memory. Keep it apart from what you teach.

A correction we ship reaches a pinned version too, and moves the system_fingerprint on its responses. A moved fingerprint is the signal to re-run the suite on the pin itself.