How we test an automation that has a model call in it

Splitting the deterministic steps from the judgement step

March gave us two papers worth arguing with. They came at the same problem from different ends: whether a model can be trusted to grade another model's output, and what a single successful run tells you about the next hundred. In the same month OpenAI shipped a new model and shut three older ones down.

Both land on one practical question for anyone running a document intake workflow: how you would know whether yours still worked after a month like that. What follows is how we answer it, and the answer is less exotic than the subject matter suggests.

The point we keep coming back to is that an automation with a language model in it is still mostly ordinary software. A document intake workflow polls a mailbox, downloads attachments, calls a model once to read each document and return structured fields, validates those fields in Python, then writes a record through the target system's API. Only the model call is non-deterministic, and every other step in that list is tested the way any other Python code is tested.

The first thing we write is a table in the repository listing every step, its classification as deterministic or judgement, and its output contract. A deterministic step gets unit tests with fixed inputs and exact assertions. A judgement step gets a graded evaluation against a dataset. Where one function does both, we split it, because the pass criteria differ. A deterministic step either passes or fails. A judgement step produces a score across a set of cases, and that score is compared against a threshold.

Microsoft's 2026 release wave 1 plan, published on 18 March and covering April to September 2026, names evaluations alongside deeper governance and multi-agent orchestration as what enables Copilot Studio agents to scale further. We think that is the right list. Our step table is what tells us which steps in a given workflow need that kind of evaluation.

Testing everything around the model with the model switched off

Most of what breaks in these systems has nothing to do with the model: pagination on the source API, an expired OAuth token, a date parsed in US order, a duplicate record created because a retry ran twice. All of it is testable without a single model call, provided the model call is made through an interface the test suite can replace.

In Python we make that interface a small protocol class, Extractor, with one method that takes bytes and returns a dictionary. The production wiring injects the implementation that calls the model API. The test suite injects a stub that reads a fixture dictionary from JSON on disk. Validation, deduplication, retry logic, error handling and the API writes then all run offline in milliseconds and need no API key.

vcrpy covers the HTTP calls on either side of the model. It records real request and response pairs once against a sandbox tenant, writes them to a cassette file, and replays them on every later run. With the responses library, the stub responses are declared in the test itself, and capturing real traffic requires its recorder module. We prefer recorded traffic to hand-written fixtures, and the reason is specific. A hand-written fixture matches the vendor documentation, and the documentation is frequently inaccurate about which fields are optional and how nulls are returned, so the fixture inherits those errors and the test passes against a response shape the API does not send.

What goes in the golden dataset

The judgement step needs real inputs with the correct output written down by someone who knows the process. We keep ours in the repository beside the code, one directory per case holding the input file and an expected.json. The set covers the common document layouts that make up most of the volume, and it also covers the cases the business has raised as problems, such as an invoice number printed in the footer rather than the header, and a credit note laid out on the same template as an invoice.

Grading runs per field, because a document-level pass rate does not show you which field failed. Invoice number and ABN compare as exact strings after normalisation, dates get parsed on both sides and compared as date objects, and amounts compare as decimals. Free text fields cannot be compared that way, which is where a second model is often used as a judge.

The paper posted to arXiv on 5 March 2026 by a RAND team, Judge Reliability Harness, is worth reading before you go down that road. Their open source library perturbs the responses a judge grades by reformatting, paraphrasing, adding verbosity and flipping the ground truth label, then measures how far the verdict moves. Across four judges and four benchmarks they found no judge that was uniformly reliable. How far that result carries over to the narrow field-level grading we do is unsettled, and our position is that the burden sits with anyone proposing the judge. We keep as many fields as possible on exact comparison, and any field that is graded by a model gets its own calibration set of cases where a person has already agreed with the verdict.

Confidence scores, and why we run every case more than once

We ask the model for a confidence value for each field alongside the extracted value. Self-reported confidence is not calibrated against real accuracy, and we read it as a sorting signal and nothing stronger. We set the threshold empirically: run the golden set, record confidence and correctness for every field, then pick the cut-off where the error rate below the line is acceptable for the process. Items below the threshold are held for a person to approve, and their corrections go back into the dataset.

Each case runs more than once, because temperature zero reduces variation without removing it and a change in the provider's serving stack can move outputs on identical inputs. We run the golden set three times and assert on the worst of the three runs, which gives a number closer to what production sees.

The paper Beyond pass@1, posted to arXiv on 31 March 2026, makes that argument across an evaluation of ten models and nearly 24,000 episodes. The authors argue that single-attempt success measures capability, that production deployments depend on consistent success across repeated attempts, and that the two rankings diverge substantially at longer horizons. They propose four metrics for it, including a Reliability Decay Curve and a Variance Amplification Factor. We agree with the argument and we still do not report those metrics by name, because they have not yet been reproduced on work that looks like ours. We report a per-case pass rate across repeated runs next to the aggregate score.

What we do when the model version changes underneath us

We pin the model ID, including the dated snapshot where the provider publishes one, and treat a change to that string as a code change: a pull request, an evaluation run on both versions, a diff of the per-case results, and a decision recorded by a person.

Two model events in March 2026 are why we want the harness in place before it is needed. OpenAI released GPT-5.4 on 5 March in standard, Thinking and Pro variants, along with a tool search mechanism that changes how tool definitions reach the model. On 26 March, gpt-4-0314, gpt-4-1106-preview and gpt-4-0125-preview reached shutdown under OpenAI's published deprecation schedule, which commits to at least six months' notice for generally available models, at least three months for specialised variants, and as little as two weeks for preview models.

Two weeks of notice leaves little room to redesign a prompt, so two weeks is the number we plan against. The golden set in the repository does that work. A forced migration becomes a run of the suite on both versions and a review of every case whose verdict changed. We review the cases that improved as well as the ones that regressed, because a newer model that extracts a field more accurately can also return it in a different type or format, which the downstream Pydantic schema will reject. An amount field is the first place we check for that.

What the suite looks like in Python

The suite has four groups, all under pytest. Unit tests cover pure functions with fixed inputs, integration tests exercise the pipeline end to end against recorded HTTP and a stubbed extractor, and both groups run on every commit. Contract tests hit the real sandbox APIs nightly to check that responses still match the recorded shapes. Evaluation tests run the golden set against the live model, marked with pytest.mark.eval and excluded from the default selection so a local run costs nothing.

Pydantic models enforce the output shape at the boundary, so a malformed response raises a validation error at the point it arrives and the bad field never reaches the target system. The evaluation job writes a JSON report for each run holding per-field accuracy, per-case results, the model ID, a prompt hash and the dataset version. Prompt text is stored in files in the repository, and every report records the git commit it ran against. When results change six weeks later, the report and the commit together identify what was different.

The part we are least sure the industry has right is how large a golden set needs to be before the score means anything, and there is no established rule for it. We size ours by coverage of the layouts and by the cases the business has already raised, which is a basis we can defend in a room. Separating how much of a score comes from the prompt and how much comes from the model takes the same suite run against two providers on one dataset, and that run is a normal part of maintaining the harness.