Who this is for
Developer or data engineer with a live model step for invoices, tagging or categories, built in n8n, Make or Python.
A prompt tweak or a provider model update changed the output and nobody noticed for weeks.
The result
Your step gets a 100-row labelled golden set and a one-command test that fails when whole-record accuracy drops below the agreed floor or the output shape changes. The test is shown to fail on 5 planted regressions and pass 3 harmless changes, and it never uses a model as the grader.
What is included
- One live model step that produces structured output
- A golden set of 100 rows, labelled with you
- A harness that runs on your own provider key or on recorded outputs
- A thresholds file holding the whole-record floor, field floors and the shape check
- A proof run: 5 planted regressions and 3 benign changes, and the cost per run
You receive
- The golden set
- The harness script and one-command run
- The thresholds file
- The planted-regression results
- Run steps with the cost per run
What is not included
- Running your production step or touching production data
- Holding your provider key
- Improving the prompt or model
- Ongoing monitoring, which is the monthly service
- A model as the grader
- Detecting a drop of a few points: 100 rows catch large drops, a changed output shape and flipped cases, not small average changes
What we need from you first
- What the step does and its output fields, as text
- Which tool it runs in (n8n, Make, Python) and how often it runs
- No real records, personal data or credentials in the first enquiry
Never send passwords, keys, customer records or confidential code in the first enquiry. Secure handover is agreed after scoping.
How we check it is done
- The test fails on each of 5 planted regressions (a prompt edit dropping a field, a model swap, a date-format change, a sign flip, truncation) and passes 3 benign changes. Two of the five regressions and one of the three benign changes are made by you and kept from us until the test run; the other three regressions and two benign changes are ours, so they show the harness works, not that it catches every real regression.
- Five live runs of your unchanged step on your own key give the same verdict, or the report shows the spread and sets the floor below it. If you run on recorded outputs only, this check is not run and the report says so.
- Grading is by exact match and field checks in code; no model grades any output.
- Cost per run is stated and under the cap agreed in writing.
- The shape check fails on a planted output with a missing field and on one with an extra field.
Your developer runs the harness, inspects the proof table and signs off in writing. Payment follows sign-off. The test covers the golden set; it is not proof about future inputs.
When we would stop or decline
- No ground truth can be set for the step
- The golden rows hold personal data that cannot be redacted
- You want access to production
- Two labellers agree on fewer than 43 of 50 rows
Questions
Why not let a model grade the output?
Models can get graded wrong too. Exact-match grading is repeatable, which is what a test needs.
Do you need my provider key?
No. You run the harness with your own key, or on recorded outputs.
Does this make my step more accurate?
No. It tells you when it gets worse.
Will it catch a small drop in accuracy?
Not reliably. 100 rows catch large drops, a changed output shape and cases that flip. A drop of a few points that stays above the floor can pass. Catching small drops needs a much larger labelled set.
Price and terms
£249 · untested offer price. £249 after the agreed checks pass and you sign off. No payment before sign-off.
This is a new service with no published client results. The price is a starting point we have not yet tested with buyers. Nothing is ordered or charged by the enquiry. The full specification is on the Synthetic Industry catalogue.