A synthetic golden-set test: eight planted changes, five that must fail and three that must pass

What this example is

Everything here is invented: a step that reads short order notes and returns a quantity, a unit and a date, with floors, changes and verdicts that are a specification for a test, not results from a real step. No real provider is involved. It shows the shape of a good test, not that any step passes it.

The setup

A golden set of 100 order notes has labelled answers. Grading is exact match on each field and on the whole record, in code, with no model grading. Three floors apply, and the output shape must match.

• Whole-record floor: 90 of 100 rows fully right.
• Field floor: no field below 95 of 100.
• Shape: exactly three fields, with the right types.

Eight planted changes and the verdict each must get

Five changes make the step worse and must fail the test. Three change it harmlessly and must pass. In a real job the buyer chooses two of the five and one of the three, unseen by the builder. If the test fails a harmless change or passes a harmful one, the test is wrong, not the step.

• 1 A prompt edit drops the unit field: must fail (shape).
• 2 A model swap lowers whole-record rows to 84 of 100: must fail (whole-record floor).
• 3 A date format changes from day-month to month-day: must fail (date field floor).
• 4 A sign flip turns returns into sales: must fail (quantity field floor).
• 5 Output is truncated after the second field: must fail (shape).
• 6 A harmless rewording of the instruction text: must pass.
• 7 A harmless re-ordering of the three output fields: must pass.
• 8 A harmless extra space in the instruction text: must pass.

Repeatability and cost

Run your unchanged step five times live, on your own key, and expect the same verdict every time or a floor set below the spread, because a test that flips on identical input cannot be trusted. A 2023 study reports that repeated runs and small rewording can change model labels, which is why grading is exact and the repeat check is part of the test. State the cost of one run and keep it under an agreed cap.

What the test does not do

It does not say the step is accurate on future inputs, only that it has not got worse on this set. It does not improve the step. Fresh rows from live use should be added to the golden set over time.