Statistical Gating for Agent Instruction Changes

2026-08-09 · 7 min read · by Michael Kaminski

Every edit to an agent's instruction file is an experiment. Mine requires a statistically significant improvement — Welch's t-test, p < 0.10, at least a 5% lift — against a 14-day rolling baseline before the change is allowed to stay. Then the power calculation showed the 14-day window can only detect a 0.97 standard-deviation shift, which makes the 5% threshold decorative.

An edit to an agent's instruction file is a deploy, and I stopped letting mine ship on a hunch. The daily tuning job that maintains my assistant's CLAUDE.md now requires a statistically significant improvement — Welch's t-test, p < 0.10, at least a 5% lift — measured against a 14-day rolling baseline, before an instruction change is allowed to stay.

Then I ran the power calculation on my own gate and found the 5% threshold is decorative. At 14 days per window the test can only detect a shift of roughly 0.97 standard deviations. Anything smaller is invisible regardless of what the lift threshold claims.

Both halves of that are this post. The mechanism is worth copying. The window size is the part I got wrong, and the arithmetic is short enough to check.

The prompt is the least-tested code in most agent systems

Every other artifact in an agent pipeline has a gate. Application code gets a test suite and a review. Infrastructure gets a plan and a diff — I have written enough Terraform modules and managed enough remote state to know nobody merges those blind.

The instruction file gets none of that. Someone notices the agent did something annoying, adds a line telling it not to, and ships. There is no baseline, no holdout, and no record of whether the previous eleven lines are still earning their tokens.

That is how instruction files rot. They accumulate rules that were true about one bad afternoon and have been costing context ever since.

You cannot test what you do not score

The gate needs a dependent variable, so the first build was a rubric, not a test. Five dimensions, each scored 0–10 per conversation: goal clarity, rework rate, context hit rate, scope discipline, and response density. The daily score is the average across that day's conversations, weighted by conversation length.

The rubric is frozen. Changing it invalidates every historical comparison, so a rubric change is itself a tracked meta-edit that resets all baselines.

Scores are anchored to observable friction rather than self-assessment. The scorer parses transcripts for five pattern classes — corrections ("no, that", "actually", "don't"), rework ("redo", "start over"), scope drift ("I didn't ask", "just do"), missed context ("I told you before", "check memory"), and praise ("exactly", "nailed it"). Friction pulls the dimension score down, praise pulls it up.

This is the load-bearing decision. An agent grading its own transcripts will drift toward generosity. Regex over the human's actual words will not.

Welch, not Student, and why that is not pedantry

Each edit gets a 14-day pre-window and a 14-day post-window of daily scores, compared with Welch's t-test. Three outcomes: p < 0.10 and lift ≥ 5% marks the edit KEPT; p < 0.10 and lift ≤ −5% triggers an automatic revert; everything else is INCONCLUSIVE and the edit stays on probation.

Welch rather than Student's t because the two windows should not have equal variance. A good instruction usually works by removing a failure mode, which compresses the bad tail — the variance drops as much as the mean rises.

Student's t assumes equal variance and over-rejects when the smaller-variance group is the larger sample. That is precisely the case you care about, so the pooled test would hand you your most confident false positives on your best edits. Welch costs a few degrees of freedom and removes the assumption.

The arithmetic that broke the design

A two-sample test at α = 0.10 and 80% power detects a minimum effect of roughly (t_α/2 + t_β) × √(2/n) standard deviations. For a 14-day window that is:

Now convert to the units the gate actually uses. Assume a 7.0 baseline on the 0–10 rubric and a day-to-day standard deviation of 0.6 points, which is unremarkable for a metric averaged over a handful of conversations.

A 5% lift is 0.35 points, or 0.58 SD. The 14-day window detects 0.97 SD, which is 0.58 points — 8.3% of baseline. The statistical test is roughly 1.7× stricter than the lift threshold sitting next to it.

So the 5% number never binds. It is doing no work. Every edit that clears the t-test has already cleared 5% by a wide margin, and every edit that fails does so on power, not on effect size.

Detecting a genuine 5% lift at that variance takes 38 days per window, not 14. And the requirement moves fast with variance:

Read the last column as the honest label on the gate. At SD 1.0 a "5% threshold" is really a 14% threshold, and the difference is entirely hidden from whoever reads the config file.

The window has to be sized to the variance, not to a calendar. Fourteen days was chosen because two weeks is a tidy number, which is not a reason.

There are two ways out and only one of them is cheap. Lengthen the window, and you wait longer per lesson. Or shrink the variance by scoring more conversations per day — the daily mean's standard error falls with √n, so tripling daily volume cuts the SD by about 42% and pulls the required window from 38 days down to roughly 17.

Put the same arithmetic the other way around. If an edit truly delivers a 5% lift, a 14-day window calls it significant 44% of the time. That is a coin flip on your own best changes. At 38 days it is 81%.

What the gate cannot do, stated plainly

It does not control for multiple comparisons. At the configured cadence of one low-risk edit every three days, that is about 122 evaluations a year. At α = 0.10 two-sided, noise alone produces roughly six spurious KEPT verdicts and six spurious auto-reverts per year. The auto-revert side is the one that stings: the system will occasionally roll back a good edit with statistical confidence.

It is also not a randomized experiment. Pre and post windows are consecutive calendar time, so a model version change, a vacation, or a month of unusually messy work lands entirely in one window and gets attributed to the edit.

That makes this a noise filter, not a causal claim. It stops the obviously-worse edits and the obviously-imaginary wins. It will not tell you why anything moved.

The safeguards do more work than the test

Three rules keep the loop from eating itself, and they matter more than the p-value.

Cooldowns. One low-risk auto-edit per three days, one high-risk proposal per week. Without a cooldown the windows overlap so badly that no edit is ever cleanly attributable.

Plateau detection. If the 14-day rolling score has not improved 2% over the prior 14 days and no friction is firing, the day's edit is skipped. The easy wins arrive early; after that, editing is mostly a way to add variance.

A risk split with a human in it. Wording and formatting changes auto-apply. Anything that adds a section or changes how the agent decides gets written to a proposals folder and waits. Every edit backs up the prior file first and lands in a revertible log, tagged, so a bad call is a one-line rollback rather than an archaeology project.

The config file has a section titled "when NOT to edit," and the last line of it is the most useful thing in the whole system: doing nothing is always a valid action.

The cost

Latency. That is the whole bill, and it is larger than I estimated when I built this.

A gated instruction file learns on a 14-day clock at best and, if the variance numbers above are right, a 38-day clock in practice. An ungated one learns in an afternoon and is wrong in ways nobody measures. I would still take the slow version, but I would not pretend the tradeoff is free, and I would not build this at all for a system I was still prototyping.

I have no KEPT verdicts to report yet, because the first honest one cannot exist until a full post-window closes. Publishing the design before the results is the point — the design is the part that is checkable, and the arithmetic above is the part I would want someone to argue with.

If you are gating prompt or instruction changes statistically at your shop, I want to know what window you landed on and what your daily variance looks like. That number is the whole ballgame and almost nobody publishes it.