[tannevaled]

cmd/ratchet judges one sweep against BASELINE. That is the right gate for
"may this land" and the wrong instrument for "what does this change do": a
baseline comparison folds in every difference between the recording machine and
this one. The 4.0 version move in #799 needed the other reading — hold the
corpus and the machine fixed, vary only the interpreter, and name every file
that moved.

$ specdelta -root ~/ruby-spec-corpus -dir core/set -quiet -repeat 3 ./rbgo-main ./rbgo-409
file                                        control   treat  delta
core/set/flatten_merge_spec.rb                    3       0     -3
core/set/to_s_spec.rb                             2       4     +2
core/set/divide_spec.rb                           9       8     -1
core/set/equal_value_spec.rb                      4       3     -1
...
57 file(s): control 175 passing, treatment 167 passing, -8
9 file(s) moved, 8 regressed

That table took the Set half of the version move from "eight files to work
through" to two defects: six of the eight are examples 4.0 stopped asking
for, which the deltas cannot say and the ruby_version_is guards can. The same
reading cut the whole move's remaining work from "−28 across 13 files" to ten
examples, and corrected its benefit from a recorded +54 to a measured
+91 — the old figure was taken against a treatment seven merged fixes behind
its control.

⛔ -repeat N, because one run cannot tell a flake from a difference

The tool's first real use reported

core/random/bytes_spec.rb      9       8     -1

between two binaries differing by one file and three constants. Three runs
of each:

	run 1	run 2	run 3
control	9	9	8
treatment	8	9	9

A coin flip, attributed to the change. That file carries no version guard at
all, which is what made the row suspicious enough to re-run — a row that did
sit behind a guard would have looked explained and gone into a report.

Above -repeat 1 each file runs N times per engine, the lowest count is
kept — not the mean and not the last, because the question is "did anything get
worse" and the answer must not rest on a lucky run — and an engine that
disagrees with itself marks the file UNSTABLE and has its delta excluded from
the verdict. The default stays 1, and the flag's help says plainly that the
regression verdict is only trustworthy above it.

Two channels, and the invariant I got wrong first

RBGO_RESULT pass=… comes out of the subject's own stdout (#843), so a
subject that miscounts can claim anything. The RBGO_DETAIL lines are the
independent channel.

My first version asserted details == fail + error and fired on its first real
run against core/set/set_spec.rb — pass=0 fail=0 error=0 skip=1 with one
skip detail, entirely well-formed. Reading spec_helper.rb rather than
assuming its shape gives five emission sites, two of which break any equality: a
guard-driven skip bumps its counter with no detail (:970), and a
before(:all) failure pushes a detail counted in no tally at all (:965,
:1020). What holds is details >= fail + error, and that last row is a hole in
the number the ratchet sums — filed as #865.

Refusals are exit 2, never 1

An engine that would not start, an unreadable RBGO_RESULT line, two result
lines in one run, and an empty population — a mistyped -dir would
otherwise print 0 files, no regressions and exit 0, which is the most
convincing possible way for this tool to tell someone their change is safe. A
file that produced no result line is named rather than counted as zero, which
is the distinction that made an endless range look like a load failure in #853.

Tests

parse first and hardest, since that is where a silent wrong answer would live:
seven malformed result lines that must each be an error, the no-result case that
must not be, the detail-shortfall table naming every shim emission site, the
empty-population refusal, four usage refusals, and end-to-end passes over fake
/bin/sh engines — an improvement exits 0 and the same pair reversed exits 1.
The instability test asserts both modes, so the difference between them is
visible rather than claimed.

The engines are shell scripts, not Ruby, because the tool knows nothing about
Ruby and a test needing an interpreter installed would measure the machine.

⛔ Two of those tests failed first time and the tool was right both times: a
"good" result line written as fail=3 error=1 with three detail lines (four
expected), and stderr 0 vs 26 bytes where warning: something\n is 19.

README: a section beside cmd/oracle's, with the core/set table as the worked
example.

🤖 Generated with Claude Code