[tannevaled]

Every parity fix in this repository starts with "what does MRI actually do
here?", and the obvious ways to ask it throw part of the answer away:

• ruby probe.rb > out 2>&1 discards the exit status, which for half these
  probes is the answer — a SyntaxError is what they are looking for;
• a pipe into grep or tail discards whichever part did not match the pattern
  the asker already had in mind, which is the part worth reading;
• running the reference and rbgo as two separate commands invites comparing a
  fresh measurement against a remembered one.

cmd/oracle parses nothing: both streams are copied through verbatim with
their byte counts, the status is printed as a number, and the only judgement it
makes is byte equality between engines.

go run ./scripts/conformance/rubyspec/cmd/oracle \
  -script scripts/conformance/rubyspec/cmd/oracle/testdata/probes/globals.rb \
  ./rbgo-main ./rbgo-branch /usr/bin/ruby

Three engines is the shape the measurement rules want: reference, control
(main), treatment (the branch). A difference between the last two that the
reference does not share is the defect, or the fix.

⛔ The row that justifies comparing BYTES and not lengths

Measured while writing #856:

=== the engines DISAGREED ===
  /usr/bin/ruby vs ./rbgo-main:   stderr 1008 vs 407 bytes
  /usr/bin/ruby vs ./rbgo-branch: stderr 1008 vs 1008 bytes

The second row is the same length and different content — the branch had
gained the warning but still differed from MRI in the line number and the quote
style. A comparison by length, which is what reading two wc -c outputs amounts
to and what I had been doing by eye, would have called that parity.

-selftest plants exactly that case ('$x' versus '$y', equal length, one
byte apart) and refuses to pass unless the comparator sees it. It runs in the
binary as well as under go test, like cmd/phantom's controls, because a
comparator stuck on "agreed" would make every measurement taken with this tool
read as parity — the most reassuring possible way for it to be broken.

Exit status, and the distinction that is tested

0 agreed (or one engine named), 1 disagreed, 2 refused — no script, an
unreadable script, no engine, or an engine that would not start.

That last one is deliberate and has its own test: "could not ask" and
"answered differently" call for opposite responses — fix the invocation, or go
read the divergence — so a missing interpreter must never arrive as a
disagreement. Same split, and same exit codes, as cmd/ratchet. A probe that
never returns is killed at -timeout (default 1m) and reported as KILLED
rather than hanging the measurement; #853 was an endless range that never came
back, first recorded as "failed to load" because a timeout had killed it
silently.

The probes, kept

Four under testdata/probes, one question each, with a README recording what the
reference answered and which upstream function owns the rule. The shapes take a
minute to write and the reference run takes a second; what costs time is working
out which function in MRI decides — parse.y does not decide
constant-ness, rb_sym_constant_char_p does; rb_gvar_get does not warn, the
undef getter it installs does — and which rows separate the plausible-but-wrong
implementation from the right one.

probe	the row that earns its keep
globals.rb	silence after $x = nil — the subject is "never assigned", not "reads as nil"
constant_names.rb	the titlecase digraphs Dž/Lj are constants and the caseless letters 日/א/ب are not, so "not lowercase" is wrong in both directions
dynamic_constant.rb	Class.new { X = 1 } inside a def is refused while define_method(:m) { X = 1 } at class level is not — the rule is syntactic, not about what will run as a method
basicobject_conversion.rb	a BasicObject has no #respond_to?, which is not the same as having an overridden one

⛔ Three of them were written, lost to a reboot that emptied the scratchpad, and
written again. They belong in the repository, which is where the rule about
durable directories was pointing all along.

The README also records the two differences against the installed 2.6.10 that
are expected and not rbgo defects — the quote style (`$x' before 3.4,
'$x' after, which is why ruby/spec matches on [`']) and 2.6's one-line
lag in a warning's position — and says plainly what that interpreter cannot
answer at all: it predates numbered parameters, core Set and everything else
3.x–4.0 added, so a probe about those gets silence, which reads exactly like a
negative.

Tests

The fake engines are /bin/sh scripts, not Ruby, on purpose: the tool knows
nothing about Ruby, and a test needing an interpreter installed would measure
the machine instead of the tool. They cover the exit status surviving, the two
streams not being swapped (a single buffer could not tell you), the three
refusals, the kill path, and — as the control for the disagreement cases — two
engines that agree.

⛔ One of those tests failed first time and the tool was right: I had written
stderr 0 vs 26 bytes where warning: something\n is 19. The byte counts being
printed at all is what made that a one-line correction instead of a theory.

README: a section under cmd/guards, with the disagreement above as the worked
example.

🤖 Generated with Claude Code