[tannevaled]

conformance: cmd/phantom — is a pass the behaviour, or the method's absence?

A spec that asserts an exception class cannot tell why it got one. #850, found
while accounting for a single unexplained pass=1:

# core/array/rfind_spec.rb
it "raises a NoMethodError if the ifnone argument does not respond to #call …" do
  -> { [1, 2, 3].rfind(42) { |x| false } }.should.raise(NoMethodError)
end

means "42 is not callable". Where Array#rfind does not exist, the call raises
NoMethodError for that reason instead and the example passes — a pass for a
method the interpreter does not have. Nothing distinguished that from
conformance.

What it reports, measured on main (3a3fb81)

$ go run ./scripts/conformance/rubyspec/cmd/phantom -root <corpus> -rbgo <rbgo>
59 NoMethodError expectation(s) over 2206 spec file(s); 1 name a method this interpreter does not define
population: describe strings of the form "X#y" or "X.y" only — operator methods and shared blocks are outside it, and an example whose receiver differs from its describe is invisible here
core/array/rfind_spec.rb:45	Array#rfind is not defined — a pass here may be the absence, not the behaviour

2206 spec files, the same population BASELINE sums. One candidate, the one
#849 implements — and BASELINE records that file as 0, so no counted pass
in today's 23,511 is bought by an absence. The point of the tool is that this
can now be re-asserted rather than hoped.

⛔ Its first run was wrong, and that is why the rule is what it is

It reported three. The other two:

describe "Complex#negative?" do
  it "is undefined" do
    c = Complex(1)
    c.methods.should_not.include?(:negative?)
    -> { c.negative? }.should.raise(NoMethodError)
  end
end

MRI does not define Complex#negative? — Complex has no ordering — and the spec
asserts that absence on purpose. rbgo passing it is conformance, and
reporting it would have sent a reader to implement a method Ruby deliberately
lacks. core/complex/positive_spec.rb is identical in shape.

So "the describe names a method the interpreter lacks" is not sufficient. A
describe block that also asserts the absence is withdrawn — by the example's
own assertion (should_not.include?, should_not respond_to,
respond_to?(…).should == false), not by a list of paths, which would have gone
stale at the next corpus bump and would not cover the next such spec. That took
the population from 82 attributable expectations to 59: 23 specs deliberately
assert a method is absent, which is worth knowing on its own.

Both false positives are test cases now, verbatim.

Attribution, and which way it is allowed to be wrong

Sites are attributed to the most recent describe naming a method. A describe it
cannot attribute ends that attribution only when it is a sibling or outer
block, judged by indentation — a describe nested inside the method's own is a
sub-grouping of the same method and attribution continues through it, which is
how ruby/spec is usually written.

Indentation is a convention, not a parse, and it is used instead of counting
do/end because the failure it risks is a miss, never a mis-attribution.
For a tool whose output sends someone to go read a spec, a false positive costs
more than a gap — so the count it prints is a lower bound, and it says so on
every run. The first implementation lacked this and attributed a shared block's
expectation to the preceding String#squeeze; its own test predicted that and
caught it.

Controls

Two, and both run in the binary rather than only under go test, because CI
runs this as a binary and never runs its tests:

• the describe-pattern control (ten lines it will meet, five that must match
  and five that must not — a constant, a shared block, an operator method, an
  it line, a bare constant name). A pattern that silently stops matching
  reports an empty census, which looks exactly like a clean corpus.
• the interpreter control, on the live corpus before any verdict:
  Array#find must come back present and a method that cannot exist must come
  back absent. Without the second half, an ask that answered "present" for
  everything would report a clean census — the most reassuring possible way for
  this tool to be broken.

Exit 2 for either, and for an unreadable corpus, an interpreter that will not
run, or a question that came back unanswered — an unanswered question would
otherwise read as "absent".

One interpreter process answers the whole census rather than one per site.

On the flag package

The fleet's rule is that command-line tools use cobra. This one uses flag,
deliberately and consistently with its two siblings cmd/ratchet and
cmd/guards in the same directory: cobra is not a direct dependency of this
module (only pflag, indirectly), and adding one to the interpreter's go.mod for
a conformance script would enlarge the dependency graph of every consumer of a
repository whose claim is pure Go and minimal. If the three should be converted,
that is one change and worth doing as one — say so and I will.

README: scripts/conformance/rubyspec/README.md gains a section, including the
measured line above as the baseline any future run is compared against, and the
Caveats section now names both directions the total can be wrong in — the stubbed
matchers make it too low, a pass bought by an absence would make it too high, and
neither is visible in the number.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

Closes #850.

🤖 Generated with Claude Code