A NoMethodError expectation can be satisfied by the method's absence: the mechanism, measured — and zero such passes in today's counted total #850

Activity

1. tannevaled commented on Oct 31, 2026

   

   tannevaled

   on Oct 31, 2026

   ContributorAuthor

   More actions

   ⛔ Correcting this issue's title and its headline claim

   I wrote "The gate counts at least one pass that comes from a method being absent". It does not. The one instance I had measured — core/array/rfind_spec.rb — scores pass=1 only with the shim at 4.0.9, which is not what the gate runs. At 3.4.1 the whole file is behind its ruby_version_is "4.0" guard, and BASELINE records it as:

   OK	core/array/rfind_spec.rb	0
   

   Zero. So the mechanism is real and demonstrated, but the claim that today's 23,511 includes such a pass was unfounded, and I should not have put it in the title. Retitled.

   The tool now exists, and it answers the question properly

   scripts/conformance/rubyspec/cmd/phantom, run against the pinned corpus and a main binary (3a3fb81):

   $ go run ./scripts/conformance/rubyspec/cmd/phantom -root <corpus> -rbgo <main's rbgo>
   59 NoMethodError expectation(s) over 2206 spec file(s); 1 name a method this interpreter does not define
   population: describe strings of the form "X#y" or "X.y" only — operator methods and shared blocks are outside it, and an example whose receiver differs from its describe is invisible here
   core/array/rfind_spec.rb:45	Array#rfind is not defined — a pass here may be the absence, not the behaviour

   2206 spec files — the same population the gate sums — and exactly one candidate, the one #849 implements. So: zero phantom passes in the counted total, and the single candidate becomes a real pass when #849 lands.

   ⛔ The tool's first run was wrong, and how it was wrong is worth recording

   It reported three, the other two being Complex#negative? and Complex#positive?. Both are false positives, and reading one of them settled it:

   describe "Complex#negative?" do
     it "is undefined" do
       c = Complex(1)
   
       c.methods.should_not.include?(:negative?)
   
       -> {
         c.negative?
       }.should.raise(NoMethodError)
     end
   end

   MRI does not define Complex#negative? — Complex has no ordering — and the spec asserts that absence on purpose. rbgo passing it is conformance. Reporting it would have sent a reader to implement a method Ruby deliberately lacks.

   So "the describe names a method the interpreter does not define" is not sufficient. The rule now withdraws any describe block that also asserts the absence (should_not.include?, should_not respond_to, respond_to?(…).should == false) — by the example's own assertion rather than by a list of file paths, which would have gone stale at the next corpus bump and would not have covered the next such spec. That withdrawal took the population from 82 attributable expectations to 59: 23 of them are specs deliberately asserting a method is absent, which is a useful number in its own right.

   Both false positives are now test cases, verbatim.

   What the tool still cannot see, and says so on every run

   • It reads the describe string, so an example whose receiver differs from what its describe names is outside the population.
   • A describe naming an operator method ([], <=>, +) is skipped, because those do not match the identifier shape.
   • Attribution stops at a sibling or outer describe, judged by indentation rather than by counting do/end. The failure that risks is a miss, never a mis-attribution — which is the right direction for a tool whose output sends someone to read a spec — so its count is a lower bound.

   It carries its describe-pattern control inside the binary, run on every invocation, because CI runs it as a binary and never runs its tests; and it refuses (exit 2) if the interpreter says Array#find is missing or says a method that cannot exist is present, since either would make every verdict meaningless.

   Why keep it

   The mechanism does not need an adversary and will matter at the version move: as the 4.0 guards open, 52 spec files arrive carrying whatever expectations they contain, and UPDATE_BASELINE=1 banks what it sees. This is the check that says whether a newly-counted pass is behaviour or absence — and it now has a measured baseline of 1 candidate, 0 counted, against which any future run can be compared.

   🤖 Generated with Claude Code
2. tannevaled

   changed the title [-]The gate counts at least one pass that comes from a method being absent: a NoMethodError expectation satisfied by the wrong NoMethodError[/-] [+]A NoMethodError expectation can be satisfied by the method's absence: the mechanism, measured — and zero such passes in today's counted total[/+] on Oct 31, 2026
3. tannevaled

   added a commit that references this issue on Nov 1, 2026

   conformance: cmd/phantom — is a pass the behaviour, or the method's a…

   ...

   80d802c
4. tannevaled

   closed this as completedin #851on Nov 1, 2026