[KyleAMathews]

Summary

The oracle-gaps CLI reported a production branch that exists only for speed as a not-observed gap. Its directive (observe the value, or widen the oracle) cannot work when the fast path and the general path give the same result. An author who followed the "remove code no law needs" advice deleted the shortcut and made the code slower. The maintainer ruled this a gap CLI bug (2026-11-12).

This PR gives such a survivor its own class, extra-work, and adds a cost law that can kill it:

• extra-work. A survivor with no changed observation, where a test whose work judges the mutant did more work. Work is the capture's count of site evaluations per test (calls), a deterministic count. Wall-clock time is never used.
• Directive. It names the test, its work before and after, the measured ratio (three decimals), and the bound to declare. The bound is the largest ratio in hundredths that is still below the measured one. The directive offers no exit. The legend line adds that a builtin call counts as one site whatever it costs, so a shortcut whose cost is in one (Object.defineProperty) is owed an observation of that cost in the driver, such as a count of those calls, judged by a law, in place of a very tight bound.
• oracleProperty({ work: { ratio } }). Under the gap CLI, a passing test whose work under a mutant is more than ratio times its unmutated work reports a verdict under law work bound on its route. On a reproducible route that kills the mutant.
• Selection fix (check.mjs). A selection that leaves out earlier tests is reused only when it reproduces each test's work as well as its observations. Before, a skipped test that warmed a cache hid a mutant's extra work.

Design

Part	Behavior
Counter	Reuses calls (one per mutation-site evaluation in a test). Call statements and calls whose value is used are sites, so an added call and a removed early return both move it. It does not see plain assignments, for...of passes, a builtin's own cost, or uninstrumented code.
Which tests judge work	The test passed, it reached the mutant (or its file loaded code that did), its observations and calls agree between the two unmutated runs, no run of it could have a count that follows the clock (row flag clocked), and its route is reproducible or random: a route that oracleProperty registers for production, which can carry a bound. A rise elsewhere leaves the mutant's class as it was.
Class order	not-judged, indeterminate, extra-work, then the static classes. A survivor whose work fell stays not-observed.
Clock (P3)	A kill must replay on an idle machine, so a test whose count could follow the clock judges no work (maintainer decision, 2026-11-12). The capture sets clocked when the test made a real timer or interval through the globals, node:timers, node:timers/promises, AbortSignal.timeout, or Atomics.wait or Atomics.waitAsync with a timeout; read a clock (Date.now, new Date() or Date() with no argument, performance.now, performance.eventLoopUtilization, process.hrtime, process.uptime, os.uptime, Temporal.Now); used monitorEventLoopDelay; or held I/O open. A read counts when any frame of its stack is the test's or production's, so a read through a vi.spyOn passthrough counts; a read whose every frame is the test runner's, fast-check's, the oracle framework's (its shrink budget), Node's, or the capture's does not, nor does one under a fake clock. A setImmediate turn reads no clock. Every wrapper keeps the own properties of what it wraps, util.promisify.custom included.
Kill	Compared exactly, in hundredths: 100 × calls > round(100 × ratio) × before. work bound failures join outcomeOf's failures in report order, so they rank as law verdicts do: killed on a reproducible route, random-only-kill on the random campaign. They outrank a timeout, a crash, and not-judged. The runner's stop-at-first-kill still reads only law verdicts.
Framework	work.ratio must be a finite number of at least 1, or registration throws (also for work: null). Every test whose route is not calibration calls __oracleWorkBound(ratio). The bound is the property's, so it judges every test the property registers, though the directive names one of them.

Maintainer decisions, 2026-11-12

1. No default bound. The bound is opt-in. A default would kill legitimate refactors that add a site evaluation. The extra-work class and its directive already make every such survivor visible. The 100× call budget remains only as the spin cutoff.
2. Plain Vitest. The bound judges nothing there, because there is no unmodified run to compare with. A recorded baseline would add state and its own flakiness. This is documented in the option's JSDoc and in the README.
3. Per test, not per history. A test that runs many histories (a generated unit) is judged by their sum, so a slowdown in a few histories is diluted. This is accepted as a limit and stated in the README.
4. The directive recommends the loosest bound that still rejects the mutant: floor((100·after − 1) / before) / 100, or 1 when the test did no work before. This is the bound least likely to fail a later legitimate change.
5. The two e2e failures listed under Limits are separate from this PR. They are fixed in a follow-up PR.

Review fixes (independent review of ba822f0)

Finding	Fix	RED at ba822f0	Mutant rejected
F1: an unregistered test got extra-work with a bound it cannot carry	Only reproducible and random routes judge work; the directive names the first registered test that rose	Classify oracle: extra-work, owed not-observed; the directive named the unregistered t0. E2e: w_fastplain	Restoring !== framework: 3 classification and 2 directive failures
F3: the float comparison disagreed with exact arithmetic	100 × calls > round(100 × ratio) × before; grids now reach 29 = 1.16 × 25	Classify oracle: killed by work bound, owed extra-work at 25 → 29 under 1.16	The float comparison: 4 failures
F5: a clock-driven count	A test that made a real timer or interval, which the capture now records, or held I/O open, judges no work	Classify oracle (grammar dimension clocked); e2e: src/sut.ts:w_fasttimed:101:if-false#0: killed, owed not-observed	Not excluding timers: 4 failures
F5, re-review at 9087746: a Date.now()/performance.now() loop, node:timers/promises, a setImmediate loop that reads a clock	The clock-read and module-timer paths above set the one clocked flag; timers and io are merged into it, and classify's route, clock, and rise checks are one closure that the bound reuses	Classify oracle 8 failed (killed laws [work bound], owed not-observed); e2e: src/sut.ts:w_fastsleep:110:if-false#0: killed, owed not-observed	Today's flag set: w_fastsleep killed, owed not-observed; no Date.now patch: w_fastclock killed, owed not-observed; immediates counted as clocks: w_fastturn not-observed, owed killed
F2: the ratio printed with two decimals	Three decimals	Classify oracle: directive mismatch	Two decimals: 4 failures
F4: work: null threw a TypeError	Reads the ratio from a null option	Self-oracle: Cannot read properties of null (reading 'ratio')	—
F7: the bound is per property	Documented in the option's JSDoc and the README	—	—
R1 (round 3): process.uptime, os.uptime, AbortSignal.timeout, Atomics.waitAsync, monitorEventLoopDelay escaped the flag	Wrapped through the same told wrapper (also eventLoopUtilization, Temporal.Now)	E2e, via w_fastclocked (tests that reach the clock by each path)	No process.uptime patch: w_fastclocked:101:if-false killed, owed not-observed
R2 (round 3): wrapping a timer dropped util.promisify.custom, so util.promisify(setTimeout) threw, in the baseline too (also the global wrapper on main)	keepProperties copies every own property onto each wrapper	E2e at b21d188: devBuild:4:binary-swap blocked, owed not-observed (file a's baseline fails on promisify(setTimeout)(0, 'v'))	A wrapper that keeps only string-keyed properties: the same RED
R3 (round 3): the first frame decided, so a user's read through a vi.spyOn passthrough was excluded	Any frame decides. Variant (a), flag every read, was measured on klona with bounds: 216 of 218 killed (json M16 and full M17 lost), because the oracle framework's own reads flag the tests that carried those rises; so the rule keeps excluding reads whose every frame is machinery	E2e, via the spied-clock test of w_fastclocked	The b21d188 first-frame rule: w_fastclocked:101:if-false killed, owed not-observed
Round 4: synchronous Atomics.wait with a timeout	Wrapped as Atomics.waitAsync is; a classify grammar entry and a w_fastclocked test	E2e at cfa10b7: w_fastclocked:101:if-false killed, owed not-observed	—
Round 4: the cost of Error.stackTraceLimit = 50	Measured, kept. A capture costs about 14 µs at 50 frames and 3 µs at 10, at a test's depth of about 40 frames (a 200,000-capture micro-benchmark). It runs only for a clock read inside a test that is not yet flagged. Klona's full bounded check took 6 s and 6 s at 50, and 7 s and 6 s at 10 (mutant phase 3.3 s and 3.5 s, against 3.8 s and 3.2 s): no measurable difference	—	—
CI over budget (round 3)	The e2e fixed-seed shards are repacked by CI costs. With the work bound derived from the plan (next row) the generated histories are main's again, so the packing follows main's CI costs at ae3184d: about 290 s, 277 s, and 260 s of CI time per job (135 s, 125 s, and 144 s locally). At 9087746 the third job ran 10.9 min, and at 661d2fd the second ran 10.8 min, both with the drawn bound's heavier histories	—	—
CI cost of the e2e grammar	A drawn work dimension in the e2e's generated test record shifted fast-check's stream, so every generated base and every constructed history built on one grew heavier (CI, main → this PR: generated 23 s → 172 s, noTypescript 4.5 s → 113 s). The stack capture was not the cost (locally: 58.7/65.2 s with it, 69.7/75.1 s flagging every read without it). The bound is now derived from the plan (1 when a generated test's input is a multiple of 3), with fc.constant(null) in the record, so the grammar's histories are main's again. Full fixed seed locally: 827 s drawn, 492 s derived	—	—
Builtin cost (maintainer)	Legend and README guidance	Classify oracle: the extra-work legend line does not say that a builtin counts as one site	—

Evidence

• Law: a survivor that changes no observation but does more work is extra-work, and its directive names the bound that rejects it. A declared work bound kills a mutant whose per-test work is more than ratio × the unmutated work. Authority: the maintainer's decision of 2026-11-12 ("this is a bug then in the gap cli and should be fixed"), and the design decided with it: a cost law, no exit, judged by a deterministic count.
• Reach of the work law by generated functions (new measured states extra work in a generated function and killed by a work bound in a generated function, summed over the 36 fixed-seed units): 0 and 0 with the drawn bound, 0 and 0 with the derived one. Deriving the bound drops no coverage: no generated function's mutant was a shortcut for speed either way, since the generated statements have no fast path that gives the general path's result. The e2e's work-law reach comes from the witness module (extra work 6 drawn and 3 derived, killed by a work bound 6 and 3, by unit count), and the classify oracle's generator covers the law broadly (counts below). A generated shortcut statement kind would reach it, at the cost of a new stream; it is not in this PR.
• Reach:
  • Classify oracle, one fixed-seed generated unit of 300 runs: extra-work 6, extra-work from no work 3, killed by a work bound 3, a work bound kill beside a changed value 1, random-only-kill by a work bound 3, a work rise where work judges nothing 4, a survivor whose work fell 3. extra-work under a declared bound is reached by its constructed history only (0 in generated runs).
  • Self-oracle: work-bound-registered and work-bound-left-out-of-calibration are reached by fixed histories. The small generated campaign reached them 0 times.
  • E2e fixed seed 20261105: every outcome class: extra work 1, killed by a work bound 1, with the witnesses w_fast, w_fastbound, w_fastcall, w_fastcallbound, and w_cold.
• RED (each on the parent commit):
  • vitest --run --config oracle-gaps/vitest.config.mjs gap-report-classify: 16 failed, for example mutant m0: ["survived","not-observed",null] owed ["survived","extra-work",null] and … owed ["killed",null,null] laws ["work bound"].
  • vitest --run --config vitest.oracle-framework.config.mjs -t "a work bound": 3 failed, with "fixed seed 7: generated" ran [1,0] with work bound null registered, owed 1.25, and ratios of 0.5 and Infinity loaded.
  • E2e -t "fixed seed 20261105: every outcome class", before the check.mjs fix: src/sut.ts:w_cold:231:drop-not#0: not-observed, owed extra-work.
• GREEN:
  • Classify oracle: 273/273.
  • Framework self-oracle, full: 851/851 (at 25ad619; the framework is unchanged since).
  • Static, classify, probe-form, and mutant-validity oracles: 33887/33887.
  • Package tests: 114/114. tsc --noEmit: clean. ESLint: clean on the changed files.
  • E2e unit fixed seed 20261105: every outcome class: passed.
  • E2e, every fixed-seed unit (-t "fixed seed 20261105: "): 36/36 at 35735d9 (581 s locally, verbose, used for the shard costs). Random-seed and other units run in CI.
  • An earlier full e2e file run (before the review fixes): 122/124. The two failures are under Limits and are not caused by this PR.
  • pr-shards check ("3 partitions split their tests exactly") and e2e-shards check ("5 shards partition the e2e tests") pass. They ran through an exec-only pnpm wrapper, with no install.
• Mutants (each one is rejected):
  • Classify oracle:
    • The classification ignores work.
    • It reads ms (wall time) instead of calls.
    • The directive offers an exit ("or accept this mutant as equivalent").
    • The bound counts a test whose work varied between the unmutated runs.
    • The bound kill also judges calibration routes.
    • The directive recommends the measured ratio instead of one below it.
    • The work-bound kill is dropped.
  • E2e:
    • The classification ignores work: w_fast:75:if-false: not-observed, owed extra-work.
    • The classification reads wall time: the same difference.
    • The capture does not record the bound: w_fastbound:84:if-false: extra-work, owed killed.
  • Self-oracle:
    • The bound is also registered on calibrations.
    • A ratio below 1 is accepted.
• Limits:
  • Work is counted per test, so a generated unit's histories are summed.
  • The count sees only site evaluations: a builtin call counts once, and a plain assignment or a for...of pass counts nothing.
  • Clock paths that are not watched: Temporal.Now is wrapped when present, which this Node (v24) cannot confirm, since it has no Temporal; a reference to a clock or timer taken before the harness ran (a module that loads before the setup file and keeps Date.now; the harness is the first setup file), a read whose every frame is an excluded library's (vitest, fast-check, @tanstack/oracles, tinyspy, tinypool, chai, or a directory named packages/oracles/src), Atomics.wait (synchronous), a native addon's clock, and a Worker's. The I/O signal is sampled every 50 ms, so a very short I/O can be missed. A read with any frame of the test file, production, or another dependency is watched, so a test loses work judging rather than giving a kill that does not replay. vi.useFakeTimers installed after the patch restores the patched functions on uninstall; that path is checked by inspection only.
  • An authored mutant's copy is not instrumented, so the work law does not judge it.
  • The e2e covers the work law only through these witnesses: an early return before a counting loop (bound 2), a push beside one that calls String (bound 1), the early return in tests that wait on a timer (bound 1) and in tests that register no route, a generated bound of 1, and one warm-cache selection witness. No e2e history has a test whose work varies between runs while its observations agree.
  • An internal error (exit 3) once in CI, during the e2e stability rerun of every outcome class on a slow runner (run 43346685892): not reproduced locally under tight limits or CPU contention, and the next occurrence's stack is kept by the driver now (error.log in the observed stderr). Tracked as oracle-gaps: an internal error (exit 3) in CI during the e2e stability rerun under load #56, where this PR's capture wrappers are named as one candidate cause.
  • Two full-e2e failures in code that this PR does not touch. Both are assigned to a follow-up PR:
    • determinism: constructed history: every outcome class. The stopped observation's sessions lists hold .oracle-gaps-<pid>, which differ between the two runs. That code came from fix(oracle-gaps): keep the daemon's state, bundles, and default output in the OS temp directory #43.
    • random seed: a failing owner under a config that --config names did not enter blocked by brokenOwner. Replay: ORACLE_SEED=520648412 ORACLE_PATH=0 ORACLE_PROPERTY='oracle-gaps.report' ORACLE_UNIT='random seed: a failing owner under a config that --config names'. Its fixed-seed twin passes, and its counterexample declares no work bound.
  • On the klona copy, a run without --seed classified all 10 shortcuts as not-judged, not extra-work. In every case the changed observation was in the old tarball's single random seed test of each mode. No test was marked nondeterministic, and no other mutant changed that test.
    • The reviewer traced it: with no --seed, oracleCampaignSeed draws the campaign seed from Math.random, so the seed can differ between runs, and the two-run agreement does not check it. It is a CLI determinism bug, investigated in the follow-up PR (a likely fix: the CLI pins one seed for every run of a check, and reports it).
    • With --seed 1 (as in the original r16 run), all 10 are extra-work.

Klona validation

This used a copy of the r16 repository. The original was not edited. The run was --seed 1, with 218 mutants.

Mutant	Test	Work before → after	Ratio	Bound to declare
json M1 key in obj → true	klona-json fixed seed 1	11156 → 13124	1.18×	1.17
json M16 && → ||	klona-json fixed seed 1	11156 → 11914	1.07×	1.06
json M30 && → ||	klona-json fixed seed 1	11156 → 12991	1.16×	1.16
lite M1 key in obj → true	klona-lite fixed seed 1	10313 → 11833	1.15×	1.14
lite M12 early return	klona-lite fixed seed 1	10313 → 16280	1.58×	1.57
default M1 key in obj → true	klona-default fixed seed 1	11270 → 12246	1.09×	1.08
default M12 early return	klona-default fixed seed 1	11270 → 21395	1.90×	1.89
full M5 !val.enumerable … → true	klona-full fixed seed 1	28088 → 28627	1.02×	1.01
full M17 early return	klona-full fixed seed 1	28088 → 31896	1.14×	1.13
full M68 hasOwnProperty … → false	klona-full fixed seed 1	28088 → 31884	1.14×	1.13

• Before this change, all 10 survivors were gaps (8 not-observed and 2 not-distinguished). After it, all 10 are extra-work.
• After the clock rule, the same bounded run again gives 218 of 218 killed and 0 gaps: the framework's own clock reads are not counted.
• I then declared work: { ratio } on the four properties (json 1.06, lite 1.14, default 1.08, full 1.01), with this PR's framework hook ported into the copy's installed build. The plain Vitest run still passes (78 tests). The gap check reports 218 of 218 mutants killed and 0 gaps, and work bound is the first law to reject 11 of them. After the review fixes, the same run again gives 218 of 218 killed and 0 gaps.
• A uniform ratio of 1.1 would leave json M16, default M1, and full M5 as gaps.

Size

Production code, compared with origin/main, minified with esbuild and gzipped:

File	Before	After	Change
oracle-gaps/classify.mjs	2410 B	2724 B	+314 B
oracle-gaps/report.mjs	4200 B	4502 B	+302 B
oracle-gaps/capture-setup.mjs	4776 B	5525 B	+749 B
oracle-gaps/check.mjs	12636 B	12643 B	+7 B
src/oracle-property.ts	10864 B	10992 B	+128 B

• The total is about +1500 B. The repository has no size:compare script, so size:compare was not measured; these are esbuild-minified, gzipped file measurements, not an app bundle. The clock rule's capture patches (every clock source, property keeping, and the stack rule) are about +670 B of it; merging the flags and folding rose into the bound's closure saved 18 B in classify. A perturbed-clock comparison would need the same patches (round 3's review), so it is not lighter. The CLI ships as source, so its shipped gzip, comments included, grows by about +1.8 KB. Including oracle-property.ts, the total is about +2.5 KB. Most of the growth after review is the legend's builtin guidance (+115 B) and the route, timer, and exact-comparison rules.
• The change reuses the existing counter and adds one class, one law, and one option. I found no state or branch that it makes redundant.
• Tests and oracles add about 700 lines. Much of the model diff is the rename of reached.add to a counting reach.

This pull request and its description were written by Isaac.

[changeset-bot]

🦋 Changeset detected

Latest commit: fa88316

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@tanstack/oracles	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[coderabbitai]

Important

• Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: fcbf0d93-67e8-4cb4-8648-28861742c0c9

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/TanStack/oracles/@tanstack/eslint-plugin-oracles@48

npm i https://pkg.pr.new/TanStack/oracles/@tanstack/oracles@48

commit: fa88316