From 92be5cdbc1a379c883452f243709c4fc855040e0 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Thu, 12 Nov 2026 10:17:28 -0700 Subject: [PATCH 01/20] test(oracle-gaps): a shortcut for speed is owed a cost law, not a not-observed gap The classify oracle gains the work law: each test's work (site evaluations, never time), whether it varied between the unmutated runs, a declared work bound, and what each mutant adds to the work. A survivor whose work rose in a test that judges work is extra-work; a test past its bound reports a verdict under law work bound on its route. The self-oracle gains oracleProperty's work option: a ratio below 1 or not finite fails to load, and every test whose route judges production registers the ratio through __oracleWorkBound, and no calibration does. The end-to-end model counts work per site evaluation and gains shortcut witnesses (an early return and a skipped String call), each with a bound twin, and a warm-cache witness for selections. RED on the parent: the classify oracle reports not-observed where extra-work and work-bound kills are owed; the self-oracle reports that no work bound is registered and that invalid ratios load. Co-authored-by: Isaac --- ...p-report-classify-oracle.property.test.mjs | 361 ++++++++++++++++-- .../oracle-gaps/gap-report-model-oracle.mjs | 288 +++++++++++--- .../gap-report-oracle.property.test.mjs | 25 +- .../oracle-property-self-oracle-fixture.ts | 28 +- .../tests/oracle-property-self-oracle.test.ts | 77 +++- 5 files changed, 685 insertions(+), 94 deletions(-) diff --git a/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs b/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs index 4fe0955..79d9023 100644 --- a/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs +++ b/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs @@ -42,6 +42,26 @@ * them all. Failing that, a verdict on the random campaign * makes it a `random-only-kill`, and one on an unknown or unregistered * route an `unregistered-kill`. + * A test's work is the number of site evaluations that its production + * code made (capture-setup.mjs `calls`), a count, never a time. A test + * whose property declares a work bound (`work: { ratio }`) and whose + * work under the mutant is more than `ratio` times its work with no + * mutant active reports a verdict under law `work bound` on its route, + * as a law violation does, so it kills on a reproducible route and is a + * `random-only-kill` or an `unregistered-kill` elsewhere. Only a test + * whose work judges the mutant reports it: it passed under the mutant, + * it reached the mutant (or its file loaded code that did), its + * observations and its work agreed between the two unmutated runs, and + * its route judges production (not the framework's own checks). The + * runner's stop at the first file that kills reads law verdicts only, so + * a kill names the laws of the files that ran up to that one, `work + * bound` among them when one of their tests exceeded its bound. + * Authority: the maintainer's decision of 2026-11-12 that a production + * branch that exists only for speed, reported as `not-observed`, is a + * gap CLI bug ("this is a bug then in the gap cli and should be fixed"), + * and the design decided with it that day: a speed shortcut is owed a + * cost law, not an exit, so a regression that deletes it must be + * catchable, by a deterministic count and never by wall-clock time. * 4. Failing that, if every failing test timed out, it is `timeout`; a * failing test outside the framework's own checks makes it `crashed`; a * failing framework check alone (the `calibration` route, which covers @@ -53,7 +73,16 @@ * `not-judged` when a reaching, deterministic owner test's asserted or * observed value changed (code that ran when a file loaded the module * reaches every test of that file, maintainer decision of 2026-11-06); - * `indeterminate` when no such test judged it; `dead-code` when the + * `indeterminate` when no such test judged it; `extra-work` when a test + * whose work judges the mutant (law 3) did more work under it than with + * no mutant active, though no observation changed: the mutant deletes a + * shortcut that exists for speed, which only a cost law sees, so its + * directive names the first such test in report order, its two work + * counts and their ratio, and the work bound to declare on its property, + * the largest ratio in hundredths below the measured one (1 when the + * test did no work before), which rejects the mutant; a mutant whose + * work fell, or rose only in a test whose work does not judge it, is + * not `extra-work`; `dead-code` when the * mutant carries the `dead code` evidence (no input distinguishes it); * `outside-annotation` when the type analysis names an input class * outside the declared types, and none of its inputs distinguishes the @@ -191,7 +220,11 @@ * names and the test's joined by a space (@vitest/runner 4.1). * * Limits: the records' shapes are those capture-setup.mjs writes; that a - * real run writes them correctly is the end-to-end oracle's. The driver + * real run writes them correctly is the end-to-end oracle's, and so is + * which work the capture counts. A test that reaches a site and did no + * work is generated, though no real run records one. An authored mutant's + * copy is not instrumented and counts no work, so the work law does not + * judge it. The driver * encodes operands as the capture does (`shown`); that the capture's * encoding is lossless is not judged here, and two long strings are told * apart by a 32-bit digest. No symbol operand is generated: two symbols of @@ -257,6 +290,15 @@ const SLOTS = MUTANTS + AUTHORED // names one of two behavioral laws, `L` and `M`, or reach agreement, so // the laws of a kill can show which owner file decided it. // +// Each test also has a work count with no mutant active (none, some, or +// much), may have done one more in its second unmutated run, may declare a +// work bound of 1, 1.5, or 2, and has, under each mutant, an amount that +// the mutant adds to its work or takes from it. The amounts straddle the +// bounds: 1 more on 10 or 100 is under every bound but 1, 7 more on 10 is +// past 1.5 and under 2, and 150 more is past every bound. Its time in +// milliseconds varies on its own, as a clock's does, so a classification +// that read time where it owes work would disagree with the model. +// // The order of the records is also generated, because a real run's order // depends on how it ran (law 7). The runner deals the files round-robin over // its contexts and joins their records context by context, so the baseline's @@ -394,6 +436,11 @@ const effect = fc.record({ law: fc.constantFrom(`L`, `M`, `reach agreement`), // The route active when a verdict fired, when it differs from the test's. firedAt: fc.option(fc.constantFrom(...routes.slice(1)), { nil: undefined }), + // What the mutant adds to the test's work (site evaluations; a negative + // amount takes work away), and the test's time in milliseconds, which + // varies as a clock does and judges nothing. + work: fc.constantFrom(0, 0, 0, -1, 1, 7, 150), + ms: fc.nat({ max: 40 }), }) const testRecord = fc.record({ route: fc.constantFrom(...routes), @@ -419,6 +466,14 @@ const testRecord = fc.record({ // The test's place among its file's records in a mutant run (ties keep the // file's order). runOrder: fc.nat({ max: 2 }), + // The test's work with no mutant active, in site evaluations; whether its + // second unmutated run did one more (its observations still agreeing); + // the work bound its property declares (`work: { ratio }`), if any; and + // its time in the two unmutated runs, which judges nothing. + work: fc.constantFrom(0, 10, 100), + workFlaky: fc.constantFrom(false, false, false, true), + bound: fc.constantFrom(undefined, undefined, 1, 1.5, 2), + ms: fc.nat({ max: 40 }), }) const records = fc.record({ files: fc.array( @@ -561,11 +616,46 @@ const swapFor = { '>': `>=`, '>=': `>`, '===': `!==`, '&&': `||` } // The model states the law above over the records, without the CLI's // functions: it reads each test's effect under the mutant directly. +// The work law (law 3). A test's work with no mutant active is `t.work` +// site evaluations (its second unmutated run did one more when +// `t.workFlaky`), and under the mutant in slot `slot` it is that plus what +// the effect adds, never below none. +const workUnder = (t, slot) => Math.max(0, t.work + t.under[slot].work) +// Whether a test's work judges operator mutant `mi`: it passed under the +// mutant, it reached the mutant or its file loaded code that did, its +// observations and its work agreed between the unmutated runs, and its +// route judges production, not the framework's own checks. +const passedKinds = new Set([`same`, `changed`, `observedChanged`, `unhandled`]) +const workJudges = (h, mi, x) => + passedKinds.has(x.t.under[mi].kind) && + (x.t.reaches[mi] || h.files[x.fi].background[mi]) && + !x.t.flaky && + !x.t.workFlaky && + kindOf(x.t.route) !== `framework` +// A test past its declared bound: more work than `ratio` times its work +// with no mutant active. +const pastBound = (h, mi, x) => + workJudges(h, mi, x) && + x.t.bound !== undefined && + workUnder(x.t, mi) > x.t.bound * x.t.work + +// The work bound that rejects a rise from `before` to `after` site +// evaluations: the largest ratio in hundredths below the measured one, and +// 1 when the test did no work before (any ratio rejects a rise from none). +function boundBelow(before, after) { + if (before === 0) return `1.00` + let hundredths = 100 + while ((hundredths + 1) * before < 100 * after) hundredths += 1 + return (hundredths / 100).toFixed(2) +} + // Laws 2 to 4 over the tests that judge a mutant, read from each test's // effect in the mutant's slot: the failure's outcome, if a failure decides, // and the tests that ran (a file that failed to load, or whose run timed -// out, ran none). -function failureOf(h, slot, tests) { +// out, ran none). `work` says whether the work law judges the slot: it +// judges operator mutants, whose copies count work, and the runner's stop +// at a kill reads no work. +function failureOf(h, slot, tests, work = false) { const under = ({ t }) => t.under[slot].kind const timedOut = tests.some((x) => under(x) === `runTimeout`) const setup = tests.some((x) => under(x) === `setupError`) @@ -580,13 +670,19 @@ function failureOf(h, slot, tests) { !x.t.flaky && [`verdict`, `assertion`, `crash`, `timeoutMessage`].includes(under(x)), ) + // A test past its work bound reports a verdict under `work bound` on its + // own route; the tests stay in report order. + const overBound = (x) => work && pastBound(h, slot, x) + const lawOf = (x) => (overBound(x) ? `work bound` : x.t.under[slot].law) const verdictsIn = (kind) => - failing.filter( + ran.filter( (x) => - under(x) === `verdict` && - kindOf(x.t.under[slot].firedAt ?? x.t.route) === kind, + (failing.includes(x) && + under(x) === `verdict` && + kindOf(x.t.under[slot].firedAt ?? x.t.route) === kind) || + (overBound(x) && kindOf(x.t.route) === kind), ) - const kills = verdictsIn(`reproducible`).map((x) => x.t.under[slot].law) + const kills = verdictsIn(`reproducible`).map(lawOf) const behavioral = kills.filter((law) => law !== `reach agreement`) const decided = (failure) => ({ failure, ran }) // A gap that names a failing test names the first one its class counts, @@ -676,21 +772,28 @@ function judgedOutcome(h, mi, owners) { const tests = targets.flatMap((fi) => h.files[fi].tests.map((t, ti) => ({ t, fi, ti })), ) - const { failure, ran } = failureOf(h, mi, tests) - // The runner runs the owner files in report order and stops at the first - // that kills, so a kill names laws of that file only (law 7). + const { failure, ran } = failureOf(h, mi, tests, true) + // The runner runs the owner files in report order and stops after the + // first whose law verdicts kill (its stop reads no work), so a kill names + // the laws of the files it ran: law verdicts of that file only, and `work + // bound` when a test of those files exceeded its bound (law 7). if (failure?.outcome === `killed`) { - const first = targets - .map( - (fi) => - failureOf( - h, - mi, - tests.filter((x) => x.fi === fi), - ).failure, - ) - .find((f) => f?.outcome === `killed`) - return { ...failure, laws: first.laws, ownerFiles } + const stop = targets.findIndex( + (fi) => + failureOf( + h, + mi, + tests.filter((x) => x.fi === fi), + ).failure?.outcome === `killed`, + ) + const ranFiles = stop < 0 ? targets : targets.slice(0, stop + 1) + const inRun = failureOf( + h, + mi, + tests.filter((x) => ranFiles.includes(x.fi)), + true, + ).failure + return { ...failure, laws: inRun.laws, ownerFiles } } if (failure) return { ...failure, ownerFiles } // A survivor. Only reaching, deterministic owner tests judge it; code that @@ -728,6 +831,20 @@ function judgedOutcome(h, mi, owners) { ], }) } + // A test whose work judges the mutant did more work, and nothing it + // observed changed: the mutant deleted a shortcut (law 5). The message + // names the first such test in report order. + const rose = compared.find( + (x) => workJudges(h, mi, x) && workUnder(x.t, mi) > x.t.work, + ) + if (rose) + return sized(`extra-work`, { + mention: [ + `\`work: { ratio: ${boundBelow(rose.t.work, workUnder(rose.t, mi))} }\``, + `\`${shownName(h, rose.fi, rose.ti)}\``, + `rose from ${rose.t.work} to ${workUnder(rose.t, mi)} site evaluation${workUnder(rose.t, mi) === 1 ? `` : `s`}${rose.t.work > 0 ? ` (${(workUnder(rose.t, mi) / rose.t.work).toFixed(2)}× the unmodified run's)` : ``}.`, + ], + }) if (m.evidence === `dead code`) return sized(`dead-code`, { mention: [`line ${m.line + 1}.`] }) // The type analysis found no input of the declared types that @@ -1160,6 +1277,8 @@ const directives = { 'not-observed': /^Observe the value this code changes in (`[^`]+`|the owner tests), and assert on it\. Where the mutated code computes the same result as the original, add a history where it computes something different\.$/, 'not-judged': /^Make a law in `[^`]+` reject the changed value\.$/, + 'extra-work': + /^Declare `work: \{ ratio: \d+\.\d\d \}` on the `oracleProperty` that registers `[^`]+`, so that its work bound rejects this mutant: there it changed no observation, and its work rose from \d+ to \d+ site evaluations?( \(\d+\.\d\d× the unmodified run's\))?\.$/, indeterminate: /^(Make `[^`]+` deterministic: control its clock and its random seed, so its failures judge this code|Add an owner test that runs line \d+ and asserts on its result)\.$/, 'dead-code': /^Remove the redundant code on line \d+\.$/, @@ -1401,6 +1520,11 @@ function baselineUnitsOf(h, pass) { ...capturedRows(h, t), route: t.route, verdicts: [], + // The capture's work count and the property's declared bound + // (capture-setup.mjs `calls` and `work`), and the test's time. + calls: t.work + (t.workFlaky && pass === 2 ? 1 : 0), + work: t.bound, + ms: t.ms + pass, }) }) rows.set(`${fileName(h, fi)} > (background)`, { @@ -1470,6 +1594,11 @@ function unitOf(h, fi, slot) { hits: [], outcomes: [], route: t.route, + // An authored mutant's copy is not instrumented, so it counts no + // work. + calls: slot < MUTANTS ? workUnder(t, slot) : 0, + work: t.bound, + ms: under(t).ms, verdicts: kind === `verdict` ? [ @@ -1745,6 +1874,53 @@ async function driver(h, observe) { } if (result.outcome === `killed` && units.some((u) => u.timedOut)) states.add(`killed beside a timeout`) + // The work law's states (laws 3 and 5): a kill by a work bound, alone + // or beside a changed value that would otherwise be not-judged, a work + // bound on the random campaign, and survivors whose work rose under a + // declared bound, rose from none, rose only where work judges nothing, + // or fell. + const workTests = targets.oracleTargets + .map(fileIndex) + .flatMap((fi) => h.files[fi].tests.map((t, ti) => ({ t, fi, ti }))) + .filter((x) => x.t.reaches[mi] || h.files[x.fi].background[mi]) + const judgedRise = workTests.find( + (x) => workJudges(h, mi, x) && workUnder(x.t, mi) > x.t.work, + ) + if (result.laws?.includes(`work bound`)) { + states.add(`${result.outcome} by a work bound`) + if ( + result.outcome === `killed` && + workTests.some( + (x) => + !x.t.flaky && + [`changed`, `observedChanged`].includes(x.t.under[mi].kind), + ) + ) + states.add(`a work bound kill beside a changed value`) + } + if (result.class === `extra-work` && judgedRise) { + if (judgedRise.t.bound !== undefined) + states.add(`extra-work under a declared bound`) + if (judgedRise.t.work === 0) states.add(`extra-work from no work`) + } + if (result.class !== undefined && result.class !== `extra-work`) { + if ( + workTests.some( + (x) => + passedKinds.has(x.t.under[mi].kind) && + !x.t.flaky && + workUnder(x.t, mi) > x.t.work && + !workJudges(h, mi, x), + ) + ) + states.add(`a work rise where work judges nothing`) + if ( + workTests.some( + (x) => workJudges(h, mi, x) && workUnder(x.t, mi) < x.t.work, + ) + ) + states.add(`a survivor whose work fell`) + } const declared = [...new Set(h.mutants[mi].owners)] orderStates(h, states, `changed test`, placesOfKeys(h, result), declared) if (namesAFailingTest.has(result.outcome)) @@ -2628,6 +2804,28 @@ const calibrations = [ return trace.map((o, i) => (i === index ? { ...o, value } : o)) }, }, + { + name: `a survivor that only did more work is reported as not observed`, + law: `classification`, + corrupt: corrupt((v) => { + const o = v.outcomes.find((x) => x.class === `extra-work`) + if (!o) return false + o.class = `not-observed` + }), + }, + { + name: `a kill by a work bound is reported as a survivor`, + law: `classification`, + corrupt: corrupt((v) => { + const o = v.outcomes.find( + (x) => x.outcome === `killed` && x.laws?.includes(`work bound`), + ) + if (!o) return false + o.outcome = `survived` + o.class = `extra-work` + o.laws = undefined + }), + }, { name: `a crash is reported as a kill`, law: `classification`, @@ -2715,6 +2913,14 @@ const required = [ `killed by one owner beside a failing owner`, `a baseline law violation with its counterexample`, `an ordinary file's failing baseline test`, + `extra-work`, + `extra-work under a declared bound`, + `extra-work from no work`, + `killed by a work bound`, + `a work bound kill beside a changed value`, + `random-only-kill by a work bound`, + `a work rise where work judges nothing`, + `a survivor whose work fell`, ] // One constructed history per outcome class: one mutant judged by one @@ -2729,13 +2935,24 @@ const plainTest = (effect = {}, test = {}) => ({ flaky: false, reaches: four(true), seen: four(`both`), - under: slots({ kind: `same`, law: `L`, firedAt: undefined, ...effect }), + under: slots({ + kind: `same`, + law: `L`, + firedAt: undefined, + work: 0, + ms: 0, + ...effect, + }), pair: four([0, 1]), suites: [], label: ``, tail: ``, fails: false, runOrder: 0, + work: 10, + workFlaky: false, + bound: undefined, + ms: 0, ...test, }) const single = ( @@ -2920,6 +3137,8 @@ const constructed = [ kind: `runTimeout`, law: `L`, firedAt: undefined, + work: 0, + ms: 0, }), }, ], @@ -2940,6 +3159,69 @@ const constructed = [ single(`a changed value`, [`not-judged`], { effect: { kind: `changed` }, }), + // The work law (laws 3 and 5): klona's shapes. A forced `if` that + // deletes an early return, and a skipped call whose absence costs work + // elsewhere, change no observation and cost more work. + single(`a forced branch that only costs work`, [`extra-work`], { + mutant: { operator: `if-false` }, + test: { work: 100 }, + effect: { work: 150 }, + }), + single(`a skipped call that only costs work`, [`extra-work`], { + effect: { work: 7 }, + }), + single( + `a rise under the declared bound`, + [`extra-work`, `extra-work under a declared bound`], + { + mutant: { operator: `if-false` }, + test: { work: 100, bound: 2 }, + effect: { work: 7 }, + }, + ), + // A test that reaches a site does work, so a rise from none comes from + // code that ran when the file loaded the module: the mutant changed what + // the loaded module does in a test that reaches none of its sites. + single(`a rise from no work`, [`extra-work`, `extra-work from no work`], { + mutant: { operator: `if-true` }, + test: { work: 0, reaches: four(false) }, + background: four(true), + backgroundInfects: four(true), + effect: { work: 1 }, + }), + single(`a rise past the declared bound`, [`killed by a work bound`], { + mutant: { operator: `if-false` }, + test: { bound: 1.5 }, + effect: { work: 7 }, + }), + single( + `a rise past the declared bound beside a changed value`, + [`killed by a work bound`, `a work bound kill beside a changed value`], + { test: { bound: 2 }, effect: { kind: `changed`, work: 150 } }, + ), + single( + `a rise past the declared bound in the random campaign`, + [`random-only-kill by a work bound`], + { test: { route: `random seed`, bound: 1 }, effect: { work: 1 } }, + ), + single( + `a rise in a test whose work varied`, + [`a work rise where work judges nothing`], + { + mutant: { operator: `if-false` }, + test: { workFlaky: true, bound: 1 }, + effect: { work: 7 }, + }, + ), + single(`a rise in a calibration`, [`a work rise where work judges nothing`], { + mutant: { operator: `if-false` }, + test: { route: `calibration`, bound: 1 }, + effect: { work: 150 }, + }), + single(`a forced branch that saves work`, [`a survivor whose work fell`], { + mutant: { operator: `if-false` }, + effect: { work: -1 }, + }), twoOwners( `a value changed in two owner files declared against file order`, [ @@ -3052,7 +3334,13 @@ const constructed = [ { effect: { kind: `verdict` }, second: { - under: slots({ kind: `verdict`, law: `M`, firedAt: undefined }), + under: slots({ + kind: `verdict`, + law: `M`, + firedAt: undefined, + work: 0, + ms: 0, + }), }, }, ), @@ -3342,22 +3630,27 @@ const constructed = [ oracleProperty({ id: `oracle-gaps.report-classify`, law: { - statement: `From the records of the runs that judged it, each mutant, operator or authored, gets the outcome the law assigns: kills only by law verdicts on reproducible routes, agreement-only, random-only, and unregistered kills, timeouts, crashes, framework and setup errors as gaps, survivors split into not-judged, indeterminate, dead-code, outside-annotation, branch-never-taken, not-infected, not-observed, and not-distinguished, and ordinary tests for information only. Every gap has one directive message from its kind's fixed set, with no exit; a not-observed directive also directs a history where the mutated code computes something different, and a not-infected message never prints two different operand values alike. A message or list that names tests or files (a gap's Owners line, the files that failed with no mutant active, and the nondeterministic tests included) names them in report order, by test file path compared by code unit and within a file in the baseline's record order, whatever order the runner joined the baseline's files in, a mutant run wrote a file's records in, or the owners were declared in; an operator mutant's owner files run in that order, and a kill names laws of the first that kills. The summary lists every gap and unowned mutant once with consistent counts; the stdout summary and the streamed gap lines cover every gap once, each on one line with its code or intent cut at 60 characters, and the results keep the full text. Each failing oracle file's baseline replay command, read by a POSIX shell, runs exactly its first failing test under Vitest 4's and Vitest 5's name joins, and the whole file when no test failed.`, - authority: `The maintainer's gap-check decisions in design runs v11, v14, and v19 (2026-11-04 and 2026-11-05), as for oracle-gaps.report; the oracle rubric's outcome kinds (ORC-006); the frozen design's oracle over the checker's pure core (v19 P10, F4); the maintainer's few-seconds budget for an incremental test run (2026-11-05); the maintainer's findings F22 and the leftovers of #12 (2026-11-10); for report order, the gap CLI's determinism requirement (oracle-gaps.report's determinism test and law 9, and the runner's "listed alike in every run") and the maintainer's decision of 2026-11-11 that the tests a report names depend on neither --contexts, worker timing, nor how owners are declared; Vitest's runner (@vitest/runner 4.1: -t is a regular expression matched against getTaskFullName, the suites' names and the test's joined by a space); POSIX.1-2017 Shell Command Language 2.2 and 2.6 for the replay's quoting.`, - limits: `That real runs produce these records is oracle-gaps.report's, and so is a real replay. The driver encodes operands as capture-setup.mjs \`shown\` does, by a copy of it; long strings are told apart by a 32-bit digest; no symbol operand is generated, and two symbols of one description print alike. Only the runner's single-file path records a test's names, and only it feeds the baseline. Test names hold no backtick and no \` > \`, and names within a file are distinct; the replay's file argument is judged as text, not as Vitest's substring filter over every file. Authored intents hold no word that the exit law forbids, and an author is not shortened. Report order within a file is the baseline's record order: that a file whose tests run concurrently writes its records in one order is not judged, and a failing test is ordered within its file as the reporter lists it, which a run cut off by a hang takes from the records. The "Changed observations in" list is the changed tests' first three, and only its first is compared. A test that only a mutant run records is not generated. A file that fails with no mutant active keeps its records and its mutants are classified; the refusal text and the summary's left-out list are not judged, only the list of failing files they print.`, + statement: `From the records of the runs that judged it, each mutant, operator or authored, gets the outcome the law assigns: kills only by law verdicts on reproducible routes, agreement-only, random-only, and unregistered kills, timeouts, crashes, framework and setup errors as gaps, survivors split into not-judged, indeterminate, extra-work, dead-code, outside-annotation, branch-never-taken, not-infected, not-observed, and not-distinguished, and ordinary tests for information only; a test whose property declares a work bound, whose work judges the mutant, and whose work (site evaluations, never time) passes the bound's ratio of its unmutated work reports a verdict under law work bound on its route, and a survivor whose work rose in such a test, with no changed observation, is extra-work, whose directive names that test, both counts, and the bound to declare that rejects it. Every gap has one directive message from its kind's fixed set, with no exit; a not-observed directive also directs a history where the mutated code computes something different, and a not-infected message never prints two different operand values alike. A message or list that names tests or files (a gap's Owners line, the files that failed with no mutant active, and the nondeterministic tests included) names them in report order, by test file path compared by code unit and within a file in the baseline's record order, whatever order the runner joined the baseline's files in, a mutant run wrote a file's records in, or the owners were declared in; an operator mutant's owner files run in that order, and a kill names laws of the first that kills. The summary lists every gap and unowned mutant once with consistent counts; the stdout summary and the streamed gap lines cover every gap once, each on one line with its code or intent cut at 60 characters, and the results keep the full text. Each failing oracle file's baseline replay command, read by a POSIX shell, runs exactly its first failing test under Vitest 4's and Vitest 5's name joins, and the whole file when no test failed.`, + authority: `The maintainer's gap-check decisions in design runs v11, v14, and v19 (2026-11-04 and 2026-11-05), as for oracle-gaps.report; the oracle rubric's outcome kinds (ORC-006); the frozen design's oracle over the checker's pure core (v19 P10, F4); the maintainer's few-seconds budget for an incremental test run (2026-11-05); the maintainer's findings F22 and the leftovers of #12 (2026-11-10); for the work law, the maintainer's decision of 2026-11-12 that a speed shortcut reported as not-observed is a gap CLI bug, and the design decided with it: a cost law, never an exit, judged by a deterministic count; for report order, the gap CLI's determinism requirement (oracle-gaps.report's determinism test and law 9, and the runner's "listed alike in every run") and the maintainer's decision of 2026-11-11 that the tests a report names depend on neither --contexts, worker timing, nor how owners are declared; Vitest's runner (@vitest/runner 4.1: -t is a regular expression matched against getTaskFullName, the suites' names and the test's joined by a space); POSIX.1-2017 Shell Command Language 2.2 and 2.6 for the replay's quoting.`, + limits: `That real runs produce these records is oracle-gaps.report's, and so is a real replay. The driver encodes operands as capture-setup.mjs \`shown\` does, by a copy of it; long strings are told apart by a 32-bit digest; no symbol operand is generated, and two symbols of one description print alike. Only the runner's single-file path records a test's names, and only it feeds the baseline. Test names hold no backtick and no \` > \`, and names within a file are distinct; the replay's file argument is judged as text, not as Vitest's substring filter over every file. Authored intents hold no word that the exit law forbids, and an author is not shortened. Report order within a file is the baseline's record order: that a file whose tests run concurrently writes its records in one order is not judged, and a failing test is ordered within its file as the reporter lists it, which a run cut off by a hang takes from the records. The "Changed observations in" list is the changed tests' first three, and only its first is compared. A test that only a mutant run records is not generated. A file that fails with no mutant active keeps its records and its mutants are classified; the refusal text and the summary's left-out list are not judged, only the list of failing files they print. Work arrives as counts: that the capture counts site evaluations, and which work it does not count, is oracle-gaps.report's; a test that reaches a site but did no work is generated, which no real run records; an authored mutant's copy counts no work, so the work law does not judge it.`, }, histories: { arbitrary: records, states: required, // An authored mutant's other outcomes, measured. measured: [ - `timeout`, - `agreement-only-kill`, - `unregistered-kill`, - `crashed`, - `framework-error`, - `setup-error`, - ].map((kind) => `authored ${kind}`), + ...[ + `timeout`, + `agreement-only-kill`, + `unregistered-kill`, + `crashed`, + `framework-error`, + `setup-error`, + ].map((kind) => `authored ${kind}`), + // A work bound comes from `oracleProperty`, which always registers a + // route, so this one is measured, not required. + `unregistered-kill by a work bound`, + ], runs: 300, fixedSeeds: [20261105], constructed, diff --git a/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs b/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs index b051428..6b045d2 100644 --- a/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs +++ b/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs @@ -405,6 +405,9 @@ const testCase = fc.record({ fc.record({ route: fc.constantFrom(...routes), input: constant }), { nil: null }, ), + // A test may declare a work bound, as an oracle property's `work` does: + // a ratio of 1 rejects a mutant that does any more work there. + work: fc.constantFrom(null, null, null, 1), }) // Law 9's edits. Each content edit changes what some evidence depends on: // a test file, a source statement, the source's line numbers, an owns list, @@ -810,6 +813,7 @@ const judging = (file, route, input, extra = {}) => ({ asProp: false, stops: false, then: null, + work: null, ...extra, }) const witness = [ @@ -902,6 +906,59 @@ const witness = [ body: [{ kind: `memoSet`, label: 21 }], tests: [judging(`a`, `fixed seed`, 4)], }, + // A cache that an earlier test of the file warmed: `w_unseen`'s test + // caches under 4 first, so in the whole file `w_cold`'s write is + // skipped. A run of the tests from `w_cold`'s on, without the one that + // warmed the cache, observes the same and does one call more, so the + // runner owes a selection only where it reproduces the whole file's work + // as well as its observations. Its `drop-not` mutant writes again in the + // whole file (`extra-work`), and writes nothing in such a selection. + { + id: `w_cold`, + owners: [`a`], + warm: true, + body: [{ kind: `memoSet` }], + tests: [judging(`a`, `fixed seed`, 4)], + }, + // Shortcuts for speed (klona's early return for a primitive, and its + // assignment in place of `Object.defineProperty`): the general path gives + // the same output for every input, with more work, so forcing the + // shortcut's `if` false changes no observation and costs work + // (`extra-work`). Each has a twin whose tests declare a work bound, which + // kills that mutant under law `work bound`: 2 for the early return, whose + // loop costs several times the shortcut, and 1 for the call, which costs + // one site evaluation more. Their inputs are 7, which takes the shortcut, + // and 1, which takes the general path. + { + id: `w_fast`, + owners: [`a`], + body: [{ kind: `shortcut`, op: `>`, k: 2, not: false }], + tests: [judging(`a`, `fixed seed`, 7), judging(`a`, `fixed seed`, 1)], + }, + { + id: `w_fastbound`, + owners: [`a`], + body: [{ kind: `shortcut`, op: `>`, k: 2, not: false }], + tests: [ + judging(`a`, `fixed seed`, 7, { work: 2 }), + judging(`a`, `fixed seed`, 1, { work: 2 }), + ], + }, + { + id: `w_fastcall`, + owners: [`a`], + body: [{ kind: `callShortcut`, op: `>`, k: 2, not: false }], + tests: [judging(`a`, `fixed seed`, 7), judging(`a`, `fixed seed`, 1)], + }, + { + id: `w_fastcallbound`, + owners: [`a`], + body: [{ kind: `callShortcut`, op: `>`, k: 2, not: false }], + tests: [ + judging(`a`, `fixed seed`, 7, { work: 1 }), + judging(`a`, `fixed seed`, 1, { work: 1 }), + ], + }, { id: `w_noobs`, owners: [`a`], @@ -1620,6 +1677,11 @@ const captureWitness = new Set([ `w_observe`, `w_expect`, `w_unseen`, + `w_cold`, + `w_fast`, + `w_fastbound`, + `w_fastcall`, + `w_fastcallbound`, `w_noobs`, `w_unreached`, `w_ordinary`, @@ -1872,6 +1934,21 @@ function renderStatement(stmt, callText) { ] case `elseIf`: return [`if (${condText(stmt)}) report();`, `else ${pushText(stmt)}`] + case `shortcut`: + return [ + `if (${condText(stmt)}) {`, + ` out.push('s' + x);`, + ` return;`, + `}`, + `let n${stmt.label} = 0;`, + `while (n${stmt.label} < x) n${stmt.label} += 1;`, + `out.push('s' + n${stmt.label});`, + ] + case `callShortcut`: + return [ + `if (${condText(stmt)}) out.push('s' + x);`, + `else out.push(String('s' + x));`, + ] case `pair`: return [ `const pair = { get: (k: number): string => String(k), set: (k: number): string => String(k + 1) };`, @@ -3085,6 +3162,30 @@ function sitesOf(plan) { }) guard(fn.id, stmt.line + 2, `'gap#p${stmt.label}'`, `true`) } + // The shortcut's `if` (its whole block is the statement that holds + // its sites), its push, then the loop's condition and comparison, and + // the push after the loop, each in the statement that holds it. + if (stmt.kind === `shortcut`) { + const block = stmt.lines.slice(0, 4).join(` `) + at(`if-true`, `true`, stmt.line, block) + at(`if-false`, `false`, stmt.line, block) + at(`binary-swap`, `swap`, stmt.line, block) + at(`delete-call`, `fast`, stmt.line + 1, stmt.lines[1]) + at(`loop-false`, `loop`, stmt.line + 5, stmt.lines[5]) + at(`binary-swap`, `loopSwap`, stmt.line + 5, stmt.lines[5]) + at(`delete-call`, `push`, stmt.line + 6, stmt.lines[6]) + } + if (stmt.kind === `callShortcut`) { + condition() + at(`delete-call`, `fast`, stmt.line, `out.push('s' + x);`) + at(`delete-call`, `push`, stmt.line + 1, `out.push(String('s' + x));`) + at( + `value-undefined`, + `string`, + stmt.line + 1, + `out.push(String('s' + x));`, + ) + } if (stmt.kind === `elseIf`) { // An `if` with a production `else` is production; its probe branch // alone is probe-only. @@ -3261,6 +3362,13 @@ function ownersOf(plan) { function interpret(plan, fnId, x, sites, active) { const trace = [] const reached = new Set() + // The run's work: one for each site evaluation, as the capture counts a + // test's `calls` (capture-setup.mjs), which a work bound judges. + let work = 0 + const reach = (id) => { + work += 1 + reached.add(id) + } const outcomes = [] // The comparison sites whose two operators disagree on the operands this // run reached: the sites an unmutated run infects (capture-setup.mjs `cmp`). @@ -3272,7 +3380,7 @@ function interpret(plan, fnId, x, sites, active) { site !== undefined && active?.site !== undefined && active.site === site.id const condition = (stmt) => { for (const role of [`true`, `false`, `not`, `swap`]) - if (siteOf(stmt, role)) reached.add(siteOf(stmt, role).id) + if (siteOf(stmt, role)) reach(siteOf(stmt, role).id) const op = stmt.kind === `throwIf` ? `>` : stmt.op const k = stmt.kind === `throwIf` ? 100 : stmt.k const swap = siteOf(stmt, `swap`) @@ -3292,7 +3400,7 @@ function interpret(plan, fnId, x, sites, active) { } const write = (stmt) => { const site = siteOf(stmt, `push`) - reached.add(site.id) + reach(site.id) if (isActive(site)) return trace.push( active?.authored === stmt.label ? `s${stmt.label}!` : `s${stmt.label}`, @@ -3301,8 +3409,13 @@ function interpret(plan, fnId, x, sites, active) { const depLog = [] // Module state as a freshly loaded module has it: an empty cache, and // the labels `w_load` wrote at load, under the same mutant. - const memo = new Map() + // A function that runs after an earlier test of its file cached its + // input (`w_cold`) finds the entry there. + const memo = new Map( + plan.functions.find((f) => f.id === fnId)?.warm ? [[x, `warm`]] : [], + ) const loaded = () => { + // The module loaded outside any test, so its load counts no work. const call = sites.find((s) => s.role === `moduleLoad`) if (call) reached.add(call.id) if (isActive(call)) return [] @@ -3319,10 +3432,10 @@ function interpret(plan, fnId, x, sites, active) { if (stmt.kind === `loop`) for (let i = 0; ; i += 1) { const loopSite = siteOf(stmt, `loop`) - reached.add(loopSite.id) + reach(loopSite.id) if (isActive(loopSite)) break const swapSite = siteOf(stmt, `swap`) - reached.add(swapSite.id) + reach(swapSite.id) if (i === 2) infected.add(swapSite.id) if (!(isActive(swapSite) ? i <= 2 : i < 2)) break write(stmt) @@ -3333,15 +3446,15 @@ function interpret(plan, fnId, x, sites, active) { if (stmt.kind === `until`) { const loopSite = siteOf(stmt, `loop`) for (;;) { - reached.add(loopSite.id) + reach(loopSite.id) if (isActive(loopSite)) break const swapSite = siteOf(stmt, `swap`) - reached.add(swapSite.id) + reach(swapSite.id) if (trace.length === 3) infected.add(swapSite.id) if (!(isActive(swapSite) ? trace.length <= 3 : trace.length < 3)) break if (isActive(siteOf(stmt, `push`))) { - reached.add(siteOf(stmt, `push`).id) + reach(siteOf(stmt, `push`).id) throw new Spin() } write(stmt) @@ -3350,15 +3463,15 @@ function interpret(plan, fnId, x, sites, active) { // The workspace dependency's log, as a freshly loaded module has it. if (stmt.kind === `depNote`) { const call = siteOf(stmt, `note`) - reached.add(call.id) + reach(call.id) if (!isActive(call)) depLog.push(`d`) } if (stmt.kind === `depSeen`) { const call = siteOf(stmt, `push`) - reached.add(call.id) + reach(call.id) if (!isActive(call)) { const seenSite = siteOf(stmt, `seen`) - reached.add(seenSite.id) + reach(seenSite.id) // Spreading undefined throws. if (isActive(seenSite)) throw new TypeError(`seen() is undefined`) trace.push(...depLog) @@ -3367,22 +3480,22 @@ function interpret(plan, fnId, x, sites, active) { if (stmt.kind === `record`) { // A deleted push builds nothing; a dropped field never evaluates. const call = siteOf(stmt, `push`) - reached.add(call.id) + reach(call.id) if (!isActive(call)) { const string = siteOf(stmt, `string`) - reached.add(string.id) + reach(string.id) if (isActive(string)) trace.push(undefined) else { const record = {} const dropN = siteOf(stmt, `dropN`) - reached.add(dropN.id) + reach(dropN.id) if (!isActive(dropN)) { const arith = siteOf(stmt, `arith`) - reached.add(arith.id) + reach(arith.id) record.n = isActive(arith) ? x - 1 : x + 1 } const dropS = siteOf(stmt, `dropS`) - reached.add(dropS.id) + reach(dropS.id) if (!isActive(dropS)) record.s = `s${stmt.label}` trace.push(JSON.stringify(record)) } @@ -3391,22 +3504,22 @@ function interpret(plan, fnId, x, sites, active) { if (stmt.kind === `flag`) { // A deleted push never evaluates its literal. const call = siteOf(stmt, `push`) - reached.add(call.id) + reach(call.id) if (!isActive(call)) { // `String(...)` replaced by undefined never evaluates its literal. const string = siteOf(stmt, `string`) - reached.add(string.id) + reach(string.id) if (isActive(string)) trace.push(undefined) else { const flip = siteOf(stmt, `flag`) - reached.add(flip.id) + reach(flip.id) trace.push(String(isActive(flip) ? !stmt.value : stmt.value)) } } } if (stmt.kind === `memoSet`) { for (const role of [`true`, `false`, `not`, `has`]) - reached.add(siteOf(stmt, role).id) + reach(siteOf(stmt, role).id) let value = !(isActive(siteOf(stmt, `has`)) ? undefined : memo.has(x)) if (isActive(siteOf(stmt, `not`))) value = !value outcomes.push([stmt.line, value]) @@ -3419,22 +3532,22 @@ function interpret(plan, fnId, x, sites, active) { if (isActive(siteOf(stmt, `false`))) value = false if (value) { const set = siteOf(stmt, `memoSet`) - reached.add(set.id) + reach(set.id) if (!isActive(set)) memo.set(x, `s${stmt.label}`) } } if (stmt.kind === `memoGet` || stmt.kind === `pushLoaded`) { const site = siteOf(stmt, stmt.kind) - reached.add(site.id) + reach(site.id) if (!isActive(site) && stmt.kind === `pushLoaded`) trace.push(...loaded()) if (!isActive(site) && stmt.kind === `memoGet`) { const string = siteOf(stmt, `string`) - reached.add(string.id) + reach(string.id) if (isActive(string)) trace.push(undefined) else { const get = siteOf(stmt, `get`) - reached.add(get.id) + reach(get.id) trace.push(String(isActive(get) ? undefined : memo.get(x))) } } @@ -3443,30 +3556,29 @@ function interpret(plan, fnId, x, sites, active) { // only then. A plain read in place of `?.` throws on no object. if (stmt.kind === `optional`) { const drop = siteOf(stmt, `dropV`) - if (x) reached.add(drop.id) + if (x) reach(drop.id) const made = x ? isActive(drop) ? {} : { v: `p${stmt.label}` } : undefined const site = siteOf(stmt, `push`) - reached.add(site.id) + reach(site.id) if (!isActive(site)) { const chain = siteOf(stmt, `chain`) - reached.add(chain.id) + reach(chain.id) if (isActive(chain) && made === undefined) throw new TypeError(`Cannot read properties of undefined`) trace.push(String(made?.v)) } } - if (stmt.kind === `innerReturned`) - reached.add(siteOf(stmt, `returned`).id) + if (stmt.kind === `innerReturned`) reach(siteOf(stmt, `returned`).id) // A read of module data, as the module evaluated it at load under the // same mutant: a dropped field reads undefined, the swapped `+` gives // 1, and the flipped literal false. if (stmt.kind === `readData`) { const site = siteOf(stmt, `push`) - reached.add(site.id) + reach(site.id) if (!isActive(site)) { const at = (role) => isActive( @@ -3475,14 +3587,14 @@ function interpret(plan, fnId, x, sites, active) { // A method's call: undefined in its place, or its `1 + 2`. const called = () => { const value = siteOf(stmt, `value`) - reached.add(value.id) + reach(value.id) if (isActive(value)) return `undefined` // Its `return`: undefined in place of what it returns. const returned = sites.find((s) => s.role === `usedReturn`) - reached.add(returned.id) + reach(returned.id) if (isActive(returned)) return `undefined` const arith = sites.find((s) => s.role === `usedArith`) - reached.add(arith.id) + reach(arith.id) return String(isActive(arith) ? -1 : 3) } trace.push( @@ -3498,16 +3610,16 @@ function interpret(plan, fnId, x, sites, active) { } if (stmt.kind === `pushTag`) { const site = siteOf(stmt, `push`) - reached.add(site.id) + reach(site.id) if (!isActive(site)) trace.push(object.tag) } // `out.push($derived(x))`, where `$derived` returns `String(x)`. if (stmt.kind === `dollar`) { const call = siteOf(stmt, `push`) - reached.add(call.id) + reach(call.id) if (!isActive(call)) { const value = siteOf(stmt, `value`) - reached.add(value.id) + reach(value.id) if (isActive(value)) trace.push(undefined) else { const [inner] = plan.functions.find((f) => f.id === `$derived`).body @@ -3516,8 +3628,8 @@ function interpret(plan, fnId, x, sites, active) { // reached when `$derived` runs, and either gives `undefined`. const string = siteOf(inner, `string`) const returned = siteOf(inner, `returned`) - reached.add(string.id) - reached.add(returned.id) + reach(string.id) + reach(returned.id) trace.push( isActive(string) || isActive(returned) ? undefined : String(x), ) @@ -3529,14 +3641,55 @@ function interpret(plan, fnId, x, sites, active) { // is installed, so none of it writes output. if (stmt.kind === `nearProbe`) for (const site of sites) - if (site.stmt === stmt.label && !site.idle) reached.add(site.id) + if (site.stmt === stmt.label && !site.idle) reach(site.id) if (stmt.kind === `primeCall`) { const site = siteOf(stmt, `call`) - reached.add(site.id) + reach(site.id) if (!isActive(site)) run(`W7.prime`, object) } if ((stmt.kind === `if` || stmt.kind === `mixed`) && condition(stmt)) write(stmt) + // A shortcut for speed: an early return that pushes what the loop + // below it would count up to, for every input it takes. + if (stmt.kind === `shortcut`) { + if (condition(stmt)) { + const fast = siteOf(stmt, `fast`) + reach(fast.id) + if (!isActive(fast)) trace.push(`s${x}`) + return + } + let n = 0 + const loopSite = siteOf(stmt, `loop`) + const swapSite = siteOf(stmt, `loopSwap`) + for (;;) { + reach(loopSite.id) + if (isActive(loopSite)) break + reach(swapSite.id) + if (n === x) infected.add(swapSite.id) + if (!(isActive(swapSite) ? n <= x : n < x)) break + n += 1 + } + const push = siteOf(stmt, `push`) + reach(push.id) + if (!isActive(push)) trace.push(`s${n}`) + } + // A shortcut that skips a call: the `else` pushes the same text + // through `String`, one call more. + if (stmt.kind === `callShortcut`) { + if (condition(stmt)) { + const fast = siteOf(stmt, `fast`) + reach(fast.id) + if (!isActive(fast)) trace.push(`s${x}`) + } else { + const push = siteOf(stmt, `push`) + reach(push.id) + if (!isActive(push)) { + const string = siteOf(stmt, `string`) + reach(string.id) + trace.push(isActive(string) ? undefined : `s${x}`) + } + } + } if (stmt.kind === `elseIf` && !condition(stmt)) write(stmt) if (stmt.kind === `probeIf` || stmt.kind === `probeBlock`) condition(stmt) if (stmt.kind === `throwIf` && condition(stmt)) throw new Error(`boom`) @@ -3551,11 +3704,11 @@ function interpret(plan, fnId, x, sites, active) { `oldNe`, `oldCall`, ]) - reached.add(siteOf(stmt, role).id) - reached.add(sites.find((s) => s.role === `devBuild`).id) + reach(siteOf(stmt, role).id) + reach(sites.find((s) => s.role === `devBuild`).id) // Its `return`, whose undefined value leaves the probe uncalled, and // no output changes. - reached.add(sites.find((s) => s.role === `devBuildReturn`).id) + reach(sites.find((s) => s.role === `devBuildReturn`).id) outcomes.push([stmt.line, true]) // The guard is true, so its if-false site changes it; and its `!==` // compares 'test' with 'production', on which `!==` and `===` @@ -3565,7 +3718,7 @@ function interpret(plan, fnId, x, sites, active) { } if (referenceKinds.has(stmt.kind) || stmt.kind === `mixedHelperCall`) { const site = siteOf(stmt, `call`) - reached.add(site.id) + reach(site.id) if (isActive(site)) continue if (stmt.kind === `valueCall` || stmt.kind === `factoryCall`) { // `apply` calls the function it was given; `make` constructs the @@ -3574,7 +3727,7 @@ function interpret(plan, fnId, x, sites, active) { (s) => s.role === (stmt.kind === `valueCall` ? `applyCall` : `makeCall`), ) - reached.add(inner.id) + reach(inner.id) if (isActive(inner)) continue } if (stmt.kind === `newCall`) { @@ -3585,7 +3738,7 @@ function interpret(plan, fnId, x, sites, active) { run(stmt.to, object) } else if (stmt.kind !== `mixedHelperCall`) run(stmt.to) else { - reached.add(helperPush.id) + reach(helperPush.id) if (!isActive(helperPush)) trace.push(`h`) } } @@ -3599,7 +3752,7 @@ function interpret(plan, fnId, x, sites, active) { if (error instanceof Spin) spun = true else threw = true } - return { trace, threw, spun, reached, outcomes, infected } + return { trace, threw, spun, reached, outcomes, infected, work } } /** A loop that never ends, as the interpreter sees one. */ @@ -3622,9 +3775,18 @@ function runsOf(plan, tests, sites, active) { let changedAny = false let slowed = false const found = [] + // The test's work, summed over its campaigns, with no mutant active and + // under the mutant. A test that only reads what an earlier test stored + // calls no production code, so it does none. + let before = 0 + let after = 0 for (const part of partsOf(t)) { const base = interpret(plan, t.fn, part.input, sites) const mutated = interpret(plan, t.fn, part.input, sites, active) + if (!t.judgesShared) { + before += base.work + after += mutated.work + } const route = routeOf(t, part) // A loop that never ends cuts the file's run off, as a hang does. if (mutated.spun) @@ -3654,12 +3816,28 @@ function runsOf(plan, tests, sites, active) { return { t, route, verdicts: found, changed: true, slowed } } } + // A test that passed: its route is the last one it registered, and + // its work is judged against the bound it declared. An authored + // mutant's copy is not instrumented, so it counts no work, and the + // work law judges only operator mutants. + const route = routeOf(t, partsOf(t).at(-1)) + const judgesWork = + found.length === 0 && + active?.site !== undefined && + route !== `framework` return { t, - route: routeOf(t, partsOf(t).at(-1)), - verdicts: found, + route, + verdicts: + judgesWork && + t.work !== null && + t.registered && + after > t.work * before + ? [{ route, law: `work bound` }] + : found, changed: changedAny, slowed, + rose: judgesWork && after > before, } }) } @@ -3963,6 +4141,11 @@ function model(plan) { if (reaching.every((t) => t.flaky)) return `indeterminate` const kind = judge(plan, tests, sites, { site: site.id }) if (kind !== `survived`) return kind + // A test that reached it did more work, though nothing it observed + // changed: the mutant deleted a shortcut for speed (maintainer decision, + // 2026-11-12), which only a work bound sees. + if (runsOf(plan, reaching, sites, { site: site.id }).some((r) => r.rose)) + return `extra-work` // The survivor classes (classify.mjs): an `if` whose line one outcome // never took, a comparison no owner test infected, and otherwise an // infected site that no observation shows. @@ -4014,6 +4197,7 @@ function model(plan) { } const survivorKinds = new Set([ `not-judged`, + `extra-work`, `branch-never-taken`, `not-infected`, `not-observed`, @@ -4223,7 +4407,7 @@ function oracleTestFile(plan, file) { `const shared = { value: 0 }`, `const stored: { value: Array } = { value: [] }`, ...owns, - `const g = globalThis as { __oracleRoute?: (route: string) => void; __oracleObserve?: (c: string, v: unknown) => void; __oracleVerdict?: (v: { property: string; law: string; checkpoint: string }) => void }`, + `const g = globalThis as { __oracleRoute?: (route: string) => void; __oracleObserve?: (c: string, v: unknown) => void; __oracleVerdict?: (v: { property: string; law: string; checkpoint: string }) => void; __oracleWorkBound?: (ratio: number) => void }`, ...(plan.pragma && file === `a` ? [ `if ((globalThis as { __gapEnvironment?: string }).__gapEnvironment !== 'marked') throw new Error('the file ran outside the environment its pragma names')`, @@ -4253,6 +4437,10 @@ function oracleTestFile(plan, file) { const out = `out${i}` return [ ...(t.registered ? [` g.__oracleRoute?.('${part.route}')`] : []), + // The framework registers a work bound beside the route. + ...(t.registered && t.work !== null + ? [` g.__oracleWorkBound?.(${t.work})`] + : []), ` const ${out}: Array = []`, ...(t.setsShared ? [` shared.value = 5`] : []), t.readsShared diff --git a/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs b/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs index ff37b5a..a20cd82 100644 --- a/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs +++ b/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs @@ -168,9 +168,21 @@ * would judge such a verdict, is not built.) A copy that a batch skipped * because an earlier copy killed the mutant runs again, through the batch, * only while no kill is in hand, so every mutant gets an outcome and no - * other missing unit is rerun. A mutant that no owner test fails is a - * survivor: a `not-judged` when an owner test's observation or asserted - * value changed, a `branch-never-taken` when an `if` on its line never took + * other missing unit is rerun. A test's work is the number of site + * evaluations that its instrumented production code made (the capture's + * call count, never a time): a call statement, a call whose value is + * used, a comparison, an `if`, each pass of a loop's condition. A + * passing, deterministic owner test that registers a work bound beside + * its route (`__oracleWorkBound`, which `oracleProperty`'s `work` + * option calls) and whose work under the mutant is more than that ratio + * of its unmutated work fails the mutant under law `work bound` on its + * route, as a law verdict does (maintainer decision, 2026-11-12: a + * shortcut for speed is owed a cost law). A mutant that no owner test + * fails is a survivor: a `not-judged` when an owner test's observation + * or asserted value changed, an `extra-work` when no observation changed + * and an owner test that reached it, on a route that judges production, + * did more work (a shortcut whose deletion only costs time), a + * `branch-never-taken` when an `if` on its line never took * one of its outcomes in an owner test, a `not-infected` when a comparison's * operators agreed on every operand an owner test reached, a * `not-distinguished` when its operator changes a value only on some @@ -822,7 +834,7 @@ function namedScratch(text, names) { // A gap line the CLI streams to stderr as its mutant is judged (stream.mjs): // its class, its line, and its id. const streamedGap = - /^(?:not-judged|indeterminate|dead-code|outside-annotation|branch-never-taken|not-infected|not-observed|not-distinguished|unreached|random-only-kill|agreement-only-kill|unregistered-kill|framework-error|timeout|setup-error|crashed|does-not-apply|blocked) L(?:\d+|\?) (\S+) / + /^(?:not-judged|extra-work|indeterminate|dead-code|outside-annotation|branch-never-taken|not-infected|not-observed|not-distinguished|unreached|random-only-kill|agreement-only-kill|unregistered-kill|framework-error|timeout|setup-error|crashed|does-not-apply|blocked) L(?:\d+|\?) (\S+) / function readRun(dir, out, status, stderr, cli, stdout) { // The output names the CLI directory that ran and the package, as scratch @@ -1649,6 +1661,9 @@ async function driver(plan, observe) { }[m.outcome] if (named) states.add(named) if (m.class === `not-judged`) states.add(`not judged`) + if (m.class === `extra-work`) states.add(`extra work`) + if (m.outcome === `killed` && m.laws?.includes(`work bound`)) + states.add(`killed by a work bound`) if (m.class === `branch-never-taken`) states.add(`branch never taken`) if (m.class === `not-infected`) states.add(`not infected`) if (m.class === `not-observed`) states.add(`not observed`) @@ -2486,6 +2501,8 @@ const required = [ `framework-only failure`, `indeterminate`, `not judged`, + `extra work`, + `killed by a work bound`, `branch never taken`, `not infected`, `not observed`, diff --git a/packages/oracles/tests/oracle-property-self-oracle-fixture.ts b/packages/oracles/tests/oracle-property-self-oracle-fixture.ts index 7a34ebd..837451e 100644 --- a/packages/oracles/tests/oracle-property-self-oracle-fixture.ts +++ b/packages/oracles/tests/oracle-property-self-oracle-fixture.ts @@ -213,6 +213,8 @@ export type OddKind = (typeof oddKinds)[number] /** The suite a `nested` spec registers its property in. */ export const outerSuite = `outer (nested)` +/** The ratio of a `bound` spec's work bound. */ +export const WORK_RATIO = 1.25 /** The extra text a `shell` label carries: a shell expands or ends at each. */ export const shellText = `$HOME \`true\` 'q' "q" \\` @@ -278,6 +280,8 @@ export type SyntheticSpec = { predictMode?: `missing` | `extra` /** The `owns` option as the gap report reads it: a list, an empty list, a string, or a list with a non-string entry. */ ownsMode?: `list` | `empty` | `string` | `bad entry` + /** The `work` option: a bound of `WORK_RATIO`, a ratio below 1, or one that is not finite. */ + workMode?: `bound` | `below one` | `not finite` /** Runs every history on a controlled clock, where a 7 spins it. */ clock?: boolean /** How the property holds a history's items; the array itself when absent. */ @@ -400,8 +404,14 @@ let route: string | undefined globalThis.__oracleRoute = (registered) => { route = registered } +// The work bound registered for the running test, as the gap report reads it. +let work: number | undefined +globalThis.__oracleWorkBound = (ratio) => { + work = ratio +} beforeEach(() => { route = undefined + work = undefined }) // Every slow driver run this file started. A slow spec's histories time out // and their judgments arrive late; its reach test starts only after every @@ -699,7 +709,7 @@ function syntheticDriver(spec: SyntheticSpec, mutant?: `first` | `last`) { if (log) appendFileSync( log, - `${JSON.stringify({ id: spec.id, test: expect.getState().currentTestName ?? ``, history, route: route ?? null })}\n`, + `${JSON.stringify({ id: spec.id, test: expect.getState().currentTestName ?? ``, history, route: route ?? null, work: work ?? null })}\n`, ) if (spec.odd) observe(`odd`, { reason: oddValue(spec.odd) }) if (spec.slow) await new Promise((resolve) => setTimeout(resolve, 300)) @@ -886,6 +896,21 @@ function syntheticCalibrations( ) } +// `work`, a work bound, in the shape the spec names. +const workOf = (spec: SyntheticSpec) => + spec.workMode === undefined + ? {} + : { + work: { + ratio: + spec.workMode === `bound` + ? WORK_RATIO + : spec.workMode === `below one` + ? 0.5 + : Number.POSITIVE_INFINITY, + }, + } + // `owns`, the literal list the gap report reads, in the shape the spec names. const ownsOf = (spec: SyntheticSpec) => spec.ownsMode === undefined @@ -1003,6 +1028,7 @@ export function syntheticProperty(spec: SyntheticSpec): void { }, ...(spec.timeout !== undefined ? { timeout: spec.timeout } : {}), ...ownsOf(spec), + ...workOf(spec), } const law = { statement: `A synthetic history holds no 3 and no 4.`, diff --git a/packages/oracles/tests/oracle-property-self-oracle.test.ts b/packages/oracles/tests/oracle-property-self-oracle.test.ts index 6eb8f26..ce02f75 100644 --- a/packages/oracles/tests/oracle-property-self-oracle.test.ts +++ b/packages/oracles/tests/oracle-property-self-oracle.test.ts @@ -17,9 +17,11 @@ * distinct histories distinct test names. A property fails to load when its id is not a registered * oracle property, a required state has no constructed history, a constructed history names an * undeclared state, a fixed or constructed history is one the arbitrary cannot take back (law - * `membership`), or `owns` is not a non-empty literal list of strings (the gap report reads it - * statically). When several of the named load errors apply, the declaration's shape reports - * first, in this order: an unregistered id, then `owns`, then `predict` on an enumerated source + * `membership`), `owns` is not a non-empty literal list of strings (the gap report reads it + * statically), or a work bound's `ratio` is not a finite number of at least 1 (a smaller one would + * reject a mutant that does no more work than the unmodified code). When several of the named load + * errors apply, the declaration's shape reports first, in this order: an unregistered id, then + * `owns`, then `work`, then `predict` on an enumerated source * (only a generated source declares states), then a history that is not data (`data`). A load * fails with its first error alone. This precedence is analyst-chosen (2026-11-10): no contract * text or maintainer decision orders these errors. The source kind decides which checks reach the @@ -181,6 +183,11 @@ * history `case`; calibrations, probe calibrations, and the determinism test * `calibration`, and so do the determinism units (TanStack/oracles#1 coordinator decisions, 2026-11-08: a * route describes the campaign that runs, in one vocabulary). + * A property that declares a work bound (`work: { ratio }`) also registers its ratio through + * `__oracleWorkBound` in every test whose route judges production, and in none whose route is + * `calibration`; a property without one registers none. The gap report reads it to judge the + * test's work under a mutant; a plain run has no unmodified run to compare with, so there the + * bound judges nothing (maintainer decision, 2026-11-12: a speed shortcut is owed a cost law). * - `seed-log`: a property logs exactly one `ORACLE_SEED` line, in its first * random unit, and no other test logs one; every random unit runs that seed * (maintainer decision, 2026-11-11, Q2: one seed per property per process). A @@ -265,6 +272,7 @@ import { shellText, syntheticArbitrary, syntheticIds, + WORK_RATIO, } from './oracle-property-self-oracle-fixture.js' import type { Verdict } from '../src/index.js' import type { @@ -361,6 +369,8 @@ type Draft = { predictMode: `match` | `missing` | `extra` /** The `owns` option: absent, a literal list, a string, or a list with a non-string entry. */ ownsMode: `none` | `list` | `empty` | `string` | `bad entry` + /** The `work` option: absent, a bound, a ratio below 1, or a ratio that is not finite. */ + workMode: `none` | `bound` | `below one` | `not finite` clock: boolean form: `plain` | `bigint` | `map` | `lossy` | `error` | `errors` | `boxed` | `trap` @@ -587,6 +597,15 @@ const draft: fc.Arbitrary = fc.record({ arbitrary: fc.constantFrom(`missing` as const, `extra` as const), }, ), + // A work bound's ratio is a finite number of at least 1. + workMode: fc.oneof( + { weight: 6, arbitrary: fc.constant(`none` as const) }, + { weight: 3, arbitrary: fc.constant(`bound` as const) }, + { + weight: 1, + arbitrary: fc.constantFrom(`below one` as const, `not finite` as const), + }, + ), // `owns` is read statically by the gap report, so it must be a literal list of strings. ownsMode: fc.oneof( { weight: 6, arbitrary: fc.constant(`none` as const) }, @@ -857,6 +876,7 @@ function resolveSuite({ drafts, seedless, prefix, environment }: Suite): Plan { ...(d.noisyProbe ? { noisyProbe: true } : {}), ...(d.predictMode !== `match` ? { predictMode: d.predictMode } : {}), ...(d.ownsMode !== `none` ? { ownsMode: d.ownsMode } : {}), + ...(d.workMode !== `none` ? { workMode: d.workMode } : {}), ...(d.timing === `slow` ? { slow: true, timeout: 100 } : {}), ...(d.timing === `slow shrink` ? { slowShrink: true, timeout: 5_000 } @@ -977,6 +997,8 @@ type LogEntry = { history: SyntheticHistory /** The route registered for the test when the history ran. */ route: string | null + /** The work bound registered for the test when the history ran. */ + work: number | null } const allowed = (item: number) => [0, 1, 2, 3, 4, 6, 7].includes(item) @@ -1516,6 +1538,11 @@ function predictSpec( loads: false, loadError: `${spec.id}: \`owns\` must be a non-empty literal list of strings`, } + if (spec.workMode === `below one` || spec.workMode === `not finite`) + return { + loads: false, + loadError: `${spec.id}: \`work.ratio\` must be a finite number of at least 1`, + } if (!generated && spec.predictMode === `extra`) return { loads: false, @@ -1960,6 +1987,8 @@ function owedRoute(name: string, replay = false): string { // every run; campaigns count them. const frameworkStates = [ `registration-error`, + `work-bound-registered`, + `work-bound-left-out-of-calibration`, `campaign-violation`, `campaign-setup-failure`, `slow-shrink-campaign-failure`, @@ -2327,6 +2356,16 @@ function frameworkReach( for (const spec of plan.specs) if (spec.overReport && spec.probe && expected.get(spec.id)!.loads) reached.add(`over-reported-reach`) + // A work bound registered beside a route that judges production, and a + // calibration of the same property that registers none. + for (const spec of plan.specs) + if (spec.workMode === `bound` && expected.get(spec.id)!.loads) { + const entries = (run?.log ?? []).filter((entry) => entry.id === spec.id) + if (entries.some((entry) => entry.work !== null)) + reached.add(`work-bound-registered`) + if (entries.some((entry) => entry.route === `calibration`)) + reached.add(`work-bound-left-out-of-calibration`) + } if (trace.some(({ checkpoint }) => checkpoint === `reach-alone`)) reached.add(`enumerated-reach-alone`) for (const { checkpoint, value } of trace) @@ -3214,7 +3253,7 @@ function checkSuite( if (seen.loaded) return violation( `run`, - `${spec.id} loaded, but its id is unregistered, a required state has no constructed history, a constructed history names an undeclared state, two of its tests would share a name, a fixed or constructed history is one the arbitrary cannot take back, \`owns\` is not a non-empty literal list of strings, or a history is not data`, + `${spec.id} loaded, but its id is unregistered, a required state has no constructed history, a constructed history names an undeclared state, two of its tests would share a name, a fixed or constructed history is one the arbitrary cannot take back, \`owns\` is not a non-empty literal list of strings, a work bound's ratio is not a finite number of at least 1, or a history is not data`, law, ) const owedText = [owed.loadError, owed.loadFix].filter( @@ -3579,7 +3618,8 @@ function checkSuite( `agreement`, { actual: late, expected: owedLate }, ) - // Every history runs under its test's route. + // Every history runs under its test's route, and with its property's + // work bound registered unless its route is the framework's own. for (const spec of plan.specs) { if (!expected.get(spec.id)!.loads || spec.slow) continue for (const entry of run.log.filter(({ id }) => id === spec.id)) { @@ -3591,6 +3631,17 @@ function checkSuite( `route`, { actual: entry.route, expected: owedRoute(test) }, ) + const owedWork = + spec.workMode === `bound` && owedRoute(test) !== `calibration` + ? WORK_RATIO + : null + if (entry.work !== owedWork) + return violation( + `run`, + `${spec.id} "${test}" ran ${JSON.stringify(entry.history)} with work bound ${JSON.stringify(entry.work)} registered, owed ${JSON.stringify(owedWork)}`, + `route`, + { actual: entry.work, expected: owedWork }, + ) } } for (const { checkpoint, probe } of probes) { @@ -3798,6 +3849,7 @@ const generatedDraft = (overrides: Partial): Draft => ({ noisyProbe: false, predictMode: `match`, ownsMode: `none`, + workMode: `none`, clock: false, form: `plain`, zeroNoise: false, @@ -4284,6 +4336,21 @@ oracleProperty({ [], generatedDraft({ ownsMode: `list` }), ), + fixedHistory( + `a work bound below 1 fails to load`, + [`registration-error`], + generatedDraft({ workMode: `below one` }), + ), + fixedHistory( + `a work bound that is not finite fails to load`, + [`registration-error`], + generatedDraft({ workMode: `not finite` }), + ), + fixedHistory( + `a work bound is registered beside each route that judges production`, + [`work-bound-registered`, `work-bound-left-out-of-calibration`], + generatedDraft({ workMode: `bound`, calibrations: [cal(`right`)] }), + ), fixedHistory( `predict needs a probe-bound state`, [`registration-error`], From 075d360539a570eee2b0b49585fa9b783639ea90 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Thu, 12 Nov 2026 10:17:39 -0700 Subject: [PATCH 02/20] fix(oracle-gaps): report a shortcut for speed as extra-work, and let a work bound kill it A survivor that changed no observation but made a test do more work (site evaluations, the capture's call count) is now extra-work, not not-observed. Its directive names the test, its work before and after, and the work bound that rejects the mutant. oracleProperty takes work: { ratio }, which it registers beside each route that judges production; under the gap CLI a test whose work under a mutant passes that ratio of its unmutated work kills it under law work bound. A plain Vitest run has no unmutated run, so there the bound judges nothing. Co-authored-by: Isaac --- .changeset/gap-cli-extra-work.md | 5 ++ README.md | 20 +++++ .../oracles/oracle-gaps/capture-setup.mjs | 16 +++- packages/oracles/oracle-gaps/classify.mjs | 83 ++++++++++++++++--- packages/oracles/oracle-gaps/report.mjs | 11 +++ packages/oracles/src/oracle-property.ts | 25 +++++- 6 files changed, 148 insertions(+), 12 deletions(-) create mode 100644 .changeset/gap-cli-extra-work.md diff --git a/.changeset/gap-cli-extra-work.md b/.changeset/gap-cli-extra-work.md new file mode 100644 index 0000000..a59e431 --- /dev/null +++ b/.changeset/gap-cli-extra-work.md @@ -0,0 +1,5 @@ +--- +'@tanstack/oracles': minor +--- + +The `oracle-gaps` report no longer reports a shortcut for speed as `not-observed`. A survivor that changed no observation but made a test do more work is now `extra-work`. Before, its directive said to observe a value or widen the oracle, which cannot work for a branch whose fast and general paths give the same result, and an author who removed the code instead made it slower. Work is the number of site evaluations that the test's instrumented production code makes, a deterministic count, never a time. The `extra-work` directive names the test, its work before and after, and the bound to declare. `oracleProperty` takes a new `work: { ratio }` option. Under the gap report, a mutant whose work in one of the property's tests is more than `ratio` times the unmodified code's work is killed under the law `work bound`. `ratio` must be a finite number of at least 1. A plain Vitest run has no unmodified run to compare with, so there the bound judges nothing. diff --git a/README.md b/README.md index 98d8831..e749ed9 100644 --- a/README.md +++ b/README.md @@ -52,6 +52,26 @@ every commit with [pkg.pr.new](https://pkg.pr.new). in its `detail` in `results.json`. Record the error in the driver, and state the law that decides that input's outcome. + A survivor that changed no observation but made a test do more work is + `extra-work`: the mutant deleted a shortcut that exists only for speed, + such as an early return or an assignment in place of a slower call. A + test's work is the number of site evaluations that its instrumented + production code makes (calls, comparisons, `if`s, and each pass of a + loop's condition), a deterministic count, never a time. A test whose work + varied between the two unmutated runs, or that is one of the framework's + own checks, does not judge work. The gap's directive names the first such + test, its work before and after, and the bound that rejects the mutant. + Declare it on the owning property, for example + `oracleProperty({ ..., work: { ratio: 1.1 } })`: a mutant whose work in + one of the property's tests is more than `ratio` times the unmodified + code's work there is killed under law `work bound`. `ratio` is a finite + number of at least 1, and 1 rejects any extra work. A plain Vitest run has + no unmodified run to compare with, so there the bound judges nothing. The + count does not see work without a site, such as a plain assignment, a + `for...of` pass, or a builtin's own work, so a call to a builtin counts + once, whatever it costs. An authored mutant's copy is not instrumented, so + a work bound does not judge it. + Exit codes: 0 the check completed, whatever its gaps (the last stdout line is `completed: gaps`); 2 the run was refused, with `refused:` and `fix:` on stderr; 3 an internal error of the CLI; 130 or 143 stopped by diff --git a/packages/oracles/oracle-gaps/capture-setup.mjs b/packages/oracles/oracle-gaps/capture-setup.mjs index 81a4388..6af40e3 100644 --- a/packages/oracles/oracle-gaps/capture-setup.mjs +++ b/packages/oracles/oracle-gaps/capture-setup.mjs @@ -252,7 +252,13 @@ const active = state.mutant // copy's budget (`callBudget`, from check.mjs: a hundred times the most calls // any unmutated test of the file made, and at least 100,000), the copy throws // from every helper call and writes a spin row, and the batch reports it as a -// timeout. A count, unlike a clock, does not depend on load. +// timeout. A count, unlike a clock, does not depend on load. It is also the +// test's work, which a work bound judges (classify.mjs): each site +// evaluation counts once, so a call statement, a call whose value is used, a +// comparison, a logical operator, an `if`, and each pass of a loop's +// condition count, and code with no site (a plain assignment, a `for...of` +// pass, a builtin's own work, code outside the instrumented sources) counts +// nothing. let calls = 0 let spun = false const hit = (id) => { @@ -656,6 +662,12 @@ let route globalThis.__oracleRoute = (registered) => { if (inTest) route = registered } +// The work bound the current test's property declares (oracle-property.ts +// `work`): classify.mjs judges the test's work, its `calls`, against it. +let workBound +globalThis.__oracleWorkBound = (ratio) => { + if (inTest) workBound = ratio +} globalThis.__oracleObserve = (checkpoint, value) => { observed.update(`${checkpoint}(${render(value)})\n`) @@ -695,6 +707,7 @@ beforeEach((context) => { histories = 0 siteHistories = new Map() route = undefined + workBound = undefined inTest = true }) @@ -734,6 +747,7 @@ function finishTest(context) { outcomes: [...outcomes], verdicts: [...verdicts.values()], route, + work: workBound, ms: Math.round(performance.now() - testStarted), // The test's own result, so a batch cut short by a hang still has the // results of the copies that finished. diff --git a/packages/oracles/oracle-gaps/classify.mjs b/packages/oracles/oracle-gaps/classify.mjs index 0f95b7c..bb3e0f4 100644 --- a/packages/oracles/oracle-gaps/classify.mjs +++ b/packages/oracles/oracle-gaps/classify.mjs @@ -36,12 +36,16 @@ function routeOf(registered) { /** * A mutant's outcome from its units. A test whose observations varied * between the two unmutated runs is nondeterministic: its failures judge - * nothing, so a mutant that only such tests fail is not killed. + * nothing, so a mutant that only such tests fail is not killed. A passing + * test for which `pastBound(key, row)` holds did more work than its + * property's work bound allows, a verdict under law `work bound` on its + * route (`judgeMutant` decides it; the runner's stop at a kill reads none). */ export function outcomeOf( units, nondeterministic = new Set(), assertions = false, + pastBound = () => false, ) { // A run that timed out recorded nothing. A kill in another run still // decides; failing that, the mutant is `timeout`. @@ -57,9 +61,19 @@ export function outcomeOf( for (const message of unit.setupErrors) failures.push({ setup: true, message }) for (const test of unit.tests) { - if (test.state !== `failed`) continue - if (nondeterministic.has(test.key)) continue const row = rows.get(test.key) + if (test.state !== `failed`) { + if (test.state === `passed` && pastBound(test.key, row)) { + const route = routeOf(row.route) + failures.push({ + test: test.fullName, + route, + verdicts: [{ law: `work bound`, route }], + }) + } + continue + } + if (nondeterministic.has(test.key)) continue // Each verdict keeps the route active when it fired. const verdicts = (row?.verdicts ?? []).map((v) => ({ ...v, @@ -236,7 +250,18 @@ export function baselineOf(firstRows, secondRows) { for (const id of row.hits) reachedBy.set(id, [...(reachedBy.get(id) ?? []), key]) const rank = new Map([...firstRows.keys()].map((key, i) => [key, i])) - return { firstRows, nondeterministic, reachedBy, rank } + // Each deterministic test's work, the site evaluations its production + // code made (capture-setup.mjs `calls`), where the two runs agree: only + // such a test's work judges a mutant. + const work = new Map() + for (const [key, row] of firstRows) + if ( + !row.background && + !nondeterministic.has(key) && + secondRows.get(key)?.calls === row.calls + ) + work.set(key, row.calls) + return { firstRows, nondeterministic, reachedBy, rank, work } } /** @@ -343,7 +368,10 @@ const changedIn = (rows, keys, baseline) => * `unreached`. A survivor gets one class from what the run measured (the * classes are in the design's decision 11): `not-judged` (a changed value * no law rejected), `indeterminate` (only nondeterministic tests reached - * it), `dead-code` (no input distinguishes it), `outside-annotation` (no + * it), `extra-work` (no observation changed, and a test did more work; + * `work` names the first such test, its work before and after, and the + * work bound that rejects the mutant), `dead-code` (no input distinguishes + * it), `outside-annotation` (no * input of the annotations distinguishes it; `input` names the input class * outside them that does), `branch-never-taken` (an * `if` one of whose outcomes never occurred; `wanted` names the missing @@ -355,11 +383,6 @@ const changedIn = (rows, keys, baseline) => export function judgeMutant(mutant, { owners, targets, units, baseline }) { if (owners.size === 0) return { outcome: `unowned` } if (targets.oracleKeys.length === 0) return { outcome: `unreached` } - const result = { - ...outcomeOf(units, baseline.nondeterministic), - seconds: units.reduce((sum, u) => sum + u.seconds, 0), - } - if (result.outcome !== `survived`) return result const rows = new Map(units.flatMap((u) => u.rows ?? [])) // Only owner tests that reach the site can judge it. Code that ran when a // file loaded the module reaches every test of that file (maintainer @@ -374,13 +397,53 @@ export function judgeMutant(mutant, { owners, targets, units, baseline }) { !rows.get(k).background && !baseline.nondeterministic.has(k), ) + // A compared test's work judges the mutant when the two unmutated runs + // agree on it and its route judges production, not the framework's own + // checks; it rose when the mutant's run did more. + const rose = (k) => + baseline.work.has(k) && + routeOf(rows.get(k).route) !== `framework` && + rows.get(k).calls > baseline.work.get(k) + const compares = new Set(compared) + const result = { + ...outcomeOf( + units, + baseline.nondeterministic, + false, + (k, row) => + compares.has(k) && + rose(k) && + row.work !== undefined && + row.calls > row.work * baseline.work.get(k), + ), + seconds: units.reduce((sum, u) => sum + u.seconds, 0), + } + if (result.outcome !== `survived`) return result result.changedTests = changedIn(rows, compared, baseline) + const roseIn = inReportOrder(compared.filter(rose), baseline)[0] if (result.changedTests.length > 0) result.class = `not-judged` else if (compared.length === 0) { result.class = `indeterminate` result.nondeterministicBy = targets.oracleKeys.filter((k) => baseline.nondeterministic.has(k), ) + } else if (roseIn !== undefined) { + // No observation changed, and the mutant's run did more work: it + // deleted a shortcut that only a work bound sees (maintainer decision, + // 2026-11-12). The bound to declare is the largest ratio in hundredths + // below the measured one, and 1 when the test did no work before. + const before = baseline.work.get(roseIn) + const after = rows.get(roseIn).calls + result.class = `extra-work` + result.work = { + test: roseIn, + before, + after, + ratio: before > 0 ? (after / before).toFixed(2) : undefined, + bound: ( + (before > 0 ? Math.floor((100 * after - 1) / before) : 100) / 100 + ).toFixed(2), + } } else if (mutant.evidence === `dead code`) result.class = `dead-code` else if (mutant.outside) { // No input of the annotations distinguishes it: the type analysis names diff --git a/packages/oracles/oracle-gaps/report.mjs b/packages/oracles/oracle-gaps/report.mjs index 7a7d1dd..3eb34f5 100644 --- a/packages/oracles/oracle-gaps/report.mjs +++ b/packages/oracles/oracle-gaps/report.mjs @@ -114,6 +114,10 @@ const directive = { 'not-distinguished': (g) => distinguishingDirective(g), 'not-judged': (g) => `Make a law in \`${testName(g.changedTests?.[0])}\` reject the changed value.`, + // No observation changed, and a test did more work: a shortcut for speed, + // which only a cost law sees (maintainer decision, 2026-11-12). + 'extra-work': ({ work }) => + `Declare \`work: { ratio: ${work.bound} }\` on the \`oracleProperty\` that registers \`${testName(work.test)}\`, so that its work bound rejects this mutant: there it changed no observation, and its work rose from ${work.before} to ${counted(work.after, `site evaluation`)}${work.ratio === undefined ? `` : ` (${work.ratio}× the unmodified run's)`}.`, indeterminate: (g) => g.nondeterministicBy?.length ? `Make \`${g.nondeterministicBy.map(testName).join(`, `)}\` deterministic: control its clock and its random seed, so its failures judge this code.` @@ -194,6 +198,13 @@ const placeholders = { operator: ``, input: ``, blockedBy: [``], + work: { + bound: ``, + test: ` > `, + before: ``, + after: ``, + ratio: ``, + }, } const legendOf = (kind) => `${meanings[kind] ?? ``}${directive[kind]({ ...placeholders, outcome: kind })}` diff --git a/packages/oracles/src/oracle-property.ts b/packages/oracles/src/oracle-property.ts index ca6b59e..037199d 100644 --- a/packages/oracles/src/oracle-property.ts +++ b/packages/oracles/src/oracle-property.ts @@ -677,6 +677,18 @@ export interface OraclePropertyOptions { * `owns`). */ owns?: ReadonlyArray + /** + * A work bound (maintainer decision, 2026-11-12: a branch that exists only + * for speed is owed a cost law). A test's work is the number of site + * evaluations that its instrumented production code makes, a count, never + * a time. Under the gap report, a mutant whose work in one of this + * property's tests is more than `ratio` times the unmodified code's work + * in that test fails it under law `work bound`, so a mutant that deletes a + * shortcut, and changes no observation, is killed. A plain Vitest run has + * no unmodified run to compare with, so there the bound judges nothing. + * `ratio` is a finite number of at least 1; 1 rejects any extra work. + */ + work?: { ratio: number } } declare global { @@ -691,6 +703,9 @@ declare global { var __oracleVerdict: | ((verdict: { property: string; law: string; checkpoint: string }) => void) | undefined + // Installed by the gap report's test setup, absent otherwise. Each test + // whose route judges production reports its property's work bound. + var __oracleWorkBound: ((ratio: number) => void) | undefined } /** The probe calls one history made: which probes it reached and which it entered. */ @@ -765,7 +780,7 @@ export function oracleProperty( // Every source names a registered property, so its replay and manifest // entries exist; an enumerated source has no random campaign to check it. assertRegisteredOracleProperty(options.id) - const { id, histories, owns } = options + const { id, histories, owns, work } = options if ( owns !== undefined && !( @@ -777,6 +792,11 @@ export function oracleProperty( throw new Error( `${id}: \`owns\` must be a non-empty literal list of strings, which the gap report reads statically.`, ) + // A ratio below 1 would reject a mutant that does no more work. + if (work !== undefined && !(Number.isFinite(work.ratio) && work.ratio >= 1)) + throw new Error( + `${id}: \`work.ratio\` must be a finite number of at least 1, the most work a mutant may do as a multiple of the unmodified code's.`, + ) const enumerated: EnumeratedHistories | undefined = Array.isArray( histories, ) @@ -817,6 +837,9 @@ export function oracleProperty( test.name, ({ task, signal }) => { oracleRoute(test.route) + // The framework's own checks judge the oracle, not production. + if (work && test.route !== `calibration`) + globalThis.__oracleWorkBound?.(work.ratio) return test.run(task, signal) }, timeoutOf(options), From f83fff6f2ce171dde9ddf6ed7e22c34ac6104d69 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Thu, 12 Nov 2026 11:44:25 -0700 Subject: [PATCH 03/20] test(oracle-gaps): a selection must reproduce the whole file's work The warm-cache witness w_cold moves to the end of file a, after every nondeterministic test, so its selection is otherwise reproducible. Its drop-not mutant writes to the cache again in the whole file (extra-work), but a selection that skips w_unseen's warming test finds the cache cold and does no extra work. RED on the parent: src/sut.ts:w_cold:231:drop-not#0: not-observed, owed extra-work. The e2e limits now state what the work law covers. Co-authored-by: Isaac --- .../oracle-gaps/gap-report-model-oracle.mjs | 30 ++++++++++--------- .../gap-report-oracle.property.test.mjs | 2 +- .../tests/oracle-property-self-oracle.test.ts | 2 +- 3 files changed, 18 insertions(+), 16 deletions(-) diff --git a/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs b/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs index a1eaa70..481b1e2 100644 --- a/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs +++ b/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs @@ -911,20 +911,6 @@ const witness = [ body: [{ kind: `memoSet`, label: 21 }], tests: [judging(`a`, `fixed seed`, 4)], }, - // A cache that an earlier test of the file warmed: `w_unseen`'s test - // caches under 4 first, so in the whole file `w_cold`'s write is - // skipped. A run of the tests from `w_cold`'s on, without the one that - // warmed the cache, observes the same and does one call more, so the - // runner owes a selection only where it reproduces the whole file's work - // as well as its observations. Its `drop-not` mutant writes again in the - // whole file (`extra-work`), and writes nothing in such a selection. - { - id: `w_cold`, - owners: [`a`], - warm: true, - body: [{ kind: `memoSet` }], - tests: [judging(`a`, `fixed seed`, 4)], - }, // Shortcuts for speed (klona's early return for a primitive, and its // assignment in place of `Object.defineProperty`): the general path gives // the same output for every input, with more work, so forcing the @@ -1657,6 +1643,22 @@ const witness = [ body: [kill], tests: [judging(`a`, `fixed seed`, 7, { asProp: true })], }, + // A cache that an earlier test of the file warmed: `w_unseen`'s test + // caches under 4 first, so in the whole file `w_cold`'s write is + // skipped. A run of the tests from `w_cold`'s on, without the one that + // warmed the cache, observes the same and does one call more, so the + // runner owes a selection only where it reproduces the whole file's work + // as well as its observations. Its `drop-not` mutant writes again in the + // whole file (`extra-work`), and writes nothing in such a selection. + // It is the last test of its file, so its selection holds no + // nondeterministic test, which would fail any selection to reproduce. + { + id: `w_cold`, + owners: [`a`], + warm: true, + body: [{ kind: `memoSet` }], + tests: [judging(`a`, `fixed seed`, 4)], + }, { id: `wg0`, file: LIB, owners: [], body: [push], tests: [] }, { id: `wg1`, file: LIB, owners: [], body: [push], tests: [] }, { id: `wg2`, file: LIB, owners: [], body: [push], tests: [] }, diff --git a/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs b/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs index c202d7a..536fccc 100644 --- a/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs +++ b/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs @@ -2748,7 +2748,7 @@ oracleProperty({ law: { statement: `The gap report enumerates the promised operator mutants, leaves out exactly the probe-only code, discovers every oracle that owns a function's call tree and refuses broken ownership, judges each mutant only by its owners' tests on their registered routes and counts only law verdicts on reproducible routes as kills, which outrank a run that never ends, anchors each mutant by its function and normalized statement, lists every gap and unowned mutant once with consistent counts, keeps keys stable under unrelated edits, reports through one daemon per package what a cold run reports at every state of a sequence of edits, and bounds the runs that confirm a mutant's cut-off copies by one budget per mutant and names each busy mutant in a heartbeat while it runs, leaves nothing in the package and writes nothing where Vitest's include could find it while it runs, leaves no process running once it is stopped, runs the package as Vitest does from any install path and any --source spelling: one instance of each workspace dependency, no refusal over an unbuildable devDependency, child processes for a native addon anywhere in the tree, a capture that a test's builtin spies cannot see, and a refusal of test.projects; leads its report with every test file's baseline, blocks only the mutants a failing oracle file owns and judges the rest, and every run that does not complete ends, cold or through the daemon, with a refusal that exits 2, prints a refused: block and one fix: directive with no stack trace, and writes the refusal to the run's output directory.`, authority: `The maintainer's decisions in design runs v11, v14, and v19 (2026-11-04 and 2026-11-05), not in this package's records, including that a kill by reach agreement alone is a gap and that an incremental run through the daemon equals a cold one (v19.3); the project's oracle guide on quiet exclusions; the oracle rubric's outcome kinds (ORC-006); the CLI's documented operator list in schemata.mjs; the maintainer's dispositions G1, G2, G4, G5, G6, and G13 of the framework papercut ledger (2026-11-11) and the v20 freeze's namespacing, one-loop, and provisional-report decisions (2026-11-10).`, - limits: `Law 12 judges refusals only: no legal history reaches an internal error, so its shape (exit 3, the stack in error.log) is not judged; argument refusals are the static oracle's; the check's rejection of a stack trace, a missing fix, and a wrong exit code is calibrated in the static oracle, through the same refusalBreach, and here only a refused run reported as complete is calibrated, because each calibration here runs real CLI histories. Hangs only as an owner test that awaits a promise that never settles, under a 1 s first budget and a 20 s full limit, and a slow run only as two witnesses whose tests wait where the output changed, 3 s in one that then judges and 10 s in two one-test owners of one that does not; busy loops only on instrumented code (cut by the call budget), no slow runs past the full limit that would end, hangs in ordinary or nondeterministic tests, setup errors other than a throw at module load, loops, multi-line conditions, two functions on one line, or await, new, and void statements; nested functions, object-literal methods, accessors, and class-field arrows only where no test calls them (their ownership is the static oracle's); two source files of one package, with no re-exports, namespace imports, or calls into another package; --only only in a stopped run and a partial run of one function's mutants, and no --sample or --diff-base; the baseline's failures only as one failing owner test that throws an error and a setup file that throws, so a baseline law violation's law and counterexample lines are the classify oracle's; a derived class's file is instrumented and its sites are judged as unowned, but no test imports or constructs the class, so its no-mutant and super-region behavior is the static oracle's (in-process), not this oracle's; package configs only as vite.config or vitest.config with one of the extensions .ts, .mts, .cts, .js, .mjs, and .cjs (not every pair), as config/gap-vitest with the same extension through --config, or none, as an object or a sync or async function, with no plugins of their own and test.projects only as a refusal, and only under the package's own package.json; a top-level await only in an ES module config and require only in a CommonJS one, as Vitest itself loads no other; a --config outside the package is refused by design, through the rule for paths outside the package (args.mjs), and no history generates one; contexts only one, four, or eleven, and process warnings judged only as MaxListenersExceededWarning on the CLI's own stderr (a daemon's process warnings go to its log); a context that fails to open gives back its listener allowance, which no observation shows (a kept allowance only raises the limit, so no warning follows), so that accounting is fixed by inspection; no probe-bound states, so reach agreement's killing of probe mutants is not checked; no gap answers or gate; law 9 judges the witness module's outcome classes along one constructed edit sequence only (a test edit, a helper edit, and a revert), and the budget witness's along one constructed second check with nothing changed, because a generated history with the witness module or the budget witness takes no steps (maintainer decision, 2026-11-11, and the runtime review's P1-a, 2026-11-12; this oracle owns the gap); law 9's edits come only between checks, so a file saved during a check is not generated (the runner invalidates a known input that changed while a run ran and does not store that run, which only inspection fixes; a witness needs a step that edits a file between a unit's start and its end), and a module that a run first loads and that changes during that same run is stored under its later hash; law 9's edits leave out files read through fs, other environment variables, new files that change how an unchanged import resolves, Vite config and installed-package changes, and concurrent checks other than two that start the daemon; cut-off charging judged only under one budget; native addons only as a binding.gyp or a prebuilt .node file, not as optional platform packages; daemon races (two CLIs reading one error file, a CLI hearing closing from the failing daemon it started, a reset connection on macOS) fixed by inspection; the runner's copies of the test files exist only in memory, and its harness and Vitest configs sit in the package's .oracle-gaps-, which the next run removes once a killed run has left it (judged after two stopped runs, not after a SIGKILL); a default --out is judged by the static oracle, and these runs name --out in the package; the namespace is checked after each run and, in a stopped run, while the run runs; a slow kill in a later owner, after earlier owners' confirmations spent the budget, becomes a timeout (owner order; not generated); the Vitest 5 path of the copies' pragma read and of the runner is verified from Vitest 5's source only, since the Vitest 5 CI job runs no gap oracle; a batch whose run reports an error outside any test reruns each copy alone (unitsBatch, runAlone) with its first budget, and a copy cut there is kept as a timeout with no confirmation, a defect older than this oracle's budget work that no history reaches (no history has an error outside any test beside a slow copy), recorded here with that trace for the maintainer; stale-session removal is scoped to this host's session directories (a host file), and is not judged across pid namespaces; the stop law covers SIGTERM and SIGINT sent to the CLI, not a SIGKILL of the CLI (its group then ends at the next check of its parent, by inspection) or a signal sent to the daemon, and its witness is one test that spins synchronously and starts one child process; the mutant budget is witnessed as two slow owners whose confirmations pass the --mutant-limit floor, not through its multiple of the owners' unmutated time, and authored mutants have no confirmations to budget; the witness w_slowpair is a wall-clock window, since the CLI's budget has no controlled clock and a fixture cannot see the time the CLI measures for its file: with one-test owner files p and q of measured unmutated time t each (a file first in its context pays the instrumented transform too), a wait W of 10 s, a 20 s --hang-limit and a 15 s --mutant-limit, it tells a budget from none only while (a) its first attempt is cut, W > 1 s + 3t, (b) its two confirmations do not fit the budget, 2W > max(15 s, 10 x 2t), and (c) each would fit its own full limit, W < 20 s + 3t; (b) is the binding one, so it holds while the two files' time stays under about 2 s together (about 1 s each, some 20 times the authoring machine's idle and 5 to 10 times its loaded measurements), and under one budget a run's overhead must stay under 10 s + 3t; CI runners measure 1.5 to 2.3 times the idle machine; beyond that w_slowpair would read not-judged; a heartbeat is owed only where a slow test waits 3 s against a 2 s interval, twice the first budget, so that no copy cut at its first budget beats, and each beat is observed as its mutant and file, not how long or how often it beat; a config only in the package's parent directory is generated one level up, in a temporary directory with no package.json (so its .js or .ts name is CommonJS), as a plain config, one that throws, one with no default export, or one that defines test.projects; a config two or more levels up, or a parent config beside a package config, is not generated, and the order in one directory follows Vitest's configFiles list by inspection; a package with no config lives in a temporary directory, where no parent has a config; a counterexample shrinks through at most six histories, so it can be larger than the smallest that fails; module-level data only as four \`const\` declarations of the main source (an object literal, an array that holds a function, a boolean, and an object of two methods), read in place, with no data imported from another file, destructured, reassigned, or declared as an enum or a default export.`, + limits: `Law 12 judges refusals only: no legal history reaches an internal error, so its shape (exit 3, the stack in error.log) is not judged; argument refusals are the static oracle's; the check's rejection of a stack trace, a missing fix, and a wrong exit code is calibrated in the static oracle, through the same refusalBreach, and here only a refused run reported as complete is calibrated, because each calibration here runs real CLI histories. Hangs only as an owner test that awaits a promise that never settles, under a 1 s first budget and a 20 s full limit, and a slow run only as two witnesses whose tests wait where the output changed, 3 s in one that then judges and 10 s in two one-test owners of one that does not; busy loops only on instrumented code (cut by the call budget), no slow runs past the full limit that would end, hangs in ordinary or nondeterministic tests, setup errors other than a throw at module load, loops, multi-line conditions, two functions on one line, or await, new, and void statements; nested functions, object-literal methods, accessors, and class-field arrows only where no test calls them (their ownership is the static oracle's); two source files of one package, with no re-exports, namespace imports, or calls into another package; --only only in a stopped run and a partial run of one function's mutants, and no --sample or --diff-base; the baseline's failures only as one failing owner test that throws an error and a setup file that throws, so a baseline law violation's law and counterexample lines are the classify oracle's; a derived class's file is instrumented and its sites are judged as unowned, but no test imports or constructs the class, so its no-mutant and super-region behavior is the static oracle's (in-process), not this oracle's; package configs only as vite.config or vitest.config with one of the extensions .ts, .mts, .cts, .js, .mjs, and .cjs (not every pair), as config/gap-vitest with the same extension through --config, or none, as an object or a sync or async function, with no plugins of their own and test.projects only as a refusal, and only under the package's own package.json; a top-level await only in an ES module config and require only in a CommonJS one, as Vitest itself loads no other; a --config outside the package is refused by design, through the rule for paths outside the package (args.mjs), and no history generates one; contexts only one, four, or eleven, and process warnings judged only as MaxListenersExceededWarning on the CLI's own stderr (a daemon's process warnings go to its log); a context that fails to open gives back its listener allowance, which no observation shows (a kept allowance only raises the limit, so no warning follows), so that accounting is fixed by inspection; no probe-bound states, so reach agreement's killing of probe mutants is not checked; no gap answers or gate; law 9 judges the witness module's outcome classes along one constructed edit sequence only (a test edit, a helper edit, and a revert), and the budget witness's along one constructed second check with nothing changed, because a generated history with the witness module or the budget witness takes no steps (maintainer decision, 2026-11-11, and the runtime review's P1-a, 2026-11-12; this oracle owns the gap); law 9's edits come only between checks, so a file saved during a check is not generated (the runner invalidates a known input that changed while a run ran and does not store that run, which only inspection fixes; a witness needs a step that edits a file between a unit's start and its end), and a module that a run first loads and that changes during that same run is stored under its later hash; law 9's edits leave out files read through fs, other environment variables, new files that change how an unchanged import resolves, Vite config and installed-package changes, and concurrent checks other than two that start the daemon; cut-off charging judged only under one budget; native addons only as a binding.gyp or a prebuilt .node file, not as optional platform packages; daemon races (two CLIs reading one error file, a CLI hearing closing from the failing daemon it started, a reset connection on macOS) fixed by inspection; the runner's copies of the test files exist only in memory, and its harness and Vitest configs sit in the package's .oracle-gaps-, which the next run removes once a killed run has left it (judged after two stopped runs, not after a SIGKILL); a default --out is judged by the static oracle, and these runs name --out in the package; the namespace is checked after each run and, in a stopped run, while the run runs; a slow kill in a later owner, after earlier owners' confirmations spent the budget, becomes a timeout (owner order; not generated); the Vitest 5 path of the copies' pragma read and of the runner is verified from Vitest 5's source only, since the Vitest 5 CI job runs no gap oracle; a batch whose run reports an error outside any test reruns each copy alone (unitsBatch, runAlone) with its first budget, and a copy cut there is kept as a timeout with no confirmation, a defect older than this oracle's budget work that no history reaches (no history has an error outside any test beside a slow copy), recorded here with that trace for the maintainer; stale-session removal is scoped to this host's session directories (a host file), and is not judged across pid namespaces; the stop law covers SIGTERM and SIGINT sent to the CLI, not a SIGKILL of the CLI (its group then ends at the next check of its parent, by inspection) or a signal sent to the daemon, and its witness is one test that spins synchronously and starts one child process; the mutant budget is witnessed as two slow owners whose confirmations pass the --mutant-limit floor, not through its multiple of the owners' unmutated time, and authored mutants have no confirmations to budget; the witness w_slowpair is a wall-clock window, since the CLI's budget has no controlled clock and a fixture cannot see the time the CLI measures for its file: with one-test owner files p and q of measured unmutated time t each (a file first in its context pays the instrumented transform too), a wait W of 10 s, a 20 s --hang-limit and a 15 s --mutant-limit, it tells a budget from none only while (a) its first attempt is cut, W > 1 s + 3t, (b) its two confirmations do not fit the budget, 2W > max(15 s, 10 x 2t), and (c) each would fit its own full limit, W < 20 s + 3t; (b) is the binding one, so it holds while the two files' time stays under about 2 s together (about 1 s each, some 20 times the authoring machine's idle and 5 to 10 times its loaded measurements), and under one budget a run's overhead must stay under 10 s + 3t; CI runners measure 1.5 to 2.3 times the idle machine; beyond that w_slowpair would read not-judged; a heartbeat is owed only where a slow test waits 3 s against a 2 s interval, twice the first budget, so that no copy cut at its first budget beats, and each beat is observed as its mutant and file, not how long or how often it beat; a config only in the package's parent directory is generated one level up, in a temporary directory with no package.json (so its .js or .ts name is CommonJS), as a plain config, one that throws, one with no default export, or one that defines test.projects; a config two or more levels up, or a parent config beside a package config, is not generated, and the order in one directory follows Vitest's configFiles list by inspection; a package with no config lives in a temporary directory, where no parent has a config; a counterexample shrinks through at most six histories, so it can be larger than the smallest that fails; module-level data only as four \`const\` declarations of the main source (an object literal, an array that holds a function, a boolean, and an object of two methods), read in place, with no data imported from another file, destructured, reassigned, or declared as an enum or a default export; the work law only through two shortcut witnesses (an early return before a counting loop, and an assignment-like push beside one that calls String), each with a bound twin (ratios 2 and 1), a generated test's bound of 1, and one warm-cache witness whose selection reproduces observations but not work; work is the capture's count of site evaluations per test, so code with no site (a plain assignment, a for...of pass, a builtin's own work) counts nothing and a test's histories are summed, and a test whose work varies between the unmutated runs while its observations agree is not generated.`, }, histories: { // A failing history shrinks through at most six smaller ones. Each runs diff --git a/packages/oracles/tests/oracle-property-self-oracle.test.ts b/packages/oracles/tests/oracle-property-self-oracle.test.ts index 62ecc91..fad6271 100644 --- a/packages/oracles/tests/oracle-property-self-oracle.test.ts +++ b/packages/oracles/tests/oracle-property-self-oracle.test.ts @@ -322,12 +322,12 @@ import { fileURLToPath } from 'node:url' import fc from 'fast-check' import { oracleProperty, pass, violation } from '../src/index.js' import { + WORK_RATIO, oddKinds, outerSuite, shellText, syntheticArbitrary, syntheticIds, - WORK_RATIO, } from './oracle-property-self-oracle-fixture.js' import type { CalibrationDifference, From 2b33a17c30911d59061bf7658411e02d6c4dc38f Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Thu, 12 Nov 2026 11:44:25 -0700 Subject: [PATCH 04/20] fix(oracle-gaps): reuse a selection only when it reproduces the tests' work A selection that leaves out the tests before a mutant's first reaching test is reused only when its unmutated run reproduces what the whole file recorded. It now also has to reproduce each test's work, the call count that a work bound and the extra-work class judge. A skipped test that warmed a cache otherwise makes a later test do more work and hides a mutant's extra work. Co-authored-by: Isaac --- packages/oracles/oracle-gaps/check.mjs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/packages/oracles/oracle-gaps/check.mjs b/packages/oracles/oracle-gaps/check.mjs index 3af4105..fa9efe3 100644 --- a/packages/oracles/oracle-gaps/check.mjs +++ b/packages/oracles/oracle-gaps/check.mjs @@ -1737,10 +1737,13 @@ export default async (env) => { return ( !run.timedOut && outcomeOf([run]).outcome === `survived` && + // Its work too, which a work bound judges (classify.mjs): a skipped + // test that warmed a cache leaves a later test more to do. selection.every( (k) => rows.get(k)?.hash === first.rows.get(k)?.hash && - rows.get(k)?.observed === first.rows.get(k)?.observed, + rows.get(k)?.observed === first.rows.get(k)?.observed && + rows.get(k)?.calls === first.rows.get(k)?.calls, ) ) } From ba822f02ba78f30175a57fb21300285248c0124c Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Thu, 12 Nov 2026 13:15:45 -0700 Subject: [PATCH 05/20] docs(oracle-gaps): state that a work bound judges a test's histories together Co-authored-by: Isaac --- README.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 9e50839..c5c9ecc 100644 --- a/README.md +++ b/README.md @@ -71,8 +71,11 @@ every commit with [pkg.pr.new](https://pkg.pr.new). no unmodified run to compare with, so there the bound judges nothing. The count does not see work without a site, such as a plain assignment, a `for...of` pass, or a builtin's own work, so a call to a builtin counts - once, whatever it costs. An authored mutant's copy is not instrumented, so - a work bound does not judge it. + once, whatever it costs. Work is counted per test, not per history: a + test that runs many histories, such as a `generated` unit, is judged by + their sum, so a slowdown in a few of its histories is diluted, and a + constructed or fixed history isolates it. An authored mutant's copy is + not instrumented, so a work bound does not judge it. Exit codes: 0 the check completed, whatever its gaps (the last stdout line is `completed: gaps`); 2 the run was refused, with `refused:` and From 539e75404a91922c53711da20ce7f61bdba135f8 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Thu, 12 Nov 2026 13:37:30 -0700 Subject: [PATCH 06/20] test(oracle-gaps): only registered, unclocked tests judge work, compared exactly Review of #48 (F1, F2, F3, F4, F5): - The classify model and the e2e model judge work only on a reproducible or random route, which can carry a bound. A grammar route that no oracleProperty registers, and a constructed case where such a test rises before a registered one, which the directive must name. - The bound grid in hundredths (1.00, 1.16, 1.50, 2.00) and the work grid (0, 10, 25, 100; +4) reach the exact boundary 29 = 1.16 x 25, which the model compares in integers. - A test that made a timer or held I/O open judges no work (classify grammar dimension; e2e witness w_fasttimed, whose tests wait on a timer, and w_fastplain, whose tests register no route). - The printed ratio has three decimals; a null work option fails to load with the work.ratio message. RED at ba822f0: classify oracle 18 failed (extra-work owed not-observed; killed by work bound owed extra-work at 29 = 1.16 x 25; the directive names the unregistered t0); self-oracle "Cannot read properties of null (reading 'ratio')". Co-authored-by: Isaac --- ...p-report-classify-oracle.property.test.mjs | 129 ++++++++++++++---- .../oracle-gaps/gap-report-model-oracle.mjs | 39 +++++- .../gap-report-oracle.property.test.mjs | 25 +++- .../oracle-property-self-oracle-fixture.ts | 26 ++-- .../tests/oracle-property-self-oracle.test.ts | 21 ++- 5 files changed, 196 insertions(+), 44 deletions(-) diff --git a/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs b/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs index 79d9023..3acabbf 100644 --- a/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs +++ b/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs @@ -439,7 +439,7 @@ const effect = fc.record({ // What the mutant adds to the test's work (site evaluations; a negative // amount takes work away), and the test's time in milliseconds, which // varies as a clock does and judges nothing. - work: fc.constantFrom(0, 0, 0, -1, 1, 7, 150), + work: fc.constantFrom(0, 0, 0, -1, 1, 4, 7, 150), ms: fc.nat({ max: 40 }), }) const testRecord = fc.record({ @@ -468,11 +468,14 @@ const testRecord = fc.record({ runOrder: fc.nat({ max: 2 }), // The test's work with no mutant active, in site evaluations; whether its // second unmutated run did one more (its observations still agreeing); - // the work bound its property declares (`work: { ratio }`), if any; and - // its time in the two unmutated runs, which judges nothing. - work: fc.constantFrom(0, 10, 100), + // the work bound its property declares (`work: { ratio }`), if any, in + // hundredths; whether it made a timer or held I/O open, so its work can + // follow a clock; and its time in the two unmutated runs, which judges + // nothing. + work: fc.constantFrom(0, 10, 25, 100), workFlaky: fc.constantFrom(false, false, false, true), - bound: fc.constantFrom(undefined, undefined, 1, 1.5, 2), + bound: fc.constantFrom(undefined, undefined, 100, 116, 150, 200), + clocked: fc.constantFrom(`none`, `none`, `none`, `timers`, `io`), ms: fc.nat({ max: 40 }), }) const records = fc.record({ @@ -623,21 +626,24 @@ const swapFor = { '>': `>=`, '>=': `>`, '===': `!==`, '&&': `||` } const workUnder = (t, slot) => Math.max(0, t.work + t.under[slot].work) // Whether a test's work judges operator mutant `mi`: it passed under the // mutant, it reached the mutant or its file loaded code that did, its -// observations and its work agreed between the unmutated runs, and its -// route judges production, not the framework's own checks. +// observations and its work agreed between the unmutated runs, it made no +// timer and held no I/O open (its work could follow a clock), and its +// route is one that `oracleProperty` registers for production, a +// reproducible or a random one, so it can carry a declared bound. const passedKinds = new Set([`same`, `changed`, `observedChanged`, `unhandled`]) const workJudges = (h, mi, x) => passedKinds.has(x.t.under[mi].kind) && (x.t.reaches[mi] || h.files[x.fi].background[mi]) && !x.t.flaky && !x.t.workFlaky && - kindOf(x.t.route) !== `framework` + x.t.clocked === `none` && + [`reproducible`, `random`].includes(kindOf(x.t.route)) // A test past its declared bound: more work than `ratio` times its work -// with no mutant active. +// with no mutant active, compared exactly, in hundredths. const pastBound = (h, mi, x) => workJudges(h, mi, x) && x.t.bound !== undefined && - workUnder(x.t, mi) > x.t.bound * x.t.work + 100 * workUnder(x.t, mi) > x.t.bound * x.t.work // The work bound that rejects a rise from `before` to `after` site // evaluations: the largest ratio in hundredths below the measured one, and @@ -842,7 +848,7 @@ function judgedOutcome(h, mi, owners) { mention: [ `\`work: { ratio: ${boundBelow(rose.t.work, workUnder(rose.t, mi))} }\``, `\`${shownName(h, rose.fi, rose.ti)}\``, - `rose from ${rose.t.work} to ${workUnder(rose.t, mi)} site evaluation${workUnder(rose.t, mi) === 1 ? `` : `s`}${rose.t.work > 0 ? ` (${(workUnder(rose.t, mi) / rose.t.work).toFixed(2)}× the unmodified run's)` : ``}.`, + `rose from ${rose.t.work} to ${workUnder(rose.t, mi)} site evaluation${workUnder(rose.t, mi) === 1 ? `` : `s`}${rose.t.work > 0 ? ` (${(workUnder(rose.t, mi) / rose.t.work).toFixed(3)}× the unmodified run's)` : ``}.`, ], }) if (m.evidence === `dead code`) @@ -1278,7 +1284,7 @@ const directives = { /^Observe the value this code changes in (`[^`]+`|the owner tests), and assert on it\. Where the mutated code computes the same result as the original, add a history where it computes something different\.$/, 'not-judged': /^Make a law in `[^`]+` reject the changed value\.$/, 'extra-work': - /^Declare `work: \{ ratio: \d+\.\d\d \}` on the `oracleProperty` that registers `[^`]+`, so that its work bound rejects this mutant: there it changed no observation, and its work rose from \d+ to \d+ site evaluations?( \(\d+\.\d\d× the unmodified run's\))?\.$/, + /^Declare `work: \{ ratio: \d+\.\d\d \}` on the `oracleProperty` that registers `[^`]+`, so that its work bound rejects this mutant: there it changed no observation, and its work rose from \d+ to \d+ site evaluations?( \(\d+\.\d{3}× the unmodified run's\))?\.$/, indeterminate: /^(Make `[^`]+` deterministic: control its clock and its random seed, so its failures judge this code|Add an owner test that runs line \d+ and asserts on its result)\.$/, 'dead-code': /^Remove the redundant code on line \d+\.$/, @@ -1523,7 +1529,11 @@ function baselineUnitsOf(h, pass) { // The capture's work count and the property's declared bound // (capture-setup.mjs `calls` and `work`), and the test's time. calls: t.work + (t.workFlaky && pass === 2 ? 1 : 0), - work: t.bound, + // The ratio the property declares, as an author writes it. + work: t.bound === undefined ? undefined : t.bound / 100, + // A timer the test made, or I/O it held open (capture-setup.mjs). + ...(t.clocked === `timers` ? { timers: true } : {}), + ...(t.clocked === `io` ? { io: true } : {}), ms: t.ms + pass, }) }) @@ -1597,7 +1607,9 @@ function unitOf(h, fi, slot) { // An authored mutant's copy is not instrumented, so it counts no // work. calls: slot < MUTANTS ? workUnder(t, slot) : 0, - work: t.bound, + work: t.bound === undefined ? undefined : t.bound / 100, + ...(t.clocked === `timers` ? { timers: true } : {}), + ...(t.clocked === `io` ? { io: true } : {}), ms: under(t).ms, verdicts: kind === `verdict` @@ -1902,6 +1914,25 @@ async function driver(h, observe) { if (judgedRise.t.bound !== undefined) states.add(`extra-work under a declared bound`) if (judgedRise.t.work === 0) states.add(`extra-work from no work`) + if ( + judgedRise.t.bound !== undefined && + 100 * workUnder(judgedRise.t, mi) === + judgedRise.t.bound * judgedRise.t.work + ) + states.add(`a rise exactly at the declared bound`) + // An unregistered test before it in report order also rose. + if ( + workTests + .slice(0, workTests.indexOf(judgedRise)) + .some( + (x) => + kindOf(x.t.route) === `unknown` && + passedKinds.has(x.t.under[mi].kind) && + !x.t.flaky && + workUnder(x.t, mi) > x.t.work, + ) + ) + states.add(`extra-work named past an unregistered rise`) } if (result.class !== undefined && result.class !== `extra-work`) { if ( @@ -2921,6 +2952,8 @@ const required = [ `random-only-kill by a work bound`, `a work rise where work judges nothing`, `a survivor whose work fell`, + `extra-work named past an unregistered rise`, + `a rise exactly at the declared bound`, ] // One constructed history per outcome class: one mutant judged by one @@ -2952,6 +2985,7 @@ const plainTest = (effect = {}, test = {}) => ({ work: 10, workFlaky: false, bound: undefined, + clocked: `none`, ms: 0, ...test, }) @@ -3175,7 +3209,7 @@ const constructed = [ [`extra-work`, `extra-work under a declared bound`], { mutant: { operator: `if-false` }, - test: { work: 100, bound: 2 }, + test: { work: 100, bound: 200 }, effect: { work: 7 }, }, ), @@ -3191,33 +3225,85 @@ const constructed = [ }), single(`a rise past the declared bound`, [`killed by a work bound`], { mutant: { operator: `if-false` }, - test: { bound: 1.5 }, + test: { bound: 150 }, effect: { work: 7 }, }), single( `a rise past the declared bound beside a changed value`, [`killed by a work bound`, `a work bound kill beside a changed value`], - { test: { bound: 2 }, effect: { kind: `changed`, work: 150 } }, + { test: { bound: 200 }, effect: { kind: `changed`, work: 150 } }, ), single( `a rise past the declared bound in the random campaign`, [`random-only-kill by a work bound`], - { test: { route: `random seed`, bound: 1 }, effect: { work: 1 } }, + { test: { route: `random seed`, bound: 100 }, effect: { work: 1 } }, ), single( `a rise in a test whose work varied`, [`a work rise where work judges nothing`], { mutant: { operator: `if-false` }, - test: { workFlaky: true, bound: 1 }, + test: { workFlaky: true, bound: 100 }, effect: { work: 7 }, }, ), single(`a rise in a calibration`, [`a work rise where work judges nothing`], { mutant: { operator: `if-false` }, - test: { route: `calibration`, bound: 1 }, + test: { route: `calibration`, bound: 100 }, effect: { work: 150 }, }), + // A test that no oracleProperty registered (an unknown route) can carry + // no bound, so its rise judges nothing, and the mutant keeps its class. + single( + `a rise on a route that carries no bound`, + [`a work rise where work judges nothing`], + { + mutant: { operator: `if-false` }, + test: { route: undefined }, + effect: { work: 7 }, + }, + ), + // The unregistered test rises first in report order; the directive names + // the registered test after it, whose bound can reject the mutant. + single( + `a rise on an unregistered route before a registered one`, + [`extra-work`, `extra-work named past an unregistered rise`], + { + mutant: { operator: `if-false` }, + test: { route: undefined }, + effect: { work: 7 }, + more: [plainTest({ work: 150 })], + }, + ), + // 29 is exactly 1.16 times 25: not past the bound, though a product in + // binary floating point is a little below 29. + single( + `a rise exactly at the declared bound`, + [ + `extra-work`, + `extra-work under a declared bound`, + `a rise exactly at the declared bound`, + ], + { + mutant: { operator: `if-false` }, + test: { work: 25, bound: 116 }, + effect: { work: 4 }, + }, + ), + // A test that made a timer, or held I/O open, can do work that follows a + // clock: a polling or a deadline loop. Its rise judges nothing, so a + // bound cannot kill on it and the mutant is not extra-work. + ...[`timers`, `io`].map((clocked) => + single( + `a rise past the bound in a test that ${clocked === `timers` ? `made a timer` : `held I/O open`}`, + [`a work rise where work judges nothing`], + { + mutant: { operator: `if-false` }, + test: { clocked, bound: 100 }, + effect: { work: 150 }, + }, + ), + ), single(`a forced branch that saves work`, [`a survivor whose work fell`], { mutant: { operator: `if-false` }, effect: { work: -1 }, @@ -3647,9 +3733,6 @@ oracleProperty({ `framework-error`, `setup-error`, ].map((kind) => `authored ${kind}`), - // A work bound comes from `oracleProperty`, which always registers a - // route, so this one is measured, not required. - `unregistered-kill by a work bound`, ], runs: 300, fixedSeeds: [20261105], diff --git a/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs b/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs index 481b1e2..3c3b5bc 100644 --- a/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs +++ b/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs @@ -408,6 +408,8 @@ const testCase = fc.record({ // A test may declare a work bound, as an oracle property's `work` does: // a ratio of 1 rejects a mutant that does any more work there. work: fc.constantFrom(null, null, null, 1), + // Only a witness waits on a timer: its work could follow the clock. + timed: fc.constant(false), }) // Law 9's edits. Each content edit changes what some evidence depends on: // a test file, a source statement, the source's line numbers, an owns list, @@ -819,6 +821,7 @@ const judging = (file, route, input, extra = {}) => ({ stops: false, then: null, work: null, + timed: false, ...extra, }) const witness = [ @@ -950,6 +953,30 @@ const witness = [ judging(`a`, `fixed seed`, 1, { work: 1 }), ], }, + // The early return again, in two tests that the work law does not judge: + // tests that wait on a timer before they run it (as a polling or a + // deadline loop does, whose count can follow the clock), with a bound of + // 1, and tests that register no route, which can carry no bound. Its + // `if-false` mutant costs work there, deterministically, and stays + // `not-observed`. + { + id: `w_fasttimed`, + owners: [`a`], + body: [{ kind: `shortcut`, op: `>`, k: 2, not: false }], + tests: [ + judging(`a`, `fixed seed`, 7, { timed: true, work: 1 }), + judging(`a`, `fixed seed`, 1, { timed: true, work: 1 }), + ], + }, + { + id: `w_fastplain`, + owners: [`a`], + body: [{ kind: `shortcut`, op: `>`, k: 2, not: false }], + tests: [ + judging(`a`, `fixed seed`, 7, { registered: false }), + judging(`a`, `fixed seed`, 1, { registered: false }), + ], + }, { id: `w_noobs`, owners: [`a`], @@ -1689,6 +1716,8 @@ const captureWitness = new Set([ `w_fastbound`, `w_fastcall`, `w_fastcallbound`, + `w_fasttimed`, + `w_fastplain`, `w_noobs`, `w_unreached`, `w_ordinary`, @@ -3828,10 +3857,14 @@ function runsOf(plan, tests, sites, active) { // mutant's copy is not instrumented, so it counts no work, and the // work law judges only operator mutants. const route = routeOf(t, partsOf(t).at(-1)) + // Only a route that oracleProperty registers for production can carry + // a bound, and a test that made a timer can do work that follows the + // clock, so neither judges work. const judgesWork = found.length === 0 && active?.site !== undefined && - route !== `framework` + [`reproducible`, `random`].includes(route) && + !t.timed return { t, route, @@ -4448,6 +4481,8 @@ function oracleTestFile(plan, file) { ...(t.registered && t.work !== null ? [` g.__oracleWorkBound?.(${t.work})`] : []), + // A test that waits on a timer before it runs production. + ...(t.timed ? [` await new Promise((r) => setTimeout(r, 0))`] : []), ` const ${out}: Array = []`, ...(t.setsShared ? [` shared.value = 5`] : []), t.readsShared @@ -4561,7 +4596,7 @@ function oracleTestFile(plan, file) { : []), t.asProp ? ` fcTest.prop([fcv.integer({ min: 0, max: 10 })], { numRuns: 20, seed: 1 })('${testName(t, index)}', async (n) => {` - : ` test('${testName(t, index)}', ${(t.hangs && !t.flaky) || t.slow || t.ioWait || lazyGated(plan, file, t) ? `async ` : ``}() => {`, + : ` test('${testName(t, index)}', ${(t.hangs && !t.flaky) || t.slow || t.ioWait || t.timed || lazyGated(plan, file, t) ? `async ` : ``}() => {`, ...partsOf(t).flatMap(campaign), ` })`, `})`, diff --git a/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs b/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs index 536fccc..5163801 100644 --- a/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs +++ b/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs @@ -172,7 +172,8 @@ * evaluations that its instrumented production code made (the capture's * call count, never a time): a call statement, a call whose value is * used, a comparison, an `if`, each pass of a loop's condition. A - * passing, deterministic owner test that registers a work bound beside + * passing, deterministic owner test on a reproducible or random route, + * that made no timer and held no I/O open, and that registers a work bound beside * its route (`__oracleWorkBound`, which `oracleProperty`'s `work` * option calls) and whose work under the mutant is more than that ratio * of its unmutated work fails the mutant under law `work bound` on its @@ -180,8 +181,11 @@ * shortcut for speed is owed a cost law). A mutant that no owner test * fails is a survivor: a `not-judged` when an owner test's observation * or asserted value changed, an `extra-work` when no observation changed - * and an owner test that reached it, on a route that judges production, - * did more work (a shortcut whose deletion only costs time), a + * and an owner test that reached it, on a reproducible or random route, + * with no timer made and no I/O held open, did more work (a shortcut + * whose deletion only costs time); a test that made a timer or + * registered no route judges no work, so its rise leaves the class as it + * was (the PR #48 review's F1 and F5, 2026-11-12), a * `branch-never-taken` when an `if` on its line never took * one of its outcomes in an owner test, a `not-infected` when a comparison's * operators agreed on every operand an owner test reached, a @@ -1688,6 +1692,19 @@ async function driver(plan, observe) { if (m.class === `extra-work`) states.add(`extra work`) if (m.outcome === `killed` && m.laws?.includes(`work bound`)) states.add(`killed by a work bound`) + // The early return whose extra work no test judges (law 4). + if ( + m.operator === `if-false` && + m.class === `not-observed` && + m.function === `w_fasttimed` + ) + states.add(`a work rise in a test that made a timer judged nothing`) + if ( + m.operator === `if-false` && + m.class === `not-observed` && + m.function === `w_fastplain` + ) + states.add(`a work rise on an unregistered route judged nothing`) if (m.class === `branch-never-taken`) states.add(`branch never taken`) if (m.class === `not-infected`) states.add(`not infected`) if (m.class === `not-observed`) states.add(`not observed`) @@ -2547,6 +2564,8 @@ const required = [ `not judged`, `extra work`, `killed by a work bound`, + `a work rise in a test that made a timer judged nothing`, + `a work rise on an unregistered route judged nothing`, `branch never taken`, `not infected`, `not observed`, diff --git a/packages/oracles/tests/oracle-property-self-oracle-fixture.ts b/packages/oracles/tests/oracle-property-self-oracle-fixture.ts index 991f361..7bad497 100644 --- a/packages/oracles/tests/oracle-property-self-oracle-fixture.ts +++ b/packages/oracles/tests/oracle-property-self-oracle-fixture.ts @@ -360,8 +360,8 @@ export type SyntheticSpec = { predictMode?: `missing` | `extra` /** The `owns` option as the gap report reads it: a list, an empty list, a string, or a list with a non-string entry. */ ownsMode?: `list` | `empty` | `string` | `bad entry` - /** The `work` option: a bound of `WORK_RATIO`, a ratio below 1, or one that is not finite. */ - workMode?: `bound` | `below one` | `not finite` + /** The `work` option: a bound of `WORK_RATIO`, a ratio below 1, one that is not finite, or `null`, which plain JavaScript can pass. */ + workMode?: `bound` | `below one` | `not finite` | `null` /** Runs every history on a controlled clock, where a 7 spins it. */ clock?: boolean /** How the property holds a history's items; the array itself when absent. */ @@ -1133,16 +1133,18 @@ function syntheticCalibrations( const workOf = (spec: SyntheticSpec) => spec.workMode === undefined ? {} - : { - work: { - ratio: - spec.workMode === `bound` - ? WORK_RATIO - : spec.workMode === `below one` - ? 0.5 - : Number.POSITIVE_INFINITY, - }, - } + : spec.workMode === `null` + ? { work: null as unknown as { ratio: number } } + : { + work: { + ratio: + spec.workMode === `bound` + ? WORK_RATIO + : spec.workMode === `below one` + ? 0.5 + : Number.POSITIVE_INFINITY, + }, + } // `owns`, the literal list the gap report reads, in the shape the spec names. const ownsOf = (spec: SyntheticSpec) => diff --git a/packages/oracles/tests/oracle-property-self-oracle.test.ts b/packages/oracles/tests/oracle-property-self-oracle.test.ts index fad6271..8f715c8 100644 --- a/packages/oracles/tests/oracle-property-self-oracle.test.ts +++ b/packages/oracles/tests/oracle-property-self-oracle.test.ts @@ -459,8 +459,8 @@ type Draft = { predictMode: `match` | `missing` | `extra` /** The `owns` option: absent, a literal list, a string, or a list with a non-string entry. */ ownsMode: `none` | `list` | `empty` | `string` | `bad entry` - /** The `work` option: absent, a bound, a ratio below 1, or a ratio that is not finite. */ - workMode: `none` | `bound` | `below one` | `not finite` + /** The `work` option: absent, a bound, a ratio below 1, a ratio that is not finite, or `null`. */ + workMode: `none` | `bound` | `below one` | `not finite` | `null` clock: boolean form: `plain` | `bigint` | `map` | `lossy` | `error` | `errors` | `boxed` | `trap` @@ -834,7 +834,11 @@ const draft: fc.Arbitrary = fc.record({ { weight: 3, arbitrary: fc.constant(`bound` as const) }, { weight: 1, - arbitrary: fc.constantFrom(`below one` as const, `not finite` as const), + arbitrary: fc.constantFrom( + `below one` as const, + `not finite` as const, + `null` as const, + ), }, ), // `owns` is read statically by the gap report, so it must be a literal list of strings. @@ -2185,7 +2189,11 @@ function predictSpec( loads: false, loadError: `${spec.id}: \`owns\` must be a non-empty literal list of strings`, } - if (spec.workMode === `below one` || spec.workMode === `not finite`) + if ( + spec.workMode === `below one` || + spec.workMode === `not finite` || + spec.workMode === `null` + ) return { loads: false, loadError: `${spec.id}: \`work.ratio\` must be a finite number of at least 1`, @@ -10378,6 +10386,11 @@ oracleProperty({ [`registration-error`], generatedDraft({ workMode: `not finite` }), ), + fixedHistory( + `a null work bound fails to load`, + [`registration-error`], + generatedDraft({ workMode: `null` }), + ), fixedHistory( `a work bound is registered beside each route that judges production`, [`work-bound-registered`, `work-bound-left-out-of-calibration`], From 809495b1619ce70de4c97c4fcf2a4e13069bc4c3 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Thu, 12 Nov 2026 14:14:53 -0700 Subject: [PATCH 07/20] fix(oracle-gaps): judge work only on registered, unclocked tests, in exact hundredths Review of #48: - F1: only a test on a reproducible or random route judges work, since only such a test can carry a bound; a rise on an unregistered route leaves the mutant's class as it was, and the directive names the first registered test that rose. - F3: the bound is compared exactly, 100 x calls > round(100 x ratio) x before, the arithmetic the recommended bound already used. - F5: a test that made a real timer or interval (the capture now records it) or held I/O open judges no work: its count can follow the clock. - F2: the measured ratio prints three decimals. - F4: work: null fails to load with the work.ratio message. - F7: the option and the README say that the bound is the property's and judges every test it registers. Co-authored-by: Isaac --- README.md | 15 +++++++++---- .../oracles/oracle-gaps/capture-setup.mjs | 22 +++++++++++++++---- packages/oracles/oracle-gaps/classify.mjs | 22 +++++++++++++------ packages/oracles/src/oracle-property.ts | 9 ++++++-- 4 files changed, 51 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index c5c9ecc..aee88b9 100644 --- a/README.md +++ b/README.md @@ -59,15 +59,22 @@ every commit with [pkg.pr.new](https://pkg.pr.new). such as an early return or an assignment in place of a slower call. A test's work is the number of site evaluations that its instrumented production code makes (calls, comparisons, `if`s, and each pass of a - loop's condition), a deterministic count, never a time. A test whose work - varied between the two unmutated runs, or that is one of the framework's - own checks, does not judge work. The gap's directive names the first such + loop's condition), a deterministic count, never a time. Only a test on a + route that `oracleProperty` registers for production (a fixed seed, a + fixed history, a case, or the random campaign) judges work, since only + such a test can carry a bound. A test whose work varied between the two + unmutated runs, or that made a real timer or interval or held I/O open + (its count can follow the clock, as a polling or a deadline loop's + does), does not judge work. A count that varies with the clock but + agreed in both unmutated runs, without a timer or I/O, is not caught. The gap's directive names the first such test, its work before and after, and the bound that rejects the mutant. Declare it on the owning property, for example `oracleProperty({ ..., work: { ratio: 1.1 } })`: a mutant whose work in one of the property's tests is more than `ratio` times the unmodified code's work there is killed under law `work bound`. `ratio` is a finite - number of at least 1, and 1 rejects any extra work. A plain Vitest run has + number of at least 1, read to the hundredth and compared exactly, and 1 + rejects any extra work. The bound is the property's, so it judges every + test the property registers, though the directive names one of them. A plain Vitest run has no unmodified run to compare with, so there the bound judges nothing. The count does not see work without a site, such as a plain assignment, a `for...of` pass, or a builtin's own work, so a call to a builtin counts diff --git a/packages/oracles/oracle-gaps/capture-setup.mjs b/packages/oracles/oracle-gaps/capture-setup.mjs index 6af40e3..65af5f3 100644 --- a/packages/oracles/oracle-gaps/capture-setup.mjs +++ b/packages/oracles/oracle-gaps/capture-setup.mjs @@ -71,7 +71,7 @@ const realUse = real.use // install over the wrappers and give them back. const pending = (globalThis.__oracleGapsPendingTimers ??= (() => { const handles = new Map() - const wrap = (make, clear, once) => { + const wrap = (make, clear, once, timed) => { globalThis[make.name] = function (callback, ...rest) { if (typeof callback !== `function`) return make(callback, ...rest) // An immediate is due at once; a timer or interval after its delay. @@ -85,6 +85,11 @@ const pending = (globalThis.__oracleGapsPendingTimers ??= (() => { ...rest, ) handles.set(handle, real.now() + delay) + // The copy now running hears of a timer or an interval: a test that + // made one can do work that follows the clock, as a polling or a + // deadline loop does (classify.mjs). An immediate waits a turn, not a + // time, as the framework's `withinTurns` does. + if (timed) globalThis.__oracleGapsTimer?.() return handle } globalThis[clear.name] = function (handle) { @@ -92,9 +97,9 @@ const pending = (globalThis.__oracleGapsPendingTimers ??= (() => { return clear(handle) } } - wrap(globalThis.setTimeout, globalThis.clearTimeout, true) - wrap(globalThis.setInterval, globalThis.clearInterval, false) - wrap(globalThis.setImmediate, globalThis.clearImmediate, true) + wrap(globalThis.setTimeout, globalThis.clearTimeout, true, true) + wrap(globalThis.setInterval, globalThis.clearInterval, false, true) + wrap(globalThis.setImmediate, globalThis.clearImmediate, true, false) return handles })()) const unit = run.units?.[globalThis.__vitest_worker__?.filepath] @@ -668,6 +673,13 @@ let workBound globalThis.__oracleWorkBound = (ratio) => { if (inTest) workBound = ratio } +// Whether the current test made a real timer or interval (the wrappers +// above; a fake clock replaces them, and its work does not follow the real +// clock). +let madeTimer = false +globalThis.__oracleGapsTimer = () => { + if (inTest) madeTimer = true +} globalThis.__oracleObserve = (checkpoint, value) => { observed.update(`${checkpoint}(${render(value)})\n`) @@ -708,6 +720,7 @@ beforeEach((context) => { siteHistories = new Map() route = undefined workBound = undefined + madeTimer = false inTest = true }) @@ -744,6 +757,7 @@ function finishTest(context) { infected: [...infected], operands: [...operands], ...(sawIo ? { io: true } : {}), + ...(madeTimer ? { timers: true } : {}), outcomes: [...outcomes], verdicts: [...verdicts.values()], route, diff --git a/packages/oracles/oracle-gaps/classify.mjs b/packages/oracles/oracle-gaps/classify.mjs index bb3e0f4..3702c02 100644 --- a/packages/oracles/oracle-gaps/classify.mjs +++ b/packages/oracles/oracle-gaps/classify.mjs @@ -251,13 +251,17 @@ export function baselineOf(firstRows, secondRows) { reachedBy.set(id, [...(reachedBy.get(id) ?? []), key]) const rank = new Map([...firstRows.keys()].map((key, i) => [key, i])) // Each deterministic test's work, the site evaluations its production - // code made (capture-setup.mjs `calls`), where the two runs agree: only - // such a test's work judges a mutant. + // code made (capture-setup.mjs `calls`), where the two runs agree and + // neither made a timer or held I/O open, after which a count can follow a + // clock: only such a test's work judges a mutant. + const clocked = (row) => row?.timers || row?.io const work = new Map() for (const [key, row] of firstRows) if ( !row.background && !nondeterministic.has(key) && + !clocked(row) && + !clocked(secondRows.get(key)) && secondRows.get(key)?.calls === row.calls ) work.set(key, row.calls) @@ -398,11 +402,14 @@ export function judgeMutant(mutant, { owners, targets, units, baseline }) { !baseline.nondeterministic.has(k), ) // A compared test's work judges the mutant when the two unmutated runs - // agree on it and its route judges production, not the framework's own - // checks; it rose when the mutant's run did more. + // agree on it, no run of it made a timer or held I/O open, and its route + // is one that oracleProperty registers for production, which can carry a + // work bound; it rose when the mutant's run did more. const rose = (k) => baseline.work.has(k) && - routeOf(rows.get(k).route) !== `framework` && + [`reproducible`, `random`].includes(routeOf(rows.get(k).route)) && + !rows.get(k).timers && + !rows.get(k).io && rows.get(k).calls > baseline.work.get(k) const compares = new Set(compared) const result = { @@ -414,7 +421,8 @@ export function judgeMutant(mutant, { owners, targets, units, baseline }) { compares.has(k) && rose(k) && row.work !== undefined && - row.calls > row.work * baseline.work.get(k), + // In hundredths, exactly: 29 is not past 1.16 times 25. + 100 * row.calls > Math.round(100 * row.work) * baseline.work.get(k), ), seconds: units.reduce((sum, u) => sum + u.seconds, 0), } @@ -439,7 +447,7 @@ export function judgeMutant(mutant, { owners, targets, units, baseline }) { test: roseIn, before, after, - ratio: before > 0 ? (after / before).toFixed(2) : undefined, + ratio: before > 0 ? (after / before).toFixed(3) : undefined, bound: ( (before > 0 ? Math.floor((100 * after - 1) / before) : 100) / 100 ).toFixed(2), diff --git a/packages/oracles/src/oracle-property.ts b/packages/oracles/src/oracle-property.ts index 3b2aa73..2ffee41 100644 --- a/packages/oracles/src/oracle-property.ts +++ b/packages/oracles/src/oracle-property.ts @@ -716,7 +716,12 @@ export interface OraclePropertyOptions { * in that test fails it under law `work bound`, so a mutant that deletes a * shortcut, and changes no observation, is killed. A plain Vitest run has * no unmodified run to compare with, so there the bound judges nothing. - * `ratio` is a finite number of at least 1; 1 rejects any extra work. + * `ratio` is a finite number of at least 1, read to the hundredth; 1 + * rejects any extra work. The bound is the property's, so it judges every + * test the property registers (each fixed seed's units and the random + * campaign's), though the gap report's directive names one of them. A + * test that made a timer or held I/O open, whose work can follow a clock, + * is not judged. */ work?: { ratio: number } } @@ -823,7 +828,7 @@ export function oracleProperty( `${id}: \`owns\` must be a non-empty literal list of strings, which the gap report reads statically.`, ) // A ratio below 1 would reject a mutant that does no more work. - if (work !== undefined && !(Number.isFinite(work.ratio) && work.ratio >= 1)) + if (work !== undefined && !(Number.isFinite(work?.ratio) && work.ratio >= 1)) throw new Error( `${id}: \`work.ratio\` must be a finite number of at least 1, the most work a mutant may do as a multiple of the unmodified code's.`, ) From fbf55ae081e71e5dfb77d8eba7428d1fced014a3 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Thu, 12 Nov 2026 14:14:53 -0700 Subject: [PATCH 08/20] test(oracle-gaps): the extra-work legend says what a builtin's cost is owed A builtin call counts as one site evaluation whatever it costs, so a shortcut whose cost is in one (defineProperty) is owed an observation of that cost, not a very tight bound (maintainer, 2026-11-12). RED: "the extra-work legend line does not say that a builtin counts as one site". Co-authored-by: Isaac --- .../gap-report-classify-oracle.property.test.mjs | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs b/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs index 3acabbf..f6984c6 100644 --- a/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs +++ b/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs @@ -1314,10 +1314,14 @@ const directives = { // What a legend line says a class is, before its directive, where the // class's name alone misleads (maintainer disposition G3, 2026-11-11): a // crash is a gap, not a kill (the v20 freeze, P3 and C1), and its error is -// in the gap's Detail line. +// in the gap's Detail line. The extra-work line says first that a builtin +// counts as one site, so a shortcut whose cost is in one is owed an +// observation of that cost, not a very tight bound (maintainer, 2026-11-12). const meanings = { crashed: /^Production threw where no law judged it: a gap, not a kill\. Its error is on the gap's Detail line in summary\.md, and in its `detail` in results\.json\. /, + 'extra-work': + /^A call to a builtin counts as one site evaluation, whatever it costs \(`Object\.defineProperty`\): where a shortcut's cost is in one, make that cost an observation of the driver, such as a count of those calls, and let a law judge it, in place of a very tight bound\. /, } // A count of things, as the report words it. const counted = (n, thing) => `${n} ${thing}${n === 1 ? `` : `s`}` @@ -2380,6 +2384,7 @@ function check(h, observations, expected) { line.startsWith(`Provisional: `), ) const crashedLegend = legend.find((line) => line.startsWith(` crashed: `)) + const workLegend = legend.find((line) => line.startsWith(` extra-work: `)) const leadProblem = stdoutLines.slice(0, baselineGot.length).join(`\n`) !== baselineGot.join(`\n`) @@ -2416,7 +2421,12 @@ function check(h, observations, expected) { crashedLegend.slice(` crashed: `.length), ) ? `the crashed legend line does not say a crash is a gap and where its error is` - : undefined + : workLegend !== undefined && + !meanings[`extra-work`].test( + workLegend.slice(` extra-work: `.length), + ) + ? `the extra-work legend line does not say that a builtin counts as one site, and what is owed for its cost` + : undefined if (leadProblem) return violation(`classified`, leadProblem, `stdout`, { actual: stdout }) // The full report leads with the same baseline, before its gaps. From 5b103316606373d2fc1e5b0494b83c8ed68b8c49 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Thu, 12 Nov 2026 14:15:36 -0700 Subject: [PATCH 09/20] fix(oracle-gaps): tell the author to observe a builtin's cost, not to tighten a bound The extra-work legend and the README now say that a builtin call counts as one site evaluation whatever it costs, so a shortcut whose cost is in one is owed an observation of that cost in the driver, such as a count of those calls, judged by a law, in place of a very tight work bound (maintainer, 2026-11-12). Co-authored-by: Isaac --- README.md | 5 ++++- packages/oracles/oracle-gaps/report.mjs | 3 +++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index aee88b9..4214167 100644 --- a/README.md +++ b/README.md @@ -78,7 +78,10 @@ every commit with [pkg.pr.new](https://pkg.pr.new). no unmodified run to compare with, so there the bound judges nothing. The count does not see work without a site, such as a plain assignment, a `for...of` pass, or a builtin's own work, so a call to a builtin counts - once, whatever it costs. Work is counted per test, not per history: a + once, whatever it costs. Where a shortcut's cost is in such a builtin + (`Object.defineProperty` and the like), make that cost an explicit + observation of the driver, such as a count of those calls, and let a law + judge it, in place of a very tight work bound. Work is counted per test, not per history: a test that runs many histories, such as a `generated` unit, is judged by their sum, so a slowdown in a few of its histories is diluted, and a constructed or fixed history isolates it. An authored mutant's copy is diff --git a/packages/oracles/oracle-gaps/report.mjs b/packages/oracles/oracle-gaps/report.mjs index 3eb34f5..3780b14 100644 --- a/packages/oracles/oracle-gaps/report.mjs +++ b/packages/oracles/oracle-gaps/report.mjs @@ -150,6 +150,9 @@ const directive = { // crash is a gap, not a kill (the v20 freeze, P3 and C1). const meanings = { crashed: `Production threw where no law judged it: a gap, not a kill. Its error is on the gap's Detail line in summary.md, and in its \`detail\` in results.json. `, + // A builtin counts as one site, so a tight bound is a poor witness of its + // cost (maintainer, 2026-11-12). + 'extra-work': `A call to a builtin counts as one site evaluation, whatever it costs (\`Object.defineProperty\`): where a shortcut's cost is in one, make that cost an observation of the driver, such as a count of those calls, and let a law judge it, in place of a very tight bound. `, } // The directive for an operator whose infection is not measured: the input From 25ad6190776b90a32f26485b7571ea4320d5599c Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Thu, 12 Nov 2026 14:17:20 -0700 Subject: [PATCH 10/20] fix(oracles): read work.ratio from a null option without a type error Co-authored-by: Isaac --- packages/oracles/src/oracle-property.ts | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/packages/oracles/src/oracle-property.ts b/packages/oracles/src/oracle-property.ts index 2ffee41..8ef46d1 100644 --- a/packages/oracles/src/oracle-property.ts +++ b/packages/oracles/src/oracle-property.ts @@ -827,8 +827,13 @@ export function oracleProperty( throw new Error( `${id}: \`owns\` must be a non-empty literal list of strings, which the gap report reads statically.`, ) - // A ratio below 1 would reject a mutant that does no more work. - if (work !== undefined && !(Number.isFinite(work?.ratio) && work.ratio >= 1)) + // A ratio below 1 would reject a mutant that does no more work. Plain + // JavaScript can pass `null`, which has no ratio. + const ratio = (work as { ratio?: unknown } | null | undefined)?.ratio + if ( + work !== undefined && + !(typeof ratio === `number` && Number.isFinite(ratio) && ratio >= 1) + ) throw new Error( `${id}: \`work.ratio\` must be a finite number of at least 1, the most work a mutant may do as a multiple of the unmodified code's.`, ) From 4f4075cfb21991037c1efa7d70776a1af263d1d2 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Thu, 12 Nov 2026 15:15:36 -0700 Subject: [PATCH 11/20] test(oracle-gaps): a count that can follow the clock never judges work Re-review of #48 (F5): a kill must replay on an idle machine (P3), so a test whose count could follow the clock judges no work (maintainer decision, 2026-11-12). The classify grammar's clocked dimension now covers a timer, a node:timers/promises timer, a clock read, and held I/O, all recorded as one row flag. The e2e gains witnesses whose tests sleep through node:timers/promises (w_fastsleep) or read Date.now() (w_fastclock), owed not-observed under a bound of 1, and one whose tests wait a turn with setImmediate and read no clock (w_fastturn), still judged, owed killed. RED at 9087746: classify oracle 8 failed ("killed laws [work bound], owed not-observed"; "extra-work, owed not-observed"). Co-authored-by: Isaac --- ...p-report-classify-oracle.property.test.mjs | 45 +++++++++++----- .../oracle-gaps/gap-report-model-oracle.mjs | 53 ++++++++++++++----- .../gap-report-oracle.property.test.mjs | 35 ++++++++++-- 3 files changed, 102 insertions(+), 31 deletions(-) diff --git a/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs b/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs index f6984c6..4a057f4 100644 --- a/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs +++ b/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs @@ -469,13 +469,24 @@ const testRecord = fc.record({ // The test's work with no mutant active, in site evaluations; whether its // second unmutated run did one more (its observations still agreeing); // the work bound its property declares (`work: { ratio }`), if any, in - // hundredths; whether it made a timer or held I/O open, so its work can - // follow a clock; and its time in the two unmutated runs, which judges - // nothing. + // hundredths; whether its count could follow the clock, and by which + // path (a timer or interval, a timer of `node:timers/promises`, a read of + // a clock such as `Date.now()`, or I/O held open), all of which the + // capture records as one flag, `clocked`, beside `io`; and its time in the + // two unmutated runs, which judges nothing. work: fc.constantFrom(0, 10, 25, 100), workFlaky: fc.constantFrom(false, false, false, true), bound: fc.constantFrom(undefined, undefined, 100, 116, 150, 200), - clocked: fc.constantFrom(`none`, `none`, `none`, `timers`, `io`), + clocked: fc.constantFrom( + `none`, + `none`, + `none`, + `none`, + `timer`, + `timers/promises`, + `clock read`, + `io`, + ), ms: fc.nat({ max: 40 }), }) const records = fc.record({ @@ -626,8 +637,10 @@ const swapFor = { '>': `>=`, '>=': `>`, '===': `!==`, '&&': `||` } const workUnder = (t, slot) => Math.max(0, t.work + t.under[slot].work) // Whether a test's work judges operator mutant `mi`: it passed under the // mutant, it reached the mutant or its file loaded code that did, its -// observations and its work agreed between the unmutated runs, it made no -// timer and held no I/O open (its work could follow a clock), and its +// observations and its work agreed between the unmutated runs, its count +// could not follow the clock (it made no timer by any path, read no clock, +// and held no I/O open: a kill must replay on an idle machine, the +// maintainer's decision of 2026-11-12 under P3), and its // route is one that `oracleProperty` registers for production, a // reproducible or a random one, so it can carry a declared bound. const passedKinds = new Set([`same`, `changed`, `observedChanged`, `unhandled`]) @@ -1536,7 +1549,7 @@ function baselineUnitsOf(h, pass) { // The ratio the property declares, as an author writes it. work: t.bound === undefined ? undefined : t.bound / 100, // A timer the test made, or I/O it held open (capture-setup.mjs). - ...(t.clocked === `timers` ? { timers: true } : {}), + ...(t.clocked === `none` ? {} : { clocked: true }), ...(t.clocked === `io` ? { io: true } : {}), ms: t.ms + pass, }) @@ -1612,7 +1625,7 @@ function unitOf(h, fi, slot) { // work. calls: slot < MUTANTS ? workUnder(t, slot) : 0, work: t.bound === undefined ? undefined : t.bound / 100, - ...(t.clocked === `timers` ? { timers: true } : {}), + ...(t.clocked === `none` ? {} : { clocked: true }), ...(t.clocked === `io` ? { io: true } : {}), ms: under(t).ms, verdicts: @@ -3300,12 +3313,18 @@ const constructed = [ effect: { work: 4 }, }, ), - // A test that made a timer, or held I/O open, can do work that follows a - // clock: a polling or a deadline loop. Its rise judges nothing, so a - // bound cannot kill on it and the mutant is not extra-work. - ...[`timers`, `io`].map((clocked) => + // A test that made a timer by any path, read a clock, or held I/O open + // can do work that follows the clock: a polling or a deadline loop. Its + // rise judges nothing, so a bound cannot kill on it, which an idle replay + // would not reproduce, and the mutant is not extra-work. + ...[ + [`timer`, `made a timer`], + [`timers/promises`, `slept through node:timers/promises`], + [`clock read`, `read the clock`], + [`io`, `held I/O open`], + ].map(([clocked, did]) => single( - `a rise past the bound in a test that ${clocked === `timers` ? `made a timer` : `held I/O open`}`, + `a rise past the bound in a test that ${did}`, [`a work rise where work judges nothing`], { mutant: { operator: `if-false` }, diff --git a/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs b/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs index 3c3b5bc..e198303 100644 --- a/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs +++ b/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs @@ -408,7 +408,8 @@ const testCase = fc.record({ // A test may declare a work bound, as an oracle property's `work` does: // a ratio of 1 rejects a mutant that does any more work there. work: fc.constantFrom(null, null, null, 1), - // Only a witness waits on a timer: its work could follow the clock. + // Only a witness waits on a timer, sleeps, reads the clock, or waits a + // turn: its work could follow the clock. timed: fc.constant(false), }) // Law 9's edits. Each content edit changes what some evidence depends on: @@ -953,21 +954,30 @@ const witness = [ judging(`a`, `fixed seed`, 1, { work: 1 }), ], }, - // The early return again, in two tests that the work law does not judge: - // tests that wait on a timer before they run it (as a polling or a - // deadline loop does, whose count can follow the clock), with a bound of - // 1, and tests that register no route, which can carry no bound. Its - // `if-false` mutant costs work there, deterministically, and stays - // `not-observed`. - { - id: `w_fasttimed`, + // The early return again, in tests that the work law does not judge, + // with a bound of 1: tests whose count could follow the clock, as a + // polling or a deadline loop's does (a kill must replay on an idle + // machine, the maintainer's decision of 2026-11-12 under P3), which wait + // on a timer, sleep through `node:timers/promises`, or read the clock + // before they run it; and tests that register no route, which can carry + // no bound. Its `if-false` mutant costs work there, deterministically, and + // stays `not-observed`. Tests that wait one turn (`setImmediate`, as + // `withinTurns` does) and read no clock still judge work, so there it is + // killed. + ...[ + [`w_fasttimed`, `timer`], + [`w_fastsleep`, `sleep`], + [`w_fastclock`, `clock`], + [`w_fastturn`, `turn`], + ].map(([id, timed]) => ({ + id, owners: [`a`], body: [{ kind: `shortcut`, op: `>`, k: 2, not: false }], tests: [ - judging(`a`, `fixed seed`, 7, { timed: true, work: 1 }), - judging(`a`, `fixed seed`, 1, { timed: true, work: 1 }), + judging(`a`, `fixed seed`, 7, { timed, work: 1 }), + judging(`a`, `fixed seed`, 1, { timed, work: 1 }), ], - }, + })), { id: `w_fastplain`, owners: [`a`], @@ -1717,6 +1727,9 @@ const captureWitness = new Set([ `w_fastcall`, `w_fastcallbound`, `w_fasttimed`, + `w_fastsleep`, + `w_fastclock`, + `w_fastturn`, `w_fastplain`, `w_noobs`, `w_unreached`, @@ -3864,7 +3877,7 @@ function runsOf(plan, tests, sites, active) { found.length === 0 && active?.site !== undefined && [`reproducible`, `random`].includes(route) && - !t.timed + (!t.timed || t.timed === `turn`) return { t, route, @@ -4424,6 +4437,9 @@ function oracleTestFile(plan, file) { ...(plan.tests.some((t) => t.file === file && (t.ioWait || t.stops)) ? [`import { spawn } from 'node:child_process'`] : []), + ...(plan.tests.some((t) => t.file === file && t.timed === `sleep`) + ? [`import { setTimeout as sleep } from 'node:timers/promises'`] + : []), ...(plan.tests.some((t) => t.file === file && t.stops) ? [ `import { writeFileSync as recordPid } from 'node:fs'`, @@ -4482,7 +4498,16 @@ function oracleTestFile(plan, file) { ? [` g.__oracleWorkBound?.(${t.work})`] : []), // A test that waits on a timer before it runs production. - ...(t.timed ? [` await new Promise((r) => setTimeout(r, 0))`] : []), + ...(t.timed + ? [ + { + timer: ` await new Promise((r) => setTimeout(r, 0))`, + sleep: ` await sleep(0)`, + clock: ` if (Date.now() < 0) throw new Error('the clock reads before 1970')`, + turn: ` await new Promise((r) => setImmediate(r))`, + }[t.timed], + ] + : []), ` const ${out}: Array = []`, ...(t.setsShared ? [` shared.value = 5`] : []), t.readsShared diff --git a/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs b/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs index 5163801..e7ea2e6 100644 --- a/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs +++ b/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs @@ -182,10 +182,14 @@ * fails is a survivor: a `not-judged` when an owner test's observation * or asserted value changed, an `extra-work` when no observation changed * and an owner test that reached it, on a reproducible or random route, - * with no timer made and no I/O held open, did more work (a shortcut - * whose deletion only costs time); a test that made a timer or - * registered no route judges no work, so its rise leaves the class as it - * was (the PR #48 review's F1 and F5, 2026-11-12), a + * whose count could not follow the clock (no timer made by any path, + * `node:timers/promises` included, no clock read, and no I/O held + * open; a wait of one turn, `setImmediate`, is no clock), did more work + * (a shortcut whose deletion only costs time); a test whose count could + * follow the clock, or that registered no route, judges no work, so its + * rise leaves the class as it was, and a bound never kills where an idle + * replay would not (the PR #48 reviews' F1 and F5; the maintainer's + * decision of 2026-11-12 under P3), a * `branch-never-taken` when an `if` on its line never took * one of its outcomes in an owner test, a `not-infected` when a comparison's * operators agreed on every operand an owner test reached, a @@ -1699,6 +1703,26 @@ async function driver(plan, observe) { m.function === `w_fasttimed` ) states.add(`a work rise in a test that made a timer judged nothing`) + if ( + m.operator === `if-false` && + m.class === `not-observed` && + m.function === `w_fastsleep` + ) + states.add( + `a work rise in a test that slept through timers/promises judged nothing`, + ) + if ( + m.operator === `if-false` && + m.class === `not-observed` && + m.function === `w_fastclock` + ) + states.add(`a work rise in a test that read the clock judged nothing`) + if ( + m.operator === `if-false` && + m.outcome === `killed` && + m.function === `w_fastturn` + ) + states.add(`a work rise in a test that waited a turn killed`) if ( m.operator === `if-false` && m.class === `not-observed` && @@ -2565,6 +2589,9 @@ const required = [ `extra work`, `killed by a work bound`, `a work rise in a test that made a timer judged nothing`, + `a work rise in a test that slept through timers/promises judged nothing`, + `a work rise in a test that read the clock judged nothing`, + `a work rise in a test that waited a turn killed`, `a work rise on an unregistered route judged nothing`, `branch never taken`, `not infected`, From b21d1882a5560296b6e3700d2fd0be627afd15a9 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Thu, 12 Nov 2026 16:49:35 -0700 Subject: [PATCH 12/20] fix(oracle-gaps): a test whose count can follow the clock judges no work Re-review of #48 (F5): a work-bound kill must replay on an idle machine (P3), so a test whose count could follow the clock judges no work (maintainer decision, 2026-11-12). The capture now records one row flag, clocked, when the test made a real timer or interval through the globals, node:timers, or node:timers/promises, read a clock (Date.now, new Date() or Date() with no argument, performance.now, process.hrtime), or held I/O open. A read by the test runner, fast-check, the oracle framework, Node, or the capture itself does not count, nor does one under a fake clock; a setImmediate turn reads no clock. classify reads the one flag, and its route, clock, and rise checks are one closure that the bound's check reuses. GREEN: classify oracle 255/255; e2e "every outcome class" passed; klona with bounds still 218/218 killed. Mutants: today's flag set ("w_fastsleep killed, owed not-observed"); no Date.now patch ("w_fastclock killed, owed not-observed"); immediates counted as clocks ("w_fastturn not-observed, owed killed"). Co-authored-by: Isaac --- README.md | 13 +++- .../oracles/oracle-gaps/capture-setup.mjs | 78 ++++++++++++++++--- packages/oracles/oracle-gaps/classify.mjs | 33 ++++---- 3 files changed, 95 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index 17dc209..aaf6fdf 100644 --- a/README.md +++ b/README.md @@ -63,10 +63,15 @@ every commit with [pkg.pr.new](https://pkg.pr.new). route that `oracleProperty` registers for production (a fixed seed, a fixed history, a case, or the random campaign) judges work, since only such a test can carry a bound. A test whose work varied between the two - unmutated runs, or that made a real timer or interval or held I/O open - (its count can follow the clock, as a polling or a deadline loop's - does), does not judge work. A count that varies with the clock but - agreed in both unmutated runs, without a timer or I/O, is not caught. The gap's directive names the first such + unmutated runs, or whose count could follow the clock, as a polling or a + deadline loop's does, does not judge work, so a work-bound kill + replays on an idle machine. A count can follow the clock when the test, + its driver, or production makes a real timer or interval (through the + globals, `node:timers`, or `node:timers/promises`), reads a clock + (`Date.now()`, `new Date()`, `performance.now()`, `process.hrtime`), or + holds I/O open. A wait of one turn (`setImmediate`) reads no clock. Reads + that the test runner, fast-check, or the oracle framework makes do not + count, and neither do reads under a fake clock. The gap's directive names the first such test, its work before and after, and the bound that rejects the mutant. Declare it on the owning property, for example `oracleProperty({ ..., work: { ratio: 1.1 } })`: a mutant whose work in diff --git a/packages/oracles/oracle-gaps/capture-setup.mjs b/packages/oracles/oracle-gaps/capture-setup.mjs index 65af5f3..7e3fc8b 100644 --- a/packages/oracles/oracle-gaps/capture-setup.mjs +++ b/packages/oracles/oracle-gaps/capture-setup.mjs @@ -25,6 +25,7 @@ // of shrinking it: a kill needs one failing history, not the smallest one. import { createHash } from 'node:crypto' import { appendFileSync, readFileSync } from 'node:fs' +import { createRequire, syncBuiltinESMExports } from 'node:module' import { afterAll, beforeEach, chai, expect, inject, vi } from 'vitest' const run = @@ -89,7 +90,7 @@ const pending = (globalThis.__oracleGapsPendingTimers ??= (() => { // made one can do work that follows the clock, as a polling or a // deadline loop does (classify.mjs). An immediate waits a turn, not a // time, as the framework's `withinTurns` does. - if (timed) globalThis.__oracleGapsTimer?.() + if (timed) globalThis.__oracleGapsClock?.() return handle } globalThis[clear.name] = function (handle) { @@ -102,6 +103,48 @@ const pending = (globalThis.__oracleGapsPendingTimers ??= (() => { wrap(globalThis.setImmediate, globalThis.clearImmediate, true, false) return handles })()) +// The other paths to the clock, wrapped once per worker as the timers are, +// each telling the copy now running (`__oracleGapsClock`): a read of +// `Date.now`, `new Date()` or `Date()` with no argument, `performance.now`, +// and `process.hrtime`, and a timer of `node:timers` or +// `node:timers/promises` (which do not go through the global functions). A +// fake clock replaces these, and its reads do not follow the real clock. +if (!globalThis.__oracleGapsClocks) { + globalThis.__oracleGapsClocks = true + const told = (read) => + function (...args) { + globalThis.__oracleGapsClock?.() + return read.apply(this, args) + } + const RealDate = Date + RealDate.now = told(RealDate.now) + const TrackedDate = new Proxy(RealDate, { + construct(target, args, newTarget) { + if (args.length === 0) globalThis.__oracleGapsClock?.() + return Reflect.construct(target, args, newTarget) + }, + apply(target, self, args) { + globalThis.__oracleGapsClock?.() + return Reflect.apply(target, self, args) + }, + }) + // A date's constructor stays the global `Date`. + RealDate.prototype.constructor = TrackedDate + globalThis.Date = TrackedDate + performance.now = told(performance.now) + const hrtime = process.hrtime + process.hrtime = Object.assign(told(hrtime), { + bigint: told(hrtime.bigint), + }) + const require = createRequire(import.meta.url) + const timers = require(`node:timers`) + const promises = require(`node:timers/promises`) + for (const module of [timers, promises]) + for (const name of [`setTimeout`, `setInterval`]) + module[name] = told(module[name]) + promises.scheduler.wait = told(promises.scheduler.wait) + syncBuiltinESMExports() +} const unit = run.units?.[globalThis.__vitest_worker__?.filepath] // The mutants that a copy earlier in this batch killed. A later copy of the // same mutant runs nothing: the mutant is decided (stop at first kill). @@ -673,12 +716,29 @@ let workBound globalThis.__oracleWorkBound = (ratio) => { if (inTest) workBound = ratio } -// Whether the current test made a real timer or interval (the wrappers -// above; a fake clock replaces them, and its work does not follow the real -// clock). -let madeTimer = false -globalThis.__oracleGapsTimer = () => { - if (inTest) madeTimer = true +// Whether the current test's count could follow the clock (maintainer +// decision, 2026-11-12, under P3: a kill must replay on an idle machine): +// it made a real timer or interval by any path, or read the clock (the +// wrappers above), or held I/O open (`sawIo`). A read that the test runner, +// fast-check, the oracle framework (its shrink budget), Node, or this file +// makes does not count: the caller is the first frame outside this file +// and Node's own. +let clocked = false +const machinery = + /[\\/]node_modules[\\/](?:vitest|@vitest|fast-check|@fast-check|tinypool|tinyspy|chai|@tanstack[\\/]oracles)[\\/]|[\\/]packages[\\/]oracles[\\/]src[\\/]/ +function fromTest() { + for (const frame of String(new Error().stack).split(`\n`).slice(1)) { + const file = + frame.match(/\((.*):\d+:\d+\)$/)?.[1] ?? + frame.match(/at (.*):\d+:\d+$/)?.[1] + if (file === undefined || file.startsWith(`node:`)) continue + if (file.endsWith(`capture-setup.mjs`)) continue + return !machinery.test(file) + } + return false +} +globalThis.__oracleGapsClock = () => { + if (inTest && !clocked && fromTest()) clocked = true } globalThis.__oracleObserve = (checkpoint, value) => { @@ -720,7 +780,7 @@ beforeEach((context) => { siteHistories = new Map() route = undefined workBound = undefined - madeTimer = false + clocked = false inTest = true }) @@ -757,7 +817,7 @@ function finishTest(context) { infected: [...infected], operands: [...operands], ...(sawIo ? { io: true } : {}), - ...(madeTimer ? { timers: true } : {}), + ...(clocked || sawIo ? { clocked: true } : {}), outcomes: [...outcomes], verdicts: [...verdicts.values()], route, diff --git a/packages/oracles/oracle-gaps/classify.mjs b/packages/oracles/oracle-gaps/classify.mjs index 3702c02..72da617 100644 --- a/packages/oracles/oracle-gaps/classify.mjs +++ b/packages/oracles/oracle-gaps/classify.mjs @@ -252,16 +252,15 @@ export function baselineOf(firstRows, secondRows) { const rank = new Map([...firstRows.keys()].map((key, i) => [key, i])) // Each deterministic test's work, the site evaluations its production // code made (capture-setup.mjs `calls`), where the two runs agree and - // neither made a timer or held I/O open, after which a count can follow a - // clock: only such a test's work judges a mutant. - const clocked = (row) => row?.timers || row?.io + // neither run's count could follow the clock (`clocked`): only such a + // test's work judges a mutant, so a kill replays on an idle machine. const work = new Map() for (const [key, row] of firstRows) if ( !row.background && !nondeterministic.has(key) && - !clocked(row) && - !clocked(secondRows.get(key)) && + !row.clocked && + !secondRows.get(key)?.clocked && secondRows.get(key)?.calls === row.calls ) work.set(key, row.calls) @@ -402,24 +401,23 @@ export function judgeMutant(mutant, { owners, targets, units, baseline }) { !baseline.nondeterministic.has(k), ) // A compared test's work judges the mutant when the two unmutated runs - // agree on it, no run of it made a timer or held I/O open, and its route - // is one that oracleProperty registers for production, which can carry a + // agree on it, no run's count could follow the clock, and its route is + // one that oracleProperty registers for production, which can carry a // work bound; it rose when the mutant's run did more. - const rose = (k) => - baseline.work.has(k) && - [`reproducible`, `random`].includes(routeOf(rows.get(k).route)) && - !rows.get(k).timers && - !rows.get(k).io && - rows.get(k).calls > baseline.work.get(k) const compares = new Set(compared) + const rose = (k, row = rows.get(k)) => + compares.has(k) && + baseline.work.has(k) && + [`reproducible`, `random`].includes(routeOf(row.route)) && + !row.clocked && + row.calls > baseline.work.get(k) const result = { ...outcomeOf( units, baseline.nondeterministic, false, (k, row) => - compares.has(k) && - rose(k) && + rose(k, row) && row.work !== undefined && // In hundredths, exactly: 29 is not past 1.16 times 25. 100 * row.calls > Math.round(100 * row.work) * baseline.work.get(k), @@ -428,7 +426,10 @@ export function judgeMutant(mutant, { owners, targets, units, baseline }) { } if (result.outcome !== `survived`) return result result.changedTests = changedIn(rows, compared, baseline) - const roseIn = inReportOrder(compared.filter(rose), baseline)[0] + const roseIn = inReportOrder( + compared.filter((k) => rose(k)), + baseline, + )[0] if (result.changedTests.length > 0) result.class = `not-judged` else if (compared.length === 0) { result.class = `indeterminate` From 96634ace8d85e6646cf0051c1db8874a63f908c3 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Thu, 12 Nov 2026 17:12:13 -0700 Subject: [PATCH 13/20] test(oracle-gaps): every path to the clock, and promisify(setTimeout), under the capture Round-3 review of #48 (R1, R2, R3). The e2e's clock witnesses become one function, w_fastclocked, whose tests each reach the clock by one path before they run production: util.promisify of the global and of node:timers's setTimeout (which must keep its value, law 3: a test that passes alone passes under the CLI), node:timers/promises, Date.now() directly and through a vi.spyOn passthrough, process.uptime(), os.uptime(), AbortSignal.timeout, Atomics.waitAsync with a timeout, and monitorEventLoopDelay. Its if-false mutant is owed not-observed under a bound of 1, so any path left unseen kills it. The classify grammar's clocked dimension names the same paths. Co-authored-by: Isaac --- ...p-report-classify-oracle.property.test.mjs | 32 +++++--- .../oracle-gaps/gap-report-model-oracle.mjs | 79 +++++++++++++------ .../gap-report-oracle.property.test.mjs | 20 +---- 3 files changed, 83 insertions(+), 48 deletions(-) diff --git a/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs b/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs index 4a057f4..516fcf7 100644 --- a/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs +++ b/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs @@ -477,15 +477,23 @@ const testRecord = fc.record({ work: fc.constantFrom(0, 10, 25, 100), workFlaky: fc.constantFrom(false, false, false, true), bound: fc.constantFrom(undefined, undefined, 100, 116, 150, 200), - clocked: fc.constantFrom( - `none`, - `none`, - `none`, - `none`, - `timer`, - `timers/promises`, - `clock read`, - `io`, + clocked: fc.oneof( + { weight: 4, arbitrary: fc.constant(`none`) }, + { + weight: 4, + arbitrary: fc.constantFrom( + `timer`, + `timers/promises`, + `clock read`, + `spied clock read`, + `process.uptime`, + `os.uptime`, + `AbortSignal.timeout`, + `Atomics.waitAsync`, + `monitorEventLoopDelay`, + `io`, + ), + }, ), ms: fc.nat({ max: 40 }), }) @@ -3321,6 +3329,12 @@ const constructed = [ [`timer`, `made a timer`], [`timers/promises`, `slept through node:timers/promises`], [`clock read`, `read the clock`], + [`spied clock read`, `read the clock through a spy`], + [`process.uptime`, `polled process.uptime`], + [`os.uptime`, `polled os.uptime`], + [`AbortSignal.timeout`, `polled an AbortSignal.timeout`], + [`Atomics.waitAsync`, `waited with Atomics.waitAsync and a timeout`], + [`monitorEventLoopDelay`, `polled monitorEventLoopDelay`], [`io`, `held I/O open`], ].map(([clocked, did]) => single( diff --git a/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs b/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs index e198303..ad9e003 100644 --- a/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs +++ b/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs @@ -957,27 +957,49 @@ const witness = [ // The early return again, in tests that the work law does not judge, // with a bound of 1: tests whose count could follow the clock, as a // polling or a deadline loop's does (a kill must replay on an idle - // machine, the maintainer's decision of 2026-11-12 under P3), which wait - // on a timer, sleep through `node:timers/promises`, or read the clock - // before they run it; and tests that register no route, which can carry - // no bound. Its `if-false` mutant costs work there, deterministically, and - // stays `not-observed`. Tests that wait one turn (`setImmediate`, as - // `withinTurns` does) and read no clock still judge work, so there it is - // killed. - ...[ - [`w_fasttimed`, `timer`], - [`w_fastsleep`, `sleep`], - [`w_fastclock`, `clock`], - [`w_fastturn`, `turn`], - ].map(([id, timed]) => ({ - id, + // machine, the maintainer's decision of 2026-11-12 under P3). Each reaches + // the clock by one path before it runs production: a timer through + // `util.promisify(setTimeout)`, the global and `node:timers`'s (which + // also checks that the capture keeps `util.promisify.custom`, law 3: a + // test that passes alone passes under the CLI), a sleep through + // `node:timers/promises`, `Date.now()` (and through a `vi.spyOn` + // passthrough, whose spy frame comes first), `process.uptime()`, + // `os.uptime()`, `AbortSignal.timeout`, `Atomics.waitAsync` with a + // timeout, and `monitorEventLoopDelay`. One untimed test runs the + // general path. A mutant is killed if any test still judges work, so its + // `if-false` mutant stays `not-observed` only while every path is seen. + // Tests that wait one turn (`setImmediate`, as `withinTurns` does) and + // read no clock still judge work, so there it is killed; and tests that + // register no route (below) can carry no bound. + { + id: `w_fastclocked`, + owners: [`a`], + body: [{ kind: `shortcut`, op: `>`, k: 2, not: false }], + tests: [ + ...[ + `timer`, + `nodetimer`, + `sleep`, + `clock`, + `spiedclock`, + `uptime`, + `osuptime`, + `abort`, + `waitasync`, + `loopdelay`, + ].map((timed) => judging(`a`, `fixed seed`, 7, { timed, work: 1 })), + judging(`a`, `fixed seed`, 1, { work: 1 }), + ], + }, + { + id: `w_fastturn`, owners: [`a`], body: [{ kind: `shortcut`, op: `>`, k: 2, not: false }], tests: [ - judging(`a`, `fixed seed`, 7, { timed, work: 1 }), - judging(`a`, `fixed seed`, 1, { timed, work: 1 }), + judging(`a`, `fixed seed`, 7, { timed: `turn`, work: 1 }), + judging(`a`, `fixed seed`, 1, { timed: `turn`, work: 1 }), ], - })), + }, { id: `w_fastplain`, owners: [`a`], @@ -1726,9 +1748,7 @@ const captureWitness = new Set([ `w_fastbound`, `w_fastcall`, `w_fastcallbound`, - `w_fasttimed`, - `w_fastsleep`, - `w_fastclock`, + `w_fastclocked`, `w_fastturn`, `w_fastplain`, `w_noobs`, @@ -4430,7 +4450,7 @@ function oracleTestFile(plan, file) { const out = [ // A per-file environment, which Vitest reads from the file's text. ...(plan.pragma && file === `a` ? [`// @vitest-environment marked`] : []), - `import { ${plan.tests.some((t) => t.file === file && t.cleanupThrows) ? `afterEach, ` : ``}describe, expect, test${plan.tests.some((t) => t.file === file && t.spies) ? `, vi` : ``} } from 'vitest'`, + `import { ${plan.tests.some((t) => t.file === file && t.cleanupThrows) ? `afterEach, ` : ``}describe, expect, test${plan.tests.some((t) => t.file === file && (t.spies || t.timed === `spiedclock`)) ? `, vi` : ``} } from 'vitest'`, ...(plan.tests.some((t) => t.file === file && t.shrinks) ? [`import fc from 'fast-check'`] : []), @@ -4440,6 +4460,14 @@ function oracleTestFile(plan, file) { ...(plan.tests.some((t) => t.file === file && t.timed === `sleep`) ? [`import { setTimeout as sleep } from 'node:timers/promises'`] : []), + ...(plan.tests.some((t) => t.file === file && t.timed && t.timed !== `turn`) + ? [ + `import { promisify } from 'node:util'`, + `import { setTimeout as nodeSetTimeout } from 'node:timers'`, + `import { uptime as osUptime } from 'node:os'`, + `import { monitorEventLoopDelay } from 'node:perf_hooks'`, + ] + : []), ...(plan.tests.some((t) => t.file === file && t.stops) ? [ `import { writeFileSync as recordPid } from 'node:fs'`, @@ -4501,9 +4529,16 @@ function oracleTestFile(plan, file) { ...(t.timed ? [ { - timer: ` await new Promise((r) => setTimeout(r, 0))`, + timer: ` if ((await promisify(setTimeout)(0, 'v')) !== 'v') throw new Error('promisify(setTimeout) lost its value')`, + nodetimer: ` if ((await promisify(nodeSetTimeout)(0, 'v')) !== 'v') throw new Error('promisify of node:timers lost its value')`, sleep: ` await sleep(0)`, clock: ` if (Date.now() < 0) throw new Error('the clock reads before 1970')`, + spiedclock: ` { const spy = vi.spyOn(Date, 'now'); if (Date.now() < 0) throw new Error('the clock reads before 1970'); spy.mockRestore() }`, + uptime: ` if (process.uptime() < 0) throw new Error('negative uptime')`, + osuptime: ` if (osUptime() < 0) throw new Error('negative uptime')`, + abort: ` if (AbortSignal.timeout(60000).aborted) throw new Error('aborted at once')`, + waitasync: ` if (Atomics.waitAsync(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 0).value !== 'timed-out') throw new Error('waitAsync did not time out')`, + loopdelay: ` { const delay = monitorEventLoopDelay(); delay.enable(); delay.disable() }`, turn: ` await new Promise((r) => setImmediate(r))`, }[t.timed], ] diff --git a/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs b/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs index e7ea2e6..8ecb918 100644 --- a/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs +++ b/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs @@ -1700,23 +1700,11 @@ async function driver(plan, observe) { if ( m.operator === `if-false` && m.class === `not-observed` && - m.function === `w_fasttimed` - ) - states.add(`a work rise in a test that made a timer judged nothing`) - if ( - m.operator === `if-false` && - m.class === `not-observed` && - m.function === `w_fastsleep` + m.function === `w_fastclocked` ) states.add( - `a work rise in a test that slept through timers/promises judged nothing`, + `a work rise in tests that reach the clock by every path judged nothing`, ) - if ( - m.operator === `if-false` && - m.class === `not-observed` && - m.function === `w_fastclock` - ) - states.add(`a work rise in a test that read the clock judged nothing`) if ( m.operator === `if-false` && m.outcome === `killed` && @@ -2588,9 +2576,7 @@ const required = [ `not judged`, `extra work`, `killed by a work bound`, - `a work rise in a test that made a timer judged nothing`, - `a work rise in a test that slept through timers/promises judged nothing`, - `a work rise in a test that read the clock judged nothing`, + `a work rise in tests that reach the clock by every path judged nothing`, `a work rise in a test that waited a turn killed`, `a work rise on an unregistered route judged nothing`, `branch never taken`, From 35735d9daf8cd5339741daf67ba53fe9fdbb8483 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Thu, 12 Nov 2026 17:32:33 -0700 Subject: [PATCH 14/20] fix(oracle-gaps): see every clock source, keep promisify(setTimeout), and judge a read by its whole stack Round-3 review of #48: - R1: the capture also tells a test that polls process.uptime, os.uptime, performance.eventLoopUtilization, AbortSignal.timeout, Atomics.waitAsync with a timeout, monitorEventLoopDelay, or Temporal.Now. - R2: every wrapper (the global timers and the clock reads) keeps the own properties of the function it wraps, util.promisify.custom included, so util.promisify(setTimeout) resolves with its value under the CLI, in the unmutated baseline too (the global wrapper on main dropped it). - R3: a clock read counts when any frame of its stack is the test's or production's, so a user's read through a vi.spyOn passthrough (tinyspy's frame first) counts. Flagging every read (variant a) was measured on klona with bounds: 216 of 218 killed, json M16 and full M17 lost, because the oracle framework's own reads flag the tests that carried those rises; so reads whose every frame is the runner's, fast-check's, or the framework's still do not count. RED at b21d188: e2e "every outcome class": "devBuild ... blocked, owed not-observed" (file a's baseline fails: promisify of the wrapped timer throws). GREEN after. Mutants: a wrapper that drops symbol properties (blocked, as at b21d188); no process.uptime patch ("w_fastclocked if-false killed, owed not-observed"); the first frame decides (the same). Klona with bounds: 218 of 218 killed. Co-authored-by: Isaac --- .../oracles/oracle-gaps/capture-setup.mjs | 90 ++++++++++++++----- 1 file changed, 66 insertions(+), 24 deletions(-) diff --git a/packages/oracles/oracle-gaps/capture-setup.mjs b/packages/oracles/oracle-gaps/capture-setup.mjs index 7e3fc8b..c782665 100644 --- a/packages/oracles/oracle-gaps/capture-setup.mjs +++ b/packages/oracles/oracle-gaps/capture-setup.mjs @@ -63,6 +63,17 @@ const realInterval = real.setInterval const realNow = real.now const realUse = real.use +// A wrapper keeps every own property of the function it wraps, such as +// `util.promisify.custom`, so `util.promisify(setTimeout)` still resolves +// with its value under the capture (law 3: a test that passes alone passes +// under the CLI). +function keepProperties(from, to) { + for (const key of Reflect.ownKeys(from)) + if (key !== `prototype`) + Object.defineProperty(to, key, Object.getOwnPropertyDescriptor(from, key)) + return to +} + // The timers that code in this worker has pending, counted where they are // made, each with when it is next due: Node's resource list does not // reliably show one entry per timer. @@ -73,7 +84,7 @@ const realUse = real.use const pending = (globalThis.__oracleGapsPendingTimers ??= (() => { const handles = new Map() const wrap = (make, clear, once, timed) => { - globalThis[make.name] = function (callback, ...rest) { + globalThis[make.name] = keepProperties(make, function (callback, ...rest) { if (typeof callback !== `function`) return make(callback, ...rest) // An immediate is due at once; a timer or interval after its delay. const delay = make === globalThis.setImmediate ? 0 : Number(rest[0]) || 0 @@ -92,7 +103,7 @@ const pending = (globalThis.__oracleGapsPendingTimers ??= (() => { // time, as the framework's `withinTurns` does. if (timed) globalThis.__oracleGapsClock?.() return handle - } + }) globalThis[clear.name] = function (handle) { handles.delete(handle) return clear(handle) @@ -106,16 +117,20 @@ const pending = (globalThis.__oracleGapsPendingTimers ??= (() => { // The other paths to the clock, wrapped once per worker as the timers are, // each telling the copy now running (`__oracleGapsClock`): a read of // `Date.now`, `new Date()` or `Date()` with no argument, `performance.now`, -// and `process.hrtime`, and a timer of `node:timers` or -// `node:timers/promises` (which do not go through the global functions). A -// fake clock replaces these, and its reads do not follow the real clock. +// `performance.eventLoopUtilization`, `process.hrtime`, `process.uptime`, +// `os.uptime`, and `Temporal.Now`; a timer of `node:timers` or +// `node:timers/promises` (which do not go through the global functions), +// of `AbortSignal.timeout`, and of `Atomics.waitAsync` with a timeout; and +// `monitorEventLoopDelay`. A fake clock replaces these, and its reads do +// not follow the real clock. A reference taken before this file ran (a +// module that loads before the harness and keeps `Date.now`) is not seen. if (!globalThis.__oracleGapsClocks) { globalThis.__oracleGapsClocks = true const told = (read) => - function (...args) { + keepProperties(read, function (...args) { globalThis.__oracleGapsClock?.() return read.apply(this, args) - } + }) const RealDate = Date RealDate.now = told(RealDate.now) const TrackedDate = new Proxy(RealDate, { @@ -132,10 +147,21 @@ if (!globalThis.__oracleGapsClocks) { RealDate.prototype.constructor = TrackedDate globalThis.Date = TrackedDate performance.now = told(performance.now) - const hrtime = process.hrtime - process.hrtime = Object.assign(told(hrtime), { - bigint: told(hrtime.bigint), - }) + performance.eventLoopUtilization = told(performance.eventLoopUtilization) + process.hrtime = told(process.hrtime) + process.hrtime.bigint = told(process.hrtime.bigint) + process.uptime = told(process.uptime) + AbortSignal.timeout = told(AbortSignal.timeout) + const waitAsync = Atomics.waitAsync + if (waitAsync) + Atomics.waitAsync = keepProperties(waitAsync, function (...args) { + if (args[3] !== undefined && args[3] !== Infinity) + globalThis.__oracleGapsClock?.() + return waitAsync.apply(this, args) + }) + const now = globalThis.Temporal?.Now + for (const name of now ? Object.getOwnPropertyNames(now) : []) + if (typeof now[name] === `function`) now[name] = told(now[name]) const require = createRequire(import.meta.url) const timers = require(`node:timers`) const promises = require(`node:timers/promises`) @@ -143,6 +169,12 @@ if (!globalThis.__oracleGapsClocks) { for (const name of [`setTimeout`, `setInterval`]) module[name] = told(module[name]) promises.scheduler.wait = told(promises.scheduler.wait) + const os = require(`node:os`) + os.uptime = told(os.uptime) + const hooks = require(`node:perf_hooks`) + hooks.monitorEventLoopDelay = told(hooks.monitorEventLoopDelay) + // `util.promisify` of a global or `node:timers` timer resolves through + // `node:timers/promises`, whose functions are told above. syncBuiltinESMExports() } const unit = run.units?.[globalThis.__vitest_worker__?.filepath] @@ -719,23 +751,33 @@ globalThis.__oracleWorkBound = (ratio) => { // Whether the current test's count could follow the clock (maintainer // decision, 2026-11-12, under P3: a kill must replay on an idle machine): // it made a real timer or interval by any path, or read the clock (the -// wrappers above), or held I/O open (`sawIo`). A read that the test runner, -// fast-check, the oracle framework (its shrink budget), Node, or this file -// makes does not count: the caller is the first frame outside this file -// and Node's own. +// wrappers above), or held I/O open (`sawIo`). A read counts when any frame +// of its stack is the test's or production's: a read whose every frame is +// the test runner's, fast-check's, the oracle framework's (its shrink +// budget), Node's, or this file's does not, while a user's read through a +// spy (tinyspy's frame first) still does. let clocked = false const machinery = /[\\/]node_modules[\\/](?:vitest|@vitest|fast-check|@fast-check|tinypool|tinyspy|chai|@tanstack[\\/]oracles)[\\/]|[\\/]packages[\\/]oracles[\\/]src[\\/]/ function fromTest() { - for (const frame of String(new Error().stack).split(`\n`).slice(1)) { - const file = - frame.match(/\((.*):\d+:\d+\)$/)?.[1] ?? - frame.match(/at (.*):\d+:\d+$/)?.[1] - if (file === undefined || file.startsWith(`node:`)) continue - if (file.endsWith(`capture-setup.mjs`)) continue - return !machinery.test(file) - } - return false + const limit = Error.stackTraceLimit + Error.stackTraceLimit = 50 + const stack = String(new Error().stack) + Error.stackTraceLimit = limit + return stack + .split(`\n`) + .slice(1) + .some((frame) => { + const file = + frame.match(/\((.*):\d+:\d+\)$/)?.[1] ?? + frame.match(/at (.*):\d+:\d+$/)?.[1] + return ( + file !== undefined && + !file.startsWith(`node:`) && + !file.endsWith(`capture-setup.mjs`) && + !machinery.test(file) + ) + }) } globalThis.__oracleGapsClock = () => { if (inTest && !clocked && fromTest()) clocked = true From a322b9141d007826f6704f7fed876a15433a7618 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Thu, 12 Nov 2026 17:51:57 -0700 Subject: [PATCH 15/20] ci(oracles): repack the e2e fixed-seed shards by measured cost The work law's witnesses grew the witness module's histories, and at 9087746 the third fixed-seed job ran 10.9 min. Measured locally at 35735d9 (36 units, 581 s), the units are packed longest first into shards of about 203 s, 197 s, and 181 s. Co-authored-by: Isaac --- packages/oracles/scripts/pr-shards.mjs | 45 ++++++++++++-------------- 1 file changed, 21 insertions(+), 24 deletions(-) diff --git a/packages/oracles/scripts/pr-shards.mjs b/packages/oracles/scripts/pr-shards.mjs index 5005b0f..b46de3c 100644 --- a/packages/oracles/scripts/pr-shards.mjs +++ b/packages/oracles/scripts/pr-shards.mjs @@ -279,11 +279,13 @@ export const partitions = { }, }, // The end-to-end gap oracle's fixed-seed units, which a pull request runs - // (`-t "fixed seed"`): 36 units, 440 s in all, measured on this - // branch in one local run under other agents' load (load average about 3). - // A unit's cost follows the generated history its constructed history is - // built from, so a grammar change can move cost between units. The units - // are packed longest first into three shards: 147 s, 147 s, and 147 s. + // (`-t "fixed seed"`): 36 units, 581 s in all, measured in one local run + // at fix/gap-cli-cost-only (2026-11-12, after the work law's witnesses + // joined the witness module; at 9087746 the old packing ran its third CI + // job for 10.9 min). A unit's cost follows the generated history its + // constructed history is built from, and the witness module's histories + // grow with its witnesses. The units are packed longest first into three + // shards: about 203 s, 197 s, and 181 s. 'oracle-gaps-fixed-seed': { args: [`--config`, `oracle-gaps/vitest.config.mjs`, `gap-report-oracle`], suite: `oracle-gaps.report`, @@ -293,31 +295,26 @@ export const partitions = { previously: `fixed seed`, shards: { 1: anyOf([ + 'an edit sequence over the witness module', 'a second daemon check of the budget witness', - 'a declaration fault: brokenOwner', - "a failing owner under its parent's config", - 'a run with eleven contexts', + 'a declaration fault: configThrows', + 'an edit that changed the cold result after a hang replaced the only context', + 'a stale authored mutant and a reformat', + 'an edit to a module that only mutated runs load', 'two authored mutants', - 'a declaration fault: setupThrows', - 'a declaration fault: missingSource', - 'a .ts config in a package with no module type that calls require', - 'a config that --config names', - 'a declaration fault: nonLiteral', - 'a declaration fault: outsideSource', + 'a failing owner under a config that --config names', + 'a run with eleven contexts', ]), 2: anyOf([ - 'an edit sequence through the daemon', + 'generated', 'every outcome class', - 'an edit that changed the cold result after a hang replaced the only context', - 'a lazy module loaded before a hang in the same batch', - 'a declaration fault: noTypescript', - 'ordinary tests left out by default', - 'an ambiguous authored mutant', + 'a declaration fault: brokenOwner', + 'a declaration fault: brokenVitest', + "a failing owner under its parent's config", + 'a declaration fault: setupThrows', 'a package whose parent directory has its config', - 'an awaiting config in a package with no module type', - 'a .cjs config that calls require', - "test.projects in a parent's config", - 'a declaration fault: configNoDefault', + 'a load failure beside a hang', + 'a package with no config', ]), 3: rest, }, From f73fd4b48ff5d78f457e59e2b5eb7acd3b9176a0 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Thu, 12 Nov 2026 18:03:08 -0700 Subject: [PATCH 16/20] test(oracle-gaps): a synchronous Atomics.wait with a timeout follows the clock Round 4 of the #48 review: w_fastclocked gains a test that waits with Atomics.wait and a timeout, and the classify grammar names it. Co-authored-by: Isaac --- .../gap-report-classify-oracle.property.test.mjs | 2 ++ packages/oracles/oracle-gaps/gap-report-model-oracle.mjs | 6 ++++-- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs b/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs index 516fcf7..257566f 100644 --- a/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs +++ b/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs @@ -489,6 +489,7 @@ const testRecord = fc.record({ `process.uptime`, `os.uptime`, `AbortSignal.timeout`, + `Atomics.wait`, `Atomics.waitAsync`, `monitorEventLoopDelay`, `io`, @@ -3333,6 +3334,7 @@ const constructed = [ [`process.uptime`, `polled process.uptime`], [`os.uptime`, `polled os.uptime`], [`AbortSignal.timeout`, `polled an AbortSignal.timeout`], + [`Atomics.wait`, `waited with Atomics.wait and a timeout`], [`Atomics.waitAsync`, `waited with Atomics.waitAsync and a timeout`], [`monitorEventLoopDelay`, `polled monitorEventLoopDelay`], [`io`, `held I/O open`], diff --git a/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs b/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs index ad9e003..0b08078 100644 --- a/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs +++ b/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs @@ -964,8 +964,8 @@ const witness = [ // test that passes alone passes under the CLI), a sleep through // `node:timers/promises`, `Date.now()` (and through a `vi.spyOn` // passthrough, whose spy frame comes first), `process.uptime()`, - // `os.uptime()`, `AbortSignal.timeout`, `Atomics.waitAsync` with a - // timeout, and `monitorEventLoopDelay`. One untimed test runs the + // `os.uptime()`, `AbortSignal.timeout`, `Atomics.wait` and + // `Atomics.waitAsync` with a timeout, and `monitorEventLoopDelay`. One untimed test runs the // general path. A mutant is killed if any test still judges work, so its // `if-false` mutant stays `not-observed` only while every path is seen. // Tests that wait one turn (`setImmediate`, as `withinTurns` does) and @@ -986,6 +986,7 @@ const witness = [ `osuptime`, `abort`, `waitasync`, + `wait`, `loopdelay`, ].map((timed) => judging(`a`, `fixed seed`, 7, { timed, work: 1 })), judging(`a`, `fixed seed`, 1, { work: 1 }), @@ -4538,6 +4539,7 @@ function oracleTestFile(plan, file) { osuptime: ` if (osUptime() < 0) throw new Error('negative uptime')`, abort: ` if (AbortSignal.timeout(60000).aborted) throw new Error('aborted at once')`, waitasync: ` if (Atomics.waitAsync(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 0).value !== 'timed-out') throw new Error('waitAsync did not time out')`, + wait: ` if (Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 0) !== 'timed-out') throw new Error('wait did not time out')`, loopdelay: ` { const delay = monitorEventLoopDelay(); delay.enable(); delay.disable() }`, turn: ` await new Promise((r) => setImmediate(r))`, }[t.timed], From 661d2fd626bb190d9b7637e8bf21d27ec54a9a36 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Thu, 12 Nov 2026 18:24:04 -0700 Subject: [PATCH 17/20] fix(oracle-gaps): a synchronous Atomics.wait with a timeout sets the clock flag Round 4 of the #48 review: Atomics.wait with a timeout returns by the clock, or by another thread's timing, so it sets clocked as Atomics.waitAsync does. RED at cfa10b7: e2e "every outcome class", "w_fastclocked:101:if-false killed, owed not-observed". GREEN after. Co-authored-by: Isaac --- .../oracles/oracle-gaps/capture-setup.mjs | 27 +++++++++++-------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/packages/oracles/oracle-gaps/capture-setup.mjs b/packages/oracles/oracle-gaps/capture-setup.mjs index c782665..4d61856 100644 --- a/packages/oracles/oracle-gaps/capture-setup.mjs +++ b/packages/oracles/oracle-gaps/capture-setup.mjs @@ -120,10 +120,11 @@ const pending = (globalThis.__oracleGapsPendingTimers ??= (() => { // `performance.eventLoopUtilization`, `process.hrtime`, `process.uptime`, // `os.uptime`, and `Temporal.Now`; a timer of `node:timers` or // `node:timers/promises` (which do not go through the global functions), -// of `AbortSignal.timeout`, and of `Atomics.waitAsync` with a timeout; and -// `monitorEventLoopDelay`. A fake clock replaces these, and its reads do -// not follow the real clock. A reference taken before this file ran (a -// module that loads before the harness and keeps `Date.now`) is not seen. +// of `AbortSignal.timeout`, and of `Atomics.wait` or `Atomics.waitAsync` +// with a timeout; and `monitorEventLoopDelay`. A fake clock replaces +// these, and its reads do not follow the real clock. A reference taken +// before this file ran (a module that loads before the harness and keeps +// `Date.now`) is not seen. if (!globalThis.__oracleGapsClocks) { globalThis.__oracleGapsClocks = true const told = (read) => @@ -152,13 +153,17 @@ if (!globalThis.__oracleGapsClocks) { process.hrtime.bigint = told(process.hrtime.bigint) process.uptime = told(process.uptime) AbortSignal.timeout = told(AbortSignal.timeout) - const waitAsync = Atomics.waitAsync - if (waitAsync) - Atomics.waitAsync = keepProperties(waitAsync, function (...args) { - if (args[3] !== undefined && args[3] !== Infinity) - globalThis.__oracleGapsClock?.() - return waitAsync.apply(this, args) - }) + // A wait with a timeout returns by the clock, or by another thread's + // timing. + for (const name of [`wait`, `waitAsync`]) { + const wait = Atomics[name] + if (wait) + Atomics[name] = keepProperties(wait, function (...args) { + if (args[3] !== undefined && args[3] !== Infinity) + globalThis.__oracleGapsClock?.() + return wait.apply(this, args) + }) + } const now = globalThis.Temporal?.Now for (const name of now ? Object.getOwnPropertyNames(now) : []) if (typeof now[name] === `function`) now[name] = told(now[name]) From 717d7ccaa140c24743539537cfafd0a45b90d017 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Thu, 12 Nov 2026 18:40:35 -0700 Subject: [PATCH 18/20] ci(oracles): repack the e2e fixed-seed shards by their CI costs At 661d2fd the second fixed-seed job ran 10.8 min: CI's relative unit costs differ from a local run's (brokenOwner, generated, and every outcome class each cost about 3 times their local time there). The units are packed longest first by the costs read from the three jobs' logs: about 427 s, 421 s, and 429 s. Co-authored-by: Isaac --- packages/oracles/scripts/pr-shards.mjs | 27 +++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/packages/oracles/scripts/pr-shards.mjs b/packages/oracles/scripts/pr-shards.mjs index b46de3c..7acacec 100644 --- a/packages/oracles/scripts/pr-shards.mjs +++ b/packages/oracles/scripts/pr-shards.mjs @@ -279,13 +279,13 @@ export const partitions = { }, }, // The end-to-end gap oracle's fixed-seed units, which a pull request runs - // (`-t "fixed seed"`): 36 units, 581 s in all, measured in one local run - // at fix/gap-cli-cost-only (2026-11-12, after the work law's witnesses - // joined the witness module; at 9087746 the old packing ran its third CI - // job for 10.9 min). A unit's cost follows the generated history its + // (`-t "fixed seed"`): 36 units, about 1,280 s in all, measured in CI at + // 661d2fd (2026-11-13, from each unit's reach line in the three jobs' + // logs; the units' relative costs in CI differ from a local run's, so the + // packing follows CI). A unit's cost follows the generated history its // constructed history is built from, and the witness module's histories // grow with its witnesses. The units are packed longest first into three - // shards: about 203 s, 197 s, and 181 s. + // shards: about 427 s, 421 s, and 429 s. 'oracle-gaps-fixed-seed': { args: [`--config`, `oracle-gaps/vitest.config.mjs`, `gap-report-oracle`], suite: `oracle-gaps.report`, @@ -294,28 +294,29 @@ export const partitions = { // select exactly these. previously: `fixed seed`, shards: { + // 225 s, 132 s, 45 s, and small units. 1: anyOf([ 'an edit sequence over the witness module', + 'a declaration fault: noTypescript', 'a second daemon check of the budget witness', 'a declaration fault: configThrows', 'an edit that changed the cold result after a hang replaced the only context', 'a stale authored mutant and a reformat', - 'an edit to a module that only mutated runs load', 'two authored mutants', - 'a failing owner under a config that --config names', - 'a run with eleven contexts', ]), + // 206 s, 201 s, and small units. 2: anyOf([ 'generated', - 'every outcome class', 'a declaration fault: brokenOwner', - 'a declaration fault: brokenVitest', - "a failing owner under its parent's config", - 'a declaration fault: setupThrows', - 'a package whose parent directory has its config', + 'a run with eleven contexts', + 'a failing owner under a config that --config names', + 'an edit to a module that only mutated runs load', 'a load failure beside a hang', 'a package with no config', ]), + // The rest: every outcome class (186 s), apiLessTypescript (112 s), the + // edit sequence through the daemon (77 s), brokenVitest (26 s), and + // small units. 3: rest, }, }, From 2717f39f27c1268a9252b6bfbb758e4bd9cb60d4 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Thu, 12 Nov 2026 20:57:41 -0700 Subject: [PATCH 19/20] test(oracle-gaps): derive a generated test's work bound from its plan, and keep error.log A drawn work dimension in the e2e's generated test record shifted fast-check's stream, so every generated base, and every constructed history built on one, changed and grew heavier: generated 23 s on main to 172 s in CI. The bound is now derived from the plan (1 when the test's input is a multiple of 3), with fc.constant(null) in the record, so the grammar's histories are main's again. Full fixed seed, locally: 827 s with the drawn bound, 492 s derived (generated 93 s to 11 s, noTypescript 73 s to 2 s). Two measured states count how often a generated function reaches the work law: 0 with the drawn bound and 0 derived, across the 36 fixed-seed units; the e2e's work-law reach comes from the witness module either way (extra work 6 and 3, killed by a work bound 6 and 3). A run that exits 3 now keeps the start of its error.log in the observed stderr, so a "did not complete" difference carries the internal error's stack (#56). Co-authored-by: Isaac --- .../oracle-gaps/gap-report-model-oracle.mjs | 13 +++++++--- .../gap-report-oracle.property.test.mjs | 24 +++++++++++++++++-- 2 files changed, 32 insertions(+), 5 deletions(-) diff --git a/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs b/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs index 0b08078..b6bfbc5 100644 --- a/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs +++ b/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs @@ -405,9 +405,10 @@ const testCase = fc.record({ fc.record({ route: fc.constantFrom(...routes), input: constant }), { nil: null }, ), - // A test may declare a work bound, as an oracle property's `work` does: - // a ratio of 1 rejects a mutant that does any more work there. - work: fc.constantFrom(null, null, null, 1), + // A test's work bound is derived from its plan (see the tests below), so + // the grammar draws nothing for it and its generated histories stay those + // of the grammar without the work law. + work: fc.constant(null), // Only a witness waits on a timer, sleeps, reads the clock, or waits a // turn: its work could follow the clock. timed: fc.constant(false), @@ -2694,6 +2695,12 @@ function resolveHistory(history) { ...history.tests.map((t) => ({ ...t, fn: generatedIds[t.fn % generatedIds.length], + // A generated test declares a work bound of 1 (an oracle property's + // `work`, which rejects a mutant that does any more work there) when + // its input is a multiple of 3: derived from the plan, not drawn, so + // the grammar's generated histories, and their cost, stay those of + // the grammar without the work law. + work: t.work ?? (t.input % 3 === 0 ? 1 : null), })), ...functions.flatMap((fn) => fn.generated ? [] : fn.tests.map((t) => ({ ...t, fn: t.fn ?? fn.id })), diff --git a/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs b/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs index 8ecb918..c1d6e22 100644 --- a/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs +++ b/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs @@ -931,7 +931,12 @@ function readRun(dir, out, status, stderr, cli, stdout) { ? undefined : endingOf(status, stderr, out, (text) => namedScratch(text, scratch)), exit: status, - stderr: named === undefined ? undefined : named.slice(-1500), + // An internal error's stack is in the run's error.log (law 12), which + // the scratch directory's removal takes with it: a failure keeps it. + stderr: + named === undefined + ? undefined + : `${status === 3 && existsSync(join(out, `error.log`)) ? `error.log: ${namedScratch(readFileSync(join(out, `error.log`), `utf8`), scratch).slice(0, 2000)}\n` : ``}${named.slice(-1500)}`, reported: results !== undefined, streamed, // Law 9 compares everything the run reports except how it was @@ -1696,6 +1701,17 @@ async function driver(plan, observe) { if (m.class === `extra-work`) states.add(`extra work`) if (m.outcome === `killed` && m.laws?.includes(`work bound`)) states.add(`killed by a work bound`) + // The same in a generated function, whose tests' bounds the plan + // derives (measured: how often generated histories reach the law). + const inGenerated = /^(?:f\d+|C\d\.)/.test(m.function ?? ``) + if (inGenerated && m.class === `extra-work`) + states.add(`extra work in a generated function`) + if ( + inGenerated && + m.outcome === `killed` && + m.laws?.includes(`work bound`) + ) + states.add(`killed by a work bound in a generated function`) // The early return whose extra work no test judges (law 4). if ( m.operator === `if-false` && @@ -2791,7 +2807,11 @@ oracleProperty({ // of those seven histories, its law the first failure's. arbitrary: fc.limitShrink(histories, 6), states: required, - measured: staticStates, + measured: [ + ...staticStates, + `extra work in a generated function`, + `killed by a work bound in a generated function`, + ], runs: 2, fixedSeeds: [20261105], constructed: withDeclaration([ From fa88316c78bbc73754173c08c7b739776b10425b Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Thu, 12 Nov 2026 21:16:40 -0700 Subject: [PATCH 20/20] ci(oracles): repack the e2e fixed-seed shards for the grammar's restored costs With the work bound derived, the generated histories are main's again, so the shards follow main's CI costs (ae3184d): about 290 s, 277 s, and 260 s of CI time, and 135 s, 125 s, and 144 s locally. Co-authored-by: Isaac --- packages/oracles/scripts/pr-shards.mjs | 41 +++++++++++--------------- 1 file changed, 17 insertions(+), 24 deletions(-) diff --git a/packages/oracles/scripts/pr-shards.mjs b/packages/oracles/scripts/pr-shards.mjs index 7acacec..2a81190 100644 --- a/packages/oracles/scripts/pr-shards.mjs +++ b/packages/oracles/scripts/pr-shards.mjs @@ -279,13 +279,14 @@ export const partitions = { }, }, // The end-to-end gap oracle's fixed-seed units, which a pull request runs - // (`-t "fixed seed"`): 36 units, about 1,280 s in all, measured in CI at - // 661d2fd (2026-11-13, from each unit's reach line in the three jobs' - // logs; the units' relative costs in CI differ from a local run's, so the - // packing follows CI). A unit's cost follows the generated history its - // constructed history is built from, and the witness module's histories - // grow with its witnesses. The units are packed longest first into three - // shards: about 427 s, 421 s, and 429 s. + // (`-t "fixed seed"`): 38 units, measured in CI on main at ae3184d (each + // unit's cost read from its reach line's time in the three jobs' logs), + // whose generated histories this branch keeps (the work law's bound in + // the grammar is derived from the plan, not drawn, 2026-11-13). A unit's + // cost follows the generated history its constructed history is built + // from, and the witness module's histories grow with its witnesses. The + // units are packed longest first into three shards: about 290 s, 277 s, + // and 260 s of CI time. 'oracle-gaps-fixed-seed': { args: [`--config`, `oracle-gaps/vitest.config.mjs`, `gap-report-oracle`], suite: `oracle-gaps.report`, @@ -294,29 +295,21 @@ export const partitions = { // select exactly these. previously: `fixed seed`, shards: { - // 225 s, 132 s, 45 s, and small units. + // 261 s, 25 s, and a small unit. 1: anyOf([ 'an edit sequence over the witness module', - 'a declaration fault: noTypescript', - 'a second daemon check of the budget witness', - 'a declaration fault: configThrows', - 'an edit that changed the cold result after a hang replaced the only context', 'a stale authored mutant and a reformat', - 'two authored mutants', + 'a failing owner under a config that --config names', ]), - // 206 s, 201 s, and small units. + // 134 s, 101 s, 27 s, and 15 s. 2: anyOf([ - 'generated', - 'a declaration fault: brokenOwner', - 'a run with eleven contexts', - 'a failing owner under a config that --config names', - 'an edit to a module that only mutated runs load', - 'a load failure beside a hang', - 'a package with no config', + 'every outcome class', + 'a second daemon check of the budget witness', + 'a declaration fault: brokenVitest', + 'an edit that changed the cold result after a hang replaced the only context', ]), - // The rest: every outcome class (186 s), apiLessTypescript (112 s), the - // edit sequence through the daemon (77 s), brokenVitest (26 s), and - // small units. + // The rest: the edit sequence through the daemon (95 s), brokenOwner + // (83 s), generated (23 s), the lazy module (12 s), and small units. 3: rest, }, },