Repository navigation
Description
The random-seed counterexample from PR #7 CI reproduces deterministically and is not a timing problem. The self-oracle's timeout law fires because it assumes that every slow spec in a plan runs its driver. A spec whose owns declaration fails to load never runs the driver, so the law reports a missing timeout presentation instead of the correct registration failure. This is a self-oracle law defect, not a framework defect: the production check at src/oracle-property.ts:828 rejects the declaration as the registration law says.
Replay
ORACLE_SEED=-1999165303 ORACLE_PATH=0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0 ORACLE_PROPERTY='self-oracle.framework' ORACLE_UNIT='random seed: generated' vitest run -t '^self-oracle\.framework(?: | > )random seed: generated$'I ran it through packages/oracles with --config vitest.oracle-framework.config.mjs --maxWorkers=2, which is how the CI shard runs it.
CI job: https://github.com/TanStack/oracles/actions/runs/43458451421/job/131998714996
Failure: self-oracle.framework violated at checkpoint "child" (law: timeout). Difference: timeout presentation was not exercised, after 26 shrinks.
Shrunk case
One draft. The fields that matter are:
timing: "slow", so the spec isslowwith a 100 ms test timeout.ownsMode: "empty", so the spec declaresowns: [].timeoutPresentation: trueon the plan.
The rest is incidental, including source: generated, runs: 2, seedless: true, nondeterministic: true, and clock: true.
Classification: deterministic, not timing
- Pinned seed on the unmodified
origin/main(affeb1a): 6 of 6 runs fail, on Vitest 4.1.12 (1 run plus 5 more) and on Vitest 5.1.0 (5 runs). Each run reports the same violation. - Unpinned random unit on Vitest 5.1.0: 1 failure in 10 runs, with
ORACLE_SEED=1279286490. It shows the same law and difference. I did not separately shrink that seed. - The machine was under heavy load during these runs (load average about 25 to 60 on 18 cores). The failure occurred anyway, and the child run finishes in about 1 second. The 100 ms timeout is never reached.
Root cause
- The generated child spec file fails to load. Its
ownsis empty, so the child reportsowns must be a non-empty literal list of strings, and the child runs zero tests. I confirmed this by temporarily logging the child's report in a scratch copy. The child reportedtests: [], and no child log was written. - Because the spec never runs, the driver never appends to the child log. The driver's log line and its
awaitfor the slow timer are attests/oracle-property-self-oracle-fixture.ts:837and:840. SoslowCallsis empty (tests/oracle-property-self-oracle.test.ts:5142), and thetimeout-presentationobservation hasexercised: false(:5260-5261). - The timeout check in
checkSuite(tests/oracle-property-self-oracle.test.ts:6356-6366) requires an exercised timeout presentation whenever the plan has a slow spec andtimeoutPresentationis set. It does not consult the model'sloadsexpectation. It also runs before the child checks that would have matched the load error, so it reports a false violation. - The model already predicts this.
predictSpecreturnsloads: falseforownsModeofempty,string, orbad entry(tests/oracle-property-self-oracle.test.ts:2184-2191), and the generator producesemptyat:845-856.
I checked the fix direction in a scratch copy, not in the worktree. Adding && expected.get(spec.id)?.loads to the timeout precondition at tests/oracle-property-self-oracle.test.ts:6356-6357 makes the pinned replay pass 3 of 3.
Fix direction
Change the timeout law's precondition from "a slow spec exists" to "a slow spec the model says loads". That obligation is grounded in the model, not in wall-clock time, so a controlled clock is not needed. Keep the registration law as the first check for such a case, so a load failure is reported as a load failure.
Before merging, run the full vitest.oracle-framework.config.mjs suite with the change. Other load-failing causes (unregistered, invalid work.ratio, predict mismatches, and others in the loads: false returns at :2178-2312) reach the same branch by the same mechanism. I reproduced only the ownsMode: empty case, so the fix should be checked against the others too.
Limits
- Only one shrunk case and one unpinned seed were examined. The fix was verified only on the pinned replay.
- The failure rate in CI is unknown beyond the 1 in 10 sample above. It is latent on
origin/mainas well, so it is not specific to PR docs(oracles): ship setup, design, and review skills, with a prompt to try them #7. - No framework code was changed or pushed.
This issue was written by Isaac.
Activity
test(oracles): declare the reach state for a slow spec that fails to …