Skip to content

self-oracle timeout law: 'timeout presentation was not exercised' on seed -1999165303 #59

Closed

Description

@KyleAMathews

The random-seed counterexample from PR #7 CI reproduces deterministically and is not a timing problem. The self-oracle's timeout law fires because it assumes that every slow spec in a plan runs its driver. A spec whose owns declaration fails to load never runs the driver, so the law reports a missing timeout presentation instead of the correct registration failure. This is a self-oracle law defect, not a framework defect: the production check at src/oracle-property.ts:828 rejects the declaration as the registration law says.

Replay

ORACLE_SEED=-1999165303 ORACLE_PATH=0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0 ORACLE_PROPERTY='self-oracle.framework' ORACLE_UNIT='random seed: generated' vitest run -t '^self-oracle\.framework(?: | > )random seed: generated$'

I ran it through packages/oracles with --config vitest.oracle-framework.config.mjs --maxWorkers=2, which is how the CI shard runs it.

CI job: https://github.com/TanStack/oracles/actions/runs/43458451421/job/131998714996

Failure: self-oracle.framework violated at checkpoint "child" (law: timeout). Difference: timeout presentation was not exercised, after 26 shrinks.

Shrunk case

One draft. The fields that matter are:

  • timing: "slow", so the spec is slow with a 100 ms test timeout.
  • ownsMode: "empty", so the spec declares owns: [].
  • timeoutPresentation: true on the plan.

The rest is incidental, including source: generated, runs: 2, seedless: true, nondeterministic: true, and clock: true.

Classification: deterministic, not timing

  • Pinned seed on the unmodified origin/main (affeb1a): 6 of 6 runs fail, on Vitest 4.1.12 (1 run plus 5 more) and on Vitest 5.1.0 (5 runs). Each run reports the same violation.
  • Unpinned random unit on Vitest 5.1.0: 1 failure in 10 runs, with ORACLE_SEED=1279286490. It shows the same law and difference. I did not separately shrink that seed.
  • The machine was under heavy load during these runs (load average about 25 to 60 on 18 cores). The failure occurred anyway, and the child run finishes in about 1 second. The 100 ms timeout is never reached.

Root cause

  1. The generated child spec file fails to load. Its owns is empty, so the child reports owns must be a non-empty literal list of strings, and the child runs zero tests. I confirmed this by temporarily logging the child's report in a scratch copy. The child reported tests: [], and no child log was written.
  2. Because the spec never runs, the driver never appends to the child log. The driver's log line and its await for the slow timer are at tests/oracle-property-self-oracle-fixture.ts:837 and :840. So slowCalls is empty (tests/oracle-property-self-oracle.test.ts:5142), and the timeout-presentation observation has exercised: false (:5260-5261).
  3. The timeout check in checkSuite (tests/oracle-property-self-oracle.test.ts:6356-6366) requires an exercised timeout presentation whenever the plan has a slow spec and timeoutPresentation is set. It does not consult the model's loads expectation. It also runs before the child checks that would have matched the load error, so it reports a false violation.
  4. The model already predicts this. predictSpec returns loads: false for ownsMode of empty, string, or bad entry (tests/oracle-property-self-oracle.test.ts:2184-2191), and the generator produces empty at :845-856.

I checked the fix direction in a scratch copy, not in the worktree. Adding && expected.get(spec.id)?.loads to the timeout precondition at tests/oracle-property-self-oracle.test.ts:6356-6357 makes the pinned replay pass 3 of 3.

Fix direction

Change the timeout law's precondition from "a slow spec exists" to "a slow spec the model says loads". That obligation is grounded in the model, not in wall-clock time, so a controlled clock is not needed. Keep the registration law as the first check for such a case, so a load failure is reported as a load failure.

Before merging, run the full vitest.oracle-framework.config.mjs suite with the change. Other load-failing causes (unregistered, invalid work.ratio, predict mismatches, and others in the loads: false returns at :2178-2312) reach the same branch by the same mechanism. I reproduced only the ownsMode: empty case, so the fix should be checked against the others too.

Limits

This issue was written by Isaac.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions