Skip to content

fix(oracles): the self-oracle timeout law applies only to slow specs that load - #63

Draft
KyleAMathews wants to merge 2 commits into
mainfrom
fix/self-oracle-timeout-load-precondition
Draft

KyleAMathews wants to merge 2 commits into
mainfrom
fix/self-oracle-timeout-load-precondition

Conversation

@KyleAMathews

@KyleAMathews KyleAMathews commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #59

Decisions

  • P2 (model independence): touched and kept. The guard reads only expected, the model's own loads prediction. It adds no production import, mutant, or probe to the model.
  • P9 (oracle-first): touched and satisfied. RED from the oracle-only commit (CI PR test(oracles): RED replay for the #59 timeout-law precondition (evidence only) #62) and the parent, GREEN on this branch, and the hostile mutants (a) and (b) listed below.
  • P4 (reach keeps randomness): touched and kept. The random seeds still run on every PR, and the guard does not gate any generated history.
  • Not in the register, flagged for coordinator and maintainer review: the timeout obligation is now scoped to slow specs that load (the model's loads expectation). This is a law-scope decision, and it has not been ratified. It was pushed before the register was read. Hold merge until it is recorded or overruled.

Law

The self-oracle's timeout law (checkSuite, packages/oracles/tests/oracle-property-self-oracle.test.ts) says that when a plan has a slow spec and timeout presentation, the run child must exercise that presentation. A slow spec whose declaration fails to load (for example owns: []) never runs its driver, so it owes no timed-out call. The authority is the model's own load prediction in predictSpec (the loads: false returns for owns shape, among others), and the registration law in src/oracle-property.ts. This change only conditions the timeout obligation on that prediction. It does not change the law's text or the model.

Reach

  • The shrunk case of self-oracle timeout law: 'timeout presentation was not exercised' on seed -1999165303 #59 is replayed as a fixed history, timeout presentation of a slow spec that fails to load, under timeout presentation with timing: slow and ownsMode: empty. It enters registration-error.
  • The defect was found by the unpinned random unit: 1 failure in 10 runs on Vitest 5.1.0 (seed 1279286490). The pinned seed -1999165303 reproduces it deterministically.
  • The guard is also exercised by every loaded slow draft in the existing timeout units (timeout continuation presentation, timeout every receiving cut, and the timeout input units). Those still require the exercised presentation.
  • Other load-failing causes (unregistered, invalid work.ratio, predict mismatches, and the other loads: false returns) reach the same branch by the same mechanism. Only ownsMode: empty is pinned here. Their coverage is in the full suite run.

RED

  • Pinned seed on origin/main (affeb1a), Vitest 4.1.12, 2 workers, with the CI config: ORACLE_SEED=-1999165303 ORACLE_PATH=0:0:…:0 ORACLE_PROPERTY='self-oracle.framework' ORACLE_UNIT='random seed: generated' vitest run -t '^self-oracle\.framework(?: | > )random seed: generated$'. 6 of 6 runs fail with timeout presentation was not exercised. On Vitest 5.1.0, 5 of 5 fail.
  • Fixed-history regression on the oracle-only commit c307c94 (guard absent): -t 'timeout presentation of a slow spec that fails to load' fails with timeout presentation was not exercised.
  • CI RED at the oracle-only commit: test(oracles): RED replay for the #59 timeout-law precondition (evidence only) #62 (draft evidence PR for c307c94).

GREEN

  • Pinned seed on this branch with the guard: 3 of 3 runs pass on Vitest 4.1.12.
  • Regression with the guard: passes.
  • Full vitest.oracle-framework.config.mjs suite, Vitest 4.1.12, 2 workers, with the guard but before the regression entry was added: 851 of 851 pass in 37 minutes. The local Vitest 5.1.0 full run was stopped, per the maintainer's rule. CI runs both Vitest 4 and Vitest 5 framework shards on this PR.

Mutant

  • (a) The guard removed (original defect), on this branch with the regression entry: the regression fails with timeout presentation was not exercised, and the pinned seed fails the same way (the earlier RED).
  • (b) The child stops reporting its timeout presentation (if (false && …) on the timeout-presentation observation). On the loaded draft unit timeout continuation presentation, the oracle rejects it: 3 timeout presentation was not exercised failures across 4 failed tests. The guard does not hide a real timeout bug.

Limits

  • Only the ownsMode: empty load failure is pinned. The other load-failing causes are argued from the same mechanism and rely on the full suite in CI.
  • The random failure rate is estimated from 10 unpinned runs, so it is a rough figure.
  • No framework (src/) code is changed. The fix is only in the oracle's precondition.

This pull request and its description were written by Isaac.

KyleAMathews and others added 2 commits November 13, 2026 10:34
…ation

The shrunk counterexample of #59: a slow spec whose owns list is empty
fails to load, runs no driver, and the timeout law reports a missing
timeout presentation. This fixed history reproduces it; the law is
fixed in the next commit.

Co-authored-by: Isaac <no-reply@databricks.com>
A slow spec that fails to load never runs its driver, so it cannot owe a
timed-out call to present. Condition the timeout obligation on the
model's load expectation, which already predicts the load failure.

Fixes #59

Co-authored-by: Isaac <no-reply@databricks.com>
@coderabbitai

coderabbitai Bot commented Nov 13, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

self-oracle timeout law: 'timeout presentation was not exercised' on seed -1999165303

1 participant