Skip to content

oracle-gaps: an internal error (exit 3) in CI during the e2e stability rerun under load #56

Open

Description

@KyleAMathews

Summary

In CI on #48 (head 717d7cc), an oracle-gaps run exited 3 with an internal error. It was the stability rerun of the e2e unit fixed seed 20261105: every outcome class, on the edited source. Law 12 says that no legal history reaches an internal error, and a slow runner is a legal condition, so this is a defect. A rerun of the same job passed.

  • Run 43346685892, job 131627616481 ("Gap report end-to-end oracle (fixed seed, 3/3)"). The job took 14m0s, and the unit took 517 s; the same unit took 142 s in the rerun and 134 s on main.
  • The violation reported: oracle-gaps.report violated at checkpoint "stability" (law: completion), the CLI did not complete on the edited source (exit 3).
  • The CLI's stdout before the crash (as the e2e driver kept it):
Difference: the CLI did not complete on the edited source (exit 3): ests/a.test.ts for 2s
busy: sut.ts:M170 in tests/a.test.ts for 2s
busy: sut.ts:M174 in tests/a.test.ts for 2s
not-infected L197 sut.ts:M161 ` > ` → ` >= `
branch-never-taken L200 sut.ts:M163 ` x > 4 ` → ` true `
timeout L200 sut.ts:M164 ` x > 4 ` → ` false `
busy: sut.ts:M174 in tests/a.test.ts for 4s
busy: sut.ts:M168 in tests/a.test.ts for 2s
busy: sut.ts:M177 in tests/a.test.ts for 2s
busy: sut.ts:M172 in tests/a.test.ts for 2s
busy: sut.ts:M175 in tests/a.test.ts for 2s
not-infected L200 sut.ts:M165 ` > ` → ` >= `
timeout L200 sut.ts:M166 ` out.push('s54'); ` → ` ; `
branch-never-taken L203 sut.ts:M167 ` x > 4 ` → ` true `
busy: sut.ts:M172 in tests/a.test.ts for 4s
not-infected L206 sut.ts:M173 ` > ` → ` >= `

The internal error's stack is in the fixture's out/error.log. The e2e driver deleted that file with its scratch directory, and CI uploads no artifact, so the stack is lost.

Reproduction attempts (all exited 0)

On a kept fixture of the same constructed history, run locally under the e2e lock:

  • With tight limits, --first-limit/--hang-limit/--mutant-limit of 100/1000/1500 ms, 200/2000/3000 ms, and 50/600/900 ms.
  • With the e2e's own limits (1000/20000/15000 ms, --contexts 4), three rounds with 14 busy node loops and three rounds with 54 busy loops on an 18-core machine. These runs printed the same timeout lines that CI printed before the crash (M164, M166, M172, M174, M210). Those mutants are the hang witnesses, which are timeouts by design.

Hypotheses, none confirmed

  1. A runner path that throws an invariant error under heavy load, for example runOwners's "has no unit and no earlier kill", or a batch whose run reports an error outside any test while copies are cut (see the e2e limits, "a batch whose run reports an error outside any test reruns each copy alone").
  2. fix(oracle-gaps): report a shortcut for speed as extra-work, and let a work bound kill it #48's new wrappers in the capture (capture-setup.mjs): the clock and timer wrappers, the Date proxy, and the stack read on clock reads. They run in every copy, and a failure there under load could surface as an internal error. This is a real candidate, and fix(oracle-gaps): report a shortcut for speed as extra-work, and let a work bound kill it #48 has not ruled it out.
  3. Something in the environment (descriptors, disk, memory) on the runner.

Diagnostic in place

From #48 onward, when a run exits 3, the e2e driver copies the start of error.log into the observed stderr. The next "did not complete (exit 3)" difference will carry the stack.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions