Repository navigation
fix(oracle-gaps): deterministic stop sessions and stdout lead, a constructed failing owner, and a copy's random stream - #54
Merged
Conversation
…tity, not pid The stop law reads only whether a later run kept an earlier run's session directory, but the driver observed the directories by name, `.oracle-gaps-<pid>`, which differs on every run, so the determinism test of "every outcome class" failed. Each directory is now the nth distinct one of its history (`session#n`). RED: determinism: constructed history: every outcome class, "two runs recorded different traces at observation 1 (stopped)". GREEN after. Mutant (the CLI keeps stale session directories): "the run after a stopped one left its session directory session#0 in the package". Co-authored-by: Isaac <no-reply@databricks.com>
…w it from the base The constructed history "a failing owner under a config that --config names" owes the state "blocked by brokenOwner", which the law owes only when the failing owner, file b, owns a mutant that no passing owner kills (law 14; maintainer disposition G5, 2026-11-11). The construction owned only file a's function and relied on its generated base for something b owns, so a base with nothing b owns missed the state. It now adds a second function that only b owns. RED: ORACLE_SEED=520648412 ORACLE_PATH=0 replay of "random seed: a failing owner under a config that --config names": "did not enter blocked by brokenOwner". GREEN: the same replay and the fixed-seed unit pass. Co-authored-by: Isaac <no-reply@databricks.com>
…iginal drew The capture seeds the ambient random source per test, so a test's draws repeat in every run. The witness w_draws observes a Math.random draw beside a cache write that nothing observes: its if-false mutant changes only the cache, so it is owed not-observed. RED: "src/sut.ts:w_draws:74:if-false#0: not-judged, owed not-observed". A mutant runs in a copy of its test file, named for its batch (.oracle-gaps-<pid>-<n>-<file>), and the capture seeded the source by the copy's path, while the unmutated runs run the original file: every copy draws another stream than the baseline. The klona run without --seed showed the same thing through oracleProperty's drawn campaign seed, so a random unit's histories changed under each mutant. Co-authored-by: Isaac <no-reply@databricks.com>
The capture seeds Math.random, crypto.getRandomValues, and crypto.randomUUID per test, so a test sees the same values in every run. It seeded them by the test's key, whose file is the copy's path in a mutant batch (.oracle-gaps-<pid>-<n>-<file>), while the unmutated runs run the original file: every mutant drew another stream. A test that observes a draw then reads as changed under a mutant that changes nothing (not-judged where not-observed is owed), and an oracleProperty campaign with no --seed draws its seed from Math.random, so its random units ran other histories under every mutant (the klona run without --seed: 10 false not-judged gaps). The batch now names each copy's original file, and the copy seeds by the key in that file. GREEN: e2e "every outcome class" passes, w_draws not-observed. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
🦋 Changeset detectedLatest commit: ba86902 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
 |
commit: |
The witness module keeps both sides: w_draws beside #48's work-law witnesses (w_fast*, w_fastclocked, w_fastturn, w_fastplain, and w_cold, still last in file a), and the test record's draws constant beside work and timed. Co-authored-by: Isaac <no-reply@databricks.com>
Since #46 the stream has no blank line, so the observed stdout lead is the whole stdout, and its notes name the run's scratch directory (.scratch-<random>), which differs between two runs of one history. The determinism test of "every outcome class" failed at observation 0 ("cli") on main after #46. Each lead line now names the scratch directory as the rest of the trace does. RED (on the merge of main): "two runs of constructed history: every outcome class recorded different traces at observation 0 (cli)", the first difference in the results note's .scratch-<id> path. GREEN after. Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews
changed the title
…t ordered names Review of #54: session#n numbered the directories in readdir order, which sorts by pid, so with two directories present the numbering depended on the pids (a replica: the kept one was not session#0 in 518 of 1000 trials). The observation now holds counts, which no name or order changes: the session directories present once the run ended, and how many of them the previous stopped run had left. The law is as strict: any kept directory is a violation. GREEN: "fixed seed 20261105: every outcome class" and its determinism unit. Mutant (the CLI keeps stale session directories): "the run after a stopped one left 1 of its session directories in the package". The stdout lead holds no timing line: the progress lines with seconds (unmutated, mutant phase) go to stderr through log, and the stdout summary (report.mjs) has none. Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews
added a commit
that referenced
this pull request
Nov 13, 2026
The fs recorder's wrappers reuse #48's keepProperties. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes three defects that the e2e gap oracle and the klona validation found while #48 was being written.
(a) Determinism of the stop law's observation. The
stoppedobservation listed session directories by name,.oracle-gaps-<pid>, and the pid differs on every run. As a result,determinism: constructed history: every outcome classfailed on main. The stop law reads only whether a later run kept an earlier run's directory, so the observation now holds counts, which no name or order changes: the session directories present once the run ended, and how many of them the previous stopped run had left. (A first version numbered themsession#nin readdir order, which sorts by pid, so the numbering of two directories still depended on the pids; the review caught it.) The stdout lead holds no timing line: the progress lines with seconds go to stderr, and the stdout summary has none.(b) A construction defect in the oracle. The constructed history "a failing owner under a config that --config names" owes the state
blocked by brokenOwner. Law 14 (maintainer disposition G5, 2026-11-11) owes that state only when the failing owner, file b, owns a mutant that no passing owner kills. The construction owned only file a's function and relied on its generated base for something that b owns. Seed 520648412 drew a base with nothing that b owns. The history now constructs that ownership: a second function that only b owns. This is a fix to the oracle, not to production.(c) A mutant's copy drew from another random stream than its original. The capture seeds
Math.random,crypto.getRandomValues, andcrypto.randomUUIDper test, by the test's key. In a mutant batch, the test file is a copy named for its batch (.oracle-gaps-<pid>-<n>-<file>), so the key held the copy's path. The unmutated runs run the original file, so every mutant copy drew a different stream than the baseline. The effects:oraclePropertycampaign with no--seeddraws its seed fromMath.random, so its random units ran other histories under every mutant.This explains the klona run without
--seedreported in fix(oracle-gaps): report a shortcut for speed as extra-work, and let a work bound kill it #48: there, all 10 speed shortcuts came out asnot-judged. The batch now names each copy's original file, and the copy seeds its streams by the test's key in that file. Pinning one seed per check, the fix proposed first, is not needed.(d) Determinism of the stdout lead (found while merging main). Since feat(oracle-gaps): every streamed line starts with
oracle-gaps: <kind>#46 the stream has no blank line, so the observed stdout lead is the whole stdout, and its notes name the run's scratch directory (.scratch-<random>). The same determinism test failed at observation 0 (cli). Each lead line now names the scratch directory as the rest of the trace does.Evidence
a run after a stopped one that left its session,blocked by brokenOwner, and the newa draw from Math.random repeats in a mutant's copy. The last is reached by the witnessw_draws, which observes aMath.randomdraw beside a cache write that nothing observes.-t "determinism: constructed history: every outcome class": "two runs ... recorded different traces at observation 1 (stopped)".ORACLE_SEED=520648412 ORACLE_PATH=0 ORACLE_PROPERTY='oracle-gaps.report' ORACLE_UNIT='random seed: a failing owner under a config that --config names': "did not enter blocked by brokenOwner".-t "fixed seed 20261105: every outcome class": "src/sut.ts:w_draws:74:if-false#0: not-judged, owed not-observed".oracle-gaps: <kind>#46, fix(oracle-gaps): report a shortcut for speed as extra-work, and let a work bound kill it #48),-t "determinism: constructed history: every outcome class": "two runs ... recorded different traces at observation 0 (cli)", first differing at the results note's.scratch-<id>path.oracle-gaps: <kind>#46 and fix(oracle-gaps): report a shortcut for speed as extra-work, and let a work bound kill it #48 too:fixed seed 20261105: every outcome class,determinism: constructed history: every outcome class,fixed seed 20261105: a failing owner under a config that --config names, and the seed 520648412 replay.-t "fixed seed 20261105: ", 446 s locally).tsc: clean. ESLint: clean on the changed files.pr-shards checkande2e-shards checkpass, run through an exec-onlypnpmwrapper with no install.--seed, the r16 copy gives 218 mutants: 208 killed, 8not-observed, and 2not-distinguished. Those are the classes of a seeded run, with nonot-judged. Before the fix, the same run gave 10not-judged.crypto.randomBytes, a native addon) is not seeded by the capture.setup-errorin a full run but were killed under--only. That is not this mechanism: the seed defect affected every copy, so--onlywould show it too. The hypothesis is builtin-prototype state carried between copies in one worker, because the reset removes only new global names. A deterministic witness needs control over batch composition, which the runner does not give today.Size
capture-setup.mjs+11 lines andcheck.mjs+3 lines (source; the CLI ships as source).This pull request and its description were written by Isaac.