Skip to content

fix(oracle-gaps): a file a test reads through fs is an input of its daemon run - #55

Open
KyleAMathews wants to merge 16 commits into
mainfrom
test/gap-daemon-midrun-edit
Open

KyleAMathews wants to merge 16 commits into
mainfrom
test/gap-daemon-midrun-edit

Conversation

@KyleAMathews

@KyleAMathews KyleAMathews commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A check through --daemon reused evidence after an edit to a file that a test reads through fs. A cold run sees the edit. This breaks law 9 of the end-to-end gap oracle ("an incremental run through the daemon equals a cold one", v19.3). It affects any package whose tests read fixtures, snapshots, or data files from disk. I found it when I debugged the mid-run edit witness: the daemon reused file a's run, so the edit never landed.

  • The fix: the capture setup (capture-setup.mjs) records the files in the package that each copy of a test file reads through fs. It records each file's hash at the copy's first read. The closing row carries the paths in loaded and their hashes in readAt. An edit to a recorded file now reruns the tests that read it.

    • Recorded reads: readFileSync, readFile (callback), existsSync, and statSync on node:fs, and readFile and stat on node:fs/promises.
    • The capture setup wraps these functions once per worker. Each wrapper keeps the function's own properties (fix(oracle-gaps): report a shortcut for speed as extra-work, and let a work bound kill it #48's keepProperties). Then syncBuiltinESMExports() makes an ES module's named imports read the wrappers.
    • The recorder skips reads outside the package, under node_modules, and of the runner's own .oracle-gaps files.
  • Read-time hashes (re-review F1): before, the runner used each input's hash at the end of the run. A test could read a file and then write it. A module could change after the run loaded it. In both cases, the runner stored the run under the new text with the old verdict. The daemon's next check then served that verdict. Now the runner stores each input under the hash of the text that the run read:

    • a file read through fs: its hash at the copy's first read (readAt)
    • a module: its hash when the run's context transformed it. A transform hook in the wrapper config asks the runner to record that hash. The runner uses its one definition for every input (servedHash, third review F-A).
    • any other input: its hash at the start of the run, or at the end

    When a path's text changed after the run read it, the runner invalidates the path and does not store the run. One rule covers the fs case and the module case. The module case is also on main (its settleIn is the same), and this PR fixes it too.

  • Reuse under a config's own pre plugin (third review, F-A): the first version of the transform hook hashed the code that it received. A config plugin with enforce: 'pre' had already rewritten that code, so the hash never matched the runner's servedHash. An unchanged check through the daemon then recomputed runs (6 of them in the witness), where main recomputes none. The hook now records servedHash.

  • A run that spun (third review, F-D): a run past its call budget still ends. So it now keeps the inputs that its closing row names. A timed-out unit is stored under the union of its file's known inputs and its own.

  • A directory, or a file removed during its hash (review items, F4): a directory that a test stats hashes as not a file. A file that disappears between the hash's stat and its read hashes as missing. Before, each case threw an internal error (EISDIR or ENOENT).

  • The mid-run edit witness (law 9): the edit midrun lands while a check through the daemon runs. The next check, with nothing changed, must equal a cold run of the state after the edit. The edit lands in one of three ways:

    • A test of file a rewrites file b's helper after its context loaded the helper. File b's runs read it later.
    • A test of file b reads its own helper and fails unless it is strict. Then it rewrites the helper.
    • A test of file b reads the data file through fs and fails unless it is open. Then it rewrites the data file.
  • The check that a mid-run edit overlaps (third review, F-C): before, the oracle threw this check away, so any output passed, even a crash. Now the check must exit as a cold run of the state before or after the edit does. It must report the same mutants and stream as any check does. Each mutant must come out as one of the two cold runs reports it. This costs one extra cold run of the state before the edit.

    • When file b's own test lands the edit, b's baseline reads the old text and b's mutant runs read the new one. So b's mutants can match neither cold run. They must still come out as a run of the edited text can: killed, survived, or blocked. They can also come out crashed, but only by the editing test's own assertion, with its name and message. No other crash or gap outcome is allowed (coordinator decision, 2026-11-13).
    • The history where file a lands the edit keeps the full comparison.
  • Docs: the README says which reads are inputs of a daemon check. It names each fs API that is not an input, and it says to run cold after an edit to such a file. A patch changeset says the same.

Evidence

  • Law: law 9 of the end-to-end gap oracle, "incremental equals cold". A check through the daemon reports, at every state of a sequence of edits, what a cold run of that state reports. A file that a test reads through fs is an input of its run, as a module that the test imports is. The runner stores a run under the text that the run read. Law 3 also applies: a test that passes alone passes under the CLI, so the wrapped functions keep their name and length. Authority: the maintainer's decision in design run v19.3, and the coordinator's decision of 2026-11-12 that the fs gap breaks law 9.
    • A second law, unchanged reuse: a check through the daemon of an unchanged state computes no run. The law applies where the cold run cut nothing off and no test reads the clock. Authority: the README's reuse contract (a run is reused while its inputs are unchanged).
    • The marker check is a witness guard, not a law-9 requirement. It is a precondition for the helper history's sensitivity. The run that lands the edit also rewrites the marker. So a run stored under the marker would not be stored at all. Then the marker would hide whether the runner stores the run under the helper's text that it read.
  • Reach:
    • The dimension fsRead makes file b's verdicts also wait on tests/data.json. File b reads it with readFileSync, fs/promises readFile, the callback readFile, or readFileSync after a statSync and an existsSync of its own directory (statDir). Each read first checks that its function keeps its name and length. The edit fsData changes only that file.
    • Four constructed histories, "an edit to data a test reads through fs ()", enter an edit to data read through <kind> changed the cold result.
    • The dimensions midrunBy and midrunTarget say which file lands a mid-run edit and what it edits. Three constructed histories enter an edit landed during a check through the daemon. Two of them also enter a run edited the helper it had read, … and a run edited the data it had read, ….
    • The dimension prePlugin gives the package's config a plugin of its own with enforce: 'pre', which appends a comment to each module that it transforms. The constructed history "an unchanged check through the daemon under a config's pre plugin" enters an unchanged check through the daemon computed nothing under a config's pre plugin.
    • Generated histories draw fsRead, midrunBy, midrunTarget, and prePlugin, and take the edits fsData, midrun, and recheck among their steps. The new grammar fields come last, so each earlier field draws as before.
  • RED:
    • The fs histories, on main (cc7bf09, without the fix): each fails law incremental equals cold, "after fsData, the daemon's report differs from the cold run's at src/sut.ts#f0:delete-call".
    • statDir and the property guard, on 2dcb803: the statDir history fails its baseline. The old wrapper drops the name and length of statSync and existsSync. With only the property copy, the sync history passes, and statDir ends in oracle-gaps internal error: EISDIR.
    • F1, on ef40cbb: both self-edit histories fail law incremental equals cold, "after midrun, the daemon's report differs from the cold run's". The daemon passes the editing test, and the cold run fails it. The cold run's errors are "the helper this run read is not strict" and "the data this run read is closed".
    • F-A, on 82b5c17: "an unchanged check through the daemon under a config's pre plugin" fails law unchanged reuse, "computed 6 runs, owed none".
    • F-C, on 82b5c17 before the narrowing: both self-edit histories fail only on file b's own mutant. The editing test crashed it, with "the helper this run read is not strict" or "the data this run read is closed". The history where file a lands the edit passes the full comparison.
  • GREEN: at 1cfab2b, the targeted fixed-seed units pass, 10 of 10. They are every outcome class, the edit sequence, the three mid-run histories, the unchanged check under the pre plugin, and the four fs histories. The four fast gap suites pass (33,891 tests). Earlier, at 82b5c17:
    • The targeted fixed-seed units pass, 9 of 9. They are every outcome class, the edit sequence through the daemon, the mid-run history, both self-edit histories, and the four fs histories. After the marker moved, the mid-run, self-edit, sync, and statDir histories pass again, 5 of 5.
    • The four fast gap suites pass at 82b5c17 (33,891 tests).
    • Prettier, e2e-shards.mjs check, and pr-shards.mjs check are clean.
  • Mutant: I put each hostile mutant into this branch, and the oracle rejected each one:
    • The fix without the promise reads: the promise history fails law incremental equals cold.
    • The fix that records the paths but not their content: the sync history fails law incremental equals cold.
    • fileHashes without its not a file case: the statDir history ends in the EISDIR internal error (law greppable stream).
    • The recorder without keepProperties: the sync history's baseline fails ("1 of 2 oracle files failed").
    • settleIn with the end-of-run hash (started ?? now): both self-edit histories fail law incremental equals cold.
    • settleIn without readAt: the data history fails, and the helper history passes.
    • settleIn without the module transform hash: the helper history fails, and the data history passes.
    • A runner that records the mid-run marker: the witness guard rejects it in the helper history. The difference is "the daemon stored a run under the mid-run marker, a file in its own session directory". This mutant removes the ownHarness filter and the recorder's .oracle-gaps exclusion.
    • The transform hook that hashes the transformed code again (F-A): the unchanged check fails law unchanged reuse, "computed 6 runs, owed none".
    • settleIn that throws when an input changed during the run (F-C): all three mid-run histories fail on the check that the edit overlapped. That check ends in an internal error, which law greppable stream rejects. The old oracle threw this check away.
    • F-D has no mutant that the oracle rejects. No history makes a timed-out run read a file that its file's unmutated run does not read, so a runner without the union passes. The limits record this.
  • The mid-run marker: the test that lands a mid-run edit reads its marker from the daemon's session directory. That directory is no input of a run, by design. Before, the marker was in the package, and the test read it through readFileSync. That read alone kept the run from being stored, and it masked the module case: the module-hash mutant survived. The oracle now checks that no stored run has the marker among its inputs.
  • CI's leftover session directory: CI on ef40cbb left .oracle-gaps-<pid>/ in "every outcome class". CI on 82b5c17 passed every job, so the failure is gone. The likely cause is the EISDIR above. On Linux, a recorded directory made the run end in an internal error, which exits before the session closes. CI shows that the failure is gone. It does not prove the cause.
  • Limits:
    • A read outside the package or under node_modules is no input. A read from a test's own worker thread or child process is no input either.
    • A read through any other fs API is no input yet. These APIs include readdir, opendir, createReadStream, realpath, lstat, access, and open. They also include a read through a file descriptor (read, readSync, readv), copyFile, copyFileSync, cp, cpSync, openAsBlob, and statfs. The rest are readlink, the callback stat, exists, watch, watchFile, and the fs/promises access and open.
    • The maintainer decided on 2026-11-13 (F2) that every fs read except watch (and watchFile) is an input. A follow-up PR from main implements that after this one merges, so this PR does not grow again. Until then, the oracle's limits record the gap with its owner, and the README says to run cold after such an edit.
    • The runner's hash memo knows a file by its size and modification time. So it does not see an edit that keeps both (F3, D10, planned in phase 2, A5). No history makes one.
    • An edit lands during a check only through the run's own tests, in the first run of the editing file. No history lands one during a mutant run, or from another process at an arbitrary time.
    • A module's transform hash is the text that its context last transformed. A module that changes and transforms again during one run keeps the later hash. No history generates one.
    • The mid-run comparison is per mutant, and it excludes the editing file's mutants when file b lands the edit, for the reason above. The stronger follow-up compares each run's raw outcome with the cold outcome of the state that the run read. It also makes the editing test take a law path instead of crashing.
    • The unchanged-reuse law applies only where the cold run cut nothing off and no test reads the clock.
    • A run cut off by its clock writes no closing row, so the runner stores it under its file's known inputs only (F-D). No history makes such a run read a file that its file's unmutated run does not read, so this limit stands by inspection.
    • An internal error still leaves the session directory, and the next run removes it. No legal history reaches an internal error (law 12).
    • The recorder also records Vitest's own reads of package files through fs in a worker (for example, a snapshot file). They add inputs, which can only make the daemon rerun more.

Code weight

Measured as minified and gzipped bytes against main:

  • capture-setup.mjs: 5,688 to 6,018 (+330), for the fs recorder with its read-time hashes
  • check.mjs: 13,472 to 13,755 (+283). This covers the module transform hash, the read-time hashes in settleIn, two new cases in fileHashes, and a spun run's inputs.

I folded the wrappers into one loop over an owner list (F8). That loop is 9 bytes smaller minified and the same in gzip. The read-time hash serves the fs case and the module case through one rule in settleIn, not two recovery paths.

This pull request and its description were written by Isaac.

KyleAMathews and others added 7 commits November 12, 2026 12:28
A new edit, midrun: a test of file a, whose context loaded file b's
helper, rewrites the helper in the baseline's first run of file a, before
file b's runs read it. That check is not compared; the next check through
the daemon must equal a cold run of the state after the edit. Constructed
history "an edit during a check through the daemon" enters "an edit
landed during a check through the daemon" and "edit midrun changed the
cold result". GREEN on main and the RED mutant are still to come.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Through the daemon, file a's text had not changed since the previous
check, so the daemon reused its baseline and its edit test never ran: the
marker is read through fs, which no run records as an input. Each mid-run
edit now also changes file a (a count of them), so the edit lands. That
fs reads are no input is law 9's own gap; the next commits judge it.

Co-authored-by: Isaac <no-reply@databricks.com>
A file that a test reads through fs is no input of its run, so a check
through the daemon reused evidence after an edit of only that file, while
a cold run sees the edit (found while the mid-run witness was debugged).
A new dimension, fsRead, makes file b's verdicts also wait on
tests/data.json, read by readFileSync, fs/promises readFile, or the
callback readFile; the new edit fsData changes only that file. Three
constructed histories, one per kind of read, enter "edit fsData changed
the cold result" and "an edit to data read through <kind> changed the
cold result".

Co-authored-by: Isaac <no-reply@databricks.com>
Law 9: after an edit of only a data file that a test reads through fs,
the daemon reused the run that read the old data, while a cold run sees
the edit. The capture setup now records each copy's reads of files in the
package (readFileSync, readFile, existsSync, and statSync, and the
fs/promises readFile and stat; under node_modules and the runner's own
files left out), and lists them with the modules the copy evaluated, so
the run is stored under their content hashes. syncBuiltinESMExports makes
an ES module's imports of node:fs and node:fs/promises read the recorded
functions.

RED on main (the three fs histories): "after fsData, the daemon's report
differs from the cold run's at src/sut.ts#f0:delete-call". GREEN: the
three fs histories and the mid-run history pass.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

# Conflicts:
#	packages/oracles/oracle-gaps/capture-setup.mjs
#	packages/oracles/oracle-gaps/gap-report-model-oracle.mjs
#	packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs
@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 2cbd1834-deac-4e04-91c7-3af44faca5b9
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

KyleAMathews and others added 3 commits November 13, 2026 07:26
… their properties, owned law 9 states

From the review of #55:
- A `statDir` read stats the test's own directory and tests that it
  exists before it reads data.json, through a constructed history. The
  recorder records the directory, and law 9 owes no crash.
- Each generated data read first checks that its fs function keeps its
  own name and length (law 3: a test that passes alone passes under the
  CLI).
- The mid-run and fs-read states are reached by their own constructed
  histories, not by the edit sequence, which enters every state in
  `incrementalStates` (CI: "an edit sequence through the daemon" did not
  enter them).
- Law 9's limits name reads from a test's worker threads or child
  processes, and a file that changes while a run first reads it.

RED on 2dcb803: "an edit to data a test reads through fs (statDir)" fails
its baseline (the wrapper loses statSync's name and length). With only
the property copy applied, the sync history passes and statDir ends in
"oracle-gaps internal error: EISDIR: illegal operation on a directory".

Co-authored-by: Isaac <no-reply@databricks.com>
… fs functions' own properties

`fileHashes()` read every recorded input with readFileSync, which throws
EISDIR for a directory that a test stats or tests for (a fixtures folder),
and the error escaped as an internal error that left the session
directory. A path that is not a regular file now hashes as `not a file`:
its input is that it is there. The recorder's fs wrappers copy the
original's own properties (name, length), so code that reads them sees
the same functions as without the CLI.

Co-authored-by: Isaac <no-reply@databricks.com>
The fs recorder's wrappers reuse #48's keepProperties.

Co-authored-by: Isaac <no-reply@databricks.com>
@pkg-pr-new

pkg-pr-new Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/TanStack/oracles/@tanstack/eslint-plugin-oracles@55
npm i https://pkg.pr.new/TanStack/oracles/@tanstack/oracles@55

commit: 1cfab2b

KyleAMathews and others added 3 commits November 13, 2026 08:21
…sed (law 9)

Re-review of #55, F1: the runner stored a run under each input's hash at
the end of the run, so a file or module that the run read and then edited
was stored under its edited text with the old verdict, and the daemon's
next check served it.

The mid-run edit gains who lands it (`midrunBy`) and what it edits
(`midrunTarget`, the helper module or the data file read through fs).
File b's own test now reads what it edits, fails unless it reads the
first state's text, and then lands the edit, so the run that lands it
passes and every run that reads the edited text fails. Two constructed
histories reach it: "a test edits the helper it read, during a check
through the daemon" and "a test edits the data it read, ...". The new
grammar fields come last, so the fields before them draw as before.

Law 9's limits name the fs APIs that are no input (F2, a maintainer
decision asked on 2026-11-13) and the hash memo's size-and-mtime key
(F3, D10).

RED on ef40cbb: both histories fail with "after midrun, the daemon's
report differs from the cold run's": the daemon passes the editor test,
and the cold run fails it ("the helper this run read is not strict",
"the data this run read is closed").

Co-authored-by: Isaac <no-reply@databricks.com>
…rectory, and is never an input

The test that lands a mid-run edit read its marker (tests/midrun.txt)
through readFileSync and rewrote it in the same run. Once a run is stored
under the text it read, that marker read alone kept the run from being
stored, so a runner without its module transform hash still passed the
helper history: the marker masked the module case.

The marker now sits in the running daemon's session directory, beside
the runner's harness, which is no input of a run by design (ownHarness).
The test finds it through the worker's first setup file; a run with no
harness or no marker edits nothing. After each mid-run step the driver
reads the daemon's evidence store, and law 9 fails if any stored run has
the marker among its inputs, so a runner that starts recording it fails
loudly.

Law 9's limits record the maintainer's F2 decision of 2026-11-13 (every
fs read except watch is an input), which a follow-up implements.

Co-authored-by: Isaac <no-reply@databricks.com>
Re-review of #55:
- F1: settleIn stored each input under its hash at the end of the run, so
  a file or module that the run read and that then changed was stored
  under the changed text with the old verdict. Now a file read through fs
  is stored under its hash at the copy's first read (the closing row's
  `readAt`), a module under the hash of its text when the run's context
  transformed it (a transform hook in the wrapper config, cleared when the
  module is invalidated), and any other input under its hash at the start
  of the run, or now. A path that changed after the run read it is
  invalidated and the run is not stored. main has the module case too;
  this fixes both with one mechanism.
- F4: a file removed between the hash's stat and its read hashes as
  `missing`, not an internal error.
- F8: the fs wrappers are one loop over an owner list (9 bytes smaller
  minified, the same gzip).
- The README names each fs API that is no input yet.

Evidence: RED on ef40cbb for both self-edit histories; GREEN for the
mid-run, self-edit, and fs histories; the mutants "end-of-run hash", "no
readAt", "no module transform hash", and "records the marker" each fail
the oracle.

Co-authored-by: Isaac <no-reply@databricks.com>
@changeset-bot

changeset-bot Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 5dd0ab4

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@tanstack/oracles Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

KyleAMathews and others added 3 commits November 13, 2026 09:42
…n check compared

Third review of #55:
- F-A: a generated config dimension, `prePlugin`, adds a config plugin of
  the package's own with `enforce: 'pre'` that appends a comment to each
  module it transforms. A new law, `unchanged reuse`: a check through the
  daemon of a state that nothing changed computes nothing, where the cold
  run cut nothing off and no test reads the clock (the README's reuse
  contract). The constructed history "an unchanged check through the
  daemon under a config's pre plugin" reaches it.
- F-C: the check that a mid-run edit overlaps is compared, not thrown
  away. It must exit as a cold run of the state before or after the edit
  does, report the same mutants, stream as any check does, and report
  each mutant as one of the two cold runs does. When file b's own test
  lands the edit, b's mutants may instead come out as a run of the edited
  text can: killed, survived, blocked, or crashed by the editing test's
  own assertion (coordinator decision, 2026-11-13). One extra cold run of
  the state before the edit.
- F-B: the marker assertion is a witness guard, a sensitivity
  precondition for law 9's helper history.
- Limits: the comparison's exclusion and the stronger follow-up, a
  timed-out run's own inputs (F-D), and the exact list of fs APIs that are
  no input (F-E).

RED on 82b5c17: the unchanged check through the daemon under the pre
plugin "computed 6 runs, owed none". Before the narrowing, both
self-edit histories failed only on file b's own mutant, crashed by the
editing test ("the helper this run read is not strict", "the data this
run read is closed"); the a-editor history passed.

Co-authored-by: Isaac <no-reply@databricks.com>
… a spun run keeps its reads

Third review of #55:
- F-A: the wrapper's transform hook hashed the code it received, which a
  config's own pre plugin had already rewritten, while servedHash hashes
  the served text, so the two never matched and an unchanged check
  through the daemon recomputed runs (6 of them in the witness). The hook
  now asks the runner to record servedHash for the module, one definition
  for both sides.
- F-D: a run that spun ends, so it now returns the inputs its closing row
  names, and a timed-out unit is stored under the union of its file's
  known inputs and its own. A run cut off by its clock writes no closing
  row (a limit, by inspection).
- F-E: the README names each fs API that is no input yet, and that a
  later release records all of them except watch and watchFile. The
  changeset also names the fix for a stale verdict after a module or file
  changed during the run that read it, which main has too.

Evidence: RED on 82b5c17 ("computed 6 runs, owed none"); GREEN for the
targeted units (10 of 10) and the fast suites (33,891); the mutants "hash
the transformed code", "settleIn throws on a changed input", and "the
runner records the marker" each fail the oracle.

Co-authored-by: Isaac <no-reply@databricks.com>
…source through fs

Fourth review of #55:
- F-A2: `unchanged reuse` no longer skips a history whose cold run cut a
  run off. It counts the computed runs that were not cut off. A new
  constructed history, "an unchanged check through the daemon while a
  test reads a source through fs", has a test of file a read the source
  under test through readFileSync and as a `?raw` import, under the
  config's pre plugin. This also gives the pre-plugin state a second
  history.
- F-C1: the editing file's exemption in the check that a mid-run edit
  overlaps holds only when the edit landed. The observation carries
  `landed`, and a calibration ("the check a mid-run edit did not reach
  flips a mutant of the editing file") binds the gate.
- F-E and F-A1: the oracle's header prose says where law 9's edits come;
  the limits name glob, globSync, promises.glob, fstat, fstatSync, and fd
  reads, and the plugin and environment inputs that the maintainer
  decided on 2026-11-13 the daemon will track. The configText JSDoc is
  back above configText.

RED on 1cfab2b (locally, -t): the new history fails `unchanged reuse`,
"computed 2 runs that were not cut off, owed none".

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant