Repository navigation
fix(oracle-gaps): a file a test reads through fs is an input of its daemon run - #55
Open
KyleAMathews wants to merge 16 commits into
Open
KyleAMathews wants to merge 16 commits into
KyleAMathews wants to merge 16 commits into
Conversation
A new edit, midrun: a test of file a, whose context loaded file b's helper, rewrites the helper in the baseline's first run of file a, before file b's runs read it. That check is not compared; the next check through the daemon must equal a cold run of the state after the edit. Constructed history "an edit during a check through the daemon" enters "an edit landed during a check through the daemon" and "edit midrun changed the cold result". GREEN on main and the RED mutant are still to come. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Through the daemon, file a's text had not changed since the previous check, so the daemon reused its baseline and its edit test never ran: the marker is read through fs, which no run records as an input. Each mid-run edit now also changes file a (a count of them), so the edit lands. That fs reads are no input is law 9's own gap; the next commits judge it. Co-authored-by: Isaac <no-reply@databricks.com>
A file that a test reads through fs is no input of its run, so a check through the daemon reused evidence after an edit of only that file, while a cold run sees the edit (found while the mid-run witness was debugged). A new dimension, fsRead, makes file b's verdicts also wait on tests/data.json, read by readFileSync, fs/promises readFile, or the callback readFile; the new edit fsData changes only that file. Three constructed histories, one per kind of read, enter "edit fsData changed the cold result" and "an edit to data read through <kind> changed the cold result". Co-authored-by: Isaac <no-reply@databricks.com>
Law 9: after an edit of only a data file that a test reads through fs, the daemon reused the run that read the old data, while a cold run sees the edit. The capture setup now records each copy's reads of files in the package (readFileSync, readFile, existsSync, and statSync, and the fs/promises readFile and stat; under node_modules and the runner's own files left out), and lists them with the modules the copy evaluated, so the run is stored under their content hashes. syncBuiltinESMExports makes an ES module's imports of node:fs and node:fs/promises read the recorded functions. RED on main (the three fs histories): "after fsData, the daemon's report differs from the cold run's at src/sut.ts#f0:delete-call". GREEN: the three fs histories and the mid-run history pass. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com> # Conflicts: # packages/oracles/oracle-gaps/capture-setup.mjs # packages/oracles/oracle-gaps/gap-report-model-oracle.mjs # packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
… their properties, owned law 9 states From the review of #55: - A `statDir` read stats the test's own directory and tests that it exists before it reads data.json, through a constructed history. The recorder records the directory, and law 9 owes no crash. - Each generated data read first checks that its fs function keeps its own name and length (law 3: a test that passes alone passes under the CLI). - The mid-run and fs-read states are reached by their own constructed histories, not by the edit sequence, which enters every state in `incrementalStates` (CI: "an edit sequence through the daemon" did not enter them). - Law 9's limits name reads from a test's worker threads or child processes, and a file that changes while a run first reads it. RED on 2dcb803: "an edit to data a test reads through fs (statDir)" fails its baseline (the wrapper loses statSync's name and length). With only the property copy applied, the sync history passes and statDir ends in "oracle-gaps internal error: EISDIR: illegal operation on a directory". Co-authored-by: Isaac <no-reply@databricks.com>
… fs functions' own properties `fileHashes()` read every recorded input with readFileSync, which throws EISDIR for a directory that a test stats or tests for (a fixtures folder), and the error escaped as an internal error that left the session directory. A path that is not a regular file now hashes as `not a file`: its input is that it is there. The recorder's fs wrappers copy the original's own properties (name, length), so code that reads them sees the same functions as without the CLI. Co-authored-by: Isaac <no-reply@databricks.com>
The fs recorder's wrappers reuse #48's keepProperties. Co-authored-by: Isaac <no-reply@databricks.com>
commit: |
…sed (law 9) Re-review of #55, F1: the runner stored a run under each input's hash at the end of the run, so a file or module that the run read and then edited was stored under its edited text with the old verdict, and the daemon's next check served it. The mid-run edit gains who lands it (`midrunBy`) and what it edits (`midrunTarget`, the helper module or the data file read through fs). File b's own test now reads what it edits, fails unless it reads the first state's text, and then lands the edit, so the run that lands it passes and every run that reads the edited text fails. Two constructed histories reach it: "a test edits the helper it read, during a check through the daemon" and "a test edits the data it read, ...". The new grammar fields come last, so the fields before them draw as before. Law 9's limits name the fs APIs that are no input (F2, a maintainer decision asked on 2026-11-13) and the hash memo's size-and-mtime key (F3, D10). RED on ef40cbb: both histories fail with "after midrun, the daemon's report differs from the cold run's": the daemon passes the editor test, and the cold run fails it ("the helper this run read is not strict", "the data this run read is closed"). Co-authored-by: Isaac <no-reply@databricks.com>
…rectory, and is never an input The test that lands a mid-run edit read its marker (tests/midrun.txt) through readFileSync and rewrote it in the same run. Once a run is stored under the text it read, that marker read alone kept the run from being stored, so a runner without its module transform hash still passed the helper history: the marker masked the module case. The marker now sits in the running daemon's session directory, beside the runner's harness, which is no input of a run by design (ownHarness). The test finds it through the worker's first setup file; a run with no harness or no marker edits nothing. After each mid-run step the driver reads the daemon's evidence store, and law 9 fails if any stored run has the marker among its inputs, so a runner that starts recording it fails loudly. Law 9's limits record the maintainer's F2 decision of 2026-11-13 (every fs read except watch is an input), which a follow-up implements. Co-authored-by: Isaac <no-reply@databricks.com>
Re-review of #55: - F1: settleIn stored each input under its hash at the end of the run, so a file or module that the run read and that then changed was stored under the changed text with the old verdict. Now a file read through fs is stored under its hash at the copy's first read (the closing row's `readAt`), a module under the hash of its text when the run's context transformed it (a transform hook in the wrapper config, cleared when the module is invalidated), and any other input under its hash at the start of the run, or now. A path that changed after the run read it is invalidated and the run is not stored. main has the module case too; this fixes both with one mechanism. - F4: a file removed between the hash's stat and its read hashes as `missing`, not an internal error. - F8: the fs wrappers are one loop over an owner list (9 bytes smaller minified, the same gzip). - The README names each fs API that is no input yet. Evidence: RED on ef40cbb for both self-edit histories; GREEN for the mid-run, self-edit, and fs histories; the mutants "end-of-run hash", "no readAt", "no module transform hash", and "records the marker" each fail the oracle. Co-authored-by: Isaac <no-reply@databricks.com>
🦋 Changeset detectedLatest commit: 5dd0ab4 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
…n check compared Third review of #55: - F-A: a generated config dimension, `prePlugin`, adds a config plugin of the package's own with `enforce: 'pre'` that appends a comment to each module it transforms. A new law, `unchanged reuse`: a check through the daemon of a state that nothing changed computes nothing, where the cold run cut nothing off and no test reads the clock (the README's reuse contract). The constructed history "an unchanged check through the daemon under a config's pre plugin" reaches it. - F-C: the check that a mid-run edit overlaps is compared, not thrown away. It must exit as a cold run of the state before or after the edit does, report the same mutants, stream as any check does, and report each mutant as one of the two cold runs does. When file b's own test lands the edit, b's mutants may instead come out as a run of the edited text can: killed, survived, blocked, or crashed by the editing test's own assertion (coordinator decision, 2026-11-13). One extra cold run of the state before the edit. - F-B: the marker assertion is a witness guard, a sensitivity precondition for law 9's helper history. - Limits: the comparison's exclusion and the stronger follow-up, a timed-out run's own inputs (F-D), and the exact list of fs APIs that are no input (F-E). RED on 82b5c17: the unchanged check through the daemon under the pre plugin "computed 6 runs, owed none". Before the narrowing, both self-edit histories failed only on file b's own mutant, crashed by the editing test ("the helper this run read is not strict", "the data this run read is closed"); the a-editor history passed. Co-authored-by: Isaac <no-reply@databricks.com>
… a spun run keeps its reads Third review of #55: - F-A: the wrapper's transform hook hashed the code it received, which a config's own pre plugin had already rewritten, while servedHash hashes the served text, so the two never matched and an unchanged check through the daemon recomputed runs (6 of them in the witness). The hook now asks the runner to record servedHash for the module, one definition for both sides. - F-D: a run that spun ends, so it now returns the inputs its closing row names, and a timed-out unit is stored under the union of its file's known inputs and its own. A run cut off by its clock writes no closing row (a limit, by inspection). - F-E: the README names each fs API that is no input yet, and that a later release records all of them except watch and watchFile. The changeset also names the fix for a stale verdict after a module or file changed during the run that read it, which main has too. Evidence: RED on 82b5c17 ("computed 6 runs, owed none"); GREEN for the targeted units (10 of 10) and the fast suites (33,891); the mutants "hash the transformed code", "settleIn throws on a changed input", and "the runner records the marker" each fail the oracle. Co-authored-by: Isaac <no-reply@databricks.com>
…source through fs Fourth review of #55: - F-A2: `unchanged reuse` no longer skips a history whose cold run cut a run off. It counts the computed runs that were not cut off. A new constructed history, "an unchanged check through the daemon while a test reads a source through fs", has a test of file a read the source under test through readFileSync and as a `?raw` import, under the config's pre plugin. This also gives the pre-plugin state a second history. - F-C1: the editing file's exemption in the check that a mid-run edit overlaps holds only when the edit landed. The observation carries `landed`, and a calibration ("the check a mid-run edit did not reach flips a mutant of the editing file") binds the gate. - F-E and F-A1: the oracle's header prose says where law 9's edits come; the limits name glob, globSync, promises.glob, fstat, fstatSync, and fd reads, and the plugin and environment inputs that the maintainer decided on 2026-11-13 the daemon will track. The configText JSDoc is back above configText. RED on 1cfab2b (locally, -t): the new history fails `unchanged reuse`, "computed 2 runs that were not cut off, owed none". Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A check through
--daemonreused evidence after an edit to a file that a test reads throughfs. A cold run sees the edit. This breaks law 9 of the end-to-end gap oracle ("an incremental run through the daemon equals a cold one", v19.3). It affects any package whose tests read fixtures, snapshots, or data files from disk. I found it when I debugged the mid-run edit witness: the daemon reused file a's run, so the edit never landed.The fix: the capture setup (
capture-setup.mjs) records the files in the package that each copy of a test file reads throughfs. It records each file's hash at the copy's first read. The closing row carries the paths inloadedand their hashes inreadAt. An edit to a recorded file now reruns the tests that read it.readFileSync,readFile(callback),existsSync, andstatSynconnode:fs, andreadFileandstatonnode:fs/promises.keepProperties). ThensyncBuiltinESMExports()makes an ES module's named imports read the wrappers.node_modules, and of the runner's own.oracle-gapsfiles.Read-time hashes (re-review F1): before, the runner used each input's hash at the end of the run. A test could read a file and then write it. A module could change after the run loaded it. In both cases, the runner stored the run under the new text with the old verdict. The daemon's next check then served that verdict. Now the runner stores each input under the hash of the text that the run read:
fs: its hash at the copy's first read (readAt)transformhook in the wrapper config asks the runner to record that hash. The runner uses its one definition for every input (servedHash, third review F-A).When a path's text changed after the run read it, the runner invalidates the path and does not store the run. One rule covers the
fscase and the module case. The module case is also onmain(itssettleInis the same), and this PR fixes it too.Reuse under a config's own pre plugin (third review, F-A): the first version of the transform hook hashed the code that it received. A config plugin with
enforce: 'pre'had already rewritten that code, so the hash never matched the runner'sservedHash. An unchanged check through the daemon then recomputed runs (6 of them in the witness), wheremainrecomputes none. The hook now recordsservedHash.A run that spun (third review, F-D): a run past its call budget still ends. So it now keeps the inputs that its closing row names. A timed-out unit is stored under the union of its file's known inputs and its own.
A directory, or a file removed during its hash (review items, F4): a directory that a test stats hashes as
not a file. A file that disappears between the hash'sstatand its read hashes asmissing. Before, each case threw an internal error (EISDIR or ENOENT).The mid-run edit witness (law 9): the edit
midrunlands while a check through the daemon runs. The next check, with nothing changed, must equal a cold run of the state after the edit. The edit lands in one of three ways:fsand fails unless it is open. Then it rewrites the data file.The check that a mid-run edit overlaps (third review, F-C): before, the oracle threw this check away, so any output passed, even a crash. Now the check must exit as a cold run of the state before or after the edit does. It must report the same mutants and stream as any check does. Each mutant must come out as one of the two cold runs reports it. This costs one extra cold run of the state before the edit.
Docs: the README says which reads are inputs of a daemon check. It names each
fsAPI that is not an input, and it says to run cold after an edit to such a file. A patch changeset says the same.Evidence
fsis an input of its run, as a module that the test imports is. The runner stores a run under the text that the run read. Law 3 also applies: a test that passes alone passes under the CLI, so the wrapped functions keep their name and length. Authority: the maintainer's decision in design run v19.3, and the coordinator's decision of 2026-11-12 that thefsgap breaks law 9.unchanged reuse: a check through the daemon of an unchanged state computes no run. The law applies where the cold run cut nothing off and no test reads the clock. Authority: the README's reuse contract (a run is reused while its inputs are unchanged).fsReadmakes file b's verdicts also wait ontests/data.json. File b reads it withreadFileSync,fs/promisesreadFile, the callbackreadFile, orreadFileSyncafter astatSyncand anexistsSyncof its own directory (statDir). Each read first checks that its function keeps its name and length. The editfsDatachanges only that file.an edit to data read through <kind> changed the cold result.midrunByandmidrunTargetsay which file lands a mid-run edit and what it edits. Three constructed histories enteran edit landed during a check through the daemon. Two of them also entera run edited the helper it had read, …anda run edited the data it had read, ….prePlugingives the package's config a plugin of its own withenforce: 'pre', which appends a comment to each module that it transforms. The constructed history "an unchanged check through the daemon under a config's pre plugin" entersan unchanged check through the daemon computed nothing under a config's pre plugin.fsRead,midrunBy,midrunTarget, andprePlugin, and take the editsfsData,midrun, andrecheckamong their steps. The new grammar fields come last, so each earlier field draws as before.fshistories, onmain(cc7bf09, without the fix): each fails lawincremental equals cold, "after fsData, the daemon's report differs from the cold run's atsrc/sut.ts#f0:delete-call".statDirand the property guard, on 2dcb803: the statDir history fails its baseline. The old wrapper drops the name and length ofstatSyncandexistsSync. With only the property copy, the sync history passes, and statDir ends inoracle-gaps internal error: EISDIR.incremental equals cold, "after midrun, the daemon's report differs from the cold run's". The daemon passes the editing test, and the cold run fails it. The cold run's errors are "the helper this run read is not strict" and "the data this run read is closed".unchanged reuse, "computed 6 runs, owed none".fshistories. The four fast gap suites pass (33,891 tests). Earlier, at 82b5c17:fshistories. After the marker moved, the mid-run, self-edit, sync, and statDir histories pass again, 5 of 5.e2e-shards.mjs check, andpr-shards.mjs checkare clean.incremental equals cold.incremental equals cold.fileHasheswithout itsnot a filecase: the statDir history ends in the EISDIR internal error (lawgreppable stream).keepProperties: the sync history's baseline fails ("1 of 2 oracle files failed").settleInwith the end-of-run hash (started ?? now): both self-edit histories fail lawincremental equals cold.settleInwithoutreadAt: the data history fails, and the helper history passes.settleInwithout the module transform hash: the helper history fails, and the data history passes.ownHarnessfilter and the recorder's.oracle-gapsexclusion.unchanged reuse, "computed 6 runs, owed none".settleInthat throws when an input changed during the run (F-C): all three mid-run histories fail on the check that the edit overlapped. That check ends in an internal error, which lawgreppable streamrejects. The old oracle threw this check away.readFileSync. That read alone kept the run from being stored, and it masked the module case: the module-hash mutant survived. The oracle now checks that no stored run has the marker among its inputs..oracle-gaps-<pid>/in "every outcome class". CI on 82b5c17 passed every job, so the failure is gone. The likely cause is the EISDIR above. On Linux, a recorded directory made the run end in an internal error, which exits before the session closes. CI shows that the failure is gone. It does not prove the cause.node_modulesis no input. A read from a test's own worker thread or child process is no input either.fsAPI is no input yet. These APIs includereaddir,opendir,createReadStream,realpath,lstat,access, andopen. They also include a read through a file descriptor (read,readSync,readv),copyFile,copyFileSync,cp,cpSync,openAsBlob, andstatfs. The rest arereadlink, the callbackstat,exists,watch,watchFile, and thefs/promisesaccessandopen.fsread exceptwatch(andwatchFile) is an input. A follow-up PR frommainimplements that after this one merges, so this PR does not grow again. Until then, the oracle's limits record the gap with its owner, and the README says to run cold after such an edit.fsin a worker (for example, a snapshot file). They add inputs, which can only make the daemon rerun more.Code weight
Measured as minified and gzipped bytes against
main:capture-setup.mjs: 5,688 to 6,018 (+330), for thefsrecorder with its read-time hashescheck.mjs: 13,472 to 13,755 (+283). This covers the module transform hash, the read-time hashes insettleIn, two new cases infileHashes, and a spun run's inputs.I folded the wrappers into one loop over an owner list (F8). That loop is 9 bytes smaller minified and the same in gzip. The read-time hash serves the
fscase and the module case through one rule insettleIn, not two recovery paths.This pull request and its description were written by Isaac.