diff --git a/.changeset/gap-cli-baseline-report.md b/.changeset/gap-cli-baseline-report.md index c778af3..f408aae 100644 --- a/.changeset/gap-cli-baseline-report.md +++ b/.changeset/gap-cli-baseline-report.md @@ -5,8 +5,8 @@ The `oracle-gaps` report leads with the baseline, and a failing oracle file no longer stops the run. - An oracle file that fails with no mutant active (a RED law, or a file that cannot load or launch) blocks only the mutants it owns. A mutant it owns that no passing owner kills is a new `blocked` gap, which names the failing file. The other mutants are judged, and the report says it is provisional. Before, the run refused with exit 2. A setup file that throws now blocks every owned mutant in the same way. -- A `blocked:` line names each failing oracle file and its failing test, and says to make it pass and check again. +- A `blocked` line names each failing oracle file and its failing test, and says to make it pass and check again. - With `--only`, the baseline runs only the owners of the selected mutants, so another file's failure no longer blocks the run. - stdout and `summary.md` start with the baseline: each test file's result with its test counts, ordinary test files included, and for a failing file its failing test, its error, the law, difference, and counterexample of a law violation, and its replay command. Ordinary test files now run once in the baseline; they run against mutants only with `--ordinary`. `results.json` gains `provisional` and `baseline`. - The legend says that a `crashed` gap is a gap, not a kill, and that its error is on the gap's Detail line in `summary.md` and in its `detail` in `results.json`. -- The last stdout line is `completed: gaps`, after `output: `. Exit codes do not change: 0 means the check completed, whatever its gaps. +- The last stdout line says that the check completed, with its gap count and its report (`oracle-gaps: done completed, gaps, report `). Exit codes do not change: 0 means the check completed, whatever its gaps. diff --git a/.changeset/gap-cli-greppable-lines.md b/.changeset/gap-cli-greppable-lines.md new file mode 100644 index 0000000..e824c18 --- /dev/null +++ b/.changeset/gap-cli-greppable-lines.md @@ -0,0 +1,18 @@ +--- +'@tanstack/oracles': minor +--- + +Every line the `oracle-gaps` CLI writes for a check now starts with `oracle-gaps: ` and its kind, on stdout and stderr alike, cold and through `--daemon`, so an agent can act on the results as they stream with `grep '^oracle-gaps: '` and filter one kind by the word after the prefix: + +- `oracle-gaps: config `, or `oracle-gaps: config none, Vitest's default discovery` +- `oracle-gaps: baseline pass : tests` and `oracle-gaps: baseline fail : …`, with the failing test, law, counterexample, and replay indented under it +- one decisive line before any mutant runs: `oracle-gaps: baseline all oracle files pass`, or `oracle-gaps: baseline of oracle files failed (provisional)` +- `oracle-gaps: blocked …` for each failing oracle file +- `oracle-gaps: gap L `, streamed and in the stdout summary +- `oracle-gaps: progress …`, `oracle-gaps: busy in for s`, and `oracle-gaps: note …` (Vite's own messages as `note vite: …`) +- the stdout summary as lines too: `oracle-gaps: summary …` for the totals, `oracle-gaps: legend : …` for each directive, and `oracle-gaps: function …` for each function's heading over its gap lines +- a last line for every run: `oracle-gaps: done completed, gaps, report ` (with `(provisional)` after the count when an oracle file failed), `oracle-gaps: done refused[, report ]` after a refusal's `refused:` and `fix:` block, `oracle-gaps: done error, log `, or `oracle-gaps: done stopped ()` + +The done line is the last line of both stdout and stderr, once each, and its status matches the exit code. Only the lines indented under a failing baseline file and a refusal's `refused:` and `fix:` block have no prefix, and no line is blank. A test's own writes to the process's stdout and stderr go to `test-output.log` in the output directory, which a note names. The stdout summary's `output: `, `Full report:`, and `completed: gaps` lines are replaced by the done line, and the refusal's `output: ` line by `oracle-gaps: done refused, report `. Exit codes do not change. + +The wrapper config now reads the package's config as Vite does: it awaits a promise default and refuses a default that is not an object. A config with a syntax error is refused with its first error, before Vite bundles it. diff --git a/.changeset/gap-cli-never-crash.md b/.changeset/gap-cli-never-crash.md index c69a26c..55c716d 100644 --- a/.changeset/gap-cli-never-crash.md +++ b/.changeset/gap-cli-never-crash.md @@ -2,6 +2,6 @@ '@tanstack/oracles': minor --- -The `oracle-gaps` CLI no longer ends a run with a bare stack trace. A problem in the author's own files or arguments (bad arguments, a `--source` that is missing or not a file, a broken `owns` declaration, no oracle file or no owner, a Vitest config that throws when it loads, has no default export, or defines `test.projects`, a Vitest that does not load, a setup file that throws, an oracle file that fails with no mutant active, an authored mutant file that is not JSON or is incomplete) refuses the run: it exits 2 and prints `refused: ` and `fix: `. When the output directory is known (even from `--out` alone, when other arguments are bad), the refusal is also written there as `results.json` (`{ "status": "refused", "cause", "directive" }`) and `summary.md`, replacing an earlier run's ending there, and the CLI prints `output: `. Any other error is a bug in the CLI: it exits 3, says so, and writes its stack to `error.log` in the output directory (or the system's temporary directory when none is known), with `results.json` status `internal-error`. A check through `--daemon` ends the same way, and now also prints `output: ` when it completes. A daemon serves only a CLI of its own code, so a second install of the CLI starts its own daemon rather than reading another version's reply. +The `oracle-gaps` CLI no longer ends a run with a bare stack trace. A problem in the author's own files or arguments (bad arguments, a `--source` that is missing or not a file, a broken `owns` declaration, no oracle file or no owner, a Vitest config that throws when it loads, has no default export, or defines `test.projects`, a Vitest that does not load, a setup file that throws, an oracle file that fails with no mutant active, an authored mutant file that is not JSON or is incomplete) refuses the run: it exits 2 and prints `refused: ` and `fix: `. When the output directory is known (even from `--out` alone, when other arguments are bad), the refusal is also written there as `results.json` (`{ "status": "refused", "cause", "directive" }`) and `summary.md`, replacing an earlier run's ending there, and the CLI's last line names that report (`oracle-gaps: done refused, report `). Any other error is a bug in the CLI: it exits 3, says so, and writes its stack to `error.log` in the output directory (or the system's temporary directory when none is known), with `results.json` status `internal-error`. A check through `--daemon` ends the same way. A daemon serves only a CLI of its own code, so a second install of the CLI starts its own daemon rather than reading another version's reply. Exit code 1 is no longer used: scripts that treated any non-zero exit as a failure keep working, and scripts that matched exit 1 must match 2 or 3. Arguments are now checked: an unknown option, an option with no value, a `--package` or `--mutants` directory that does not exist, a `--seed` that is not an integer, a count or limit that is not a positive whole number, and a `--pool` other than `threads` or `forks` each refuse the run instead of being ignored or failing later. diff --git a/README.md b/README.md index 8a7e94c..00e14fc 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ every commit with [pkg.pr.new](https://pkg.pr.new). up leaves that directory, and the next run removes it. While one copy of a test file runs for 30 s or more, the CLI prints - `busy: in for s` every 30 s. A run of many fast + `oracle-gaps: busy in for s` every 30 s. A run of many fast copies prints none, and a check through `--daemon` prints its heartbeats only when the check ends. The runs that confirm a mutant's cut-off copies share one budget: the larger of `--mutant-limit` (30 s by default) and ten @@ -46,7 +46,7 @@ every commit with [pkg.pr.new](https://pkg.pr.new). failing file its failing test, its error, the law and counterexample of a law violation, and a `replay:` command. An oracle file that fails with no mutant active blocks only the mutants it owns: each that no passing owner - kills is a `blocked` gap, a `blocked:` line names the file and its failing + kills is a `blocked` gap, a `blocked` line names the file and its failing test, the rest are judged, and the report says it is provisional. With `--only `, the baseline runs only the owners of the selected mutants. A `crashed` gap is not a kill: production threw where no law @@ -54,10 +54,59 @@ every commit with [pkg.pr.new](https://pkg.pr.new). in its `detail` in `results.json`. Record the error in the driver, and state the law that decides that input's outcome. - Exit codes: 0 the check completed, whatever its gaps (the last stdout line - is `completed: gaps`); 2 the run was refused, with `refused:` and - `fix:` on stderr; 3 an internal error of the CLI; 130 or 143 stopped by - SIGINT or SIGTERM. + Every line the CLI writes for a check, on stdout and stderr alike, starts + with `oracle-gaps: ` and its kind, so an agent can act on the results as + they stream with `grep '^oracle-gaps: '`, and filter one kind: + + | Kind | Line | + | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | + | `config` | `oracle-gaps: config vitest.config.ts`, or `oracle-gaps: config none, Vitest's default discovery` | + | `baseline` | `oracle-gaps: baseline pass tests/a.test.ts: 3 tests`, `oracle-gaps: baseline fail tests/b.test.ts: 1 of 2 tests failed` | + | `baseline` (decisive, before any mutant) | `oracle-gaps: baseline all 4 oracle files pass`, or `oracle-gaps: baseline 1 of 4 oracle files failed (provisional)` | + | `blocked` | `oracle-gaps: blocked fails with no mutant active at , so the mutants it owns wait for it: …` | + | `gap` | `oracle-gaps: gap L ` | + | `progress` | `oracle-gaps: progress judged 25 of 120 mutants: 7 gaps so far, 18 not gaps` | + | `busy` | `oracle-gaps: busy in for s` | + | `note` | `oracle-gaps: note …`: the `--hang-limit` note, Vite's own messages (`note vite: …`), ordinary files left out, the provisional count, unowned counts, the results' path, and where the tests' console output went (`note test output in `) | + | `summary` | `oracle-gaps: summary : 120 mutants, 98 killed, 12 gaps, 10 unowned (warnings). …` | + | `legend` | `oracle-gaps: legend : `, once for each class of gap the run found | + | `function` | `oracle-gaps: function fn () (owners: )`, the heading over that function's gap lines | + | `done` (last) | `oracle-gaps: done completed, 12 gaps, report /summary.md` (`(provisional)` after the count when an oracle file failed), `oracle-gaps: done refused[, report ]`, `oracle-gaps: done error, log `, or `oracle-gaps: done stopped (SIGINT)` | + + Only two kinds of line have no prefix: the lines indented under a failing + baseline file (`failing:`, `law:`, `counterexample:`, `replay:`), and a + refusal's `refused:` and `fix:` block on stderr. No line is blank. The + done line is the last line of both stdout and stderr, once each, for + every ending, and its status matches the exit code. A test's own output + to the process's stdout and stderr goes to `test-output.log` in the output + directory, which a note names; `summary.md` holds the report for reading. + A run through the daemon prints the same lines, at the end of its check. + `--help` is not a check, so its usage text has no prefix. + + For contributors: two environment variables in the `ORACLE_GAPS_TEST_` + namespace are test-only hooks for the CLI's own oracle, not settings for + users. `ORACLE_GAPS_TEST_HOLD_EXIT_MS` holds the run's exit after its done + line, and `ORACLE_GAPS_TEST_WARM_PROBE` makes the first config warm-up + wait where a stop signal can reach it. + + A few endings print no done line, or one more line: a SIGKILL of the CLI, + or a signal that comes while Node is still starting the CLI, before its + first line runs (a few tens of milliseconds), ends it with no done line. + When the run's own process ends without its ending, by a signal sent to + it rather than the CLI or by a crash (a V8 out-of-memory abort is a + SIGABRT), the CLI writes `oracle-gaps: done error, the run's process +exited with ` and exits 3; after an out-of-memory crash + the line says to give Node more memory (`NODE_OPTIONS=--max-old-space-size=`) + or to run fewer `--contexts`. An internal error's own message lines + (`oracle-gaps internal error: …`) have no prefix. A config's syntax error + is refused with its first error; a syntax error in a module the config + imports still lets esbuild print its own error block, which has no + prefix. + + Exit codes: 0 the check completed, whatever its gaps (its last line is + `oracle-gaps: done completed, …`); 2 the run was refused, with `refused:` + and `fix:` on stderr; 3 an internal error of the CLI; 130 or 143 stopped + by SIGINT or SIGTERM. The check runs in a process group of its own, with every Vitest worker and every process that a test starts, except one started detached in a session diff --git a/packages/oracles/oracle-gaps/args.mjs b/packages/oracles/oracle-gaps/args.mjs index 07a53ea..bd4b15b 100644 --- a/packages/oracles/oracle-gaps/args.mjs +++ b/packages/oracles/oracle-gaps/args.mjs @@ -52,10 +52,19 @@ The paths in --source and --config are relative to --package. Without --config, the run loads the Vitest config that Vitest discovers for the package: in its directory, or else in the nearest parent directory that has one. Run the CLI from any directory. +Every line the CLI writes for a check starts with \`oracle-gaps: \` and its +kind (config, note, baseline, blocked, progress, gap, busy, done), on +stdout and stderr alike, so \`grep '^oracle-gaps: '\` finds them. Before any +mutant runs, one line says whether the baseline passed: +\`oracle-gaps: baseline all oracle files pass\`, or +\`oracle-gaps: baseline of oracle files failed (provisional)\`. +The last line is \`oracle-gaps: done \`. Exit codes: 0 the check completed, whatever its gaps: its last line is -\`completed: gaps\`; 2 refused, with the fix on stderr; -3 an internal error of oracle-gaps; 130 or 143 stopped by SIGINT or -SIGTERM, once every process the run started has ended.` +\`oracle-gaps: done completed, gaps, report \`; 2 refused, with +the fix on stderr and \`oracle-gaps: done refused\` last; 3 an internal +error of oracle-gaps (\`oracle-gaps: done error, log \`); 130 or 143 +stopped by SIGINT or SIGTERM (\`oracle-gaps: done stopped ()\`), +once every process the run started has ended.` const help = `as \`oracle-gaps --help\` shows` /** * The options in `argv`. Paths are resolved against `cwd`, the caller's diff --git a/packages/oracles/oracle-gaps/capture-setup.mjs b/packages/oracles/oracle-gaps/capture-setup.mjs index 81a4388..266835d 100644 --- a/packages/oracles/oracle-gaps/capture-setup.mjs +++ b/packages/oracles/oracle-gaps/capture-setup.mjs @@ -25,6 +25,7 @@ // of shrinking it: a kill needs one failing history, not the smallest one. import { createHash } from 'node:crypto' import { appendFileSync, readFileSync } from 'node:fs' +import { dirname, join } from 'node:path' import { afterAll, beforeEach, chai, expect, inject, vi } from 'vitest' const run = @@ -151,6 +152,26 @@ if (run.units && !skippedCopy) { vi.resetModules() } const state = unit ? { ...run, ...unit } : run +// A test's own writes to the process's stdout and stderr (a debugging +// `process.stdout.write`) go to the run's test-output.log, beside its +// capture, not to the CLI's streams, whose every line is a line of the +// gap report's stream (review of #46, F2). The check names the log. +if (state.capture) + globalThis.__oracleGapsOutput = join( + dirname(state.capture), + `test-output.log`, + ) +if (!globalThis.__oracleGapsStreams) { + globalThis.__oracleGapsStreams = true + for (const stream of [process.stdout, process.stderr]) + stream.write = (chunk, encoding, callback) => { + const done = typeof encoding === `function` ? encoding : callback + const output = globalThis.__oracleGapsOutput + if (output) appendFileSync(output, chunk) + if (typeof done === `function`) done() + return true + } +} // A batch's watchdog reads this row to know which copy is running. A copy // whose test file fails to load runs no `afterAll` and so writes no closing // row; the next copy's start ends its turn. diff --git a/packages/oracles/oracle-gaps/check.mjs b/packages/oracles/oracle-gaps/check.mjs index 3af4105..3521a5f 100644 --- a/packages/oracles/oracle-gaps/check.mjs +++ b/packages/oracles/oracle-gaps/check.mjs @@ -19,7 +19,7 @@ // the harness, the Vite config and the files it loads, the package's // manifest and the lockfile, the Node version, and the options that change a // run (the seed, the test timeout, and bail). -import { spawnSync } from 'node:child_process' +import { spawn, spawnSync } from 'node:child_process' import { createHash } from 'node:crypto' import { appendFileSync, @@ -35,7 +35,7 @@ import { writeFileSync, } from 'node:fs' import { createRequire, isBuiltin, syncBuiltinESMExports } from 'node:module' -import { hostname } from 'node:os' +import { constants, hostname } from 'node:os' import { basename, dirname, join, relative as relativePath } from 'node:path' import { fileURLToPath, pathToFileURL } from 'node:url' import { applyAuthored, loadAuthored } from './authored.mjs' @@ -51,6 +51,8 @@ import { blockedLines, gapLine, progressLine, + streamLine, + verdictLine, testNamesOf, } from './stream.mjs' import { analyze, isRunnerCopy } from './analysis.mjs' @@ -452,10 +454,24 @@ export async function openSession(pkg) { wrapper, `import { realpathSync } from 'node:fs' import { mergeConfig } from 'vitest/config' -${config ? `import base from ${JSON.stringify(relativePath(home, join(pkg, config)))}` : `const base = {}`} -// Vitest accepts a config object or a function that returns one. +${ + config + ? `import * as config from ${JSON.stringify(relativePath(home, join(pkg, config)))} +// The package's config, read from its namespace at run time: a config with +// no default export then fails as it fails under Vitest, when it loads, and +// not when Vite bundles this wrapper, which esbuild reports on stderr +// itself (a static \`config.default\` is a warning there). +const base = Reflect.get(config, 'default')` + : `const base = {}` +} +// Vitest accepts a config object, a promise of one, or a function that +// returns one, as Vite reads it: a function is called and awaited, anything +// else is awaited, and what is not an object is refused. export default async (env) => { - const merged = mergeConfig(typeof base === 'function' ? await base(env) : base, { + const loaded = typeof base === 'function' ? await base(env) : await base + if (loaded === null || typeof loaded !== 'object') + throw new Error(${JSON.stringify(`${config ?? `the config`}: config must export or return an object.`)}) + const merged = mergeConfig(loaded, { plugins: [{ name: 'oracle-gaps-schemata', enforce: 'pre', @@ -557,7 +573,78 @@ export default async (env) => { } const closeContext = (context) => context.close().finally(() => allowListeners(-1)) + // Vite's logger, writing each message as `oracle-gaps: note vite: …`. + const viteNote = (message) => { + for (const line of String(message) + .replace(/\x1b\[[0-9;]*m/g, ``) + .split(`\n`)) + if (line.trim() !== ``) pulse.log(streamLine(`note`, `vite: ${line}`)) + } + const viteNotes = { + hasWarned: false, + info: () => {}, + warn: viteNote, + warnOnce: viteNote, + error: viteNote, + clearScreen: () => {}, + hasErrorLogged: () => false, + } + // Whether the config's own text parses, by the esbuild that Vite bundles + // configs with, with its messages silenced: a syntax error rejects with + // its first error. A Vite that does not load is the context's to report. + async function configParses(config) { + // The esbuild of the Vite that the package's Vitest loads. + let esbuild + try { + const requireVite = createRequire( + createRequire(requireTarget.resolve(`vitest/node`)).resolve(`vite`), + ) + esbuild = await import(pathToFileURL(requireVite.resolve(`esbuild`)).href) + } catch { + return + } + try { + await esbuild.transform(readFileSync(join(pkg, config), `utf8`), { + loader: /\.[cm]?ts$/.test(config) ? `ts` : `js`, + logLevel: `silent`, + sourcefile: config, + }) + } catch (error) { + const first = error?.errors?.[0] + throw new Error( + first + ? `${config}:${first.location?.line}:${first.location?.column}: ${first.text}` + : String(error?.message ?? error), + ) + } + } + // A test-only hook for the CLI's own oracle, not for users + // (ORACLE_GAPS_TEST_WARM_PROBE, a marker's path): + // the first warm-up stands still as a config load can, on a service in + // the run's process group, which a stop signal ends (as it ends the + // esbuild service that bundles a config), while a handler that exits + // 200 ms after the signal holds the process (as Vitest's does). It marks + // the path when it starts, so the oracle sends its signal then. + let warmProbed = false + async function warmProbe() { + const marker = process.env.ORACLE_GAPS_TEST_WARM_PROBE + if (!marker || warmProbed) return + warmProbed = true + const exitLater = (signal) => + setTimeout(() => process.exit(128 + constants.signals[signal]), 200) + process.once(`SIGINT`, exitLater) + process.once(`SIGTERM`, exitLater) + const service = spawn( + process.execPath, + [`-e`, `setInterval(() => {}, 1000)`], + { stdio: `ignore` }, + ) + writeFileSync(marker, ``) + await new Promise((resolve) => service.on(`exit`, resolve)) + throw new Error(`The service was stopped`) + } async function warmContext(settings, index) { + await fromConfig(index, settings.config, warmProbe()) writeWrapper(index, settings.config) // A context that fails to open gives its listener allowance back: one // that never opened, at once, and one that opened, when it closes. @@ -565,38 +652,45 @@ export default async (env) => { const context = await fromConfig( index, settings.config, - createVitest(`test`, { - root: pkg, - config: wrapperFor(index, settings.config), - watch: false, - maxWorkers: 1, - // Every test in a file runs, so a kill is attributed to each route - // that produces it (see routeOf). A property still stops at its first - // failure. - bail: settings.bail, - testTimeout: settings.testTimeout, - hookTimeout: settings.testTimeout, - teardownTimeout: settings.testTimeout, - typecheck: { enabled: false }, - coverage: { enabled: false }, - // Vitest 5 caches transformed modules on disk by default, keyed by - // the file, not by what this runner's plugin serves for it, so a - // patched or instrumented source could come back as an earlier text. - // Vitest 4 caches nothing unless asked. - fsModuleCache: false, - // Every module each test file imports, which a unit records as its - // inputs. - experimental: { importDurations: { limit: 1_000_000 } }, - // A batch runs copies of test files in one worker, sharing its - // environment (runBatch), in the order given, so a hang is attributed - // to the copy that was running. - isolate: false, - // Worker threads start much faster than child processes, and each - // batch starts one (`--pool`; see args.mjs). - pool: settings.pool, - sequence: { sequencer: KeepOrder }, - reporters: [{}], - }).catch((error) => { + createVitest( + `test`, + { + root: pkg, + config: wrapperFor(index, settings.config), + watch: false, + maxWorkers: 1, + // Every test in a file runs, so a kill is attributed to each route + // that produces it (see routeOf). A property still stops at its first + // failure. + bail: settings.bail, + testTimeout: settings.testTimeout, + hookTimeout: settings.testTimeout, + teardownTimeout: settings.testTimeout, + typecheck: { enabled: false }, + coverage: { enabled: false }, + // Vitest 5 caches transformed modules on disk by default, keyed by + // the file, not by what this runner's plugin serves for it, so a + // patched or instrumented source could come back as an earlier text. + // Vitest 4 caches nothing unless asked. + fsModuleCache: false, + // Every module each test file imports, which a unit records as its + // inputs. + experimental: { importDurations: { limit: 1_000_000 } }, + // A batch runs copies of test files in one worker, sharing its + // environment (runBatch), in the order given, so a hang is attributed + // to the copy that was running. + isolate: false, + // Worker threads start much faster than child processes, and each + // batch starts one (`--pool`; see args.mjs). + pool: settings.pool, + sequence: { sequencer: KeepOrder }, + reporters: [{}], + }, + // Vite's own messages, such as a config that fails to load, as notes + // of the stream (stream.mjs), so that every line the run writes has + // the stream's prefix. + { customLogger: viteNotes }, + ).catch((error) => { allowListeners(-1) throw error }), @@ -693,7 +787,10 @@ export default async (env) => { if (Math.floor(elapsed / pulse.every) <= sent) return sent = Math.floor(elapsed / pulse.every) pulse.log( - `busy: ${label} in ${relativePath(pkg, file)} for ${Math.floor(elapsed / 1000)}s`, + streamLine( + `busy`, + `${label} in ${relativePath(pkg, file)} for ${Math.floor(elapsed / 1000)}s`, + ), ) } } @@ -1165,7 +1262,7 @@ export default async (env) => { seedVariable = `${options.envPrefix}_SEED` if (options.hangLimit > options.mutantLimit) log( - `note: --hang-limit ${options.hangLimit} ms is above --mutant-limit ${options.mutantLimit} ms; a mutant's confirmations together get at most the larger of --mutant-limit and ten times its owner files' unmutated time`, + `oracle-gaps: note --hang-limit ${options.hangLimit} ms is above --mutant-limit ${options.mutantLimit} ms; a mutant's confirmations together get at most the larger of --mutant-limit and ten times its owner files' unmutated time`, ) pulse = { log, @@ -1175,6 +1272,9 @@ export default async (env) => { // An earlier run's internal error; a completed check writes the other // files of an ending over the earlier ones (ending.mjs). rmSync(join(options.out, `error.log`), { force: true }) + // The tests' own output to the process's streams (capture-setup.mjs). + const testOutput = join(options.out, `test-output.log`) + rmSync(testOutput, { force: true }) profileFile = process.env.ORACLE_GAPS_PROFILE ? join(options.out, `profile.jsonl`) : undefined @@ -1201,6 +1301,11 @@ export default async (env) => { // Vitest discovers; undefined for none. config: options.config ?? discoveredConfig(pkg), } + // A config that does not parse is refused here, with the error, before + // Vite bundles it: esbuild, which bundles it, prints its own error on + // the CLI's streams (review of #46, F7). + if (settings.config !== undefined) + await fromConfig(0, settings.config, configParses(settings.config)) // Contexts are warmed again when their settings or the Vite config they // loaded changed. const configHash = () => @@ -1221,9 +1326,10 @@ export default async (env) => { // The run names the config its contexts loaded, before any refusal that // the analysis or the baseline makes. log( - settings.config === undefined - ? `no config file: Vitest's default discovery` - : `config: ${settings.config}`, + streamLine( + `config`, + settings.config ?? `none, Vitest's default discovery`, + ), ) for (const context of contexts) await globSpecs(context) endPhase(`find test files`) @@ -1682,6 +1788,7 @@ export default async (env) => { })) for (const line of [ ...baselineLines(baselineFiles, options.config), + verdictLine(baselineFiles), ...blockedLines(baselineFiles), ]) log(line) @@ -1698,7 +1805,12 @@ export default async (env) => { (file) => !oracleFiles.has(file), ) if (excluded.length > 0) - log(`left out, failing with no mutant active: ${excluded.join(`, `)}`) + log( + streamLine( + `note`, + `left out, failing with no mutant active: ${excluded.join(`, `)}`, + ), + ) for (const file of firstFiles) if ( first.failed.has(file) || @@ -1844,7 +1956,7 @@ export default async (env) => { !valueChanges.get(file)?.has(mutant.number), ) && targets.oracleKeys.every((key) => !nondeterministic.has(key)) log( - `unmutated: ${[...first.rows.values()].filter((r) => !r.background).length} tests in ${first.seconds.toFixed(1)}s and ${second.seconds.toFixed(1)}s; nondeterministic: ${nondeterministic.size}`, + `oracle-gaps: progress unmutated: ${[...first.rows.values()].filter((r) => !r.background).length} tests in ${first.seconds.toFixed(1)}s and ${second.seconds.toFixed(1)}s; nondeterministic: ${nondeterministic.size}`, ) // 3–5. Mutants. @@ -2174,7 +2286,10 @@ export default async (env) => { if (result.authored && gapsOf([result]).length > 0) log(gapLine(result)) const mutantSeconds = (Date.now() - started) / 1000 log( - `mutant phase: ${results.length} mutants in ${mutantSeconds.toFixed(1)}s`, + streamLine( + `progress`, + `mutant phase: ${results.length} mutants in ${mutantSeconds.toFixed(1)}s`, + ), ) await store.flush?.() writeFileSync( @@ -2217,6 +2332,8 @@ export default async (env) => { config: options.config, }), ) + if (existsSync(testOutput) && statSync(testOutput).size > 0) + log(streamLine(`note`, `test output in ${testOutput}`)) return renderStdout({ results, run: runIdentity, diff --git a/packages/oracles/oracle-gaps/ending.mjs b/packages/oracles/oracle-gaps/ending.mjs index c1cb88c..f26a94c 100644 --- a/packages/oracles/oracle-gaps/ending.mjs +++ b/packages/oracles/oracle-gaps/ending.mjs @@ -5,7 +5,8 @@ // A refusal is a problem the author can fix. It exits 2 and prints one block, // `refused: ` and `fix: `, and, when the run knows its // output directory, writes the refusal there as results.json and summary.md -// and names the directory. Any other error is a bug in the CLI. It exits 3, +// and names its report there in its last line, `oracle-gaps: done refused`. +// Any other error is a bug in the CLI. It exits 3, // says so, and keeps its stack in error.log, in the output directory when // the run knows one. Each ending replaces the files of another ending that // an earlier run left in the directory (check.mjs removes error.log when a @@ -33,9 +34,17 @@ export function endRun(error, out = error?.out) { if (error instanceof Refusal) { const block = [`refused: ${message}`, `fix: ${error.fix}`] const results = { status: `refused`, cause: message, directive: error.fix } - if (write(out, results, `Refused: ${message}\n\nFix: ${error.fix}\n`)) - block.push(`output: ${out}`) - return { exitCode: 2, stderr: `${block.join(`\n`)}\n` } + // The last line says the run refused, and where its report is once the + // run knows its output directory (the stream's done line, stream.mjs); + // it ends stdout too (index.mjs). + const done = write( + out, + results, + `Refused: ${message}\n\nFix: ${error.fix}\n`, + ) + ? `oracle-gaps: done refused, report ${join(out, `summary.md`)}` + : `oracle-gaps: done refused` + return { exitCode: 2, stderr: `${[...block, done].join(`\n`)}\n`, done } } const results = { status: `internal-error`, message } const stack = `${error?.stack ?? message}\n` @@ -44,9 +53,11 @@ export function endRun(error, out = error?.out) { log = join(tmpdir(), `oracle-gaps-error-${process.pid}.log`) writeFileSync(log, stack) } + const done = `oracle-gaps: done error, log ${log}` return { exitCode: 3, - stderr: `oracle-gaps internal error: ${message}\nThis is a bug in oracle-gaps, not in your oracle. Its stack is in ${log}.\n`, + stderr: `oracle-gaps internal error: ${message}\nThis is a bug in oracle-gaps, not in your oracle. Its stack is in ${log}.\n${done}\n`, + done, } } diff --git a/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs b/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs index 4fe0955..035bd1b 100644 --- a/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs +++ b/packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs @@ -143,14 +143,27 @@ * points, so an astral-plane character is never split), cut to its first * 57 and `...`. An intent is the author's text, so it prints as a code * span, which the exit law does not read. The results keep the full text. - * The stdout leads with the baseline (law 9's lines), then one `blocked:` + * Every line of the stdout is a line of the stream, with the stream's + * prefix and its kind (the maintainer's request of 2026-11-12, and the + * coordinator's decision on review F1): `oracle-gaps: summary` for the + * totals, `oracle-gaps: legend :` for each directive, + * `oracle-gaps: function` for each function's heading, `oracle-gaps: + * gap`, `oracle-gaps: baseline`, `oracle-gaps: blocked`, `oracle-gaps: + * note` (the provisional count, the unowned counts, the files left out, + * and the results' path), and `oracle-gaps: done`; only the lines + * indented under a failing baseline file have none, and no line is + * blank. + * The stdout leads with the baseline (law 9's lines), then the one + * decisive baseline line (all oracle files pass, or how many of them + * failed, provisional), then one `oracle-gaps: blocked` * line for each failing oracle file that names it, its failing test, and * the step that unblocks it (G13), then, when an oracle file failed, * `Provisional:` with the count of blocked mutants; summary.md leads with * the same lines, before its gaps (the freeze's one loop, 2026-11-10). * The legend line for `crashed` says first that a crash is a gap, not a * kill, and that its error is on the gap's Detail line (G3; the freeze, - * P3 and C1). The last stdout line is `completed: gaps` (G6). A replay + * P3 and C1). The last stdout line is `oracle-gaps: done completed, + * gaps[ (provisional)], report ` (G6). A replay * line is a command, which the exit law does not read. * 9. Before any mutant, each test file's baseline is printed, an ordinary * file marked `(ordinary)`: `pass` with its test count, or `fail` with how @@ -2101,25 +2114,46 @@ function check(h, observations, expected) { return violation(`classified`, `no compressed stdout summary`, `stdout`) const stdoutGaps = stdout .split(`\n`) - .filter((line) => /^ \S+ L(\d+|\?) (m\d+|authored:a\d+) /.test(line)) + .filter((line) => + /^oracle-gaps: gap \S+ L(\d+|\?) (m\d+|authored:a\d+) /.test(line), + ) const kinds = new Set(gapsOfModel.map((o) => o.class ?? o.outcome)) // Gaps are grouped by function, and the authored ones in one group. const groupOf = (o) => o.authored ? `authored mutants (src/x.ts)` : `fn ${o.function} (src/x.ts)` const functions = new Set(gapsOfModel.map(groupOf)) - // The legend's lines: those under its heading, before the blank line - // after it (the baseline above it has indented lines too). - const legendAt = stdout + // Every line of the stdout is a line of the stream: its prefix and kind, + // or an indented line under a failing baseline file (law 15 of the + // end-to-end oracle; the coordinator's decision on review F1, 2026-11-12). + // A blank line is none. + const stray = stdout + .replace(/\n$/, ``) .split(`\n`) - .indexOf(`Legend, one directive per class:`) + .find( + (line) => + !/^oracle-gaps: (?:config|note|baseline|blocked|progress|gap|busy|summary|legend|function|done) /.test( + line, + ) && !/^ \S/.test(line), + ) + if (stray !== undefined) + return violation( + `classified`, + `the stdout has a line that is not a line of the stream: ${JSON.stringify(stray)}`, + `stdout`, + { actual: stdout }, + ) + // The legend's lines, one per class: `oracle-gaps: legend : …`. const legend = stdout .split(`\n`) - .slice(legendAt + 1) - .filter((line, i, rest) => rest.slice(0, i + 1).every((l) => l !== ``)) - .filter((line) => /^ [a-z-]+: /.test(line)) + .filter((line) => /^oracle-gaps: legend [a-z-]+: /.test(line)) + // Each function's heading over its gap lines. const headings = stdout .split(`\n`) - .filter((line) => line.startsWith(`fn `) || line.startsWith(`authored `)) + .filter( + (line) => + line.startsWith(`oracle-gaps: function fn `) || + line.startsWith(`oracle-gaps: function authored `), + ) const stdoutProblem = !stdout.includes(`, ${gapsOfModel.length} gaps,`) ? `the totals do not state ${gapsOfModel.length} gaps` : !stdout.includes(`out/summary.md`) @@ -2134,7 +2168,7 @@ function check(h, observations, expected) { (o) => stdoutGaps.filter((line) => line.startsWith( - ` ${o.class ?? o.outcome} L${o.line} ${o.id} `, + `oracle-gaps: gap ${o.class ?? o.outcome} L${o.line} ${o.id} `, ), ).length !== 1, ) @@ -2167,49 +2201,65 @@ function check(h, observations, expected) { const failingOracle = expected.baseline.filter((b) => b.fails && !b.ordinary) const blockedCount = gapsOfModel.filter((o) => o.outcome === `blocked`).length const blockedLines = stdoutLines.filter((line) => - line.startsWith(`blocked: `), + line.startsWith(`oracle-gaps: blocked `), ) + // The one decisive baseline line, right after the baseline's lines (law + // 15 of the end-to-end oracle; the maintainer's request of 2026-11-12): + // every oracle file passed, or how many failed. + const oracleFiles = expected.baseline.filter((b) => !b.ordinary).length + const files = (n) => `${n} oracle file${n === 1 ? `` : `s`}` + const verdict = + failingOracle.length === 0 + ? `oracle-gaps: baseline all ${files(oracleFiles)} pass` + : `oracle-gaps: baseline ${failingOracle.length} of ${files(oracleFiles)} failed (provisional)` + const done = `oracle-gaps: done completed, ${counted(gapsOfModel.length, `gap`)}${failingOracle.length > 0 ? ` (provisional)` : ``}, report out/summary.md` const provisional = stdoutLines.filter((line) => - line.startsWith(`Provisional: `), + line.startsWith(`oracle-gaps: note provisional: `), + ) + const crashedLegend = legend.find((line) => + line.startsWith(`oracle-gaps: legend crashed: `), ) - const crashedLegend = legend.find((line) => line.startsWith(` crashed: `)) const leadProblem = stdoutLines.slice(0, baselineGot.length).join(`\n`) !== baselineGot.join(`\n`) ? `the stdout does not lead with the baseline` - : blockedLines.length !== failingOracle.length - ? `${blockedLines.length} blocked lines for ${failingOracle.length} failing oracle files` - : failingOracle.find((b) => { - const named = blockedLines.find( - (l) => - readSpan(l.slice(`blocked: `.length))?.content === - relativeName(h, b.fi), - ) - return ( - named === undefined || - (b.test !== undefined && - !named.includes( - ` at ${codeSpanOf(shownName(h, b.fi, b.test))}`, - )) || - !named.endsWith(`then check again.`) - ) - }) - ? `a failing oracle file's blocked line does not name it, its failing test, and the step that unblocks it` - : provisional.length !== (failingOracle.length > 0 ? 1 : 0) || - (failingOracle.length > 0 && - !provisional[0].startsWith( - `Provisional: ${counted(blockedCount, `mutant`)} ${blockedCount === 1 ? `is` : `are`} blocked`, - )) - ? `the stdout does not say that the report is provisional, with ${blockedCount} blocked mutants` - : stdoutLines.at(-1) !== - `completed: ${counted(gapsOfModel.length, `gap`)}` - ? `the last stdout line is ${JSON.stringify(stdoutLines.at(-1))}, not the completion with ${gapsOfModel.length} gaps` - : crashedLegend !== undefined && - !meanings.crashed.test( - crashedLegend.slice(` crashed: `.length), - ) - ? `the crashed legend line does not say a crash is a gap and where its error is` - : undefined + : stdoutLines[baselineGot.length] !== verdict || + stdoutLines.filter((line) => line === verdict).length !== 1 + ? `the stdout's line after the baseline is ${JSON.stringify(stdoutLines[baselineGot.length])}, not the decisive ${JSON.stringify(verdict)}` + : blockedLines.length !== failingOracle.length + ? `${blockedLines.length} blocked lines for ${failingOracle.length} failing oracle files` + : failingOracle.find((b) => { + const named = blockedLines.find( + (l) => + readSpan(l.slice(`oracle-gaps: blocked `.length)) + ?.content === relativeName(h, b.fi), + ) + return ( + named === undefined || + (b.test !== undefined && + !named.includes( + ` at ${codeSpanOf(shownName(h, b.fi, b.test))}`, + )) || + !named.endsWith(`then check again.`) + ) + }) + ? `a failing oracle file's blocked line does not name it, its failing test, and the step that unblocks it` + : provisional.length !== (failingOracle.length > 0 ? 1 : 0) || + (failingOracle.length > 0 && + !provisional[0].startsWith( + `oracle-gaps: note provisional: ${counted(blockedCount, `mutant`)} ${blockedCount === 1 ? `is` : `are`} blocked`, + )) + ? `the stdout does not say that the report is provisional, with ${blockedCount} blocked mutants` + : stdoutLines.at(-1) !== done + ? `the last stdout line is ${JSON.stringify(stdoutLines.at(-1))}, not ${JSON.stringify(done)}` + : crashedLegend !== undefined && + !meanings.crashed.test( + crashedLegend.slice( + `oracle-gaps: legend crashed: `.length, + ), + ) + ? `the crashed legend line does not say a crash is a gap and where its error is` + : undefined if (leadProblem) return violation(`classified`, leadProblem, `stdout`, { actual: stdout }) // The full report leads with the same baseline, before its gaps. @@ -2235,9 +2285,9 @@ function check(h, observations, expected) { unownedFns.set(groupOf(o), (unownedFns.get(groupOf(o)) ?? 0) + 1) const unownedLines = stdout .split(`\n`) - .filter((line) => line.startsWith(` unowned `)) + .filter((line) => line.startsWith(`oracle-gaps: note unowned `)) const unownedText = (group, n) => - ` unowned ${group}: ${n} ${n === 1 ? `mutant` : `mutants`}` + `oracle-gaps: note unowned ${group}: ${n} ${n === 1 ? `mutant` : `mutants`}` if ( unownedLines.length !== unownedFns.size || [...unownedFns].some( @@ -2267,7 +2317,7 @@ function check(h, observations, expected) { { actual: got.killsSection }, ) // The streamed lines are the stdout's gap lines, each once. - const printed = stdoutGaps.map((line) => line.slice(2)).sort() + const printed = [...stdoutGaps].sort() const streamed = got.streamed if (!Array.isArray(streamed)) return violation(`classified`, `no streaming printer`, `stream`) @@ -2282,7 +2332,7 @@ function check(h, observations, expected) { // operator mutant's original and replacement as two code spans, an // authored mutant's intent as a code span and its author. The results keep the full text. for (const owed of gapsOfModel) { - const head = `${owed.class ?? owed.outcome} L${owed.line} ${owed.id} ` + const head = `oracle-gaps: gap ${owed.class ?? owed.outcome} L${owed.line} ${owed.id} ` const line = streamed.find((l) => l.startsWith(head)) const rest = line?.slice(head.length) let shown @@ -2350,7 +2400,8 @@ function baselineBreach(h, observations, expected) { const file = `${relativeName(h, owed.fi)}${owed.ordinary ? ` (ordinary)` : ``}` const at = lines.findIndex( (line) => - line.startsWith(`pass ${file}: `) || line.startsWith(`fail ${file}: `), + line.startsWith(`oracle-gaps: baseline pass ${file}: `) || + line.startsWith(`oracle-gaps: baseline fail ${file}: `), ) if (at < 0) return breach(`no baseline line for ${file}`) // Each file's line gives its test counts; a failing file's lines under @@ -2360,7 +2411,10 @@ function baselineBreach(h, observations, expected) { for (let i = at + 1; lines[i]?.startsWith(` `); i += 1) block.push(lines[i]) if (!owed.fails) { - if (lines[at] !== `pass ${file}: ${counted(owed.tests, `test`)}`) + if ( + lines[at] !== + `oracle-gaps: baseline pass ${file}: ${counted(owed.tests, `test`)}` + ) return breach( `${file} passes ${owed.tests} tests, and its line is ${lines[at]}`, ) @@ -2368,8 +2422,8 @@ function baselineBreach(h, observations, expected) { } const head = owed.test === undefined - ? `fail ${file}: did not load: ` - : `fail ${file}: ${owed.failed} of ${counted(owed.tests, `test`)} failed` + ? `oracle-gaps: baseline fail ${file}: did not load: ` + : `oracle-gaps: baseline fail ${file}: ${owed.failed} of ${counted(owed.tests, `test`)} failed` if ( owed.test === undefined ? !lines[at].startsWith(head) : lines[at] !== head ) @@ -2545,10 +2599,11 @@ const calibrations = [ name: `the stdout leaves a failing oracle file's blocked line out`, law: `stdout`, corrupt: corruptStdout((stdout) => - stdout.includes(`\nblocked: `) || stdout.startsWith(`blocked: `) + stdout.includes(`\noracle-gaps: blocked `) || + stdout.startsWith(`oracle-gaps: blocked `) ? stdout .split(`\n`) - .filter((line) => !line.startsWith(`blocked: `)) + .filter((line) => !line.startsWith(`oracle-gaps: blocked `)) .join(`\n`) : undefined, ), @@ -2556,13 +2611,15 @@ const calibrations = [ { name: `the stdout's baseline comes after its gaps`, law: `stdout`, + // The baseline's lines, everything before the totals, moved after the + // rest of the stdout. corrupt: corruptStdout((stdout) => { const lines = stdout.split(`\n`) - const blank = lines.indexOf(``) - if (blank <= 0) return undefined - return [...lines.slice(blank + 1), ``, ...lines.slice(0, blank)].join( - `\n`, + const totals = lines.findIndex((line) => + line.startsWith(`oracle-gaps: summary `), ) + if (totals <= 0) return undefined + return [...lines.slice(totals), ...lines.slice(0, totals)].join(`\n`) }), }, { @@ -2610,7 +2667,7 @@ const calibrations = [ const value = structuredClone(trace[index].value) const line = value.stdout ?.split(`\n`) - .find((l) => /^ \S+ L\d+ m\d+ /.test(l)) + .find((l) => /^oracle-gaps: gap \S+ L\d+ m\d+ /.test(l)) if (line === undefined) return undefined value.stdout = value.stdout.replace(`${line}\n`, ``) return trace.map((o, i) => (i === index ? { ...o, value } : o)) diff --git a/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs b/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs index dc96d11..1958c9f 100644 --- a/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs +++ b/packages/oracles/oracle-gaps/gap-report-model-oracle.mjs @@ -594,6 +594,10 @@ const drawn = fc.record({ // it loads. Each is the author's to fix, so each refuses the run. `configThrows`, `configNoDefault`, + // A config whose default export is null, and one with a syntax error, + // which Vitest refuses too (review of #46, F5 and F7). + `configNull`, + `configSyntax`, `setupThrows`, ), stability: fc.constantFrom( @@ -616,7 +620,12 @@ const drawn = fc.record({ // The package's Vitest config: its file, and whether it exports an object // or a function (sync or async) that returns one. Vitest accepts each, so // the runner must too. - config: fc.constantFrom(`object`, `object`, `function`, `async`), + // Whether file a has a test that writes to the process's own streams + // (review of #46, F2; re-review N2: a run whose tests wrote nothing names + // no log). + consoleOutput: fc.boolean(), + // A promise of a config, which Vite awaits (review of #46, F5). + config: fc.constantFrom(`object`, `object`, `function`, `async`, `promise`), // A config may read the environment Vitest's CLI sets before it loads // one (VITEST, TEST, and NODE_ENV), as svelte-db's picks its resolve // conditions by VITEST. This one finds its tests only under it. @@ -657,8 +666,15 @@ const drawn = fc.record({ fc.record({ signal: fc.constantFrom(`SIGTERM`, `SIGINT`), pool: fc.constantFrom(`forks`, `threads`), + // When the signal comes: while the stop witness spins (`run`), + // once the run has printed its completed line, while its process + // holds its exit (`after`, review of #46, F3), or while the run + // warms its config (`warm`, re-review N1). Or no signal: a test + // aborts the run's process (`abort`, as a V8 out-of-memory abort + // does; the coordinator's decision of 2026-11-12 on P10). + when: fc.constantFrom(`run`, `run`, `after`, `warm`, `abort`), }), - { minLength: 1, maxLength: 2 }, + { minLength: 1, maxLength: 5 }, ), }, ), @@ -2067,9 +2083,14 @@ const formatOf = (file, packageType) => */ function configOf(history) { const needsConfig = - [`configThrows`, `configNoDefault`, `projects`, `setupThrows`].includes( - history.fault, - ) || + [ + `configThrows`, + `configNoDefault`, + `configNull`, + `configSyntax`, + `projects`, + `setupThrows`, + ].includes(history.fault) || history.configEnv === true || history.setupGlobal === true const place = @@ -2630,6 +2651,10 @@ function resolveHistory(history) { stability: history.stability, steps: history.steps ?? [], lazyGate: history.lazyGate === true, + consoleOutput: history.consoleOutput === true, + // Whether a run aborts its process (a stop run `abort`): then file a has + // the test that aborts when the driver asks. + aborts: (history.stopRuns ?? []).some((stop) => stop.when === `abort`), ordinary: history.ordinary === true, // An `owns` key in an assertion's arguments, in the ordinary file: its // assertion form and whether its list is a literal, or none. @@ -3801,7 +3826,11 @@ function refusal(plan) { detail: `the fixture config broke`, runtime: true, } - if (plan.fault === `configNoDefault`) + if ( + plan.fault === `configNoDefault` || + plan.fault === `configNull` || + plan.fault === `configSyntax` + ) return { cause: plan.configPath, runtime: true } // A run with no oracle file, or with no owns entry, owns no selected // source, so every mutant would read as unowned. The analysis refuses it @@ -3849,7 +3878,12 @@ function blockingOf(plan) { function baselineOf(plan) { const failing = blockingOf(plan) return [...oracleFilesOf(plan), `c`].map((file) => { - const own = plan.tests.filter((t) => t.file === file).length + // File a's test that writes to the console streams runs in every + // baseline too. + const own = + plan.tests.filter((t) => t.file === file).length + + (file === `a` && plan.consoleOutput ? 1 : 0) + + (file === `a` && plan.aborts ? 1 : 0) const broken = failing.find((f) => f.file === file) // The failing owner's suite holds two tests and a third of its name. const extra = broken?.test === undefined ? 0 : 3 @@ -3881,6 +3915,48 @@ const declarationFaults = [ const reachesRun = (history) => !declarationFaults.includes(history.fault ?? `none`) +// Law 15, the stream's lines (the maintainer's request of 2026-11-12, +// relayed by the coordinator: an agent acts on a check's results as they +// stream, so one `grep '^oracle-gaps: '` finds every line the CLI writes +// for it, and the kind's word right after the prefix filters one kind; the +// same prefixes on stdout and stderr, in a cold run and through the +// daemon). The kinds: +// The stdout summary's lines have kinds of their own: its totals +// (`summary`), one directive per gap class (`legend`), and each function's +// heading over its gap lines (`function`) (the coordinator's decision of +// 2026-11-12 on review F1: every line of either stream is a line of the +// stream). +const streamKinds = [ + `config`, + `note`, + `baseline`, + `blocked`, + `progress`, + `gap`, + `busy`, + `summary`, + `legend`, + `function`, + `done`, +] +// The one decisive baseline line, before any mutant runs: every oracle +// file passed, or how many of them failed, which makes the report +// provisional. +const baselineVerdict = + /^oracle-gaps: baseline (?:all \d+ oracle files? pass|\d+ of \d+ oracle files? failed \(provisional\))$/ +const countedFiles = (n) => `${n} oracle file${n === 1 ? `` : `s`}` +/** The decisive baseline line of a run with `files` oracle files, `failing` of them failing. */ +const verdictLine = (files, failing) => + failing === 0 + ? `oracle-gaps: baseline all ${countedFiles(files)} pass` + : `oracle-gaps: baseline ${failing} of ${countedFiles(files)} failed (provisional)` +/** + * The last line of a completed run's stdout: its status, its gap count, and + * the path of its full report in output directory `out`. + */ +const doneLine = (gaps, provisional, out) => + `oracle-gaps: done completed, ${gaps} gap${gaps === 1 ? `` : `s`}${provisional ? ` (provisional)` : ``}, report ${out}/summary.md` + function model(plan) { const refused = refusal(plan) // The config the run names (F12) once its contexts have loaded it: the @@ -3893,13 +3969,15 @@ function model(plan) { `brokenVitest`, `configThrows`, `configNoDefault`, + `configNull`, + `configSyntax`, `projects`, ].includes(plan.fault) ? [] : [ plan.configPath === undefined - ? `no config file: Vitest's default discovery` - : `config: ${plan.configPath}`, + ? `oracle-gaps: config none, Vitest's default discovery` + : `oracle-gaps: config ${plan.configPath}`, ] if (refused) return { refused, configLines } const sites = sitesOf(plan) @@ -4392,9 +4470,32 @@ function oracleTestFile(plan, file) { `})`, `test('fixed seed 999', () => {})`, ) + // A test that ends the run's process by SIGABRT when the driver asks + // (`GAP_ABORT`), as a V8 out-of-memory abort does: in worker threads its + // process is the run's, and `process.abort()` is unavailable there. It + // runs and passes in every other run. + if (file === `a` && plan.aborts) + out.push( + `test('aborts the process when asked', () => {`, + ` if (process.env.GAP_ABORT === '1') process.kill(process.pid, 'SIGABRT')`, + `})`, + ) // A file with no tests still loads, so the CLI can read what it owns. - if (!plan.tests.some((t) => t.file === file)) + if ( + !plan.tests.some((t) => t.file === file) && + !(file === `a` && (plan.consoleOutput || plan.aborts)) + ) out.push(`test.skip('no tests', () => {})`) + // A test that writes to the process's own streams, as a test's debugging + // output does: none of it may reach the CLI's streams (law 15; review of + // #46, F2). It calls nothing of the source under test. + if (file === `a` && plan.consoleOutput) + out.push( + `test('writes to the console streams', () => {`, + ` process.stdout.write('raw stdout from a test\\n')`, + ` process.stderr.write('raw stderr from a test\\n')`, + `})`, + ) return `${out.join(`\n`)}\n` } @@ -4537,13 +4638,24 @@ const altConfigPath = (plan) => */ function configText(plan, test) { const commonjs = plan.configSyntax === `cjs` - const body = `defineConfig(${ - { object: ``, function: `() => (`, async: `async () => (` }[plan.config] + const config = `defineConfig(${ + { object: ``, function: `() => (`, async: `async () => (`, promise: `` }[ + plan.config + ] }{ test: ${ plan.fault === `projects` ? `{ projects: [{ test: { name: 'unit', ...${test} } }] }` : test - } }${plan.config === `object` ? `` : `)`})` + } }${plan.config === `object` || plan.config === `promise` ? `` : `)`})` + // A promise of the config, a null default, or a config cut off mid-object. + const body = + plan.fault === `configNull` + ? `null` + : plan.fault === `configSyntax` + ? `defineConfig({ test: {` + : plan.config === `promise` + ? `Promise.resolve(${config})` + : config return [ commonjs ? `const { defineConfig } = require('vitest/config')` @@ -5019,7 +5131,7 @@ function applyEdit(state, edit) { * one. `name` replaces the scratch paths in a text, so two runs of one * history in different directories record the same ending. */ -function endingOf(exit, stderr, out, name = (text) => text) { +function endingOf(exit, stderr, out, name = (text) => text, stdout = ``) { const text = `${stderr ?? ``}` const line = (label) => text.match(new RegExp(`^${label}: (.*)$`, `m`))?.[1] const cause = text.match(/^refused: ([\s\S]*?)\nfix: /m)?.[1] @@ -5035,11 +5147,22 @@ function endingOf(exit, stderr, out, name = (text) => text) { } catch { // No output directory is known, or the run wrote no results.json. } + // The run's last line: what follows `oracle-gaps: done ` there (law 15). + const last = text.split(`\n`).filter(Boolean).at(-1) ?? `` + const done = last.startsWith(`oracle-gaps: done `) + ? name(last.slice(`oracle-gaps: done `.length)) + : undefined + // The stdout's lines: a run that does not complete prints only its done + // line there (law 15). + const printed = `${stdout ?? ``}`.split(`\n`).filter(Boolean) return { exit, cause: cause && name(cause).slice(0, 2000), fix: line(`fix`) && name(line(`fix`)), - namesOutput: out !== undefined && line(`output`) === out, + done, + stdout: printed.map(name), + namesOutput: + out !== undefined && done === name(`refused, report ${out}/summary.md`), // An internal error's log beside a refusal would be another run's. errorLog: out !== undefined && existsSync(join(out, `error.log`)), stackFrames: /^\s+at /m.test(text), @@ -5064,9 +5187,22 @@ function refusalBreach(ending, { cause, outKnown }) { // problem in place. if (/\b(ignore|equivalent|skip)/i.test(ending.fix)) return `the refusal's fix offers a way around it: ${ending.fix}` + // Its last line says that it refused (law 15), with its report once the + // run knows its output directory. + if ( + ending.done === undefined || + !/^refused(?:, report .+)?$/.test(ending.done) + ) + return `the refusal's last line is not \`oracle-gaps: done refused\`` + // The done line ends stdout too, as its only line. + if ( + ending.stdout.length !== 1 || + ending.stdout[0] !== `oracle-gaps: done ${ending.done}` + ) + return `the refusal's stdout is ${JSON.stringify(ending.stdout)}, not its done line` if (!outKnown) return undefined if (!ending.namesOutput) - return `the refusal does not name its output directory` + return `the refusal does not name its report in its output directory: done ${ending.done}` const { written } = ending if (written?.status !== `refused`) return `the output's results.json has status ${written?.status ?? `(none)`}, not refused` @@ -5171,6 +5307,10 @@ export { histories, witnessSteps, budgetSteps, + streamKinds, + baselineVerdict, + verdictLine, + doneLine, kill, push, judging, diff --git a/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs b/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs index b74f4fc..f3f3a63 100644 --- a/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs +++ b/packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs @@ -312,8 +312,25 @@ * RED baseline gives a provisional report). With `--only`, the baseline * runs only the selected mutants' owners (and the ordinary files with * `--ordinary`), and each selected mutant is judged as the full run - * judges it. The last stdout line is `completed: gaps` (G6): exit 0 + * judges it. The last stdout line is the run's done line (law 15), + * `oracle-gaps: done completed, gaps, report ` (G6): exit 0 * means the check ran to its end, not that it found no gaps. + * 15. **The stream's lines are greppable** (the maintainer's request of + * 2026-11-12, relayed by the coordinator, with its decisions of the same + * day: the same prefixes on stdout and stderr, and a line for a stopped + * run). An agent acts on a check's results as they stream, so every line + * the CLI writes for the check starts with `oracle-gaps: ` and then its + * kind: `config`, `note`, `baseline`, `blocked`, `progress`, `gap`, + * `busy`, or `done`. Only an indented line under a baseline failure, a + * refusal's `refused:` and `fix:` block, and a line Node writes for the + * process are not. The baseline ends with one decisive line before any + * mutant runs: `oracle-gaps: baseline all oracle files pass`, or + * `oracle-gaps: baseline of oracle files failed (provisional)`. + * The stdout's baseline, blocked, and gap lines carry the same prefixes. + * The run's last line is its done line: `completed, gaps[ + * (provisional)], report ` on stdout, `refused[, report + * ]` after a refusal's block, or `stopped ()` + * after a stop. A check through the daemon streams the same lines. * * Authority: the maintainer's decisions in the gap-check design runs v11, v14, * and v19 (2026-11-04 and 2026-11-05; the design notes are kept by the project @@ -462,6 +479,10 @@ import { histories, witnessSteps, budgetSteps, + streamKinds, + baselineVerdict, + verdictLine, + doneLine, push, judging, witness, @@ -653,8 +674,28 @@ const strays = (dir, before, running = false) => * the package's `.oracle-gaps` (law 10's namespace, while the run runs, not * only after it closes), sends the signal to the CLI, and waits for the CLI * to end. A second later, each recorded process must have ended. + * + * With `when` `after`, the run is not stopped while it runs: the witness + * does not spin, the run's process holds its exit for 500 ms once it has + * printed its done line (`ORACLE_GAPS_TEST_HOLD_EXIT_MS`, a hook for this + * driver), and the driver sends the signal once that line is on stdout + * (review of #46, F3). The run then ends as it completed, with exactly one + * done line. With `warm`, the driver sends the signal while the run warms + * its config (`ORACLE_GAPS_TEST_WARM_PROBE`, a hook that writes its marker there + * and then waits, as a config load does, on a service in the run's group + * that the signal ends, with a handler that takes 200 ms to exit, as + * Vitest's does; re-review N1). The run then ends stopped, with exactly + * one done line and no refusal. */ -async function stoppedRun(dir, files, how, ids, { signal, pool }) { +async function stoppedRun(dir, files, how, ids, { signal, pool, when }) { + const after = when === `after` + const warm = when === `warm` + // With `abort`, no signal comes: a test of file a aborts the run's + // process (`GAP_ABORT`), in worker threads, so the abort ends the process + // that runs the check, as a V8 out-of-memory abort does. The run then + // ends with one error line per stream that names how its process ended, + // and exits 3 (the coordinator's decision of 2026-11-12 on P10). + const abort = when === `abort` writeFixture(dir, files) const before = new Set(filesUnder(dir)) const pids = mkdtempSync(join(realpathSync(tmpdir()), `gap-stop-`)) @@ -663,33 +704,68 @@ async function stoppedRun(dir, files, how, ids, { signal, pool }) { outName: `out-stop`, contexts: 1, only: ids, - pool, + pool: abort ? `threads` : pool, }) const child = spawn(process.execPath, args, { cwd: repo, - env: { ...cleanEnv(), GAP_STOP_DIR: pids }, - stdio: `ignore`, + env: after + ? { ...cleanEnv(), ORACLE_GAPS_TEST_HOLD_EXIT_MS: `500` } + : warm + ? { ...cleanEnv(), ORACLE_GAPS_TEST_WARM_PROBE: join(pids, `warming`) } + : abort + ? { ...cleanEnv(), GAP_ABORT: `1` } + : { ...cleanEnv(), GAP_STOP_DIR: pids }, + stdio: [`ignore`, `pipe`, `pipe`], + }) + let stdout = `` + let stderr = `` + child.stdout.on(`data`, (chunk) => { + stdout += chunk + }) + child.stderr.on(`data`, (chunk) => { + stderr += chunk }) let exited = false + let exitCode const exit = new Promise((resolvePromise) => - child.on(`exit`, () => { + child.on(`exit`, (code) => { exited = true + exitCode = code resolvePromise(true) }), ) const sleep = (ms) => new Promise((r) => setTimeout(r, ms)) for ( let waited = 0; - !exited && readdirSync(pids).length < 2 && waited < 120_000; + !exited && + !abort && + (after + ? !stdout.includes(`oracle-gaps: done completed`) + : warm + ? !existsSync(join(pids, `warming`)) + : readdirSync(pids).length < 2) && + waited < 120_000; waited += 50 ) await sleep(50) - const spun = readdirSync(pids).length >= 2 + // Spun: the witness's test spins (with `after`, the done line came while + // the run's process still held its exit; with `warm`, the run was warming + // its config). + // With `abort`, the run's own end is the event: it ended unstopped. + if (abort) await Promise.race([exit, sleep(120_000)]) + const spun = abort + ? exited + : after + ? stdout.includes(`oracle-gaps: done completed`) && !exited + : warm + ? existsSync(join(pids, `warming`)) && !exited + : readdirSync(pids).length >= 2 const strayWhileRunning = strays(dir, before, true) - child.kill(signal) + if (!abort) child.kill(signal) const ended = await Promise.race([exit, sleep(30_000).then(() => false)]) await sleep(1000) const alive = readdirSync(pids) + .filter((name) => /^\d+$/.test(name)) .map(Number) .filter((pid) => { try { @@ -716,6 +792,23 @@ async function stoppedRun(dir, files, how, ids, { signal, pool }) { ended, survivors: alive.length, strayWhileRunning, + when: when ?? `run`, + // Whether a refusal's block reached stderr: a stop is never a refusal. + refused: stderr.split(`\n`).some((line) => line.startsWith(`refused: `)), + exit: exitCode, + // Both streams' lines (law 15), with the scratch paths named. + stream: streamOf(stderr.split(`\n`), (text) => + namedScratch(text, [ + [how.cli ?? here, ``], + [dir, ``], + ]), + ), + stdoutStream: streamOf(stdout.split(`\n`), (text) => + namedScratch(text, [ + [how.cli ?? here, ``], + [dir, ``], + ]), + ), // The session directories in the package once it has ended: a stopped // run leaves its own, and the next run removes it. sessions: readdirSync(dir).filter((name) => @@ -792,9 +885,15 @@ async function runClisAtOnce(dir, files, hows) { const child = spawn(process.execPath, args, { cwd: repo, env: { ...cleanEnv(), ...how.env }, - stdio: [`ignore`, `ignore`, `pipe`], + stdio: [`ignore`, `pipe`, `pipe`], }) + // Both streams, as runCli reads them: law 15 judges each run's lines, + // its stdout's done line included. + let stdout = `` let stderr = `` + child.stdout.on(`data`, (chunk) => { + stdout += chunk + }) child.stderr.on(`data`, (chunk) => { stderr += chunk }) @@ -805,7 +904,7 @@ async function runClisAtOnce(dir, files, hows) { resolvePromise(code) }) }) - return readRun(dir, out, status, stderr, how.cli ?? here) + return readRun(dir, out, status, stderr, how.cli ?? here, stdout) }), ) } @@ -835,9 +934,67 @@ function namedScratch(text, names) { /** What one run of the CLI reported, from its exit status and its files. */ // A gap line the CLI streams to stderr as its mutant is judged (stream.mjs): -// its class, its line, and its id. +// its class, its line, and its id, after the stream's prefix (law 15). const streamedGap = - /^(?:not-judged|indeterminate|dead-code|outside-annotation|branch-never-taken|not-infected|not-observed|not-distinguished|unreached|random-only-kill|agreement-only-kill|unregistered-kill|framework-error|timeout|setup-error|crashed|does-not-apply|blocked) L(?:\d+|\?) (\S+) / + /^oracle-gaps: gap (?:not-judged|indeterminate|dead-code|outside-annotation|branch-never-taken|not-infected|not-observed|not-distinguished|unreached|random-only-kill|agreement-only-kill|unregistered-kill|framework-error|timeout|setup-error|crashed|does-not-apply|blocked) L(?:\d+|\?) (\S+) / + +/** + * What law 15 judges of a run's stderr, as relations that do not depend on + * the order in which its contexts finished: + * - `strays`: each line, once, that is neither a line of a stream kind + * (`oracle-gaps: `), an indented line under a baseline failure, a + * refusal's block (`refused:` up to its `fix:` line), nor a line Node + * writes for the process (a warning); + * - `verdicts`: the decisive baseline lines; + * - `mutantBeforeVerdict`: whether a gap line, a count of judged mutants, + * or a heartbeat that names a mutant came before the first verdict; + * - `last`: the last line, which ends a run that did not complete. + */ +function streamOf(lines, name) { + const kinds = new RegExp(`^oracle-gaps: (?:${streamKinds.join(`|`)}) `) + const strays = new Set() + let inRefusal = false + for (const line of lines) { + if (line === ``) continue + if (line.startsWith(`refused: `)) inRefusal = true + const ending = inRefusal + if (line.startsWith(`fix: `)) inRefusal = false + if ( + ending || + kinds.test(line) || + line.startsWith(` `) || + /^\((?:node:\d+|Use `node)/.test(line) + ) + continue + strays.add(name(line).slice(0, 300)) + } + const verdict = (line) => baselineVerdict.test(line) + const first = lines.findIndex(verdict) + const mutantLine = (line) => + streamedGap.test(line) || + line.startsWith(`oracle-gaps: progress judged `) || + (line.startsWith(`oracle-gaps: busy `) && + !line.startsWith(`oracle-gaps: busy baseline `)) + const before = first < 0 ? lines : lines.slice(0, first) + return { + strays: [...strays].sort().slice(0, 10), + verdicts: lines.filter(verdict), + mutantBeforeVerdict: before.some(mutantLine), + last: name(lines.filter((line) => line !== ``).at(-1) ?? ``), + // How many done lines the stream has: exactly one, its last. + dones: lines.filter((line) => line.startsWith(`oracle-gaps: done `)).length, + // The notes, each once, named as the trace names paths, and with the + // output directory's name left out: a cold run and a check through the + // daemon name different ones. + notes: [ + ...new Set( + lines + .filter((line) => line.startsWith(`oracle-gaps: note `)) + .map((line) => name(line).replace(/\/out(?:-[\w-]+)?\//g, `//`)), + ), + ].sort(), + } +} function readRun(dir, out, status, stderr, cli, stdout) { // The output names the CLI directory that ran and the package, as scratch @@ -877,7 +1034,7 @@ function readRun(dir, out, status, stderr, cli, stdout) { const heartbeats = [ ...new Set( lines.flatMap((line) => { - const beat = line.match(/^busy: (\S+) in (\S+) for \d+s$/) + const beat = line.match(/^oracle-gaps: busy (\S+) in (\S+) for \d+s$/) return beat ? [JSON.stringify({ label: beat[1], file: beat[2] })] : [] }), ), @@ -892,24 +1049,54 @@ function readRun(dir, out, status, stderr, cli, stdout) { const blank = printed.indexOf(``) return { lead: blank < 0 ? printed : printed.slice(0, blank), - lastLine: printed.at(-1), + lastLine: printed.at(-1) && namedScratch(printed.at(-1), scratch), + // The stdout's own lines of the stream's kinds: the gap lines under + // their functions, as the stream prints them (law 15). + stdoutGaps: printed.filter((line) => streamedGap.test(line)).length, baselineFiles: lines.flatMap((line) => { - const file = line.match(/^(?:pass|fail) (\S+)(?: \(ordinary\))?: /)?.[1] + const file = line.match( + /^oracle-gaps: baseline (?:pass|fail) (\S+)(?: \(ordinary\))?: /, + )?.[1] return file === undefined ? [] : [file] }), heartbeats, // The config the run says it used (law 3), and the warnings Node printed // for the process. configLines: lines.filter((line) => - /^(config: |no config file: )/.test(line), + line.startsWith(`oracle-gaps: config `), + ), + // The stream's grammar (law 15), recorded as relations so that a trace + // does not depend on the order in which contexts finished. + stream: streamOf(lines, (text) => namedScratch(text, scratch)), + // The stdout's lines, judged as the stderr's are (review of #46, F1 and + // F2): every line of either stream is a line of the stream. + stdoutStream: streamOf(`${stdout ?? ``}`.split(`\n`), (text) => + namedScratch(text, scratch), ), + // The output directory the run was given, as the trace names it: the + // final line names its report there. + outDir: namedScratch(out, scratch), + // Whether the run captured its tests' own console output there. + captured: (() => { + try { + return readFileSync(join(out, `test-output.log`)).length > 0 + } catch { + return false + } + })(), warnings: lines.filter((line) => /MaxListenersExceededWarning/.test(line)), // How a run that did not complete ended, read from the whole of its // stderr and its output directory (law 12). ending: status === 0 ? undefined - : endingOf(status, stderr, out, (text) => namedScratch(text, scratch)), + : endingOf( + status, + stderr, + out, + (text) => namedScratch(text, scratch), + stdout, + ), exit: status, stderr: named === undefined ? undefined : named.slice(-1500), reported: results !== undefined, @@ -1100,12 +1287,16 @@ async function incremental(dir, plan, cli, copy, observe, states) { exit: coldRun.exit, refusal: coldRun.refusal, report: coldRun.report, + notes: coldRun.stream.notes, }, daemon: { exit: run.exit, refusal: run.refusal, ending: run.ending, report: run.report, + // The stream through the daemon (law 15): both streams' lines. + stream: run.stream, + stdoutStream: run.stdoutStream, // The first line of a failure, for the message; readRun has named // the scratch package and CLI copy in it. stderr: run.stderr?.split(`\n`)[0], @@ -1113,6 +1304,12 @@ async function incremental(dir, plan, cli, copy, observe, states) { }) if (twice && step === `start`) states.add(`two checks started the daemon at once`) + if ( + daemonConfigFaults.includes(plan.fault) && + daemon.exit === 2 && + coldRun.exit === 2 + ) + states.add(`a config refusal through the daemon`) // A session that cannot open, heard by two checks at once; one that // fails while the CLI that started its daemon is connected and unread; // and one whose failure, larger than a socket's buffer, a queued @@ -1382,6 +1579,8 @@ async function driver(plan, observe) { observe(`replay`, { command: replay, ran: replayed(dir, replay) }) // Reach comes from what the CLI reported, mapped onto the plan. const states = new Set() + if (cli.exit === 0 && !plan.consoleOutput) + states.add(`a completed run whose tests wrote nothing`) // Law 14: a partial run of one owned function's mutants, whose baseline // runs only their owners. const chosen = cli.mutants.find( @@ -1423,6 +1622,18 @@ async function driver(plan, observe) { states.add(`a run after a stopped one that left its session`) previousSessions = stopped.sessions if (!stopped.spun) continue + if (stop.when === `after`) { + states.add(`a stop signal after the completed line`) + continue + } + if (stop.when === `warm`) { + states.add(`a stop signal while the config warms`) + continue + } + if (stop.when === `abort`) { + states.add(`a run whose process aborted`) + continue + } states.add(`a run stopped by ${stop.signal} while a worker spun`) states.add( `a stopped run whose workers are ${stop.pool === `forks` ? `child processes` : `threads`}`, @@ -1874,6 +2085,43 @@ function check(plan, observations, expected) { `config`, { actual: cli.configLines, expected: expected.configLines }, ) + const stream = streamBreach(`cli`, cli, expected, plan) + if (stream) return stream + // Through the daemon, each check streams the same kinds of line, and a + // completed one ends with its done line (law 15). The state of each step + // differs, so its verdict and counts are judged by law 9's comparison. + for (const { value } of observations.filter( + (o) => o.checkpoint === `incremental`, + )) { + const { daemon, cold } = value + const ends = endsBreach(daemon) + if (ends) + return violation( + `incremental`, + `through the daemon at ${value.step}: ${ends.difference}`, + `greppable stream`, + ends.extra, + ) + // A refusal through the daemon prints the cold run's notes (review of + // #46, F4): the notes of the session that failed reach the CLI. + if ( + cold.exit !== 0 && + daemon.exit === cold.exit && + JSON.stringify(daemon.stream.notes) !== JSON.stringify(cold.notes) + ) + return violation( + `incremental`, + `the refusal through the daemon at ${value.step} prints the notes ${JSON.stringify(daemon.stream.notes)}, and the cold run ${JSON.stringify(cold.notes)}`, + `greppable stream`, + { actual: daemon.stream.notes, expected: cold.notes }, + ) + if (daemon.stream.mutantBeforeVerdict) + return violation( + `incremental`, + `through the daemon at ${value.step}, a line about a mutant came before the decisive baseline line`, + `greppable stream`, + ) + } if (expected.refused) { if (cli.exit === 0) return violation( @@ -2115,6 +2363,34 @@ function check(plan, observations, expected) { `stopping`, { actual: value.survivors, expected: 0 }, ) + // A stopped run ends both streams with its stopped line, and one that + // had printed its completed line ends as it completed (law 15). + const ends = value.spun ? endsBreach(value) : undefined + if (ends) + return violation( + `stopped`, + `the run sent ${how}${value.when === `run` ? `` : ` (${value.when})`}: ${ends.difference}`, + `greppable stream`, + ends.extra, + ) + const owedEnd = + value.when === `after` + ? `oracle-gaps: done completed` + : value.when === `abort` + ? `oracle-gaps: done error, the run's process exited with SIGABRT` + : `oracle-gaps: done stopped (${value.signal})` + if (value.spun && value.when === `warm` && value.refused) + return violation( + `stopped`, + `the run stopped while its config warmed reported a refusal`, + `greppable stream`, + ) + if (value.spun && !value.stream.last.startsWith(owedEnd)) + return violation( + `stopped`, + `the stopped run's last line is ${JSON.stringify(value.stream.last)}, not ${owedEnd}`, + `greppable stream`, + ) } const lead = leadBreach(plan, cli, observations, expected) if (lead) return lead @@ -2144,6 +2420,103 @@ function check(plan, observations, expected) { return incrementalCheck(observations) ?? daemonCheck(observations) ?? pass } +/** + * Law 15: the stream's lines are greppable. Every line a run writes to + * stderr is a line of one of the stream's kinds, `oracle-gaps: ` + * (config, note, baseline, blocked, progress, gap, busy, done), an indented + * line under a baseline failure, a refusal's block, or a line Node writes + * for the process. A run that judges mutants prints its one decisive + * baseline line before any gap line, count of judged mutants, or heartbeat + * that names a mutant, and its stdout prints each gap once, on a line of + * the gap kind. The ends of a run (`oracle-gaps: done …`) are judged with + * the lead (completed), the refusal law (refused), and the stop law + * (stopped). `run` is a run as readRun records it, observed at + * `checkpoint`. + */ +function streamBreach(checkpoint, run, expected, plan) { + const breach = (difference, extra) => + violation(checkpoint, difference, `greppable stream`, extra) + const ends = endsBreach(run) + if (ends) return breach(ends.difference, ends.extra) + if (run.stream.mutantBeforeVerdict) + return breach( + `a line about a mutant came before the decisive baseline line`, + ) + if (run.exit !== 0 || expected.refused) return undefined + // A note names the log once the run captured its tests' console output + // (review of #46, F2; re-review N2), and only then. File a's test that + // writes to the process's streams runs once file a loads, so its output + // must be captured; a setup file that throws fails every file before any + // test runs, and a file a with no such test writes nothing. + const writes = + plan.consoleOutput && + expected.baseline.find((b) => b.file === `tests/a.test.ts`)?.loads + if (writes && !run.captured) + return breach( + `file a's test wrote to the console, and the run captured none of it`, + ) + const outputNotes = run.stream.notes.filter((note) => + /^oracle-gaps: note test output in \S+test-output\.log$/.test(note), + ) + const owedNotes = run.captured ? 1 : 0 + if (outputNotes.length !== owedNotes) + return breach( + `the run names its tests' console output in ${outputNotes.length} notes, owed ${owedNotes} (captured: ${run.captured}): ${JSON.stringify(run.stream.notes)}`, + ) + const verdict = verdictLine( + expected.baseline.filter((b) => !b.ordinary).length, + expected.blocking.length, + ) + if (JSON.stringify(run.stream.verdicts) !== JSON.stringify([verdict])) + return breach( + `the stream's decisive baseline lines are ${JSON.stringify(run.stream.verdicts)}, owed ${JSON.stringify([verdict])}`, + { actual: run.stream.verdicts, expected: [verdict] }, + ) + if (run.stdoutGaps !== expected.gaps.length) + return breach( + `the stdout has ${run.stdoutGaps} gap lines for ${expected.gaps.length} gaps`, + ) + return undefined +} + +/** + * What law 15 judges of every run, on both streams, whatever its outcome: + * no stray line, and exactly one done line, which ends each stream, the + * same on both, of the kind its exit status says (0 completed, 2 refused, + * 3 error, and a signal's code stopped). Undefined when it holds. + */ +function endsBreach(run) { + const streams = [ + [`stderr`, run.stream], + [`stdout`, run.stdoutStream], + ] + for (const [where, stream] of streams) + if (stream.strays.length > 0) + return { + difference: `the run wrote a line on ${where} that is not a line of the stream: ${stream.strays[0]}`, + extra: { actual: stream.strays, expected: [] }, + } + for (const [where, stream] of streams) + if (stream.dones !== 1 || !stream.last.startsWith(`oracle-gaps: done `)) + return { + difference: `${where} has ${stream.dones} done lines and ends with ${JSON.stringify(stream.last)}`, + } + if (run.stream.last !== run.stdoutStream.last) + return { + difference: `stderr ends with ${JSON.stringify(run.stream.last)} and stdout with ${JSON.stringify(run.stdoutStream.last)}`, + } + const kind = run.stream.last + .slice(`oracle-gaps: done `.length) + .match(/^\w+/)?.[0] + const owed = + { 0: `completed`, 2: `refused`, 3: `error` }[run.exit] ?? `stopped` + if (kind !== owed) + return { + difference: `the run exited ${run.exit} and its done line says ${kind}, not ${owed}`, + } + return undefined +} + /** * Law 14: the report leads with the baseline. The stdout opens with each * test file's result and its test counts, a failing file's failing test and @@ -2164,14 +2537,15 @@ function leadBreach(plan, cli, observations, expected) { const name = `${b.file}${b.ordinary ? ` (ordinary)` : ``}` const at = cli.lead.findIndex( (line) => - line.startsWith(`pass ${name}: `) || line.startsWith(`fail ${name}: `), + line.startsWith(`oracle-gaps: baseline pass ${name}: `) || + line.startsWith(`oracle-gaps: baseline fail ${name}: `), ) if (at < 0) return breach(`the stdout gives no baseline line for ${name}`) const owed = !b.loads - ? `fail ${name}: did not load: ` + ? `oracle-gaps: baseline fail ${name}: did not load: ` : b.failed > 0 - ? `fail ${name}: ${b.failed} of ${counted(b.ran, `test`)} failed` - : `pass ${name}: ${counted(b.ran, `test`)}${b.skipped > 0 ? `, ${b.skipped} skipped` : ``}` + ? `oracle-gaps: baseline fail ${name}: ${b.failed} of ${counted(b.ran, `test`)} failed` + : `oracle-gaps: baseline pass ${name}: ${counted(b.ran, `test`)}${b.skipped > 0 ? `, ${b.skipped} skipped` : ``}` if (b.loads ? cli.lead[at] !== owed : !cli.lead[at].startsWith(owed)) return breach( `the baseline line of ${name} is ${cli.lead[at]}, owed ${owed}`, @@ -2184,14 +2558,27 @@ function leadBreach(plan, cli, observations, expected) { `the baseline of ${name} does not name its failing test ${b.test} and its error`, ) } - const blocked = cli.lead.filter((line) => line.startsWith(`blocked: `)) + // The one decisive baseline line (law 15): every oracle file passed, or + // how many failed. + const verdict = verdictLine( + expected.baseline.filter((b) => !b.ordinary).length, + expected.blocking.length, + ) + const verdicts = cli.lead.filter((line) => baselineVerdict.test(line)) + if (verdicts.length !== 1 || verdicts[0] !== verdict) + return breach( + `the stdout's decisive baseline lines are ${JSON.stringify(verdicts)}, owed ${JSON.stringify([verdict])}`, + ) + const blocked = cli.lead.filter((line) => + line.startsWith(`oracle-gaps: blocked `), + ) if (blocked.length !== expected.blocking.length) return breach( `${blocked.length} blocked lines for ${expected.blocking.length} failing oracle files`, ) for (const f of expected.blocking) { const at = f.test === undefined ? `` : ` at ${span(f.test)}` - const line = `blocked: ${span(`tests/${f.file}.test.ts`)} fails with no mutant active${at}, ` + const line = `oracle-gaps: blocked ${span(`tests/${f.file}.test.ts`)} fails with no mutant active${at}, ` if ( !blocked.some( (l) => l.startsWith(line) && l.endsWith(`then check again.`), @@ -2205,22 +2592,27 @@ function leadBreach(plan, cli, observations, expected) { (e) => e.kind === `blocked`, ).length const provisional = cli.lead.filter((line) => - line.startsWith(`Provisional: `), + line.startsWith(`oracle-gaps: note provisional: `), ) if ( provisional.length !== (expected.blocking.length > 0 ? 1 : 0) || (provisional[0] !== undefined && !provisional[0].startsWith( - `Provisional: ${counted(blockedCount, `mutant`)} `, + `oracle-gaps: note provisional: ${counted(blockedCount, `mutant`)} `, )) ) return breach( `the stdout does not say that the report is provisional, with ${blockedCount} blocked mutants`, ) - if (cli.lastLine !== `completed: ${counted(expected.gaps.length, `gap`)}`) + const done = doneLine( + expected.gaps.length, + expected.blocking.length > 0, + cli.outDir, + ) + if (cli.lastLine !== done) return violation( `cli`, - `the last stdout line is ${JSON.stringify(cli.lastLine)}, not the completion with ${expected.gaps.length} gaps`, + `the last stdout line is ${JSON.stringify(cli.lastLine)}, not ${JSON.stringify(done)}`, `baseline lead`, ) // The failing owner's replay runs exactly its failing test (law 3). @@ -2299,9 +2691,20 @@ const calibrations = [ { name: `a refused run is reported as complete`, law: `ownership`, - corrupt: corruptCli((cli) => - cli.exit === 0 ? undefined : { ...cli, exit: 0 }, - ), + // A refused run that says it completed: exit 0, and the completed done + // line that ends both its streams (law 15's shape of a completion), so + // only the law that the run must refuse rejects it. + corrupt: corruptCli((cli) => { + if (cli.exit === 0) return undefined + const done = `oracle-gaps: done completed, 0 gaps, report ${cli.outDir}/summary.md` + const completed = (stream) => ({ ...stream, last: done, dones: 1 }) + return { + ...cli, + exit: 0, + stream: completed(cli.stream), + stdoutStream: completed(cli.stdoutStream), + } + }), }, { name: `a mutant is dropped from the enumeration`, @@ -2465,6 +2868,100 @@ calibrations.push( } }), }, + // Law 15: a line without the stream's prefix, the decisive baseline line + // after a mutant's line, a completed run that ends with the old + // completion line, and a stopped run that ends with no done line. + { + name: `a streamed line has no prefix`, + law: `greppable stream`, + corrupt: corruptCli((cli) => ({ + ...cli, + stream: { ...cli.stream, strays: [`judged 25 of 100 mutants`] }, + })), + }, + { + name: `the decisive baseline line comes after a mutant's line`, + law: `greppable stream`, + corrupt: corruptCli((cli) => + cli.exit !== 0 + ? undefined + : { ...cli, stream: { ...cli.stream, mutantBeforeVerdict: true } }, + ), + }, + { + name: `a completed run ends without its done line`, + law: `baseline lead`, + corrupt: corruptCli((cli) => + cli.exit !== 0 ? undefined : { ...cli, lastLine: `completed: 3 gaps` }, + ), + }, + { + name: `a raw line reaches the stdout`, + law: `greppable stream`, + corrupt: corruptCli((cli) => ({ + ...cli, + stdoutStream: { + ...cli.stdoutStream, + strays: [`raw stdout from a test`], + }, + })), + }, + { + name: `a run ends with two done lines`, + law: `greppable stream`, + corrupt: corruptCli((cli) => ({ + ...cli, + stream: { ...cli.stream, dones: 2 }, + })), + }, + { + name: `a refusal through the daemon drops the cold run's notes`, + law: `greppable stream`, + // A config the session cannot load, checked cold and through the + // daemon: its notes are Vite's. + history: { constructed: `a declaration fault: configThrows` }, + corrupt: (trace) => { + const index = trace.findIndex( + (o) => + o.checkpoint === `incremental` && + o.value.cold.exit !== 0 && + o.value.cold.notes.length > 0, + ) + if (index < 0) return undefined + const value = structuredClone(trace[index].value) + value.daemon.stream.notes = [] + return trace.map((o, i) => (i === index ? { ...o, value } : o)) + }, + }, + { + // A note whenever file a loads, whether or not a test wrote (re-review + // N2). + name: `a run names a test-output log it never captured`, + law: `greppable stream`, + history: { constructed: `a load failure beside a hang` }, + corrupt: corruptCli((cli) => + cli.exit !== 0 || cli.captured + ? undefined + : { + ...cli, + stream: { + ...cli.stream, + notes: [ + ...cli.stream.notes, + `oracle-gaps: note test output in //test-output.log`, + ], + }, + }, + ), + }, + { + name: `a stopped run ends without its done line`, + law: `greppable stream`, + corrupt: corruptStopped((value) => ({ + ...value, + stream: { ...value.stream, last: `` }, + })), + }, ) // Law 9's states: reuse, a mix of reuse and recomputation, reuse from disk @@ -2586,6 +3083,11 @@ const required = [ `a heartbeat named a busy mutant`, `a run stopped by SIGTERM while a worker spun`, `a run stopped by SIGINT while a worker spun`, + `a stop signal after the completed line`, + `a stop signal while the config warms`, + `a run whose process aborted`, + `a config refusal through the daemon`, + `a completed run whose tests wrote nothing`, `a stopped run whose workers are child processes`, `a stopped run whose workers are threads`, `a run after a stopped one that left its session`, @@ -2623,6 +3125,8 @@ const required = [ `refused brokenVitest`, `refused configThrows`, `refused configNoDefault`, + `refused configNull`, + `refused configSyntax`, `blocked by setupThrows`, `a partial run of one function`, `a partial run beside a failing owner`, @@ -2649,6 +3153,15 @@ const configFaults = [ `projects`, `configThrows`, `configNoDefault`, + `configNull`, + `configSyntax`, +] +// The config faults whose refusal is also checked through the daemon. +const daemonConfigFaults = [ + `configThrows`, + `configNoDefault`, + `configNull`, + `configSyntax`, ] const faults = [ `badEntry`, @@ -2660,6 +3173,8 @@ const faults = [ `brokenVitest`, `configThrows`, `configNoDefault`, + `configNull`, + `configSyntax`, `setupThrows`, ...analyzedFaults, ] @@ -2671,6 +3186,8 @@ const separate = [ `a setup error beside a timeout under one budget`, `stale authored mutant`, `ambiguous authored mutant`, + `a config refusal through the daemon`, + `a completed run whose tests wrote nothing`, ...faults .filter( (fault) => @@ -2729,8 +3246,8 @@ const withDeclaration = (constructed) => oracleProperty({ id: `oracle-gaps.report`, law: { - statement: `The gap report enumerates the promised operator mutants, leaves out exactly the probe-only code, discovers every oracle that owns a function's call tree and refuses broken ownership, judges each mutant only by its owners' tests on their registered routes and counts only law verdicts on reproducible routes as kills, which outrank a run that never ends, anchors each mutant by its function and normalized statement, lists every gap and unowned mutant once with consistent counts, keeps keys stable under unrelated edits, reports through one daemon per package what a cold run reports at every state of a sequence of edits, and bounds the runs that confirm a mutant's cut-off copies by one budget per mutant and names each busy mutant in a heartbeat while it runs, leaves nothing in the package and writes nothing where Vitest's include could find it while it runs, leaves no process running once it is stopped, runs the package as Vitest does from any install path and any --source spelling: one instance of each workspace dependency, no refusal over an unbuildable devDependency, child processes for a native addon anywhere in the tree, a capture that a test's builtin spies cannot see, and a refusal of test.projects; leads its report with every test file's baseline, blocks only the mutants a failing oracle file owns and judges the rest, and every run that does not complete ends, cold or through the daemon, with a refusal that exits 2, prints a refused: block and one fix: directive with no stack trace, and writes the refusal to the run's output directory.`, - authority: `The maintainer's decisions in design runs v11, v14, and v19 (2026-11-04 and 2026-11-05), not in this package's records, including that a kill by reach agreement alone is a gap and that an incremental run through the daemon equals a cold one (v19.3); the project's oracle guide on quiet exclusions; the oracle rubric's outcome kinds (ORC-006); the CLI's documented operator list in schemata.mjs; the maintainer's dispositions G1, G2, G4, G5, G6, and G13 of the framework papercut ledger (2026-11-11) and the v20 freeze's namespacing, one-loop, and provisional-report decisions (2026-11-10).`, + statement: `The gap report enumerates the promised operator mutants, leaves out exactly the probe-only code, discovers every oracle that owns a function's call tree and refuses broken ownership, judges each mutant only by its owners' tests on their registered routes and counts only law verdicts on reproducible routes as kills, which outrank a run that never ends, anchors each mutant by its function and normalized statement, lists every gap and unowned mutant once with consistent counts, keeps keys stable under unrelated edits, reports through one daemon per package what a cold run reports at every state of a sequence of edits, and bounds the runs that confirm a mutant's cut-off copies by one budget per mutant and names each busy mutant in a heartbeat while it runs, leaves nothing in the package and writes nothing where Vitest's include could find it while it runs, leaves no process running once it is stopped, runs the package as Vitest does from any install path and any --source spelling: one instance of each workspace dependency, no refusal over an unbuildable devDependency, child processes for a native addon anywhere in the tree, a capture that a test's builtin spies cannot see, and a refusal of test.projects; leads its report with every test file's baseline, blocks only the mutants a failing oracle file owns and judges the rest, prints every line of its stream with the prefix oracle-gaps: and its kind, one decisive baseline line before any mutant, and a done line last, and every run that does not complete ends, cold or through the daemon, with a refusal that exits 2, prints a refused: block and one fix: directive with no stack trace, and writes the refusal to the run's output directory.`, + authority: `The maintainer's decisions in design runs v11, v14, and v19 (2026-11-04 and 2026-11-05), not in this package's records, including that a kill by reach agreement alone is a gap and that an incremental run through the daemon equals a cold one (v19.3); the project's oracle guide on quiet exclusions; the oracle rubric's outcome kinds (ORC-006); the CLI's documented operator list in schemata.mjs; the maintainer's dispositions G1, G2, G4, G5, G6, and G13 of the framework papercut ledger (2026-11-11) and the v20 freeze's namespacing, one-loop, and provisional-report decisions (2026-11-10); the maintainer's request for a greppable stream (2026-11-12, relayed by the coordinator with its decisions of that day: the same prefixes on stdout and stderr, a done line for a stopped run, and no --max-gaps).`, limits: `Law 12 judges refusals only: no legal history reaches an internal error, so its shape (exit 3, the stack in error.log) is not judged; argument refusals are the static oracle's; the check's rejection of a stack trace, a missing fix, and a wrong exit code is calibrated in the static oracle, through the same refusalBreach, and here only a refused run reported as complete is calibrated, because each calibration here runs real CLI histories. Hangs only as an owner test that awaits a promise that never settles, under a 1 s first budget and a 20 s full limit, and a slow run only as two witnesses whose tests wait where the output changed, 3 s in one that then judges and 10 s in two one-test owners of one that does not; busy loops only on instrumented code (cut by the call budget), no slow runs past the full limit that would end, hangs in ordinary or nondeterministic tests, setup errors other than a throw at module load, loops, multi-line conditions, two functions on one line, or await, new, and void statements; nested functions, object-literal methods, accessors, and class-field arrows only where no test calls them (their ownership is the static oracle's); two source files of one package, with no re-exports, namespace imports, or calls into another package; --only only in a stopped run and a partial run of one function's mutants, and no --sample or --diff-base; the baseline's failures only as one failing owner test that throws an error and a setup file that throws, so a baseline law violation's law and counterexample lines are the classify oracle's; a derived class's file is instrumented and its sites are judged as unowned, but no test imports or constructs the class, so its no-mutant and super-region behavior is the static oracle's (in-process), not this oracle's; package configs only as vite.config or vitest.config with one of the extensions .ts, .mts, .cts, .js, .mjs, and .cjs (not every pair), as config/gap-vitest with the same extension through --config, or none, as an object or a sync or async function, with no plugins of their own and test.projects only as a refusal, and only under the package's own package.json; a top-level await only in an ES module config and require only in a CommonJS one, as Vitest itself loads no other; a --config outside the package is refused by design, through the rule for paths outside the package (args.mjs), and no history generates one; contexts only one, four, or eleven, and process warnings judged only as MaxListenersExceededWarning on the CLI's own stderr (a daemon's process warnings go to its log); a context that fails to open gives back its listener allowance, which no observation shows (a kept allowance only raises the limit, so no warning follows), so that accounting is fixed by inspection; no probe-bound states, so reach agreement's killing of probe mutants is not checked; no gap answers or gate; law 9 judges the witness module's outcome classes along one constructed edit sequence only (a test edit, a helper edit, and a revert), and the budget witness's along one constructed second check with nothing changed, because a generated history with the witness module or the budget witness takes no steps (maintainer decision, 2026-11-11, and the runtime review's P1-a, 2026-11-12; this oracle owns the gap); law 9's edits come only between checks, so a file saved during a check is not generated (the runner invalidates a known input that changed while a run ran and does not store that run, which only inspection fixes; a witness needs a step that edits a file between a unit's start and its end), and a module that a run first loads and that changes during that same run is stored under its later hash; law 9's edits leave out files read through fs, other environment variables, new files that change how an unchanged import resolves, Vite config and installed-package changes, and concurrent checks other than two that start the daemon; cut-off charging judged only under one budget; native addons only as a binding.gyp or a prebuilt .node file, not as optional platform packages; daemon races (two CLIs reading one error file, a CLI hearing closing from the failing daemon it started, a reset connection on macOS) fixed by inspection; the runner's copies of the test files exist only in memory, and its harness and Vitest configs sit in the package's .oracle-gaps-, which the next run removes once a killed run has left it (judged after two stopped runs, not after a SIGKILL); a default --out is judged by the static oracle, and these runs name --out in the package; the namespace is checked after each run and, in a stopped run, while the run runs; a slow kill in a later owner, after earlier owners' confirmations spent the budget, becomes a timeout (owner order; not generated); the Vitest 5 path of the copies' pragma read and of the runner is verified from Vitest 5's source only, since the Vitest 5 CI job runs no gap oracle; a batch whose run reports an error outside any test reruns each copy alone (unitsBatch, runAlone) with its first budget, and a copy cut there is kept as a timeout with no confirmation, a defect older than this oracle's budget work that no history reaches (no history has an error outside any test beside a slow copy), recorded here with that trace for the maintainer; stale-session removal is scoped to this host's session directories (a host file), and is not judged across pid namespaces; the stop law covers SIGTERM and SIGINT sent to the CLI, not a SIGKILL of the CLI (its group then ends at the next check of its parent, by inspection) or a signal sent to the daemon, and its witness is one test that spins synchronously and starts one child process; the mutant budget is witnessed as two slow owners whose confirmations pass the --mutant-limit floor, not through its multiple of the owners' unmutated time, and authored mutants have no confirmations to budget; the witness w_slowpair is a wall-clock window, since the CLI's budget has no controlled clock and a fixture cannot see the time the CLI measures for its file: with one-test owner files p and q of measured unmutated time t each (a file first in its context pays the instrumented transform too), a wait W of 10 s, a 20 s --hang-limit and a 15 s --mutant-limit, it tells a budget from none only while (a) its first attempt is cut, W > 1 s + 3t, (b) its two confirmations do not fit the budget, 2W > max(15 s, 10 x 2t), and (c) each would fit its own full limit, W < 20 s + 3t; (b) is the binding one, so it holds while the two files' time stays under about 2 s together (about 1 s each, some 20 times the authoring machine's idle and 5 to 10 times its loaded measurements), and under one budget a run's overhead must stay under 10 s + 3t; CI runners measure 1.5 to 2.3 times the idle machine; beyond that w_slowpair would read not-judged; a heartbeat is owed only where a slow test waits 3 s against a 2 s interval, twice the first budget, so that no copy cut at its first budget beats, and each beat is observed as its mutant and file, not how long or how often it beat; a config only in the package's parent directory is generated one level up, in a temporary directory with no package.json (so its .js or .ts name is CommonJS), as a plain config, one that throws, one with no default export, or one that defines test.projects; a config two or more levels up, or a parent config beside a package config, is not generated, and the order in one directory follows Vitest's configFiles list by inspection; a package with no config lives in a temporary directory, where no parent has a config; a counterexample shrinks through at most six histories, so it can be larger than the smallest that fails; module-level data only as four \`const\` declarations of the main source (an object literal, an array that holds a function, a boolean, and an object of two methods), read in place, with no data imported from another file, destructured, reassigned, or declared as an enum or a default export.`, }, histories: { @@ -2781,9 +3298,14 @@ oracleProperty({ // Stopped while the stop witness spins, once by each signal and // in each pool (law 13). stopRuns: [ - { signal: `SIGTERM`, pool: `forks` }, - { signal: `SIGINT`, pool: `threads` }, + { signal: `SIGTERM`, pool: `forks`, when: `run` }, + { signal: `SIGINT`, pool: `threads`, when: `run` }, + { signal: `SIGINT`, pool: `threads`, when: `after` }, + { signal: `SIGINT`, pool: `threads`, when: `warm` }, + { signal: `SIGINT`, pool: `threads`, when: `abort` }, ], + // File a writes to the console, so the run captures it. + consoleOutput: true, }), }, { @@ -2826,9 +3348,12 @@ oracleProperty({ enters: [ `a setup error`, `a setup error beside a timeout under one budget`, + `a completed run whose tests wrote nothing`, ], build: (generated) => ({ ...generated, + // No test writes to the console: no log, and no note. + consoleOutput: false, functions: [ [ { kind: `push`, quote: `'` }, @@ -3194,6 +3719,11 @@ oracleProperty({ `a check after the session's failure was fixed`, ] : []), + // A config the session cannot load is refused through the + // daemon with the cold run's lines (law 15; review F4). + ...(daemonConfigFaults.includes(fault) + ? [`a config refusal through the daemon`] + : []), ], // A session that cannot open is checked through the daemon too, and // fixed once; a long failure is heard by two checks at once, then by @@ -3213,6 +3743,9 @@ oracleProperty({ witness: false, } : {}), + ...(daemonConfigFaults.includes(fault) + ? { steps: [`recheck`], witness: false } + : {}), }), })), { diff --git a/packages/oracles/oracle-gaps/gap-report-static-oracle.property.test.mjs b/packages/oracles/oracle-gaps/gap-report-static-oracle.property.test.mjs index bd2214a..d2ef2d0 100644 --- a/packages/oracles/oracle-gaps/gap-report-static-oracle.property.test.mjs +++ b/packages/oracles/oracle-gaps/gap-report-static-oracle.property.test.mjs @@ -489,8 +489,12 @@ function argumentsRunOf(dir, plan) { encoding: `utf8`, timeout: 120_000, }) - return endingOf(run.status, run.stderr, out, (text) => - text.replaceAll(dir, ``), + return endingOf( + run.status, + run.stderr, + out, + (text) => text.replaceAll(dir, ``), + run.stdout, ) } @@ -1230,6 +1234,11 @@ const calibrations = [ law: `report-shaped ending`, corrupt: corruptArguments(() => ({ fix: undefined, cause: undefined })), }, + { + name: `a refusal ends without its done line`, + law: `report-shaped ending`, + corrupt: corruptArguments(() => ({ done: undefined })), + }, { name: `a refusal's fix offers a way around it`, law: `report-shaped ending`, @@ -1411,8 +1420,8 @@ const required = [ oracleProperty({ id: `oracle-gaps.report-static`, law: { - statement: `The gap report's static half keeps every Svelte rune where Svelte allows it, gives each run's default output a directory of its own under its package's state, so runs of one package and of two packages never collide, enumerates the promised operator mutants, marks exactly the probe-only code, refuses broken owns declarations, reads no owns from an assertion's arguments (an expect chain, Vitest's assert or chai, or expectTypeOf), gives each function the owners whose call trees contain it, anchors each mutant's key, keeps keys, probe marks, and ownership stable under unrelated edits, and keeps each derived class's \`super(...)\` call unconditional. In a derived constructor, no mutant site lies before the first statement that calls super or inside that statement (an arrow inherits super, so an arrow's call counts), and statements after it keep their sites. So no mutant, when active, prevents or delays the super call. With no mutant active the instrumented class behaves as its source does, and under every mutant the constructor completes with its base class's tag unless an early return ends it first. Bad arguments (a missing --package or --source, a --package that does not exist, a source outside the package, a bad --env-prefix, --seed, count, or limit, an unknown --pool, a --mutants directory that does not exist, an option with no value, an unknown option, one long past a pipe's buffer included, a --source that is missing or is a directory, a --config that is missing or is a directory, and a --mutants file that is not JSON, lacks a field, or repeats an id) refuse the real CLI, cold or with --daemon, with exit 2 and a refused: block that names the cause, then one fix: directive that offers no way around the fix, and no stack trace; with --out, the refusal replaces an earlier run's results.json there and removes its error.log.`, - authority: `The maintainer's gap-check decisions in design runs v11, v14, and v19 (2026-11-04 and 2026-11-05), as for oracle-gaps.report; the frozen design's plan for an oracle over the checker's pure core (v19 P10, F4); the maintainer's few-seconds budget for an incremental test run (2026-11-05); for rune placement, Svelte 5's placement rule for runes (state_invalid_placement, effect_invalid_placement) and the maintainer's report of svelte-db's useLiveQuery.svelte.ts failing to compile under instrumentation (2026-11-08); for derived classes, the gap report's contract that instrumentation with no mutant active does not change behavior and that a mutant is a legal program variant (the operator contract in schemata.mjs), ECMAScript's rule that a derived constructor calls \`super\` before it reads \`this\` and before it returns, the maintainer's report of TanStack DB's subscription.ts failing with no mutant active (G11, 2026-11-09), and the maintainer's decision of 2026-11-09 on the derived constructor's site region (every shape of super, including an early return before it, is covered by the one rule); for the report-shaped ending, the maintainer's decision of 2026-11-10 ("we should never just crash -- the cli should also return something sensible") and the usage text in args.mjs, the CLI's documented interface.`, + statement: `The gap report's static half keeps every Svelte rune where Svelte allows it, gives each run's default output a directory of its own under its package's state, so runs of one package and of two packages never collide, enumerates the promised operator mutants, marks exactly the probe-only code, refuses broken owns declarations, reads no owns from an assertion's arguments (an expect chain, Vitest's assert or chai, or expectTypeOf), gives each function the owners whose call trees contain it, anchors each mutant's key, keeps keys, probe marks, and ownership stable under unrelated edits, and keeps each derived class's \`super(...)\` call unconditional. In a derived constructor, no mutant site lies before the first statement that calls super or inside that statement (an arrow inherits super, so an arrow's call counts), and statements after it keep their sites. So no mutant, when active, prevents or delays the super call. With no mutant active the instrumented class behaves as its source does, and under every mutant the constructor completes with its base class's tag unless an early return ends it first. Bad arguments (a missing --package or --source, a --package that does not exist, a source outside the package, a bad --env-prefix, --seed, count, or limit, an unknown --pool, a --mutants directory that does not exist, an option with no value, an unknown option, one long past a pipe's buffer included, a --source that is missing or is a directory, a --config that is missing or is a directory, and a --mutants file that is not JSON, lacks a field, or repeats an id) refuse the real CLI, cold or with --daemon, with exit 2 and a refused: block that names the cause, then one fix: directive that offers no way around the fix, and no stack trace, and a last line oracle-gaps: done refused, which names the refusal's summary.md with --out; with --out, the refusal replaces an earlier run's results.json there and removes its error.log.`, + authority: `The maintainer's gap-check decisions in design runs v11, v14, and v19 (2026-11-04 and 2026-11-05), as for oracle-gaps.report; the frozen design's plan for an oracle over the checker's pure core (v19 P10, F4); the maintainer's few-seconds budget for an incremental test run (2026-11-05); for rune placement, Svelte 5's placement rule for runes (state_invalid_placement, effect_invalid_placement) and the maintainer's report of svelte-db's useLiveQuery.svelte.ts failing to compile under instrumentation (2026-11-08); for derived classes, the gap report's contract that instrumentation with no mutant active does not change behavior and that a mutant is a legal program variant (the operator contract in schemata.mjs), ECMAScript's rule that a derived constructor calls \`super\` before it reads \`this\` and before it returns, the maintainer's report of TanStack DB's subscription.ts failing with no mutant active (G11, 2026-11-09), and the maintainer's decision of 2026-11-09 on the derived constructor's site region (every shape of super, including an early return before it, is covered by the one rule); for the report-shaped ending, the maintainer's decision of 2026-11-10 ("we should never just crash -- the cli should also return something sensible") and the usage text in args.mjs, the CLI's documented interface; for the done line, the maintainer's request for a greppable stream (2026-11-12, relayed by the coordinator).`, limits: `The grammar and model of oracle-gaps.report, with its limits. Outcomes that need test runs, and the failing-owner refusal, belong to oracle-gaps.report. Same-named functions, accessors, and field arrows are judged for ownership only; runes only as $state, $state.raw, $derived, $derived.by, $effect, and $effect.pre in a .svelte.ts module, judged by a restatement of Svelte's placement rule rather than the compiler. A derived class is judged in this oracle's in-process run and in the end-to-end oracle's static half: the end-to-end oracle instruments its file and judges its sites as unowned, but no test imports or constructs the class, so no test runs it. The in-process run transpiles the file to CommonJS at ES2022; other downlevel targets and Vitest's own transform are not exercised. The mutant switch (switchFor) is a hand copy of the switch's contract in schemata.mjs, not the capture setup's runtime: the capture setup imports Vitest and installs timers and globals when it loads, so it cannot run in-process. The derived grammar has one derived class per history, with parameter properties, a field, statements before and after super, an early return before it, and the four forms of the super call (a statement, an arrow argument, and the right operand of \`&&\` or \`||\`). It has no super call in a method, a nested class, or a \`super.method()\` call, and no default parameter. Module-level data is four \`const\` declarations of the main source (an object literal, an array that holds \`devBuild\` as a value, a boolean, and an object of two methods), and a top-level loop of the \`..gen/\` file whose variable is no unit, each read in place by generated or witness functions or named by an owns entry; data imported from another file, \`let\` data, enums, and default exports are not generated. A destructured declaration (\`const { a } = obj\`) stays \`\`, and a function made by a call (\`const h = memo(() => …)\`) now belongs to the unit \`h\`; neither is generated. Unnamed exports are seven files of a helper and one exported function each (an ES default function, a default arrow bare and in parentheses, CommonJS \`module.exports\` alone and chained through \`exports\`, \`exports.default\`, and \`exports.run\`), each calling its helper, owned through an owns entry that names them or through an importer that imports and calls each but \`exports.default\` (a default import of a CommonJS module resolves to the module, not its \`default\`, so a reference to \`exports.default\` through an import is not judged). TypeScript's \`export = \` is named \`default\` by the same rule but is not generated, nor is \`export default class {}\` or \`module.exports = { ... }\` (whose methods are named by their properties). A mutant of these forms was keyed \`\` before and is keyed \`default\` (or its export's name) now: a one-time change of saved keys. A method that module data holds references nothing, so whether a method's references stay out of the declaration's tree is not judged. The ending law is judged here only for argument refusals, one fault per history; refusals from the analysis and the runner, and the daemon's own path, are the end-to-end oracle's. No legal history reaches an internal error, so its shape (exit 3, the stack in error.log) is not judged; a calibration shows that the check rejects a stack trace on stderr.`, }, histories: { diff --git a/packages/oracles/oracle-gaps/index.mjs b/packages/oracles/oracle-gaps/index.mjs index 7aca2dc..d2639fe 100755 --- a/packages/oracles/oracle-gaps/index.mjs +++ b/packages/oracles/oracle-gaps/index.mjs @@ -105,9 +105,12 @@ // the config it loaded, or says it found none) // oracle-gaps --stop-daemon --package // oracle-gaps --help (prints the usage text, args.mjs) +// Every line the CLI writes for a check starts with `oracle-gaps: ` and its +// kind (stream.mjs), and the last is `oracle-gaps: done `. // Exit codes: 0 the check completed, whatever its gaps (the last stdout -// line is `completed: gaps`), 2 refused, 3 an internal error, and 130 -// or 143 when SIGINT or SIGTERM stopped it. +// line is `oracle-gaps: done completed, gaps, report `), 2 +// refused, 3 an internal error, and 130 or 143 when SIGINT or SIGTERM +// stopped it. // // No process outlives the CLI (maintainer disposition G2, 2026-11-11). The // CLI starts a check as a child in a process group of its own, which every @@ -147,38 +150,107 @@ function signalGroup(group, signal) { * with 128 plus the signal's number. */ function superviseRun() { - // An inspector flag would bind the CLI's own port a second time. - const child = spawn( - process.execPath, - [ - ...process.execArgv.filter((flag) => !/^--inspect/.test(flag)), - ...process.argv.slice(1), - ], - { - detached: true, - stdio: `inherit`, - env: { ...process.env, ORACLE_GAPS_GROUP: String(process.pid) }, - }, - ) + let child let stopped + // How the run ended: its exit code, or the signal that ended it. + let ended + // Whether the run said that it writes its ending (a message it sends + // before it writes): then its done line is the last on each stream, and + // it ends with its own code, whatever stop reaches it. + let ending = false + // Whether the run's stderr, which the CLI passes on, said that V8 ran + // out of memory. + let outOfMemory = false + let stderrTail = `` + // Whether the run's streams and channel have closed. + let closed = false + // Ends the CLI once the run has ended (or the grace has passed), with + // what is left of the group killed, and once the run's stderr has been + // passed on (at most 2 s more). A run that writes its ending wrote its + // done line, so the CLI ends with its code and writes none. Every other + // end gets one done line from the CLI, on both streams: a run that a stop + // ended stopped (review of #46, F3; re-review N1); a run whose process + // ended on its own, by a signal or a code, with no ending (a V8 + // out-of-memory abort is a SIGABRT) is an error, exit 3, with the remedy + // when it ran out of memory (the coordinator's decision of 2026-11-12 on + // P10). A write to a pipe is asynchronous on macOS, so the process exits + // once it is written. + let finishing = false + const finish = () => { + if (finishing) return + finishing = true + if (child) signalGroup(child.pid, `SIGKILL`) + let decided = false + const decide = () => { + if (decided) return + decided = true + if (ending) + return process.exit( + ended?.code ?? 128 + constants.signals[ended?.signal ?? stopped], + ) + const how = ended?.signal ?? (ended ? `code ${ended.code}` : `no exit`) + const line = stopped + ? `oracle-gaps: done stopped (${stopped})\n` + : `oracle-gaps: done error, the run's process exited with ${how}${outOfMemory ? `: it ran out of memory; give Node more (NODE_OPTIONS=--max-old-space-size=) or run fewer --contexts` : ``}\n` + process.stdout.write(line, () => + process.stderr.write(line, () => + process.exit(stopped ? 128 + constants.signals[stopped] : 3), + ), + ) + } + if (child === undefined || closed) return decide() + child.once(`close`, decide) + setTimeout(decide, 2000).unref() + } const stop = (signal) => { if (stopped) return stopped = signal + if (child === undefined) return finish() signalGroup(child.pid, signal) const deadline = Date.now() + STOP_GRACE_MS const poll = setInterval(() => { - if (signalGroup(child.pid, 0) && Date.now() < deadline) return + if ( + (ended === undefined || signalGroup(child.pid, 0)) && + Date.now() < deadline + ) + return clearInterval(poll) - signalGroup(child.pid, `SIGKILL`) - process.exit(128 + constants.signals[signal]) + finish() }, 50) } + // The stop handlers come first, so a signal that comes while the run + // starts is a stop too (re-review N3). process.on(`SIGINT`, stop) process.on(`SIGTERM`, stop) + // An inspector flag would bind the CLI's own port a second time. + child = spawn( + process.execPath, + [ + ...process.execArgv.filter((flag) => !/^--inspect/.test(flag)), + ...process.argv.slice(1), + ], + { + detached: true, + stdio: [`inherit`, `inherit`, `pipe`, `ipc`], + env: { ...process.env, ORACLE_GAPS_GROUP: String(process.pid) }, + }, + ) + child.on(`message`, (message) => { + if (message === `ending`) ending = true + }) + // The run's stderr, passed on as it comes, and read for V8's + // out-of-memory message. + child.stderr.on(`data`, (chunk) => { + process.stderr.write(chunk) + stderrTail = (stderrTail + chunk).slice(-200) + if (/heap out of memory/i.test(stderrTail)) outOfMemory = true + }) + child.on(`close`, () => { + closed = true + }) child.on(`exit`, (code, signal) => { - if (stopped) return - signalGroup(child.pid, `SIGKILL`) - process.exit(code ?? 128 + constants.signals[signal]) + ended = { code, signal } + if (!stopped) finish() }) } @@ -252,17 +324,40 @@ else if (main) { // Ends the run once. A write to a pipe is asynchronous on macOS, and // exiting at once cuts it off at the pipe's buffer, so the process exits // once both streams have written everything before it. + // A test-only hook for the CLI's own oracle, not for users: + // ORACLE_GAPS_TEST_HOLD_EXIT_MS holds the exit after the done line, so the + // CLI's oracle can send a signal in that window. const end = (exitCode) => { if (ended) return ended = true + const hold = Number(process.env.ORACLE_GAPS_TEST_HOLD_EXIT_MS ?? 0) process.stdout.write(``, () => - process.stderr.write(``, () => process.exit(exitCode)), + process.stderr.write(``, () => + setTimeout(() => process.exit(exitCode), hold), + ), ) } + // A stop ends the run at once, with no line: an error that the stop + // causes (a config load that it cuts off) is no refusal, and the + // supervisor writes the stopped line (re-review N1). Once the run writes + // its ending, it says so to the supervisor, and a stop no longer cuts it + // off: its done line is then the last line of each stream. + let writingEnding = false + const onStop = (signal) => { + if (!writingEnding) process.exit(128 + constants.signals[signal]) + } + process.on(`SIGINT`, onStop) + process.on(`SIGTERM`, onStop) + const holdStops = () => { + writingEnding = true + process.send?.(`ending`) + } const fail = (error) => { if (ended) return const ending = endRun(error, options?.out) + holdStops() process.stderr.write(ending.stderr) + process.stdout.write(`${ending.done}\n`) end(ending.exitCode) } // An error thrown outside the awaited check, by a timer or a promise no @@ -274,11 +369,14 @@ else if (main) { if (options.help) console.log(usage) else if (options.stop) await stopDaemon(options.pkg) else { - // The summary ends with the output directory and the completion - // line (report.mjs). - console.log( - await (options.daemon ? checkThroughDaemon : checkCold)(options), + const summary = await (options.daemon ? checkThroughDaemon : checkCold)( + options, ) + holdStops() + // The summary ends with the done line (report.mjs), which ends stderr + // too. + console.log(summary) + process.stderr.write(`${summary.split(`\n`).at(-1)}\n`) } end(0) } catch (error) { diff --git a/packages/oracles/oracle-gaps/report.mjs b/packages/oracles/oracle-gaps/report.mjs index 7a7d1dd..97d89db 100644 --- a/packages/oracles/oracle-gaps/report.mjs +++ b/packages/oracles/oracle-gaps/report.mjs @@ -27,6 +27,8 @@ import { escapeControls, gapLine, kindOf, + streamLine, + verdictLine, } from './stream.mjs' /** Outcomes that are gaps. Only a law verdict on a reproducible route kills. */ @@ -212,7 +214,11 @@ function baselineSection(results, baseline, config) { const blocking = blockedLines(baseline) const blocked = gapsOf(results).filter((r) => r.outcome === `blocked`).length return { - lines: [...baselineLines(baseline, config), ...blocking], + lines: [ + ...baselineLines(baseline, config), + verdictLine(baseline), + ...blocking, + ], provisional: blocking.length === 0 ? undefined @@ -402,43 +408,61 @@ export function renderStdout({ const relative = (file) => file.startsWith(`${pkg}/`) ? file.slice(pkg.length + 1) : file const section = baselineSection(results, baseline, config) + // Every line is a line of the stream (stream.mjs): the totals, one + // directive per gap class, each function's heading over its gap lines, + // and notes; summary.md holds the report for reading. const lines = [ ...section.lines, - ...(section.provisional ? [section.provisional] : []), - ``, - `Oracle gap check ${run.id}: ${results.length} mutants, ${killed.length} killed, ${gaps.length} gaps, ${unowned.length} unowned (warnings). ${run.inputs}.`, - ``, - `Legend, one directive per class:`, + ...(section.provisional + ? [ + streamLine( + `note`, + `provisional: ${section.provisional.replace(/^Provisional: /, ``)}`, + ), + ] + : []), + streamLine( + `summary`, + `${run.id}: ${results.length} mutants, ${killed.length} killed, ${gaps.length} gaps, ${unowned.length} unowned (warnings). ${run.inputs}.`, + ), ] for (const kind of new Set(gaps.map(kindOf))) - lines.push(` ${kind}: ${legendOf(kind)}`) - lines.push(``, `Gaps, by function:`) - if (gaps.length === 0) lines.push(` none`) + lines.push(streamLine(`legend`, `${kind}: ${legendOf(kind)}`)) for (const [group, rows] of groupBy(gaps, groupOf)) { const owners = [ ...new Set(rows.flatMap((g) => g.oracleFiles ?? []).map(relative)), ].sort() lines.push( - `${group}${owners.length > 0 ? ` (owners: ${owners.join(`, `)})` : ``}`, + streamLine( + `function`, + `${group}${owners.length > 0 ? ` (owners: ${owners.join(`, `)})` : ``}`, + ), ) - for (const gap of rows) lines.push(` ${gapLine(gap)}`) - } - if (unowned.length > 0) { - lines.push(``, `Unowned warnings, by function:`) - for (const [group, rows] of groupBy(unowned, groupOf)) - lines.push( - ` unowned ${group}: ${rows.length} ${rows.length === 1 ? `mutant` : `mutants`}`, - ) + for (const gap of rows) lines.push(gapLine(gap)) } + for (const [group, rows] of groupBy(unowned, groupOf)) + lines.push( + streamLine( + `note`, + `unowned ${group}: ${rows.length} ${rows.length === 1 ? `mutant` : `mutants`}`, + ), + ) if (excluded.length > 0) lines.push( - ``, - `Left out, failing with no mutant active: ${excluded.join(`, `)}.`, + streamLine( + `note`, + `left out, failing with no mutant active: ${excluded.join(`, `)}`, + ), ) - lines.push(``, `Full report: ${paths.report}`, `Results: ${paths.results}`) - if (paths.out !== undefined) lines.push(`output: ${paths.out}`) - // The last line says the check ran to its end, and how many gaps it - // found (G6): exit 0 means the run completed, not that it found none. - lines.push(`completed: ${counted(gaps.length, `gap`)}`) + lines.push(streamLine(`note`, `results ${paths.results}`)) + // The last line says the check ran to its end, how many gaps it found, + // and where its full report is (G6, and the stream's done line): exit 0 + // means the run completed, not that it found none. + lines.push( + streamLine( + `done`, + `completed, ${counted(gaps.length, `gap`)}${section.provisional ? ` (provisional)` : ``}, report ${paths.report}`, + ), + ) return lines.join(`\n`) } diff --git a/packages/oracles/oracle-gaps/stream.mjs b/packages/oracles/oracle-gaps/stream.mjs index 8171445..ed78725 100644 --- a/packages/oracles/oracle-gaps/stream.mjs +++ b/packages/oracles/oracle-gaps/stream.mjs @@ -5,6 +5,16 @@ // // The change a mutant makes is shown as its code, fenced so that a backtick // in the code cannot close the fence. +// +// Every line the CLI writes for a check starts with `oracle-gaps: ` and +// then its kind: `config`, `note`, `baseline`, `blocked`, `progress`, `gap`, +// `busy`, or `done` (the maintainer's request of 2026-11-12: an agent acts +// on the results as they stream, and one `grep '^oracle-gaps: '` finds +// them, on stdout and stderr alike). Only the indented lines under a +// baseline failure, and a refusal's `refused:` and `fix:` block, are not. + +/** A line of the stream: its kind after the prefix, then its text. */ +export const streamLine = (kind, text) => `oracle-gaps: ${kind} ${text}` /** The gap's kind: its survivor class, or its outcome. */ export const kindOf = (result) => result.class ?? result.outcome @@ -42,12 +52,18 @@ export function change(result) { * and again in the summary. */ export function gapLine(result) { - return `${kindOf(result)} L${result.line ?? `?`} ${result.id} ${change(result)}` + return streamLine( + `gap`, + `${kindOf(result)} L${result.line ?? `?`} ${result.id} ${change(result)}`, + ) } /** The periodic count while mutants are judged. */ export function progressLine({ judged, total, gaps }) { - return `judged ${judged} of ${total} mutants: ${gaps} gaps so far, ${judged - gaps} not gaps` + return streamLine( + `progress`, + `judged ${judged} of ${total} mutants: ${gaps} gaps so far, ${judged - gaps} not gaps`, + ) } /** A test's names, from Vitest's TestCase: its suites', then its own. */ @@ -135,18 +151,27 @@ export function baselineLines(files, config) { const skipped = tests.length - ran.length if (failure === undefined) return [ - `pass ${name}: ${counted(ran.length, `test`)}${skipped > 0 ? `, ${skipped} skipped` : ``}`, + streamLine( + `baseline`, + `pass ${name}: ${counted(ran.length, `test`)}${skipped > 0 ? `, ${skipped} skipped` : ``}`, + ), ] const failed = tests.filter((t) => t.state === `failed`) const first = failed[0] if (first === undefined) return [ - `fail ${name}: ${tests.length === 0 ? `did not load` : `failed outside its tests`}: ${codeSpan(escapeControls(failure.split(`\n`)[0]))}`, + streamLine( + `baseline`, + `fail ${name}: ${tests.length === 0 ? `did not load` : `failed outside its tests`}: ${codeSpan(escapeControls(failure.split(`\n`)[0]))}`, + ), ` replay: ${vitest} ${shellWord(file)}`, ] const message = first.message ?? `` return [ - `fail ${name}: ${failed.length} of ${counted(ran.length, `test`)} failed`, + streamLine( + `baseline`, + `fail ${name}: ${failed.length} of ${counted(ran.length, `test`)} failed`, + ), ` failing: ${codeSpan(escapeControls((first.names ?? [first.fullName]).join(` > `)))}: ${codeSpan(escapeControls(message.split(`\n`)[0]))}`, ...violationLines(message), ` replay: ${vitest} ${shellWord(file)}${first.names === undefined ? `` : ` -t ${shellWord(escapeControls(replayPattern(first.names)))}`}`, @@ -154,6 +179,22 @@ export function baselineLines(files, config) { }) } +/** + * The one decisive baseline line, before any mutant runs: every oracle file + * passed, or how many of them failed, so the report is provisional. + */ +export function verdictLine(files) { + const oracle = files.filter((f) => !f.ordinary) + const failed = oracle.filter((f) => f.failure !== undefined).length + const all = counted(oracle.length, `oracle file`) + return streamLine( + `baseline`, + failed === 0 + ? `all ${all} pass` + : `${failed} of ${all} failed (provisional)`, + ) +} + /** * One line per oracle file that failed with no mutant active (G13): the * file, its failing test, and the step that unblocks the mutants it owns. @@ -167,7 +208,10 @@ export function blockedLines(files) { ? `` : ` at ${codeSpan(escapeControls((first.names ?? [first.fullName]).join(` > `)))}` return [ - `blocked: ${codeSpan(file)} fails with no mutant active${at}, so the mutants it owns wait for it: make it pass (its replay is above), then check again.`, + streamLine( + `blocked`, + `${codeSpan(file)} fails with no mutant active${at}, so the mutants it owns wait for it: make it pass (its replay is above), then check again.`, + ), ] }) } diff --git a/packages/oracles/scripts/e2e-shards.mjs b/packages/oracles/scripts/e2e-shards.mjs index e445bd0..986c404 100644 --- a/packages/oracles/scripts/e2e-shards.mjs +++ b/packages/oracles/scripts/e2e-shards.mjs @@ -85,6 +85,8 @@ export const shards = { `calibration: a refused run is reported as complete`, `determinism: constructed history: a .cjs config that calls require`, `determinism: constructed history: a declaration fault: configNoDefault`, + `determinism: constructed history: a declaration fault: configNull`, + `determinism: constructed history: a declaration fault: configSyntax`, `determinism: constructed history: a declaration fault: outsideSource`, `determinism: constructed history: a declaration fault: setupThrows`, `determinism: constructed history: a load failure beside a hang`, @@ -140,11 +142,15 @@ export const shards = { `fixed seed 20261105: a declaration fault: badEntry`, `fixed seed 20261105: a declaration fault: brokenVitest`, `fixed seed 20261105: a declaration fault: configNoDefault`, + `fixed seed 20261105: a declaration fault: configNull`, + `fixed seed 20261105: a declaration fault: configSyntax`, `fixed seed 20261105: a declaration fault: nonLiteral`, `fixed seed 20261105: a failing owner under a config that --config names`, `fixed seed 20261105: a load failure beside a hang`, `random seed: a .cjs config that calls require`, `random seed: a declaration fault: configNoDefault`, + `random seed: a declaration fault: configNull`, + `random seed: a declaration fault: configSyntax`, `random seed: an awaiting config in a package with no module type`, `calibration: production code is reported as probe code`, `calibration: the daemon reports another outcome than the cold run`, @@ -157,6 +163,14 @@ export const shards = { `fixed seed 20261105: a second daemon check of the budget witness`, `calibration: a stopped run leaves a process it started running`, `calibration: a busy mutant is named in no heartbeat`, + // Law 15's calibrations (the greppable stream), measured at ab18c60: + // 79 s and 70 s; the daemon-notes one, bound to its history, 2 s (it + // took 433 s searching for one). + `calibration: a streamed line has no prefix`, + `calibration: a refusal through the daemon drops the cold run's notes`, + // Bound to its history (re-review N2), not yet measured. + `calibration: a run names a test-output log it never captured`, + `calibration: a completed run ends without its done line`, `determinism: constructed history: a second daemon check of the budget witness`, // The witness module's edit sequence (option M): 97 s here, measured // with the daemon fix of fix/daemon-current-inputs. @@ -188,6 +202,12 @@ export const shards = { `random seed: a second daemon check of the budget witness`, `calibration: a stopped run had written a copy beside the tests`, `calibration: a heartbeat names a file that does not run its mutant`, + // Law 15's calibrations (the greppable stream), measured at ab18c60: + // 69 s, 58 s, 73 s, and 69 s. + `calibration: the decisive baseline line comes after a mutant's line`, + `calibration: a stopped run ends without its done line`, + `calibration: a raw line reaches the stdout`, + `calibration: a run ends with two done lines`, // The witness module's edit sequence (option M): about 97 s here, measured // with the daemon fix of fix/daemon-current-inputs. `random seed: an edit sequence over the witness module`,