feat(oracle-gaps): every streamed line starts with oracle-gaps: <kind>

.changeset/gap-cli-baseline-report.md

@@ -5,8 +5,8 @@
 The `oracle-gaps` report leads with the baseline, and a failing oracle file no longer stops the run.
 
 - An oracle file that fails with no mutant active (a RED law, or a file that cannot load or launch) blocks only the mutants it owns. A mutant it owns that no passing owner kills is a new `blocked` gap, which names the failing file. The other mutants are judged, and the report says it is provisional. Before, the run refused with exit 2. A setup file that throws now blocks every owned mutant in the same way.
-- A `blocked:` line names each failing oracle file and its failing test, and says to make it pass and check again.
+- A `blocked` line names each failing oracle file and its failing test, and says to make it pass and check again.
 - With `--only`, the baseline runs only the owners of the selected mutants, so another file's failure no longer blocks the run.
 - stdout and `summary.md` start with the baseline: each test file's result with its test counts, ordinary test files included, and for a failing file its failing test, its error, the law, difference, and counterexample of a law violation, and its replay command. Ordinary test files now run once in the baseline; they run against mutants only with `--ordinary`. `results.json` gains `provisional` and `baseline`.
 - The legend says that a `crashed` gap is a gap, not a kill, and that its error is on the gap's Detail line in `summary.md` and in its `detail` in `results.json`.
-- The last stdout line is `completed: <n> gaps`, after `output: <dir>`. Exit codes do not change: 0 means the check completed, whatever its gaps.
+- The last stdout line says that the check completed, with its gap count and its report (`oracle-gaps: done completed, <n> gaps, report <path>`). Exit codes do not change: 0 means the check completed, whatever its gaps.

.changeset/gap-cli-greppable-lines.md

@@ -0,0 +1,18 @@
+---
+'@tanstack/oracles': minor
+---
+
+Every line the `oracle-gaps` CLI writes for a check now starts with `oracle-gaps: ` and its kind, on stdout and stderr alike, cold and through `--daemon`, so an agent can act on the results as they stream with `grep '^oracle-gaps: '` and filter one kind by the word after the prefix:
+
+- `oracle-gaps: config <file>`, or `oracle-gaps: config none, Vitest's default discovery`
+- `oracle-gaps: baseline pass <file>: <n> tests` and `oracle-gaps: baseline fail <file>: …`, with the failing test, law, counterexample, and replay indented under it
+- one decisive line before any mutant runs: `oracle-gaps: baseline all <n> oracle files pass`, or `oracle-gaps: baseline <k> of <n> oracle files failed (provisional)`
+- `oracle-gaps: blocked <file> …` for each failing oracle file
+- `oracle-gaps: gap <kind> L<line> <id> <change>`, streamed and in the stdout summary
+- `oracle-gaps: progress …`, `oracle-gaps: busy <mutant id> in <file> for <n>s`, and `oracle-gaps: note …` (Vite's own messages as `note vite: …`)
+- the stdout summary as lines too: `oracle-gaps: summary …` for the totals, `oracle-gaps: legend <class>: …` for each directive, and `oracle-gaps: function …` for each function's heading over its gap lines
+- a last line for every run: `oracle-gaps: done completed, <n> gaps, report <path>` (with `(provisional)` after the count when an oracle file failed), `oracle-gaps: done refused[, report <path>]` after a refusal's `refused:` and `fix:` block, `oracle-gaps: done error, log <path>`, or `oracle-gaps: done stopped (<signal>)`
+
+The done line is the last line of both stdout and stderr, once each, and its status matches the exit code. Only the lines indented under a failing baseline file and a refusal's `refused:` and `fix:` block have no prefix, and no line is blank. A test's own writes to the process's stdout and stderr go to `test-output.log` in the output directory, which a note names. The stdout summary's `output: <dir>`, `Full report:`, and `completed: <n> gaps` lines are replaced by the done line, and the refusal's `output: <dir>` line by `oracle-gaps: done refused, report <path>`. Exit codes do not change.
+
+The wrapper config now reads the package's config as Vite does: it awaits a promise default and refuses a default that is not an object. A config with a syntax error is refused with its first error, before Vite bundles it.

.changeset/gap-cli-never-crash.md

@@ -2,6 +2,6 @@
 '@tanstack/oracles': minor
 ---
 
-The `oracle-gaps` CLI no longer ends a run with a bare stack trace. A problem in the author's own files or arguments (bad arguments, a `--source` that is missing or not a file, a broken `owns` declaration, no oracle file or no owner, a Vitest config that throws when it loads, has no default export, or defines `test.projects`, a Vitest that does not load, a setup file that throws, an oracle file that fails with no mutant active, an authored mutant file that is not JSON or is incomplete) refuses the run: it exits 2 and prints `refused: <cause>` and `fix: <one step>`. When the output directory is known (even from `--out` alone, when other arguments are bad), the refusal is also written there as `results.json` (`{ "status": "refused", "cause", "directive" }`) and `summary.md`, replacing an earlier run's ending there, and the CLI prints `output: <dir>`. Any other error is a bug in the CLI: it exits 3, says so, and writes its stack to `error.log` in the output directory (or the system's temporary directory when none is known), with `results.json` status `internal-error`. A check through `--daemon` ends the same way, and now also prints `output: <dir>` when it completes. A daemon serves only a CLI of its own code, so a second install of the CLI starts its own daemon rather than reading another version's reply.
+The `oracle-gaps` CLI no longer ends a run with a bare stack trace. A problem in the author's own files or arguments (bad arguments, a `--source` that is missing or not a file, a broken `owns` declaration, no oracle file or no owner, a Vitest config that throws when it loads, has no default export, or defines `test.projects`, a Vitest that does not load, a setup file that throws, an oracle file that fails with no mutant active, an authored mutant file that is not JSON or is incomplete) refuses the run: it exits 2 and prints `refused: <cause>` and `fix: <one step>`. When the output directory is known (even from `--out` alone, when other arguments are bad), the refusal is also written there as `results.json` (`{ "status": "refused", "cause", "directive" }`) and `summary.md`, replacing an earlier run's ending there, and the CLI's last line names that report (`oracle-gaps: done refused, report <path>`). Any other error is a bug in the CLI: it exits 3, says so, and writes its stack to `error.log` in the output directory (or the system's temporary directory when none is known), with `results.json` status `internal-error`. A check through `--daemon` ends the same way. A daemon serves only a CLI of its own code, so a second install of the CLI starts its own daemon rather than reading another version's reply.
 
 Exit code 1 is no longer used: scripts that treated any non-zero exit as a failure keep working, and scripts that matched exit 1 must match 2 or 3. Arguments are now checked: an unknown option, an option with no value, a `--package` or `--mutants` directory that does not exist, a `--seed` that is not an integer, a count or limit that is not a positive whole number, and a `--pool` other than `threads` or `forks` each refuse the run instead of being ignored or failing later.

README.md

@@ -29,7 +29,7 @@ every commit with [pkg.pr.new](https://pkg.pr.new).
   up leaves that directory, and the next run removes it.
 
   While one copy of a test file runs for 30 s or more, the CLI prints
-  `busy: <mutant id> in <test file> for <n>s` every 30 s. A run of many fast
+  `oracle-gaps: busy <mutant id> in <test file> for <n>s` every 30 s. A run of many fast
   copies prints none, and a check through `--daemon` prints its heartbeats
   only when the check ends. The runs that confirm a mutant's cut-off copies
   share one budget: the larger of `--mutant-limit` (30 s by default) and ten
@@ -46,18 +46,67 @@ every commit with [pkg.pr.new](https://pkg.pr.new).
   failing file its failing test, its error, the law and counterexample of a
   law violation, and a `replay:` command. An oracle file that fails with no
   mutant active blocks only the mutants it owns: each that no passing owner
-  kills is a `blocked` gap, a `blocked:` line names the file and its failing
+  kills is a `blocked` gap, a `blocked` line names the file and its failing
   test, the rest are judged, and the report says it is provisional. With
   `--only <id,...>`, the baseline runs only the owners of the selected
   mutants. A `crashed` gap is not a kill: production threw where no law
   judged it, and the error is on the gap's Detail line in `summary.md` and
   in its `detail` in `results.json`. Record the error in the driver, and
   state the law that decides that input's outcome.
 
-  Exit codes: 0 the check completed, whatever its gaps (the last stdout line
-  is `completed: <n> gaps`); 2 the run was refused, with `refused:` and
-  `fix:` on stderr; 3 an internal error of the CLI; 130 or 143 stopped by
-  SIGINT or SIGTERM.
+  Every line the CLI writes for a check, on stdout and stderr alike, starts
+  with `oracle-gaps: ` and its kind, so an agent can act on the results as
+  they stream with `grep '^oracle-gaps: '`, and filter one kind:
+
+  | Kind                                     | Line                                                                                                                                                                                                                                                        |
+  | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+  | `config`                                 | `oracle-gaps: config vitest.config.ts`, or `oracle-gaps: config none, Vitest's default discovery`                                                                                                                                                           |
+  | `baseline`                               | `oracle-gaps: baseline pass tests/a.test.ts: 3 tests`, `oracle-gaps: baseline fail tests/b.test.ts: 1 of 2 tests failed`                                                                                                                                    |
+  | `baseline` (decisive, before any mutant) | `oracle-gaps: baseline all 4 oracle files pass`, or `oracle-gaps: baseline 1 of 4 oracle files failed (provisional)`                                                                                                                                        |
+  | `blocked`                                | `oracle-gaps: blocked <file> fails with no mutant active at <test>, so the mutants it owns wait for it: …`                                                                                                                                                  |
+  | `gap`                                    | `oracle-gaps: gap <kind> L<line> <id> <change>`                                                                                                                                                                                                             |
+  | `progress`                               | `oracle-gaps: progress judged 25 of 120 mutants: 7 gaps so far, 18 not gaps`                                                                                                                                                                                |
+  | `busy`                                   | `oracle-gaps: busy <mutant id or baseline> in <test file> for <n>s`                                                                                                                                                                                         |
+  | `note`                                   | `oracle-gaps: note …`: the `--hang-limit` note, Vite's own messages (`note vite: …`), ordinary files left out, the provisional count, unowned counts, the results' path, and where the tests' console output went (`note test output in <path>`)            |
+  | `summary`                                | `oracle-gaps: summary <run id>: 120 mutants, 98 killed, 12 gaps, 10 unowned (warnings). …`                                                                                                                                                                  |
+  | `legend`                                 | `oracle-gaps: legend <class>: <what it means and what to do>`, once for each class of gap the run found                                                                                                                                                     |
+  | `function`                               | `oracle-gaps: function fn <name> (<source>) (owners: <files>)`, the heading over that function's gap lines                                                                                                                                                  |
+  | `done` (last)                            | `oracle-gaps: done completed, 12 gaps, report <path>/summary.md` (`(provisional)` after the count when an oracle file failed), `oracle-gaps: done refused[, report <path>]`, `oracle-gaps: done error, log <path>`, or `oracle-gaps: done stopped (SIGINT)` |
+
+  Only two kinds of line have no prefix: the lines indented under a failing
+  baseline file (`failing:`, `law:`, `counterexample:`, `replay:`), and a
+  refusal's `refused:` and `fix:` block on stderr. No line is blank. The
+  done line is the last line of both stdout and stderr, once each, for
+  every ending, and its status matches the exit code. A test's own output
+  to the process's stdout and stderr goes to `test-output.log` in the output
+  directory, which a note names; `summary.md` holds the report for reading.
+  A run through the daemon prints the same lines, at the end of its check.
+  `--help` is not a check, so its usage text has no prefix.
+
+  For contributors: two environment variables in the `ORACLE_GAPS_TEST_`
+  namespace are test-only hooks for the CLI's own oracle, not settings for
+  users. `ORACLE_GAPS_TEST_HOLD_EXIT_MS` holds the run's exit after its done
+  line, and `ORACLE_GAPS_TEST_WARM_PROBE` makes the first config warm-up
+  wait where a stop signal can reach it.
+
+  A few endings print no done line, or one more line: a SIGKILL of the CLI,
+  or a signal that comes while Node is still starting the CLI, before its
+  first line runs (a few tens of milliseconds), ends it with no done line.
+  When the run's own process ends without its ending, by a signal sent to
+  it rather than the CLI or by a crash (a V8 out-of-memory abort is a
+  SIGABRT), the CLI writes `oracle-gaps: done error, the run's process
+exited with <signal or code>` and exits 3; after an out-of-memory crash
+  the line says to give Node more memory (`NODE_OPTIONS=--max-old-space-size=<MB>`)
+  or to run fewer `--contexts`. An internal error's own message lines
+  (`oracle-gaps internal error: …`) have no prefix. A config's syntax error
+  is refused with its first error; a syntax error in a module the config
+  imports still lets esbuild print its own error block, which has no
+  prefix.
+
+  Exit codes: 0 the check completed, whatever its gaps (its last line is
+  `oracle-gaps: done completed, …`); 2 the run was refused, with `refused:`
+  and `fix:` on stderr; 3 an internal error of the CLI; 130 or 143 stopped
+  by SIGINT or SIGTERM.
 
   The check runs in a process group of its own, with every Vitest worker and
   every process that a test starts, except one started detached in a session

packages/oracles/oracle-gaps/args.mjs

@@ -52,10 +52,19 @@ The paths in --source and --config are relative to --package. Without
 --config, the run loads the Vitest config that Vitest discovers for the
 package: in its directory, or else in the nearest parent directory that
 has one. Run the CLI from any directory.
+Every line the CLI writes for a check starts with \`oracle-gaps: \` and its
+kind (config, note, baseline, blocked, progress, gap, busy, done), on
+stdout and stderr alike, so \`grep '^oracle-gaps: '\` finds them. Before any
+mutant runs, one line says whether the baseline passed:
+\`oracle-gaps: baseline all <n> oracle files pass\`, or
+\`oracle-gaps: baseline <k> of <n> oracle files failed (provisional)\`.
+The last line is \`oracle-gaps: done <status>\`.
 Exit codes: 0 the check completed, whatever its gaps: its last line is
-\`completed: <n> gaps\`; 2 refused, with the fix on stderr;
-3 an internal error of oracle-gaps; 130 or 143 stopped by SIGINT or
-SIGTERM, once every process the run started has ended.`
+\`oracle-gaps: done completed, <n> gaps, report <path>\`; 2 refused, with
+the fix on stderr and \`oracle-gaps: done refused\` last; 3 an internal
+error of oracle-gaps (\`oracle-gaps: done error, log <path>\`); 130 or 143
+stopped by SIGINT or SIGTERM (\`oracle-gaps: done stopped (<signal>)\`),
+once every process the run started has ended.`
 const help = `as \`oracle-gaps --help\` shows`
 /**
  * The options in `argv`. Paths are resolved against `cwd`, the caller's

packages/oracles/oracle-gaps/capture-setup.mjs

@@ -25,6 +25,7 @@
 // of shrinking it: a kill needs one failing history, not the smallest one.
 import { createHash } from 'node:crypto'
 import { appendFileSync, readFileSync } from 'node:fs'
+import { dirname, join } from 'node:path'
 import { afterAll, beforeEach, chai, expect, inject, vi } from 'vitest'
 
 const run =
@@ -151,6 +152,26 @@ if (run.units && !skippedCopy) {
   vi.resetModules()
 }
 const state = unit ? { ...run, ...unit } : run
+// A test's own writes to the process's stdout and stderr (a debugging
+// `process.stdout.write`) go to the run's test-output.log, beside its
+// capture, not to the CLI's streams, whose every line is a line of the
+// gap report's stream (review of #46, F2). The check names the log.
+if (state.capture)
+  globalThis.__oracleGapsOutput = join(
+    dirname(state.capture),
+    `test-output.log`,
+  )
+if (!globalThis.__oracleGapsStreams) {
+  globalThis.__oracleGapsStreams = true
+  for (const stream of [process.stdout, process.stderr])
+    stream.write = (chunk, encoding, callback) => {
+      const done = typeof encoding === `function` ? encoding : callback
+      const output = globalThis.__oracleGapsOutput
+      if (output) appendFileSync(output, chunk)
+      if (typeof done === `function`) done()
+      return true
+    }
+}
 // A batch's watchdog reads this row to know which copy is running. A copy
 // whose test file fails to load runs no `afterAll` and so writes no closing
 // row; the next copy's start ends its turn.