Repository navigation
docs(oracles): ship setup, design, and review skills, with a prompt to try them - #7
Open
KyleAMathews wants to merge 123 commits into
Open
KyleAMathews wants to merge 123 commits into
KyleAMathews wants to merge 123 commits into
Conversation
…checklist
The package now carries what an adopter's coding agent needs, inside
node_modules:
- skills/oracles-setup: install both pinned packages, configure lint,
write the AGENTS.md block, make the skills findable through repository
or global pointer skills, and run a first oracle and gap check.
- skills/oracles-design: law sheet first, model choice, a witness for
every distinction, histories and reach, driver and check, RED, GREEN,
hostile mutant, and closing every gap with no exits.
- skills/oracles-review: rebuild one observation, the shortcut table,
the weaknesses tools cannot see, and the finding format.
- docs/oracle-checklist.md: ORC-001 to ORC-016 (version 1), ported from
TanStack DB's rubric with the same IDs and a table of every change.
- docs/glossary.md, docs/guide.md, and five instrument cards.
- READMEs for both packages; `files` now includes skills and docs.
The skills teach `oracleProperty({ owns })` and no `law.owner`, so this
waits for those framework changes before it merges.
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
🦋 Changeset detectedLatest commit: 522aa90 The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
 |
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Title check |  |
This assessment has not completed. The check will be retried on the next pre-merge check run. |
-
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled.
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
commit: |
`oracles-design` step 3 named six model styles but showed no example of any but a pure function. docs/model-styles.md gives a small, generic, type-checked example of each: a pure function, a naive reference structure, a transition table with derived reach, a pure step function, a relation between runs with its calibration, and an allowed set. It also states two rules from the model prototypes: each model rule is one sentence of the law, and legality lives in a separate pure guard. Co-authored-by: Isaac <no-reply@databricks.com>
Oracle reviews were sometimes read as security work. The skills and docs now avoid words that suggest it: the adversarial-review card is the challenge-review card, attacks are challenges, a hostile mutant is a mutant (as TanStack DB's glossary already asks), a fault-injected step is a step that simulates a signal, and "bypass", "defeat", and "payload" are reworded. Links follow the renamed card. Co-authored-by: Isaac <no-reply@databricks.com>
Design step 2 now asks the agent to name the model's parts and rules, map each rule to a sentence of the law, rebuild the known examples before trusting the model, and record rule conflicts as open questions. docs/instruments/design-grammar.md carries the heavier method for a hard law or a growing model: a stated preservation relation, parts admitted only for a job, overlaps, reconstruction and simplification by a different representation, a two-way comparison for every simplification with refactor, refine, and revise kept apart, and an optional TLA+ check with a weakened-configuration control. The design and review skills and the card index route to it. Co-authored-by: Isaac <no-reply@databricks.com>
- New skill oracles-evaluate-review: a lossless ledger of review findings, enforcement gaps fixed and law changes brought to the user, one oracle plan for the whole review run against unchanged production, each bug turned into a family of histories with a nearby passing case, and a loss audit. - oracles-design: the approved law sheet becomes the law block at the top of the oracle file; a run-budget heuristic (critical states by construction and probes, starting runs, fast files, longer sweeps by the run multiplier, a lowered budget is a weakening); lint after each edit; a production cleanup step after GREEN; the evidence block goes in the pull request description. - oracles-review: findings for a lowered budget and for inserted rather than refactored code; a law block that disagrees with the law option. - oracles-setup: the AGENTS.md block names the new skill and asks for lint after each edit; pointer skills include the new skill. - Skill descriptions and a few sentences follow the STE structural rules. Co-authored-by: Isaac <no-reply@databricks.com>
#6 made `recommended` turn off inline directives and widened the `production` glob to `src/**`, so setup now adds both presets as they are, without the workaround blocks. The plugin README lists no-oracle-disable and the new glob. The design skill now asks for at least one fixed seed (an empty list will be refused) and treats a random-only kill as a gap until its history is kept. Co-authored-by: Isaac <no-reply@databricks.com>
Two fresh agents set up the package in vercel/ms (Jest and Biome, no ESLint) and developit/mitt (mocha, ESLint 7), using only the packed packages and the skills. mitt reached zero gaps. ms stopped correctly on a law that nobody can decide for a library the adopter does not own. Setup now covers what they hit: - the Node version against CI, the lint setup, and the command runner - `eslint@^9` when the repository has none, a stable parser major, and TypeScript 5 or newer for the oracle files, kept out of an older `tsc` - where oracle files go, and a Vitest config beside another runner - a separate ESLint 9 run for an ESLint 7 or 8 repository, with a check that the rules are live - `owns` entries that name one function - `--source` selecting files, an explicit `--seed`, `--out` inside the repository, and `.oracle-gaps/` in `.gitignore` - dead code that the gap check finds counts as a production change The design skill teaches function-level `owns`, dead-code removal as a production change, and what a law that kills nothing means. Co-authored-by: Isaac <no-reply@databricks.com>
- Evidence by kind of change: a defect fix shows RED, GREEN, and the defect as a mutant; a behavior-preserving refactor shows the same oracle passing on old and new production with kill sets compared; a new oracle over correct code shows production passing and calibrations failing. Review no longer treats a refactor without RED as a shortcut. - Approval follows the semantics: strengthening an adopted law reuses it without a new sheet; a law change needs a law delta; a new law or missing authority needs the full sheet; an approval the task already gave stands. - A boundary-selection step before the law sheet, with an example. - Model state is justified by a legal continuation (one or more actions), not only the next action, in the design skill, ORC-008, the guide, and the law-discovery card. - Dead code needs a contract reason before removal; a surviving mutant only shows what the current histories did not expose. - A survivor that the adopted law already allows is reported as an allowed alternative, not sent back as a law question. - Setup: pointers and the AGENTS.md block read the chosen package's installed copy in a workspace; the TypeScript parser covers production source as well as oracle files. - The round-trip example compares tags element by element and names the two corruptions the joined comparison accepted as calibrations. Co-authored-by: Isaac <no-reply@databricks.com>
The framework judges a mutant over the domain the operand types admit, so a narrower domain stated only in prose does not make a mutant invalid. Co-authored-by: Isaac <no-reply@databricks.com>
Four fresh agents ran the branch's skills on four workflow branches: a behavior-preserving refactor, a review with two findings, a planted defect, and a package-local workspace. Each took the right branch. Their findings, plus the framework session's answers: - Refactor evidence follows the gate's rule: oracle files unchanged or only strengthened and passing on old and new code with the same seed; shared killed keys still killed; new keys in changed functions killed or acted on. - Design: an allowed alternative is its own outcome; a gap that existed before a change still counts in a changed function; widening histories inside the stated domain is strengthening; a grammar that covers part of the law's domain says so; constructed histories must fit the arbitrary; the driver records production errors; own the whole file when an owned function reads module data; several oracles may own one function; public state needs a law or a limit; an exhaustiveness-guard survivor is a known CLI bug, recorded without stopping or removing the guard. - Review: a law-dependent state in `measured` is an unreachable claim. - Setup: name the oracle config `vitest.config.ts` (the gap check loads only that name), pnpm 11's allowBuilds placeholder, refresh pointers on upgrade, the AGENTS.md block in a package-local workspace, keep oracle files out of the repository's own lint or match its style, install before a pointer gives up, and a path for runs with no user. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
With #11 merged, the README sample type-checks and the gap check counts it as an oracle file. It still left one gap: `>` to `>=` differs at clamp(-0, 0), and the histories never generated -0. The example now covers its type's domain, as the design skill teaches: -0 is in the grammar, a constructed history enters "negative zero at max", and the gap check kills all three mutants. Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews
marked this pull request as ready for review
November 11, 2026 01:13
Three fresh agents adopted the package in p-limit (async JavaScript, xo, AVA), dequal (uvu with esm), and nanostores (TypeScript 7, oxlint). All three followed the method, and dequal's agent found real library defects. Their findings, plus the framework session's answers: - Setup: offer a separate Node 20 job when CI is older; use the repository's package manager for install commands; place oracle files where the existing runner will not load them (their own directory, or a `*.oracle.ts` name); probe-form for root-level source; xo repositories; the TypeScript 7 plugin bug as a temporary note; the live-rule check through stdin before any oracle file exists; choose a function with a documented promise; a first oracle that is RED found a defect (report it, never weaken the oracle, run the gap check on a scratch fix until the CLI judges a failing baseline); group the open gaps in the report. - Design: an analogy is not authority; laws people forget (no throw on valid input, the algebraic laws of the kind of function); constructed values derived from the generated history; whole-file entries that pull in another contract; owned code that serves another module; where `oracleProperty`'s options are typed; do not copy exit wording into the agent's own report; a synchronous-spin timeout is a CLI limitation. Co-authored-by: Isaac <no-reply@databricks.com>
…mmar
Agents go from examples to a general law only when made to write it
down, and no agent in three validation rounds opened the design-grammar
card, because it triggered on judgment ("a hard law").
- Design step 1: before the law sheet, each law gets a written block:
the examples with sources, what varies and what stays, the law as one
"For every ..." sentence with its edges, a new case it predicts, and
every example rebuilt from the sentence alone.
- Five concrete triggers send the agent to the design-grammar card; the
evidence says which fired.
- Review: "Law as examples" and "Skipped design grammar" findings.
- Evaluate-review: the causal condition is written as a sentence over
histories, with a second history it predicts.
- Setup: TypeScript 7 is an environment limit (typescript-eslint), and
the plugin's `unparsedModule` report for a `.js` oracle file that
imports TypeScript (#13).
- Design: the interim exit-wording bullet is gone with #12; the rule not
to write those words in a report stays.
Co-authored-by: Isaac <no-reply@databricks.com>
- Design step 1, item 6: sort each law into dynamics (how things change, stated by the model's rules), constraints (what holds in every history), and boundary conditions (the edges, where real behavior and most defects live). Every boundary gets a constructed history or an open question; it is never left out of the grammar. Every defect the validation agents found was at a boundary. - Adjacent forms keep design diversity: the design-grammar card's new step 8 sketches two to four forms that keep every adopted law, for the user to pick; design step 3 names a second fitting model style; design step 7 sketches alternative cleanup designs instead of keeping the first. - Glossary: dynamics, constraint, boundary condition, adjacent form. Co-authored-by: Isaac <no-reply@databricks.com>
The law sheet is now a template that the agent fills in for each law. Its examples, law sentence, new case, and dynamics, constraints, and boundary conditions are fields, and the summary table is only an index. In round 4, one of four agents skipped the written block because it was prose before a table. Other fixes from round 4 (validation findings V64 to V96): - design: arbitraries must take histories back (`.map` needs its inverse); string units; observations are data; brand checks, not instanceof; methods of a returned object are owned through the enclosing function; the whole-file entry only when module data affects a law; authored mutants when operators are sparse; triggers checked again after each gap check, counting every row. - setup: the no-user default covers setup questions only; the end condition is the first oracle's own contract; keep an existing Vitest include; register the plugin in a custom probe-form block; xo and TS 7 installs; the live-rule probe needs the fast-check import; placeholders on a stopped run; scratch copy wording. - review: the shortcut rows match the template and the later trigger check. Co-authored-by: Isaac <no-reply@databricks.com>
…Script The framework credits a kill to the first law that a history violates, so a later law can look idle. The design skill now says to aim hand mutants at that law before restating or deleting it. The setup skill reports a TypeScript copy that npm hoisted from @tanstack/oracles as a known packaging bug, so it is not taken as the repository's own TypeScript. Both are temporary: the framework fixes are queued. Co-authored-by: Isaac <no-reply@databricks.com>
The law block in an oracle file is now prose for a reader who has not seen the sheet: a question for a title, how the code works, the documented promises, how the model states them, and what the driver observes. The sheet's fields stay in the review record. The AGENTS.md block (template 2) says who owns what. The maintainers own the laws. The agent owns the production code that the oracles cover, and makes it the smallest, clearest code that passes them. Round-5 validation fixes: - A stated limit no longer closes a gap. Setup's end condition, its report groups, design step 8, and the review's exit row now say so. - A block whose domain is its own example is a fixed history, not a law. - The string-unit advice names `binary` (every code point except lone surrogates), since `grapheme` has no control characters. - A guard that TypeScript narrowing makes constant (the last member of an exhaustive union) is a known CLI bug. Other mutants on its line still count. - A `not-observed` survivor can need a history where the branches diverge. - The design-grammar trigger after a gap check is a stop, not a note. - An idle law that another law covers is merged. No empty tests in oracle files. `vitest.config.mts` in a CommonJS package. The AGENTS.md lint line when lint cannot run. Co-authored-by: Isaac <no-reply@databricks.com>
The law sheet is now the new oracle file before it has any code: an opening comment that states the laws as prose (ORC-003), and a review-notes comment with the per-law template that turns examples into a general law. After approval, the review notes are deleted, and the code is added under the prose. A law delta edits the prose in place. This removes the step that copied an approved sheet into the file. Co-authored-by: Isaac <no-reply@databricks.com>
… never by name A validation agent used pkill -f oracle-gaps to stop its own runs and killed other agents' runs on the same machine. Co-authored-by: Isaac <no-reply@databricks.com>
The ownership line now says: whenever you change production code, do a good job, not a lazy one, as an optimizing compiler for the oracle would; big refactors are fine when they make the code better; and don't go looking for things to optimize when the task is something else. Co-authored-by: Isaac <no-reply@databricks.com>
Improve the code the work touches, well; offer a larger win instead of turning the task into a long optimization project. Co-authored-by: Isaac <no-reply@databricks.com>
…ck (template 4) The always-loaded block now carries the working rules that blind probes and in-session baselines showed agents lack (Opus, Sonnet, Haiku, Codex; klona and quick-lru): probe first, propose laws then build only approved ones, red first then fix, never weaken or quarantine, the strongest model stating the user's rule rather than the code's algorithm, calibrations, run only through the gap check and read its output as it streams, stop runs by PID, no equivalent mutants, and classes of bugs with measured shares. Co-authored-by: Isaac <no-reply@databricks.com>
…the feedback file Show the evidence (a hand-applied copy that the oracle catches, or a mutant that never ran), and with the developer's agreement record it in oracles-feedback.md for the framework, then move on, without rewriting production or weakening the oracle. Co-authored-by: Isaac <no-reply@databricks.com>
…unners and lint autofix in problems.md From the r18 Codex run's feedback log: owns index.js#default was refused for `export default function pLimit` (a named default export is named pLimit); ava picked up *.oracle.test.js; an XO autofix changed a rejection observation. Co-authored-by: Isaac <no-reply@databricks.com>
…rea, any time The same tested workflow (probe, propose, build, fix oracle first, close gaps) now runs on a named area (`/oracles-explore the clearQueue API`) or the whole repository. The onboarding layer (the opening, a walk-through after the first fix, the setup choice) runs only in a first session: a repository with no oracle files yet, or one the developer says is new to them. The AGENTS.md block, setup, design, and the README prompt point to the new name. Co-authored-by: Isaac <no-reply@databricks.com>
… opt-in log, review subagents From the second usability test and the loss audit, checked against the design-grammar run 2 preservation list (21 items): - the first message comes before reading the AGENTS block, and says what will happen, what gets added, the milestones, a rough length, and that everything runs locally - questions only at decision stops; the probe plan is "what I'll find out next (no answer needed)", in statements - milestones marked with elapsed time, and a step-away note for long runs - findings said in what a caller sees, reproduced through the public API - what a pass means; an early red is often the oracle's own setup - the feedback log is opt-in; deleted files are named - after the first oracle, suggest oracles-review subagents, evaluated with oracles-evaluate-review; other spots offered as parallel sessions - mutant and mutation testing in the word table; recount the design trigger when the developer adds a law; fictional examples replace ms and mitt; problems.md covers resuming after an install and an oracle that throws Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…n true, walk-through and wrap-up tied to the milestones, fast-check in scratch probes Co-authored-by: Isaac <no-reply@databricks.com>
… in-house test helpers Co-authored-by: Isaac <no-reply@databricks.com>
… for a stop Co-authored-by: Isaac <no-reply@databricks.com>
… law that would hold it, or name its spot Co-authored-by: Isaac <no-reply@databricks.com>
…ine reason in terms of their design Co-authored-by: Isaac <no-reply@databricks.com>
…ed path under /tmp Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…ify it, not a gap-check bug Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…its one done line (#46) Co-authored-by: Isaac <no-reply@databricks.com>
…the work bound (#48) Co-authored-by: Isaac <no-reply@databricks.com>
… mutant gap search Codex's safety filter flags prompts that pile up words like hunt and attack around code probing. The skills used hunt for the mutant card and the review step; this renames them (and the card's file) with no change in meaning. Co-authored-by: Isaac <no-reply@databricks.com>
…ates while the oracle is written, and drop "laws people forget" From a usability session: - "The laws people forget" read oddly when the agent repeated it to the developer. The two common laws are now listed with the others. - After the findings, the agent says that probes only sample the design, and that the oracle, once it runs, explores the whole space and often finds more bugs or design questions. - Writing the oracle can take a while, so the agent sends a short update at each part, or every few minutes, on what it is building and learning. Co-authored-by: Isaac <no-reply@databricks.com>
…s changeset Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A developer can now paste one prompt into a coding agent and see what oracle testing finds in their own code. The agent probes the code first and shows the bugs it found. Then it proposes an oracle, builds it with the developer, and fixes each bug that the developer accepts, oracle first. The package ships five skills and the docs they cite inside
node_modules, so the instructions always match the installed version.Before this change, the method lived only in TanStack DB, in a 1,175-line rubric that agents did not read in full. Outside agents skipped the required reading, wrote code before they stated a law, and called surviving mutants "equivalent". These skills replace that rubric for adopters. They close the exits that agents took, and they keep the decisions that need a person with that person.
Try it
The package README now starts with this prompt. A developer pastes it into an agent at the root of their repository:
The prompt works only after this PR merges, because
maindoes not have the skills yet. Before the merge, replace<sha>with this branch's head.What the session looks like
oracles-exploreis the entry point. It works toward four milestones, and it tells the developer each time it reaches one, with the time since the start:The agent asks a question only where a decision is the developer's. Everything else is narration. With the first decision, it asks whether it may keep a short
oracles-feedback.mdof rough spots for the maintainers.Who owns what
The
AGENTS.mdblock thatoracles-setupwrites states the split. The maintainers own the laws: what the code promises, the authority for each promise, and its limits. The agent owns the production code that the oracles cover. It makes that code the smallest, clearest code that satisfies the oracles, and it restructures freely while they pass.oracles-exploreoracles-setupAGENTS.mdblock, makes the skills findable, and runs the gap check.oracles-designoracles-evaluate-revieworacles-reviewHow the skills keep the agent on the path
The skills make the right move the obvious one. They do not rely on the agent to judge when a rule applies.
What the validation runs found
Fresh agents (Opus, Haiku, and Codex) ran the skills on more than 12 open-source libraries over 19 rounds. Each agent got only the packed packages and one library it had not seen. Each round's findings changed the skills, or went to the framework as an issue. From round 19, each run records its clock time, its dollar cost, and the bug classes it proved.
DataViewcopy that loses its window, and an LRU cache whoseonEvictionreports a live key. Nobody filed these upstream yet.oracle-gaps: <kind>output stream (feat(oracle-gaps): every streamed line starts withoracle-gaps: <kind>#46), the extra-work class for shortcuts (fix(oracle-gaps): report a shortcut for speed as extra-work, and let a work bound kill it #48), and module-level data that an oracle can own (feat(oracles): module-level data is ownable; an owned function that reads it owns it #21).Work continues outside this PR on faster bug finding: cheap parallel workers that sweep the code before the oracle exists. That work does not change these skills yet.
The checklist and the docs
docs/oracle-checklist.mdholds ORC-001 to ORC-016, ported from TanStack DB's rubric as version 1. The IDs did not change, so existing citations still work. ORC-003 requires each oracle file to be a literate program. A table at the end lists every difference from the rubric and its reason.docs/intro.mdexplains what an oracle is, and how it differs from unit tests, property tests, fuzzing, and formal models.docs/glossary.mddefines the oracle terms.docs/model-styles.mdgives one small example, which passestsc, of each of six model styles.docs/guide.mdholds the rubric's technique sections, made generic. It adds no requirements.docs/instruments/holds six reasoning cards, such as the design grammar and the mutant gap search.Interim rules that remain
The design skill has two temporary rules for CLI limits. A mutant that makes production loop forever shows as a timeout, and the agent reports it as a known CLI limit. A gap on a guard that TypeScript narrowing makes constant is an invalid mutant. The agent records it with its line, and it keeps the guard, because the guard protects untyped callers.
Merge order
mainalready has everything that the skills teach, through affeb1a (feat(oracles): owns on oracleProperty; a run that judges nothing fails; namespaced output #11 to fix(oracle-gaps): deterministic stop sessions and stdout lead, a constructed failing owner, and a copy's random stream #54).docs/oracles-package-skills) bumps DB's pin to this commit and merges. It turns DB'soracle-reviewandoracle-authoringinto addenda that load these skills.Verification
oracle-gapscommand. The last two that Isaac installed were 136f02e and 30b2a93.tests/package.test.tsfails whennpm packleaves outskills/ordocs/. Prettier passes on every changed file.main. A rerun passed.Where to start reading
Start with
packages/oracles/skills/oracles-explore/SKILL.md, thenpackages/oracles/skills/oracles-design/SKILL.mdandpackages/oracles/docs/oracle-checklist.md. The setup skill and the package README show the API as an adopter uses it.This pull request and its description were written by Isaac.