Repository navigation
benchmark rig: the TS/Postgres baseline against LMDB, numbers in docs/rewrite/benchmarks.md #1131
Description
The last remaining slice of #864 (closed): the rig that produces docs/rewrite/benchmarks.md, the
numbers the cutover gate (#869) reads before it can retire the TypeScript implementation.
#864 held four slices. The store landed as job 024-lmdb-store (PR #1073), NIP-50 as job
028-nip50-fts (PR #1113, pr-open), and the migration as job 029-pg-to-lmdb (PR #1125). This
is the fourth: a measurement harness, not a feature.
What it must produce
- A reproducible rig that measures the LMDB backend and the TypeScript/Postgres implementation
on the same workload, so the two columns are comparable rather than evocative. docs/rewrite/benchmarks.mdwith the numbers, the workload that produced them, the machine and
the date — a document#869can cite and a later reader can re-run.- The workload definition in the repository, not in prose: what events, what mix of reads and
writes, what concurrency, and what is deliberately excluded.
What it must not do
- No number invented, extrapolated or copied from an earlier document. Where a measurement cannot
be taken in the environment available, the document says so and says why. - No threshold gate. The gate is [Rewrite P6] Hardening: fuzz, benchmark gate, 72h soak, security review #868's slice; this job produces evidence, not a pass/fail.
Detail — the workload, the harness shape, the environment limits and the acceptance criteria —
belongs in the job's spec.md. Filed as #864's remainder rather than reopening it, the same way
#1115 was filed for the migration slice.
Activity
chappie-daemon commentedon Nov 14, 2026
Opened as job
030-benchmark-rig(factory 0.12.18), cut fromrewrite/mainat509b403. Expected cost $2.40 from the measured units inestimates.md; recon starting now.chappie-daemon commentedon Nov 14, 2026
Scope decision at spec time, from recon (
.factory/jobs/030-benchmark-rig/recon.md#sizing-one-subsystem-or-two): this job builds the rig and measures both Rust backends through one harness — LMDB here, Postgres in CI wherechecks.ymlalready declarespostgres:16-alpine— because the trait seam makes them the same code and the same workload. The TypeScript relay column is split out to #1132:src/on this branch is 3.11.0 diverged fromorigin/main, and there is no Docker, nopsqland nonode_modulesin the member that runs the job.benchmarks.mdcarries that column asnot measuredwith the reason.chappie-daemon commentedon Nov 15, 2026
PR opened: #1141 (
3ad5686, baserewrite/main). Merge is Cameri's.Cost: $4.15 actual against $2.40 expected (+72.8%) — five review rounds, four builder resumes, one approver escalation and a five-anchor citation correction in the job record. The unit prices in
estimates.mdheld (28 sessions, $0.148 average, inside the builder band); what the estimate got wrong was the round count, because each round's fix introduced the next round's defect — an empty-range measurement, a store-reuse guard validating a path the run did not use, and an exclusion list with three false claims. Theregretline inruns.jsonlsays so.What shipped: the harness (
examples/bench.rs),--verifyasserting exact expected totals,--repeats Nwith per-shape bands,docs/rewrite/benchmarks.md+ the committed 50k/seed-7 artifact, two CI steps (benchmark unit tests; the Postgres arm), and the ADR 0012 amendment. The TypeScript column is #1132; three wording defects in D9 are #1138–#1140.chappie-daemon commentedon Nov 15, 2026
Post-PR rework complete; re-approved at
d808694. PR #1141, five automated-review findings fixed (one a P1), two stale document figures corrected.Final cost: $4.69 actual against $2.40 expected (+95%), 31 sessions. The operator's own words for the overrun, recorded in
regret: the cause is my tolerance, not your build — every extension was authorised after a named finding, and the pre-PR projection (+33%) was met before the automated reviewer found a P1 that five rounds and eighteen lens readings had passed. Unit prices never moved: $0.148 per session.What the rework changed:
--verifycomparedtiming.per_second.max(1)against the truth, so a shape whose integer rate rounded to 0 reported failure on a good measurement. It now assertsops > 0. The oracle has been wrong in both directions: too weak at round 1 (rows > 0passed acountshape that measured an empty range 49 times of 50), too strict here.--outrefused on its parent, so--out /tmppassed and failed after the whole run; now refused at the destination, probing without truncating an existing artifact.u64::MAXboundary test in both profiles.workload.readsgainedby_id_stride/by_id_offset(schema 2), so two by-id workloads cannot look alike.create_many26% faster, counts unchanged. The document quotes the new run; two figures that still came from the old one (48/24416 and a stale date) were caught by the gate lens and corrected.Approver's second pass, at the final head: recomputed AC15 independently (181 comparisons, 0 mismatches), re-ran AC4/AC5 and the Oracle (
--verifyexit 0),git merge-treeclean. Follow-ups: #1143 (the refusal list omits three refusals the harness makes) alongside #1132 and #1138–#1140.The process record — cost by stage, findings by instrument, and why a cold automated read beat eighteen briefed lens readings — is
.factory/jobs/030-benchmark-rig/process.md.