Repository navigation
nostream-cli migrate pg-to-lmdb — resumable Postgres→LMDB migration with a checksum manifest and measured throughput #1115
Description
The remainder of #864, filed fresh
#864 was closed on 2026-11-08 when the LMDB backend slice landed (PR #1073, 68be6f3a). Its own
scope was four slices, and two of them were never issued: the Postgres → LMDB migration and the
benchmark rig. The board's ordering note still says #864 stays open until the store, NIP-50, the
migration and the rig all land, so this is that note's remainder rather than new work. The store
landed (job 024), LMDB NIP-50 is PR #1113, the rig is board row 1d.
This issue is the migration slice and nothing else.
What exists at rewrite/main today
- Both backends are complete in
nostream-storage.PostgresStore(src/postgres/, against the
37-migration schema) andLmdbStore(src/lmdb/, 30 databases) implement the same nine
repository traits (src/repositories.rs: events, invoices, users, NIP-05 verifications, invite
codes, DVM jobs, reports, notification delivery log, notification outbox). nostream-storage/src/lib.rsnames this tool as deliberately unowned: "the daemon's backend
factory and thenostream-cli migrate pg-to-lmdbtool" are outside the crate.- A batching precedent exists:
nostream-cli import(src/commands/import.rs) streams a file
throughEventImportServicein 1 000-event batches with per-line errors and live progress.
exportis still a stub (CliError::NotImplemented). crates/nostream-cli/src/commands/mod.rshas nomigratemodule.
Scope
nostream-cli migrate pg-to-lmdb, as board row 1c states it:
- resumable — an interrupted run continues rather than restarting, and what "continues" means
(a cursor, a manifest, a per-database completion mark) is the spec's decision, not this body's; - a checksum manifest — the run leaves a record that lets a second invocation, or a person,
decide whether the target holds what the source held; - throughput measured — the command reports events/second itself.
Reader and writer are the two existing backends, through the repository traits where they reach and
directly where they do not. No new storage shape, no schema change on either side: the LMDB layout is
ADR 0011's and the Postgres schema is the migrations'.
The ordering conflict, stated so the spec resolves it
Row 1c's acceptance criterion is "≥ 50k events/s", and row 1d is the rig that would measure it. The
rig does not exist. Do not invent a throughput number and do not put a criterion in the spec that
only the rig could settle. The delivery has to be a measurement the CLI reports itself — a rate
over an input the test can construct — and the spec says which measurement settles the criterion and
what fails at base.
Out of scope
- The benchmark rig against real data (row 1d, [Rewrite P6] Hardening: fuzz, benchmark gate, 72h soak, security review #868's gate reads it).
- NIP-50 index migration specifically: an LMDB store built by this tool gets its search index from
the index's own rebuild path once PR feat(storage): answer NIP-50 from an events_by_term index, parity-tested against the fixture #1113 lands, not from the migration. - The cutover itself ([Rewrite P7] Cutover: release, migration run, parallel run, TS retirement #869), which includes the real data run.
Done means
A migration that can be interrupted and resumed, leaves a manifest that can be checked, reports its
own throughput, and is covered by tests that fail on the base commit. The issue's acceptance criteria
are written in the job's spec.md; this body is the scope the spec is written against.
Activity
chappie-daemon commentedon Nov 12, 2026
Recon finished (job
029-pg-to-lmdb, factory 0.12.18) at baseb42f833.What it settled: the subcommand lands in
crates/nostream-cli/src/commands/migrate.rswith aCommandvariant and aCliErrorarm (main.rs:120-139is exhaustive over three variants);EventImportService's batching/progress shape is reusable but the service is not (it is file-shaped and re-validates signatures); both backends open independently and nothing in the workspace opens both today.What it found that changes the design: the traits cannot carry a faithful migration. No trait scans a table;
find_by_filtersexcludes soft-deleted and expired events (postgres/sql.rs:265); and every write stampsfirst_seen,created_at,balance, invite claims and report ids with "now", so a trait-only migration silently loses them. The spec therefore has to choose between a lossy trait-only port and a minimal migration seam innostream-storage.Resumability: LMDB gives
Database::len/iter/keyset and one write transaction per batch; Postgres orders stably onevents.event_id(unique, indexed) but not onfirst_seen(no index). The residual the spec must handle is that the manifest and the batch cannot share a transaction, so a replayed batch has to be harmless or detectable.Throughput:
≥ 50k events/s(row 1c) cannot be settled without row 1d's rig. The CLI can report its own rate over a constructed input, and the criterion will assert the reported rate is consistent withinserted / elapsedwith no floor — row 1d settles the 50k figure.Measured at base:
cargo build -p nostream-cli --all-features65 s;cargo test -p nostream-cli77 s, 18 tests;nostream-cli migrate pg-to-lmdbexits 2 withunrecognized subcommand; the proposed oraclecargo test -p nostream-cli --test migrate_cliexits 101 ("no test target namedmigrate_cli"). No Postgres is reachable from the build member, so the Postgres arm is#[ignore]d in the repo's existing convention and the deciding check needs no database.chappie-daemon commentedon Nov 12, 2026
Spec written (606 lines, digest
8ac627929a2a17d9), red-team running.Design it binds: D1 — a migration-only seam, not the repository traits. A new feature-free
nostream-storage::migrationmodule (MigratedTable,MigratedUser,MigrationRow,MigrationSourceonPostgresBackend,MigrationTargetonLmdbBackend), because no trait enumerates a table and every trait write stamps, clears or drops a column. Eight of nine LMDB per-table writers already exist; only the delivery log needs a new exact one.event_tags(the LMDBtagsindex is written from the event row) anduser_subscriptions(no LMDB database, no trait).events.id, the v4 uuid LMDB never stores.users.tos_accepted_atis preserved throughStoredUser, which the publicUseromits.<target>/migration-manifest.json,format: 1, per tablerows/checksum/after/complete, temp-and-rename after each batch. Checksum is a 256-bit multiset sum ofSHA-256(len ‖ key ‖ canonical)— order-free and composable across a resume, stated as a corruption check, not a commitment.Migrated {n} rows in {ms} msandEvents throughput: {rate} events/s, and the criterion assertsrate == n * 1000 / max(ms, 1)exactly with no floor —≥ 50 000 events/sis row 1c and is settled by row 1d's rig.15 acceptance criteria: 11 commands, 2
[judged](the Postgres arm,#[ignore]d in the repo's convention, judged on the PR's CI log; and the ADR0012-pg-to-lmdb-migration.mdcarrying D1–D12), 2[guard]disclosed as passing at base. Oraclecargo test -p nostream-cli --all-features --test migrate_cli, measured at base as exit 101 ("no test target namedmigrate_cli").chappie-daemon commentedon Nov 12, 2026
Spec red-teamed twice and frozen (digest
13d27a7d1f9dea11, 734 lines). Build round 1 is running.Pass 1 was run at base and found nine ways a broken implementation would still pass: a resume that restarts and recomputes produces a byte-identical manifest, an
Err-returning source is defeated by mapping errors toOk(vec![]), one checksum criterion changed two fields at once so dropping either passed, the throughput identity was self-referential, the deleted/expired tail indexes went unasserted, and an--exacttest name that matches nothing exits 0 with0 passed.Pass 2 found five ways a correct build would have failed: the
events_expiringfixture was unsatisfiable (a preserved 2030 expiry writes its own entry),claim_batchrewrites the row it returns so it is not a read-back, a resume assertion silently depended on--batch-size 1, a rationale was still false, and the canonical-projection criterion had no inequality. All five are fixed in the spec; the test seam the criteria need (migrate_with/verify_with) is now R14.Verdict count after pass 2: 12 criteria fail at base (proven, mostly the absent test targets), one claimed, and only the two disclosed guards pass at base. No unguarded criterion passes before the work.
Estimate re-cut at spec time to $4.00 (
estimates.mdhas no per-slice row; the #864 row is $5.00 for three slices): two modules, nine readers and nine writers, two new test targets, an ADR and a CI step extension.chappie-daemon commentedon Nov 12, 2026
Review round at the frozen head
8d3e4c9854840a39fce7216e9110482d462a3383: four lenses (code,security,ux,architecture) reading it, with static clippy clean at that head (ran, 0 findings) and fourteen static rounds recorded in the job. Thirteen rounds of fixes were applied by hand after the builder budget was spent; the tree is frozen for this read.Open at this stage:
#1119— theuserscolumn list is still restated betweenpostgres/users.rsand the migration seam reader; the only catching tests need a live Postgres.chappie-daemon commentedon Nov 12, 2026
PR open: #1125 —
603b477f6c440dd67e54fa5ca33e6d7bbadf7b21, againstrewrite/main, 40 commits, 33 files, +5362/−59.Job record: actual $6.40 vs expected $4.00 (+59.9%), measured by
factory-spans.ts(35 sessions, 13.26 h summed wall). The overrun is re-reads, not scope: six builder passes, the budget spent at 5/5, then every fix by hand — nineteen static rounds and five review rounds, each reading a tree that had changed since the last.Static at the PR head: clippy and greptile both clean (round 19). Reviews:
code,security,ux,architecture,performanceat a frozen head, then confirmation reads of each fix delta.Follow-ups filed from this job: #1119 (column list), #1121 (Unix gating), #1122 (cap test), #1124 (
ORDER BYon text ids), #1126 (refusal tightening).Limits stated in the PR: no end-to-end resume test without Postgres, a present-but-foreign store is trusted, the command's file handling is Unix-only, and the probe's ten-second cap is proven by inversion rather than by a test.
The merge is Cameri's.