[chappie-daemon]

Closes nothing by keyword: base is rewrite/main, not the repository's default branch. #1109 is closed by hand after the merge.

On a limited search, this store ranks by (span, id) where Postgres keeps created_at DESC. The set is the capture's; the order is not, and it is the first thing to know about this change. Everything below is the record of why, what it cost, and what it does not do.

NIP-50 search was refused on the LMDB path — StorageError::NotImplemented { module: "storage::lmdb::nip50_search" } — so Filter::search was a field a client could send and a store could not answer. This adds a term index (events_by_term, the 30th named database, key term ‖ event_id, empty values), a tokeniser shared by the write and read paths, and a rank, so a searched REQ, COUNT and reconciliation all answer from the same walk.

What a reader should check first

The oracle is fixture entry 21, the search row the differential replay had been excluding. It is replayed now:

base b42f833, parity arm edited:  exit 101  op 21 (events.find_by_filters) failed:
                                            not implemented in Phase 1 (job 005): storage::lmdb::nip50_search
head 06301de:                     LMDB_PARITY_COMPLETE replayed=114 excluded=48,50,112,115   43 passed

Entry 21 asks search: "bitcoin lightning" and the fixture records four rows. The token rule (lowercase, then maximal alphanumeric runs, every term a run of content) selects exactly those four; the capture's in-memory substring rule selects a fifth (f8ef9793…, bitcoininveniice / #LightningNetwork). The capture disagrees with itself here — its SQL path uses to_tsvector/ts_rank, its memory path substrings — and this store reproduces the SQL half's predicate, not its ranker. The fixture's recorded order is ts_rank's; (min cover span, event_id) reproduces it on this corpus (spans 2, 2, 2, 4; ids ascending within a span), and a term-frequency rank does not.

The store's shape changed, so it has a decision record

docs/rewrite/decisions/0012-nip50-search-index.md (D1–D9) with 0011 trimmed to a pointer plus the facts it owns. The index is derived: every key rebuilds from events, search-index-version in meta versions the shape so a change forces a rebuild rather than a migration, and dropping the database costs a rebuild and nothing else. The rebuild runs inside open, in 1000-row batches, marker written last, behind a per-path lock so two concurrent opens rebuild once (batches == 2, markers == 1; the un-serialised build measures 4 / 2).

Validation

cargo fmt --all --check                                                exit 0
cargo clippy --workspace --all-targets --all-features -D warnings       exit 0
cargo clippy --workspace --lib --bins --all-features (strict five)      exit 0
cargo test --workspace --all-features     1267 passed / 0 failed / 4 ignored   (80 result lines)
cargo build --workspace --all-features --release                        exit 0
cargo test -p nostream-storage --all-features --test parity_lmdb        43 passed

Static review at this head: 0 findings from 1 provider with status ran — a verdict, not an absence. Two rounds of lenses: code (gate), architecture, performance, security, all PASS. Round 1's code lens found three defects that are fixed here: the ADR sat in the wrong document, proven was decided in three places with one test, and the tokeniser ran split-then-lowercase against the specified lowercase-then-split. Static round 3 then found the concurrent-rebuild race, measured and fixed above.

Known limits — read these before comparing against Postgres

1. Order, not values, and not the same order on every corpus. ts_rank produces numbers; this store produces a rank of (smallest cover span, event_id). On the fixture corpus the two agree on the four rows' order. A frequency-based rank does not reproduce it (it sorts 5454369c, 9ae1fef4, c004, cb41289d), and on a corpus where term frequencies differ the two orders may part with nothing here able to see it.
2. A limited search diverges. LMDB ranks by (span, id); Postgres keeps created_at DESC on a limited search. The unlimited set is the capture's; a limited branch's set can differ.
3. The tag-key residual is a coverage limit, not a defect. The index carries content terms only. An implementation that also indexed tag keys would answer every criterion correctly (the verdict never reads a tag) and pay index space — so no criterion would notice either way.
4. A one-term query still walks a prefix range (breadth, not duplication). The walk visits every posting sharing the term's first bytes and rejects the rest by term identity (storage: a one-term search proposes every posting sharing the term's first bytes #1112): "a" visits 414 candidates for 49 matches on the fixture corpus, "c" 254 for 2, and each survivor costs a body decode. Against a million-row store that is roughly 9% of all postings for one REQ. Filed as storage: a one-term search proposes every posting sharing the term's first bytes #1112 with the key-shape fix named.
5. The first open after an upgrade blocks. The rebuild runs on a blocking-pool thread that open awaits: 20 000 events ≈ 1.4 s in release, 100 000 ≈ 7 s, 1 000 000 ≈ 70 s on the host measured (240 000 postings in 1.37 s). A store where that is unacceptable on a restart needs the incremental or offline rebuild, not a background build that would answer from a half-built index.
6. A second handle in one process is outside the contract. The rebuild lock is taken by open, not by writers, so a write landing inside a rebuild through a second handle can lose postings. ADR D9 records the rule (one handle per store per process; no writes overlapping a rebuild) and storage: a second in-process LMDB handle can lose search postings during a rebuild #1111 carries the failing test that would close it.

Fixed in review

A posting whose term merely shared the first term's bytes was counted twice — for an event whose content is cat cats, a search for cat reached the keeper once per posting, so a count over-reported and a limited page could spend two slots on one row. walk now compares the posting's term slice with the walked term and skips a difference before the other-term probes and before any body decode (307fe09). Read-side only: SEARCH_INDEX_VERSION stays 1, the key layout is untouched, so no store rebuilds for it. Three tests pin it, two of which were proved red without the comparison.

What the lenses measured, so round 3 does not have to ask

measurement	value
postings per event, fixture corpus	9.5 (max 46), 4 592 postings
key writes, index vs the rest	3.1× total, index alone 2.1×
fixture store on disk	+39% (851 968 → 1 183 744 B, ≈72 B/posting)
create_many(483)	34.4 → 52.3 ms (control read +14%; ≈1.33× normalised)
the per-path lock, steady state	free — heed already serialises writers; open 0.15 ms
hostile shapes against the read path	21, no panic, no page written on any read (&RoTxn), page/count agree
write amplification, token-bomb content	≈15×
the 64-byte term cut	load-bearing: a 99 999-char token would otherwise break LMDB's key limit and fail a valid event

Do not remove the term truncation. It reads like a limitation and is the opposite: without it a client can make a legitimate event unindexable with one very long token.

Out of scope

nostream-cli migrate pg-to-lmdb and the benchmark rig — the other two slices of #864.

[changeset-bot]

🦋 Changeset detected

Latest commit: 2901660

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 0 packages

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 0517d171-12dd-4477-89b9-a5d05f4410c9

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• Trigger review

Pre-merge checks: 1 · 1

Failed checks: 1 inconclusive

Check name	Status	Explanation	Resolution
Title check		This assessment has not completed. The check will be retried on the next pre-merge check run.

Passed checks: 1 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

[greptile-apps]

Confidence Score: 4/5

[Medium impact] The change affects NIP-50 reads and event writes in the LMDB storage backend.

Fix the missing field in Databases::open before merging; the LMDB-enabled build cannot compile.

Findings

1.  LMDB build fails ▶

Summary

Adds events_by_term for LMDB search, shared query terms, ranked pages, and an open-time rebuild.

• LMDB searches event content and ranks matching rows.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Write[Event write] --> Index[events_by_term]
  Open[Open store] --> Marker{Marker current?}
  Marker -->|No| Rebuild[Rebuild term index]
  Query[Search filter] --> Walk[Walk term postings]
  Index --> Walk
  Walk --> Match[Check body and other criteria]
  Match --> Result[Ranked page or count]

Loading

_{Reviews (5) · Last reviewed commit: "Merge branch 'rewrite/main' into factory..." · Reviewed by Greptile}

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

[greptile-apps]

LMDB build fails

Databases now requires events_by_term, but Databases::open still supplies only the old fields. Rust rejects that constructor with a missing-field error, so nostream-storage cannot compile with LMDB enabled. Open SEARCH_INDEX_DB in Databases::open too.