Add node-based catamorphism POC by Adam-Vandervorst · Pull Request #31 · Adam-Vandervorst/PathMap

Adam-Vandervorst · 2026-01-09T23:38:56Z

I purpose we use this paradigm under the cata interface.

… far.

…zer more to work with. ~5% improvement to saturated val_count benchmark

…dable, and <5%

…ad memory when retrieving the refcount of the empty node

…h a test

…-head

Adam-Vandervorst · 2026-04-22T16:42:38Z

@luketpeterson any problem merging this?

luketpeterson · 2026-04-23T00:03:43Z

@luketpeterson any problem merging this?

In concept no. There is a (small) bit of work to make the API is consistent with the cata that's there already.

adamv-symbolica · 2026-08-16T22:04:21Z

Giving Fable some time with it:

Blocking

B1 — `goat_val_count` double-counts the root value

trie_map.rs:509–521. PathMap::recursive_cata already ends with
collapse_f(self.root_val(), Some(w), &[]), so the closure counts the root value; the caller
then adds root_val again.

let mut map = PathMap::new();
map.insert(b"", ());
map.insert(b"a", ());
assert_eq!(map.val_count(), 2);      // ok
assert_eq!(map.goat_val_count(), 2); // FAILS: returns 3

Fix: drop the + root_val (and the now-pointless match self.root()), or make the collapse
closure ignore the final root-val invocation.

B2 — `LineListNode` "Case 10" (Val, Child with different first bytes) violates the branch contract

line_list_node.rs:2952–2977. The PR's own tests establish the contract:
at a branch, branch_f is called once per branch in ascending mask-bit order with the branch
mask, so the algebra can attribute each W to its byte (mask.indexed_bit(acc.idx)). Case 10
breaks it twice:

the first branch_f call passes &ByteMask::new() (empty) instead of &mask
(line 2960 — contrast with Cases 4/5/8 which pass &mask on every call);
it processes the child slot first, but slots are stored sorted by first byte, so for
(Val@a, Child@b) the calls arrive in descending byte order.

// keys {"a", "b1", "b2"} -> root pair node (Val@'a', Child@'b')
// Reconstructing paths via the documented mask contract yields:
//   [[98], [238,49], [238,50]]   (0xEE = sentinel for "mask had no bit at idx")
// instead of [[97], [98,49], [98,50]]

The value gets attributed to byte b, and the child subtrie to an empty mask. Any
mask-sensitive algebra (including the PR's own recursive_cata_jumping_total_len bench closure,
which would panic on .unwrap()) is wrong or crashes on tries that contain this node shape.
Fix: pass &mask on both calls and emit the value (lower byte) before the child.

B3 — Zipper `recursive_cata` silently returns "empty" for a mid-node focus

morphisms.rs:513–521. The blanket impl uses get_focus().0.borrow(), which
returns Some only for the BorrowedRc/OwnedRc variants of AbstractNodeRef. A zipper
focused part-way into a node (BorrowedTiny/BorrowedDyn — exactly what TinyRefNode exists
for) gets None and is treated as an empty trie:

let mut map = PathMap::new();
map.insert(b"abc1", ());
map.insert(b"abc2", ());
let mut rz = map.read_zipper();
rz.descend_to(b"ab");                       // path exists
rz.recursive_cata::<..>(count_vals ...)     // returns 0, expected 2

Fix: handle the remaining variants — e.g. go through as_tagged()/try_as_tagged() and add a
TaggedNodeRef-level entry point, or fall back to into_option() (accepting the clone), or
panic loudly rather than returning a wrong answer.

B4 — Branch-byte convention differs between node types, so results depend on physical layout

DenseByteNode includes the branch byte in the collapse_f prefix
(dense_byte_node.rs:415, core::slice::from_ref(&key_byte)) and
represents it in the mask given to branch_f. LineListNode strips the byte from the prefix
(&key0[1..]) and represents it only in the mask. An algebra therefore cannot know whether
prefix[0] is the branch byte or a distinct following byte:

// Path-reconstruction algebra (byte taken from mask, prefix appended):
//   correct on LineListNode pair shapes,
//   duplicates the first byte on DenseByteNode: [1,7,13] -> [1,1,7,13]

The same logical trie yields different W depending on which physical nodes back it. The PR's
own sum-digits tests only pass because the algebra threads a bool flag ("value was at empty
prefix") through W to compensate — a workaround that no external user will discover from the
docs. This is the "API consistency with the existing cata" work the reviewer already flagged on
the PR. Fix: pick one convention (LLN's byte-in-mask-only matches the existing
into_cata_jumping_* sub_path semantics best) and align ByteNode::node_recursive_cata;
then document it on Summarization.

B5 — Unbounded recursion: stack overflow on deep tries; PR's own test aborts the `all_dense_nodes` suite

recursive_cata_cached recurses once per physical node. The PR's
recursive_cata_stack_overflow_smoke (morphisms.rs:2232) documents overflow
between 8–10 KB of path depth on default features — and under --features all_dense_nodes
(1 byte per node) the very same test overflows and SIGABRTs the whole test binary at
PATH_LEN = 8_000. As a public API this is a panic-free-abort footgun on adversarial/deep data,
and as merged it leaves a red test config. Minimum: gate or shrink the smoke test per feature
and document the depth limit prominently; proper fix: explicit work-stack or segmented stacks
(e.g. stacker::maybe_grow) in recursive_cata_cached.

Non-blocking cleanups

Dead code: node_goat_val_count (trait method + 6 impls) lost its only consumer when
traverse_physical was removed; either delete the chain or keep goat_val_count on it for
the comparison's sake — not both.
Commented-out blocks left in line_list_node.rs (old generic implementation, old
node_goat_val_count) and dense_byte_node.rs.
unreachable_unchecked on header patterns (line_list_node.rs:2980):
header values 1–7 (slot1 used, slot0 free) are assumed impossible; if that invariant is ever
violated this is UB rather than a panic. A debug_assert!/unreachable! in debug builds
would be cheap insurance.
Option<Acc> dance in ByteNode::node_recursive_cata (Some(Acc::default()) +
unwrap_unchecked) — ws is always Some; a plain Acc binding works.
pub(crate) values on ByteNode is only applied to the non-nightly field variant and no
code outside the module reads it — revert the visibility change.
Docs: Summarization docs still carry GOAT/dev-branch placeholders;
recursive_cata_stepping links to [Catamorphism::recursive_cata] (wrong trait); the
COMPUTE_PATH=false caveat ("no reliable child_masks") deserves a loud, user-facing warning
since it silently changes what the closures receive.
Bench honesty: recursive_cata_jumping_total_len asserts only the count, not the length —
the two implementations being compared do not agree on total length today (a consequence of B4).
slim_dispatch's TaggedNodeRef::node_val_count now takes the gxhash map while the node impls
take std — moot while the feature is bitrotted on master, but worth aligning if it's revived;
same for the missing BridgeNode arm in recursive_cata_dispatch under bridge_nodes.

adamv-symbolica · 2026-08-16T22:04:50Z

Path byte should not be represented in prefix.

…ld cata, and picking up another 5% perf

…from parent node

… wasn't offering a meaningful speedup when switched off

adamv-symbolica · 2026-09-03T00:08:51Z

One more

Iterative engine hangs on a LineListNode shape left behind by join_k_path_into (found by a randomized edit-program run against 3839f31)

join_k_path_into shortens two value keys in place: (Val@"aaa", Val@"bab") → (Val@"aa", Val@"ab"). factor_prefix accepts that (rule A: "slot 0 contains a value"), but insertion never builds it — a fresh {aa, ab} is Child@"a" -> {a, b} — and nothing that indexes children can walk it: descend_indexed_byte fails (master's side-effecting cata trips its debug_assert at morphisms.rs:~844), the recursive engine trips Case 7's debug_assert_eq!(key0.len(), 1), and CatamorphismCachedIterative loops forever (summarize/fold on byte a, never advancing to b). Plain iter() works, which is why it went unnoticed. The shape is pre-existing master behavior; the hang is new with the iterative engine.

Standalone test (drop into tests/); the assertions are ordered so it fails before it can hang:

use pathmap::PathMap;
use pathmap::zipper::*;
use pathmap::morphisms::{CatamorphismCached, CatamorphismCachedIterative};

#[test]
fn drop_head_shared_first_byte_shape() {
    let mut m = PathMap::<()>::new();
    m.set_val_at(b"aaa", ());
    m.set_val_at(b"bab", ());
    assert!(m.write_zipper().join_k_path_into(1, false));
    assert_eq!(m.iter().map(|(k, _)| k).collect::<Vec<_>>(), vec![b"aa".to_vec(), b"ab".to_vec()]);

    let mut rz = m.read_zipper();
    rz.descend_to_byte(b'a');
    assert_eq!(rz.child_count(), 2);
    assert!(rz.descend_indexed_byte(1).is_some(), "indexed descent cannot reach the second child"); // fails here today

    assert_eq!(CatamorphismCached::val_count(&m), 2);           // debug build: Case 7 assert
    assert_eq!(CatamorphismCachedIterative::val_count(&m), 2);  // never returns
}

Fix that makes this pass (and keeps both full suites green): in LineListNode::factor_prefix, rule A should also require a 1-byte slot-0 key —

let legal_overlap = overlap == 1 && (
    (!self.is_child_ptr::<0>() && key0.len() == 1) ||
    (!self.is_child_ptr::<1>() && key0.len()==1 && key1.len()==1 ));

— so the shape gets factored into the canonical Child@"a" -> {a, b}. Alternatively the engines could learn to traverse it, but since indexed descent itself can't, canonicalizing at the producer seems right. Worth adding to the recursive-vs-iterative equivalence suite either way.

Values never reach summarize_f: a value with nothing below it goes to map_f(&V, prefix), a value above a subtrie goes to collapse_f(&V, W) on the value-free W of that subtrie, in both the recursive and iterative drivers. Shared nodes are therefore summarized once regardless of the value the parent slot stores for them. Removes the single-function cached catas and the CataChildren adapter, ports hash/hash_with/val_count, the debug cata, benches and tests. Splits LineListNode traversal into a header-dispatched 1-byte-key fast path (tail-recursion friendly) and the general path, and gives every node type its own recursion frame. Fixes value placement for a (Val, Child) LineListNode whose child key shares its first byte and is longer than two bytes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

merkleize identified nodes by hashing their physical layout, keyed with the value the parent slot stores for them, so byte-identical subtries reached through a valued and an unvalued slot never merged, and neither did logically equal subtries laid out in different nodes or key segments. Nodes are now identified by the cached cata's value-free logical hash, which is the same function as CatamorphismCached::hash (the shared scheme lives in morphisms::trie_hash, and MerkleizeResult::hash equals the map's cata hash). The cata cache became a crate-internal trait, CataCache, whose ordinary implementation is the HashMap it always was; merkleization plugs in a cache that records every node's hash in post-order. A bottom-up rewrite pass then reads those hashes back in the same order and replaces each repeated subtrie with the first node that produced its hash, visiting physically shared nodes once. Adds merkleize tests (hash equals the cata hash and contents are kept, different layouts merge, idempotence, shared input untouched), a divan bench and a viz example that reports sharing before and after. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…t read in the cata Make the trie hash's primitive pluggable and model the hash in Lean so the differential harness can check that it is a function of the logical trie alone. * `morphisms::trie_hash` is now public: a `HashScheme<V>` trait with the production `GxHashScheme`, `GxHashWith` (behind the existing `hash_with`), and `Fnv1a64Scheme` for fuzzing. Both cached cata traits gain `hash_with_scheme`. `merkleize_root` is generic over the scheme but stays crate-private; `PathMap::merkleize` remains pinned to gxhash, since a caller-supplied scheme with collisions would merge subtries that differ. * `lean/PathMapModel/Hash.lean` defines the same hash over the flat trie model. Ops 56-59 (`hash`, `hash_iter`, `map_hash`, `merkleize`) are added to both front ends (NOPS 60), with `#guard`s in Check.lean, six hasher mutants in mutants.toml, and README/FINDINGS entries. * The first fuzz run found a memory-safety bug (FINDINGS.md #17): the ordinary `CataCache::skips` read `refcount()` of the empty-node sentinel when a two-slot node had a dangling child, a segfault in release reachable from `hash`, `val_count` and `merkleize`. `skips` now accepts the sentinel without reading it, the trait documents the contract, `refcount` gets a `debug_assert`, and a minimal reproducer test is added. Also on this branch (adam2/merkle-cata + master): * Port the structural-sharing tests from `merkleize-fix` into the merkleize test module, plus a physical-layout dumper and topology exploration tests. * Fix test-only fallout from merging master: `val_count` moved to the cached cata traits (a `FuzzValCount` helper in the differential crate), the removed `Catamorphism::into_cata_cached` in one write_zipper test, and `assert_eq!(x, &[])` ambiguities. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…h messages lean/PathMapModel/HashSecurity.lean proves the collision reduction for the logical trie hash, for an arbitrary primitive H: if two positions whose subtries differ (`LogicalEq` fails) hash alike, then two different messages collide under H, or two different values under the value hash, and the proof constructs them (`collision_of_hash_eq`). `hashWith_inj_imp_logicalEq` is the ideal-primitive form and `fnv_collision_of_hashU64_eq` the instance for the harness's PathMap<u64> maps. The reduction rests on the messages being decodable: a tag byte separates node from value messages, the 32-byte mask bitmap determines the child set and hence how many fixed-width digests follow, and running out of fuel hashes the empty message. The primitive itself is treated as replaceable; nothing is claimed about gxhash or FNV. To make the proven layout the production layout, `GxHashScheme` now writes `NODE_TAG`/`VALUE_TAG` (b'N'/b'V') before its messages, shared with `Fnv1a64Scheme`. The mask stays a 32-byte bitmap. Hash.lean defines the bitmap through `digits` (bit b % 8 of byte b / 8) and `leBytes64` as an explicit list, both guarded against hand-computed bytes, and returns `fnv []` when fuel runs out so an inadequate fuel can never look like a real hash. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

lean/PathMapModel/GraftNoop.lean: `graft_noop_of_hash_eq` shows that for an ideal primitive, `hash(wz) = hash(rz)` at the two foci makes `wz.graft(&rz)` leave every Entry of the write map as it was. `collision_of_graft_ne` is the unconditional contrapositive (a hash-equal graft that changes the trie exhibits a collision of H or of the value hash) and `graft_noop_of_hashU64_eq` the instance for the harness's PathMap<u64> maps under Fnv1a64Scheme. The hypotheses are the canonical-form invariants: the write map is prefix-closed with an existing root, the source is prefix-closed. The hashing half is HashSecurity.lean's soundness theorem. The rest characterizes the model's normalizer `mk'` (`valAt_mk'`: the first binding in the supplied list, via Perm-invariance of lookup on unique keys; `pathExists_mk'`: prefixes of the root, of a given path, or of a valued path) and, from it, `removeBelow`, `graftBelow`, `setVal`, `removeVal` and `subtrie` as observations at any path. Those lemmas need no hashing and are available to Spec.lean. Check.lean gains executable guards: a hash-equal graft at an interior focus and at the root leaves a fixture unchanged, a non-hash-equal graft changes it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

luketpeterson · 2026-09-08T23:20:18Z

ACT should get a fast recursive implementation of cata, modeled on the ACT val_count in master, and val_count benchmarks that can easily be back-ported to master.

Implement subtraction-aware recursive cached morphisms for SubtractZipper, delegate unaffected subtrees, and keep value counting in the morphism API. Adapt incoming tests to the new API.

…ht by miri

# Conflicts: # src/write_zipper.rs

…cover it. (~3-5x uplift)

github-actions · 2026-11-02T07:59:22Z

Bench A/B vs base: done

job log · 2026-11-11 04:53:39 UTC

base 8172d18 → head 1a794c0, 3 round(s), median of each run averaged; negative is faster

bench	cases	geomean	largest gain	largest loss	>5% faster	>5% slower
shakespeare	7	-7.0%	-58.3% `shakespeare/shakespeare_sentences_val_count_act`	+27.9% `shakespeare/shakespeare_words_val_count`	1	2
cities	5	-4.3%	-23.6% `cities/cities_val_count_act`	+3.6% `cities/cities_val_count`	1	0
sparse_keys	102	-0.9%	-10.7% `sparse_zipper_cursor/400`	+6.1% `sparse_k_path_iter/1600`	12	2
binary_keys	83	+0.1%	-13.8% `binary_descend_until/2000`	+17.9% `binary_meet/800`	9	6
superdense_keys	102	+2.4%	-10.0% `superdense_insert/1600`	+54.8% `superdense_drop_bench/200`	4	17
act_paths	46	-1.5%	-9.2% `shape_map_to_act_cata/random_long`	+3.2% `size_map_to_act_cata/200000`	2	0
zipper_head_owned	16	+0.9%	-1.0% `zipper_head_owned/map_reader_is_val`	+5.1% `zipper_head_owned/owned_head_read_creation`	0	2
product_zipper	4	-3.2%	-5.3% `product_zipper/introspecting_pathmap_pathmap`	-2.3% `product_zipper/generic_pathmap_pathmap`	1	0

59 case(s) moved more than 5%

bench	case	base	head	change
shakespeare	`shakespeare/shakespeare_sentences_val_count_act`	37.01 ms	15.43 ms	-58.3%
superdense_keys	`superdense_drop_bench/200`	67 ns	103 ns	+54.8%
superdense_keys	`superdense_drop_bench/100`	50 ns	66 ns	+33.5%
shakespeare	`shakespeare/shakespeare_words_val_count`	312.23 µs	399.27 µs	+27.9%
cities	`cities/cities_val_count_act`	52.78 ms	40.33 ms	-23.6%
binary_keys	`binary_meet/800`	79.12 µs	93.30 µs	+17.9%
binary_keys	`binary_meet/400`	38.58 µs	45.48 µs	+17.9%
binary_keys	`binary_meet/200`	20.48 µs	24.06 µs	+17.5%
binary_keys	`binary_meet/100`	10.24 µs	12.02 µs	+17.4%
binary_keys	`binary_meet/1600`	154.77 µs	180.83 µs	+16.8%
binary_keys	`binary_meet/50`	5.24 µs	6.11 µs	+16.5%
binary_keys	`binary_descend_until/2000`	210.03 µs	181.13 µs	-13.8%
superdense_keys	`superdense_zipper_prev_step_iter/3200`	18.99 µs	21.57 µs	+13.6%
superdense_keys	`superdense_zipper_prev_step_iter/1600`	10.49 µs	11.86 µs	+13.1%
shakespeare	`shakespeare/shakespeare_sentences_val_count`	2.79 ms	3.14 ms	+12.3%
binary_keys	`binary_descend_until_max_bytes/2000`	163.47 µs	144.00 µs	-11.9%
superdense_keys	`superdense_zipper_prev_cursor/3200`	32.88 µs	36.52 µs	+11.1%
sparse_keys	`sparse_zipper_cursor/400`	5.81 µs	5.19 µs	-10.7%
binary_keys	`binary_zipper_iter/1600`	38.23 µs	34.17 µs	-10.6%
sparse_keys	`binary_drop_head/100`	7.93 µs	7.12 µs	-10.2%
superdense_keys	`superdense_insert/1600`	42.15 µs	37.94 µs	-10.0%
superdense_keys	`superdense_zipper_prev_cursor/1600`	17.71 µs	19.42 µs	+9.7%
act_paths	`shape_map_to_act_cata/random_long`	43.67 ms	39.67 ms	-9.2%
superdense_keys	`superdense_meet_low_overlap/32000`	23.50 µs	25.53 µs	+8.7%
superdense_keys	`superdense_drop_head/32000`	112.53 µs	122.07 µs	+8.5%
sparse_keys	`sparse_zipper_cursor/200`	2.58 µs	2.36 µs	-8.4%
superdense_keys	`superdense_zipper_cursor/800`	5.38 µs	5.83 µs	+8.3%
sparse_keys	`sparse_zipper_prev_cursor/400`	65.18 µs	59.87 µs	-8.2%
superdense_keys	`superdense_join/32000`	105.70 µs	113.80 µs	+7.7%
sparse_keys	`sparse_zipper_cursor/800`	12.54 µs	11.61 µs	-7.4%
sparse_keys	`sparse_zipper_prev_cursor/50`	7.35 µs	6.81 µs	-7.3%
sparse_keys	`sparse_zipper_cursor/100`	1.27 µs	1.18 µs	-7.1%
act_paths	`size_paths_to_act/100000`	14.82 ms	13.78 ms	-7.0%
sparse_keys	`sparse_zipper_prev_cursor/200`	31.66 µs	29.50 µs	-6.8%
sparse_keys	`sparse_zipper_prev_cursor/100`	15.09 µs	14.06 µs	-6.8%
binary_keys	`binary_zipper_prev_iter/50`	46.72 µs	43.58 µs	-6.7%
binary_keys	`binary_zipper_prev_iter/100`	91.19 µs	85.18 µs	-6.6%
superdense_keys	`superdense_zipper_prev_cursor/400`	6.00 µs	6.39 µs	+6.5%
superdense_keys	`superdense_join/16000`	62.03 µs	66.02 µs	+6.4%
sparse_keys	`sparse_zipper_cursor/50`	624 ns	584 ns	-6.4%
sparse_keys	`sparse_zipper_cursor/1600`	23.52 µs	22.01 µs	-6.4%
superdense_keys	`superdense_meet_after_join/32000`	92.70 µs	98.56 µs	+6.3%
superdense_keys	`superdense_k_path_iter/1000`	6.21 µs	6.58 µs	+6.1%
sparse_keys	`sparse_k_path_iter/1600`	26.21 µs	27.80 µs	+6.1%
superdense_keys	`superdense_k_path_iter/2000`	12.36 µs	13.11 µs	+6.1%
binary_keys	`binary_zipper_prev_iter/200`	177.67 µs	166.90 µs	-6.1%
superdense_keys	`superdense_k_path_iter/500`	3.10 µs	3.29 µs	+6.0%
binary_keys	`binary_zipper_prev_iter/400`	344.03 µs	323.37 µs	-6.0%
superdense_keys	`superdense_join/1000`	16.52 µs	15.56 µs	-5.8%
binary_keys	`binary_zipper_prev_iter/1600`	1.29 ms	1.21 ms	-5.6%
superdense_keys	`superdense_insert/3200`	71.98 µs	68.10 µs	-5.4%
binary_keys	`binary_zipper_prev_iter/800`	665.53 µs	629.70 µs	-5.4%
product_zipper	`product_zipper/introspecting_pathmap_pathmap`	450.17 ms	426.33 ms	-5.3%
superdense_keys	`superdense_zipper_cursor/200`	1.09 µs	1.15 µs	+5.2%
zipper_head_owned	`zipper_head_owned/owned_head_read_creation`	11.76 µs	12.36 µs	+5.1%
sparse_keys	`sparse_k_path_iter/800`	13.92 µs	14.62 µs	+5.1%
sparse_keys	`sparse_zipper_prev_cursor/800`	127.67 µs	121.23 µs	-5.0%
superdense_keys	`superdense_insert/200`	2.19 µs	2.08 µs	-5.0%
zipper_head_owned	`zipper_head_owned/shared_parent_reader_val`	1.47 µs	1.54 µs	+5.0%

Full tables per bench are in the job log and the bench-out artifact.

… (comment)

… the stem of a prefix zipper too)

luketpeterson · 2026-11-02T08:46:03Z

I think this PR is finally ready to go. However, the one last piece of validation I want to do is see if the bespoke val_counts that are getting replaces got any slower by moving the implementation over to the recursive catamorphism.

luketpeterson · 2026-11-02T12:04:54Z

Here are the complete benchmark results.
recursive-cata-benchmark-results.zip

Bottom line, recursive cata is massively faster than iterative cata, as discussed above.

Since val_count is now subsumed by recursive cata, the val_count benchmark results are a bit more nuanced. There is an essential tradeoff in the implementation whether to count the value first or to visit the downstream first. Recursive cata made a different tradeoff than the hard-coded val_count resulting in better performance for catamorphism workloads, and about a ~10-20% uplift on "real" tries (binary trie shape), roughly neutral on the superdense benchmark, but a similar penalty on the "sparse" tries (a bunch of non-branching paths).

On ACT, there was no recursive cata at all. Implementing one gave a 3-8x speedup over the iterative cata but a 5-40x perf improvement in val_count vs the implementation on the ACTZipper!

Preserve the original Merkle lineage and its Adam/Igor commits while adapting logical hashing and Merkleization to the current three-callback cata API. Record full and value-free node hashes through a private cache abstraction, including reuse of shared nodes reached through valued parent slots. Retain the current public cata API and its recursive ACT implementation. Integrate the logical hash model, collision and graft proofs, regression tests, benchmark and example. Append hash/Merkle fuzz operations at IDs 60-63 so the existing versioned input formats retain their meaning. Update reproducer codegen and mutation targets for the current API. Validation: - cargo test --offline --workspace --features viz,arena_compact,random - lake build, including HashSecurity/GraftNoop proofs and build-time guards - Python input-format tests and the new Rust harness integration test - 500 targeted hash/Merkle programs against Lean on each of native and ACT - 1,000 general ACT differential inputs (seed 2), all agree - All six adapted hash/cache mutants compile and are detected - Benchmarks/examples compile; dense-node Merkle regression tests pass General native differential seed 1 also exposes the existing meet value-bias issue: the reduced meet_into changes [3,1] from 0 to the source value 1. This was reproduced on pre-merge 1e95979 and is outside this integration. Building without slim_ptrs also fails on the pre-merge commit and remains unchanged.

luketpeterson · 2026-11-11T03:40:35Z

imlvts/merkle-cata-sharing was already descended from merkle-cata, so merging it already got all that work.

Cata API adaptation was straightforward.

Bring in master's meet, graft, prune, relative-write and empty-focus fixes. Preserve the Merkle integration, current catamorphism API and roadmap comments. Resolve the fuzz opcode collision by retaining Merkle operations at 60-63 and appending master's set_val_at at 64. New inputs use 65 selectors; inputs from master's 61-selector format retain their set_val_at meaning through explicit Rust/Lean decoding. Preserve the 64-selector Merkle format as well. Add Rust and Lean regression checks for both historical formats, and update reproducer codegen, Python selector count and input-format documentation. Validation: - cargo test --offline --workspace --features viz,arena_compact,random - lake build (including proofs and format regression guards) - Python input-format tests - 2,000 native differential inputs: 45 known findings, no new divergences - 2,000 ACT differential inputs: all agree - 500 targeted hash/Merkle differential programs: all agree

Retain set_val_at at opcode 60 and append hash, hash_iter, map_hash and merkleize at 61-64. Restore plain modulo decoding in both Rust and Lean, and remove the intermediate-format compatibility tests and documentation. Update reproducer generation, retained harness tests and mutation comments. Intermediate Merkle/integration runs can be regenerated; no migration is kept. Validation: differential crate tests, Python input-format tests, lake build, 500 fresh targeted inputs on each native and ACT, and 1,000 general native inputs (33 known findings, no new divergences).

github-actions · 2026-11-11T04:38:55Z

Differential fuzz vs Lean model: success

job log · 2026-11-11 04:38:54 UTC

WARNING: 1 new divergence(s) relative to base

Base was built with its own harness and model (head's did not build against it), so harness changes may show up as differences.

crate: 20000 inputs, seed 7

agree = model and crate match; known = the divergence matches a classified bug in differential.py; new = it matches none. Only the new set is compared input by input below.

side	agree	known	new divergences
head	19769/20000	230	1
base	19724/20000	276	0

1 input(s) diverge on head but not on base; 0 diverge on base but not on head.

Known-class hits that changed

known class	base	head
AlgebraicStatus::Identity is not returned reliably when nothing changed (finding 8); reproducer in l	233	200
AlgebraicStatus::Identity is not returned reliably when nothing changed (finding 8); meet_into/subtr	22	17
an algebraic op keeps a dangling child the spec drops: an empty child node meets/subtracts to itself	8	3
which value survives a collision follows node layout rather than the left-biased spec (join/meet and	12	8
subtract_into() drops a value under a path present in the source [subtract_drops_value]	1	2

Newly diverging inputs (head only): 1 input(s), 1 kind(s)

first differs at	inputs	first example
`map_hash`	1	`random#19434`: line 9

Reproducers: first 1 distinct kind(s), shrunk

random#19434: first differs at map_hash, 257 -> 52 bytes

line 9
lean: 9 map_hash ret=72ea3f0c77e181a3 W=020001010302 o020001010302 e0 v- c0 n0 f02 R=0203000202 o0203000202 e1 v25 c0 n1 f02
crate: 9 map_hash ret=59361e69864350de W=020001010302 o020001010302 e0 v- c0 n0 f02 R=0203000202 o0203000202 e1 v25 c0 n1 f02

//! Generated by `pathmap_trace --repro`.  Reproduces a fuzzer input as
//! plain `pathmap` API calls.
//!
//! `wz` writes into map0 and `rz` reads map1; they are separate maps so
//! both can be held at once.  Values are `u64`.

use pathmap::PathMap;
use pathmap::utils::ByteMask;
use pathmap::zipper::*;

fn main() {
    let mut map0 = PathMap::<u64>::new();
    map0.set_val_at(&[], 0);
    map0.set_val_at(&[], 0);
    map0.set_val_at(&[0u8], 0);
    map0.set_val_at(&[], 0);
    map0.set_val_at(&[], 0);
    let mut map1 = PathMap::<u64>::new();
    map1.set_val_at(&[], 0);
    map1.set_val_at(&[], 0);
    map1.set_val_at(&[], 0);
    map1.set_val_at(&[0u8], 1);
    map1.set_val_at(&[0u8, 0, 0, 0, 0], 0);
    map1.set_val_at(&[], 0);
    map1.set_val_at(&[1u8, 0], 0);

    let mut wz = map0.write_zipper_at_path(&[]);
    let mut rz = map1.read_zipper_at_path(&[]);

    /*   0 */ wz.descend_to(&[]);
    /*   1 */ wz.join_map_into(rz.make_map());
    /*   2 */ { let path = wz.path().to_vec(); drop(wz); pathmap::morphisms::CatamorphismCached::<u64>::hash_with_scheme(&map0, &pathmap::morphisms::trie_hash::Fnv1a64Scheme); wz = map0.write_zipper_at_path(&[]); wz.descend_to(&path); }

    drop(wz);

    // What the maps hold at the end -- compare against the trace's MAP0/MAP1.

    println!("MAP0 {:?}", map0.iter().collect::<Vec<_>>());

    println!("MAP1 {:?}", map1.iter().collect::<Vec<_>>());
}

act: 5000 inputs, seed 7

agree = model and crate match; known = the divergence matches a classified bug in differential.py; new = it matches none. Only the new set is compared input by input below.

side	agree	known	new divergences
head	4997/5000	3	0
base	3584/5000	1416	0

0 input(s) diverge on head but not on base; 0 diverge on base but not on head.

Known-class hits that changed

known class	base	head
ACTZipper::val_count() counts from the zipper root, not the focus [act: val_count_ignores_focus]	1416	0
which value survives a collision follows node layout rather than the left-biased spec (join/meet and	0	3

Adam-Vandervorst and others added 15 commits January 9, 2026 21:19

Add node-based catamorphism POC

981b041

Thrashing towards fully functional recursive caching cata.

949d2d6

Implementing new API structure for recursive cata. No loss in perf so…

54b8daa

… far.

Slight tweak to node_recursive_cata for byte node, to give the optimi…

cd6ea5c

…zer more to work with. ~5% improvement to saturated val_count benchmark

Getting rid of dead code

3fb75fe

Working through all the cases in pair node (on paper)

69bce41

Handling every case in the PairNode for recursive_cata (still no paths)

3a8e826

Adding correct path handling to node_recursive_cata for PairNode

42c3587

Re-adding caching to recursive_cata. Slight perf hit, but it's unavoi…

542c283

…dable, and <5%

Putting empty-node check back into recursive cata, to avoid reading b…

32973bf

…ad memory when retrieving the refcount of the empty node

Filling in remaining node type branches for recursive_cata

ce6a8f8

Adding recursive_cata_stack_overflow_smoke test

73130ea

Adding a test for recursive cata, and adding the stepping version wit…

58ea1f1

…h a test

Reorganizing recursive_cata so it can be part of the public API

df4e48a

Adding catamorphism benchmark to put two cata implementations head-to…

2567f33

…-head

luketpeterson added 11 commits August 22, 2026 23:48

Merge branch 'master' into osplit-valcount

a8b0ee9

Collapsing recursive_cata_stepping implementation into a trait default

95cdb38

Tweaking the Summarization API to support a cheap(ish) shim for the o…

ee5ce1b

…ld cata, and picking up another 5% perf

Adding some more correctness tests for the summarize mechanism

c63191a

Fixing bug in new summarize API caused by not carrying value forward …

9699447

…from parent node

Improving documentation for recursive_cata callback args

7342db5

Renaming closures in Summarization

f0104fa

Adding fallible exit path to recursive cata

640b82b

Removing COMPUTE_MASKS generic constant in recursive cata, because it…

2a3a4f5

… wasn't offering a meaningful speedup when switched off

Increasing val_count benchmark sizes

d4d3fba

Adding iterative zipper-backed implementation of Summarization trait

939a817

adamv-symbolica mentioned this pull request Sep 2, 2026

join_k_path_into leaves two values on one key when shortened keys collide #84

Closed

adamv-symbolica and others added 6 commits September 7, 2026 01:33

Merge branch 'master' into merkle-cata-sharing

8beb320

This was referenced Oct 6, 2026

Merkleize byte-identical node allocation #90

Open

Merkleize only hashes physically identical node layouts to the same value #95

Open

luketpeterson added 5 commits November 1, 2026 20:10

Merge master into osplit-valcount

89fa8b9

Implement subtraction-aware recursive cached morphisms for SubtractZipper, delegate unaffected subtrees, and keep value counting in the morphism API. Adapt incoming tests to the new API.

Fixing a braindead place where a mut ptr was used unnecessarily, caug…

a768470

…ht by miri

Merge branch 'master' into osplit-valcount

88c17d8

# Conflicts: # src/write_zipper.rs

Adding separate recursive vs. iterative val_count benchmarks

9d17784

Adding recursive cata impl to ACTZipper, and expanding benchmarks to …

e346e66

…cover it. (~3-5x uplift)

luketpeterson mentioned this pull request Nov 2, 2026

ACTZipper needs sharing awareness #174

Open

Updating harness to deal with val_count moving to a different trait

7fab6cf

luketpeterson added 2 commits November 2, 2026 01:16

Adding tests to exercise the issues reported in #31 (comment) and #31…

078bb65

… (comment)

Fixing outstanding test failures in recursive cata (and grafting from…

1e95979

… the stem of a prefix zipper too)

luketpeterson added 2 commits November 10, 2026 20:02

Adding clarifications that were lost in the merge

75a6577

luketpeterson added 2 commits November 10, 2026 20:40

adamv-symbolica mentioned this pull request Nov 13, 2026

Add wiki_prefixes bench: catamorphisms on prefix-closed Wikipedia titles #187

Draft

Navigation Menu

Repository navigation

Add node-based catamorphism POC - #31

Add node-based catamorphism POC#31
Adam-Vandervorst wants to merge 70 commits into
masterfrom
osplit-valcount

Adam-Vandervorst commented Jan 9, 2026

Uh oh!

Adam-Vandervorst commented Apr 22, 2026

Uh oh!

luketpeterson commented Apr 23, 2026

Uh oh!

adamv-symbolica commented Aug 16, 2026

Uh oh!

adamv-symbolica commented Aug 16, 2026

Uh oh!

adamv-symbolica commented Sep 3, 2026

Uh oh!

luketpeterson commented Sep 8, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 2, 2026 •

edited

Loading

Uh oh!

luketpeterson commented Nov 2, 2026

Uh oh!

luketpeterson commented Nov 2, 2026 •

edited

Loading

Uh oh!

luketpeterson commented Nov 11, 2026

Uh oh!

github-actions Bot commented Nov 11, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Adam-Vandervorst commented Jan 9, 2026

Uh oh!

Adam-Vandervorst commented Apr 22, 2026

Uh oh!

luketpeterson commented Apr 23, 2026

Uh oh!

adamv-symbolica commented Aug 16, 2026

Blocking

B1 — goat_val_count double-counts the root value

B2 — LineListNode "Case 10" (Val, Child with different first bytes) violates the branch contract

B3 — Zipper recursive_cata silently returns "empty" for a mid-node focus

B4 — Branch-byte convention differs between node types, so results depend on physical layout

B5 — Unbounded recursion: stack overflow on deep tries; PR's own test aborts the all_dense_nodes suite

Non-blocking cleanups

Uh oh!

adamv-symbolica commented Aug 16, 2026

Uh oh!

adamv-symbolica commented Sep 3, 2026

Uh oh!

luketpeterson commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bench A/B vs base: done

Uh oh!

luketpeterson commented Nov 2, 2026

Uh oh!

luketpeterson commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

luketpeterson commented Nov 11, 2026

Uh oh!

github-actions Bot commented Nov 11, 2026

Differential fuzz vs Lean model: success

crate: 20000 inputs, seed 7

Known-class hits that changed

Newly diverging inputs (head only): 1 input(s), 1 kind(s)

Reproducers: first 1 distinct kind(s), shrunk

act: 5000 inputs, seed 7

Known-class hits that changed

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

B1 — `goat_val_count` double-counts the root value

B2 — `LineListNode` "Case 10" (Val, Child with different first bytes) violates the branch contract

B3 — Zipper `recursive_cata` silently returns "empty" for a mid-node focus

B5 — Unbounded recursion: stack overflow on deep tries; PR's own test aborts the `all_dense_nodes` suite

luketpeterson commented Sep 8, 2026 •

edited

Loading

github-actions Bot commented Nov 2, 2026 •

edited

Loading

luketpeterson commented Nov 2, 2026 •

edited

Loading