[fuzzyrichie]

Background

json_string_unescape allocates its output buffer with the size of the raw (escaped) input — a correct upper bound, since unescape can only shrink (\uXXXX: 6 raw bytes → at most 4 output bytes, 12 → 4 for a surrogate pair, \n-style: 2 → 1) — but then finishes with:

rb_str_set_len(result, buffer - bufferStart);

rb_str_set_len sets the length and never releases capacity, so every parsed string that contains escapes retains the raw size as buffer capacity for as long as the parsed document is alive. Documents produced by Python's json.dumps(ensure_ascii=True) (Python's default) hit this hardest, since all non-ASCII comes out as \uXXXX.

Measured (ruby 3.4.12, json 3.1.0, a 12.5 MB document of 200 × 50 KB escaped strings):

retained string capacity: 12.52 MB   (content 7.75 MB, slack 4.78 MB = 61% of content)

Fix

One line: use rb_str_resize, whose shrink path implements MRI's own policy — strings that now fit inline are embedded (their heap buffer freed), and still-larger buffers are realloced down to the exact length when the waste exceeds MRI's threshold. No json-side threshold needed, no new allocations.

-    rb_str_set_len(result, buffer - bufferStart);
+    // The buffer was sized for the raw (escaped) input, but the unescaped
+    // content can be much shorter ... rb_str_resize shrinks oversized
+    // buffers (embedding short strings and reallocating large ones to
+    // the exact length), so parsed strings only retain the bytes they
+    // actually hold.
+    rb_str_resize(result, (rb_len_t)(buffer - bufferStart));

I first tried the obvious alternative — build an exact-size copy when the waste is at least a malloc quantum — but measured it at +19% parse time on \u-heavy documents (one extra malloc per escaped string) and dropped it. rb_str_resize achieves the same retention win allocation-free: realloc-in-place / embed, no churn.

Measurements (same machine, stock ruby 3.4.12)

metric	before	after
retained capacity, 12.5 MB escaped doc	12.52 MB (61% slack)	7.76 MB (0% slack)
parse time, 130 KB heavily-escaped doc	0.125–0.129 ms	0.131 ms (within stock's own run-to-run noise)
allocations per parse	—	unchanged
documents without escapes (fastpath)	—	byte-identical benchmark results (fastpath never enters unescape)

Validation

• Full test suite on the patched build: 612 tests, 4,040 assertions, 0 failures, 0 errors (parser, ext parser, minefield, resumable, float fallback, encoding, fixtures, coder, generator, common interface)
• Parse results vs stock: byte-for-byte identical (Marshal dump comparison over mixed escaped/plain/surrogate-pair documents)
• No behavior change: rb_str_resize maintains the terminator and encoding; the shrink clears coderange on a buffer whose coderange was unknown anyway

Compatibility note

rb_str_resize's embed-on-shrink is the modern-MRI behavior I measured (3.4); on the gem's minimum (2.7) the shrink path is less aggressive (realloc only above MRI's waste threshold), so on older rubies the win is smaller — but never worse than rb_str_set_len.

Found during a memory-consumption pass over the parser (allocation-site review + instrumented benchmarks); the only other significant retained-memory sites (container presizing, name interning, generator buffer) were already exact.

[byroot]

This is behavior is intended, it's not work risking a realloc for a string that will likely be thrown away soon.