Repository navigation
Fix integer overflow in the number parser digit counter - #1134
Conversation
|
Thanks for the patch, but |
json_parse_digits returns the number of digits consumed as an
(int)(cursor - start). A JSON number whose digit run is 2^31 or
longer (a >= 2GiB document) wraps the counter negative, and the
mantissa_digits <= MAX_FAST_INTEGER_SIZE fast-path check on a long
wrapped uint64 accumulator then returns a small garbage Integer
instead of the number's true value:
$ ruby -rjson -e 'p JSON.parse(File.read("huge.json"))[0].bit_length'
63 # a 2_147_483_664 digit number cannot be 63 bits
The same wrap also defeats the exponent_digits >= 20 guard, letting
a wrapped exponent accumulator reach the exponent.
Widen the digit counters from int to ssize_t, so the routing
comparisons (<= 19, == 20, > 18, >= 20) see the true count and
route huge numbers to the existing large-number paths, which
already derive their lengths from the raw text. ssize_t keeps the
signed semantics the int counters relied on -- decimal_point_pos
starts at -1, and json_decode_float does signed arithmetic in
mantissa_digits + exponent < -307 with a negative int64 exponent --
which an unsigned size_t would break, while still being wide enough
to hold the true digit count.
Not feasible to regression-test from Ruby: the wrap needs a
>= 2GiB digit run, and the corrected path converts it through
rb_cstr2inum which is quadratic in the digit count. Validated
empirically against a 2_147_483_664 digit document: before this
change it returns a 63 bit Integer in 0.5s; after it correctly
enters the large-number path. All numeric boundary probes
(uint64 max, 19/20/21 digits, subnormals, 100k digit bignum) are
byte-identical before and after, and the full test suite passes.
Note: java/src/json/ext/Parser.java keeps the same int counters;
I could not build JRuby here so I left it out of this change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
0cb79ca to
e4dfd77
Compare
|
Thanks, makes sense — switched the digit counters from I went with The pre-existing |
Background
json_parse_digitsreturns the number of digits it consumed as(int)(state->cursor - start)(ext/json/ext/parser/parser.c). The counter is then used for all the fast-path routing of numbers:A JSON number whose digit run reaches 2^31 digits (a ≥ 2GiB document) wraps the counter negative, so
mantissa_digits <= 19is true on auint64accumulator that long since wrapped mod 2^64 — and the parser silently returns a small garbage Integer instead of the number's true value:The same wrap also defeats the
exponent_digits >= 20guard (parser.c:1519), letting a wrappeduint64exponent accumulator through theINT64_MAXcheck.Fix
Widen the digit counters to
rb_len_t— the same direction as #1133:json_parse_digitsreturnsrb_len_t(both return sites)mantissa_digits,fractional_digits,exponent_digits,decimal_point_posin the number entryjson_decode_integer/json_decode_floattakerb_len_t mantissa_digitsThe routing comparisons (
<= 19,== MAX_FAST_UINT64_SIZE,> 18,>= 20,+ exponent < -307/-324) then see the true count and route huge numbers to the existing large-number paths (json_decode_large_integer/json_decode_large_float), which already derive lengths from the raw text asrb_len_t— no new limits, no behavior change for any input that fits the old counters.Validation
[+1×2³¹⁺¹⁶ +]:rb_cstr2inum)1e308/1e309,5e-324/4.9e-324, 100k-digit bignum (EXACT vsKernel::Integerboth sides)Notes
rb_cstr2inum, which is quadratic in the digit count — neither side of the test is feasible in CI. Happy to add one if you want a unit-level C harness instead.java/src/json/ext/Parser.javakeeps the sameintcounters (parseDigits→int mantissa_digits/int exponent_digits, lines ~633/~659); I could not build JRuby here, so I left it out of this change rather than ship an untested patch. Happy to follow up.Follows #1133 (
rb_len_tmigration).