$substring throws IndexOutOfBoundsException on strings with non-BMP characters when length is omitted or too large #461

Problem

$substring(str, start) throws java.lang.IndexOutOfBoundsException when str contains a non-BMP character (a code point above U+FFFF, stored in Java as a surrogate pair). This includes emoji such as 😀 (U+1F600) and also mathematical letters such as 𝒳 (U+1D4B3).

JSONata JS counts code points and returns the substring.

Reproduce

$substring('😀bc', 1)
$substring('𝒳bc', 1)
$substring('😀bc', 0, 10)

$substringAfter('😀-x', '-') throws the same exception.

Cause

In JSONataUtils.substr, start is converted from code points, but length is clamped to the UTF-16 length and then converted from offset 0 instead of from start:

if (length == null) {
    length = strData.length();          // UTF-16 units
} else if (length > strData.length()) {
    length = strData.length();          // UTF-16 units
}
length = strData.offsetByCodePoints(0, length);   // throws if length > code point count

For '😀bc' the UTF-16 length is 4 but there are only 3 code points, so offsetByCodePoints(0, 4) throws. Because the offset is counted from 0 rather than from start, a non-BMP character after start can also be cut in half.

SubstringAfterFunction and SubstringBeforeFunction contain copies of this helper with the same logic.

Suggested fix

Clamp length to the code points that remain after start, and convert it relative to start:

int remaining = strData.codePointCount(start, strData.length());
if (length == null) {
    length = remaining;
} else if (length < 0) {
    return "";
} else if (length > remaining) {
    length = remaining;
}
length = strData.offsetByCodePoints(start, length) - start;

This is the fix applied in dashjoin/jsonata-java, which shares this code: dashjoin/jsonata-java#123 (fixed in dashjoin/jsonata-java#126).