[raviston]

Rationale for this change

In the corrupt pages reported in #3712, the length prefix of the repetition levels and the page header both match the size() of the level stream, but the serialized payload is shorter. @edgarRd's analysis shows that a contiguous range ending exactly at a slab boundary of CapacityByteArrayOutputStream is missing. The analysis reproduces the stored page byte for byte when the position of an already full slab is lowered before serialization. size() comes from the independent bytesUsed counter, while writeTo() writes slab.position() bytes per slab, and nothing checks the two against each other. So the writer silently produced files that no reader can decode.

This PR does not identify what lowered the slab position. It makes that state fail the write instead of corrupting the file, and it removes the code paths in this class that change the position of a slab after it was written.

What changes are included in this PR?

• writeTo and writeInto check that the slabs hold exactly size() bytes before they write anything, and throw an IllegalStateException otherwise. The check is one pass over the slab list.
• writeInto and the direct buffer path of writeTo read the slabs through duplicate(). Before, they called flip() on the slabs themselves, so serializing changed the slabs' position and limit.
• setByte used slab.limit() to find the slab holding a byte, which assumes that every slab but the last is full. Since writeInt/writeLong were added (GH-3530: Optimize PLAIN encoding and decoding with direct ByteBuffer I/O #3565), a slab can be followed by a new one before it is full, and then setByte patched the wrong byte. It now uses slab.position(). Today the only caller of setByte is RunLengthBitPackingHybridEncoder, which never calls writeInt/writeLong, so no current code path reaches this. It is fixed because it is the same kind of size/content mismatch.

Are these changes tested?

Yes. The new TestCapacityByteArrayOutputStream runs with heap and direct allocators:

• setByte after a partially filled slab. Fails without this change.
• Serializing repeatedly while writing gives the same content through writeTo, writeInto and BytesInput.
• The Page header uncompressed_page_size exceeds the actual decompressed payload by 2 bytes on ZSTD + dictionary-encoded ARRAY<ARRAY<BIGINT>> columns #3712 state: lowering the position of a full slab makes writeTo, writeInto and BytesInput.toByteArray() throw, and nothing is written to the output stream. Fails without this change.

The existing tests of parquet-common (346), parquet-encoding (34) and parquet-column (973) pass, as do the parquet-hadoop writer tests (TestParquetWriter*, TestColumnChunkPageWriteStore, TestParquetFileWriter, TestDataPageChecksums, TestStatistics, TestDirectCodecFactory).

Are there any user-facing changes?

A write that would have produced a corrupt page now fails with an IllegalStateException whose message gives both sizes.

Refs #3712. This does not close the issue, because the root cause is still unknown.

🤖 Generated with Claude Code