fix(writer): declare the real zone stride so vortex-jni prunes correctly (#418) - #421
Merged
Merged
Conversation
…tly (#418) The legacy vortex.stats zone map holds one zone per chunk but declared WriteOptions#chunkSize() as its zone length. Rust reads that length as a uniform stride (row r in zone r / len), so whenever batches were not exactly chunkSize rows a filtered vortex-jni read pruned on the wrong zone's stats: with default options and small batches it returned no rows at all. Declare the shared chunk length when every chunk but the last has it, and 0 otherwise, which Rust treats as "no stride" and reads unpruned. vortex-java's own reader places legacy zones on physical chunks and is unaffected. Closes #418 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dfa1
force-pushed
the
fix/418-zone-length
branch
from
October 1, 2026 20:08
c193f95 to
8087363
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #418. Stacked on #420. The filtered vortex-jni reads in these tests abort the JVM without #420's alignment fix. Merge #420 first; this PR then retargets to
main.Bug
Our legacy
vortex.statszone map holds one zone per chunk (one chunk perwriteChunkcall), but declaredWriteOptions#chunkSize()(default 65 536) as its zone length. Rust reads the declared length as a uniform stride: rowrbelongs to zoner / zone_len(ZonedReader::zone_range). So a filtered vortex-jni read pruned using the wrong zone's stats:[5,5,5,3], default optionsid >= 7[]7..17[4,4,2,4]id >= 10[]10..13With small batches every row maps to zone 0, whose max is chunk 0's, so everything gets pruned.
Fix
ZoneMapStatCodec.uniformZoneLength(rowCounts):0, Rust's "no stride" (LegacyStatsLayout::zoned_reader). Rust then reads the column unpruned, which is correct and only slower.vortex-java's reader doesn't use the declared length for this layout; it places legacy zones on the physical chunks (#419). So nothing changes on our side.
Not done
vortex.zoned(what Writer emits the legacy vortex.stats zone-map layout; Rust writes vortex.zoned with aggregate-spec metadata #418 originally asked for): not needed for correctness.vortex.statsstays the more compatible choice for older readers.WriteOptions.chunkSizeis now unused. Its only effect was this wrong zone length;writeChunknever splits batches. It should either be removed or start splitting batches; that's a separate decision.Tests
ZoneMapStatCodecTest.UniformZoneLength: 9 shape cases (short tail, short middle chunk, longer tail, empty tail, single chunk, no rows, u32 overflow).javaWriter_jniReader_zoneMapped_filterKeepsEveryMatch: three chunk shapes through a filtered vortex-jni read. The two multi-chunk shapes return[]with the previous writer (verified by reverting it). All pass now../mvnw verifyis green.🤖 Generated with Claude Code