fix(core): close gaps for PyPaimon native Parquet writers - #970
Conversation
|
Pushed 0684f18 to fix the two stale unit-test assertions without reverting the intended implementation behavior:
Validation: |
leaves12138
left a comment
There was a problem hiding this comment.
LGTM after re-review of 0684f18. The two outdated assertions are corrected, and the additional Format Table regression verifies that a schema error preserves previously accepted rows and allows subsequent valid writes. No further blocking issues found in the reviewed RowKind filtering, first-row reducer, Parquet page-index, type-default, or Python binding changes. Verified all 14 checks are successful; the Ubuntu core suite reports 3496 passed and 6 ignored, and the Python binding suite reports 283 passed, including all 29 write tests. The previously failing tests and the new recovery test all pass in CI.
Purpose
Close the remaining core and binding gaps exposed by enabling PyPaimon native Parquet writers for data-evolution tables, merge engines, nested input and primary-key value statistics.
Companion PyPaimon PR: apache/paimon#10202 (Native CI follows Rust main, so this Rust PR merges first).
Brief change log
TableWrite, allowing compatible nested child names and fixed-size binary input. Preserve schema-error recovery for the shared format-table writer path._VALUE_KINDinput before bucketing and changelog writing. Accept globalignore-deleteand Java fallback aliases for aggregation, and allowrowkind.fieldwith supported merge engines.FirstRowMergeFunctionfor multi-row key groups during flush so every retract is checked. Preserve Java reducer behavior for singleton groups and the insert-only fast path.parquet.write-page-index.enabledfor both data and changelog files, retaining row-group statistics when page indexes are disabled and rejecting invalid settings before output creation.VARCHAR,BINARYandVARBINARYnames with Java's default length of 1, including nested Variant shredding schemas.Tests
cargo fmt --all --check.cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings.API and Format
Python write signatures remain unchanged; normalization and filtering live in Rust core. Adds a
CoreOptionsaccessor for the existing Parquet option. Storage remains standard Paimon/Parquet.Documentation
No new user-facing option: this implements existing Java/PyPaimon contracts.