Skip to content

fix(core): close gaps for PyPaimon native Parquet writers - #970

Merged
JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/native-writer-capabilities
Sep 27, 2026
Merged

JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/native-writer-capabilities

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Close the remaining core and binding gaps exposed by enabling PyPaimon native Parquet writers for data-evolution tables, merge engines, nested input and primary-key value statistics.

Companion PyPaimon PR: apache/paimon#10202 (Native CI follows Rust main, so this Rust PR merges first).

Brief change log

  • Delegate Python batch validation and normalization to TableWrite, allowing compatible nested child names and fixed-size binary input. Preserve schema-error recovery for the shared format-table writer path.
  • Apply the core row-kind filter to both generated and explicit _VALUE_KIND input before bucketing and changelog writing. Accept global ignore-delete and Java fallback aliases for aggregation, and allow rowkind.field with supported merge engines.
  • Reuse FirstRowMergeFunction for multi-row key groups during flush so every retract is checked. Preserve Java reducer behavior for singleton groups and the insert-only fast path.
  • Honor parquet.write-page-index.enabled for both data and changelog files, retaining row-group statistics when page indexes are disabled and rejecting invalid settings before output creation.
  • Parse bare VARCHAR, BINARY and VARBINARY names with Java's default length of 1, including nested Variant shredding schemas.

Tests

  • Rust targeted tests: KV writer (27), aggregation options (21), row-kind schema validation (7), row-kind integration (21), table writer (69), types (22), Variant shredding (7), Parquet writer and physical data/changelog page-index regression.
  • Python binding write tests: 29 passed, including normalization, explicit row kinds and schema-error recovery.
  • cargo fmt --all --check.
  • cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings.
  • PyPaimon integration results will be recorded in the companion PR.

API and Format

Python write signatures remain unchanged; normalization and filtering live in Rust core. Adds a CoreOptions accessor for the existing Parquet option. Storage remains standard Paimon/Parquet.

Documentation

No new user-facing option: this implements existing Java/PyPaimon contracts.

@leaves12138

Copy link
Copy Markdown

Pushed 0684f18 to fix the two stale unit-test assertions without reverting the intended implementation behavior:

  • The aggregation schema rejection test now checks genuinely unsupported options rather than the newly supported global ignore-delete option.
  • The Format Table test now expects an empty successful prepare after a rejected schema. Added a regression proving that valid rows accepted before and after an invalid batch are preserved and committed.

Validation: cargo test --locked -p paimon --lib passed (3398 passed, 6 ignored); the RowKind and physical Parquet page-index integration tests passed (22 tests); cargo fmt --all --check and git diff --check passed. The new GitHub CI run is still in progress.

@QuakeWang QuakeWang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after re-review of 0684f18. The two outdated assertions are corrected, and the additional Format Table regression verifies that a schema error preserves previously accepted rows and allows subsequent valid writes. No further blocking issues found in the reviewed RowKind filtering, first-row reducer, Parquet page-index, type-default, or Python binding changes. Verified all 14 checks are successful; the Ubuntu core suite reports 3496 passed and 6 ignored, and the Python binding suite reports 283 passed, including all 29 write tests. The previously failing tests and the new recovery test all pass in CI.

@JingsongLi
JingsongLi merged commit 5c87ec5 into apache:main Sep 27, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants