Skip to content

fix(core): complete native update partition and deletion vector paths - #969

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/native-update-ci-parity
Sep 26, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/native-update-ci-parity

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Complete the Rust core gaps exposed when enabling every native switch in PyPaimon update CI (apache/paimon#10185), in one PR.

The original Native CI run had 7056 passing tests and 8 failures, all in deletion-vector path tests. Enabling partitioned incremental updates also exposed the existing rejection of unchanged partition columns.

  • Write deletion vectors to configured bucket external paths or global-index.external-path, and persist the actual location in IndexFileMeta.external_path. Match Java's location precedence, round-robin, specific-fs, weight-robin and entropy-inject strategies, including Guava's entropy hash and comma-list parsing.
  • Reuse the existing legacy table/index DV resolver when preparing a subsequent delete. Keep the original manifest identity in removal entries, and always honor explicit external paths even after output options change.
  • Clean up staged deletion-vector files when a later bucket or output write fails.
  • Allow row-ID update inputs to carry unchanged partition columns. Normalize them using the existing core conversion rules and compare against typed partition metadata before writing any files. Reject actual partition moves; preserve SQL assignment restrictions, overlap checks, and prior staged messages.
  • Match complete-row upserts within each partition, even when the caller omits partition columns from the upsert keys. Keep source last-write-wins and target fan-out semantics in core.

No new public factory or Python-side execution engine is introduced. TIME/TIMESTAMP upsert key types and unsupported legacy partition layouts retain their existing limitations.

Verification

  • PyPaimon update, incremental row-ID, key-upsert, deletion-vector paths, native writer, Ray update and global-index update suites with all five native flags enabled: 853 passed, 26 subtests passed. Counts: row-ID 134, grouped 51, predicate 58, upsert 84, incremental 9; native plans 716, reads 1014, writes 19, commits 2.
  • Core table_update_test: 21 passed; new table_update_paths_test: 7 passed, including all external path strategies, repeated deletes, historical reads, explicit-path precedence, abort, legacy manifest identity, nullable/composite partitions and failure atomicity.
  • External-path unit selection: 19 passed.
  • DataFusion UPDATE/MERGE/append MERGE: 83 passed.
  • cargo clippy --locked -p pypaimon_rust -p paimon --all-targets -- -D warnings, formatting and diff checks passed.
  • Independent reviews covered partition semantics and DV persistence/cleanup. Corrected both reproduced configuration parsing findings before submission.

The paired Python PR activates these capabilities and pins its Native CI build to this Rust revision. Hosted Rust CI also passed all 14 checks on this revision: https://github.com/apache/paimon-rust/actions/runs/36243350815. The paired PyPaimon workflow also passed on commit 3b1a15e9abed20e6db8351bf0c0baee06ed08f46: https://github.com/apache/paimon/actions/runs/36243417621. Its full Native lane verified this exact Rust revision and finished with 7065 passed, 225 skipped, 0 failed. Native update counts were row-ID 136, grouped 51, predicate 62, upsert 85 and incremental 9.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 780bc79. No blocking findings.

Checked typed partition identity validation and partition-scoped upsert matching, preservation of legacy DV manifest identities, explicit-path precedence, and cleanup after a later bucket fails. Compared external-path placement and entropy hashing with the Java implementations.

Local validation against this exact Rust revision:

  • Core library: 3,391 passed, 6 ignored; table_update_test and table_update_paths_test: 28 passed.
  • Rebuilt the Python extension and ran the paired PyPaimon revision 3b1a15e9abed20e6db8351bf0c0baee06ed08f46 with all five native switches enabled: 853 passed, no skips. Native operation counters confirm the new update paths were exercised.
  • DataFusion UPDATE/MERGE regression suites: 83 passed.
  • 13 additional independent integration checks passed: actual Java reads of Rust-updated partition columns and externally stored DVs across all four bucket strategies plus the global-index path; historical reads after subsequent upserts; Java entropy-path comparison; abort preserving committed external files; typed/null partition identity and rejected moves; and cleanup when a later external bucket directory cannot be created.
  • cargo fmt --all -- --check passed.

LGTM.

@JingsongLi
JingsongLi merged commit 50725c8 into apache:main Sep 26, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants