fix(core): complete native update partition and deletion vector paths - #969
Merged
Merged
Conversation
leaves12138
approved these changes
Sep 26, 2026
leaves12138
left a comment
There was a problem hiding this comment.
Reviewed 780bc79. No blocking findings.
Checked typed partition identity validation and partition-scoped upsert matching, preservation of legacy DV manifest identities, explicit-path precedence, and cleanup after a later bucket fails. Compared external-path placement and entropy hashing with the Java implementations.
Local validation against this exact Rust revision:
- Core library: 3,391 passed, 6 ignored; table_update_test and table_update_paths_test: 28 passed.
- Rebuilt the Python extension and ran the paired PyPaimon revision 3b1a15e9abed20e6db8351bf0c0baee06ed08f46 with all five native switches enabled: 853 passed, no skips. Native operation counters confirm the new update paths were exercised.
- DataFusion UPDATE/MERGE regression suites: 83 passed.
- 13 additional independent integration checks passed: actual Java reads of Rust-updated partition columns and externally stored DVs across all four bucket strategies plus the global-index path; historical reads after subsequent upserts; Java entropy-path comparison; abort preserving committed external files; typed/null partition identity and rejected moves; and cleanup when a later external bucket directory cannot be created.
- cargo fmt --all -- --check passed.
LGTM.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Complete the Rust core gaps exposed when enabling every native switch in PyPaimon update CI (apache/paimon#10185), in one PR.
The original Native CI run had 7056 passing tests and 8 failures, all in deletion-vector path tests. Enabling partitioned incremental updates also exposed the existing rejection of unchanged partition columns.
global-index.external-path, and persist the actual location inIndexFileMeta.external_path. Match Java's location precedence, round-robin, specific-fs, weight-robin and entropy-inject strategies, including Guava's entropy hash and comma-list parsing.table/indexDV resolver when preparing a subsequent delete. Keep the original manifest identity in removal entries, and always honor explicit external paths even after output options change.No new public factory or Python-side execution engine is introduced. TIME/TIMESTAMP upsert key types and unsupported legacy partition layouts retain their existing limitations.
Verification
table_update_test: 21 passed; newtable_update_paths_test: 7 passed, including all external path strategies, repeated deletes, historical reads, explicit-path precedence, abort, legacy manifest identity, nullable/composite partitions and failure atomicity.cargo clippy --locked -p pypaimon_rust -p paimon --all-targets -- -D warnings, formatting and diff checks passed.The paired Python PR activates these capabilities and pins its Native CI build to this Rust revision. Hosted Rust CI also passed all 14 checks on this revision: https://github.com/apache/paimon-rust/actions/runs/36243350815. The paired PyPaimon workflow also passed on commit
3b1a15e9abed20e6db8351bf0c0baee06ed08f46: https://github.com/apache/paimon/actions/runs/36243417621. Its full Native lane verified this exact Rust revision and finished with 7065 passed, 225 skipped, 0 failed. Native update counts were row-ID 136, grouped 51, predicate 62, upsert 85 and incremental 9.