Use Rust core for native Arrow upserts and table updates

[JingsongLi]

Summary

Enable native update CI and wrap the public Rust core table update APIs. Based on merged apache/paimon-rust#961 and the follow-up core fixes in apache/paimon-rust#969. Native CI is pinned to Rust revision 780bc7957f24a78828b7910369e079bbff30e92c.

• Delegate direct/grouped row-ID updates, predicate matching and assignments, complete-row key upserts, and supported row-ID deletes to core. Core owns snapshot/file indexing, input normalization, matching, overlap detection, file preparation and cleanup; Python converts values/commit messages and runs user callbacks when invoked by core.
• Adapt new_update_by_row_id() to the incremental core update_columns(data, columns) and cumulative commit_messages API for batch and stream tables.
• Enable partitioned incremental updates and supported complete-row partitioned upserts. Matching is scoped to the input partition even when partition columns are omitted from the upsert keys. Carrying unchanged partition values succeeds; actual row-ID partition changes are rejected in core.
• Keep Python execution for unsupported types, partial row objects, row-aware Blob operations, custom/legacy partition paths and precomputed Python indexes. Ray planning uses RowIdFileIndex directly instead of constructing an updater.
• Remove update API compatibility probes/version skips. Set PYPAIMON_TEST_NATIVE_UPDATE=1 together with native plan/read/write/commit, and fail the native lane if row-ID, grouped, predicate, upsert or incremental operations are not exercised.
• Make the legacy DV fixture explicitly use the old Python writer for its initial index file; all subsequent eligible operations run natively.

No private matcher or matched-writer bridge is exposed by this integration. The Rust binding wraps the core operation interfaces. Some PyPaimon operations, including MERGE/shard and predicate-delete orchestration, remain in Python.

Test findings and paired Rust fixes

The first complete hosted Native run had 7056 passed, 8 failed, 225 skipped. All eight failures concerned external/legacy deletion-vector paths. The paired Rust PR fixes these paths and manifest identities, output cleanup, unchanged partition-column validation and partition-scoped upserts. It also includes Java-compatible external path strategy and comma-list parsing regressions.

Verification

With the rebuilt paired Rust binding, PyArrow 19.0.1 and all five native switches enabled:

• Update, chunked row-ID, key-upsert, deletion-vector path, native writer, Ray update and global-index update suites: 853 passed, 26 subtests passed. Native calls: row-ID 134, grouped 51, predicate 58, upsert 84, incremental 9; plans 716, reads 1014, writes 19, commits 2. Three existing Ray warnings remain.
• Earlier activation check of Ray MERGE, row-ID range planning/read and conflict rewriting: 205 passed, 33 subtests passed.
• Paired core update suites: 28 passed; external-path unit selection: 19 passed; DataFusion UPDATE/MERGE: 83 passed.
• Configured Python flake8, Rust Clippy (--all-targets -- -D warnings), formatting and diff checks passed.
• Independent reviews covered partition semantics, DV persistence, historical reads and rollback. Both reproduced configuration parsing findings were fixed in the paired Rust PR.

Hosted CI result

Both workflows passed:

• Python PR at 3b1a15e9abed20e6db8351bf0c0baee06ed08f46: https://github.com/apache/paimon/actions/runs/36243417621. All Python version, requirements/Ray, PyTorch, Native and aggregate checks passed.
• The full Native lane verified Rust commit 780bc7957f24a78828b7910369e079bbff30e92c and finished with 7065 passed, 225 skipped, 0 failed. Counts: plans 3766, reads 3784, writes 7837, commits 503; updates row-ID 136, grouped 51, predicate 62, upsert 85, incremental 9.
• Paired Rust PR: all 14 checks passed at https://github.com/apache/paimon-rust/actions/runs/36243350815.

[leaves12138]

LGTM. Reviewed the native/Python dispatch boundaries, commit identity and message conversion, snapshot handling, and Ray RowIdFileIndex integration. An additional 805 tests passed with native plan/read/write/update/commit enabled, covering Ray MERGE, conflict rewriting, Blob fallbacks, and native operations. Also checked the existing same-revision paired Rust validation, including Java interoperability. No blocking issues found.