feat(table): expire snapshots after commit - #966
Open
zhuxiangyi wants to merge 5 commits into
Open
zhuxiangyi wants to merge 5 commits into
zhuxiangyi wants to merge 5 commits into
Conversation
Port Java's ExpireSnapshotsImpl and SnapshotDeletion. Snapshots are chosen with snapshot.num-retained.min/max, snapshot.time-retained (a snapshot expires once its successor is older than the cut-off), consumer protection, and snapshot.expire.limit. Expiring [earliest, end) deletes data files removed by the deltas of (earliest, end] that the closest earlier tag does not read, changelog files, and manifest lists, manifests, index files, statistics and reassign plans that neither end nor a tag in the range references, then the snapshot files and finally moves the EARLIEST hint. Read failures only ever make a run delete less. Expose it as Table::new_expire_snapshots() and the DataFusion procedure sys.expire_snapshots with Java's arguments.
6 tasks
…ation Extend the file-set invariant to changelog files and index files kept in bucket directories, and add cases for deletion-vector files in both index layouts, changelog and hash index files of a primary-key table, external data files, unreadable deltas, tags and skipping sets (each must delete less, never more), the closest earlier tag among several, a snapshot missing from the range, the slowest of several consumers, and older_than given as a timestamp string.
Like Java's TableCommitImpl, run snapshot expiration with the table options after every commit, append, overwrite and truncate alike. It is skipped for write-only tables (write-only, falling back to write.compaction-skip) and while changelogs are configured to outlive snapshots, which needs a changelog manager first. The commit has already succeeded when expiration runs, so a failure is logged and retried by the next commit instead of failing the commit.
Global index builds commit through commit_if_latest_snapshot, which skipped the post-commit maintenance. Run it there too, and cover every commit path (truncate, drop partition, filter_and_commit and an index build) with a test.
zhuxiangyi
force-pushed
the
feat/expire-after-commit
branch
from
September 26, 2026 13:11
db14497 to
e006561
Compare
While a table has no more snapshots than snapshot.num-retained.min, no snapshot can expire. Check that from the LATEST hint before running expiration, so commits to a young table no longer list consumers or look up the earliest snapshot. Besides the saved I/O, this keeps commits from listing the snapshot directory, which opendal's local lister cannot do for Windows short temp paths.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: #964 (part 2 of 4)
Java runs snapshot expiration after each commit (
TableCommitImpl#maintain), which is what keepstables from growing without bound. paimon-rust commits never did. With the core expiration from
#965, this PR runs expiration after every commit, like Java does.
Brief change log
TableCommitcallsmaintain()after every successful commit: append, overwrite, partitiontruncate or drop, table truncate,
filter_and_commit, and thecommit_if_latest_snapshotusedby global index builds.
maintain()runsExpireSnapshotswith the tableoptions. Like Java's
expireForEmptyCommitdefault, this does not depend on whether the commitcreated a snapshot.
write-onlyis set (falling back towrite.compaction-skip), as in Java;changelog.num-retained.min/maxorchangelog.time-retainedabove the snapshot settings). Java then moves expired snapshots intolong-lived changelogs, and paimon-rust has no changelog manager yet. Expiring here would drop
changelogs the table is configured to keep.
snapshot.num-retained.min, nothing can expire, somaintain()returns after reading the LATEST hint. Commits to a young table then skip theconsumer listing and the EARLIEST lookup. This also keeps commits from listing the snapshot
directory, which opendal's local lister cannot do for the 8.3 short temporary paths on Windows
CI.
retried by the next commit instead of failing the commit. Java's batch mode would surface it.
Returning an error for a commit that did publish its snapshot invites callers to retry and
duplicate data.
CoreOptions:write_only()andchangelog_lifecycle_decoupled().tables and existing tests are unaffected.
Tests
commit_expires_snapshots: withsnapshot.num-retained.max = 2, four commits leave snapshots3 and 4. The file-set invariant from feat(table): expire snapshots and add sys.expire_snapshots #965 holds and the data is correct.
commit_keeps_recent_snapshots_by_default: 12 quick commits keep every snapshot.commit_skips_expiration:write-only,write.compaction-skip, and a decoupled changelog eachkeep every snapshot.
failed_expiration_does_not_fail_the_commit: an invalid retention option does not failcommits.
commit_does_not_look_for_expirable_snapshots_in_a_young_table: 10 commits don't attemptexpiration (no EARLIEST hint is written); the 11th does.
every_commit_path_expires: table truncate, drop partition,filter_and_commit, and a globalindex build each leave only the latest snapshot.
changelog_lifecycle_decoupledandwrite_only.write-only, because they need commits not to expire.cargo test -p paimon --all-targets --features fulltext,vortexpasses (3793 tests), and so doesclippy with
-D warnings.paimon-datafusion --all-targetspasses except for tests that needthe Spark-provisioned tables (
make docker-up) or the Lumina native library, which fail the sameway on
mainlocally.API and Format
No new public API beyond two
CoreOptionsaccessors. Behavior change: commits now expiresnapshots according to the table's retention options, as in Java.
Documentation
docs/src/sql.md(expire_snapshotssection) describes automatic expiration and how to turn itoff with
write-only.