Skip to content

feat(table): expire snapshots after commit - #966

Open
zhuxiangyi wants to merge 5 commits into
apache:mainfrom
zhuxiangyi:feat/expire-after-commit
Open

zhuxiangyi wants to merge 5 commits into
apache:mainfrom
zhuxiangyi:feat/expire-after-commit

Conversation

@zhuxiangyi

@zhuxiangyi zhuxiangyi commented Sep 26, 2026 •

Copy link
Copy Markdown

Purpose

Linked issue: #964 (part 2 of 4)

Stacked on #965. Only the last three commits, starting with "feat(table): expire snapshots after
commit", are new here.

Java runs snapshot expiration after each commit (TableCommitImpl#maintain), which is what keeps
tables from growing without bound. paimon-rust commits never did. With the core expiration from
#965, this PR runs expiration after every commit, like Java does.

Brief change log

  • TableCommit calls maintain() after every successful commit: append, overwrite, partition
    truncate or drop, table truncate, filter_and_commit, and the commit_if_latest_snapshot used
    by global index builds. maintain() runs ExpireSnapshots with the table
    options. Like Java's expireForEmptyCommit default, this does not depend on whether the commit
    created a snapshot.
  • Skipped when:
    • write-only is set (falling back to write.compaction-skip), as in Java;
    • the changelog lifecycle is decoupled (changelog.num-retained.min/max or
      changelog.time-retained above the snapshot settings). Java then moves expired snapshots into
      long-lived changelogs, and paimon-rust has no changelog manager yet. Expiring here would drop
      changelogs the table is configured to keep.
  • While the table has no more snapshots than snapshot.num-retained.min, nothing can expire, so
    maintain() returns after reading the LATEST hint. Commits to a young table then skip the
    consumer listing and the EARLIEST lookup. This also keeps commits from listing the snapshot
    directory, which opendal's local lister cannot do for the 8.3 short temporary paths on Windows
    CI.
  • The commit has already succeeded when expiration runs, so an expiration error is logged and
    retried by the next commit instead of failing the commit. Java's batch mode would surface it.
    Returning an error for a commit that did publish its snapshot invites callers to retry and
    duplicate data.
  • New CoreOptions: write_only() and changelog_lifecycle_decoupled().
  • With the defaults (keep at least 10 snapshots, and anything younger than 1 h), short-lived
    tables and existing tests are unaffected.

Tests

  • commit_expires_snapshots: with snapshot.num-retained.max = 2, four commits leave snapshots
    3 and 4. The file-set invariant from feat(table): expire snapshots and add sys.expire_snapshots #965 holds and the data is correct.
  • commit_keeps_recent_snapshots_by_default: 12 quick commits keep every snapshot.
  • commit_skips_expiration: write-only, write.compaction-skip, and a decoupled changelog each
    keep every snapshot.
  • failed_expiration_does_not_fail_the_commit: an invalid retention option does not fail
    commits.
  • commit_does_not_look_for_expirable_snapshots_in_a_young_table: 10 commits don't attempt
    expiration (no EARLIEST hint is written); the 11th does.
  • every_commit_path_expires: table truncate, drop partition, filter_and_commit, and a global
    index build each leave only the latest snapshot.
  • Unit tests for changelog_lifecycle_decoupled and write_only.
  • Two tests from feat(table): expire snapshots and add sys.expire_snapshots #965 now set write-only, because they need commits not to expire.
  • cargo test -p paimon --all-targets --features fulltext,vortex passes (3793 tests), and so does
    clippy with -D warnings. paimon-datafusion --all-targets passes except for tests that need
    the Spark-provisioned tables (make docker-up) or the Lumina native library, which fail the same
    way on main locally.

API and Format

No new public API beyond two CoreOptions accessors. Behavior change: commits now expire
snapshots according to the table's retention options, as in Java.

Documentation

docs/src/sql.md (expire_snapshots section) describes automatic expiration and how to turn it
off with write-only.

Port Java's ExpireSnapshotsImpl and SnapshotDeletion. Snapshots are
chosen with snapshot.num-retained.min/max, snapshot.time-retained (a
snapshot expires once its successor is older than the cut-off),
consumer protection, and snapshot.expire.limit. Expiring [earliest,
end) deletes data files removed by the deltas of (earliest, end] that
the closest earlier tag does not read, changelog files, and manifest
lists, manifests, index files, statistics and reassign plans that
neither end nor a tag in the range references, then the snapshot
files and finally moves the EARLIEST hint. Read failures only ever
make a run delete less.

Expose it as Table::new_expire_snapshots() and the DataFusion
procedure sys.expire_snapshots with Java's arguments.
…ation

Extend the file-set invariant to changelog files and index files kept
in bucket directories, and add cases for deletion-vector files in both
index layouts, changelog and hash index files of a primary-key table,
external data files, unreadable deltas, tags and skipping sets (each
must delete less, never more), the closest earlier tag among several,
a snapshot missing from the range, the slowest of several consumers,
and older_than given as a timestamp string.
Like Java's TableCommitImpl, run snapshot expiration with the table
options after every commit, append, overwrite and truncate alike. It
is skipped for write-only tables (write-only, falling back to
write.compaction-skip) and while changelogs are configured to outlive
snapshots, which needs a changelog manager first. The commit has
already succeeded when expiration runs, so a failure is logged and
retried by the next commit instead of failing the commit.
Global index builds commit through commit_if_latest_snapshot, which
skipped the post-commit maintenance. Run it there too, and cover every
commit path (truncate, drop partition, filter_and_commit and an index
build) with a test.
@zhuxiangyi
zhuxiangyi force-pushed the feat/expire-after-commit branch from db14497 to e006561 Compare September 26, 2026 13:11
While a table has no more snapshots than snapshot.num-retained.min,
no snapshot can expire. Check that from the LATEST hint before running
expiration, so commits to a young table no longer list consumers or
look up the earliest snapshot. Besides the saved I/O, this keeps
commits from listing the snapshot directory, which opendal's local
lister cannot do for Windows short temp paths.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant