feat(table): remove orphan files and add sys.remove_orphan_files

[zhuxiangyi]

Purpose

Linked issue: #964 (part 4 of 4)

Stacked on #965, which provides the shared helpers (try_get_snapshot, parse_older_than,
reassign-plan detection). Only the last three commits are new here: "feat(table): remove orphan
files and add sys.remove_orphan_files" and two test follow-ups.

Failed or interrupted writes leave files that no snapshot references, and nothing in paimon-rust
removes them. This PR ports Java's LocalOrphanFilesClean and adds CALL sys.remove_orphan_files.

Brief change log

• Table::new_remove_orphan_files() / RemoveOrphanFiles (table/orphan_files_clean.rs):

  • Candidates: files last modified before older_than in these places:

    • the manifest, index and statistics directories;
    • every bucket directory, walking partition_keys levels of key=value directories, as
      Java's listFileDirs does;
    • bucket directories under data-file.external-paths.

  • Non-snapshot files in each branch's snapshot and changelog directories (temporary files of
    interrupted commits) are removed too, as in Java's cleanBranchSnapshotDir.

  • Used files: everything referenced by the snapshots, tags, and long-lived changelogs
    (changelog/changelog-*, the snapshot JSON format) of every branch:

    • manifest lists, manifests and their extra files;
    • index manifests and index files;
    • statistics files and reassign plans;
    • data files and their extra files from all manifest entries.

    Files are matched by name, as in Java.

  • Safety:

    • older_than defaults to one day ago and must be in the past;
    • a store that reports no modification time never exposes a file;
    • a branch without a schema aborts the run (Java);
    • managed BLOB packs (*.managed.blob) are skipped (Java);
    • directories are never deleted.

  • Unlike Java, which reads a missing manifest as empty, a missing metadata file of a snapshot or
    changelog that still exists aborts the run. Otherwise that snapshot's data files would look
    unreferenced and be deleted. If the owning snapshot vanished at the same time (concurrent
    expiration), it is simply skipped.

  • dry_run reports without deleting; parallelism bounds concurrent reads and deletions.

• DataFusion CALL sys.remove_orphan_files(table, older_than, dry_run, parallelism, mode) returns
  deletedFileCount and deletedFileTotalLenInBytes, like Java. It supports mode => 'local' and
  one table per call; db.* returns a clear "not supported yet" error.
• Not included: removing empty directories. It is left for later, because on object stores
  "check empty, then delete" races with writers.

Tests

• table::orphan_files_clean::tests (13 tests) run on a real local file system, because they need
  modification times. Like the repository's other local-listing tests, they are skipped on Windows,
  where CI's 8.3 short temporary paths make opendal's local lister panic:
  • planted orphans (a data file, a manifest, an index file, a statistics file, and a temporary
    snapshot file) are found by a dry run, then removed. Afterwards the directory holds exactly
    the referenced files, the table reads correctly, and a second run finds nothing;
  • recent files are kept, and a future older_than is rejected;
  • after a snapshot is removed without cleanup, only its own manifest lists become orphans. Its
    data file is still named by the next delta's DELETE entry, as in Java;
  • a tag, a branch, and a long-lived changelog each protect their files; once the changelog is
    removed, its files become orphans;
  • partitioned bucket directories are scanned, and managed BLOB packs are skipped;
  • a missing manifest of a live snapshot aborts the run with nothing deleted;
  • a branch without a schema aborts the run;
  • deletion vectors in bucket directories and global index files that snapshots reference are
    kept, while orphans planted next to them are removed;
  • data-file.external-paths directories are scanned, keeping referenced files and removing
    orphans;
  • two levels of partition directories are walked;
  • temporary changelog files are removed while hint files stay;
  • a snapshot that vanished concurrently is skipped, while a live snapshot or a tag with a missing
    manifest aborts the run.
• Each of these changes makes a test fail: ignoring branches, not counting referenced index files
  as used, not scanning external paths, and walking only one partition level.
• DataFusion tests/procedures.rs:
  • an end-to-end test: the default cut-off keeps a fresh orphan; a dry run reports it; a real run
    with parallelism and mode => 'local' removes it and the table stays readable;
  • argument validation: db.*, mode => 'distributed', a future older_than, a bad dry_run,
    and parallelism => 0.
• cargo test -p paimon --all-targets --features fulltext,vortex passes (3835 tests), and so does
  clippy with -D warnings.

API and Format

New public API: Table::new_remove_orphan_files(), RemoveOrphanFiles, OrphanFilesCleanResult;
new sys.remove_orphan_files procedure. No format change.

Documentation

docs/src/sql.md documents remove_orphan_files: its arguments, what counts as an orphan, and
the safety rules.