[rust] Cache decoded BLOB indexes across readers

[XiaoHongbo-Hope]

Purpose

Opening the same immutable .blob file with a new reader currently reloads its footer and index, adding two object-store range reads each time.

This caches the decoded index across readers, following the bounded process-local cache used by PyPaimon in apache/paimon#9547.

Changes

• Cache up to 16 decoded BLOB indexes in a process-local LRU.
• Share immutable indexes with Arc; readers and BLOB payloads are not cached.
• Scope cache entries to the originating FileIO storage context.
• Keep cache locks out of asynchronous I/O.
• Verify cache reuse within one FileIO and isolation across independent storage contexts.

Trade-offs

• Cold reads are unchanged.
• Cache hits avoid two range requests per BLOB file.
• Up to 16 decoded indexes remain in memory.
• This relies on Paimon data file paths being immutable; there is no API or format change.

Tests

• cargo test -p paimon --lib blob (140 passed)
• cargo test -p paimon --lib file_io (48 passed)
• cargo clippy -p paimon --lib --tests -- -D warnings
• cargo fmt --all --check
• git diff --check

[QuakeWang]

The same path can refer to different files when two FileIO instances use different storage roots. I tested this with two BLOB files of the same size and row count: opening the second reused the first file’s index and returned NULL instead of its actual value. The cache key needs to distinguish the storage namespace too.

[XiaoHongbo-Hope]

The same path can refer to different files when two FileIO instances use different storage roots. I tested this with two BLOB files of the same size and row count: opening the second reused the first file’s index and returned NULL instead of its actual value. The cache key needs to distinguish the storage namespace too.

Thanks. I hesitated because Paimon files use UUID names, so two files within one storage namespace are extremely unlikely to have the same path, and PyPaimon also keys by path. I’ll fix the cache key and add a regression test.

[QuakeWang]

+1

[JingsongLi]

+1