Skip to content

datalake_agent: the gRPC contract between the extension and the agent - #2071

Draft
MisterRaindrop wants to merge 1 commit into
apache:mainfrom
MisterRaindrop:feature/datalake-proto
Draft

MisterRaindrop wants to merge 1 commit into
apache:mainfrom
MisterRaindrop:feature/datalake-proto

Conversation

@MisterRaindrop

Copy link
Copy Markdown
Contributor

What does this PR do?

The contract between datalake_fdw and datalake_agent: the proto files
both sides will build from, before either has code that uses them. Nothing
here is compiled into the extension yet, and the PostgreSQL build is
unchanged.

Closes #2010. Part of #2008 (B0).

  • contrib/datalake_agent/proto/: common.proto, iceberg_catalog.proto
    (IcebergCatalogService), catalog_mgmt.proto
    (CatalogManagementService), and a README.
  • contrib/datalake_fdw/src/meta/fragment.proto: fragments, written-file
    reports and pushed-down predicates -- what the C side reads and writes
    directly, so it lives with the C side, in the same package.
  • .github/workflows/datalake-proto.yml: runs the C++ and the Java gRPC
    generators on every change to these files.

Decisions worth a look:

  • One RPC per IcebergMetaEngine entry, and the README maps them in both
    directions. alter_table and truncate_table have none yet: an engine
    without them leaves the capability bits unset and the central dispatch
    answers DL_ERR_NOT_SUPPORTED without calling it. Each gets its RPC in the
    change that implements it.
  • What a statement acts on is fixed in the request, not resolved on
    arrival.
    LoadTable and CreateTable return the exact metadata location
    and the table UUID; GetFragment plans against that location; every write
    and DropTable must name the UUID, so a table dropped and recreated under
    the same name is never the one written to or purged; UPDATE, DELETE and
    rewrite commits name the snapshot they were planned against, so conflict
    detection starts there and not at whatever is current at commit time.
  • Commit state has one authority, the three-way CommitOutcome. A commit
    whose answer was lost is neither applied nor not; a boolean would have to
    lie one way or the other.
  • CommitFileGroups streams both ways, one group per message, so a
    rewrite of any size stays under gRPC's message limit.
  • Versions are checked by both sides. The client sends
    x-cloudberry-client-version; every response -- streamed batches included
    -- carries the server's version and the oldest client it accepts. The
    server refuses a client that is too old, and the client refuses a server
    that is too old for it: a field an older server does not know arrives as
    zero, and for several fields here zero means "none".
  • Errors end in a google.rpc.Status with an ErrorDetail whose
    business_code a client branches on; no stack trace leaves the server.
  • Temporal literals count from the Unix epoch, as Iceberg's do. The field
    comments give the offset from PostgreSQL's 2000-01-01, because a predicate
    off by 10957 days prunes the wrong files and no recheck can bring them back.
  • Nothing generated is committed. Each side runs protoc at build time
    with both directories as include roots.

Type of Change

  • Bug fix (non-breaking change)
  • New feature (non-breaking change)
  • Breaking change (fix or feature with breaking changes)
  • Documentation update

Test Plan

  • The workflow's steps, extracted from the YAML and run unchanged in
    ubuntu:24.04: C++ with the distribution's protoc 3.21.12 and
    grpc_cpp_plugin, Java with protoc 3.25.5 and protoc-gen-grpc-java
    1.81.0 -- the versions the agent's Maven build will pin -- all with
    --fatal_warnings. Clean on both.
  • The checks fail when they should: an unused import fails the run under
    --fatal_warnings, and a file without java_package fails the
    package assertion (its classes land in cloudberry/datalake/v1/).
  • Apache RAT: every new file is AL, 0 unknown.
  • Unit tests added/updated (nothing to run yet; B1 and B2 bring them)
  • Passed make installcheck (no C change)

Impact

Dependencies: none for the PostgreSQL build. The workflow downloads two
generators from Maven Central, pinned by version and checked against SHA-256
digests that match Maven Central's published SHA-1.

User-facing changes: none.

Checklist

Additional Context

Known limit, in the README: AppendResponse and UpdateResponse carry
outcome and committed_metadata_location, which only a commit can
truthfully report. Append and Update stage; the table changes at
CommitAppend and CommitUpdate, and commit state is read from those.

B1 (#2011, the Java service framework) builds on this and follows as its own PR.

The proto files datalake_fdw and datalake_agent agree on, before either
side has code that uses them.  IcebergCatalogService has one RPC per
IcebergMetaEngine entry the agent will carry; CatalogManagementService
creates and lists catalogs and namespaces; fragment.proto holds the
messages the C side reads and writes directly -- fragments, written-file
reports, pushed-down predicates -- and so lives with it, in the same
package.

alter_table and truncate_table have no RPC yet: an engine without them
leaves the capability unset and the dispatch refuses the call.  Each
gets its RPC in the change that implements it.

What a statement acts on is fixed in the request rather than resolved
when the request arrives.  LoadTable and CreateTable return the exact
metadata location and the table's UUID; GetFragment plans against that
location; every write and DropTable must name the UUID, so a table
dropped and recreated under the same name is never the one written to
or purged; UPDATE, DELETE and rewrite commits name the snapshot they
were planned against.  Commit state has one authority, the three-way
outcome, since a commit whose answer was lost is neither applied nor
not.  CommitFileGroups streams both ways, one group per message, so a
rewrite of any size fits gRPC's message limit.

Errors carry an ErrorDetail whose business_code a client branches on.
The client sends its version in x-cloudberry-client-version; every
response, streamed batches included, carries the server's version and
the oldest client it accepts, and each side refuses the other when it
is too old.  Temporal literals count from the Unix epoch, as Iceberg's
do, not PostgreSQL's.

Nothing generated is committed.  A workflow runs the C++ and the Java
gRPC generators on every change to these files, with warnings fatal,
and fails if any Java class lands outside the agent's package; the
Java generators are pinned and their digests checked.
@MisterRaindrop MisterRaindrop added the datalake contrib/datalake_fdw and contrib/datalake_agent: Iceberg lake tables label Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

datalake contrib/datalake_fdw and contrib/datalake_agent: Iceberg lake tables

Projects

None yet

Development

Successfully merging this pull request may close these issues.

datalake_agent: the gRPC contract (proto files)

1 participant