datalake_agent: the gRPC contract between the extension and the agent - #2071
Draft
MisterRaindrop wants to merge 1 commit into
Draft
MisterRaindrop wants to merge 1 commit into
MisterRaindrop wants to merge 1 commit into
Conversation
The proto files datalake_fdw and datalake_agent agree on, before either side has code that uses them. IcebergCatalogService has one RPC per IcebergMetaEngine entry the agent will carry; CatalogManagementService creates and lists catalogs and namespaces; fragment.proto holds the messages the C side reads and writes directly -- fragments, written-file reports, pushed-down predicates -- and so lives with it, in the same package. alter_table and truncate_table have no RPC yet: an engine without them leaves the capability unset and the dispatch refuses the call. Each gets its RPC in the change that implements it. What a statement acts on is fixed in the request rather than resolved when the request arrives. LoadTable and CreateTable return the exact metadata location and the table's UUID; GetFragment plans against that location; every write and DropTable must name the UUID, so a table dropped and recreated under the same name is never the one written to or purged; UPDATE, DELETE and rewrite commits name the snapshot they were planned against. Commit state has one authority, the three-way outcome, since a commit whose answer was lost is neither applied nor not. CommitFileGroups streams both ways, one group per message, so a rewrite of any size fits gRPC's message limit. Errors carry an ErrorDetail whose business_code a client branches on. The client sends its version in x-cloudberry-client-version; every response, streamed batches included, carries the server's version and the oldest client it accepts, and each side refuses the other when it is too old. Temporal literals count from the Unix epoch, as Iceberg's do, not PostgreSQL's. Nothing generated is committed. A workflow runs the C++ and the Java gRPC generators on every change to these files, with warnings fatal, and fails if any Java class lands outside the agent's package; the Java generators are pinned and their digests checked.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
The contract between
datalake_fdwanddatalake_agent: the proto filesboth sides will build from, before either has code that uses them. Nothing
here is compiled into the extension yet, and the PostgreSQL build is
unchanged.
Closes #2010. Part of #2008 (B0).
contrib/datalake_agent/proto/:common.proto,iceberg_catalog.proto(
IcebergCatalogService),catalog_mgmt.proto(
CatalogManagementService), and a README.contrib/datalake_fdw/src/meta/fragment.proto: fragments, written-filereports and pushed-down predicates -- what the C side reads and writes
directly, so it lives with the C side, in the same package.
.github/workflows/datalake-proto.yml: runs the C++ and the Java gRPCgenerators on every change to these files.
Decisions worth a look:
IcebergMetaEngineentry, and the README maps them in bothdirections.
alter_tableandtruncate_tablehave none yet: an enginewithout them leaves the capability bits unset and the central dispatch
answers
DL_ERR_NOT_SUPPORTEDwithout calling it. Each gets its RPC in thechange that implements it.
arrival.
LoadTableandCreateTablereturn the exact metadata locationand the table UUID;
GetFragmentplans against that location; every writeand
DropTablemust name the UUID, so a table dropped and recreated underthe same name is never the one written to or purged; UPDATE, DELETE and
rewrite commits name the snapshot they were planned against, so conflict
detection starts there and not at whatever is current at commit time.
CommitOutcome. A commitwhose answer was lost is neither applied nor not; a boolean would have to
lie one way or the other.
CommitFileGroupsstreams both ways, one group per message, so arewrite of any size stays under gRPC's message limit.
x-cloudberry-client-version; every response -- streamed batches included-- carries the server's version and the oldest client it accepts. The
server refuses a client that is too old, and the client refuses a server
that is too old for it: a field an older server does not know arrives as
zero, and for several fields here zero means "none".
google.rpc.Statuswith anErrorDetailwhosebusiness_codea client branches on; no stack trace leaves the server.comments give the offset from PostgreSQL's 2000-01-01, because a predicate
off by 10957 days prunes the wrong files and no recheck can bring them back.
protocat build timewith both directories as include roots.
Type of Change
Test Plan
ubuntu:24.04: C++ with the distribution'sprotoc3.21.12 andgrpc_cpp_plugin, Java withprotoc3.25.5 andprotoc-gen-grpc-java1.81.0 -- the versions the agent's Maven build will pin -- all with
--fatal_warnings. Clean on both.--fatal_warnings, and a file withoutjava_packagefails thepackage assertion (its classes land in
cloudberry/datalake/v1/).make installcheck(no C change)Impact
Dependencies: none for the PostgreSQL build. The workflow downloads two
generators from Maven Central, pinned by version and checked against SHA-256
digests that match Maven Central's published SHA-1.
User-facing changes: none.
Checklist
Additional Context
Known limit, in the README:
AppendResponseandUpdateResponsecarryoutcomeandcommitted_metadata_location, which only a commit cantruthfully report.
AppendandUpdatestage; the table changes atCommitAppendandCommitUpdate, and commit state is read from those.B1 (#2011, the Java service framework) builds on this and follows as its own PR.