Skip to content

GH-3832: Preserve component paths in column configuration - #3833

Draft
costas-db wants to merge 2 commits into
apache:masterfrom
costas-db:gh-3832-structured-column-config-paths
Draft

costas-db wants to merge 2 commits into
apache:masterfrom
costas-db:gh-3832-structured-column-config-paths

Conversation

@costas-db

Copy link
Copy Markdown

What changes?

Add a backward-compatible structured form for per-column Hadoop configuration keys. Each UTF-8 path component is encoded independently, so the valid paths ["a.b"] and ["a", "b"] no longer collapse to the same "a.b" identity.

The PR adds:

  • component-aware parsing and key generation in ColumnConfigParser
  • String[] path overloads for Bloom-filter and statistics properties
  • public ParquetOutputFormat setters for those structured paths
  • round-trip coverage for dotted, nested, empty, and Unicode path components
  • an end-to-end statistics regression using ordinary string columns

Legacy dot-string keys continue to work. Structured overrides are applied afterward, so callers can opt into unambiguous paths without a compatibility break. This does not change the Parquet file format.

Why is this needed?

Per-column configuration currently passes a flattened string into ColumnPath.fromDotString. A top-level field literally named a.b is therefore indistinguishable from nested field b under a; configuration intended for one column can be applied to the other. The issue is datatype-independent.

TDD verification

The first commit adds the reproduction before production changes. On master, it fails because the top-level ["a.b"] override is ignored. The second commit adds the structured-path implementation.

Final verification:

  • TestParquetProperties: 11 tests passed
  • TestParquetWriter: 32 tests passed
  • TestColumnConfigParser: 1 test passed
  • Spotless passed
  • Apache RAT passed

Fixes #3832

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Per-column Hadoop configuration cannot distinguish dotted and nested column paths

1 participant