[CurtHagenlocher]

Note

Draft until the [VOTE] on dev@ for format version 1.6 passes. It implements the format change proposed in apache/arrow#53994, which may still change before the vote. Companion PRs: apache/arrow-go#1381, apache/arrow-rs#11622, apache/arrow-nanoarrow#972.

What's Changed

In IPC, dictionary<values=extension<X>> and extension<X, storage=dictionary<...>> serialize identically today, with ARROW:extension:name in Field.custom_metadata (apache/arrow#49704). Format 1.6 adds an optional DictionaryEncoding.custom_metadata that holds the dictionary value type's metadata. It's the IPC counterpart of ArrowSchema.dictionary->metadata, which #474 already handles for C Data.

Format

• format/Schema.fbs gets the new custom_metadata: [ KeyValue ] member of DictionaryEncoding, as proposed upstream.
• Flatbuf/DictionaryEncoding.cs was regenerated with flatc 23.5.9, the version of the checked-in code. The diff is limited to the new member.

Writer

• When a dictionary's value type is an ExtensionType, its ARROW:extension:name and ARROW:extension:metadata are written to DictionaryEncoding.custom_metadata instead of Field.custom_metadata.
• If the field metadata names that same extension, those keys are dropped from the field. Earlier versions of the IPC reader left them there, and on the field they would mean an extension over the whole dictionary. Other field metadata is kept.
• An extension type whose storage is a dictionary keeps its keys on the field. Writing one used to throw in DictionaryMemo.AddField, which only accepted a DictionaryType field.
• A dictionary whose value type is an extension type with nested storage now writes the storage's children.

Reader

• DictionaryEncoding.custom_metadata resolves the value type against the registry before it's wrapped in the DictionaryType. DictionaryType has nowhere to keep value-type metadata, so, as in Arrow C++, other keys and an unresolved extension name are dropped.
• Field.custom_metadata is then resolved against the whole type, so on a dictionary-encoded field it names an extension whose storage is the dictionary, as in Arrow C++ and arrow-go.
• Breaking change: a field-level extension name on a dictionary isn't applied to the values any more. Files with a dictionary of an extension type written by earlier Apache.Arrow versions, and by every other writer before 1.6, read as a dictionary of the storage type, with the keys left in the field metadata. As decided in Support the proposed DictionaryEncoding.custom_metadata (format 1.6) for dictionaries of extension types in IPC #475, there's no fallback. This needs a release note.

Tests

• .NET:
  • the flatbuffer the writer produces, including dropping field metadata that names the value type;
  • round trips with and without a registry;
  • a dictionary of an extension type nested in a list;
  • field-level metadata on a dictionary not being applied to the values;
  • an extension type whose storage is a dictionary.
• Full suite: passes on net8.0, net462 and net472, pyarrow tests included.
• Cross-implementation, run manually:
  • flatc --json decodes a schema written by .NET with the new member as expected.
  • .NET reads a schema message that flatc built from JSON.
  • pyarrow 24, which predates 1.6, reads .NET output as dictionary<values=fixed_size_binary[16]>.

There's no Go toolchain here, so I couldn't test against apache/arrow-go#1381.

Not included

• Integration test app: it doesn't support the archery dictionary_extension case yet, so archery has to keep skipping .NET for it. The app has no extension types; it passes field-level extension metadata through unchanged. A dictionary's value metadata has nowhere to live in the .NET object model, so the app needs a pass-through extension type before it can produce the case.

Closes #475.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com