[C++] Arrow ipc format supports deduplication of dictionaries with dict_id, but there is no way to exercise it #49646

Activity

1. cooldome

   added

   Type: enhancement

   on Apr 2, 2026
2. github-actions

   added

   Component: C++

   on Apr 2, 2026
3. thisisnic

   changed the title [-]Arrow ipc format supports deduplication of dictionaries with dict_id, but there is no way to exercise it[/-] [+][C++] Arrow ipc format supports deduplication of dictionaries with dict_id, but there is no way to exercise it[/+] on Apr 2, 2026
4. pitrou commented on Jun 11, 2026

   

   pitrou

   on Jun 11, 2026

   Member

   More actions

   Sorry for the lack of earlier response. Did you encounter a use case where this would bring significant benefits?

   Intuitively, in most cases the space savings won't be large, but deduplicating dictionaries will be costly in the write side.
5. pitrou commented on Jun 11, 2026

   

   pitrou

   on Jun 11, 2026

   Member

   More actions

   Also note that deduplicating consecutive dictionaries inside a single column is supported:
   

   arrow/cpp/src/arrow/ipc/writer.cc

   Lines 1280 to 1315 in 4f3b0f0

   	// If a dictionary with this id was already emitted, check if it was the same.
   	auto* last_dictionary = &last_dictionaries_[dictionary_id];
   	const bool dictionary_exists = (*last_dictionary != nullptr);
   	int64_t delta_start = 0;
   	if (dictionary_exists) {
   	if ((*last_dictionary)->data() == dictionary->data()) {
   	// Fast shortcut for a common case.
   	// Same dictionary data by pointer => no need to emit it again
   	continue;
   	}
   	const int64_t last_length = (*last_dictionary)->length();
   	const int64_t new_length = dictionary->length();
   	if (new_length == last_length &&
   	((*last_dictionary)->Equals(dictionary, equal_options))) {
   	// Same dictionary by value => no need to emit it again
   	// (while this can have a CPU cost, this code path is required
   	// for the IPC file format)
   	continue;
   	}
   	// (the read path doesn't support outer dictionary deltas, don't emit them)
   	if (new_length > last_length && options_.emit_dictionary_deltas &&
   	!HasNestedDict(*dictionary->data()) &&
   	((*last_dictionary)
   	->RangeEquals(dictionary, 0, last_length, 0, equal_options))) {
   	// New dictionary starts with the current dictionary
   	delta_start = last_length;
   	}
   	if (is_file_format_ && !delta_start) {
   	return Status::Invalid(
   	"Dictionary replacement detected when writing IPC file format. "
   	"Arrow IPC files only support a single non-delta dictionary for "
   	"a given field across all batches.");
   	}
   	}
6. cooldome commented on Oct 20, 2026

   

   cooldome

   on Oct 20, 2026

   Last edited by cooldome

   Author

   More actions

   Hi, I missed your request for comment.
   Here is our use case. We are storing trades in arrow ipc files. Main assets like currency codes, equity names, interest rate curves, bond cusip codes are dictionarised. In this file currency code column appear more than 180 times. Main offender is the column with trade economics, it is the dense union column with trade types like IR swap, Equity Future, Bond Future, FX Option and etc. Dense union has around 70 alternatives. For these 70 alternatives: currency code appears more than 100 times, bond cusips 5 times, equity name 35 times, interest rate 60 times and etc. Bond dictionary has 200k+ elements repeated 5 times. Equity dictionary has 100k elements repeated 35 times.
   We don't use plain strings because everything gets loaded into pytorch tensors in the end and pytorch doesn't work with strings.

   We estimated that we would save 2.2 gb every day if would stop duplicating dictionaries