[rustyconover]

Rebased onto current main; prerequisites #928 and #968 have merged. This PR now contains only the dictionary value deduplication change.

ArrowIpcWriterWriteArrayViewWithDictionaries() (#968) writes every dictionary before every RecordBatch. Its input is borrowed, so it can't do what ArrowIpcWriterWriteArrayStream() does (#928): keep the previous array alive and compare dictionaries by buffer address. A producer that writes batches one at a time with the same dictionary therefore sends that dictionary again with every batch. In a file, where a dictionary ID may be written only once, it can't write a second batch with dictionaries at all.

The writer now keeps a copy of each dictionary this function writes, keyed by dictionary ID, and compares the next batch's values with it:

• First it checks whether the borrowed values are physically identical to the copy (ArrowArrayViewCompare(..., NANOARROW_COMPARE_IDENTICAL, ...)). This is the common case, e.g. when the producer builds each batch's dictionary the same way, and it needs no copy.
• Otherwise it copies the values into the same canonical form as the saved copy (zero offset, buffers sized to the values, cleared bitmap padding bits, computed null count) and compares the two. A slice, an unknown null count or garbage in padding bits therefore doesn't make an unchanged dictionary look changed.
• A dictionary whose values are unchanged isn't written. A changed one is written in full, as before (in a file that returns ENOTSUP, as before).
• A dictionary whose values contain a nested dictionary that was written is written again even if its own values are unchanged, since a reader resolves a DictionaryBatch against the nested dictionaries current at that point.
• Values that can't be copied (ArrowArrayAppendStorageFromArrayView() doesn't support dense unions) are written every time, as before.
• ArrowIpcWriterWriteDictionaryBatch() drops the saved copy for its ID, so a dictionary written that way is never skipped by mistake, and ArrowIpcWriterWriteSchema() drops all of them.

ArrowIpcWriterWriteArrayStream() is unchanged. The function's documentation now describes this, including the cost: a copy of each dictionary that is written.

Cost: 1000 batches of 2048 rows, dictionary<int32, string> with 10,000 values, written to a memory buffer (Release, Linux aarch64; best of 3):

	#968	this PR
Dictionary unchanged	126 ms, 258.5 MB	23 ms, 8.6 MB
Dictionary grows by 16 values a batch	205 ms, 458.3 MB	531 ms, 458.3 MB
Unchanged, 100 values	6.4 ms, 11.0 MB	5.1 ms, 8.4 MB

A dictionary that changes on every batch costs a copy per batch. The follow-up, #980, writes a dictionary that only grew as a delta, which brings that row to 346 ms and 9.2 MB.

Tests (writer_test.cc):

• WriteArrayViewWithDictionariesSkipsUnchanged: separately built arrays with dictionaries [foo, bar], [foo, bar], [foo, baz], [foo, baz] write two DictionaryBatches. Read back with nanoarrow and Arrow C++.
• WriteArrayViewWithDictionariesComparesValues: an int64 dictionary [1, null, 3], then the same values as a slice of a longer array with an unknown null count and padding bits set, write one DictionaryBatch.
• WriteArrayViewWithDictionariesNested: when only a nested dictionary changes, both it and the dictionary that refers to it are written (IDs 1, 0, 1, 0 over three batches).
• WriteArrayViewWithDictionariesAfterWriteDictionaryBatch: a dictionary written with ArrowIpcWriterWriteDictionaryBatch() in between isn't mistaken for the saved copy.
• WriteArrayViewWithDictionariesUncopyableValues: a dense union dictionary is written for every batch.
• WriteArrayViewWithDictionariesInFile now writes a second, unchanged batch to the file, checks that a changed dictionary returns ENOTSUP, and reads the file back with Arrow C++.

Against #968's writer.c, the first three new tests and the updated file test fail.

ctest passes (Linux aarch64, Debug, 878 tests with NANOARROW_BUILD_TESTS_WITH_ARROW), and 403 tests under ASan/UBSan with clang. The writer tests are clean under valgrind. The changed files are formatted with clang-format 16.0.6, the pre-commit version.

This PR's code and description were written by AI (Claude Code) and reviewed by a human maintainer.

🤖 Generated with Claude Code

Validation after rebasing (October 26): macOS arm64 Debug, 426 CTest cases, no failures; 16 compression cases skipped because compression is disabled in this build.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 85.10638% with 21 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.56%. Comparing base (08d23f6) to head (8f8e296).

Files with missing lines	Patch %	Lines
src/nanoarrow/ipc/writer.c	85.10%	1 Missing and 20 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #979      +/-   ##
==========================================
+ Coverage   78.52%   78.56%   +0.04%     
==========================================
  Files         106      106              
  Lines       17195    17334     +139     
  Branches     2064     2108      +44     
==========================================
+ Hits        13502    13619     +117     
- Misses       2431     2432       +1     
- Partials     1262     1283      +21     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.