[jbearak]

Summary

C_dtatools_gather_numeric_columns() gathers rows of several Stata numeric columns at once. Its Rust kernel started a scoped worker thread for each column, up to the processor count, on every call with more than one column. A grouped verb gathers each group's few rows of every column in its own call, so nest_by(), group_nest() and group_split() paid for starting those threads once for each group. Gathering two rows of eight columns took about 46 µs, where one thread takes 1.5 µs.

This PR starts one worker for each 16,384 values a gather copies, up to the column count and the processor count, and gathers on the calling thread when that comes to one worker. A gather of one column or few values returns before it asks the system for the processor count, which a grouped read of one column would otherwise ask once for each group.

Behavior

Results are unchanged. Each worker writes the same bytes to its own columns, so the number of workers changes only the time. No ADR records the threads, so none changes.

Timing

Best of three runs in fresh R sessions, in seconds, with the builds interleaved, over the 42,739 women in the first 100,000 rows of the India 2021 DHS births file. Before is main.

Call	Before	After
nest_by(b, caseid)	5.8	3.3
group_split(group_by(b, caseid))	8.7	5.6
group_nest(b, caseid)	5.7	3.4
.split_dta_column() of a byte column into 50,000 chunks of three rows	0.010	0.010

One call that gathers sorted random rows of eight columns of the births file took these times, in microseconds:

Rows	Before	After	Workers after
2	46	1.5	1
128	46	3.1	1
1,024	57	16	1
4,096	82	66	2
16,384	200	200	8
262,144	2,000	2,000	8

Tests

• A Rust test checks the worker count: one for a few rows of many columns, for 2,048 rows of eight columns and for one column of any length, two for 4,096 rows of eight columns, and the processor count for wide and long gathers.
• The full suite passes: 169 files, 2,281 tests. The native manifest and repository guard tests pass. Codex review found that the first version asked for the processor count on every serial gather, which slowed a split into 50,000 chunks from 0.010 to 0.025 seconds. This PR fixes that, and the second review is clean.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: jbearak/dta-parser
• Branch: claude/gather-small-serial
• File: .coderabbit.yaml
• Commit: dffec59
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Team
• Run ID: 951b2dee-386e-401a-b1a8-784dc83da94e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}