[jbearak]

Summary

#456 builds a grouped mutate() of arithmetic from the columns' rows when the tree holds only numbers, (, columns, summaries of columns and the four operators. A tree with lag() or lead() of a column, first(), last() or nth() of one, or n() still went to ADR 0065's plan of the chunks, which split each column and returned a result for each group. Over the 42,739 women in the first 100,000 rows of the India 2021 DHS births file, mutate(b, k = b3 - lag(b3), .by = caseid) took about 0.14 seconds, where a tibble takes about 0.46.

This PR lets the rows evaluator of C_dtatools_arith_plan_rows() take these operands, as the plan of the chunks computes them for each group:

• lag(x, n) or lead(x, n) gives each group's values of the column n rows before or after, within the group, and the default or a missing value where the shift leaves a row, in the column's storage. shift_default_number() must find that the storage holds the default, as tree_shift() requires.
• first(x), last(x) or nth(x, n) gives each group's value of the column at that position, or a missing value past either end. As tree_pick() does, the call declines where a group has the position and the first group does not.
• n() gives each group's size as a bare integer. Arithmetic of a Stata numeric with it takes the Stata numeric's storage as the minimum, as with a number. Arithmetic of two bare numbers still goes to the plan of the chunks.
• A shift or pick declines where some group's slice would hold a value the storage no longer takes, such as a byte of 101 from an old file. The restore of that slice in the plan of the chunks declines there too.

slice_chunk_admitted()'s checks move to slice_storage_admitted(), which takes the chunk's size as an argument. The evaluator then checks the gathered rows once, with every group shorter than the size gate. .arith_planned() now tries the rows first for trees with shifts, picks and sizes.

Behavior

Values, attributes, ALTREP forms, warnings and errors are unchanged. ADR 0119 records the decision. row_number() takes no arithmetic, as before, and x[1] and x[n()] still go to the plan of the chunks.

Timing

Best of three runs in fresh R sessions, in seconds, with the builds interleaved, over the first 100,000 rows of the India 2021 DHS births file. Before is #456's head after its merge of main.

Call	Before	After	Tibble
mutate(b, k = b3 - lag(b3), .by = caseid)	0.14	0.008	0.46
mutate(b, k = b3 - first(b3), .by = caseid)	0.14	0.008	0.30
mutate(b, k = b11 / n(), .by = caseid)	0.086	0.009	0.12

GC time fell from about 0.015, 0.015 and 0.008 seconds to none in each call.

Tests

• A test runs 12 shapes of each compact column, such as x - lag(x), lead(x, 3, default = 1L) * 2, x - nth(x, -2), x / n() and (x - lag(x)) / n() + first(x). Each must give the same value, warnings and result form with the plans as without them. 77 build their column from the rows. The 12th shape, x - nth(x, 3), reads a third row the first group lacks, so the loop runs, as before.
• A test runs 80 random trees of columns, numbers, summaries, shifts, picks and n() over random data at each storage's limits.
• A test reads the old file's byte column, which holds 101 and 102. last() of a group that ends in one errors in the loop, with the plan too. lag() and lead() keep such a value, but the plan declines a slice that holds one, and builds the column where no slice does.
• Two existing tests now expect the rows to build int / n() and byte - lag(byte). The existing case of b - first(b) over the old file still errors in the loop, with the plan declining.
• The full suite passes: 168 files, 2,266 tests. The native manifest and repository guard tests pass. Codex review is clean.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: jbearak/dta-parser
• Branch: claude/arith-rows-shifts
• File: .coderabbit.yaml
• Commit: 4c4c313
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Team
• Run ID: 16b2c665-5bc1-4951-be56-55922548f799

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}