[mo4islona]

Implements sets, the filter on value sets too long for a list, as spec/ specifies them (ADR-17, still Proposed), and turns the request bound into a 16 MiB backstop. The research and the measurements behind the design are in #84. Part of #82.

The request

{
  "type": "evm",
  "sets": {
    "pairs": {
      "fuse8": "<base 64>",
      "canaries": ["0xB4e16d0168e52d35CaCD2c6185b44281Ec28C9Dc", "0x0d4a11d5…"]
    }
  },
  "fields": { "log": { "address": true } },
  "logs": [{ "address": [{ "set": "pairs" }], "topic0": ["0x1c41…"] }]
}

A set is declared once under sets, and filters refer to it by name, as an element of their list. An engine without sets refuses the sets key with UnknownTable, so the portal can send a request that carries one only to an engine that serves it.

Use cases

A set is for a list too long to send. A list of a few thousand values still fits, and it is the better choice: it prunes row groups by their statistics and returns no extra rows. A set prunes none, and about one value in 256 that it does not hold passes, so the client drops the extra rows with its own exact check.

• Every pool of a factory. 400 000 addresses are 18 MB as a list and 612 KB as a set: "address": [{ "set": "pairs" }].
• Pools created after the set was built. The factory keeps adding pools. The client sends the new ones as values beside the set and rebuilds the set only now and then: "address": ["0x<new pool>", "0x<new pool>", { "set": "pairs" }].
• Several protocols in one query. Sets built and cached per protocol join in one list, without a new build: "address": [{ "set": "uniswap-v2" }, { "set": "sushiswap" }].
• One set on several columns. A wallet tracker asks for transactions from or to its wallets with two item requests that refer to one set, and the set travels once: "transactions": [{ "from": [{ "set": "wallets" }] }, { "to": [{ "set": "wallets" }] }]. The same goes for the two owners of a Solana token balance, preOwner and postOwner.

A set works only on columns the response renders as text (hexBytes, base58, utf8): addresses, hashes, Solana accounts. The request must select the filtered field, because the client can drop only the extra rows it sees.

What the engine does

• Validates before reading data. UnknownSet for an undeclared name, InvalidSet for an unused set or a malformed fuse8, InvalidFilterValue for a reference on a filter that is not over text, SetFieldNotSelected when the response does not render the column, and RequestTooLarge past 64 references a request: each is a lookup on every scanned row, and a set is cheap to declare again under another name.
• Checks the canaries (SetCanaryMismatch). A set built from the wrong text (checksummed, without 0x, raw bytes, another seed) is refused instead of matching almost nothing.
• Looks up the text the response renders (§4.2.11), once per row however many sets a list names, and once per run of equal values, which is feat: sets, a filter on large value sets (spec, prototype, measurements) #84's skipRepeats made unconditional. A list with a set prunes no row group. xorf only builds sets in tests.
• Raises P-MAX-REQUEST-BYTES to a 16 MiB backstop (second commit). The bounds that matter are the portal's admission and the network's 4 MiB message, and the engine's bound now sits above them. The worst 16 MiB request takes 452 MiB and 0.3 s to parse and compile. ADR-13 gets a revision section.
• Refuses a catalog table or alias named after a reserved request key, such as sets (INV-D10).

Tests

Invariant	Test
INV-Q13	set_references_are_bounded_in_number: either side of the cap in one list and across items; one set named from more items than the cap is refused, a name repeated in one list counts once
INV-Q15	a_set_is_declared_used_and_well_formed; soundness unit tests in src/sets.rs
INV-Q16	a_set_stands_only_in_a_list_of_text: a reference on every filter of every bundled catalog, tables and aliases, alone and beside a value
INV-Q17	a_sets_canaries_are_values_it_holds: five wrong builds, case folding per column, canary bounds
INV-Q18	a_set_filtered_column_reaches_the_response
INV-P17	a_set_passes_exactly_what_its_lookup_passes: on every fixture chunk and every filter that takes a set, the answer equals the answer to a list of every value xorf's own lookup passes, and holds the members' answer; then the same members as a list of a third of them beside two sets of the rest. The sweep asserts that it saw false positives. Unit tests cover dictionaries, byte columns, verdict reuse and a list of values beside two sets.

The traceability matrix moves to 70 C, 21 P, 0 U of 91 (0.77).

Benchmarks

One core, on two real chunks as the network serves them now: evm (ethereum-mainnet-5, 88 blocks, 33 MB) and solana-hot (solana-mainnet-3, 48 slots). The set holds the addresses that emit Sync in the chunk (on Solana, one token-balance owner in twenty), padded with random values to 400 000. The other structures are #84's prototype, measured in the same session; its numbers are with skipRepeats on, its better setting.

logs.address, 400 000 values	On the wire	Extra rows	Predicate, ns/row	Parse and compile	Request
inList, 40 000 values	1.80 MB	0	4.3	7.57 ms	10.7 ms
fuse8, this PR	612 KB	28	3.9	0.24 ms	1.63 ms
fuse8, prototype (xorf)	612 KB	28	5.0	0.24 ms	1.70 ms
Bloom, RFC draft	826 KB	43	5.7	0.32 ms	1.92 ms
fuse16	1.22 MB	0	4.9	0.59 ms	1.98 ms

What the numbers say:

• The set is still larger than the answer it selects. On this chunk the 975 member rows render to 75 KB, and the set is 612 KB, sent with every chunk request. Fuse8 is the smallest structure, which is why it is the one specified.
• The lookup written from §4.2.11 is faster than xorf's: 7.6 against 8.5 ns per row on transactions.from, where no value repeats.
• Looking a repeated value up once pays beyond the columns a table is sorted by. It halves the predicate on logs.address, and on tokenBalances.postOwner, which is not in its table's sort key but repeats beside the account that is. It costs 0.55 ns a row on transactions.from, where values do not repeat. A first version did it only on sort-key columns; this is why it now does it everywhere.
• A set the network's 4 MiB carries holds 2.7 million values and parses and compiles in 1.5 ms.

All cases

Predicate cost per row in ns, by when a repeated value is looked up once (this PR):

Column	Rows	Never	Always	Only on sort-key columns
logs.address	65 487	7.35	3.90	3.90
transactions.from	27 668	7.57	8.12	7.57
tokenBalances.postOwner	132 837	7.43	4.02	7.43

All structures, 400 000 values. "Request" is the whole request on one page, "ns/row" the predicate alone over the column; the prototype's rows are without and with skipRepeats.

Target	Case	Wire	Rows (members)	Request, ms	Parse + compile, ms	ns/row
logs	no filter	–	65 487	5.93	–	–
logs	inList 40k	1.80 MB	68 (68)	10.7	7.57	4.3
logs	Bloom (RFC)	826 KB	1 018 (975)	2.31 / 1.92	0.32	12.0 / 5.7
logs	split-block Bloom	827 KB	1 024 (975)	2.82 / 1.96	0.33	19.2 / 5.8
logs	fuse16	1.22 MB	975 (975)	2.18 / 1.98	0.59	8.1 / 4.9
logs	fuse8, prototype	612 KB	1 003 (975)	1.91 / 1.70	0.24	8.1 / 5.0
logs	fuse8, this PR	612 KB	1 003 (975)	1.63	0.24	3.9
logs	fuse8, this PR, 2.7M	4.06 MB	1 300 (975)	3.18	1.54	4.0
transactions	no filter	–	27 668	2.87	–	–
transactions	inList 40k	1.80 MB	0 (0)	10.9	7.66	5.5
transactions	Bloom (RFC)	826 KB	42 (0)	2.60 / 2.62	0.31	22.7 / 24.4
transactions	fuse16	1.22 MB	0 (0)	1.62 / 1.65	0.59	8.8 / 9.7
transactions	fuse8, prototype	612 KB	81 (0)	2.02 / 2.06	0.25	8.5 / 9.7
transactions	fuse8, this PR	612 KB	81 (0)	1.98	0.24	7.6 → 8.1 always
tokenBalances	no filter	–	132 837	25.8	–	–
tokenBalances	inList 40k	1.88 MB	75 (75)	12.8	7.54	4.3
tokenBalances	Bloom (RFC)	826 KB	7 531 (7 312)	7.14 / 5.83	0.32	17.0 / 6.9
tokenBalances	fuse16	1.22 MB	7 313 (7 312)	6.10 / 5.71	0.59	8.4 / 5.2
tokenBalances	fuse8, prototype	612 KB	7 662 (7 312)	5.82 / 5.42	0.25	8.2 / 5.1
tokenBalances	fuse8, this PR	612 KB	7 662 (7 312)	5.69	0.24	7.4 → 4.0 always

The set is built from pair addresses, and no pair sends a transaction, so on transactions.from every row it passes is extra: it measures the predicate and the false positives alone. inList carries the first 40 000 values, the most that fit the old 2 MiB bound. Extra rows are 28 at 400 000 values and 325 at 2.7 million: the share of values that pass by chance does not depend on the set's size, so the difference is which addresses happened to pass and how many rows each has. The "this PR" request times were taken before reuse became unconditional, so transactions and tokenBalances used the sort-key rule there.

Gates

Per commit, on the pinned Rust 1.89. The full suite is cargo test --workspace --features legacy-query -- --include-ignored with fixture chunks required, the reference comparisons included.

Commit	fmt, lint, clippy	spec-check, spec-test	make test	Full suite
1e55d8b sets	pass	pass	pass	–
3062113 16 MiB backstop	pass	pass	pass	pass (conformance 233/233)

A mutant that drops a list's values beside its sets fails the sweep.

Not in this PR

• Portal. Routing on sets, and a query size limit that admits large sets. The portal's limit is set below the engine's today, so it decides how large a set a client can send.
• Client builders. The Pipes SDK side: building sets, naming them and picking canaries. feat: sets, a filter on large value sets (spec, prototype, measurements) #84 measured a WASM build of the Rust builder and a TypeScript port.
• ADR-17 status. It stays Proposed until the team accepts it.

🤖 Generated with Claude Code