[mo4islona]

Proposes sets, a filter on value sets too large for an inList (#82), and measures the options behind them. The PR has three parts:

• the API, specified in spec/ as ADR-17, which is Proposed;
• a research prototype behind the membership-filter build feature, written before the API was settled;
• the benches and measurements the choices rest on.

Not meant to merge as it is. The default build does not change, and the prototype has to be rewritten to the spec once ADR-17 is accepted.

API

The API is specified in spec/: ADR-17 (Proposed), the request in §2.7, the lookup in §4.2.11, four new errors in §6.2, and five invariants, INV-Q15 to INV-Q18 and INV-P17.

{
  "type": "evm",
  "sets": {
    "pairs": { "fuse8": "<base 64>", "canaries": ["0xB4e16d…", "0x0d4a11d5…"] }
  },
  "fields": { "log": { "address": true }, "transaction": { "from": true, "to": true } },
  "logs": [{ "address": { "set": "pairs" }, "topic0": ["0x1c41…"] }],
  "transactions": [{ "from": { "set": "pairs" } }, { "to": { "set": "pairs" } }]
}

• Sets are declared once and referred to by name. One set serves from and to, or both owners of a Solana token balance, and its bytes travel once. The name means nothing to the engine; an SDK numbers its sets as it serialises the query, so its users never see one.
• One format, fuse8. The spec fixes the lookup, not the construction, so any builder whose output the lookup accepts is correct.
• Canaries are required, from one to 64 (a proposed bound). A canary the column cannot render, or one the structure does not hold, is SetCanaryMismatch.
• Any inList filter on a hexBytes, base58 or utf8 column takes a set. Every chain gets sets from its catalog, with no code.
• The filtered field must be selected (SetFieldNotSelected), so the client can see the false positives it has to drop.
• New errors: UnknownSet, InvalidSet, SetCanaryMismatch, SetFieldNotSelected.
• Routing. An engine without sets refuses a request carrying sets with UnknownTable. The portal can send every such request to an engine that supports them, and nothing answers one without its filter.

The prototype below predates this API. It takes the structure inside the filter value, accepts four formats and a skipRepeats flag, and does not check the selected field. spec/GAPS.md lists the differences.

What it adds

• The spec: ADR-17, the request in §2.7, the lookup in §4.2.11, four errors, the invariants INV-Q15 to INV-Q18 and INV-P17 (unchecked until the engine implements them), the parameter P-MAX-SET-CANARIES, and an entry in GAPS.md for where the prototype differs.
• A prototype behind membership-filter. It takes the structure inside the filter value, e.g. "address": {"fuse8": "<base64>", "canaries": ["0xAbC…", …]}, and accepts four structures so that they can be compared: the RFC draft's Bloom filter, Parquet's split-block Bloom filter, and binary fuse filters with 8- and 16-bit fingerprints (xorf). The structure never prunes a row group (INV-P16). An optional skipRepeats reuses the previous row's verdict when the value repeats.
• Canaries in the prototype work as the spec says: a few members as the client had them. The engine normalises each one the way the column's filters are normalised (case folding, the 0x prefix, the base58 alphabet) and looks it up in the structure. A miss refuses the request before any row is read.
• Two benches: membership (Gungraun, with branch simulation) and membership_wall (criterion: the whole request, the parse alone, the predicate alone). Before it times anything, membership_wall pages through the chunk like a client and asserts that no member row is missing.
• A client-side build in benches/membership_client: the Rust builder compiled to WASM and a TypeScript port, each checked byte for byte against xorf, with benches for both and for native Rust.

Results

The set has 400 000 values. On Ethereum it is every address that emits a V2 Sync in evm-big (3 039); on Solana it is 5% of the token-balance owners in solana-small (392). Random values fill the rest: a member the chunk never mentions changes a structure's size and nothing else. An inList takes the first 40 000, about the most a 2 MiB request carries. Every number is from one pinned core.

No structure dropped a member row on any chunk.

400k set unless noted	Filter	Junk rows, 224-block Ethereum chunk	Predicate ns/row, logs.address / transactions.from	Parse per request
inList, 40k	1 800 KB	0	4.3 / 5.7	7.6 ms
RFC Bloom, 0.26%	826 KB	607	15.8 / 23.2	0.32 ms
Fuse8	612 KB	343	8.2 / 8.5	0.24 ms
Fuse16	1 223 KB	0	8.3 / 8.8	0.59 ms

The others: the split-block Bloom filter is as large as the RFC one, with 1.5–4× its junk and 19 ns per row. skipRepeats brings every structure to about 5 ns per row on a column the table is sorted by, and adds about 1 ns elsewhere.

What the numbers say

The filter's size matters more than its junk. The portal sends the whole query in every worker request, one request per chunk, and recent chunks are small: 224 blocks on Ethereum, 48 slots on Solana. On the 224-block chunk the 400k filter is 4–8× the answer it selects (162 KB), while Fuse8's junk adds 25 KB. Over a stream:

	Chunk requests	Fuse8 filter traffic
Ethereum, a year of history at 224 blocks per chunk	~11 700	~7 GB
Solana, a day of history at 48 slots per chunk	~4 500	~2.7 GB

Large inList filters pay this today. The 40k list is 1.8 MB per chunk, and parsing it is 7.6 ms of an 11.8 ms request on the 224-block chunk.

So Fuse8. It is the smallest structure at about 0.4% false positives: 9.2 bits per element, against about 11.5 for a Bloom filter at the same rate. The bound is n·log2(1/p) bits, and Fuse8 is within 15% of it, so no other format does much better. The bits are random, so compression does not help either.

Not Fuse16, though it is faster. It returned no junk here and made the fastest requests, because a junk row costs more than its bytes: the worker reads back the output columns of the row group it sits in. On transactions in evm-big, 628 junk rows from a Bloom filter made that read-back 36% of the request. But Fuse16 is twice the size of Fuse8, and the size is paid on every chunk.

Drop both Bloom filters. Each is larger than Fuse8 at the same rate and slower per row. The RFC's k seeded hashes cost 3–4× a hash-set lookup on a column the table is not sorted by. A Bloom filter can also be set to match everything, so it would need a density check. A fuse8 lookup cannot be made to pass every value, whatever bytes it carries, because part of each fingerprint comes from hash bits no index reads.

Canaries make a wrong build loud. Without them each of these mistakes compiles and answers an empty or partial result: mixed-case input, no 0x prefix, the 20 raw bytes instead of the text, another hash, other seeds, the other byte order. With them each is an error, and each has a test. Each part of the check was also removed in turn, to confirm a test fails without it. A wrong structure passes four canaries only by false positives, with a probability of about 2 × 10⁻¹⁰ (Fuse8). The one mistake they cannot see is a filter on the wrong field, such as an address set on a topic column, because the canaries are valid values there too.

Why not a per-worker filter cache

A cache would send the filter to each worker once and refer to it by digest after that. It does not pay. A factory's set grows while the client discovers its children, so the filter, and with it the digest, changes every time the client rebuilds it. Each change misses on every worker, and each miss costs one more round trip to send the bytes.

Client build time

A fuse filter takes no inserts, so a client rebuilds it whenever its set grows. Most clients are TypeScript, so benches/membership_client builds the fuse8 value there in two ways: through the Rust builder compiled to WASM (25 KB), and through a TypeScript port. Both give the same bytes as xorf on 516 xxh3 vectors, on every set below, and on every set size from 0 to 3 000. 33 of those sizes needed a second attempt at construction. None needed a third, so that path is untested.

From a list of strings to the base64 value, Node 24, median:

Set	Rust	WASM from TypeScript	TypeScript
10k	0.19 ms	0.54 ms	2.1 ms
40k	0.73 ms	2.3 ms	8.5 ms
100k	1.9 ms	5.9 ms	21 ms
400k, Ethereum	7.9 ms	26 ms	87 ms
400k, Solana	7.9 ms	26 ms	86 ms
1M	24 ms	83 ms	221 ms
2M	64 ms	216 ms	512 ms

A rebuild is cheap. 400k values take 26 ms through WASM, within the range of a single chunk request on the worker in the tables above. The first build in a fresh process takes 50–60 ms through WASM and 150 ms in TypeScript.

Half of the WASM path is strings, not the filter. Packing 400k strings into one buffer takes 17 ms, and the build itself 14 ms. A client that keeps its set packed, and appends each new member to it, pays only the second.

WASM memory does not shrink. It grows to the largest build so far and stays: 32 MB after 400k values, 152 MB after 2M.

2M values do not fit one request. Fuse8 over 2M values is 3.0 MB of base64, over the 2 MiB bound of ADR-13. About 1.4M values fit.

So WASM. It is 2.4–3.9× faster, and it is the engine's own code, so client and engine cannot drift apart. The port is fast enough, but it is a second implementation. A bug that spoils only some values, such as one input length or one retry path, passes the canaries and loses rows without a sign. With the port, the vectors and the size sweep would have to run in CI.

How often a client rebuilds. A stream restarts at the first block of a new member, with a set that holds every member its last batch found. So each restart moves past the end of the last batch, and a client rebuilds at most once per batch. A first pass over the factory's creation events makes it once per backfill.

Found on the way

• QueryOutput::read_through() is the last block a page read, not the last it served: a page can read blocks its budget then leaves out. A caller that resumes from it loses rows; the first version of the report here lost 19. The worker resumes from last_block(), so it is not affected, but the doc comment invites the mistake.
• A topic one hex digit short matches nothing without an error (ADR-16). Inside an opaque structure the same mistake cannot even be seen, which is why the canaries check that each one is a value the column can hold.

Not measured

• What the per-chunk signature costs: two SHA-512 passes over the whole query on the portal and one on the worker.
• The real-time path. Hot blocks must refuse a set until they support it, not ignore it.
• More than one machine, one core and two output fields. Wider field selections make junk dearer.

Running it

SQD_MEMBERSHIP_DIR=sets taskset -c 2 cargo bench --features membership-filter --bench membership_wall
SQD_MEMBERSHIP_DIR=sets cargo bench --features membership-filter --bench membership -- --nocapture

SQD_MEMBERSHIP_TARGETS (logs,transactions,tokenBalances), SQD_MEMBERSHIP_CHUNK (default evm-big), SQD_MEMBERSHIP_SOLANA_CHUNK (default solana-small) and SQD_MEMBERSHIP_CASES (a case-name substring) narrow a run. The first run derives the set files into SQD_MEMBERSHIP_DIR.

The client-side build needs wasm-pack and Node 22.18 or later. prepare.sh builds the WASM module and writes the sets and the native reference outputs:

SQD_MEMBERSHIP_DIR=sets benches/membership_client/prepare.sh
(cd benches/membership_client/ts && node verify.ts && taskset -c 2 node bench.ts)
(cd benches/membership_client && taskset -c 2 cargo bench --bench build)

node cold.ts <wasm|ts> <set> times the first build in a fresh process.

Full tables

Request and predicate times on evm-big were measured before the canary check was added; it runs once per request and costs four lookups.

Ethereum evm-big (1 592 blocks), logs.address

Case	Filter KB	Rows	Members	Junk	Answer KB	Request ms	Predicate ns/row
inList.40000	1,800	8,484	8,484	0	661	22.63	4.3
bloom.40000	83	9,531	8,484	1,047	738	22.86	11.9
bloom+skip.40000	83	9,531	8,484	1,047	738	17.75	5.5
splitBlock.40000	83	12,855	8,484	4,371	981	29.08	19.2
splitBlock+skip.40000	83	12,855	8,484	4,371	981	18.08	5.1
fuse8.40000	66	9,547	8,484	1,063	741	20.02	8.1
fuse8+skip.40000	66	9,547	8,484	1,063	741	17.08	4.7
fuse16.40000	131	8,484	8,484	0	661	16.80	8.1
fuse16+skip.40000	131	8,484	8,484	0	661	14.01	4.7
bloom.400000	826	97,740	97,481	259	7,075	33.92	15.8
bloom+skip.400000	826	97,740	97,481	259	7,075	25.92	5.8
splitBlock.400000	827	97,946	97,481	465	7,089	37.32	19.2
splitBlock+skip.400000	827	97,946	97,481	465	7,089	26.26	5.2
fuse8.400000	612	101,677	97,481	4,196	7,357	29.12	8.2
fuse8+skip.400000	612	101,677	97,481	4,196	7,357	26.26	4.8
fuse16.400000	1,223	97,481	97,481	0	7,056	26.80	8.3
fuse16+skip.400000	1,223	97,481	97,481	0	7,056	23.71	4.9

Ethereum evm-big (1 592 blocks), transactions.from

Case	Filter KB	Rows	Members	Junk	Answer KB	Request ms	Predicate ns/row
inList.40000	1,800	0	0	0	0.1	17.12	5.7
bloom.40000	83	628	0	628	73	18.47	22.4
bloom+skip.40000	83	628	0	628	73	18.56	21.7
splitBlock.40000	83	926	0	926	102	24.64	19.4
splitBlock+skip.40000	83	926	0	926	102	20.84	16.4
fuse8.40000	66	882	0	882	98	17.02	8.4
fuse8+skip.40000	66	882	0	882	98	15.48	9.7
fuse16.40000	131	2	0	2	0.3	8.17	8.5
fuse16+skip.40000	131	2	0	2	0.3	8.38	9.4
bloom.400000	826	531	0	531	61	18.91	23.2
bloom+skip.400000	826	531	0	531	61	18.72	22.6
splitBlock.400000	827	1,121	0	1,121	122	18.41	19.8
splitBlock+skip.400000	827	1,121	0	1,121	122	17.62	16.7
fuse8.400000	612	1,118	0	1,118	121	15.72	8.5
fuse8+skip.400000	612	1,118	0	1,118	121	15.96	9.7
fuse16.400000	1,223	4	0	4	0.6	9.14	8.8
fuse16+skip.400000	1,223	4	0	4	0.6	9.28	9.7

Ethereum evm-small (224 blocks), logs.address

Case	Filter KB	Rows	Members	Junk	Answer KB	Request ms	Parse ms
inList.40000	1,800	197	197	0	17	11.81	7.581
bloom.40000	83	285	197	88	24	4.71	0.034
bloom+skip.40000	83	285	197	88	24	3.74	0.034
splitBlock.40000	83	480	197	283	40	6.27	0.035
splitBlock+skip.40000	83	480	197	283	40	4.01	0.034
fuse8.40000	66	318	197	121	27	4.24	0.029
fuse8+skip.40000	66	318	197	121	27	3.70	0.029
fuse16.40000	131	197	197	0	17	2.92	0.063
fuse16+skip.40000	131	197	197	0	17	2.39	0.063
bloom.400000	826	2,747	2,140	607	206	5.70	0.318
bloom+skip.400000	826	2,747	2,140	607	206	4.69	0.318
splitBlock.400000	827	2,252	2,140	112	170	6.85	0.326
splitBlock+skip.400000	827	2,252	2,140	112	170	4.61	0.326
fuse8.400000	612	2,483	2,140	343	186	4.80	0.243
fuse8+skip.400000	612	2,483	2,140	343	186	4.26	0.243
fuse16.400000	1,223	2,140	2,140	0	162	4.19	0.585
fuse16+skip.400000	1,223	2,140	2,140	0	162	3.63	0.585

Ethereum evm-small (224 blocks), transactions.from

Case	Filter KB	Rows	Members	Junk	Answer KB	Request ms	Parse ms
inList.40000	1,800	0	0	0	0.1	11.79	7.580
bloom.40000	83	133	0	133	14	5.53	0.035
bloom+skip.40000	83	133	0	133	14	5.50	0.035
splitBlock.40000	83	328	0	328	32	5.39	0.035
splitBlock+skip.40000	83	328	0	328	32	5.25	0.035
fuse8.40000	66	207	0	207	22	4.48	0.029
fuse8+skip.40000	66	207	0	207	22	4.58	0.029
fuse16.40000	131	0	0	0	0.1	2.38	0.064
fuse16+skip.40000	131	0	0	0	0.1	2.45	0.064
bloom.400000	826	124	0	124	14	6.01	0.315
bloom+skip.400000	826	124	0	124	14	6.02	0.315
splitBlock.400000	827	308	0	308	31	5.77	0.327
splitBlock+skip.400000	827	308	0	308	31	5.63	0.327
fuse8.400000	612	316	0	316	31	4.95	0.244
fuse8+skip.400000	612	316	0	316	31	5.02	0.244
fuse16.400000	1,223	0	0	0	0.1	3.03	0.586
fuse16+skip.400000	1,223	0	0	0	0.1	3.07	0.586

Solana solana-small (48 slots), tokenBalances.postOwner

Case	Filter KB	Rows	Members	Junk	Answer KB	Request ms	Parse ms	Predicate ns/row
inList.40000	1,880	1,155	1,155	0	166	13.42	7.741	4.4
bloom.40000	83	1,292	1,155	137	185	4.60	0.036	14.4
bloom+skip.40000	83	1,292	1,155	137	185	3.67	0.036	6.7
splitBlock.40000	83	1,781	1,155	626	254	5.22	0.036	18.4
splitBlock+skip.40000	83	1,781	1,155	626	254	3.76	0.036	6.0
fuse8.40000	66	1,527	1,155	372	218	3.86	0.029	8.1
fuse8+skip.40000	66	1,527	1,155	372	218	3.53	0.029	5.1
fuse16.40000	131	1,155	1,155	0	166	3.69	0.065	8.0
fuse16+skip.40000	131	1,155	1,155	0	166	3.35	0.065	5.1
bloom.400000	826	6,126	6,073	53	868	6.39	0.322	17.2
bloom+skip.400000	826	6,126	6,073	53	868	5.19	0.322	7.1
splitBlock.400000	827	6,216	6,073	143	881	6.45	0.331	18.5
splitBlock+skip.400000	827	6,216	6,073	143	881	5.02	0.331	6.1
fuse8.400000	612	6,438	6,073	365	912	5.19	0.246	8.2
fuse8+skip.400000	612	6,438	6,073	365	912	4.85	0.246	5.2
fuse16.400000	1,223	6,073	6,073	0	861	5.49	0.594	8.3
fuse16+skip.400000	1,223	6,073	6,073	0	861	5.14	0.594	5.3

🤖 Generated with Claude Code

[mo4islona]

Superseded by #85, which implements sets as the spec describes them. The measurements in this PR's description stay here as the record behind ADR-17.