Skip to content

fix(catalog): render a stored catalog by the shipped one - #80

Closed
mo4islona wants to merge 1 commit into
masterfrom
fix/render-stored-catalog-as-shipped
Closed

mo4islona wants to merge 1 commit into
masterfrom
fix/render-stored-catalog-as-shipped

Conversation

@mo4islona

Copy link
Copy Markdown
Collaborator

Closes #79.

Summary

A consumer can keep the catalog a chunk was written with, and pin the chunk to it. That copy never changes. A copy made before a8f6e03 has no struct members, so the engine rendered two fields as numbers where the reference writes decimal strings:

  • solana transactions[].transactionConfig.priorityFee
  • evm transactions[].authorizationList[].nonce

sqd-metadata now embeds the catalogs from metadata/. After parse_dataset_description loads a catalog, it takes encoding and members from the shipped catalog with the same name, for each column with the same name and the same stored type. Everything else in the copy is used as written: tables, columns, types, weights and column order. The strict loader checks keys against the document as written, and then applies the same rendering.

A consumer needs no code change: a pin bump is enough.

Tests

  • test_a_copy_of_a_shipped_catalog_renders_as_shipped: the solana and evm catalogs with their members removed, and solana with fee's column encoding removed, render as shipped, through both loaders. A column stored as another type keeps its own rendering.
  • Mutations: with the overlay off, or with the type check removed, the test fails.
  • make fmt, make lint, make spec-check and make test pass: 444 passed, 0 failed.
  • Ignored real-chunk tests: all pass, except 33 query tests and the concurrency test, whose chunks were not available locally.
  • End to end, not committed: solana_devnet/v1_transaction with a stored copy of the solana catalog from before a8f6e03. Before: "priorityFee":5000. With this change: "priorityFee":"5000". With sorted keys, the whole answer (501 lines) is identical to the answer with the shipped catalog.

Not in this PR

  • The order of keys in an object still follows the stored copy's column order. A copy with reordered columns gives the same values in a different key order.
  • Weights are not taken from the shipped catalog. They change page sizes, not answers, and a weight can name a column that an older copy does not have.

🤖 Generated with Claude Code

A consumer can keep the catalog a chunk was written with and pin the chunk to
it, and that copy never changes. A copy made before struct members carried
encodings rendered solana `transactionConfig.priorityFee` and evm
`authorizationList[].nonce` as numbers where the reference writes decimal
strings. The loader now takes each column's `encoding` and `members` from the
catalog this release ships under the same name, where the copy stores the
column as the same type.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@mo4islona

Copy link
Copy Markdown
Collaborator Author

Closing without merging. Taking the rendering from catalogs built into the engine would make every new output rule wait for an engine release in every consumer. Catalogs are distributed by their provider so that a change of this kind needs no release, and the fix belongs there. See #79.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A pinned dataset description from before a8f6e03 changes the JSON of struct members

1 participant