[foysavas]

Why

On 2026-10-18 the public testnet had no shared chain: no sequencer answered /status.json, node0 was frozen at tip 4887, and node1, node2, and node3 each mined alone (10196, 15330, 11762) at the difficulty floor, while a non-bootstrap miner gossiped a fourth chain near 18.6k.

Every gossiped miner block ran a fork-choice pass that decoded the whole store several times and did work quadratic in stored blocks, under the one datastore mutex. Nothing was pruned, so after 15 days a pass took about five minutes. Status, the Identify handler on the networking task, and Shoal broadcasts all waited on that lock. Peers stopped seeing each other's blocks and each node mined its own branch. This is the third failure of this shape; earlier fixes made steps of the same pass cheaper without removing the per-block rescan.

What changes

Nothing waits on scoring

• /status.json comes from a snapshot retaken every 5 s in the background, with status_age_secs. A wedged node answers with a growing age instead of hanging. Before the first snapshot exists the endpoint returns 503.
• Identify stores peer info off the networking task.
• Shoal sends wait at most 2 s for the swarm.

Incremental fork choice

• An exact in-memory mirror of the three miner stores, with a change journal. All multi_store writes update it, and scans read it in the same key order.
• A tree over that mirror tracks:
  • linked work for each block;
  • the winner and canonical spine tips;
  • the tips of linked branches.
    It relinks only what changed. The target check is memoized per hash and walks back only to the previous epoch.
• select_best_stored_chain and adopt_connected_extensions decide on the tree, with the same guards and log lines. They fall back to the full scan for cases the tree cannot decide.
• A block off every linked walk is stored without being scored.

Bounded untrusted gossip

• Per-peer rate limit on miner blocks: 1/s, burst 120.
• Parked-block caps: 1024 per author, 8192 in total.
• Parents are requested from the block's author, not only from bootstrappers, at most once per minute per peer.

Pruning

• Blocks that are not canonical and sit two epochs under the latest checkpoint are pruned.
• Chain validation indexes blocks instead of searching per index.

Tests

• Scoring one gossiped block: about 250 µs (debug build) at both 2k and 20k stored blocks.
• Randomized, 600 stores each:
  • tree select vs. full scan (including checkpoint floors and missed targets);
  • canonical set vs. verified_spine;
  • adoption leaves no branch that beats the chain;
  • incremental tree vs. rebuilt tree.
• The randomized tests show two full-scan bugs the tree does not repeat:
  • a block demoted off the spine that is on the winner stays demoted, leaving a hole under the tip;
  • store-order adoption can take a lighter branch and then orphan the heavier one.
• cargo test -p modality-node -p modality-datastore -p modality-observer -p modality-cli-node, workspace check (including tests and examples) with -D warnings, and clippy -D warnings.
• Local 3-node hybrid devnet: one genesis, one tip, rounds rising after epoch 2, status answering in under 1 ms with age ≤ 5 s, and a restart on existing data that kept the chain.
• tests/network/11-hybrid-devnet3 and 04-sync-miner-blocks fail the same way on main:
  • node1 is still in its 30 s chain-info timeout at the test's 15 s check;
  • the 04 test config predates config_version 2.

Follow-ups

• Gossipsub message validation and peer scoring, so floods also stop propagating.
• Select's fills_parents rule can still adopt a stored chain that contains a block below its target, whenever that chain extends the local one. Peer batches do not check proofs.

🤖 Generated with Claude Code

[foysavas]

Landed on main directly: 0c35814 merges 842d45b (testnet release 20261018_182417-842d45b8), and d76ac02 carries the Rust tree of c3f8109 (20261018_201039-c3f81097, tag testnet-mint-memo). The examples/multi-agent-smax commits that rode on this branch are kept on the examples/multi-agent-smax branch.

🤖 Generated with Claude Code