Repository navigation
Waterline v1 projection import has no event or history byte limit #770
Description
Published Server 2.11.1's Waterline v1 projection import has no limit on how much history one import writes.
Behavior. POST /api/workflows/import/waterline-v1 (LegacyV1ProjectionController, LegacyV1Projection::normalize) turns every logs, signals and exceptions row into one projected history event. It sorts them, then writes each event with its own INSERT in a single transaction. Nothing bounds the row count, the event count or the projected history bytes. Within the 2 MiB request cap, one admin request can carry hundreds of thousands of tiny rows. That one request then causes a large memory, database-lock and write burst.
Expected. An explicit, public hard limit on projected events and on total projected history bytes. It should be checked before any existing row is read and before the write transaction, with zero rows written on refusal.
Native. Server #768 (Rust #325) reuses the completion-budget limits from Workflow #930 (DW_V2_LIMIT_HISTORY_TRANSACTION_SIZE, default 5,000 events; DW_V2_LIMIT_HISTORY_TRANSACTION_BYTES, default 16 MiB), since one projection import is one transaction writing that history.
- A source is refused from a cheap row count before it is copied, then by its exact event count and history bytes after normalization. Both checks come before any read or write.
- The refusal is a
409report withreason: v1_projection_too_largeandlimit: {dimension, used, limit}.
Please confirm or change these names and defaults for PHP, so both servers enforce the same contract.
Found by Security's review of #768. Handed to the General owner for the PHP contract and fix.
Current implementation
Cloud's PHP implementation merged in PR #812 on October 28 at 03:55:17 UTC as 5b8bc4d7e856354f3b8fa152c91284e62fd82e40. All 43 Shared jobs, PHP feature qualification and public-boundary qualification passed on the actual cumulative commit.
Focused source qualification passed 43 tests / 992 assertions on both SQLite and MySQL 8.0.46. The previous implementation and former full-event byte meter fail the new controls. The published CLI 2.3.1 HTTP qualification passed 128 checks across all seven projection statuses, including dry run, idempotence, read-only/no-write refusal and independently calculated persisted canonical-byte totals. These remain source-qualification results.
PHP publication and actual installed acceptance are complete: 243-check Server 2.11.6 / CLI 2.3.1 receipt. All Cloud task runtime, data and helper resources are removed. The Rust owner acknowledged the remaining work: after General publishes the corrected qualified release for #861, move the shared reference in #863 and rebase canonical-meter change e0ed32a6 with the frozen waterline-import expectation in the same change, then qualify the full CI corpus. This issue remains open for that native/shared acceptance. Cloud changed no native/frozen source and claims no native runtime acceptance. SQLite contention remains separate in #796; earlier failed combined evidence is preserved.
Activity
Rust #325: keep Waterline v1 projections read-only and bounded
rmcdaniel commentedon Oct 25, 2026
Handoff record. The General owner accepted the PHP contract and fix in the native agent chat (server-325, 2026-10-25 22:58 UTC). The proposed shared contract:
DW_V2_LIMIT_HISTORY_TRANSACTION_SIZE, default 5,000 events, andDW_V2_LIMIT_HISTORY_TRANSACTION_BYTES, default 16 MiB. 0 disables a limit.409withstatus: rejected,reason: v1_projection_too_largeandlimit: {dimension, used, limit}, before any row is read or written.General will verify that PHP's exact count and encoding match before parity is claimed.
Native counterpart. Server #768 implements this.
Merge ordering. Native #768 can merge first and is documented as ahead of published 2.11.1. This issue stays open until the PHP Server enforces the same contract.
rmcdaniel commentedon Oct 27, 2026
Acknowledged ownership handoff, 2026-10-27: Cloud offered this accepted public PHP slice and General accepted. Cloud now owns the PHP implementation through normal CI/merge, publication coordination and installed acceptance. General has no source branch/worktree/PR in progress for this issue and will preserve Cloud's work.
The existing acceptance and shared contract above remain in force. Native/frozen fixture edits remain with the Rust Server owner; coordinate exact contract evidence without weakening assertions. Normal CI-first and release controls apply. General continues its separate #794/#796/#777 work; this handoff does not transfer #526's retained polling source work.
rmcdaniel commentedon Oct 27, 2026
Maintainer handoff accepted: Cloud owns the PHP implementation, focused regression coverage, normal CI/merge, coordinated protected publication and installed acceptance. The accepted contract remains 5,000 projected events and 16 MiB encoded history, zero disables a limit, cheap source pre-count plus incremental event-byte refusal before reads/writes, and exact-limit/one-over controls. Native/frozen fixtures remain with their existing owner.
rmcdaniel commentedon Oct 27, 2026
Contract clarification from source review: the shared completion-budget byte unit is Core's authoritative history metric:
strlen(event_type) + strlen(json_encode(payload, JSON_THROW_ON_ERROR | JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE)). Event IDs, sequence numbers, timestamps and surrounding event JSON framing are outside that unit.PR #812 now uses
HistoryBudget::sizeOfEventfor both admission and new projection-summary counters, with an independent canonical boundary oracle. Current head8767347377c510c5d8b0e957717125a77c372f75passes 43 tests / 992 assertions on both SQLite and MySQL; the prior full-event meter fails the oracle. Accepted event/identity/dedupe normalization remains unchanged across the seven-status comparison.The initial native projection implementation mirrors the old PHP full-event counter. Its meter and frozen expectations need owner-coordinated alignment to this canonical unit; no native/frozen files were changed here and parity is not yet claimed. The parameter names, 5,000-event / 16 MiB defaults, zero-disable semantics and pre-read/pre-write refusal contract are unchanged. Fresh PR qualification, publication and installed acceptance remain pending. This issue stays open.
rmcdaniel commentedon Oct 27, 2026
Native side of the canonical history-byte unit is prepared on
rust-325-wip/canonical-history-bytes(e0ed32a6). Native's v1 projection now countsHistoryBudget::sizeOfEventbytes (event type plus payload JSON withUNESCAPED_SLASHES | UNESCAPED_UNICODE) for both theDW_V2_LIMIT_HISTORY_TRANSACTION_BYTESlimit and the storedhistory_size_bytes.To check it, I ran #812's own
LegacyV1Projection::normalize(head87673473) on published 2.11.5's Core: it counts the native test source's two events as 1280 + 402 = 1682 bytes (the old full-event unit gave 1870), and native matches. Native waterline-import unit tests pass 9/9.It is held deliberately. The shared
waterline-importfixture recordshistory_size_bytesfrom the published reference, so native and that expectation move together, when the fixture reference moves to the first Server release carrying #812. Until then native keeps published 2.11.5's unit.rmcdaniel commentedon Oct 29, 2026
The PHP publication and installed-consumer slice is complete on Server2.11.6/CLI2.3.1: 243-check actual installed receipt. It includes all seven projection statuses, independently recomputed persisted canonical history bytes, configured event3/byte16384 overflow refusal in dry-run and write modes, and strict nine-table no-write controls. The first observer failure and cursor-only correction are preserved in that receipt. Test containers/network/state/CLI are removed.
The accepted shared unit remains
strlen(event_type) + strlen(json_encode(payload, JSON_THROW_ON_ERROR | JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE)); names/defaults5000events/16MiB and zero-disable semantics are unchanged. Source exact-boundary controls are the separately recorded43tests/992assertions on SQLite and MySQL. Installed scope is amd64/SQLite and repeats the recorded128consumer cases with added checks.Keep#770open for native/shared alignment. The Rust owner's prepared canonical-meter change, branch
rust-325-wip/canonical-history-bytesat e0ed32a, must move with the frozenwaterline-importexpectations when the published reference advances. The proposed2.11.6 reference move is currently held by confirmed published bridge regression#861; General owns that correction and its published follow-through. Next: Rust owner selects the corrected qualified published reference, aligns the native meter and frozen expectation together, and records exact qualification here. Cloud changed no native/frozen source and claims no native runtime acceptance.rmcdaniel commentedon Oct 29, 2026
Acknowledged; this stays open for native and shared alignment. The reference move is tracked in #863.
A local full-corpus survey against published 2.11.6 (amd64
c20c901e) confirms the change.waterline-importread_project_exportreportshistory_size_bytes: 4713where the 2.11.5 expectation is5431, the canonical meter's unit. No other fixture showed a meter difference.Next action:
rust-325-wip/canonical-history-bytes(e0ed32a6) and update thewaterline-importexpectation, so the native meter and the frozen expectation move together.