Repository navigation
feat(relay): record where media is shared and who uploaded it - #8451
ngoodger-block wants to merge 9 commits into
Conversation
Add `media_shares` and `media_uploaders` (migration 0067) and start filling them, without changing who can read media yet. Each local `/media/<sha256>` path in a stored event shares that blob into the event's channel (or with the whole community when channel-less), in the same transaction as the event, when its author can read the blob or uploaded those bytes; each upload records its uploader under the serving-write lease. `buzz-admin backfill-media-shares` records shares for events stored before this, so channel-scoped reads can be enforced next without hiding existing media. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nikolaj Goodger <ngoodger@squareup.com>
🔐 Codex Security Review
|
Clients upload a file before attaching it, so record the uploader first and keep the test valid once reads deny files not yet shared into any channel. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nikolaj Goodger <ngoodger@squareup.com>
wpfleger96
left a comment
There was a problem hiding this comment.
🤖 thanks for splitting this out from the enforcement PR. two blocking issues, both about how a share gets created, plus a schema question and some smaller items inline.
- a file with no channel shares yet can be made community-wide by anyone who has its hash, because the "no channel shares" arm of
readable_sql!also decides who may create a share. backfill-media-sharesreplays every stored event without the author check, so it turns references the live path refused into community-wide shares.
we reproduced both in a throwaway Postgres using migration 0067 and the production SQL pulled out of media_reference.rs. a non-uploader's kind:1 note created a public share before the backfill and before a fresh uploader's first post. for a reference the live path had refused (0 rows written), the backfill then wrote the public share (1 row). that was a SQL-level check, not a full relay run. #8444's stricter read check still returned true for that user in all three cases, so it can't undo grants this PR already created. these need fixing here before enforcement in #8444 protects anything.
|
@buzz-security-review 86525c2 |
Trust references only from events received before migration 0067 was installed; later events get the live, checked recording, so a backfill run after rollout never shares a blob that live recording refused. Insert shares in (hash, channel) order from both the backfill and live recording so the two cannot deadlock, and read media paths in JSON content that escapes `/`. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nikolaj Goodger <ngoodger@squareup.com>
Require upload proof or a recorded share when recording new references. Let an agent publish only its profile picture using its registered owner's upload proof. Keep the historical backfill cutoff on communities for both schema setup paths, and advance the deletion fixture through migration 67. Signed-off-by: Nikolaj Goodger <ngoodger@squareup.com>
Signed-off-by: Nikolaj Goodger <ngoodger@squareup.com>
Signed-off-by: Nikolaj Goodger <ngoodger@squareup.com>
Signed-off-by: Nikolaj Goodger <ngoodger@squareup.com>
|
🤖 Both blocking grant-creation issues are fixed in this PR with regression coverage, and source-event provenance now ensures deleted messages stop granting file access while surviving shares continue working. |
wpfleger96
left a comment
There was a problem hiding this comment.
🤖 thanks for the quick turnaround. this round closes nearly everything from the first review: hash-only sharing is gone, shares now carry their source event so deletion revokes them, the savepoint ordering is fixed, and the migration-67 fixture is updated. one blocker is left, in the backfill (inline). two nonblocking notes are inline too.
the deferred follow-ups (lease-lost logging, engram 30174, excluded-kind coverage, uploader recovery from audit_log) are fine as follow-ups.
| for (event, hashes, channel) in recorded_live { | ||
| let mut tx = self.begin_media_share_transaction(community).await?; | ||
| added += | ||
| record_media_shares(tx.conn(), community, &hashes, &event, channel).await?; | ||
| tx.commit().await?; | ||
| } |
There was a problem hiding this comment.
🤖 blocking: this re-checks post-cutoff events against the permissions people have when the backfill runs, not the decision live recording already made. that means a refusal can turn into a grant later:
- an outsider posts a channel-less kind:1 with a leaked
/media/<hash>URL from a private channel. live recording correctly writes nothing. - later they're legitimately added to that private channel.
- someone runs
backfill-media-shares(documented as safe to rerun). it replays the old note, the author can read the file now, and it writes a community-wide share. - that share is sourced from the outsider's own note, so it survives their removal from the channel and the deletion of the original private post.
nobody published anything new, but the file is now readable community-wide. the author could repost while they're a member, sure, but here the backfill publishes on their behalf from a note we once refused, and the grant outlives the access that justified it. this was reproduced with the production insert/read SQL and migration 0067 in an isolated Postgres: community-wide shares went 0 to 1 with no new event.
could live recording persist an "evaluated" marker per referencing event in the same transaction, including when it records zero shares, and have the backfill skip marked events and mark the ones it evaluates? events old pods stored during the rolling deploy stay unmarked, so they still get their one evaluation. skipping every post-cutoff event would be simpler, but attachments posted through old pods during the deploy window would then never get a share and would 404 once #8444 enforces. a regression test for refused → gains access → backfill → still no community share would pin it; the current tests keep permissions fixed so they miss this.
There was a problem hiding this comment.
🤖 We’re accepting this as a nonblocking limitation for recording and log-only rollout and have documented that reruns can grant previously refused references after access changes, with enforcement still requiring an explicit decision.
| /// Shares come from the same rule as live writes. Events received before | ||
| /// share recording was installed predate the readability check, so their | ||
| /// references are recorded as they stand; later events get the live, | ||
| /// checked recording, so the backfill never shares what live recording | ||
| /// refused. Idempotent and resumable: rerunning only fills missing shares. |
There was a problem hiding this comment.
🤖 this guarantee only holds while permissions stay the same (see the comment on the recorded_live loop). worth updating along with that fix.
There was a problem hiding this comment.
🤖 Updated the backfill API docs and CLI help to state that reruns recheck post-cutoff references using current permissions and can grant previously refused references after access changes.
|
|
||
| state | ||
| .db | ||
| .record_media_uploader( |
There was a problem hiding this comment.
🤖 nonblocking: nothing in this PR fails if this call is deleted, since reads aren't enforced until #8444 and the media_reference tests seed media_uploaders directly. a relay test that uploads through the handler and asserts the uploader row (or that a later post records its share) would cover it here.
| event_id BYTEA NOT NULL, | ||
| event_created_at TIMESTAMPTZ NOT NULL, | ||
| created_at TIMESTAMPTZ NOT NULL DEFAULT now(), | ||
| PRIMARY KEY (community_id, sha256, event_id) |
There was a problem hiding this comment.
🤖 nonblocking: rows sourced from replaced or hard-deleted events stay behind forever, one per profile or emoji-set update. they grant nothing since the read joins the live event, but nothing ever cleans them up. a periodic sweep or a delete in the purge/replace path would keep the table from growing with dead rows.
Signed-off-by: Nikolaj Goodger <ngoodger@squareup.com>
* origin/main: feat(audio-client): accept a client request in connect (block#8497) fix(mobile): lazily lay out new-DM recipients (block#8435) fix(mobile): lazily lay out channel member groups (block#8437) perf(mobile): retain preference crypto across reconnects (block#8442) fix(deps): bump webpki crates for RUSTSEC-2026-0381/0382/0383 (block#8499) fix(mobile): simplify and refine the quick actions menu (block#8491) fix(sdk): reject malformed mention pubkeys before signing p tags (block#6372) fix(mobile): resolve DM display labels once per sort (block#8488) docs(db): database target architecture and schema-change rules (block#8273) fix(mobile): restore channel icons and align chat colors with desktop (block#8478) test(desktop): read only the open effort menu in effort-picker e2e (block#8476) fix(relay): advance the partition catch-all by default (block#8464) feat(relay): write the NIP-43 member list in the membership transaction (block#8482) fix(relay): delete the startup NIP-33 d_tag backfill (block#8468) fix(mobile): restore Solo chats from channel labels (block#8466) fix(relay): show only legacy channel types to reads that name no channel (block#8469) fix(relay): take NIP-43 snapshot reconcile off the startup path (block#8467) Signed-off-by: Nikolaj Goodger <ngoodger@squareup.com> # Conflicts: # crates/buzz-db/src/store/relay_members.rs
wesbillman
left a comment
There was a problem hiding this comment.
🤖 Star Lord’s automated source review via Wes’s account, integrating independent Rocket (systems/lifecycle), Mantis (correctness/security/concurrency), and Groot (product/operations) reviews.
Two new P2 findings inline: retaining a legacy avatar during a profile update before backfill loses its grant, and deleting an edited message leaves the edit’s independent grant active. These affect the grant population created here; download enforcement remains in #8444.
The existing permission-drift finding in #8451 (comment) still applies; this review does not duplicate it. The rolling-deploy gap also remains an enforcement prerequisite: schema installation precedes the last legacy writer, so old-pod uploads/references after the cutoff may have neither proof nor shares. Preserve the accepted historical-uploader recovery follow-up and resolve that gap before relying on complete coverage. Merely moving the cutoff later can trust references the new recorder deliberately refused.
Public material: please remove the internal tracker URL from the public description’s Related issue section, replacing it with a public description of the work. This is a P3 publication cleanup, not a credential finding. No attached images were present in the description/discussion inspected; the linked issue’s contents were unavailable to this review.
Minimalism 9/10; elegance 9/10; correctness 7/10 because the two inline lifecycle cases need explicit authorization provenance and regression coverage. Existing deferred logging, kind-exclusion and cleanup suggestions remain follow-ups.
Reviewed head 7f7f21c7ea0a5c6ddd39d807dcadf13233ea6723 against base/merge-base 6e6173acd58029bac800583c4cc7482dafed693a. Source only: no PR code, tests, SQL, builds, apps, live workflows or benchmarks executed. One head-bound CI snapshot showed 55 successful and 30 skipped checks; this does not establish runtime correctness. Database races, rollout recovery, performance, and native upload/error/retry focus behavior remain unverified; the PR itself records its Desktop workflow smoke test as pending. Non-blocking COMMENT review, not approval or merge authorization.
| kind_i32, | ||
| channel_id, | ||
| // Authorize retained references while the previous head is still live. | ||
| crate::store::event_follow_up::after_admitted_insert(&mut tx, event, channel_id) |
There was a problem hiding this comment.
🤖 [P2] Bootstrap retained legacy references before retiring their source
A pre-upgrade kind:0 avatar can have neither an uploader row nor a media share when a name-only profile update reaches this call before backfill. The authorization check therefore records no share for the unchanged picture; the following UPDATE retires the old head. Backfill excludes that deleted pre-cutoff row and subjects the new post-cutoff row to the same empty-proof check, so rerunning it cannot restore the grant. The parameterized replacement path has the same problem for legacy emoji sets. Once #8444 enforces reads, an ordinary name update can make the retained avatar unreadable.
While the old head is still live, materialize its eligible pre-cutoff historical references before authorizing the replacement, or gate replacement until those grants exist. Keep this inside the replacement transaction/savepoint. Add a regression starting with an unbackfilled pre-cutoff profile/emoji set, replace it while retaining the media, then backfill and verify the surviving head still authorizes the blob.
| WHERE s.community_id = $1 AND s.sha256 = b.sha256 AND s.channel_id IS NULL) \ | ||
| OR EXISTS (SELECT 1 FROM media_shares s \ | ||
| JOIN events e ON e.community_id = s.community_id AND e.id = s.event_id \ | ||
| AND e.created_at = s.event_created_at AND e.deleted_at IS NULL \ |
There was a problem hiding this comment.
🤖 [P2] Tie edit-derived grants to the original message’s lifetime
Kind:40003 edits pass shared_blob_hashes and are stored with their own event ID, so an edit retaining or adding an attachment creates a separate share. Deleting the displayed message only tombstones its original ID (side_effects.rs:2050 and event.rs:1494–1503); the edit row stays undeleted. This join therefore continues authorizing the edit’s share after the message disappears. Desktop explicitly suppresses edits whose target was deleted (formatTimelineMessages.ts:275–277), so the visible deletion and the new grant model disagree. The same remaining grant can authorize a later share even before download enforcement lands.
Record the edit target as authorization provenance and require that target to remain live, or revoke the associated edit grants transactionally on original-message deletion. Apply the same lifetime rule during backfill so it cannot recreate them. Cover an attachment-bearing edit followed by both author and moderator deletion of the original message.
|
🤖 @wpfleger96 Please re-review the corrected backfill guarantees and the explicitly accepted limitation for recording/log-only rollout at |
wesbillman
left a comment
There was a problem hiding this comment.
🤖 Star Lord’s automated source review via Wes’s account, integrating independent Rocket (systems/lifecycle), Mantis (correctness/security/concurrency), and Groot (product/operator documentation) reviews.
One new P2 integration finding inline: the main merge brings in the database architecture contract from #8273, while this PR still introduces the fence triggers and foreign keys that contract prohibits. This is a source-demonstrated repository-contract conflict, not a newly demonstrated runtime exploit.
The six previous feature patches remain unchanged. The NIP-43 snapshot merge resolution uses the updated event hook consistently; no new runtime defect found in that integration. The backfill documentation now describes current-permission reevaluation. Optional wording: use “any backfill run” rather than “reruns” in the admin help and Db docs, since the first backfill can also reconsider a reference refused by live recording.
Prior review findings remain outstanding; this documentation/merge follow-up does not resolve the legacy-avatar or edit-lifetime issues or the public-description cleanup. They are not duplicated here. The stated acceptance of permission drift for recording/log-only rollout does not establish enforcement readiness.
Reviewed head 61ef918 against base 7e64612, focusing on changes since 7f7f21c. Source only: no tests, builds, SQL, app or CLI execution. CI snapshot: 44 successful, 29 skipped, 2 in progress, and Run Codex Security Review failed; failure cause unverified. Runtime migration parity, rollout behavior and Desktop edit/upload/profile/error-retry flows remain unverified. Public description inspected, no attached images; linked issue content not inspected. Non-blocking COMMENT review; no approval or merge authorization.
| PRIMARY KEY (community_id, sha256, event_id) | ||
| ); | ||
|
|
||
| SELECT attach_community_write_fence('media_uploaders'); |
There was a problem hiding this comment.
🤖 [P2] Reconcile both schema paths with the newly merged database contract
The main merge now includes docs/database.md Rules 1–2: new tables must not call attach_community_write_fence or add foreign keys, and tenant-owned tables must wait for the prerequisite removal of blanket fence attachment from desired-state bootstrap. These calls still add two triggers; lines 11 and 22 also introduce REFERENCES communities, repeated in schema/tables/public/media_{uploaders,shares}.sql. Explicit schema/triggers files and schema/community_write_fence.sql also install the fences, so removing just these migration calls leaves the bootstrap paths inconsistent.
This is new integration evidence from #8273: both paths add objects prohibited by the current schema contract and unsupported by its stated out-of-band apply model. Land the prerequisite bootstrap change, then remove the new fence/FK declarations consistently while retaining application transaction admission and deletion coverage. An intentional exception needs an explicit repository-owner decision. This finding does not claim a new runtime authorization failure.
Why?
Media is readable by anyone in a community who has its URL, even when it was posted in a private channel or DM. Restricting reads to the channels a file is shared in needs a record of where each file is shared — including everything posted before — so enforcement can land without hiding existing media.
What?
Migration 0067 adds uploader proof and a separate file grant for each source event. Each grant records the source message’s
event_idandevent_created_at, plus its channel or the whole community for a channel-less reference. A leaked URL creates no grant: the author must have uploaded the exact bytes or already have a readable share. Author-only and recipient-gated events share nothing. An agent’s kind:0 picture may use its registered owner’s upload proof; other fields cannot.Deleted or purged source events cannot authorize new shares. Profile and emoji replacements preserve references they are still entitled to publish before retiring the old head. The backfill skips deleted events and preserves each surviving reference as an independent grant. Download behavior stays unchanged here; #8444 uses these records to enforce reads.
Run
buzz-admin backfill-media-sharesafter every relay pod records shares. It trusts events received before the cutoff stored on each community and checks later references like live writes. The cutoff exists in both desired-state and SQLx-managed databases. Reruns deduplicate rows but re-check later references using current permissions, so an earlier refusal can become a grant after access changes. This migration limitation is accepted for recording and log-only rollout. Before enabling enforcement, explicitly accept this behavior or prevent reevaluation.Related issue
First half of BUZZ-358.
Testing
Bootstrapped isolated PostgreSQL 17 with
pgschema applyand the reconciliation script, then ranbuzz-admin backfill-media-shares --batch-size 1without a SQLx ledger. On the stacked local relay, backed by PostgreSQL, Redis, and an S3 HTTP fixture, all 27 media end-to-end tests passed. They cover private reads, owner-uploaded agent avatars, and author/moderator deletions that revoke only the removed post’s grant. The profile journey also retains a copied avatar across a name update, then revokes access when the picture is removed. The actual Desktop edit/upload/profile workflow smoke test is pending.Generated with Codex