Multimodal (image/document) input: forward A2A file/data parts to the model instead of silently dropping them

Problem

The A2A envelope models multi-part messages with media — but the runtime silently drops everything except text before it reaches the model. So a client can attach a document or image and get a normal-looking response that never actually considered the attachment, with no error and no signal that it was ignored.

What the protocol already supports

Message.Parts[] where Part.Kind is text / data / file (forge-core/a2a/types.go:57-59), and a file part carries media:

type FileContent struct {
    Name     string
    MimeType string   // image/png, application/pdf, ...
    URI      string   // by reference
    Bytes    []byte   // inline (base64 on the wire)
}

POST /tasks/send accepts and stores such a message fine.

Where it's dropped

When the loop projects the A2A message into the LLM request it keeps only text parts:

// forge-core/runtime/loop.go:943
var textParts []string
for _, p := range msg.Parts {
    if p.Kind == a2a.PartKindText && p.Text != "" {
        textParts = append(textParts, p.Text)
    }
}
return llm.ChatMessage{Role: role, Content: strings.Join(textParts, "\n")}

File and data parts never reach the model.
llm.ChatMessage.Content is a plain string (forge-core/llm/types.go:17) — no multimodal content array — and there's no image/vision path in the OpenAI/Anthropic providers, so the client type couldn't carry an image even if the loop forwarded it.
Inbound file bytes aren't persisted anywhere the agent could reach them either (the filesDir / .forge/files is agent output, not inbound), so there's no side-channel workaround.

(For contrast, the reverse direction works: an agent can return a file part — file_create emits PartKindFile, loop.go:804-822.)

Footgun to fix regardless of the full feature

Even before multimodal lands, accepted-but-dropped is a correctness footgun: a client attaching a PDF/image gets a 200 and a plausible answer that ignored the attachment, with nothing telling them. Minimum bar: when compression/projection drops non-text parts on input, either

return a clear error / 4xx ("this agent does not accept file/image input"), or
surface a warning field on the response / an audit event (input_parts_dropped with the dropped kinds + mimeTypes),

so the ignored-attachment case is visible instead of silent.

Proposed three-layer plan (full multimodal input)

LLM abstraction — give llm.ChatMessage a structured content-parts representation (text blocks + image blocks with {mime_type, bytes|url}), keeping the plain-string Content as the common/back-compat case.
Provider layer — translate image/document parts into each provider's native multimodal wire format (OpenAI image_url content parts; Anthropic image source blocks — anthropicContentBlock already exists as the seam). Gate on model capability so a non-vision model gets a clean error, not a malformed request.
Runtime projection — in loop.go, project file (and where relevant data) parts into image/document content blocks by mimeType instead of dropping them; decide the policy for unsupported types (reject vs. describe-as-text).

Scope / open questions

Which media to support first — images (vision) is the obvious first cut; PDFs/documents may need extraction or provider-native document support.
bytes (inline base64) vs uri (fetch) handling, and how uri interacts with the egress allowlist.
Size caps for inline bytes (mirror the audit/tracing capture caps posture).
Capability detection per model/provider so unsupported combos fail loudly.

Context

Accepted: forge-core/a2a/types.go (Part / FileContent)
Dropped: forge-core/runtime/loop.go:943 (text-only projection)
Client type: forge-core/llm/types.go:17 (ChatMessage.Content string)
Provider seam: forge-core/llm/providers/anthropic.go (anthropicContentBlock), forge-core/llm/providers/responses.go (responsesContentPart)
Output already works: forge-core/runtime/loop.go:804-822 (file_create -> PartKindFile)

fix(runtime): align REST inbound body cap with the JSON-RPC transport…

feat(runtime): raise inbound body cap to 32 MiB on both transports fo…

DoS checklist for the multimodal-ingest phases

Captured from the #527 (Phase 0) review so the media-consumption phases land with the right controls. A flat body cap (MaxBytesReader) is necessary but far from sufficient once we actually accept media — it only bounds one request's total bytes and doesn't catch decode bombs. The raise from 2 → 32 MiB belongs here, co-located with the items below (not in Phase 0, where media is rejected).

Threat model (once media is accepted)

Memory: json.Decode buffers the whole JSON value; base64-in-JSON inflates ~33% and then decodes to raw bytes. N concurrent 32 MiB requests ⇒ 32N MiB resident.
Decode bombs: a ~100 KB PNG can decode to GBs of pixels; a small PDF can carry an enormous page count. Byte limits don't catch this.
Storage: persisting inbound to .forge/files ⇒ disk exhaustion.
Cost amplification: forwarding large media upstream ⇒ token/$ blowup.
Slow-body (Slowloris): drip-feeding a large body ties up a goroutine.

Controls to land with media consumption

Split envelope cap from media cap. Keep the JSON text envelope small (~1–2 MiB); carry media out of JSON via a separate multipart/upload endpoint that streams bytes to a size-capped temp file in .forge/files and returns a file id the A2A message references by URI. Avoids base64-buffering media in RAM. (If base64-in-JSON must stay, use a streaming decoder with a per-part limit, not Decode-into-struct.)
Per-part limits: cap the number of file parts per message, and cap each part's decoded size with per-mime-type limits (e.g. image ≤ 5 MiB, doc ≤ 20 MiB). Check declared size/Content-Length before decoding.
Decode-dimension bounds (bomb defense): read image headers only (image.DecodeConfig) and reject on a pixel budget (e.g. > 40 MP) before full decode; bound PDF page count.
Concurrency semaphore for media requests so worst-case memory = maxBodyBytes × maxConcurrentMedia fits the pod's memory limit — pick the cap and concurrency together against that budget.
Per-principal rate limiting using the invoker identity from feat(audit): emit channel invoker identity in auth_verify #525 (tie media quotas to the authenticated caller, not just a global limit).
Body read deadline (beyond ReadHeaderTimeout) so a slow large body can't hold a connection open.
Pre-check provider limits (Anthropic/OpenAI image size/count) before forwarding — fail fast instead of paying for a rejected upstream call.

Phasing

Phase 0 (feat(runtime): reject inbound file/media parts loudly (#255 Phase 0) #527): media rejected ⇒ keep the body cap at 2 MiB on both transports.
Media phase: flip (capable model + supported mime) reject→accept, and in the same change raise the cap to the media ceiling with items 1–6 above. The big cap and the defenses that make it safe must ship together.

fix(runtime): keep inbound body cap at 2 MiB for Phase 0 (#255 review)

Merge pull request #527 from initializ/feat/multimodal-ingest-guard

Addendum (from #528 Phase-1 review) — Phase-4 symlink confinement

Adding one item to the DoS/security checklist above, specific to Phase 4 (persist inbound files to .forge/files):

8. Symlink-safe inbound file persistence. The read-side confinement (confinedFilesPath, used by RehydrateMedia and the file tools) is lexical — it rejects ..//absolute/out-of-dir via Abs+Rel, but does not EvalSymlinks. So a symlink inside .forge/files pointing outward would be followed by os.ReadFile. Harmless in Phases 0–1 (no untrusted URIs are populated yet), but once Phase 4 writes inbound, attacker-influenced media into that dir it becomes reachable. Mitigate by either:

writing inbound files with generated names (e.g. a content hash / uuid) into a dedicated subdir, never honoring an attacker-supplied filename, and never creating symlinks there; and/or
adding filepath.EvalSymlinks to the confinement check so a symlink that escapes the dir is rejected at read time.

Pairs with items 1–3 (per-part size/type limits + decode-dimension bounds): together they ensure inbound media is size-bounded, decode-bounded, and location-confined.

Merge pull request #528 from initializ/feat/multimodal-chatmessage-parts

Addendum (from #531 Phase-2 review) — image content is not scanned

Phase 2 (#531) forwards image input to vision models. One security dimension to track beyond the DoS checklist above:

9. Image-borne prompt injection bypasses the text scanners. a2aMessageToLLM keeps Content = PromptText() (which omits file bytes) as the text-of-record, so the guardrail/intent scanners see only the text of a turn — instructions embedded inside an image (screenshot of text, etc.) reach the vision model unscanned.

Compensating control (holds today): the action-layer guardrails — PathValidator (file confinement), egress enforcement, and the PDP — do not depend on prompt scanning. So an image can influence the model's reasoning but cannot bypass what the model is allowed to do. This bounds the blast radius: image injection ≈ persuading the model, not escalating its permissions.

Future hardening (optional, threat-model dependent): if image-borne injection becomes a concern, add image-content scanning before the model call — OCR-then-run-the-existing-text-scanners, or a vision-moderation pass — so image text is subject to the same guardrail/intent checks as message text. Not required for Phase 2 (documented as an accepted limitation), captured here so the decision is explicit.

Merge pull request #531 from initializ/feat/multimodal-image-input

Merge pull request #532 from initializ/feat/multimodal-dos-controls

Merge pull request #534 from initializ/feat/multimodal-document-input

Addendum (from #535 Phase-4 review) — inbound-media retention/GC (realizes item 3)

Phase 4 (#535) persists inbound media to <FilesDir>/inbound/<sha256>.<ext> (content-addressed, generated names — the item-8 write-side confinement is done correctly). But it lands the storage-exhaustion concern from item 3 with no cleanup story:

3b. Retention/GC for inbound/. Content-addressed files accumulate indefinitely — no TTL, size cap, or refcount. Content-addressing makes naive session-scoped deletion unsafe (a file is shared across sessions/turns, so deleting on one session's end can break another's replay). A long-running agent that receives media grows inbound/ without bound → eventual disk exhaustion. Per-request DoS is already bounded (body cap + per-part limits, #532); this is the long-term dimension.

Options for a follow-up:

Size-capped LRU over inbound/ (evict oldest by atime/mtime once total exceeds a budget) — simple, content-addressing-friendly (a re-upload just re-persists).
TTL sweep (delete files older than N days) — a re-upload after eviction re-persists on demand.
Refcount tied to which sessions reference each hash — most precise but most complex; probably overkill.

Since a re-upload re-persists idempotently, an LRU/TTL that occasionally evicts a still-referenced file degrades gracefully (that turn's replay loses the media → the provider serializer skips it → degrades to text, per #535's best-effort design). So a simple size-cap LRU is likely enough. Not required for Phase 4 (the persistence mechanism is correct); tracking so the disk-growth dimension isn't lost.

Merge pull request #535 from initializ/feat/multimodal-persist-inbound

Addendum (from #536 Phase-5 review) — item 3b also covers generated media

Phase 5 (#536) persists model-generated images via the same persistInboundMedia → <FilesDir>/inbound/<sha256>.<ext> path used for inbound media. So the retention/GC follow-up in item 3b is not inbound-only: inbound/ now accumulates both received and generated media, so any agent that generates images grows it too. Same fix (size-capped LRU / TTL) covers both — just noting the scope so the eventual cleanup isn't written as inbound-only.

Minor cleanup to fold in with 3b: the store is now shared by inbound + generated media, so persistInboundMedia / the inbound/ dir name are slightly misleading — a rename (persistMedia) and/or a generated/ subdir would read more clearly when the retention work lands.

#255 status: all core phases (0–5) are implemented (image + PDF input, DoS controls, cross-turn persistence, image output). Remaining follow-ups tracked here: retention/GC (3b, both directions), OpenAI Responses input_file document input, Gemini/Bedrock media, text-extraction fallback (+ its scan-coverage, item 9), remote-session media replay, and a live-API verification of the Responses image_generation_call wire-shape.

refactor(runtime): persistMedia with inbound/generated subdirs (#536 …

Merge pull request #536 from initializ/feat/multimodal-image-output

Navigation Menu

Repository navigation

Multimodal (image/document) input: forward A2A file/data parts to the model instead of silently dropping them #255

Problem

What the protocol already supports

Where it's dropped

Footgun to fix regardless of the full feature

Proposed three-layer plan (full multimodal input)

Scope / open questions

Context

DoS checklist for the multimodal-ingest phases

Threat model (once media is accepted)

Controls to land with media consumption

Phasing

Addendum (from #528 Phase-1 review) — Phase-4 symlink confinement

Addendum (from #531 Phase-2 review) — image content is not scanned

Addendum (from #535 Phase-4 review) — inbound-media retention/GC (realizes item 3)

Addendum (from #536 Phase-5 review) — item 3b also covers generated media

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Multimodal (image/document) input: forward A2A file/data parts to the model instead of silently dropping them #255

Description

Problem

What the protocol already supports

Where it's dropped

Footgun to fix regardless of the full feature

Proposed three-layer plan (full multimodal input)

Scope / open questions

Context

Activity

initializ-mk commented on Sep 24, 2026

DoS checklist for the multimodal-ingest phases

Threat model (once media is accepted)

Controls to land with media consumption

Phasing

initializ-mk commented on Sep 25, 2026

Addendum (from #528 Phase-1 review) — Phase-4 symlink confinement

initializ-mk commented on Sep 25, 2026

Addendum (from #531 Phase-2 review) — image content is not scanned

initializ-mk commented on Sep 28, 2026

Addendum (from #535 Phase-4 review) — inbound-media retention/GC (realizes item 3)

initializ-mk commented on Sep 28, 2026

Addendum (from #536 Phase-5 review) — item 3b also covers generated media

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions