Skip to content

fix(imap): normalize subjects before creating reply drafts - #22

Draft
MakerFlyFly wants to merge 4 commits into
audit/h09-basefrom
fix/imap-subject-normalization
Draft

MakerFlyFly wants to merge 4 commits into
audit/h09-basefrom
fix/imap-subject-normalization

Conversation

@MakerFlyFly

@MakerFlyFly MakerFlyFly commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

Summary

Folded root subjects prevented IMAP reply drafts, and RFC 2047 encoded reply prefixes could produce Re: Re: ….
Normalize the shared Subject before reply-prefix handling, retaining continuation spaces/tabs and the existing draft/threading rules.

Use lightweight standard-library decoding for ASCII headers, with a compatibility path for mixed raw 8-bit text. Normalize separators before and after decoding, tolerate invalid charset names locally, and return the already normalized text when decoding raises HeaderParseError.

Decode at most 8192 characters of unfolded Subject text: count after removing raw CR/LF, before decoding or separator normalization. Exactly 8192 still decodes; longer values bypass both decoders and retain the complete unfolded text with separator normalization. This is a Magpie resource tradeoff, not a protocol header-size limit. It introduces no truncation, suffix, or public configuration.

All four read operations (list_recent_threads, read_thread, list_drafts, list_sent_since) return unfolded, decoded subject values within the budget; oversized values can retain RFC 2047 transport encoding. CLI arguments and JSON field names/types remain compatible. Oversized subjects can degrade decoded display, prefilter matching, and reply-prefix deduplication. The budget covers summary decoding, not the runtime of the existing draft serializer.

Type of change

  • Python package (tools/mail-source/imap)

Review disposition

Review item Decision and reason
Decoded line separators can prevent drafts Normalize the complete separator set recognized by str.splitlines(), including separators produced by charset decoding.
NUL in a charset name prevents folder operations Catch (LookupError, ValueError) only around charset conversion and retain UTF-8 replacement fallback. UnicodeError is already a ValueError. Synthetic-byte reproduction is not a claim that a compliant production IMAP server returns NUL.
Crafted Subject decoding is expensive Keep the 8192-character budget before both decoder paths. Complete text is preserved, and no custom RFC parser is introduced.
Raw VT/FF and other separators disappear next to encoded words Normalize before decoding as well as afterwards.
Redundant replacement on HeaderParseError Removed. The fallback returns the already normalized value; a malformed-base64/FF test preserves its content and spacing.
Lower the budget to 4096 to reduce bounded per-message cost Not adopted. The 8192 tradeoff remains accepted; lowering it would expand the range of transport-encoded display/filtering without new business evidence. Timing remains observational, not a latency guarantee.
Oversized subjects reaching draft creation Add three formal CLI/FakeMailbox cases for encoded CRLF plus Bcc text, U+2028, and VT at 8193 characters. Existing behavior is preserved; these are not newly discovered defects.
A malformed base64 word causes whole-line fallback Deferred. Per-word recovery needs to preserve UTF-8 split across adjacent words. Malformed mixed input can still produce a duplicate Re:; this is not a regression from the upstream baseline.
Read values, controls, SPDX, and other headers Document and test all four read operations. Preserve NUL/ESC content and JSON escaping, retain the required SPDX line, and leave address parsing/global compat32/raw UTF-8 support outside this change.

Test plan

  • Complete IMAP suite: 165 passed on each of Python 3.11.15, 3.12.13, 3.13.9, 3.14.5, plus Linux.
  • Four new third-round cases also pass against previous head 9de8b1f8; the complete final suite passes there as well. This revision simplifies one production line and strengthens coverage.
  • Oversized draft cases use the shared CLI executor: exit 0, valid JSON, one saved draft, one Subject, no Bcc field, complete trailing text, matching serialized/returned subject, recipients, body, latest-parent In-Reply-To, References, and sent: false.
  • Existing coverage retains folding whitespace, Q/B decoding, full separator handling, NUL/unknown charset fallback, unrelated malformed mail across all five operations, all four read operations, 8191/8192/8193 boundaries, raw 8-bit compatibility, cross-word UTF-8, single decoding, and JSON escaping.
  • Fixed-seed random fuzz: 200,000 inputs per Python version, including ASCII/non-ASCII inputs with encoded-word markers; no uncaught exceptions or surviving separators.
  • Charset corpus: 575 / 575 / 576 / 583 names, each with Q/B payloads; no uncaught exceptions.
  • Nine bounded probes per version: six summary boundary cases and three oversized CLI draft cases. Each summary child has a 4-second timeout; each draft child has a 30-second timeout. Output integrity is checked; unit tests have no millisecond thresholds.
  • Ruff, format checking, and mypy.
  • Linux prek run --all-files --skip workspace-pytest.
  • prek run workspace-pytest --from-ref audit/h09-base --to-ref HEAD.
  • Real commit hooks and Magpie pre-first-pr-check: all categories pass, zero blocking/advisory items.

All mail tests use synthetic data and FakeMailbox. POSIX hooks and the hooked commit ran in an isolated Linux checkout; its exact Git tree was matched to the Windows staged tree before import.

Performance boundary and remaining serializer cost

The timed probes measure parse_message plus summarise for summaries and CLI draft creation for drafts. At exactly 8192 characters, the Python 3.12 observations were 0.041s for unterminated ASCII words and 0.034s for mixed raw 8-bit text; at 8193, both returned complete text in about 0.003s. Timings vary with runtime and scheduling.

A separate, same-host Python 3.12 comparison sends 20,000 unterminated encoded-word openers (140,000 characters) through the full draft path, with a 30-second timeout per child:

Subject implementation Draft creation
Upstream baseline dcfe0ad5 8.805s
Previous head 9de8b1f8 8.898s
Current head 72a05d08 8.941s

All three save a complete draft and return exit 0. This remains an existing serializer limitation for the targeted draft; the summary budget does not remove that cost. These single-run observations are not performance guarantees.

RFC-AI-0004 compliance

  • Vendor neutrality — standard-library handling in the IMAP Subject summary.
  • Write-access discipline — drafts remain drafts; recipients and thread-parent behavior are verified.

Linked issues

Related issue: #21.

Notes for reviewers

Fork audit only: fix/imap-subject-normalization targets audit/h09-base at dcfe0ad562b55d4df41bc69aee1f0f8289ee205b. Current head is 72a05d0879503d512b6db2c50445da81ee15d84c.

The four commits change only the Subject implementation and its tests. Published history, the audit baseline, and fork main are retained. No upstream issue or PR has been opened, and this PR remains a draft for human audit.

CI on the revised head passed: tests, prek, rat, Actions analysis, and labeler. The actual pytest (imap) job and Run pytest step succeeded; prek's normal checks, whole-repository lychee, and Claude Code mods steps also succeeded. Final code-check snapshot: 2026-10-25 14:17 UTC. The committed tree passed all local Linux gates above; earlier cancelled checks are not used as validation.

Gen-AI disclosure

AI-generated with Codex (GPT-6), following the AI contribution policy.
Published as a draft for human audit; no human review is claimed. All four commits carry Generated-by: Codex (GPT-6).

Unfold and decode the shared Subject summary before reply-prefix handling.
Replace line breaks produced by encoded words with spaces so draft headers
remain single-line. Cover folded, encoded and empty subjects with offline
read and draft round-trip regressions.

Generated-by: Codex (GPT-6)
@github-actions github-actions Bot added family:tools contract:mail-source Tool capability: inbound-mail ingestion (mbox / IMAP / ...) labels Oct 25, 2026
@MakerFlyFly MakerFlyFly added the bug Something isn't working label Oct 25, 2026 — with ChatGPT Codex Connector
Decode RFC 2047 subjects without constructing a header object on the normal
path. Normalize all line separators rejected by EmailMessage, retain raw
8-bit compatibility, and cover all four read operations.

Keep subjects intact and leave address parsing and compat32 unchanged.

Generated-by: Codex (GPT-6)
Skip Subject decoding above 8192 unfolded characters to protect folder
scans while retaining the entire header text. Normalize raw separators
before decoding and tolerate NUL-bearing charset names locally.

Cover both decoder paths, the budget boundary, and all operations through
the shared CLI executor in the presence of unrelated malformed mail.

Generated-by: Codex (GPT-6)
Return the already normalized Subject when RFC 2047 decoding fails,
removing a redundant separator substitution without changing behavior.

Add four checks for oversized draft structure and malformed base64
fallback, preserving the agreed 8192-character decoding budget.

Generated-by: Codex (GPT-6)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working contract:mail-source Tool capability: inbound-mail ingestion (mbox / IMAP / ...) family:tools

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant