Skip to content

fix(email): honor shared keyword, timeframe and OR filters in email fetch - #3611

Open
chubes4 wants to merge 4 commits into
mainfrom
fix/issue-3609-data-machine
Open

chubes4 wants to merge 4 commits into
mainfrom
fix/issue-3609-data-machine

Conversation

@chubes4

@chubes4 chubes4 commented Oct 13, 2026

Copy link
Copy Markdown
Member

Closes #3609

Why

The email fetch handler exposed the shared fetch settings (search, exclude_keywords, timeframe_limit) but ignored or misread them, so email flows looked filtered when they weren't. This came up while setting up a Festival Wire flow that should pick up festival lineup press releases from an inbox getting about 54 messages a day.

What changed

  • Keyword filters work and run before the cap. FetchEmailAbility applies the shared HasApplyKeywordSearch / HasApplyKeywordExclusion traits. It checks the subject first and reads the body only when the subject doesn't decide it. Body reads use FT_PEEK and never download attachments. It walks UIDs newest first and stops once offset + max_messages matches are found. The scan is bounded (datamachine_email_keyword_scan_limit, default 500) and logs a warning when the bound is hit. has_more reports whether unscanned candidates remain.
  • No change when no keywords are set. It's plain UID pagination as before, with no extra fetches.
  • Timeframes use the shared filter. timeframe_limit resolves through datamachine_timeframe_limit, so 72_hours (offered in the UI, used by every Reddit wire flow) now adds a SINCE instead of silently meaning "all time". Unknown values log a warning.
  • Several subjects or senders in one flow. Comma-separated from_filter / subject_filter expand to one criteria string per combination, because c-client imap_search() rejects OR (verified on production: Unknown search criterion: OR). The ability merges and dedupes the UIDs, newest first. A single value still produces a single string, as before. IMAP quoted strings are now escaped.
  • Ability schema. search_criteria accepts a string or an array of strings, and search and exclude_keywords are declared, so CLI and MCP callers can use them too.

Verification

The homeboy review test data-machine gate currently runs zero tests on clean main (PHPUNIT_ZERO_TESTS cause=recipe_run_no_executions / WP Codebox canonical PHPUnit discovery failed, tracked in Extra-Chill/homeboy#15333). Two Homeboy cooks for this issue failed on that infrastructure, not on the code. At the operator's direction, I finished the change outside Homeboy and verified it directly:

  • New tests/email-fetch-filters-smoke.php: 21 checks against an in-memory IMAP fixture. It covers include/exclude on subject and body, no fetches without keywords, body reads only when needed and always with peek, early stop and offset paging, the scan limit and its warning, OR merge/dedupe/order, 72_hours / all_time / unknown timeframe, the multi-value criteria expansion, and quote escaping. All pass.
  • Existing email-registration-contract-smoke.php and email-mailbox-runtime-smoke.php pass. named-mailbox-security-contract-smoke.php fails identically on clean main, which is unrelated and filed as tests: named-mailbox-security-contract-smoke fails on main (authRefProperty count 3, expects >= 10) #3610.
  • phpcs with the Homeboy WordPress ruleset: the changed source files are clean. The smoke file follows the existing multi-namespace smoke-test pattern.
  • php -l passes on all changed files.
  • Live, read-only check against the production inbox (headers only, nothing marked read), using the patched ability with two OR'd subject searches, search: lineup, exclude_keywords: WordCamp and a 7-day window. It returned High Water (both emails), Big Ears, WinterWonderGrass, Piedmont Stomp, Rooster Walk and Brandi Carlile's Girls Just Wanna Weekend, and it dropped WordCamp.

Note for flow config: exclude keywords also check the email body, so keep them narrow. In the live check, excluding film also dropped Big Ears, whose press release mentions film.

The Homeboy run evidence is recorded in cook runs agent-task-b13bd486… and agent-task-ec9dba4a…. The final commit replaces both cook candidates.

AI disclosure: written by Extra Chill Bot (Claude, via OpenCode), building on two Homeboy cook attempts (opencode / gpt-6-luna). Chris reviews and merges.

…etch

- Apply include/exclude keywords (subject, then body only when needed)
  before the max_messages cap, stopping once the page is filled and
  bounding the scan (datamachine_email_keyword_scan_limit, default 500).
- Resolve timeframe_limit through the shared datamachine_timeframe_limit
  filter so 72_hours works; unknown values log a warning.
- Comma-separated from_filter/subject_filter run one IMAP search per
  combination (c-client rejects OR) and merge UIDs newest first.
- Declare search, exclude_keywords and array search_criteria in the
  ability schema.

Closes #3609
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Email fetch handler ignores shared keyword filters, mis-maps 72_hours timeframe, and can't OR subjects/senders

1 participant