Skip to content

docs(spec-loop): propose a per-item failure budget and harness-outage exclusion for batch skills - #1930

Merged
potiuk merged 3 commits into
mainfrom
batch-failure-budget-proposal
Nov 1, 2026
Merged

potiuk merged 3 commits into
mainfrom
batch-failure-budget-proposal

Conversation

@justinmclean

Copy link
Copy Markdown
Member

Summary

  • Every batch (sweep) skill documents "stop and surface" for any per-item error, so one bad issue sinks a whole sweep and a forge outage reads as a failure of every item it touched.
  • This PR proposes, as a proposed spec plus two plan items, a two-step fix: a per-item failure budget on the read/classify phase (the write phase keeps stop-and-surface so a retried post can never double-comment), and a harness-outage exclusion that classifies 429 / 5xx / reset / timeout / auth-expiry failures separately, retries once, and trips a breaker that reports an outage with no item marked failed.
  • The two steps are proposed together because the exclusion is meaningless without the budget. No skill, tool or eval changes here.

Type of change

  • Documentation (docs/, README.md, CONTRIBUTING.md)
  • Other: spec-loop spec and implementation plan

Test plan

  • uv run --project tools/spec-validator spec-validate tools/spec-loop/specs/ passes with the new spec
  • python3 tools/dev/check-doc-sync.py passes (spec indexed in specs/README.md and specs/overview.md)
  • prek run --files <changed files> passes (markdownlint, typos, lychee, spec-validate, check-doc-sync)

RFC-AI-0004 compliance

  • HITL — the proposal keeps every write gated and explicitly rules out retrying writes
  • Conversational + correctable — budget and breaker are proposed as adopter-tunable config keys with template defaults

Linked issues

None yet; the plan items batch-failure-budget and harness-outage-exclusion (blocked on the first) track the implementation.

Notes for reviewers (optional)

  • The failure classifier (which status codes mean "outage" on GitHub, GitLab, Forgejo, Jira, Bitbucket) is left to the tracker capability contract so each adapter maps its own; the spec does not pin per-adapter tables.
  • Later plan items were renumbered (5 to 9) and their cross-references updated.
  • Generative-AI disclosure: authored with Claude Code (Fable 5.1) under maintainer direction; commits carry the Generated-by: trailer.

@github-actions github-actions Bot added family:tools tools/* family:docs Docs, MISSION.md, READMEs substrate:framework-dev Tool substrate: build / validate / eval the framework itself labels Oct 31, 2026
@potiuk
potiuk force-pushed the batch-failure-budget-proposal branch from 90c77c4 to 71999df Compare November 1, 2026 00:33

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I rebased this branch onto main (90c77c4 → 71999df) to clear the conflict in IMPLEMENTATION_PLAN.md: main had replaced the plan's work-item list, so I kept main's list and appended your two items as 7 and 8 (batch-failure-budget, then harness-outage-exclusion, blocked on 7), dropping the renumbering of items that no longer exist. Please pull before pushing again. The proposal itself reads well; splitting the budget (read phase) from the never-retried write phase is the right line. Two points below, the first applying to your other open PRs too.

Agents are assistants, not authors (tools/spec-loop/specs/batch-failure-budget.md)

MUST NOT use Co-Authored-By: with an AI agent as co-author. Agents are assistants, not authors — attributing them as authors misrepresents contribution and is contrary to ASF policy on AI-assisted contributions.

The commit on this branch is authored as Claude <noreply@anthropic.com>; the Generated-by: trailer is right, but the author field makes the agent the author outright. It is the same on #1928, #1929 and #1931. When a PR is squash-merged, GitHub adds a Co-authored-by: trailer for every commit author other than the PR author, so the squash commit on main would carry Co-authored-by: Claude <noreply@anthropic.com>, which is the attribution this rule forbids (and which the commit-trailer guard cannot see, because the trailer is written by GitHub, not by a local git commit). None of the last 300 commits on main is authored that way.

Suggested fix: Reset the author on each commit to yourself, keeping the Generated-by: trailer: git rebase --exec 'git commit --amend --no-edit --reset-author' origin/main, then force-push; and set user.name / user.email in the environment the agent commits from so new commits are yours.

Smaller observations

  • See inline comments on tools/spec-loop/specs/batch-failure-budget.md:126.

This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The findings
below are observations, not blockers; an Apache Magpie
maintainer — a real person — will take the next look at the
PR. If you think a finding is mis-applied, please reply on
the PR and a maintainer will weigh in.

More on how Apache Magpie handles maintainer review:
Contributing guide.

@justinmclean
justinmclean force-pushed the batch-failure-budget-proposal branch 2 times, most recently from b24ba44 to 4720add Compare November 1, 2026 01:15
@justinmclean

Copy link
Copy Markdown
Member Author

Thanks Jarek. On the author point: ASF policy doesn't forbid this. The generative tooling page (https://www.apache.org/legal/generative-tooling.html) recommends either "Generated-by: " or "Co-authored-by: " in the commit message. The no-Co-Authored-By rule is this repo's own convention in AGENTS.md, not ASF policy.

@potiuk
potiuk force-pushed the batch-failure-budget-proposal branch from 2720ed6 to 2ebe0dc Compare November 1, 2026 07:18

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — the 403 rule now reads exactly as suggested, so a 403 on one unreadable item can no longer trip the breaker as a false outage. I rebased the branch onto main and pushed one small fixup commit, 2ebe0dc (please pull before pushing again):

  • tools/spec-loop/specs/batch-failure-budget.md: new acceptance criterion 6 pinning the rule (a 403 on one item while the auth check still passes is an item failure and does not advance the breaker); the later criteria are renumbered.
  • tools/spec-loop/IMPLEMENTATION_PLAN.md item 8: the harness bucket now carries the same 403 rule, so the classifier is not coded from the plan alone with every 403 as harness.

The PR description's "later plan items were renumbered (5 to 9)" no longer matches the diff against current main (it only adds items 7 and 8); the squash message below drops it.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.

More on how Apache Magpie handles maintainer review:
Contributing guide.

@potiuk
potiuk enabled auto-merge (squash) November 1, 2026 07:23
claude and others added 3 commits November 1, 2026 09:14
… exclusion for batch skills

Every batch (sweep) skill documents "stop and surface" for any per-item
error, so one bad issue sinks a whole sweep and a forge outage reads as a
failure of every item it touched. This is a proposal only; no skill
changes here.

- New `proposed` spec batch-failure-budget.md: step 1 wraps the
  read/classify phase so an item failure records `failed` and the sweep
  continues until a skill-declared budget is spent; the write phase keeps
  stop-and-surface because a retried post can duplicate a comment or
  label. Step 2 classifies failures as item or harness (429 / 5xx / reset
  / timeout / auth expiry), excludes harness failures from the budget,
  retries once after a back-off, and trips a separate consecutive-failure
  breaker that stops the sweep as an outage with no item marked failed.
  The exclusion ships only together with the budget.
- Two plan items, `batch-failure-budget` then `harness-outage-exclusion`
  (blocked on the first); later items renumbered.
- Spec indexed in specs/README.md and specs/overview.md.

Generated-by: Claude Code (Fable 5.1)
…s too

GitHub returns 403 for one item the token cannot read and for its
secondary rate limit, not only for a dead session. Classing every 403
as harness would let three unreadable items in a row trip the breaker
as a false outage with none recorded as failed. A 403 is now harness
only when its body says rate limit or the pre-flight auth check, re-run
straight after it, also fails; otherwise it is an item failure.

Generated-by: Claude Code (Opus 5.5)
Review fixups:

- Add an acceptance criterion: a 403 on one item while the pre-flight
  auth check still passes is an item failure and does not advance the
  breaker. Later criteria renumbered.
- Plan item 8 now carries the same 403 rule, so the classifier is not
  coded from the plan alone with every 403 as harness.

Generated-by: Claude Opus 5.5
@potiuk
potiuk force-pushed the batch-failure-budget-proposal branch from 2ebe0dc to 30df1d1 Compare November 1, 2026 09:52
@potiuk
potiuk merged commit 6fd069a into main Nov 1, 2026
9 checks passed
@potiuk
potiuk deleted the batch-failure-budget-proposal branch November 1, 2026 09:58
potiuk added a commit that referenced this pull request Nov 1, 2026
…ly real candidates

Review fixups, after moving the change onto main (#1930's 403 rule and
acceptance criterion 6 kept; the six skills restamped over the write-lock
block #1931 added):

- pr-stale-sweep's recap is built in code and ignored the new failed and
  unfinished fields, so a PR the sweep recorded as failed and carried on
  past disappeared from the final recap the skill prints as-is. The recap
  now leads with "N PR(s) failed and M not reached", lists each failed PR
  with its reason and each unreached one, and names the rerun selector.
- The sweep sized the budget on, and took unfinished from, every loaded PR,
  so drafts, bot PRs and PRs outside the label or the explicit numbers
  landed in the rerun selector. It now filters first and budgets only the
  PRs it would classify.
- The shared block pointed every skill at issue-tracker-config.md; it now
  names the file each family already reads (pr-management-config.md for
  the PR skills, project.md for security), which also drops the two config
  table rows that claimed pr-stale-sweep and security issue-triage read
  issue-tracker-config.md.
- The eval README lists the new item-failure-budget suite per skill, and
  the overview marks the spec experimental with step 1 shipped.

Generated-by: Claude Opus 5.5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

family:docs Docs, MISSION.md, READMEs family:tools tools/* substrate:framework-dev Tool substrate: build / validate / eval the framework itself

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants