Repository navigation
docs(spec-loop): propose a per-item failure budget and harness-outage exclusion for batch skills - #1930
Conversation
90c77c4 to
71999df
Compare
potiuk
left a comment
There was a problem hiding this comment.
I rebased this branch onto main (90c77c4 → 71999df) to clear the conflict in IMPLEMENTATION_PLAN.md: main had replaced the plan's work-item list, so I kept main's list and appended your two items as 7 and 8 (batch-failure-budget, then harness-outage-exclusion, blocked on 7), dropping the renumbering of items that no longer exist. Please pull before pushing again. The proposal itself reads well; splitting the budget (read phase) from the never-retried write phase is the right line. Two points below, the first applying to your other open PRs too.
Agents are assistants, not authors (tools/spec-loop/specs/batch-failure-budget.md)
MUST NOT use
Co-Authored-By:with an AI agent as co-author. Agents are assistants, not authors — attributing them as authors misrepresents contribution and is contrary to ASF policy on AI-assisted contributions.
The commit on this branch is authored as Claude <noreply@anthropic.com>; the Generated-by: trailer is right, but the author field makes the agent the author outright. It is the same on #1928, #1929 and #1931. When a PR is squash-merged, GitHub adds a Co-authored-by: trailer for every commit author other than the PR author, so the squash commit on main would carry Co-authored-by: Claude <noreply@anthropic.com>, which is the attribution this rule forbids (and which the commit-trailer guard cannot see, because the trailer is written by GitHub, not by a local git commit). None of the last 300 commits on main is authored that way.
Suggested fix: Reset the author on each commit to yourself, keeping the Generated-by: trailer: git rebase --exec 'git commit --amend --no-edit --reset-author' origin/main, then force-push; and set user.name / user.email in the environment the agent commits from so new commits are yours.
Smaller observations
- See inline comments on
tools/spec-loop/specs/batch-failure-budget.md:126.
This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The findings
below are observations, not blockers; an Apache Magpie
maintainer — a real person — will take the next look at the
PR. If you think a finding is mis-applied, please reply on
the PR and a maintainer will weigh in.More on how Apache Magpie handles maintainer review:
Contributing guide.
b24ba44 to
4720add
Compare
|
Thanks Jarek. On the author point: ASF policy doesn't forbid this. The generative tooling page (https://www.apache.org/legal/generative-tooling.html) recommends either "Generated-by: " or "Co-authored-by: " in the commit message. The no-Co-Authored-By rule is this repo's own convention in AGENTS.md, not ASF policy. |
2720ed6 to
2ebe0dc
Compare
potiuk
left a comment
There was a problem hiding this comment.
Thanks — the 403 rule now reads exactly as suggested, so a 403 on one unreadable item can no longer trip the breaker as a false outage. I rebased the branch onto main and pushed one small fixup commit, 2ebe0dc (please pull before pushing again):
tools/spec-loop/specs/batch-failure-budget.md: new acceptance criterion 6 pinning the rule (a 403 on one item while the auth check still passes is an item failure and does not advance the breaker); the later criteria are renumbered.tools/spec-loop/IMPLEMENTATION_PLAN.mditem 8: the harness bucket now carries the same 403 rule, so the classifier is not coded from the plan alone with every 403 as harness.
The PR description's "later plan items were renumbered (5 to 9)" no longer matches the diff against current main (it only adds items 7 and 8); the squash message below drops it.
This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.More on how Apache Magpie handles maintainer review:
Contributing guide.
… exclusion for batch skills Every batch (sweep) skill documents "stop and surface" for any per-item error, so one bad issue sinks a whole sweep and a forge outage reads as a failure of every item it touched. This is a proposal only; no skill changes here. - New `proposed` spec batch-failure-budget.md: step 1 wraps the read/classify phase so an item failure records `failed` and the sweep continues until a skill-declared budget is spent; the write phase keeps stop-and-surface because a retried post can duplicate a comment or label. Step 2 classifies failures as item or harness (429 / 5xx / reset / timeout / auth expiry), excludes harness failures from the budget, retries once after a back-off, and trips a separate consecutive-failure breaker that stops the sweep as an outage with no item marked failed. The exclusion ships only together with the budget. - Two plan items, `batch-failure-budget` then `harness-outage-exclusion` (blocked on the first); later items renumbered. - Spec indexed in specs/README.md and specs/overview.md. Generated-by: Claude Code (Fable 5.1)
…s too GitHub returns 403 for one item the token cannot read and for its secondary rate limit, not only for a dead session. Classing every 403 as harness would let three unreadable items in a row trip the breaker as a false outage with none recorded as failed. A 403 is now harness only when its body says rate limit or the pre-flight auth check, re-run straight after it, also fails; otherwise it is an item failure. Generated-by: Claude Code (Opus 5.5)
Review fixups: - Add an acceptance criterion: a 403 on one item while the pre-flight auth check still passes is an item failure and does not advance the breaker. Later criteria renumbered. - Plan item 8 now carries the same 403 rule, so the classifier is not coded from the plan alone with every 403 as harness. Generated-by: Claude Opus 5.5
2ebe0dc to
30df1d1
Compare
…ly real candidates Review fixups, after moving the change onto main (#1930's 403 rule and acceptance criterion 6 kept; the six skills restamped over the write-lock block #1931 added): - pr-stale-sweep's recap is built in code and ignored the new failed and unfinished fields, so a PR the sweep recorded as failed and carried on past disappeared from the final recap the skill prints as-is. The recap now leads with "N PR(s) failed and M not reached", lists each failed PR with its reason and each unreached one, and names the rerun selector. - The sweep sized the budget on, and took unfinished from, every loaded PR, so drafts, bot PRs and PRs outside the label or the explicit numbers landed in the rerun selector. It now filters first and budgets only the PRs it would classify. - The shared block pointed every skill at issue-tracker-config.md; it now names the file each family already reads (pr-management-config.md for the PR skills, project.md for security), which also drops the two config table rows that claimed pr-stale-sweep and security issue-triage read issue-tracker-config.md. - The eval README lists the new item-failure-budget suite per skill, and the overview marks the spec experimental with step 1 shipped. Generated-by: Claude Opus 5.5
Summary
proposedspec plus two plan items, a two-step fix: a per-item failure budget on the read/classify phase (the write phase keeps stop-and-surface so a retried post can never double-comment), and a harness-outage exclusion that classifies 429 / 5xx / reset / timeout / auth-expiry failures separately, retries once, and trips a breaker that reports an outage with no item marked failed.Type of change
docs/,README.md,CONTRIBUTING.md)Test plan
uv run --project tools/spec-validator spec-validate tools/spec-loop/specs/passes with the new specpython3 tools/dev/check-doc-sync.pypasses (spec indexed inspecs/README.mdandspecs/overview.md)prek run --files <changed files>passes (markdownlint, typos, lychee, spec-validate, check-doc-sync)RFC-AI-0004 compliance
Linked issues
None yet; the plan items
batch-failure-budgetandharness-outage-exclusion(blocked on the first) track the implementation.Notes for reviewers (optional)
Generated-by:trailer.