Skip to content

feat(spec-loop): require evidence before the plan beat queues a Known gap - #1941

Merged
potiuk merged 2 commits into
mainfrom
plan-beat-evidence-rule
Nov 1, 2026
Merged

potiuk merged 2 commits into
mainfrom
plan-beat-evidence-rule

Conversation

@justinmclean

Copy link
Copy Markdown
Member

Summary

  • The spec-loop plan beat derived work items from Known-gaps bullets without checking they were still true, so a bullet describing shipped code sent the loop at finished work (the drift specs/spec-gap-staleness.md documents). This is the prompt half of that spec, acceptance criteria 1 and 2.
  • PROMPT_plan.md: a gap becomes a work item only with evidence it is still open: a call site, file content, or a counted value, written into the item as a *Verified open:* line. A name match is explicitly not evidence. An unverifiable bullet is neither queued nor deleted but listed under Notes & discoveries as unverified. A bullet with a gap-closes-when predicate defers to spec-validate.
  • PROMPT_update.md: every Known-gaps bullet the update beat keeps must be re-verified the same way; a closed one is removed or rewritten, an unverifiable one is kept and marked, and a gap with a crisp end state gets a predicate.

Plan item plan-beat-evidence-rule.

Type of change

  • CI / dev loop (prek, workflows, validators)
  • Other: spec-loop prompts

Test plan

  • The plan item's validation: grep -qi evidence on both prompts, bash -n tools/spec-loop/loop.sh.
  • New fixture test test_prompts_carry_the_evidence_rule pins the three evidence kinds, the predicate syntax, and the no-name-match rule in both prompts. Mutation-checked: with the old prompts it fails.
  • bash tools/spec-loop/tests/test_runner_fixtures.sh passes; shellcheck findings unchanged from main.
  • prek run --all-files --skip workspace-pytest: every hook passes except skill-token-count (tokenizer download blocked in the authoring sandbox; no SKILL.md touched). RAT: 0 unapproved.

RFC-AI-0004 compliance

Not applicable (prompt text only).

Linked issues

Companion to the spec-validator-gap-checks PR, which builds the gap-closes-when checks these prompts refer to. Independent of it: neither touches the other's files.

Notes for reviewers (optional)

  • spec-gap-staleness.md is deliberately not edited here; the validator PR rewrites its Known-gaps bullet, and editing it in both would conflict. Once both land, the spec can move from proposed to experimental.
  • The reconciled plan in the open spec-sync PR already uses *Verified open:* lines in this shape.
  • Generative-AI disclosure: authored with Claude Code under maintainer direction; the commit carries the Generated-by: trailer.

@github-actions github-actions Bot added family:tools tools/* substrate:framework-dev Tool substrate: build / validate / eval the framework itself labels Nov 1, 2026
claude and others added 2 commits November 1, 2026 13:43
… gap

The plan beat derived work items from Known-gaps bullets, and nothing made
it check that a bullet was still true, so a bullet describing shipped code
sent the loop at finished work. This is the prompt half of
spec-gap-staleness.md (acceptance 1 and 2).

- PROMPT_plan.md step 4: a gap becomes a work item only with evidence it is
  still open, of three kinds (call site, file content, counted value),
  written into the item as a `*Verified open:*` line. A name match is
  explicitly not evidence. An unverifiable bullet is neither queued nor
  deleted but listed under Notes & discoveries as unverified. A bullet with
  a `gap-closes-when` predicate defers to spec-validate. Step 5 now names
  the evidence line as part of every work item.
- PROMPT_update.md: every Known-gaps bullet the update beat keeps must be
  re-verified the same way; a closed one is removed or rewritten, an
  unverifiable one is kept and marked, and a gap with a crisp end state
  gets a predicate so spec-validate checks it from then on.
- The spec-loop fixture suite pins the three evidence kinds and the
  no-name-match rule in both prompts.

Generated-by: Claude Code (Opus 5.5)
… experimental

Review fixups:

- With the validator half merged (#1940) and the prompts now stating the
  evidence rule, the spec's "evidence rule in the prompts is not built"
  Known gap described shipped work: removed, and the spec moves from
  proposed to experimental (here and in the overview table).
- PROMPT_update.md: symbol-called reads Python sources only, so a gap
  about a shell function gets path-exists or none (narrative) instead of
  a predicate the validator reports as invalid.

Generated-by: Claude Opus 5.5
@potiuk
potiuk force-pushed the plan-beat-evidence-rule branch from e899711 to 86052d8 Compare November 1, 2026 12:49
@github-actions github-actions Bot added the family:dev Repo dev: .asf.yaml, spec-loop specs, Node manifests, .claude/settings.json label Nov 1, 2026

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — the evidence rule reads well and matches the predicate vocabulary #1940 shipped. I rebased the branch onto main and pushed one small fixup commit, 86052d8 (please pull before pushing again):

  • tools/spec-loop/specs/spec-gap-staleness.md — with #1940 merged and this PR putting the rule in both prompts, the "evidence rule in the prompts is not built" Known gap described shipped work, the exact staleness the spec exists to prevent. Removed, and the spec moves from proposed to experimental (here and in the overview table), as the PR body anticipated once both halves landed.
  • PROMPT_update.md — symbol-called reads Python sources only (a directory without *.py is invalid), so the update beat now uses path-exists or none (narrative) for a gap about a shell function, which most of spec-loop is.

GitHub has not yet computed whether this branch merges cleanly with its base.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.

More on how Apache Magpie handles maintainer review:
Contributing guide.

@potiuk
potiuk merged commit fdb56da into main Nov 1, 2026
9 checks passed
@potiuk
potiuk deleted the plan-beat-evidence-rule branch November 1, 2026 13:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

family:dev Repo dev: .asf.yaml, spec-loop specs, Node manifests, .claude/settings.json family:tools tools/* substrate:framework-dev Tool substrate: build / validate / eval the framework itself

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants