Skip to content

feat(skill-and-tool-validator): validate organization.md content, not just presence - #1942

Merged
potiuk merged 2 commits into
mainfrom
org-manifest-content-check
Nov 1, 2026
Merged

potiuk merged 2 commits into
mainfrom
org-manifest-content-check

Conversation

@justinmclean

Copy link
Copy Markdown
Member

Summary

  • validate_organization_structure only required README.md and organization.md to exist, so an empty manifest, or one missing a key, passed, and every key it omitted resolved silently to the framework default instead of the organization's value.
  • New validate_organization_manifests derives the required set from organizations/_template/organization.md: every key its uncommented yaml blocks declare, which is the organization identity (id, name, url, logo) and the governance vocabulary (seven keys). The template comments out the capability bundle, so those keys stay optional. Each adapter must declare every required key (null is a declared value, as independent uses it), organization_identity.id must match the directory name, and a manifest declaring no keys fails.
  • HARD, in the existing organization category, wired into run_validation. Stdlib-only parsing (fenced yaml blocks, two levels deep, comments declare nothing). A tree without the template skips the check.

Plan item org-manifest-content-check (specs/organization-adapters.md).

Type of change

  • Python package (tools/*/ with pyproject.toml)
  • CI / dev loop (prek, workflows, validators)

Test plan

  • 10 new tests: keys parsed from yaml fences only, inline comments stripped, commented blocks declare nothing; complete manifest clean with null values; an omitted required key named; optional blocks may be omitted or added; empty manifest fails; id must match the directory; no template skips; a missing manifest is left to the structure check; reached through run_validation; the in-tree adapters pass.
  • Live tree: the ASF and independent adapters pass as they stand (confirmed before landing, as the plan item asks); skill-and-tool-validate exits 0.
  • uv run --directory tools/skill-and-tool-validator --group dev pytest: 502 passed, 1 skipped; mypy clean; ruff clean.
  • prek run --all-files --skip workspace-pytest: every hook passes except skill-token-count (tokenizer download blocked in the authoring sandbox; no SKILL.md touched). RAT: 0 unapproved.

RFC-AI-0004 compliance

Not applicable (validator only).

Linked issues

None.

Notes for reviewers (optional)

  • The template never says "required" in words; it says omitted keys fall through. The live skeleton vs commented-out bundle is the split it does draw, and it is what the website relies on for identity. If a key should be optional after all, comment it out in the template and the check follows.
  • Generative-AI disclosure: authored with Claude Code under maintainer direction; the commit carries the Generated-by: trailer.

@github-actions github-actions Bot added family:tools tools/* substrate:framework-dev Tool substrate: build / validate / eval the framework itself labels Nov 1, 2026
claude and others added 2 commits November 1, 2026 11:26
… just presence

validate_organization_structure only required README.md and organization.md
to exist, so an empty manifest, or one missing a key, passed, and every key
it omitted resolved silently to the framework default instead of the
organization's value.

validate_organization_manifests derives the required set from
organizations/_template/organization.md: every key its uncommented yaml
blocks declare, which is the organization identity and the governance
vocabulary. The template comments out the capability bundle, so those keys
stay optional. Each adapter must declare every required key (null is a
declared value), organization_identity.id must match the directory name, and
a manifest declaring no keys fails. All three are HARD, in the existing
organization category, wired into run_validation. Parsing stays stdlib only:
the fenced yaml blocks, two levels deep, comments declaring nothing. A tree
without the template skips the check.

The ASF and independent adapters pass as they stand.

Generated-by: Claude Code (Opus 5.5)
…ad any indent

Review fixups:

- The template told authors "any key you omit falls through to the
  framework default", which the new HARD check now contradicts. It and
  organizations/README.md now say the identity and governance-vocabulary
  keys are required (null allowed) and only the capability bundle falls
  through.
- The parser read only two-space indented block YAML with LF endings, so
  a valid four-space or CRLF manifest was reported as declaring no keys.
  Any indent and CRLF are now read; the violation says the validator reads
  block-style YAML, since a flow mapping still is not.
- organization-adapters.md: the "presence, not content" Known gap is
  closed; the enforcement section describes the content check.

Generated-by: Claude Opus 5.5
@potiuk
potiuk force-pushed the org-manifest-content-check branch from e7ddaf5 to 0122d4c Compare November 1, 2026 11:25
@github-actions github-actions Bot added family:docs Docs, MISSION.md, READMEs family:dev Repo dev: .asf.yaml, spec-loop specs, Node manifests, .claude/settings.json labels Nov 1, 2026

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — deriving the required keys from the template keeps the check from drifting, and ASF and independent both pass. I rebased the branch onto main and pushed one fixup commit, 0122d4c (please pull before pushing again):

  • organizations/_template/organization.md and organizations/README.md — the template still said "any key you omit falls through to the framework default", which this HARD check now contradicts for the identity and governance-vocabulary keys. Both now say those keys are required (null allowed) and only the commented-out capability bundle falls through.
  • organization_manifest_keys read only two-space indented block YAML with LF endings, so a valid four-space or CRLF manifest was reported as "declares no keys". Any indent and CRLF are now read (with a test); the omitted-key message says the validator reads block-style YAML, since a flow mapping (block: { key: value }) still is not.
  • tools/spec-loop/specs/organization-adapters.md — the "presence, not content" Known gap is closed, and the enforcement section describes the content check.

GitHub has not yet computed whether this branch merges cleanly with its base.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.

More on how Apache Magpie handles maintainer review:
Contributing guide.

@potiuk
potiuk merged commit dc0da34 into main Nov 1, 2026
11 checks passed
@potiuk
potiuk deleted the org-manifest-content-check branch November 1, 2026 12:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

family:dev Repo dev: .asf.yaml, spec-loop specs, Node manifests, .claude/settings.json family:docs Docs, MISSION.md, READMEs family:tools tools/* substrate:framework-dev Tool substrate: build / validate / eval the framework itself

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants