feat: validate skill reference files (#318)

* feat: validate skill reference files

Check every Markdown file under a skill's references/ directory for frontmatter, a direct link from SKILL.md, and code examples that typecheck. Require a skill tree entry to list its skill's reference files, and check the list against the files.

* perf: scan skill links only for skills with reference files

Author: LadyBluenotes

.changeset/validate-skill-references.md

@@ -0,0 +1,5 @@
+---
+'@tanstack/intent': patch
+---
+
+`intent validate` now checks every Markdown file under a skill's `references/` directory: the file must not begin with frontmatter, the `SKILL.md` body must link to it directly, and its TypeScript and JavaScript examples are checked with the skill's examples. A `skill_tree.yaml` entry for a skill with reference files must list them in a `references` key, and the list must match the files. Existing repositories now fail validation for reference files that `SKILL.md` does not link to directly, reference frontmatter, reference examples that do not typecheck, and tree entries that do not list their skill's reference files.

docs/cli/intent-validate.md

@@ -110,6 +110,39 @@ TypeScript 5.0 or newer must be available in the repository for code checking. I
 
 Relative Markdown links outside fenced examples must point to an existing file or directory. External URLs and anchors are not checked. Link checks still run when TypeScript is unavailable. Repeated validations read current source files and link targets.
 
+### Reference files
+
+Every `.md` file under a skill's `references/` directory, at any depth, is checked with its skill. No planning artifacts are required.
+
+- The file is plain Markdown: it does not begin with YAML frontmatter
+- The body of the owning `SKILL.md` links to the file with a direct relative Markdown link outside fenced examples, such as `[retry behavior](references/retries.md)`. A path in backticks or a link from another reference file does not count, because an agent finds a reference only through that link. A reference file can also link to another reference file
+- TypeScript and JavaScript fences are checked with the skill's fences, under the rules in [Code examples and links](#code-examples-and-links). Errors name the reference file and line. Relative links inside a reference file are not checked
+
+A `skill_tree.yaml` entry lists the skill's reference files in a `references` key, as paths relative to the skill directory:
+
+```yaml
+skills:
+  - name: React Table State
+    slug: table-state
+    package: packages/react-table
+    path: skills/table-state/SKILL.md
+    references:
+      - references/reactivity.md
+```
+
+Intent reads the tree from `<dir>/_artifacts` and, in a monorepo, from `_artifacts` at the workspace root. It matches an entry to a skill by the resolved `path`, so two packages can use the same slug. When a skill has a tree entry, the entry must agree with the files:
+
+- An entry for a skill that has reference files has a `references` key. The error shows the YAML lines to add
+- `references`, when present, is a list of strings
+
+- Each path has the form `references/<name>.md`, stays inside the skill directory (no absolute path and no `..` segment), and is listed once
+- Each listed file exists
+- Each `.md` file under `references/` is listed
+
+A skill without reference files needs no `references` key. A skill without a tree entry, or a repository without planning artifacts, gets only the file checks above.
+
+Upgrade impact: a repository that already has reference files fails validation when a reference file is not linked directly from its `SKILL.md`, begins with frontmatter, or contains a code example that does not typecheck, or when a matching `skill_tree.yaml` entry does not list the skill's reference files.
+
 ### Artifacts
 
 When `<dir>/_artifacts` exists, Intent also checks:

packages/intent/meta/generate-skill/SKILL.md

@@ -19,7 +19,7 @@ Work in the maintainer's library repository with their existing coding agent. Pr
 
 Use the current conversation, supplied diff, identified code/docs change, review report, or selected skill tree entry. Reuse the task, decisions, and evidence already established in this session; do not ask the maintainer to paste them again. Identify what developers need help doing and which package owns that task. If there is no usable task or review input, ask one question: “What do developers need help doing with this library?” Wait for the answer.
 
-Check repository instructions and Git status before edits; preserve unrelated changes. Read [maintainer commands](references/maintainer-commands.md) when setting up the repository, adding a skill, or preparing a batch for review. Use those commands for registration, metadata synchronization, and completion checks. Discover the package name, version, repository, skill root, and vocabulary from the repository. Use an established custom root; otherwise use `skills/` inside the owning package, including in monorepos.
+Check repository instructions and Git status before edits; preserve unrelated changes. Read [maintainer commands](references/maintainer-commands.md) when setting up the repository, adding a skill, or preparing a batch for review. When installing or updating CI, read [workflow security](references/workflow-security.md) before changing pins, triggers, or permissions. Use those commands for registration, metadata synchronization, and completion checks. Discover the package name, version, repository, skill root, and vocabulary from the repository. Use an established custom root; otherwise use `skills/` inside the owning package, including in monorepos.
 
 Before choosing or changing repository exports, follow [repository distribution](references/distribution.md). For an export-only change, complete that procedure against existing skill records; use the authoring steps below only if guidance must change.
 

packages/intent/meta/generate-skill/references/planning-records.md

@@ -35,11 +35,13 @@ skills:
     purpose: '[descriptive explanation; synchronized from metadata.purpose]'
     requires: ['[other skill slugs]'] # omit if none
     sources: ['src/[path].ts', 'owner/repo:docs/[path].md']
+    references: ['references/[file].md'] # required when the skill has reference files; relative to the skill directory
 ```
 
 - `domain_map.yaml` owns the domain/task relationships, supported failure modes, cross-references, tensions, and knowledge gaps. Keep task slugs and package ownership aligned with the skills.
 - `skill_spec.md` is the human-readable coverage and decision record. Retain the existing inventories and add a **Coverage and batch history** section recording the assessed scope, each completed batch or behavior change, its source revision/version, consequential decisions and reasons, check outcomes, and remaining work. Keep entries concise; do not store transcripts. Distinguish implemented guidance from planned work and unassessed areas.
 - `skill_tree.yaml` owns skill identities, paths, prerequisites, source mappings, and package placement. Keep existing planned entries and their status or explanatory notes. Add each new skill, and update entries when their names, paths, dependencies, or sources change. Resolve `path` relative to the owning package when `package` is present; otherwise resolve it relative to the library root. Keep `generated_from` links accurate for the actual artifact location.
+- `references` lists the `.md` files under the skill's `references/` directory. `intent validate` requires the key when the skill has reference files, and requires the list to match those files: each path has the form `references/<name>.md`, is listed once, and exists, and no file is left out. Omit the key, or use an empty list, for a skill without reference files. `intent validate` also checks every reference file as described in [the skill format](skill-format.md).
 
 Keep each tree entry's `description` aligned with its skill's activation description and its `purpose` aligned with `metadata.purpose`. Existing map/spec task explanations retain their meaning; do not replace them with trigger text. Record a purpose migration and its preserved source text in the batch review without discarding prior decisions.
 

packages/intent/meta/generate-skill/references/skill-format.md

@@ -61,6 +61,8 @@ Example pointer (create the reference only for a real retry branch):
 When configuring retries, read [retry behavior](references/retries.md) before choosing a policy.
 ```
 
+`intent validate` checks every `.md` file under `references/`. Write each reference as plain Markdown without frontmatter. Link to it from the `SKILL.md` body with a direct relative Markdown link; a link from another reference file is not sufficient. Its TypeScript and JavaScript fences are checked with the skill's fences.
+
 For a verification task (security, go-live, migration audit), use checks instead of setup/patterns: state what to inspect, the expected result, the failure condition, and the remediation. Include sourced mistakes and a final completion check. Use the applicable checklist template in [tree-generator's writing reference](../../tree-generator/references/write-skills.md#step-7--write-checklistaudit-skills-where-applicable) only when that detailed format is needed.
 
 For an entry selected from a full-library tree, retain its package placement, dependencies, failure-mode status, and cross-skill relationships. Read the applicable type template in [the tree writing reference](../../tree-generator/references/write-skills.md) when generating overview registries, framework trees, compositions, or cross-domain tension notes. The focused procedure remains in [generate-skill](../SKILL.md).

packages/intent/src/commands/validate.ts

@@ -1,10 +1,21 @@
-import { appendFileSync, existsSync, readFileSync } from 'node:fs'
+import {
+  appendFileSync,
+  existsSync,
+  readFileSync,
+  readdirSync,
+  statSync,
+} from 'node:fs'
 import { basename, dirname, join, relative, resolve } from 'node:path'
 import { fail, isCliFailure } from '../shared/cli-error.js'
 import { resolveProjectContext } from '../core/project-context.js'
 import { findWorkspacePackages } from '../setup/workspace-patterns.js'
 import { createIntentFsCache } from '../discovery/fs-cache.js'
-import { checkSkillBlocks, summarizeSkillExamples } from '../validate/blocks.js'
+import {
+  checkSkillBlocks,
+  skillLinks,
+  summarizeSkillExamples,
+} from '../validate/blocks.js'
+import { readIntentArtifacts } from '../staleness/artifact-coverage.js'
 import { writeChanges } from '../maintainer/files.js'
 import { repositoryWritePath } from '../shared/write-path.js'
 import {
@@ -15,6 +26,7 @@ import { printWarnings } from './support.js'
 import type { SkillBlockCache } from '../validate/blocks.js'
 import type { FileChange } from '../maintainer/files.js'
 import type { ProjectContext } from '../core/project-context.js'
+import type { IntentArtifactSkill } from '../shared/types.js'
 
 interface ValidationError {
   file: string
@@ -184,6 +196,47 @@ function collectPackagingWarnings(
   return warnings
 }
 
+function findReferenceFiles(skillDir: string): Array<string> {
+  const referencesDir = join(skillDir, 'references')
+  if (!statSync(referencesDir, { throwIfNoEntry: false })?.isDirectory())
+    return []
+  return readdirSync(referencesDir, { recursive: true, withFileTypes: true })
+    .filter((entry) => entry.isFile() && entry.name.endsWith('.md'))
+    .map((entry) =>
+      relative(skillDir, join(entry.parentPath, entry.name)).replaceAll(
+        '\\',
+        '/',
+      ),
+    )
+    .sort()
+}
+
+function readTreeEntries(
+  context: ProjectContext,
+  skillsDir: string,
+  artifacts: Map<string, ReturnType<typeof readIntentArtifacts>>,
+): Map<string, IntentArtifactSkill> {
+  const entries = new Map<string, IntentArtifactSkill>()
+  for (const [artifactRoot, root] of [
+    [skillsDir, context.packageRoot ?? context.cwd],
+    [context.workspaceRoot, context.workspaceRoot],
+  ]) {
+    if (!artifactRoot || !root) continue
+    if (!artifacts.has(artifactRoot))
+      artifacts.set(artifactRoot, readIntentArtifacts(artifactRoot))
+    for (const entry of artifacts.get(artifactRoot)?.skills ?? []) {
+      if (entry.artifactKind !== 'skill-tree' || !entry.path) continue
+      entries.set(
+        entry.package && !entry.path.startsWith(`${entry.package}/`)
+          ? resolve(root, entry.package, entry.path)
+          : resolve(root, entry.path),
+        entry,
+      )
+    }
+  }
+  return entries
+}
+
 function formatWarning({ file, message }: ValidationWarning): string {
   return `${file}: ${message}`
 }
@@ -404,6 +457,7 @@ async function runValidateCommandInternal(
   const setVersionPlans: Array<SetVersionPlan> = []
   let validatedCount = 0
   const validatedFiles = new Set<string>()
+  const artifacts = new Map<string, ReturnType<typeof readIntentArtifacts>>()
 
   if (skillsDirs.length === 0) {
     console.log('No skills/ directory found — skipping validation.')
@@ -425,7 +479,9 @@ async function runValidateCommandInternal(
       file: string
       content: string
       library: string | undefined
+      references: Array<{ file: string; content: string }>
     }> = []
+    const treeEntries = readTreeEntries(validateContext, skillsDir, artifacts)
     for (const filePath of skillFiles) {
       const rel = relative(process.cwd(), filePath)
       const content = readFileSync(filePath, 'utf8')
@@ -538,10 +594,101 @@ async function runValidateCommandInternal(
         ...collectAgentSkillSpecWarnings({ fm, rel }).map(formatWarning),
       )
 
+      const skillDir = dirname(filePath)
+      const skillName = basename(skillDir)
+      const referenceFiles = findReferenceFiles(skillDir)
+      const linked = new Set(
+        referenceFiles.length
+          ? skillLinks(match[2]!).map((link) => resolve(skillDir, link.path))
+          : [],
+      )
+      const references = referenceFiles.map((reference) => {
+        const referencePath = join(skillDir, reference)
+        const file = relative(process.cwd(), referencePath)
+        const referenceContent = readFileSync(referencePath, 'utf8')
+        if (
+          /^---\r?\n(?:[\s\S]*?\r?\n)?---[ \t]*(?:\r?\n|$)/.test(
+            referenceContent,
+          )
+        )
+          errors.push({
+            file,
+            message:
+              'Reference files must be plain Markdown without YAML frontmatter',
+          })
+        if (!linked.has(referencePath))
+          errors.push({
+            file,
+            message: `Not linked from SKILL.md — add a direct relative Markdown link such as [${basename(reference, '.md')}](${reference}) so an agent can find it`,
+          })
+        return { file, content: referenceContent }
+      })
+
+      const declared = treeEntries.get(filePath)?.references
+      if (declared === undefined) {
+        if (treeEntries.has(filePath) && referenceFiles.length)
+          errors.push({
+            file: rel,
+            message: `The skill_tree.yaml entry for skill "${skillName}" must list its reference files. Add to the entry:\n${[
+              'references:',
+              ...referenceFiles.map((reference) => `  - ${reference}`),
+            ]
+              .map((line) => `      ${line}`)
+              .join('\n')}`,
+          })
+      } else if (
+        !Array.isArray(declared) ||
+        declared.some((reference) => typeof reference !== 'string')
+      ) {
+        errors.push({
+          file: rel,
+          message: `The "references" value of the skill_tree.yaml entry for skill "${skillName}" must be a list of reference file paths`,
+        })
+      } else {
+        const seen = new Set<string>()
+        for (const reference of declared as Array<string>) {
+          if (seen.has(reference)) {
+            errors.push({
+              file: rel,
+              message: `Reference "${reference}" is declared more than once in skill_tree.yaml`,
+            })
+            continue
+          }
+          seen.add(reference)
+          if (
+            !/^references\/.+\.md$/.test(reference) ||
+            reference.includes('\\') ||
+            reference
+              .split('/')
+              .some((part) => part === '' || part === '.' || part === '..')
+          ) {
+            errors.push({
+              file: rel,
+              message: `Declared reference "${reference}" must be a references/<name>.md path inside the skill directory`,
+            })
+          } else if (!existsSync(join(skillDir, reference))) {
+            errors.push({
+              file: relative(process.cwd(), join(skillDir, reference)),
+              message:
+                'Declared in skill_tree.yaml but the file does not exist',
+            })
+          }
+        }
+        for (const reference of referenceFiles) {
+          if (!seen.has(reference))
+            errors.push({
+              file: relative(process.cwd(), join(skillDir, reference)),
+              message:
+                'Not declared in the "references" list of its skill_tree.yaml entry',
+            })
+        }
+      }
+
       checkedSkills.push({
         file: rel,
         content,
         library: readScalarField(fm, 'library'),
+        references,
       })
 
       const lineCount = content.split(/\r?\n/).length
@@ -576,6 +723,7 @@ async function runValidateCommandInternal(
           packageDir: validateContext.packageRoot,
           library,
           skills,
+          references: skills.flatMap((skill) => skill.references),
           cache: blockCache,
         })
         if (exampleSummaries)

packages/intent/src/shared/types.ts

@@ -136,6 +136,7 @@ export interface IntentArtifactSkill {
   packages: Array<string>
   sources: Array<string>
   covers: Array<string>
+  references?: unknown
 }
 
 export interface IntentArtifactCoverageIgnore {

packages/intent/src/staleness/artifact-coverage.ts

@@ -97,6 +97,7 @@ function parseSkills(
       packages: [...new Set(packages)].sort((a, b) => a.localeCompare(b)),
       sources: stringArray(skill.sources),
       covers: stringArray(skill.covers),
+      ...('references' in skill ? { references: skill.references } : {}),
     })
   }
 

packages/intent/src/validate/blocks.ts

@@ -448,13 +448,10 @@ async function splitExamples(
   return { content: lines.join(''), suggestions }
 }
 
-function checkSkillLinks(
-  root: string,
-  file: string,
+export function skillLinks(
   content: string,
-): Array<SkillBlockFinding> {
-  const findings: Array<SkillBlockFinding> = []
-  const absolute = resolve(root, file)
+): Array<{ target: string; path: string; line: number }> {
+  const links: Array<{ target: string; path: string; line: number }> = []
   // Blank out fenced examples, keeping newlines so line numbers still match.
   const lines = content.split(/\r?\n/)
   for (const fence of codeFences(content))
@@ -470,10 +467,27 @@ function checkSkillLinks(
     if (/^[a-z][a-z0-9+.-]*:/i.test(target) || target.startsWith('#')) continue
     const path = target.replace(/[#?].*$/, '')
     if (!path) continue
+    links.push({
+      target,
+      path,
+      line: prose.slice(0, match.index).split('\n').length,
+    })
+  }
+  return links
+}
+
+function checkSkillLinks(
+  root: string,
+  file: string,
+  content: string,
+): Array<SkillBlockFinding> {
+  const findings: Array<SkillBlockFinding> = []
+  const absolute = resolve(root, file)
+  for (const { target, path, line } of skillLinks(content)) {
     if (!existsSync(resolve(dirname(absolute), path)))
       findings.push({
         file,
-        line: prose.slice(0, match.index).split('\n').length,
+        line,
         message: `Link target not found: ${target}`,
         severity: 'error',
       })
@@ -626,6 +640,7 @@ export async function checkSkillBlocks(
     packageDir: string
     library: string
     skills: Array<{ file: string; content: string }>
+    references?: Array<{ file: string; content: string }>
     cache?: SkillBlockCache
   },
   ts?: typeof TS | null,
@@ -634,8 +649,8 @@ export async function checkSkillBlocks(
   const findings = options.skills.flatMap((skill) =>
     checkSkillLinks(root, skill.file, skill.content),
   )
-  const blocks = options.skills.flatMap((skill) =>
-    extractCodeBlocks(skill.file, skill.content),
+  const blocks = [...options.skills, ...(options.references ?? [])].flatMap(
+    (skill) => extractCodeBlocks(skill.file, skill.content),
   )
   const result = (skipped?: string): SkillBlockCheck => ({
     blocks: blocks.length,