Sync sensei skill + GEPA integration from upstream by spboyer · Pull Request #15097 · Azure/azure-sdk-tools

spboyer · 2026-04-13T16:25:57Z

Summary

Syncs the .github/skills/sensei folder with the upstream source at microsoft/GitHub-Copilot-for-Azure.

Fixes #15096

Changes

Sensei Skill Updates (v1.0.0 → v1.0.5)

SKILL.md: Rich help banner, batch mode, --skip-integration, trigger-overlap disambiguation
SCORING.md: Routing-regression guard logic, overlap exception rules, guard pseudocode
README.md / LOOP.md / EXAMPLES.md: testPathPatterns fix, link alignment

GEPA Integration (new)

scripts/gepa/auto_evaluator.py — Quality scoring evaluator (score, optimize, score-all)
references/TOKEN-INTEGRATION.md — Token budget integration docs
.github/workflows/gepa-quality-score.yml — CI pipeline for SKILL.md quality scoring
.github/workflows/gepa-quality-score-comment.yml — Posts score comments on PRs

Path Adjustments

Pipeline refs updated from plugin/skills/ → .github/skills/ for this repo's layout

Preserved (not modified)

eval.yaml, evals/, tasks/ — local eval configs kept as-is

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Syncs the Sensei skill content with upstream and adds GEPA-based quality scoring automation in CI for SKILL.md changes.

Changes:

Added GEPA quality scoring workflow + a follow-up workflow to comment results on PRs.
Introduced a Python auto-evaluator that scores (and can optionally optimize) skills based on content heuristics and discovered trigger tests.
Refreshed Sensei documentation (SKILL.md + references) with updated loop/scoring guidance.

Show a summary per file

File	Description
.github/workflows/gepa-quality-score.yml	Adds CI job to score skills and upload score JSON artifact.
.github/workflows/gepa-quality-score-comment.yml	Adds workflow_run job to post/update a PR comment with score results.
.github/skills/sensei/scripts/gepa/auto_evaluator.py	Adds GEPA evaluator CLI to score/optimize skills using discovered test harness data.
.github/skills/sensei/references/TOKEN-INTEGRATION.md	Documents token-budget workflow and CLI usage.
.github/skills/sensei/references/SCORING.md	Updates scoring guidance (anti-trigger exception rules, links, examples).
.github/skills/sensei/references/LOOP.md	Updates test command examples and token-budget references/links.
.github/skills/sensei/references/EXAMPLES.md	Updates examples + skill name references (e.g., azure-cost).
.github/skills/sensei/SKILL.md	Updates Sensei skill definition, help banner, loop steps, GEPA mode, references.
.github/skills/sensei/README.md	Updates documentation + Jest CLI flag (`testPathPatterns`) and links.

Copilot's findings

Files reviewed: 9/9 changed files
Comments generated: 13

ronniegeraghty · 2026-04-13T17:44:58Z

Hey @spboyer — nice work pulling this together! Heads up on the failing CI check:

The score job in .github/workflows/gepa-quality-score.yml is failing because it installs gepa==0.7.0, but that version doesn't exist on PyPI — the latest published version is 0.1.1.

The good news is the score job doesn't actually need the gepa package at all. The score and score-all commands in auto_evaluator.py are fully self-contained local heuristics — gepa is only used lazily inside the optimize code path.

Two options to fix:

Remove the gepa install from the workflow entirely — the score job never calls it. You can add it back later when/if an optimize job gets added.
Pin to the actual version (gepa>=0.1.0 or gepa==0.1.1) if you want it pre-installed for future optimize support.

Option 1 is probably cleanest since it avoids carrying an unused dependency. Let me know if you want me to push a suggestion!

spboyer · 2026-04-13T18:36:49Z

All 13 review threads addressed in 0e9516f. Fixed all remaining plugin/skills → .github/skills references. CI optimization suggestion noted for future. @ronniegeraghty — ready for another look.

Updates sensei skill from v1.0.0 to v1.0.5 and adds missing GEPA integration. Changes synced from microsoft/GitHub-Copilot-for-Azure. Updated files: - SKILL.md: v1.0.5 with help banner, batch mode, overlap disambiguation - SCORING.md: routing-regression guard, trigger-overlap rules - README.md, LOOP.md, EXAMPLES.md: testPathPatterns fix, link updates New files: - references/TOKEN-INTEGRATION.md: separate token integration docs - scripts/gepa/auto_evaluator.py: GEPA quality scoring evaluator - .github/workflows/gepa-quality-score.yml: CI quality scoring - .github/workflows/gepa-quality-score-comment.yml: PR score comments Path references updated from plugin/skills/ to .github/skills/ for this repository's layout. Fixes Azure#15096 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Updates path references across sensei skill docs, GEPA auto_evaluator defaults/usage examples, and workflow PR comment help text to match this repository's .github/skills/ layout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The upstream source pinned gepa==0.7.0 which is not available on PyPI. Latest available is 0.1.1. Using >= to pick up future releases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

spboyer · 2026-04-15T17:23:32Z

/skip check-enforcer

spboyer

2 findings (0 HIGH, 2 MEDIUM, 0 LOW)

Syncs the sensei skill from upstream with path corrections and adds the GEPA auto-evaluator for CI-based quality scoring. Documentation updates are clean and the path migration from plugin/skills/ to .github/skills/ looks complete. Two medium-severity items below on the new code.

ronniegeraghty · 2026-04-16T17:08:49Z

/check-enforcer override

spboyer requested review from Copilot and ronniegeraghty April 13, 2026 16:25

spboyer force-pushed the sync-sensei-from-upstream branch from 07f8f41 to 3b0e1d8 Compare April 13, 2026 16:29

spboyer enabled auto-merge (squash) April 13, 2026 16:29

Copilot AI reviewed Apr 13, 2026

View reviewed changes

Copilot started reviewing on behalf of spboyer April 13, 2026 16:37 View session

spboyer and others added 3 commits April 14, 2026 11:48

fix: update gepa dependency to available version (0.7.0 → >=0.1.1)

31f2b63

The upstream source pinned gepa==0.7.0 which is not available on PyPI. Latest available is 0.1.1. Using >= to pick up future releases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

spboyer force-pushed the sync-sensei-from-upstream branch from 1df2728 to 31f2b63 Compare April 14, 2026 15:48

ronniegeraghty approved these changes Apr 14, 2026

View reviewed changes

spboyer commented Apr 15, 2026

View reviewed changes

Comment thread .github/workflows/gepa-quality-score.yml

Comment thread .github/skills/sensei/scripts/gepa/auto_evaluator.py

spboyer merged commit a411192 into Azure:main Apr 16, 2026
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sync sensei skill + GEPA integration from upstream#15097

Sync sensei skill + GEPA integration from upstream#15097
spboyer merged 3 commits intoAzure:mainfrom
spboyer:sync-sensei-from-upstream

spboyer commented Apr 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ronniegeraghty commented Apr 13, 2026

Uh oh!

spboyer commented Apr 13, 2026

Uh oh!

spboyer commented Apr 15, 2026

Uh oh!

spboyer left a comment

Uh oh!

Uh oh!

Uh oh!

ronniegeraghty commented Apr 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

spboyer commented Apr 13, 2026

Summary

Changes

Sensei Skill Updates (v1.0.0 → v1.0.5)

GEPA Integration (new)

Path Adjustments

Preserved (not modified)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ronniegeraghty commented Apr 13, 2026

Uh oh!

spboyer commented Apr 13, 2026

Uh oh!

spboyer commented Apr 15, 2026

Uh oh!

spboyer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ronniegeraghty commented Apr 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants