Repository navigation
Compare agent skill sets in one run - #34
RobertCrupa wants to merge 1 commit into
Conversation
| })) as unknown as { | ||
| data?: { value?: unknown; subject?: { id?: string } }[]; | ||
| }; | ||
| const prev = (page.data ?? []).find((sc) => sc.subject?.id && sc.subject.id !== s.datasetRunId); |
There was a problem hiding this comment.
The digest compares this run against the newest pass_rate it finds, whatever experiment that is. In a run with two skill sets, the second variant's "previous run" is the first variant from the same Actor run. The digest then shows the difference between the two skill sets as if the suite had changed since the last run. The lookup should only consider runs with the same model and skill set.
| let flushFailed = false; | ||
|
|
||
| for (const model of models) { | ||
| const variants = models.flatMap((model) => skillSets.map((skillSet) => ({ model, skillSet }))); |
There was a problem hiding this comment.
The report picks one experiment per day and model. Both variants have the same model and fullScope: true. On a day without a scheduled run they cancel each other out and the day shows nothing. We should either add the skill set to the report key or state that comparison runs stay out of the report.
| // minute would otherwise merge into one experiment (three did on 22 Sept). | ||
| const runTag = (process.env.APIFY_ACTOR_RUN_ID ?? '').slice(-5).toLowerCase(); | ||
| if (runTag) nameParts.push(runTag); | ||
| const effectiveRunName = runName ?? nameParts.join(' · '); |
There was a problem hiding this comment.
When runName is set all variants get the same name. Langfuse then merges them into one experiment, same as on 22 Sept. Append the skills label or reject runName with more than one skill set. (Already true for models on main, I hit it today.)
A run can inject agent skills into the session and compare skill sets side by side: every model x skill set is its own experiment. - Skills come from apify/agent-skills at the commit pinned in DEFAULT_SKILLS_REF, downloaded once per run. `skillsRef` takes another GitHub ref, SKILLS_DIR a local directory. A run without skills downloads nothing; an unknown skill name fails before any session. - Inputs `skillSets`, `skillCombinations`, `skillsRef`, `maxSessions` (exit 15 when the run would start more sessions). - Claude Code starts with `--setting-sources project`, so personal skills never leak into the `none` variant. The span records `agentSkills` and `skillsLoaded`; traces get the tag `skills:<label>`. - Profiles list `agentSkills`; store-actors offers apify-ultimate-scraper.
058ce2c to
b0a20c4
Compare
3 participants
Stack 2 of 4, on top of #33.
A run can inject agent skills into the session and compare skill sets side by side. Each model × skill set is its own experiment.
What changes
apify/agent-skillsonce per run, at the commit pinned inDEFAULT_SKILLS_REF(d522584). It is a full commit hash, so the runner does not call the GitHub API.skillsReftakes another GitHub ref, andSKILLS_DIRtakes a local directory. The repo holds no copy of the skills.skillSets(none,a,a+b),skillCombinationsandskillsRef.maxSessionsstops a run with exit 15 when models × skill sets × scenarios × repeats is larger than the limit.--setting-sources project, so personal skills never get into thenonevariant.agentSkillsandskillsLoaded, traces get the tagskills:<label>, and OUTPUT hasvariants, best first.agentSkills.store-actorsoffersapify-ultimate-scraper.This PR does not change the judge. The judge learns about the agent setup in #35.
Tests: workflow-runner 76, tools 9.