[feat]: run auto evaluation from traces

[ashrafchowdury]

What's new??

Run auto evaluation from traces, previously running evaluation on trace were only for the live evaluation. Now we are run auto evaluation from traces as well.

• Demo

QA scope

Evaluation and Observability

1. Running evaluations should work as expected (no changes there)
2. Go to observability and select some traces
3. Click + Add button and then Run auto evaluation
4. This should open the create new evaluation modal (the selected tab should be evaluator)
5. You should be able to see the traces tab and go there and deselect and select the already selected traces from the observability
6. Run the evaluation and it should work as expected (no changes there)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
agenta-documentation	Ready	Preview, Comment	Jun 24, 2026 1:21pm

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: f34ac490-07cc-4b04-be1f-8a160a926855

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/run-auto-evaluation-from-traces-queries

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Railway Preview Environment

Preview URL	https://gateway-production-0796.up.railway.app/w
Project	agenta-oss-pr-4794
Image tag	pr-4794-f2d7912
Status	Deployed
Railway logs	Open logs
Workflow logs	View workflow run
Updated at 2026-06-24T13:48:37.224Z

[ardaerzin]

good overall, thanks for the great work @ashrafchowdury.

however we can improve the ux a bit, currently we're silently filling query registry with auto generated "trace-eval" queries

[ashrafchowdury]

Fixed the issue with the following fixes:

1. Added alert message on the "New evaluation modal > Trace tab"
2. Display a toast message upon evaluation creation
3. Added meta.source when creating a query from the evaluation creation.
4. Added a new column to the Query registry name source to display the source of the creation