Add live golden-path (Tier 2) pipeline for azd ai agent extension#8758
Open
v1212 wants to merge 14 commits into
Open
Add live golden-path (Tier 2) pipeline for azd ai agent extension#8758v1212 wants to merge 14 commits into
v1212 wants to merge 14 commits into
Conversation
Adds eng/pipelines/ext-azure-ai-agents-live.yml, an on-demand/weekly Azure DevOps pipeline that drives the real 'azd ai agent' CLI through tmux against live Azure (TME), exercising init -> provision -> deploy -> invoke -> down for both code and container deploy modes. This is the live counterpart to the PR-gate checks (Tier 0 offline + Tier 1 recording/playback in #8754). Per Azure SDK EngSys / SFI guidance, live access stays out of the automatic PR pipeline (trigger: none) and runs only via '/azp run ext-azure-ai-agents-live' or the weekly schedule. The Tier 2 tmux driver (test_full_e2e.py, test_tier2.py) is migrated from the #8692 prototype; CI auth detection is extended to recognize Azure DevOps (TF_BUILD) and an explicit E2E_USE_AZ_CLI_AUTH override.
v1212
added
ext-agents
added 2 commits
June 22, 2026 19:54
…zSessionActive The azure-sdk-tests service connection uses Workload Identity Federation, whose az session is isolated to the task's private AZURE_CONFIG_DIR and expires after ~10 min. Running the ~50 min golden-path test (and the cleanup) as plain bash steps after a separate login step would fail auth on both counts. Run them inside AzureCLI@2 with keepAzSessionActive:true (matching build-cli.yml) so the session stays refreshed and reaches azd (auth.useAzCliAuth) through tmux, which inherits AZURE_CONFIG_DIR. Subscription/tenant are now read in-script via az account show instead of cross-step pipeline variables.
test_tier2.py always ran sequentially, but kept a tautological if-condition (len==1 or len>1), an unused concurrent.futures import, a no-op --serial flag, and a docstring/print claiming parallel execution. Simplify to an explicit sequential loop and update the docstring to match. Also fix test_full_e2e.py's module docstring to point at README.md (LOCAL-TEST-GUIDE.md does not exist).
v1212
requested review from
JeffreyCA,
RickWinter,
danieljurek,
glharper,
tg-msft,
therealjohn,
trangevi,
trrwilson and
vhvb1989
as code owners
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a dedicated Azure DevOps pipeline and supporting Python drivers/docs to run the Tier 2 live golden-path E2E for the azure.ai.agents azd extension (init → provision → deploy → invoke → down) against real Azure resources, outside of the GitHub PR gate.
Changes:
- Added an on-demand + weekly ADO pipeline (
ext-azure-ai-agents-live) to run Tier 2 live E2E for bothcodeandcontainerdeploy modes. - Added a tmux-driven Python Tier 2 runner (
test_tier2.py) and a full golden-path driver (test_full_e2e.py) adapted for ADO CI auth detection. - Added documentation for running the live Tier 2 tests locally and in CI.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| eng/pipelines/ext-azure-ai-agents-live.yml | New ADO pipeline wiring to build azd + extension, run Tier 2 live E2E under AzureCLI@2, and publish logs/cleanup. |
| cli/azd/extensions/azure.ai.agents/tests/e2e-live/test_tier2.py | Tier 2 orchestrator: runs code + container golden paths sequentially with isolation and timeout handling. |
| cli/azd/extensions/azure.ai.agents/tests/e2e-live/test_full_e2e.py | tmux-driven end-to-end golden path driver (init/provision/deploy/invoke/down) with CI/local auth switching. |
| cli/azd/extensions/azure.ai.agents/tests/e2e-live/README.md | Docs for CI setup (ADO registration/service connection/secrets) and local WSL execution. |
- Use the ambient azure-sdk org secret `azuresdk-github-pat` for GH_TOKEN instead of an empty `GitHubPat` placeholder variable (mirrors eval-waza.yml); removes a misleading masked variable and the need for admin PAT setup. - Harden the AzureCLI@2 inline script: `set -euo pipefail` and assign-then-verify subscription/tenant so an `az account show` failure fails fast (a plain `export X=$(...)` would have masked the error from set -e). - Reword the extension-install comment to be self-contained (it no longer inaccurately claims to mirror lint-ext-azure-ai-agents.yml). - Clarify the test_full_e2e.py auth prerequisite: only local WSL runs leave auth.useAzCliAuth unset; CI auto-enables az CLI auth. - Clear tmux scrollback after env setup so the exported GH token cannot leak into capture() output on failures/timeouts. - _cleanup_leaked_resources now checks azd down's return code and reports failures instead of always printing "Cleanup complete".
- Stream child E2E output live with a watchdog-enforced hard timeout instead of buffering everything via capture_output - Shell-escape the GitHub token (shlex.quote) before exporting in tmux - Clean up the per-mode AZD_CONFIG_DIR temp copy unless E2E_KEEP_ARTIFACTS - Use sha256 instead of md5 for the agent-name uniqueness suffix - Derive the agent binary arch from uname -m instead of hard-coding amd64
4 tasks
- Shell-escape HOME/PATH/TENANT, the cd target, and the agent name with shlex.quote() (consistent with the earlier token fix) - On Tier 2 timeout, kill the child's detached tmux server so reused CI agents do not accumulate orphaned tmux sockets
… (Copilot round 7)
📋 Prioritization NoteThanks for the contribution! The linked issue isn't in the current milestone yet. |
…out pipe (Copilot round 9)
…um (Copilot round 10)
…ned python in CI (Copilot round 11)
… macro (Copilot round 12)
Collaborator
Author
|
/check-enforcer evaluate |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #8759
Summary
eng/pipelines/ext-azure-ai-agents-live.yml: an on-demand / weekly Azure DevOps pipeline that runs the Tier 2 live golden path (init → provision → deploy → invoke → down) for theazure.ai.agentsextension against a real Azure (TME) subscription, for bothcodeandcontainerdeploy modes.test_full_e2e.py,test_tier2.py) from the test: add static E2E tests for azure.ai.agents extension #8692 prototype intocli/azd/extensions/azure.ai.agents/tests/e2e-live/, adapting CI auth detection for Azure DevOps (TF_BUILD/E2E_USE_AZ_CLI_AUTH).README.mddocumenting how to run (CI + local) and the one-time ADO setup.Why a separate live pipeline
Per Azure SDK EngSys / SFI guidance, live Azure access must stay out of the automatic PR pipeline. This pipeline is
trigger: none/pr: noneand runs only:/azp run ext-azure-ai-agents-live(requires write permission), orTogether with the PR-gate tests in #8754 (Tier 0 offline + Tier 1 recording/playback), this covers Tier 0/1/2 from the original prototype (#8692).
Next steps to land this (admin / EngSys — cannot be done in PR code)
These are required to actually exercise the pipeline; the PR itself is inert in GitHub CI by design.
ext-azure-ai-agents-live(the exact name/azp runuses) against this repo + YAML path.azure-sdk-testsservice connection (theserviceConnectionparameter default) maps to the TME subscription with RBAC to create Foundry projects and deploy models (Contributor + Azure AI Developer + Cognitive Services Contributor, or equivalent)./azp run ext-azure-ai-agents-live— the first live validation of thekeepAzSessionActiveauth path (test + cleanup run insideAzureCLI@2so the WIFazsession survives the full multi-minute run).The GitHub token for
initis already wired via the ambientazuresdk-github-patorg secret ($(azuresdk-github-pat)), so no extra secret setup is needed.Testing
py_compileclean;README.mdpassescspell; the streaming / timeout / cleanup logic and the invoke-assertion regex in the Tier 2 driver were validated with standalone simulations.set -euo pipefail+ arch-derived binary name, shell-escaping (shlex.quote) of all interpolated values, a watchdog-enforced hard timeout with live-streamed child output, tmux-server teardown on timeout, per-modeAZD_CONFIG_DIRcleanup, a destructive-rm -rfguardrail, boundedsetup()subprocess calls, and a robust standalone-token check for the live invoke result (accepts4orfour, ignores incidental4s such asgpt-4o-mini/4.1/404).