Add PR gate tests for azd ai agent extension#8754
Open
v1212 wants to merge 4 commits into
Open
Conversation
v1212
requested review from
JeffreyCA,
RickWinter,
glharper,
hemarina,
therealjohn,
trangevi and
trrwilson
as code owners
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a PR-gate functional test suite for the azure.ai.agents extension using the existing recording/playback infrastructure, so ARM-dependent init flows can be validated in CI without Azure credentials.
Changes:
- Adds
record-tagged functional tests covering Tier 0 (offline) and Tier 1 (playback)azd ai agentscenarios. - Introduces a small
recordproxyhelper to route extension HTTP + ARM SDK traffic through the recorder proxy when running record/playback builds. - Extends the extension CI workflow to build record-tagged binaries and run the new Tier 0 + Tier 1 tests in PRs.
Reviewed changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| cli/azd/test/functional/ai_agent_test.go | Adds record-tagged functional tests for azd ai agent Tier 0 + Tier 1 playback coverage. |
| cli/azd/test/azdcli/cli.go | Adjusts mock credential server token expiry timestamp formatting for compatibility with extension parsing. |
| cli/azd/extensions/azure.ai.agents/internal/pkg/recordproxy/transport.go | Defines recordproxy.Transport as nil in non-record builds. |
| cli/azd/extensions/azure.ai.agents/internal/pkg/recordproxy/transport_record.go | Implements proxy-aware HTTP transport + sets http.DefaultTransport in record builds. |
| cli/azd/extensions/azure.ai.agents/internal/pkg/azure/client_options.go | Injects the recording proxy transport into ARM SDK client options when present. |
| .github/workflows/lint-ext-azure-ai-agents.yml | Adds a PR job to build record-tagged azd/extension and run Tier 0 + Tier 1 playback tests. |
v1212
force-pushed
the
wujia/ai-agent-pr-gate-tests
branch
3 times, most recently
from
d6ae7c8 to
030a36d
Compare
github-actions
Bot
added
the
ext-foundry
v1212
force-pushed
the
wujia/ai-agent-pr-gate-tests
branch
2 times, most recently
from
8ba6663 to
2d83046
Compare
Add recording-based integration tests for the azure.ai.agents extension: Tier 0 (offline, no Azure): - Version command - Help command - Init with deferred model lookup - Missing flags validation - Sample list (text and JSON output) - Doctor help Tier 1 (recording/playback): - Init with --project-id: validates full ARM flow (project lookup, model catalog, deployment selection) via cassette replay Infrastructure: - recordproxy package: injects proxy transport in record-tagged builds - client_options.go: routes ARM SDK calls through recording proxy - cli.go: fix UTC time format for fake credential server - CI workflow: builds azd + extension, runs Tier 0 and Tier 1 playback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
v1212
force-pushed
the
wujia/ai-agent-pr-gate-tests
branch
from
2d83046 to
8a0b29a
Compare
…egative control - WithProject: assert .env file has AZURE_AI_MODEL_DEPLOYMENT_NAME resolved value (not a placeholder), proving ARM calls in cassette were consumed - WithProject: cross-check agent.yaml has resolved value (not \ placeholder) - Add NegativeControl_BadCassette test: empty cassette → init MUST fail → proves Tier 1 isn't false-green - Update CI workflow regex to include negative control test in Tier 1 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tatic cassette - Pin .env assertion to exact cassette output (gpt-4.1), not loose Contains - Use filepath.Glob to discover .env dir (no hardcoded env name) - Fix negative control comments: failure is at first proxy call (registry/ARM), not specifically ARM - Add require.Contains for 'requested interaction not found' to lock failure reason - Replace runtime cassette creation with pre-committed static empty cassette (same pattern as Defer test) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Jun 22, 2026
📋 Prioritization NoteThanks for the contribution! The linked issue isn't in the current milestone yet. |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collaborator
Azure Dev CLI Install InstructionsInstall scriptsMacOS/Linux
bash: pwsh: WindowsPowerShell install MSI install Standalone Binary
MSI
Documentationlearn.microsoft.com documentationtitle: Azure Developer CLI reference
|
trangevi
requested changes
Jun 22, 2026
| @@ -0,0 +1,303 @@ | |||
| // Copyright (c) Microsoft Corporation. All rights reserved. | |||
Member
There was a problem hiding this comment.
Does this need to live in the broader azd test directory, or can it be moved inside the extension? Here it will require azd signoff for every change, not to mention that it would be best to keep the tests more closely tied to the extension
| require.Contains(t, result.Stdout, "doctor") | ||
| } | ||
|
|
||
| // --- Tier 1: Recording tests (ARM calls replayed from cassette) --- |
Member
There was a problem hiding this comment.
NIT: Perhaps the tier 1 tests should go in a separate file, to better distinguish between the tests which need recordings and the ones which don't?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #8760
Summary
Adds automated PR gate tests for the
azd ai agentextension using azd's existing recording/playback proxy infrastructure. These tests run in CI on every PR touching the extension, catching regressions without requiring live Azure credentials.Test Architecture
4-Process Recording Chain
The recording proxy intercepts all outbound HTTP from the extension, storing request/response pairs in YAML cassette files. During playback, the proxy replays stored responses — no network access needed.
Test Tiers
AZURE_RECORD_MODE=playbackinit --no-promptwith ARM calls replayed from cassette — model catalog queries, project resolution, scaffold generationTests (9 total, all passing)
Tier 0 — Offline (6 tests):
Test_AIAgent_Version— extension version commandTest_AIAgent_Help— help outputTest_AIAgent_Init_NoPrompt_MissingFlags— missing required flags → proper errorTest_AIAgent_SampleList— sample template listingTest_AIAgent_SampleList_JSON— JSON output with proper structure validation (json.Unmarshal + key check)Test_AIAgent_Doctor_Help— doctor subcommand helpTier 1 — Recording Playback (3 tests):
Test_AIAgent_Init_NoPrompt_Defer— init with--deploy-mode defer(no ARM calls, scaffold only)Test_AIAgent_Init_NoPrompt_WithProject— full init with--project-id, resolves Foundry project via ARM, auto-selects model from catalog, generates complete scaffoldTest_AIAgent_Init_NegativeControl_BadCassette— negative control: empty cassette → init fails with "requested interaction not found", proving the cassette is genuinely consumedKey Assertions (WithProject)
azure.yaml,agent.yaml, agent source directory.envfile containsAZURE_AI_MODEL_DEPLOYMENT_NAME="gpt-4.1"(resolved from cassette's model catalog, not a placeholder)agent.yamlhas resolved value (not${...}` env placeholder)Files Changed
New files
cli/azd/test/functional/ai_agent_test.gocli/azd/test/functional/testdata/manifests/foundry-toolbox.yamlcli/azd/test/functional/testdata/recordings/Test_AIAgent_Init_NoPrompt_WithProject.yamlcli/azd/test/functional/testdata/recordings/Test_AIAgent_Init_NoPrompt_Defer.yamlcli/azd/test/functional/testdata/recordings/Test_AIAgent_Init_NegativeControl_BadCassette.yamlcli/azd/extensions/azure.ai.agents/internal/pkg/recordproxy/transport.gocli/azd/extensions/azure.ai.agents/internal/pkg/recordproxy/transport_record.gohttp.DefaultTransportto route through proxy.github/workflows/lint-ext-azure-ai-agents.ymlModified files
cli/azd/extensions/azure.ai.agents/internal/pkg/azure/client_options.goproxyTransporterstruct —*http.Clientdirectly satisfiespolicy.Transporterinterfacecli/azd/test/azdcli/cli.go+00:00fromtime.RFC3339)cli/azd/extensions/azure.ai.agents/cspell.yamlrecordproxyto dictionaryCI Workflow Design
Key CI considerations:
azdis missing (Tier 0). Withazdpresent but not logged in → hard fail. The symlink is only created for Tier 1 where fake auth via recording session works.-mflag points to committedtestdata/manifests/foundry-toolbox.yamlinstead of GitHub URL, avoiding device-code auth in clean CI.~/.azd/config.jsonwith extension path, matching azd's discovery mechanism.Cassette Size
The
WithProjectcassette is 2.4 MB (47 interactions). For context, the repo's existing 29 cassettes have median 0.97 MB and max 4.80 MB — this is within normal range. The cassette was trimmed from 26 MB by keeping only target models (gpt-4o, gpt-4.1, gpt-4o-mini) in model catalog responses.How to Re-record
The cassette file is updated in-place. Trim if needed (model catalog responses are large).