Export tool metadata after NuGet releases by conniey · Pull Request #3867 · microsoft/mcp

Connie Yau (conniey) · 2026-10-06T16:36:32Z

What does this PR do?

Runs ToolMetadataExporter after Azure MCP NuGet releases and publishes generated tool-change metadata as a pipeline artifact.
Adds release configuration, historical-version export tooling, malformed JSON recovery, tests, and updated documentation.
Adds per server querying and ingestion
Adds support for "dry-run" when running with Template.Mcp.Server

Validation:

dotnet test -c Debug --project .\tests\ToolMetadataExporter.Tests\ToolMetadataExporter.Tests.csproj --filter-class "ToolMetadataExporter.Tests.UtilityTests" — passed (2 tests).
dotnet build .\src\ToolMetadataExporter.csproj --no-restore --nologo — passed with 0 warnings.

GitHub issue number?

https://github.com/microsoft/mcp-pr/issues/268

Pre-merge Checklist

azure-pipelines · 2026-10-06T16:36:56Z

Azure Pipelines: Successfully started running 1 pipeline(s). There may be pipelines that require an authorized user to comment /azp run to run.

Copilot

Copilot review overview

🟡 Changes recommended

JSON recovery remains incomplete, and the historical export script has unsafe defaults and misleading success behavior.

Review effort: Balanced
Findings: 4

Open (4)

Require PackagesDirectory instead of using a developer-specific default · New
Honor WhatIf by deferring ledger creation until successful export · New
Make package directory enumeration failures terminating · New
Escape all control characters inside JSON strings during recovery · New

What changed in this PR

Adds post-release Azure MCP tool metadata export and historical backfill support.

Changes:

Adds release pipeline export and artifact publication.
Adds historical-version processing and malformed JSON recovery.
Expands tests and documentation.

File	Description
`eng/tools/ToolMetadataExporter/tests/ToolMetadataExporter.Tests/UtilityTests.cs`	Tests malformed JSON recovery.
`eng/tools/ToolMetadataExporter/src/Utility.cs`	Escapes malformed JSON control characters.
`eng/tools/ToolMetadataExporter/src/ToolMetadataExporter.csproj`	Enables user-secrets configuration.
`eng/tools/ToolMetadataExporter/src/README.md`	Removes superseded nested documentation.
`eng/tools/ToolMetadataExporter/src/Export-DownloadedVersions.ps1`	Adds historical-version exports.
`eng/tools/ToolMetadataExporter/README.md`	Documents configuration and workflows.
`eng/scripts/Invoke-ReleaseToolMetadataExport.ps1`	Exports metadata from signed packages.
`eng/pipelines/templates/variables/tool-metadata-exporter.yml`	Defines release datastore settings.
`eng/pipelines/templates/jobs/release.yml`	Adds the metadata export release job.
`eng/pipelines/templates/common.yml`	Loads metadata exporter variables.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

🟡 Changes recommended

The release path cannot locate the Windows executable or output directory and may corrupt metadata during non-Azure releases.

2 open findings

Scope exporter job to Azure.Mcp.Server · New
Include platform extension when locating executable · New

4 resolved since last review

Escape all control characters inside JSON strings during recovery
Make package directory enumeration failures terminating
Honor WhatIf by deferring ledger creation until successful export
Require PackagesDirectory instead of using a developer-specific default

🧠 Review effort: Balanced

Connie Yau (conniey) · 2026-10-08T05:32:54Z

+    /// Escapes raw control characters inside JSON strings while preserving structural whitespace.
+    /// Historical azmcp versions emitted characters such as U+001A unescaped inside tool descriptions.
+    /// </summary>
+    private static string EscapeInvalidJsonStringControlCharacters(string json)


added because 3.0.0-beta.21 has control characters that are not escaped correctly when returning JSON resulting in a JSON parse exceptions.

Alan Zimmer (alzimmermsft) · 2026-10-13T18:30:07Z

+    ServerName: ${{ parameters.ServerName }}
+    DependsOn:
+    - TagRepository
+    - PublishNugetPackage


PublishNugetPackage already depends on TagRepository, do we need to include it as a dependency here as well?

Nah. I just wanted to look the same as the other ones and then you'd see there's an additional stage it needed. ;P

Alan Zimmer (alzimmermsft) · 2026-10-13T18:41:10Z

Higher level thought, not sure where to put it, but I'm going to put it here.

We do a lot of effort to build and run tool listing in quite a few places, I'm wondering if we'll be better off by having a source code parser that can get us this information. That'll also allow us to not worry about tools which require external applications to be installed (AZD, azqr, etc).

That would be nice and a future improvement but since a lot of our existing analyses use the tool at runtime to check tools, I kept it at that.

Copilot

🟡 Changes recommended

The release path can ingest without producing its artifact, and forced WhatIf previews incorrectly skip prior exports.

2 open findings

2 resolved since last review

🧠 Review effort: Balanced

+    throw "ToolMetadataExporter build failed with exit code $LASTEXITCODE."
+}
+
+$env:AppConfig__WorkDirectory = [System.IO.Path]::GetFullPath($ArtifactDirectory)


+$successfulPaths = [System.Collections.Generic.HashSet[string]]::new(
+    [System.StringComparer]::OrdinalIgnoreCase)
+
+if (Test-Path -LiteralPath $SuccessfulPathsFile -PathType Leaf) {


Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Connie Yau (conniey) requested review from a team as code owners October 6, 2026 16:36

Connie Yau (conniey) requested review from KarishmaGhiya, Vinay Gera (g2vinay), msalaman, Sai Koumudi Kaluvakolanu (saikoumudi), Sandeep Sen (sandeep-sen), Tom Meschter (tmeschter) and Jeff Fisher (xirzec) and a balanced review from Copilot October 6, 2026 16:36

Copilot started reviewing on behalf of Connie Yau (conniey) October 6, 2026 16:37

View session

Copilot AI reviewed Oct 6, 2026

View reviewed changes

Connie Yau (conniey) force-pushed the conniey/ci-tool-metadata branch from 25af259 to b560620 Compare October 7, 2026 19:35

Connie Yau (conniey) requested a balanced review from Copilot October 7, 2026 20:36

Copilot started reviewing on behalf of Connie Yau (conniey) October 7, 2026 20:37

View session

Copilot AI reviewed Oct 7, 2026

View reviewed changes

Comment thread eng/pipelines/templates/jobs/release.yml Outdated

Comment thread eng/scripts/Invoke-ReleaseToolMetadataExport.ps1 Outdated

Connie Yau (conniey) force-pushed the conniey/ci-tool-metadata branch from b560620 to 78b3497 Compare October 7, 2026 22:23

Connie Yau (conniey) commented Oct 8, 2026

View reviewed changes

Alan Zimmer (alzimmermsft) reviewed Oct 13, 2026

View reviewed changes

Connie Yau (conniey) requested a balanced review from Copilot October 13, 2026 19:20

Copilot started reviewing on behalf of Connie Yau (conniey) October 13, 2026 19:20

View session

Copilot AI reviewed Oct 13, 2026

View reviewed changes

Connie Yau (conniey) force-pushed the conniey/ci-tool-metadata branch from 588e477 to 94e1f37 Compare October 16, 2026 01:54

Connie Yau (conniey) added 5 commits October 19, 2026 13:18

Adds logic to fix JSON with control characters.

5b8cc94

- Update README.

ae7636a

Add support for secrets.

44a1fd4

Add script for updating.

7c63dfa

Update README.md

a1b5920

Connie Yau (conniey) and others added 17 commits October 19, 2026 13:18

Update to publish metadata during release build.

0ad9225

Add variables for ingestion

28ac027

Update README.

174f461

Update exporter tool into a job file.

d002db6

Only create paths/directories when successful

7fe90e9

Accept whiteline characters within JSON string.

5cc66f6

Add query support for multiple servers.

718f298

Update binary search to include file extension

4b5320f

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Add support for Template.Mcp.Server as IsDryRun

ee77aa5

Update comments.

1c5a0a3

Do not fail the release stage.

f26b603

Throw when more than one server or runtime is found.

0732767

Remove overlapping.

0df6a0d

Make child directory if it does not exist.

ecc5590

Add Kusto to networkpolicy isolation

6d194b4

Make binary executable on linux.

b31eee7

Add Kusto to common.yml

c28196f

Connie Yau (conniey) force-pushed the conniey/ci-tool-metadata branch from 94e1f37 to c28196f Compare October 19, 2026 20:19

Conversation

Connie Yau (conniey) commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

GitHub issue number?

Pre-merge Checklist

Uh oh!

azure-pipelines Bot commented Oct 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Connie Yau (conniey) Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Alan Zimmer (alzimmermsft) Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Connie Yau (conniey) Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Alan Zimmer (alzimmermsft) Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Connie Yau (conniey) Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Connie Yau (conniey) commented Oct 6, 2026 •

edited

Loading