Repository navigation
Conversation
michaelkedar
left a comment
michaelkedar
left a comment
Member
There was a problem hiding this comment.
Sorry for the delay in reviewing this
michaelkedar
left a comment
michaelkedar
left a comment
Member
There was a problem hiding this comment.
LGTM, but one more thing
| } | ||
| if sys, ok := provider.Get(ecosystem); ok { | ||
| name = eco.NormalizePackageName(sys, name) | ||
| name = eco.NormalizeMatchName(sys, name) |
Member
There was a problem hiding this comment.
Just to make rollover easier on our end, can you gate this change behind an environment variable or something so we can enable this API change after we've migrated all the data
Author
There was a problem hiding this comment.
Done, the API now only uses the match name when OSV_NORMALIZE_MATCH_NAMES is set to true, and it's off by default. Until then, queries are normalized the same way as before. The worker still writes the extra rows, so once NuGet and crates.io have been reprocessed, setting the variable turns it on. I also rebased onto master, since the API setup changed there.
NuGet package IDs are case-insensitive, and crates.io treats crate names case-insensitively with '-' and '_' as equivalent. Add PackageNameMatcher and MatchPackageName for names that should match without being rewritten: NuGet lowercases the ID, and crates.io also maps '_' to '-'. Only ASCII letters are folded, so lookalike IDs that the registry keeps distinct stay distinct, and invalid UTF-8 is kept as is. computeAffectedVersions now also writes a copy of a package's AffectedVersions rows under its match name when that differs, marked match_only. Queries skip match_only rows for now, so this commit does not change any query results. Published records are unchanged, since the namenormalize enricher still only applies PackageNameNormalizer. Once NuGet and crates.io records have been reprocessed, queries can use the match name. Refs google#6039
Fold the package name with MatchPackageName when a query gives an ecosystem, and let such queries use the match_only rows, so that a NuGet ID matches in any case and a crate name matches in any case with either '-' or '_'. For example, Microsoft.OpenApi.Kiota@1.31.0 currently misses GHSA-7f3j-j7jj-r3vr, which spells the package two ways, and lol_html misses GHSA-c3x7-354f-4p2x, which spells it lol-html. Queries without an ecosystem still only match the package name. NuGet and crates.io records need to be reprocessed with the previous commit before this is deployed, or ecosystem queries miss records that have no match_only rows yet. Fixes google#6039
Rename the interface to MatchNameNormalizer, and its method and the package-level helper to NormalizeMatchName, since they normalize a name rather than match it. Move unwrap from tuxcare.go to ecosystem.go, now that the name normalizers use it. unwrap was the only part of tuxcare.go that dhi.go does not have, so without it dupl reports the two files as duplicates. Mark both with nolint:dupl, since each delegates to an ecosystem resolved on demand.
The flag only kept queries without an ecosystem, which do not fold the name, from matching the rows under the match name. Such queries already match any ecosystem's package with the same name, so they can match those rows too, and the AffectedVersions models stay unchanged. Build the rows under the match name in one place by copying the package's rows once they are computed.
MatchPackages now only uses NormalizeMatchName when the store is created with NormalizeMatchNames, which the API and its devserver set from the OSV_NORMALIZE_MATCH_NAMES environment variable. It is off by default, so the API keeps normalizing queries as before until NuGet and crates.io records have been reprocessed.
kanokwut
force-pushed
the
match-nuget-crates-names
branch
from
October 22, 2026 06:11
1e6e047 to
8d71063
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
NuGet package IDs are case-insensitive, and crates.io treats crate names case-insensitively with
-and_as equivalent, but the query API matches these names exactly. Valid spellings therefore miss advisories today. For example,Microsoft.OpenApi.Kiota@1.31.0does not match CVE-2026-59862 because GHSA-7f3j-j7jj-r3vr spells the package two ways, andlol_htmldoes not match GHSA-c3x7-354f-4p2x, which spells itlol-html.This makes NuGet and crates.io names match case-insensitively, as agreed in #6039, without changing any published record. npm and Packagist are unchanged.
Fixes #6039
Details
The change only affects matching. Reusing
PackageNameNormalizerwould also rewrite the published names, since thenamenormalizeenricher writes them back into the record. For NuGet that would rename 1,828 of 1,910 records and break exact-name consumers such as osv-scalibr's offline matcher, so this adds a separate matching path instead.MatchNameNormalizerandecosystem.NormalizeMatchNamereturn the name used to index and query a package. NuGet lowercases the ID, and a newcratesIOEcosystemalso maps_to-. Only ASCII letters are folded, so lookalike IDs with characters such as U+0130 or U+212A stay distinct, and invalid UTF-8 is kept as is.computeAffectedVersionsalso writes a copy of a package'sAffectedVersionsrows under its match name when that differs. The original rows are unchanged.MatchPackagesfolds the name when an ecosystem is given and the API runs withOSV_NORMALIZE_MATCH_NAMES=true. Queries without an ecosystem still match the name as written, and they can also hit the copied rows, the same way they already match any ecosystem's package with the same name.MatchNameNormalizer, and the interface doc explains why. Echo:NuGet and TuxCare are unchanged, since their wrappers do not forward it.Rollout
The query change is gated behind
OSV_NORMALIZE_MATCH_NAMESon the API, which is off by default:newtonsoft.json.OSV_NORMALIZE_MATCH_NAMES=trueon the API.If the variable is set before the reprocess finishes, ecosystem queries miss records that have no copied rows yet, even with the exact published spelling. That affects about 96% of NuGet records, including MAL ones. The docs change in
post-v1-query.mddescribes the behavior with the variable on.Other notes:
AffectedVersionsentities. The largest record grows from 297 to 591 rows, about 200 KB per write.getNextFixVersioncompares names exactly, so for the newly matched spellings it shows the vulnerability without a fix version. That is not a regression, since these queries returned nothing before.Microsoft.OpenApi.Kiota@1.31.0should return GHSA-7f3j-j7jj-r3vr, andNewtonsoft.Json@12.0.1should still return GHSA-5crp-9r3c-p9vr.Testing
NormalizeMatchName, the copied rows incomputeAffectedVersions, published names staying unchanged in thenamenormalizeenricher, andMatchPackageson the Datastore emulator with the variable on and off, covering case variants,-and_, versions, npm, and queries without an ecosystem.go test ./...ingo/passes with the Datastore emulator.go vetandgofmt -sreport no issues, andgolangci-lintv2.13.0 reports no new issues on thegomodule.This change was prepared with AI assistance.
Generated with Claude Code: https://claude.com/claude-code