Skip to content

perf: use osv batch api for dep analysis#665

Merged
danielroe merged 3 commits intomainfrom
perf/osv-batch
Feb 1, 2026
Merged

perf: use osv batch api for dep analysis#665
danielroe merged 3 commits intomainfrom
perf/osv-batch

Conversation

@danielroe
Copy link
Copy Markdown
Member

@danielroe danielroe commented Feb 1, 2026

this updates osv analysis to use their batch api which significantly speeds up resolution for packages with large dep trees

@vercel
Copy link
Copy Markdown

vercel Bot commented Feb 1, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
npmx.dev Ready Ready Preview, Comment Feb 1, 2026 7:31pm
2 Skipped Deployments
Project Deployment Actions Updated (UTC)
docs.npmx.dev Ignored Ignored Preview Feb 1, 2026 7:31pm
npmx-lunaria Ignored Ignored Feb 1, 2026 7:31pm

Request Review

@danielroe
Copy link
Copy Markdown
Member Author

@serhalp are you investigating?

Copy link
Copy Markdown
Member

@serhalp serhalp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 LGTM, great find!

N=1 not very scientific but loading the netlify-cli page in prod vs. this branch took 130s vs. 15s 🎉

Comment thread server/utils/dependency-analysis.ts Outdated
Comment on lines +204 to +205
const detailResults = await Promise.all(
vulnerablePackageInfos.map(pkg => queryOsvDetails(pkg)),
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In egregious cases, this might fire off too many requests in parallel. Perhaps we could use a util here that limits concurrency to something like 25?

}

return { status: 'ok', data: { name, version, depth, path, vulnerabilities, counts } }
return {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like the batch endpoint is also paginated.

It seems unlikely we'd ever have incomplete results in the first page:

  • An individual query within the queryset returns more than 1,000 vulnerabilities
  • The entire queryset returns more than 3,000 vulnerabilities total

but maybe just check for a non-nil next_page_token and log a warning/error for future visibility?

Copy link
Copy Markdown
Contributor

@garthdw garthdw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feels faster on projen and @babel/core packages

@danielroe danielroe added this pull request to the merge queue Feb 1, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Feb 1, 2026
@danielroe danielroe added this pull request to the merge queue Feb 1, 2026
Merged via the queue into main with commit 0df32a3 Feb 1, 2026
16 checks passed
@danielroe danielroe deleted the perf/osv-batch branch February 1, 2026 19:52
taskylizard pushed a commit to taskylizard/npmx.dev that referenced this pull request Feb 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants