Skip to content

perf(requirements): reuse validated specifier tokens - #1495

Open
DicksonWu654 wants to merge 1 commit into
pypa:mainfrom
DicksonWu654:perf/requirement-specifiers
Open

DicksonWu654 wants to merge 1 commit into
pypa:mainfrom
DicksonWu654:perf/requirement-specifiers

Conversation

@DicksonWu654

Copy link
Copy Markdown

Requirement.__init__ builds SpecifierSet(parsed.specifier), which splits the string and runs each part through Specifier._regex again, although the requirement parser has just matched every part with the same specifier pattern. This adds a private SpecifierSet._from_validated() that splits the parser's string and sets the same fields that SpecifierSet.__init__ and Specifier.__init__ set: same spelling, order and duplicates, new objects for every requirement, empty caches. The public constructors don't change.

Strings that contain === still use the normal constructor. The arbitrary-equality token may contain commas, so splitting it can produce parts that must still be rejected with the same InvalidRequirement (for example demo===x,y).

Trade-off: _from_validated() sets the slots of both classes directly, so it has to be updated when __init__ gains a field. The new tests compare str, repr, hash, pickling, filter(), contains() and prereleases handling with SpecifierSet(text) for a range of specifier strings, and check that two requirements never share specifier objects.

Performance

CPU time per pass over the workload, this branch vs main (713d0d7). Paired AB/BA: two worker processes, one per tree, pinned to the same core and timed alternately; each sample is the median of 5 passes; 15 pairs; speedup is the median per-pair ratio. The host was shared, so CPU time is reported. The Airflow file is the 3.1.0 Python 3.12 constraints file; the 1,169 real requirements add the Requires-Dist of the latest apache-airflow, jupyterlab, pandas, poetry, requests, scipy, setuptools and torch.

Workload Python main (ms) branch (ms) speedup p10–p90 faster pairs
Airflow constraints (673) CPython 3.12 18.14 14.38 1.18× 1.09–1.45 15/15
Real requirements (1,169) CPython 3.12 44.43 37.39 1.23× 0.88–1.50 11/15
benchmarks/dist_sample.txt (500) CPython 3.12 21.08 18.29 1.10× 0.98–1.37 13/15
Airflow constraints (673) CPython 3.15 13.82 12.67 1.09× 0.65–1.25 11/15
Real requirements (1,169) CPython 3.15 33.34 29.76 1.11× 1.04–1.46 14/15
benchmarks/dist_sample.txt (500) CPython 3.15 18.60 17.74 1.09× 0.95–1.32 12/15
Airflow constraints (673) PyPy 3.11 3.73 2.87 1.21× 1.12–2.09 15/15
Real requirements (1,169) PyPy 3.11 11.97 8.30 1.37× 1.19–1.68 15/15
benchmarks/dist_sample.txt (500) PyPy 3.11 9.59 5.48 1.51× 1.11–2.26 15/15

It is independent of the marker parsing PR (#1494); with that one applied first, the same measurement gives Airflow 1.38× / 1.19× and real requirements 1.12× / 1.11× on CPython 3.12 / 3.15 (PyPy 1.22× / 1.33×, 12 pairs).

Testing

  • Full test suite passes on this branch on CPython 3.14 with 100% coverage, and together with the other two performance PRs on CPython 3.10, 3.11, 3.12, 3.13, 3.15 and PyPy 3.11. prek run --all-files (ruff, ruff format, mypy, typos, ...) passes.
  • A differential run against main (together with the other two open performance PRs) compared 82,478 cases on CPython 3.12 and PyPy 3.11, covering real requirements, generated and malformed input, exception types and messages, warnings, hashes, ordering, pickling, string subclasses and mutation of the resulting objects: no differences.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant