Repository navigation
Measure setup benchmarks with CPU time after explicit warmup - #1356
Conversation
1393377 to
45e4b35
Compare
mrbean-bremen
left a comment
There was a problem hiding this comment.
Thanks for that! It is certainly a good idea to measure CPU time instead of elapsed time.
I had hoped that this would make the measurements more stable than before, but that seems not to be the case (based on three test runs). Still it is better this way.
Also, I would still like to have some (albeit imperfect) check for some limits to make sure that a change does not decrease the performance sufficiently (I had introduced the check precisely because that had happened and I didn't notice at the time).
Could you please re-introduce this assert bases on the current results?
mrbean-bremen
left a comment
There was a problem hiding this comment.
Ok, thanks! Ideally, we would have some performance graph over the builds (there are tools for this), but this would be overkill, I think.
Describe the changes
Measure warmed-up Patcher setup/cleanup with process CPU time while keeping a coarse regression guard, as requested in the maintainer review. Library behavior and the thread-race work in issue #1321 are unchanged.
Keep
TEST_PERFORMANCE=1 python -m pyfakefs.tests.performance_test. Each cache mode performs one warmup, followed by 100 complete construction/setup/cleanup cycles. Report meantime.process_time()milliseconds per cycle, with strict CPython limits of 2 ms cached / 50 ms uncached. PyPy still reports measurements without the CPython limits, matching the previous runtime exclusion.The limits use 34 existing CPython measurement pairs across Ubuntu/Windows and Python 3.10–3.15. Observed maxima were 1.250 ms cached and 30.952 ms uncached, leaving approximately 60% headroom. Sources: current-head attempt 2, attempt 3, and available benchmark steps from the older head's attempt 2. The older workflow failed on an unrelated Python 3.14 glob test; it is not presented as a successful current-head matrix. macOS skips this CI benchmark step. Gated attempts with no jobs are not measurements.
These are coarse absolute regression tripwires, not portable performance guarantees. The same OS/Python configuration varied up to 2.665× cached and 2.106× uncached across those observations. No stability improvement or speedup is claimed; slower hosts or future runner variance can still fail.
Seven deterministic controls cover CPU-clock selection/normalization, excluded warmup/included cleanup, fresh filesystems/restored patching, invalid iteration counts, strict regression boundaries and PyPy reporting. The new guard controls reproduce four failing subtests before the follow-up and pass afterward.
Validation on macOS arm64 / CPython 3.13.1:
TEST_REAL_FS=1 python -bb -m pyfakefs.tests.all_tests: 2,498 tests, OK (352 skipped, 2 expected failures).git diff --checkpass.AI assistance was used for implementation, test/evidence preparation and independent automated review. Automated review is not an owner human-review attestation.
Tasks