Test collection is 10x slower on 9.1.1 compared to 9.0.3

Hello!

After upgrading to 9.1.1 from 9.0.3 (and no other code changes, only one config change), running pytest --collect-only is around 10x slower.

On 9.0.3: 30406/30495 tests collected (89 deselected) in 30.22s

On 9.1.1: 30406/30495 tests collected (89 deselected) in 303.61s (0:05:03)

The only thing that changed between the two runs was adding this to filterwarnings (as we have a lot of class based fixtures)

ignore:.*Class-scoped fixture defined as instance method.*:pytest.PytestRemovedIn10Warning

I did some profiling with pyinstrument, it seems that most of the time is now spent in is_visibility_more_specific from FixtureManager.parsefactories

The codebase has a lot of class-based tests, but I can't share the code unfortunately.

I do appreciate there is not a lot to go on here, please lmk what other info I could provide to make the debugging easier!

Hmm sounds painful!

Do you think you can try to somehow see which part of is_visibility_more_specific is causing the slowdown, the part inside the if or the return outside the if?

Do you use a lot of overriding fixtures -- that is, fixtures which use the same name (e.g. one at module level, one at class level, etc)?

I reproduced the regression and instrumented is_visibility_more_specific function. The slowdown is on the final Node-based return. But the main problem is how many times it's called.

With 4000 unrelated sibling test classes defining the same fixture name:

pytest 9.0.3: 2.57s
pytest 9.1.1: 9.12s

final Node-based comparisons: 7,998,000
returned True: 0
returned False: 7,998,000

_register_fixture scans all previously registered fixtures with that name for every new definition. Since fixtures on sibling classes have no visibility relationship, every comparison fails and the whole list is scanned. This results N * (N - 1) /2 comparisons even though there are no actual override relationships in this reproducer.

The regression also reproduces without the warning filter and with plugin autoload disabled.

Reproducer and instrumentation

Create test_many_classes.py:

import pytest


def make_test_class(number):
    class TestClass:
        @pytest.fixture
        def value(self):
            return number

        def test_value(self, value):
            assert value == number

    TestClass.__name__ = f"TestClass{number}"
    return TestClass


for number in range(4000):
    globals()[f"TestClass{number}"] = make_test_class(number)

Create conftest.py:

import _pytest.fixtures


counts = {
    "if branch": 0,
    "final return": 0,
    "return true": 0,
    "return false": 0,
}
original = _pytest.fixtures.is_visibility_more_specific


def counted(candidate, other):
    result = original(candidate, other)

    if candidate.node is None or other.node is None:
        counts["if branch"] += 1
    else:
        counts["final return"] += 1
        if result:
            counts["return true"] += 1
        else:
            counts["return false"] += 1

    return result


_pytest.fixtures.is_visibility_more_specific = counted


def pytest_terminal_summary(terminalreporter):
    terminalreporter.write_line(f"visibility comparisons: {counts}")

I used Python 3.12 with two isolated environments:

python3.12 -m venv /tmp/pytest-903-env
/tmp/pytest-903-env/bin/python -m pip install pytest==9.0.3

python3.12 -m venv /tmp/pytest-911-env
/tmp/pytest-911-env/bin/python -m pip install pytest==9.1.1

Compare collection times without loading the diagnostic conftest.py:

PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 /usr/bin/time -f '%e seconds' \
    /tmp/pytest-903-env/bin/python -m pytest \
    --collect-only -qq --noconftest -c /dev/null test_many_classes.py

PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 /usr/bin/time -f '%e seconds' \
    /tmp/pytest-911-env/bin/python -m pytest \
    --collect-only -qq --noconftest -c /dev/null test_many_classes.py

Run the instrumentation on 9.1.1:

PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 \
    /tmp/pytest-911-env/bin/python -m pytest \
    --collect-only -qq -c /dev/null test_many_classes.py

The instrumentation output is:

visibility comparisons: {'if branch': 0, 'final return': 7998000, 'return true': 0, 'return false': 7998000}

Do you think you can try to somehow see which part of is_visibility_more_specific is causing the slowdown, the part inside the if or the return outside the if?

Sadly pyinstrument doesn't go that deep!

Do you use a lot of overriding fixtures -- that is, fixtures which use the same name (e.g. one at module level, one at class level, etc)?

In the codebase in general there are not many conftest.py files, so the global context is not generally very full but I think in some cases there will be some of these overrides, but not "many".

The vast majority of tests are in classes, which mostly inherit from one base class which does define a quite a few fixtures, which are then used in a lot of the final classes. In some cases there are a bunch of fixtures as well which get overridden from the base class in the final class

@RonnyPfannschmidt

Profiling notes. This analysis and the numbers below were produced by Claude Opus 5 via Claude Code, driven by me (@RonnyPfannschmidt). I have reviewed them; the reproducer and the timings are reproducible with the snippets given here.

@Amine-LG's diagnosis is correct, and there is a second, smaller regression from the same release hiding behind it.

Reproducer

test_many_classes.py, N sibling classes each defining the same fixture name (the shape of mikicz's suite: one base class with fixtures, inherited by thousands of test classes — parsefactories registers a fixturedef per class):

import os
import pytest


def make_test_class(number):
    class TestClass:
        @pytest.fixture
        def value(self):
            return number

        def test_value(self, value):
            assert value == number

    TestClass.__name__ = f"TestClass{number}"
    return TestClass


for number in range(int(os.environ["NCLASSES"])):
    globals()[f"TestClass{number}"] = make_test_class(number)

$ NCLASSES=4000 python -m pytest --collect-only -qq --noconftest -c /dev/null test_many_classes.py

Cause 1 — `_register_fixture` scan (the 10x)

_register_fixture scans the whole faclist for the name on every registration to find the insertion point (added in 7186cd4 / #14513):

for i, existing in enumerate(faclist):
    if is_visibility_more_specific(existing, fixture_def):
        faclist.insert(i, fixture_def)
        break
else:
    faclist.append(fixture_def)

N=4000 → 7,998,000 calls, all returning False, each walking iter_parents(). cProfile on main:

ncalls   tottime  cumtime  function
7998000    5.857   10.447  fixtures.py:139(is_visibility_more_specific)
80232000   4.611    4.611  nodes.py:286(iter_parents)
   4020    0.895   11.490  fixtures.py:2048(_register_fixture)

11.5s of a 16.2s profiled run.

Fix: an existing fixturedef can only be more specific than the new one if its node is strictly deeper in the collection tree. Track the deepest registration per fixture name and skip the scan when nothing registered so far can be deeper. Fixturedefs with a legacy string baseid have no node, so a name that has seen one falls back to the full scan.

Collection is top-down, so in practice the scan almost never fires — it only exists for plugins that register fixtures out of tree order (which is exactly what test_register_fixture_ordered_by_visibility covers, and that test still passes: registering node=item before node=session makes max_depth > depth and the scan runs).

Cause 2 — `Node.hash` in `_matchfactories`

With cause 1 fixed, N=4000 is 4.62s profiled vs. 9.0.3's 2.13s. The remainder:

ncalls   tottime  cumtime  function
   8000    1.080    2.038  fixtures.py:2392(_matchfactories)
8162022    0.674    0.969  nodes.py:277(__hash__)

_matchfactories is O(number of fixturedefs with that name) per lookup in both 9.0.3 and 9.1 — that quadratic is not new. What is new is node-based matching:

parent_nodes = set(node.iter_parents())
...
if fixturedef.node in parent_nodes:

Node.__hash__ is a Python-level function (hash(self._nodeid)), so the inner loop went from "one attribute + one str set lookup" (9.0.3 matched fixturedef.baseid in parentnodeids) to a Python call per fixturedef per lookup. Same 8M iterations, 0.363s in 9.0.3 vs 1.080s on main.

Fix: precompute a _match_key on FixtureDef — id(node) for node-based defs (nodes compare by identity, and the fixturedef holds the node alive so the id cannot be reused), the baseid string for legacy ones. int and str never compare equal, so one set holds both and the loop is a single lookup again.

Numbers

Wall clock, --collect-only, CPython 3.10, not under a profiler:

N classes	9.0.3	main (9.1.x)	patched
1000	0.38s	0.98s	0.56s
2000	0.67s	2.22s	0.94s
4000	1.53s	7.41s	1.83s

Not fixed here

_matchfactories remains O(defs-with-that-name) per lookup, i.e. quadratic in N, in 9.0.3 and after this patch alike. Killing it properly means indexing _arg2fixturedefs[name] by node and iterating the (short) parent chain instead of the (long) fixturedef list — which would also subsume the _register_fixture ordering, since the visible set for a given node is a chain and therefore totally ordered. That is a bigger change and is left out of the cheap regression fix.

@mikicz — the patch is on a branch; a PR will follow. Trying it against your suite would be useful: does it get you back to ~30s, or does something else in your layout still bite?

I installed your latest patch and sadly it hasn't improved things (I did verify I did in fact install the correct patch by going to source code in my venv)

Our main with 9.0.3: 30572/30661 tests collected (89 deselected) in 48.52s
Our main with the patch: 30572/30661 tests collected (89 deselected) in 308.60s (0:05:08)

I ran the profiling on a much smaller sample using pyinstrument (6594 tests, only about 3x slower on the branch compared to 9.0.3), attaching the pyinstrument output:

  _     ._   __/__   _ _  _  _ _/_   Recorded: 17:37:40  Samples:  19316
 /_//_/// /_\ / //_// / //_'/ //     Duration: 40.182    CPU time: 41.727
/   _/                      v5.1.2

Program: pytest --collect-only <folder>

39.987 <module>  pytest/__main__.py:1
└─ 39.987 _console_main  _pytest/config/__init__.py:254
   └─ 39.987 _main  _pytest/config/__init__.py:212
      ├─ 35.973 HookCaller.__call__  pluggy/_hooks.py:497
      │  └─ 35.973 PytestPluginManager._hookexec  pluggy/_manager.py:111
      │     └─ 35.973 pytest_cmdline_main  _pytest/main.py:376
      │        └─ 35.973 wrap_session  _pytest/main.py:317
      │           ├─ 35.428 _main  _pytest/main.py:380
      │           │  └─ 35.428 HookCaller.__call__  pluggy/_hooks.py:497
      │           │     └─ 35.428 PytestPluginManager._hookexec  pluggy/_manager.py:111
      │           │        └─ 35.426 pytest_collection  _pytest/main.py:393
      │           │           └─ 35.426 Session.perform_collect  _pytest/main.py:789
      │           │              └─ 35.119 Session.genitems  _pytest/main.py:1047
      │           │                 └─ 35.110 Session.genitems  _pytest/main.py:1047
      │           │                    └─ 35.062 Session.genitems  _pytest/main.py:1047
      │           │                       └─ 34.964 Session.genitems  _pytest/main.py:1047
      │           │                          ├─ 33.825 Session.genitems  _pytest/main.py:1047
      │           │                          │  ├─ 21.979 Session.genitems  _pytest/main.py:1047
      │           │                          │  │  ├─ 19.526 Session._collect_one_node  _pytest/main.py:886
      │           │                          │  │  │  └─ 19.526 collect_one_node  _pytest/runner.py:586
      │           │                          │  │  │     └─ 19.524 HookCaller.__call__  pluggy/_hooks.py:497
      │           │                          │  │  │        └─ 19.520 PytestPluginManager._hookexec  pluggy/_manager.py:111
      │           │                          │  │  │           └─ 19.500 pytest_make_collect_report  _pytest/runner.py:388
      │           │                          │  │  │              └─ 19.500 CallInfo.from_call  _pytest/runner.py:340
      │           │                          │  │  │                 └─ 19.496 collect  _pytest/runner.py:389
      │           │                          │  │  │                    └─ 19.401 Class.collect  _pytest/python.py:777
      │           │                          │  │  │                       ├─ 15.389 FixtureManager.parsefactories  _pytest/fixtures.py:2288
      │           │                          │  │  │                       │  └─ 14.811 FixtureManager._register_fixture  _pytest/fixtures.py:2071
      │           │                          │  │  │                       │     ├─ 11.860 is_visibility_more_specific  _pytest/fixtures.py:139
      │           │                          │  │  │                       │     │  ├─ 6.646 [self]  _pytest/fixtures.py
      │           │                          │  │  │                       │     │  └─ 5.214 Class.iter_parents  _pytest/nodes.py:286
      │           │                          │  │  │                       │     ├─ 1.777 [self]  _pytest/fixtures.py
      │           │                          │  │  │                       │     └─ 1.134 FixtureDef.__init__  _pytest/fixtures.py:1128
      │           │                          │  │  │                       │        └─ 1.016 getfuncargnames  _pytest/compat.py:108
      │           │                          │  │  │                       │           └─ 0.878 signature  _pytest/compat.py:68
      │           │                          │  │  │                       │              └─ 0.872 signature  inspect.py:3342
      │           │                          │  │  │                       │                    [4 frames hidden]  inspect
      │           │                          │  │  │                       └─ 4.010 Class.collect  _pytest/python.py:420
      │           │                          │  │  │                          └─ 3.968 HookCaller.__call__  pluggy/_hooks.py:497
      │           │                          │  │  │                             └─ 3.926 PytestPluginManager._hookexec  pluggy/_manager.py:111
      │           │                          │  │  │                                └─ 3.690 pytest_pycollect_makeitem  _pytest/python.py:239
      │           │                          │  │  │                                   └─ 3.598 Class._genfunctions  _pytest/python.py:471
      │           │                          │  │  │                                      ├─ 2.400 FunctionDefinition.from_parent  _pytest/python.py:1734
      │           │                          │  │  │                                      │  └─ 2.396 FunctionDefinition.from_parent  _pytest/nodes.py:209
      │           │                          │  │  │                                      │     └─ 2.390 FunctionDefinition._create  _pytest/nodes.py:99
      │           │                          │  │  │                                      │        └─ 2.378 FunctionDefinition.__init__  _pytest/python.py:1682
      │           │                          │  │  │                                      │           └─ 2.277 FixtureManager.getfixtureinfo  _pytest/fixtures.py:1836
      │           │                          │  │  │                                      │              └─ 2.147 FixtureManager.getfixtureclosure  _pytest/fixtures.py:1975
      │           │                          │  │  │                                      │                 └─ 1.919 traverse_fixture_closure  _pytest/fixtures.py:412
      │           │                          │  │  │                                      │                    └─ 1.907 process_argname  _pytest/fixtures.py:429
      │           │                          │  │  │                                      │                       ├─ 1.009 process_argname  _pytest/fixtures.py:429
      │           │                          │  │  │                                      │                       │  ├─ 0.521 getfixturedefs  _pytest/fixtures.py:1990
      │           │                          │  │  │                                      │                       │  │  └─ 0.515 FixtureManager.getfixturedefs  _pytest/fixtures.py:2381
      │           │                          │  │  │                                      │                       │  │     └─ 0.500 FixtureManager._matchfactories  _pytest/fixtures.py:2401
      │           │                          │  │  │                                      │                       │  └─ 0.443 process_argname  _pytest/fixtures.py:429
      │           │                          │  │  │                                      │                       └─ 0.811 getfixturedefs  _pytest/fixtures.py:1990
      │           │                          │  │  │                                      │                          └─ 0.781 FixtureManager.getfixturedefs  _pytest/fixtures.py:2381
      │           │                          │  │  │                                      │                             └─ 0.745 FixtureManager._matchfactories  _pytest/fixtures.py:2401
      │           │                          │  │  │                                      └─ 1.134 HookCaller.call_extra  pluggy/_hooks.py:541
      │           │                          │  │  │                                            [6 frames hidden]  pluggy, pytest_lazy_fixtures
      │           │                          │  │  │                                               0.407 _getfixtureclosure  pytest_lazy_fixtures/utils.py:19
      │           │                          │  │  │                                               └─ 0.407 FixtureManager.getfixtureclosure  _pytest/fixtures.py:1975
      │           │                          │  │  └─ 2.425 Session.genitems  _pytest/main.py:1047
      │           │                          │  │     └─ 2.335 Session._collect_one_node  _pytest/main.py:886
      │           │                          │  │        └─ 2.333 collect_one_node  _pytest/runner.py:586
      │           │                          │  │           └─ 2.333 HookCaller.__call__  pluggy/_hooks.py:497
      │           │                          │  │              └─ 2.333 PytestPluginManager._hookexec  pluggy/_manager.py:111
      │           │                          │  │                 └─ 2.333 pytest_make_collect_report  _pytest/runner.py:388
      │           │                          │  │                    └─ 2.331 CallInfo.from_call  _pytest/runner.py:340
      │           │                          │  │                       └─ 2.331 collect  _pytest/runner.py:389
      │           │                          │  │                          └─ 2.325 Class.collect  _pytest/python.py:777
      │           │                          │  │                             ├─ 1.860 FixtureManager.parsefactories  _pytest/fixtures.py:2288
      │           │                          │  │                             │  └─ 1.777 FixtureManager._register_fixture  _pytest/fixtures.py:2071
      │           │                          │  │                             │     └─ 1.429 is_visibility_more_specific  _pytest/fixtures.py:139
      │           │                          │  │                             │        ├─ 0.814 [self]  _pytest/fixtures.py
      │           │                          │  │                             │        └─ 0.615 Class.iter_parents  _pytest/nodes.py:286
      │           │                          │  │                             └─ 0.464 Class.collect  _pytest/python.py:420
      │           │                          │  │                                └─ 0.458 HookCaller.__call__  pluggy/_hooks.py:497
      │           │                          │  │                                   └─ 0.452 PytestPluginManager._hookexec  pluggy/_manager.py:111
      │           │                          │  │                                      └─ 0.428 pytest_pycollect_makeitem  _pytest/python.py:239
      │           │                          │  │                                         └─ 0.414 Class._genfunctions  _pytest/python.py:471
      │           │                          │  └─ 11.833 Session._collect_one_node  _pytest/main.py:886
      │           │                          │     └─ 11.831 collect_one_node  _pytest/runner.py:586
      │           │                          │        └─ 11.825 HookCaller.__call__  pluggy/_hooks.py:497
      │           │                          │           └─ 11.823 PytestPluginManager._hookexec  pluggy/_manager.py:111
      │           │                          │              └─ 11.799 pytest_make_collect_report  _pytest/runner.py:388
      │           │                          │                 └─ 11.795 CallInfo.from_call  _pytest/runner.py:340
      │           │                          │                    └─ 11.791 collect  _pytest/runner.py:389
      │           │                          │                       └─ 11.376 Class.collect  _pytest/python.py:777
      │           │                          │                          ├─ 9.338 FixtureManager.parsefactories  _pytest/fixtures.py:2288
      │           │                          │                          │  └─ 8.880 FixtureManager._register_fixture  _pytest/fixtures.py:2071
      │           │                          │                          │     ├─ 7.129 is_visibility_more_specific  _pytest/fixtures.py:139
      │           │                          │                          │     │  ├─ 4.035 [self]  _pytest/fixtures.py
      │           │                          │                          │     │  └─ 3.093 Class.iter_parents  _pytest/nodes.py:286
      │           │                          │                          │     ├─ 1.017 [self]  _pytest/fixtures.py
      │           │                          │                          │     └─ 0.712 FixtureDef.__init__  _pytest/fixtures.py:1128
      │           │                          │                          │        └─ 0.658 getfuncargnames  _pytest/compat.py:108
      │           │                          │                          │           └─ 0.586 signature  _pytest/compat.py:68
      │           │                          │                          │              └─ 0.582 signature  inspect.py:3342
      │           │                          │                          │                    [3 frames hidden]  inspect
      │           │                          │                          └─ 2.034 Class.collect  _pytest/python.py:420
      │           │                          │                             └─ 2.012 HookCaller.__call__  pluggy/_hooks.py:497
      │           │                          │                                └─ 1.978 PytestPluginManager._hookexec  pluggy/_manager.py:111
      │           │                          │                                   └─ 1.806 pytest_pycollect_makeitem  _pytest/python.py:239
      │           │                          │                                      └─ 1.733 Class._genfunctions  _pytest/python.py:471
      │           │                          │                                         ├─ 1.069 FunctionDefinition.from_parent  _pytest/python.py:1734
      │           │                          │                                         │  └─ 1.067 FunctionDefinition.from_parent  _pytest/nodes.py:209
      │           │                          │                                         │     └─ 1.063 FunctionDefinition._create  _pytest/nodes.py:99
      │           │                          │                                         │        └─ 1.051 FunctionDefinition.__init__  _pytest/python.py:1682
      │           │                          │                                         │           └─ 1.013 FixtureManager.getfixtureinfo  _pytest/fixtures.py:1836
      │           │                          │                                         │              └─ 0.951 FixtureManager.getfixtureclosure  _pytest/fixtures.py:1975
      │           │                          │                                         │                 └─ 0.813 traverse_fixture_closure  _pytest/fixtures.py:412
      │           │                          │                                         │                    └─ 0.799 process_argname  _pytest/fixtures.py:429
      │           │                          │                                         └─ 0.599 HookCaller.call_extra  pluggy/_hooks.py:541
      │           │                          │                                            └─ 0.595 PytestPluginManager._hookexec  pluggy/_manager.py:111
      │           │                          └─ 1.133 Session._collect_one_node  _pytest/main.py:886
      │           │                             └─ 1.131 collect_one_node  _pytest/runner.py:586
      │           │                                └─ 1.125 HookCaller.__call__  pluggy/_hooks.py:497
      │           │                                   └─ 1.125 PytestPluginManager._hookexec  pluggy/_manager.py:111
      │           │                                      └─ 1.113 pytest_make_collect_report  _pytest/runner.py:388
      │           │                                         └─ 1.111 CallInfo.from_call  _pytest/runner.py:340
      │           │                                            └─ 1.109 collect  _pytest/runner.py:389
      │           │                                               └─ 1.033 Module.collect  _pytest/python.py:583
      │           │                                                  └─ 0.939 Module._register_setup_module_fixture  _pytest/python.py:589
      │           │                                                     └─ 0.939 Module.obj  _pytest/python.py:305
      │           │                                                        └─ 0.939 Module._getobj  _pytest/python.py:580
      │           │                                                           └─ 0.939 importtestmodule  _pytest/python.py:521
      │           │                                                              └─ 0.935 import_path  _pytest/pathlib.py:546
      │           │                                                                 └─ 0.895 import_module  importlib/__init__.py:71
      │           │                                                                    └─ 0.726 AssertionRewritingHook.exec_module  _pytest/assertion/rewrite.py:147
      │           └─ 0.476 Config._ensure_unconfigure  _pytest/config/__init__.py:1279
      └─ 4.014 _prepareconfig  _pytest/config/__init__.py:395
         └─ 3.981 HookCaller.__call__  pluggy/_hooks.py:497
            └─ 3.981 PytestPluginManager._hookexec  pluggy/_manager.py:111
               └─ 3.981 Config.pytest_cmdline_parse  _pytest/config/__init__.py:1300
                  └─ 3.981 Config.parse  _pytest/config/__init__.py:1584
                     └─ 3.237 HookCaller.__call__  pluggy/_hooks.py:497

If you'd like, I can do more profiling on some more examples, possibly using other methods compared to pyinstrument?
Unfortunately I can only probably do for the next hour or and then I am not back until Tuesday

i may not be avaliable much - it would be great if you could create a reproducer script i could integrate in my profiling later as my initial testing makes somethings faster

the stacked pr on top also helps some parts but breaks plugins

I shall try next Tuesday!

@mikicz

@mikicz thanks — that profile is exactly what was needed, and it says my patch is wrong. This analysis was again produced by Claude Opus 5 via Claude Code, driven and reviewed by me.

You did install the right thing

Every line number in your pyinstrument output matches the #14949 branch rather than main: _register_fixture at 2071 (main: 2048), parsefactories 2288 (2220), getfixturedefs 2381 (2341), _matchfactories 2401 (2361), getfixtureinfo 1836, getfixtureclosure 1975. So no stale install — the patch simply does not help your suite.

Why it does not help

The skip condition in #14949 is "no already-registered fixturedef can be deeper in the collection tree than this one", tracked as a running maximum depth per fixture name. That holds only while registrations arrive in non-decreasing depth.

My reproducer was 4000 sibling classes in a single module — all at identical depth — so the guard skipped essentially every scan and the fix looked complete. A real suite has test classes at many different directory depths, and pytest collects depth-first, so the depth of successive registrations goes up and down continuously. Once any deep class has registered a name, the running maximum stays high, and every subsequent shallower registration of that name scans the whole list again.

Instrumented on a suite with classes at depths 1–4: 2920 of 4018 registrations still scan, 5,959,340 is_visibility_more_specific() calls. The guard skips 27% of registrations and buys almost nothing.

Numbers

Same 4000 classes each inheriting the same fixture from a shared base class; only the shape of the directory tree differs. --collect-only, CPython 3.10, wall clock.

tree shape	9.0.3	main	#14949	#14950 (node index)
all classes at one depth	1.53s	7.41s	1.83s	1.52s
depths 1–4, interleaved	1.72s	8.72s	6.81s	1.62s
depths 1–4, monotonically increasing	3.03s	—	3.23s	—

The third row is the control: take the same classes at the same depths and order them so depth never decreases, and #14949 works again. So the failure is specifically depth non-monotonicity, not depth.

Where that leaves it

#14949 has two halves. The _matchfactories half — precomputing a match key so the per-fixturedef loop stops hashing Nodes — is a real fix and stands. The registration-scan guard is close to worthless on real trees and should come out; optimising for a shape my reproducer had and real suites do not is exactly the trap.

The approach that actually fixes this is #14950: index the fixturedefs by the node they are visible from and walk the requesting node's (short) parent chain, instead of scanning every definition of that name. That removes the quadratic outright rather than trying to skip it, and it also subsumes the ordering machinery #14513 added. Its problem is different — it assumes _register_fixture() is the only way fixturedefs enter the manager, and it is not: pytest-bdd, pytest-bdd-ng, pytest-psqlgraph, pytest-codspeed, pytest-keyring, pytest-fixture-forms and pykiso all write into FixtureManager._arg2fixturedefs directly. That is being worked on.

@mikicz no need to spend more of your time profiling — the reproducer above reproduces your numbers, so there is something concrete to iterate against now. If you have a spare moment when you are back Tuesday, trying #14950 against your suite would be the useful data point.

fixtures: keep the visibility index in sync with external mutation

@mikicz

@mikicz in case you are still around - take a look at the followup pr to see if that more reliably solves the issue

the new mechanism is a bit more fragile and needs a few extra rounds of due diligence

Hi, you just caught me before I needed to leave the computer, I ran against exp-14942-index-fixturedefs-by-node from #14950; and the collection is back to it's normal speed (a tiny bit faster even, but that could be just my computer not running something else - the order of magnitude slowdown is no longer present, either way)

30572/30661 tests collected (89 deselected) in 43.69s

I can run more tests on tuesday if the code changes by then

fixtures: avoid hashing Nodes when matching fixturedefs

fixtures: index fixturedefs by visibility instead of scanning

@RonnyPfannschmidt

Hmm the many classes scenario is not one I thought about. It does indeed register a bunch of "overrides", though they are all mutually exclusive. That suggests we want a better data structure for the global fixture list, that scales with the depth of the tree rather than the absolute number of fixtures registered with the same name. The problem ~kinda existed before, in that the fixturedefs list would be very long, and _matchfactories would then need to filter through them. The is_visibility_more_specific change made it a lot worse and (hmm) visible :)

@RonnyPfannschmidt regarding the slow Node hash, as part of #14758 we are going to switch to identity-based hash for Nodes: #14758 (comment). Sounds like it would make things simpler for you.

Regarding pytest-bdd, it does use register_fixture these days (well actually still using the private _register_fixture but that's the same). But maybe you are referring to the inject_fixturedefs_for_step code it is using:

https://github.com/pytest-dev/pytest-bdd/blob/c17dbb2de74bebd4d16cfc865773286460e06e34/src/pytest_bdd/scenario.py#L133-L165

Interestingly it looks like they're hand-rolling some version of visibility-ordered insertion.

I'm not sure it would be completely sane to keep the existing pytest-bdd code working, if we choose the replace the data structure. We always to not break plugins but don't guarantee it for internal stuff. In other words, in this I think it would be better if pytest-bdd add compat for new pytest (which we can help with) then for pytest to carry very complex compat.

If the compat is bearable we can have it until the next major im still trying to find a good balance

WIP: fixtures: use a better data structure for storing FixtureDefs

fixtures: use a better data structure for storing FixtureDefs

I went ahead and made my own proposed fix in #14984. It's simpler than #14950, mainly due to avoiding complex compat, which I think is too much for this.

fixtures: use a better data structure for storing FixtureDefs

Navigation Menu

Repository navigation

Test collection is 10x slower on 9.1.1 compared to 9.0.3 #14942

Reproducer

Cause 1 — `_register_fixture` scan (the 10x)

Cause 2 — `Node.hash` in `_matchfactories`

Numbers

Not fixed here

You did install the right thing

Why it does not help

Numbers

Where that leaves it

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Test collection is 10x slower on 9.1.1 compared to 9.0.3 #14942

Description

Activity

bluetech commented on Aug 26, 2026

Amine-LG commented on Aug 26, 2026

mikicz commented on Aug 28, 2026

RonnyPfannschmidt commented on Aug 28, 2026

Reproducer

Cause 1 — _register_fixture scan (the 10x)

Cause 2 — Node.__hash__ in _matchfactories

Numbers

Not fixed here

mikicz commented on Aug 28, 2026

RonnyPfannschmidt commented on Aug 28, 2026

mikicz commented on Aug 28, 2026

RonnyPfannschmidt commented on Aug 28, 2026

You did install the right thing

Why it does not help

Numbers

Where that leaves it

RonnyPfannschmidt commented on Aug 28, 2026

mikicz commented on Aug 28, 2026

bluetech commented on Sep 5, 2026

RonnyPfannschmidt commented on Sep 5, 2026

bluetech commented on Sep 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Cause 1 — `_register_fixture` scan (the 10x)

Cause 2 — `Node.hash` in `_matchfactories`