Parametrizing test class with class-scope

[mikicz]

Hello, I've got a question about class parametrisation. I'm looking for a pytest native way to parametrize a test class with a class-scoped fixture, which doesn't make fixture exist multiple times at once with each of the parameters.

The code examples are also stored in this repo with all imports: https://github.com/mikicz/symmetrical-engine

Setup

Let's say I've got some test where class-based fixture is used to make the test faster, something like this. The fixture some_fixture generates and inserts some object into the database, and it takes a while, let's say ten seconds. The whole class therefore takes 10 seconds to run, as the some_fixture fixture gets reused for the two tests.

class TestSomething:
    @pytest.fixture(scope="class", autouse=True)
    def some_fixture(self):
        time.sleep(10)  # let's say this takes ten seconds, e.g. creating some object in the database
        pass

    def test_1(self, some_fixture):
        pass

    def test_2(self, some_fixture):
        pass

Test output

======================== test session starts =========================
platform linux -- Python 3.11.3, pytest-7.3.1, pluggy-1.0.0 -- /home/miki/xelix/experiments/symmetrical-engine/venv/bin/python
cachedir: .pytest_cache
rootdir: /home/miki/xelix/experiments/symmetrical-engine
collected 2 items                                                    

test_default.py::TestSomething::test_1 PASSED                  [ 50%]
test_default.py::TestSomething::test_2 PASSED                  [100%]

========================= 2 passed in 10.00s =========================

Now, let's say the code I'm testing has changed and there's two variants and I want to test both using the same test, just with some parameter. The obvious to do this is to use pytest.mark.parametrize("variant", ["a", "b"]) on the class. The trouble now starts. By default, this adds a variant fixture to the class with function scope. This means, that I need to remove class scope from some_fixture, which makes the test twice as slow. But at least it will work fine in most cases.

@pytest.mark.parametrize("variant", ["a", "b"])
class TestSomething:
    @pytest.fixture(autouse=True)
    def some_fixture(self, variant):
        time.sleep(10)  # generate the object using the variant fixture
        pass

    def test_1(self, some_fixture):
        pass

    def test_2(self, some_fixture):
        pass

Test output

======================== test session starts =========================
platform linux -- Python 3.11.3, pytest-7.3.1, pluggy-1.0.0 -- /home/miki/xelix/experiments/symmetrical-engine/venv/bin/python
cachedir: .pytest_cache
rootdir: /home/miki/xelix/experiments/symmetrical-engine
collected 4 items                                                    

test_parametrize.py::TestSomething::test_1[a] PASSED           [ 25%]
test_parametrize.py::TestSomething::test_1[b] PASSED           [ 50%]
test_parametrize.py::TestSomething::test_2[a] PASSED           [ 75%]
test_parametrize.py::TestSomething::test_2[b] PASSED           [100%]

========================= 4 passed in 40.01s =========================

There's an option to add a scope="class" parameter to the pytest.mark.parametrize, which makes variant a class-scope fixture. This does however break the tests if the tests assume that there's only one object in the database created by some_fixture.

Desired behaviour

What I would like to do is to run the whole TestSomething with each of the variants, but not such that the some_fixture fixture being available in the scope at the same time with each of the variants, basically make it safe for the test to still assume that there's only one object in the database.

Attempted solutions

Base Class

One solution is to define a base class for the tests, and then subclass for each of the variants, something like this:

class BaseSomethingTest(abc.ABC):
    @property
    @abc.abstractmethod
    def variant(self):
        pass

    @pytest.fixture(scope="class", autouse=True)
    def some_fixture(self):
        time.sleep(10)  # generate the object using self.variant
        pass

    def test_1(self, some_fixture):
        pass

    def test_2(self, some_fixture):
        pass


class TestSomethingA(BaseSomethingTest):
    variant = "a"


class TestSomethingB(BaseSomethingTest):
    variant = "b"

Test output

======================== test session starts =========================
platform linux -- Python 3.11.3, pytest-7.3.1, pluggy-1.0.0 -- /home/miki/xelix/experiments/symmetrical-engine/venv/bin/python
cachedir: .pytest_cache
rootdir: /home/miki/xelix/experiments/symmetrical-engine
collected 4 items                                                    

test_subclass.py::TestSomethingA::test_1 PASSED                [ 25%]
test_subclass.py::TestSomethingA::test_2 PASSED                [ 50%]
test_subclass.py::TestSomethingB::test_1 PASSED                [ 75%]
test_subclass.py::TestSomethingB::test_2 PASSED                [100%]

========================= 4 passed in 20.01s =========================

This does work, but there's downsides:

• One needs to completely rewrite the tests to not use variant as a fixture, but to use the property
• It is quite verbose - for many parameters it requires a lot of new classes to be explicitly defined

Cursed Metaprogramming

I did develop a way to basically automate the subclass approach, but it's quite ugly metaprogramming:

def generate_fixture(someparam):
    @pytest.fixture(scope="class")
    def my_fixture(self):
        return someparam

    return my_fixture


def generate_tests_cls_parametrize(cls: type, parameter_names: list[str], values: list[list], ids: list[str]):
    base_cls_name = "Test" + cls.__name__.removeprefix("Base").removesuffix("Test")
    for test_values, test_id in zip(values, ids, strict=True):
        cls_name = base_cls_name + test_id
        globals()[cls_name] = type(
            cls_name,
            (cls,),
            {
                parameter_name: generate_fixture(value)
                for parameter_name, value in zip(parameter_names, test_values, strict=True)
            },
        )


class BaseSomethingTest:
    @pytest.fixture(scope="class", autouse=True)
    def some_fixture(self, variant):
        time.sleep(10)  # generate the object using the variant fixture
        pass

    def test_1(self, some_fixture):
        pass

    def test_2(self, some_fixture):
        pass


generate_tests_cls_parametrize(
    BaseSomethingTest,
    ["variant"],
    [["a"], ["b"]],
    ["A", "B"],
)

Test output

======================== test session starts =========================
platform linux -- Python 3.11.3, pytest-7.3.1, pluggy-1.0.0 -- /home/miki/xelix/experiments/symmetrical-engine/venv/bin/python
cachedir: .pytest_cache
rootdir: /home/miki/xelix/experiments/symmetrical-engine
collected 4 items                                                    

test_meta.py::TestSomethingA::test_1 PASSED                    [ 25%]
test_meta.py::TestSomethingA::test_2 PASSED                    [ 50%]
test_meta.py::TestSomethingB::test_1 PASSED                    [ 75%]
test_meta.py::TestSomethingB::test_2 PASSED                    [100%]

========================= 4 passed in 20.01s =========================

So this solves the two issues I have with the base class approach, but it's not the prettiest solution.

Conclusion

So the question I have is, is there a native pytest way to do what I did in the attempted solutions?

[portyu9]

Hi @mikicz

Yes. The native pytest solution is to make the parameter class-scoped and make the dependent fixture class-scoped as well:

import pytest


@pytest.mark.parametrize(
    "variant",
    ["a", "b"],
    scope="class",
)
class TestSomething:
    @pytest.fixture(scope="class", autouse=True)
    def some_fixture(self, variant):
        obj = create_object_in_database(variant)

        yield obj

        delete_object_from_database(obj)

    def test_1(self, some_fixture):
        pass

    def test_2(self, some_fixture):
        pass

Current pytest uses the parametrization scope to group tests by parameter instance, so the effective lifecycle is:

SETUP variant=a
test_1[a]
test_2[a]
TEARDOWN variant=a

SETUP variant=b
test_1[b]
test_2[b]
TEARDOWN variant=b

In other words, you do not need to generate subclasses yourself for this.

The important part is the teardown.

If your fixture only does:

@pytest.fixture(scope="class")
def some_fixture(variant):
    create_object_in_database(variant)

then changing fixture instances does not magically remove the previous database object. The external side effect remains, which can make it look as though both parametrizations are "active".

Use a yield fixture (or request.addfinalizer) to clean up the first parameter's resource before pytest moves to the second one:

@pytest.fixture(scope="class")
def some_fixture(variant):
    obj = create_object_in_database(variant)

    yield obj

    delete_object_from_database(obj)

Pytest's fixture machinery only caches one instance of a parametrized fixture at a time, and class-scoped parametrization is specifically used to group the tests belonging to the same parameter value.

So your desired behavior is supported natively; the combination is:

@pytest.mark.parametrize(..., scope="class")

plus:

@pytest.fixture(scope="class")

with proper teardown of any external resources created by the fixture.

Please mark this answer as accepted if it helped, thank you.