Test property to be added to AssertionError texts

[atatarn]

For infrastructure tests it is nice to have a link to the infrastructure documentation e.g. "how to fix this failure" in AssertionError message.
But to properly implement this all the assertion messages must be updated with relevant urls. And the next time an engineer will change the test logic and/or assertion message - url will be gone.

Is it possible to somehow assign a text property with an url to the test function / test class and then have some hook which will take assertion message from the test and append the property to it if the property exists?

[portyu9]

Hi @atatarn

I would attach the URL as a custom marker and add it to the test report, rather than trying to rewrite every AssertionError.

There is no general public hook for safely rewriting the text of every arbitrary assertion failure. pytest_runtest_makereport is a better fit because it lets you augment the failure while preserving pytest's original assertion diagnostics.

For example:

# conftest.py
import pytest


def pytest_configure(config):
    config.addinivalue_line(
        "markers",
        "fix_doc(url): documentation explaining how to fix this test",
    )


@pytest.hookimpl(wrapper=True)
def pytest_runtest_makereport(item, call):
    report = yield

    if report.when == "call" and report.failed:
        mark = item.get_closest_marker("fix_doc")
        if mark and mark.args:
            report.sections.append(
                ("How to fix this failure", str(mark.args[0]))
            )

    return report

Then:

@pytest.mark.fix_doc("https://example.com/docs/how-to-fix-this")
def test_something():
    assert False

get_closest_marker() also makes this work when the marker is placed on a class rather than every individual test.

The failure keeps its normal traceback/assertion explanation, and the documentation URL is added as a separate failure-report section, so future changes to the assertion itself cannot accidentally remove it.

Please mark this answer as accepted if it helped, thank you.