Add an error_later warning filter action

changelog/15135.feature.rst

@@ -0,0 +1,11 @@
+Added an ``error_later`` warning filter action, valid anywhere a warning filter is accepted
+(:option:`-W`, the :confval:`filterwarnings` ini option and
+:ref:`@pytest.mark.filterwarnings <filterwarnings>`).
+
+``error`` raises at the :func:`warnings.warn` call site, aborting the code under test
+halfway through and reporting the failure at the frame that emitted the warning.
+``error_later`` still turns the warning into an error, but records it first: the code under
+test runs to completion, and pytest raises afterwards. The new :confval:`error_later_report`
+option chooses what fails -- ``test`` (the default) fails the test that emitted the warning,
+``session`` lets the tests pass and fails the run at the end with the new
+``ExitCode.LATE_WARNING_ERROR``.

doc/en/how-to/capture-warnings.rst

@@ -327,6 +327,78 @@ See :ref:`@pytest.mark.filterwarnings <filterwarnings>` and
     by calling :func:`warnings.simplefilter` (see :issue:`2430` for an example of that).
 
 
+.. _`error-later-warnings`:
+
+Erroring on warnings after the test instead of during it
+---------------------------------------------------------
+
+.. versionadded:: 9.2
+
+The ``error`` action raises at the :func:`warnings.warn` call site. That aborts the
+code under test halfway through, and the failure is reported at the frame that emitted
+the warning rather than where the test was going.
+
+pytest adds an ``error_later`` action for the cases where that is not what you want.
+The warning still becomes an error, but it is recorded first: the code under test runs
+to completion, and pytest raises afterwards.
+
+.. tab:: pyproject.toml
+
+    .. code-block:: toml
+
+        [tool.pytest.ini_options]
+        filterwarnings = [
+            'error_later::DeprecationWarning',
+        ]
+
+.. tab:: ini
+
+    .. code-block:: ini
+
+        [pytest]
+        filterwarnings =
+            error_later::DeprecationWarning
+
+``error_later`` is valid anywhere a warning filter is, including :option:`-W` and
+:ref:`@pytest.mark.filterwarnings <filterwarnings>`, and follows the same precedence
+rules as every other action: the last matching filter wins.
+
+The :confval:`error_later_report` option decides what fails:
+
+* ``test`` (the default) fails the test that emitted the warning, once the setup, call
+  or teardown phase that emitted it finishes. If that phase failed on its own, the
+  warnings are added to its failure report instead of failing it a second time.
+* ``session`` lets the tests pass, lists the warnings in a ``late warning errors``
+  section at the end of the run, and exits with :class:`pytest.ExitCode`
+  ``LATE_WARNING_ERROR`` (code ``7``), unless the run already failed for another reason.
+
+Warnings emitted where there is no test to fail end up in that end-of-run section in
+either mode: during collection, while plugins and ``conftest.py`` files are loaded, in
+``pytest_configure`` and ``pytest_sessionfinish``, and in a test that is skipped. Warnings
+emitted after the end-of-run report, in ``pytest_unconfigure`` for example, are too
+late to fail the run and are shown as usual.
+
+With :pypi:`pytest-xdist`, the workers pass their late warnings to the controller,
+which reports them and sets the exit code.
+
+.. note::
+
+    Python's own filter matching decides whether ``error_later`` applies, exactly as it
+    decides for ``error``. The message, category, module and line fields all work, and
+    module and line match the way :mod:`warnings` matches them::
+
+        filterwarnings = error_later::DeprecationWarning:some\.module
+
+    For the same reason, filters that the code under test installs itself, with
+    :func:`warnings.simplefilter` or :class:`warnings.catch_warnings` for example, take
+    precedence over an ``error_later`` filter, just as they take precedence over ``error``.
+    Warnings re-emitted by :func:`pytest.warns` carry a module name derived from the
+    file path, so a filter restricted to a module does not match them.
+
+    ``error_later`` is a pytest action: :envvar:`python:PYTHONWARNINGS` and ``python -W`` do
+    not accept it.
+
+
 .. _`ensuring a function triggers a deprecation warning`:
 
 .. _ensuring_function_triggers:

doc/en/reference/exit-codes.rst

@@ -3,7 +3,7 @@
 Exit codes
 ========================================================
 
-Running ``pytest`` can result in seven different exit codes:
+Running ``pytest`` can result in eight different exit codes:
 
 :Exit code 0: All tests were collected and passed successfully
 :Exit code 1: Tests were collected and run but some of the tests failed
@@ -12,6 +12,7 @@ Running ``pytest`` can result in seven different exit codes:
 :Exit code 4: pytest command line usage error, including a plugin that cannot be found or a ``conftest.py`` that fails to import
 :Exit code 5: No tests were collected
 :Exit code 6: Maximum number of warnings exceeded (see :option:`--max-warnings`)
+:Exit code 7: Warnings matched an ``error_later`` filter and no test failed for them (see :ref:`error-later-warnings`)
 
 They are represented by the :class:`pytest.ExitCode` enum. The exit codes being a part of the public API can be imported and accessed directly using:
 

doc/en/reference/reference.rst

@@ -1707,6 +1707,35 @@ passed multiple times. The expected format is ``name=value``. For example::
    into errors. For more information please refer to :ref:`warnings`.
 
 
+.. confval:: error_later_report
+   :type: ``"test" | "session"``
+   :default: ``"test"``
+
+   .. versionadded:: 9.2
+
+   What a warning matching an ``error_later`` filter fails.
+
+   ``test`` fails the test that emitted the warning, once the phase that emitted it
+   finishes. ``session`` lets the tests pass, lists the warnings at the end of the run
+   and exits with :class:`pytest.ExitCode` ``LATE_WARNING_ERROR`` (code ``7``).
+
+   .. tab:: toml
+
+       .. code-block:: toml
+
+            [tool.pytest.ini_options]
+            error_later_report = "session"
+
+   .. tab:: ini
+
+       .. code-block:: ini
+
+            [pytest]
+            error_later_report = session
+
+   See :ref:`error-later-warnings` for more information.
+
+
 .. confval:: max_warnings
    :type: ``int | str``
 

src/_pytest/config/__init__.py

@@ -81,6 +81,16 @@
 from _pytest.pathlib import resolve_package_path
 from _pytest.pathlib import safe_exists
 from _pytest.stash import Stash
+from _pytest.warning_late_error import catching_warnings
+from _pytest.warning_late_error import collect_or_show
+from _pytest.warning_late_error import ERROR_LATER_ACTION
+from _pytest.warning_late_error import install_warning_filter
+from _pytest.warning_late_error import late_warning_state_key
+from _pytest.warning_late_error import LateWarningLog
+from _pytest.warning_late_error import LateWarningState
+from _pytest.warning_late_error import recording_warnings
+from _pytest.warning_late_error import select_late_warnings
+from _pytest.warning_late_error import WarningFilter
 from _pytest.warning_types import PytestConfigWarning
 from _pytest.warning_types import warn_explicit_for
 
@@ -127,6 +137,8 @@ class ExitCode(enum.IntEnum):
     NO_TESTS_COLLECTED = 5
     #: All tests pass, but maximum number of warnings exceeded.
     MAX_WARNINGS_ERROR = 6
+    #: All tests pass, but warnings matching an ``error_later`` filter were emitted.
+    LATE_WARNING_ERROR = 7
 
     __module__ = "pytest"
 
@@ -1316,7 +1328,7 @@ def _catch_configured_warnings(
         self,
         *,
         record: bool,
-    ) -> Generator[list[warnings.WarningMessage] | None]:
+    ) -> Generator[LateWarningLog | None]:
         """Apply configured filters in a warnings-catching context.
 
         Defined here instead of _pytest.warnings as _do_configure uses
@@ -1325,7 +1337,10 @@ def _catch_configured_warnings(
         """
         config_filters = self.getini("filterwarnings")
         cmdline_filters = self.known_args_namespace.pythonwarnings or []
-        with warnings.catch_warnings(record=record) as log:
+        catcher: contextlib.AbstractContextManager[LateWarningLog | None] = (
+            recording_warnings() if record else catching_warnings()
+        )
+        with catcher as log:
             if not sys.warnoptions:
                 # If user is not explicitly configuring warning filters, show deprecation warnings by default (#2908).
                 warnings.filterwarnings("always", category=DeprecationWarning)
@@ -1335,6 +1350,12 @@ def _catch_configured_warnings(
             # warnings.filterwarnings("error", category=pytest.PytestRemovedIn10Warning)
 
             apply_warning_filters(config_filters, cmdline_filters)
+            state = self.stash.setdefault(late_warning_state_key, LateWarningState())
+            if log is None:
+                # Nothing records here, so an 'error_later' warning would just be
+                # printed. Collect it instead; nested recording contexts swap
+                # showwarning out and back, so this only sees unrecorded warnings.
+                warnings.showwarning = collect_or_show(state, warnings.showwarning)
             yield log
 
     @contextlib.contextmanager
@@ -1346,6 +1367,8 @@ def _capture_plugin_import_warnings(self) -> Iterator[None]:
             try:
                 yield
             finally:
+                state = self.stash[late_warning_state_key]
+                state.collected.extend(select_late_warnings(records, 0, nodeid=""))
                 for warning_message in records:
                     self.hook.pytest_warning_recorded.call_historic(
                         kwargs=dict(
@@ -1834,12 +1857,14 @@ def issue_config_time_warning(self, warning: Warning, stacklevel: int) -> None:
         cmdline_filters = self.known_args_namespace.pythonwarnings or []
         config_filters = self.getini("filterwarnings")
 
-        with warnings.catch_warnings(record=True) as records:
+        with recording_warnings() as records:
             warnings.simplefilter("always", type(warning))
             apply_warning_filters(config_filters, cmdline_filters)
             warnings.warn(warning, stacklevel=stacklevel)
 
         if records:
+            state = self.stash.setdefault(late_warning_state_key, LateWarningState())
+            state.collected.extend(select_late_warnings(records, 0, nodeid=""))
             frame = sys._getframe(stacklevel - 1)
             location = frame.f_code.co_filename, frame.f_lineno, frame.f_code.co_name
             self.hook.pytest_warning_recorded.call_historic(
@@ -2374,9 +2399,7 @@ def _strtobool(val: str) -> bool:
 
 
 @lru_cache(maxsize=50)
-def parse_warning_filter(
-    arg: str, *, escape: bool
-) -> tuple[warnings._ActionKind, str, type[Warning], str, int]:
+def parse_warning_filter(arg: str, *, escape: bool) -> WarningFilter:
     """Parse a warnings filter string.
 
     This is copied from warnings._setoption with the following changes:
@@ -2417,10 +2440,14 @@ def parse_warning_filter(
     while len(parts) < 5:
         parts.append("")
     action_, message, category_, module, lineno_ = (s.strip() for s in parts)
-    try:
-        action: warnings._ActionKind = warnings._getaction(action_)  # type: ignore[attr-defined]
-    except warnings._OptionError as e:
-        raise UsageError(error_template.format(error=str(e))) from None
+    action: warnings._ActionKind | Literal["error_later"]
+    if action_ == ERROR_LATER_ACTION:
+        action = ERROR_LATER_ACTION
+    else:
+        try:
+            action = warnings._getaction(action_)  # type: ignore[attr-defined]
+        except warnings._OptionError as e:
+            raise UsageError(error_template.format(error=str(e))) from None
     try:
         category: type[Warning] = _resolve_warning_category(category_)
     except ImportError:
@@ -2478,24 +2505,35 @@ def _resolve_warning_category(category: str) -> type[Warning]:
 
 def apply_warning_filters(
     config_filters: Iterable[str], cmdline_filters: Iterable[str]
-) -> None:
-    """Applies pytest-configured filters to the warnings module"""
+) -> list[WarningFilter]:
+    """Applies pytest-configured filters to the warnings module.
+
+    Returns the filters that were applied, in application order, so that the
+    ``error_later`` action can resolve precedence the way ``warnings.filters`` does.
+    """
+    applied: list[WarningFilter] = []
     # Filters should have this precedence: cmdline options, config.
     # Filters should be applied in the inverse order of precedence.
     for arg in config_filters:
         try:
-            warnings.filterwarnings(*parse_warning_filter(arg, escape=False))
+            parsed = parse_warning_filter(arg, escape=False)
         except ImportError as e:
             warnings.warn(
                 f"Failed to import filter module '{e.name}': {arg}", PytestConfigWarning
             )
             continue
+        install_warning_filter(parsed)
+        applied.append(parsed)
 
     for arg in cmdline_filters:
         try:
-            warnings.filterwarnings(*parse_warning_filter(arg, escape=True))
+            parsed = parse_warning_filter(arg, escape=True)
         except ImportError as e:
             warnings.warn(
                 f"Failed to import filter module '{e.name}': {arg}", PytestConfigWarning
             )
             continue
+        install_warning_filter(parsed)
+        applied.append(parsed)
+
+    return applied

src/_pytest/main.py

@@ -148,6 +148,13 @@ def pytest_addoption(parser: Parser) -> None:
         type=int | str,
         default=None,
     )
+    parser.addini(
+        "error_later_report",
+        default="test",
+        help="What an 'error_later' warning filter fails: "
+        "'test' fails the test that emitted the warning, "
+        "'session' lists them and fails at the end of the session",
+    )
 
     group = parser.getgroup("collect", "collection")
     group.addoption(

src/_pytest/terminal.py

@@ -53,6 +53,7 @@
 from _pytest.reports import BaseReport
 from _pytest.reports import CollectReport
 from _pytest.reports import TestReport
+from _pytest.warning_late_error import LATE_WARNING_ERRORS_STAT
 
 
 if TYPE_CHECKING:
@@ -70,6 +71,7 @@
     "xpassed",
     "warnings",
     "error",
+    LATE_WARNING_ERRORS_STAT,
     "subtests passed",
     "subtests failed",
     "subtests skipped",
@@ -1436,7 +1438,7 @@ def _get_main_color(self) -> tuple[str, list[str]]:
 
     def _determine_main_color(self, unknown_type_seen: bool) -> str:
         stats = self.stats
-        if "failed" in stats or "error" in stats:
+        if "failed" in stats or "error" in stats or LATE_WARNING_ERRORS_STAT in stats:
             main_color = "red"
         elif "warnings" in stats or "xpassed" in stats or unknown_type_seen:
             main_color = "yellow"
@@ -1641,6 +1643,7 @@ def _folded_skips(
 _color_for_type = {
     "failed": "red",
     "error": "red",
+    LATE_WARNING_ERRORS_STAT: "red",
     "warnings": "yellow",
     "passed": "green",
     "subtests passed": "green",
@@ -1651,13 +1654,13 @@ def _folded_skips(
 
 def pluralize(count: int, noun: str) -> tuple[int, str]:
     # No need to pluralize words such as `failed` or `passed`.
-    if noun not in ["error", "warnings", "test"]:
+    if noun not in ["error", "warnings", "test", LATE_WARNING_ERRORS_STAT]:
         return count, noun
 
     # The `warnings` key is plural. To avoid API breakage, we keep it that way but
     # set it to singular here so we can determine plurality in the same way as we do
-    # for `error`.
-    noun = noun.replace("warnings", "warning")
+    # for `error`. The late warning errors key follows the same pattern.
+    noun = noun.replace("warnings", "warning").replace("errors", "error")
 
     return count, noun + "s" if count != 1 else noun