diff --git a/changelog/15135.feature.rst b/changelog/15135.feature.rst new file mode 100644 index 00000000000..410a349fa78 --- /dev/null +++ b/changelog/15135.feature.rst @@ -0,0 +1,11 @@ +Added an ``error_later`` warning filter action, valid anywhere a warning filter is accepted +(:option:`-W`, the :confval:`filterwarnings` ini option and +:ref:`@pytest.mark.filterwarnings `). + +``error`` raises at the :func:`warnings.warn` call site, aborting the code under test +halfway through and reporting the failure at the frame that emitted the warning. +``error_later`` still turns the warning into an error, but records it first: the code under +test runs to completion, and pytest raises afterwards. The new :confval:`error_later_report` +option chooses what fails -- ``test`` (the default) fails the test that emitted the warning, +``session`` lets the tests pass and fails the run at the end with the new +``ExitCode.LATE_WARNING_ERROR``. diff --git a/doc/en/how-to/capture-warnings.rst b/doc/en/how-to/capture-warnings.rst index d9336c0a144..172cd35d613 100644 --- a/doc/en/how-to/capture-warnings.rst +++ b/doc/en/how-to/capture-warnings.rst @@ -327,6 +327,78 @@ See :ref:`@pytest.mark.filterwarnings ` and by calling :func:`warnings.simplefilter` (see :issue:`2430` for an example of that). +.. _`error-later-warnings`: + +Erroring on warnings after the test instead of during it +--------------------------------------------------------- + +.. versionadded:: 9.2 + +The ``error`` action raises at the :func:`warnings.warn` call site. That aborts the +code under test halfway through, and the failure is reported at the frame that emitted +the warning rather than where the test was going. + +pytest adds an ``error_later`` action for the cases where that is not what you want. +The warning still becomes an error, but it is recorded first: the code under test runs +to completion, and pytest raises afterwards. + +.. tab:: pyproject.toml + + .. code-block:: toml + + [tool.pytest.ini_options] + filterwarnings = [ + 'error_later::DeprecationWarning', + ] + +.. tab:: ini + + .. code-block:: ini + + [pytest] + filterwarnings = + error_later::DeprecationWarning + +``error_later`` is valid anywhere a warning filter is, including :option:`-W` and +:ref:`@pytest.mark.filterwarnings `, and follows the same precedence +rules as every other action: the last matching filter wins. + +The :confval:`error_later_report` option decides what fails: + +* ``test`` (the default) fails the test that emitted the warning, once the setup, call + or teardown phase that emitted it finishes. If that phase failed on its own, the + warnings are added to its failure report instead of failing it a second time. +* ``session`` lets the tests pass, lists the warnings in a ``late warning errors`` + section at the end of the run, and exits with :class:`pytest.ExitCode` + ``LATE_WARNING_ERROR`` (code ``7``), unless the run already failed for another reason. + +Warnings emitted where there is no test to fail end up in that end-of-run section in +either mode: during collection, while plugins and ``conftest.py`` files are loaded, in +``pytest_configure`` and ``pytest_sessionfinish``, and in a test that is skipped. Warnings +emitted after the end-of-run report, in ``pytest_unconfigure`` for example, are too +late to fail the run and are shown as usual. + +With :pypi:`pytest-xdist`, the workers pass their late warnings to the controller, +which reports them and sets the exit code. + +.. note:: + + Python's own filter matching decides whether ``error_later`` applies, exactly as it + decides for ``error``. The message, category, module and line fields all work, and + module and line match the way :mod:`warnings` matches them:: + + filterwarnings = error_later::DeprecationWarning:some\.module + + For the same reason, filters that the code under test installs itself, with + :func:`warnings.simplefilter` or :class:`warnings.catch_warnings` for example, take + precedence over an ``error_later`` filter, just as they take precedence over ``error``. + Warnings re-emitted by :func:`pytest.warns` carry a module name derived from the + file path, so a filter restricted to a module does not match them. + + ``error_later`` is a pytest action: :envvar:`python:PYTHONWARNINGS` and ``python -W`` do + not accept it. + + .. _`ensuring a function triggers a deprecation warning`: .. _ensuring_function_triggers: diff --git a/doc/en/reference/exit-codes.rst b/doc/en/reference/exit-codes.rst index fff25c96ea0..f5399a816c6 100644 --- a/doc/en/reference/exit-codes.rst +++ b/doc/en/reference/exit-codes.rst @@ -3,7 +3,7 @@ Exit codes ======================================================== -Running ``pytest`` can result in seven different exit codes: +Running ``pytest`` can result in eight different exit codes: :Exit code 0: All tests were collected and passed successfully :Exit code 1: Tests were collected and run but some of the tests failed @@ -12,6 +12,7 @@ Running ``pytest`` can result in seven different exit codes: :Exit code 4: pytest command line usage error, including a plugin that cannot be found or a ``conftest.py`` that fails to import :Exit code 5: No tests were collected :Exit code 6: Maximum number of warnings exceeded (see :option:`--max-warnings`) +:Exit code 7: Warnings matched an ``error_later`` filter and no test failed for them (see :ref:`error-later-warnings`) They are represented by the :class:`pytest.ExitCode` enum. The exit codes being a part of the public API can be imported and accessed directly using: diff --git a/doc/en/reference/reference.rst b/doc/en/reference/reference.rst index 4e0f40f4f93..6344500cc26 100644 --- a/doc/en/reference/reference.rst +++ b/doc/en/reference/reference.rst @@ -1707,6 +1707,35 @@ passed multiple times. The expected format is ``name=value``. For example:: into errors. For more information please refer to :ref:`warnings`. +.. confval:: error_later_report + :type: ``"test" | "session"`` + :default: ``"test"`` + + .. versionadded:: 9.2 + + What a warning matching an ``error_later`` filter fails. + + ``test`` fails the test that emitted the warning, once the phase that emitted it + finishes. ``session`` lets the tests pass, lists the warnings at the end of the run + and exits with :class:`pytest.ExitCode` ``LATE_WARNING_ERROR`` (code ``7``). + + .. tab:: toml + + .. code-block:: toml + + [tool.pytest.ini_options] + error_later_report = "session" + + .. tab:: ini + + .. code-block:: ini + + [pytest] + error_later_report = session + + See :ref:`error-later-warnings` for more information. + + .. confval:: max_warnings :type: ``int | str`` diff --git a/src/_pytest/config/__init__.py b/src/_pytest/config/__init__.py index 60ba5f54a89..f6eb6ddc74e 100644 --- a/src/_pytest/config/__init__.py +++ b/src/_pytest/config/__init__.py @@ -81,6 +81,16 @@ from _pytest.pathlib import resolve_package_path from _pytest.pathlib import safe_exists from _pytest.stash import Stash +from _pytest.warning_late_error import catching_warnings +from _pytest.warning_late_error import collect_or_show +from _pytest.warning_late_error import ERROR_LATER_ACTION +from _pytest.warning_late_error import install_warning_filter +from _pytest.warning_late_error import late_warning_state_key +from _pytest.warning_late_error import LateWarningLog +from _pytest.warning_late_error import LateWarningState +from _pytest.warning_late_error import recording_warnings +from _pytest.warning_late_error import select_late_warnings +from _pytest.warning_late_error import WarningFilter from _pytest.warning_types import PytestConfigWarning from _pytest.warning_types import warn_explicit_for @@ -127,6 +137,8 @@ class ExitCode(enum.IntEnum): NO_TESTS_COLLECTED = 5 #: All tests pass, but maximum number of warnings exceeded. MAX_WARNINGS_ERROR = 6 + #: All tests pass, but warnings matching an ``error_later`` filter were emitted. + LATE_WARNING_ERROR = 7 __module__ = "pytest" @@ -1316,7 +1328,7 @@ def _catch_configured_warnings( self, *, record: bool, - ) -> Generator[list[warnings.WarningMessage] | None]: + ) -> Generator[LateWarningLog | None]: """Apply configured filters in a warnings-catching context. Defined here instead of _pytest.warnings as _do_configure uses @@ -1325,7 +1337,10 @@ def _catch_configured_warnings( """ config_filters = self.getini("filterwarnings") cmdline_filters = self.known_args_namespace.pythonwarnings or [] - with warnings.catch_warnings(record=record) as log: + catcher: contextlib.AbstractContextManager[LateWarningLog | None] = ( + recording_warnings() if record else catching_warnings() + ) + with catcher as log: if not sys.warnoptions: # If user is not explicitly configuring warning filters, show deprecation warnings by default (#2908). warnings.filterwarnings("always", category=DeprecationWarning) @@ -1335,6 +1350,12 @@ def _catch_configured_warnings( # warnings.filterwarnings("error", category=pytest.PytestRemovedIn10Warning) apply_warning_filters(config_filters, cmdline_filters) + state = self.stash.setdefault(late_warning_state_key, LateWarningState()) + if log is None: + # Nothing records here, so an 'error_later' warning would just be + # printed. Collect it instead; nested recording contexts swap + # showwarning out and back, so this only sees unrecorded warnings. + warnings.showwarning = collect_or_show(state, warnings.showwarning) yield log @contextlib.contextmanager @@ -1346,6 +1367,8 @@ def _capture_plugin_import_warnings(self) -> Iterator[None]: try: yield finally: + state = self.stash[late_warning_state_key] + state.collected.extend(select_late_warnings(records, 0, nodeid="")) for warning_message in records: self.hook.pytest_warning_recorded.call_historic( kwargs=dict( @@ -1834,12 +1857,14 @@ def issue_config_time_warning(self, warning: Warning, stacklevel: int) -> None: cmdline_filters = self.known_args_namespace.pythonwarnings or [] config_filters = self.getini("filterwarnings") - with warnings.catch_warnings(record=True) as records: + with recording_warnings() as records: warnings.simplefilter("always", type(warning)) apply_warning_filters(config_filters, cmdline_filters) warnings.warn(warning, stacklevel=stacklevel) if records: + state = self.stash.setdefault(late_warning_state_key, LateWarningState()) + state.collected.extend(select_late_warnings(records, 0, nodeid="")) frame = sys._getframe(stacklevel - 1) location = frame.f_code.co_filename, frame.f_lineno, frame.f_code.co_name self.hook.pytest_warning_recorded.call_historic( @@ -2374,9 +2399,7 @@ def _strtobool(val: str) -> bool: @lru_cache(maxsize=50) -def parse_warning_filter( - arg: str, *, escape: bool -) -> tuple[warnings._ActionKind, str, type[Warning], str, int]: +def parse_warning_filter(arg: str, *, escape: bool) -> WarningFilter: """Parse a warnings filter string. This is copied from warnings._setoption with the following changes: @@ -2417,10 +2440,14 @@ def parse_warning_filter( while len(parts) < 5: parts.append("") action_, message, category_, module, lineno_ = (s.strip() for s in parts) - try: - action: warnings._ActionKind = warnings._getaction(action_) # type: ignore[attr-defined] - except warnings._OptionError as e: - raise UsageError(error_template.format(error=str(e))) from None + action: warnings._ActionKind | Literal["error_later"] + if action_ == ERROR_LATER_ACTION: + action = ERROR_LATER_ACTION + else: + try: + action = warnings._getaction(action_) # type: ignore[attr-defined] + except warnings._OptionError as e: + raise UsageError(error_template.format(error=str(e))) from None try: category: type[Warning] = _resolve_warning_category(category_) except ImportError: @@ -2478,24 +2505,35 @@ def _resolve_warning_category(category: str) -> type[Warning]: def apply_warning_filters( config_filters: Iterable[str], cmdline_filters: Iterable[str] -) -> None: - """Applies pytest-configured filters to the warnings module""" +) -> list[WarningFilter]: + """Applies pytest-configured filters to the warnings module. + + Returns the filters that were applied, in application order, so that the + ``error_later`` action can resolve precedence the way ``warnings.filters`` does. + """ + applied: list[WarningFilter] = [] # Filters should have this precedence: cmdline options, config. # Filters should be applied in the inverse order of precedence. for arg in config_filters: try: - warnings.filterwarnings(*parse_warning_filter(arg, escape=False)) + parsed = parse_warning_filter(arg, escape=False) except ImportError as e: warnings.warn( f"Failed to import filter module '{e.name}': {arg}", PytestConfigWarning ) continue + install_warning_filter(parsed) + applied.append(parsed) for arg in cmdline_filters: try: - warnings.filterwarnings(*parse_warning_filter(arg, escape=True)) + parsed = parse_warning_filter(arg, escape=True) except ImportError as e: warnings.warn( f"Failed to import filter module '{e.name}': {arg}", PytestConfigWarning ) continue + install_warning_filter(parsed) + applied.append(parsed) + + return applied diff --git a/src/_pytest/main.py b/src/_pytest/main.py index d43a68b4679..dc5e6870003 100644 --- a/src/_pytest/main.py +++ b/src/_pytest/main.py @@ -148,6 +148,13 @@ def pytest_addoption(parser: Parser) -> None: type=int | str, default=None, ) + parser.addini( + "error_later_report", + default="test", + help="What an 'error_later' warning filter fails: " + "'test' fails the test that emitted the warning, " + "'session' lists them and fails at the end of the session", + ) group = parser.getgroup("collect", "collection") group.addoption( diff --git a/src/_pytest/terminal.py b/src/_pytest/terminal.py index b77a649d28a..920638b44c5 100644 --- a/src/_pytest/terminal.py +++ b/src/_pytest/terminal.py @@ -53,6 +53,7 @@ from _pytest.reports import BaseReport from _pytest.reports import CollectReport from _pytest.reports import TestReport +from _pytest.warning_late_error import LATE_WARNING_ERRORS_STAT if TYPE_CHECKING: @@ -70,6 +71,7 @@ "xpassed", "warnings", "error", + LATE_WARNING_ERRORS_STAT, "subtests passed", "subtests failed", "subtests skipped", @@ -1436,7 +1438,7 @@ def _get_main_color(self) -> tuple[str, list[str]]: def _determine_main_color(self, unknown_type_seen: bool) -> str: stats = self.stats - if "failed" in stats or "error" in stats: + if "failed" in stats or "error" in stats or LATE_WARNING_ERRORS_STAT in stats: main_color = "red" elif "warnings" in stats or "xpassed" in stats or unknown_type_seen: main_color = "yellow" @@ -1641,6 +1643,7 @@ def _folded_skips( _color_for_type = { "failed": "red", "error": "red", + LATE_WARNING_ERRORS_STAT: "red", "warnings": "yellow", "passed": "green", "subtests passed": "green", @@ -1651,13 +1654,13 @@ def _folded_skips( def pluralize(count: int, noun: str) -> tuple[int, str]: # No need to pluralize words such as `failed` or `passed`. - if noun not in ["error", "warnings", "test"]: + if noun not in ["error", "warnings", "test", LATE_WARNING_ERRORS_STAT]: return count, noun # The `warnings` key is plural. To avoid API breakage, we keep it that way but # set it to singular here so we can determine plurality in the same way as we do - # for `error`. - noun = noun.replace("warnings", "warning") + # for `error`. The late warning errors key follows the same pattern. + noun = noun.replace("warnings", "warning").replace("errors", "error") return count, noun + "s" if count != 1 else noun diff --git a/src/_pytest/warning_late_error.py b/src/_pytest/warning_late_error.py new file mode 100644 index 00000000000..ae1aa873bc2 --- /dev/null +++ b/src/_pytest/warning_late_error.py @@ -0,0 +1,318 @@ +"""Warnings that become errors after the fact. + +The ``error`` warning filter raises at the ``warnings.warn()`` call site, which +aborts whatever the code under test was doing halfway through and reports the +failure at the frame that emitted the warning. The ``error_later`` action is +pytest's alternative: the warning still becomes an error, but it is recorded +first, the code under test runs to completion, and pytest raises afterwards at a +defined point. + +This module holds the pieces both :mod:`_pytest.config` and :mod:`_pytest.warnings` +need, so that neither has to import the other. +""" + +from __future__ import annotations + +from collections.abc import Callable +from collections.abc import Iterator +import contextlib +import dataclasses +import re +import sys +import threading +from typing import Any +from typing import cast +from typing import Final +from typing import Literal +from typing import TextIO +import warnings + +from _pytest.stash import StashKey + + +#: Filter action implemented by pytest rather than by the :mod:`warnings` module. +ERROR_LATER_ACTION: Final = "error_later" + +#: Key of the terminal reporter's stats, and so of the final summary line. +LATE_WARNING_ERRORS_STAT: Final = "late warning errors" + +#: A parsed warning filter: action, message regex, category, module regex, lineno. +WarningFilter = tuple[ + "warnings._ActionKind | Literal['error_later']", str, type[Warning], str, int +] + + +@dataclasses.dataclass(frozen=True) +class LateWarning: + """A warning that matched an ``error_later`` filter, rendered down to plain data. + + Everything is pre-rendered so that no warning instance, and nothing the + warning referenced, is kept alive until the report is written, and so that + it can travel from a pytest-xdist worker to the controller. + """ + + message: str + category: str + filename: str + lineno: int + nodeid: str + + def format(self, *, with_nodeid: bool = False) -> str: + location = f"{self.filename}:{self.lineno}" + if with_nodeid and self.nodeid: + location = f"{location} ({self.nodeid})" + return f"{location}: {self.category}: {self.message}" + + +@dataclasses.dataclass +class LateWarningState: + """Per-session state for the ``error_later`` action.""" + + #: Warnings held over to the end of the session. + collected: list[LateWarning] = dataclasses.field(default_factory=list) + #: How many of ``collected`` have been written to the terminal. + reported: int = 0 + #: Set once the session can no longer report; warnings after that point + #: are shown the ordinary way instead of being collected and lost. + closed: bool = False + + +late_warning_state_key: StashKey[LateWarningState] = StashKey() + + +# How the stdlib is made to decide ``error_later`` itself +# --------------------------------------------------------- +# +# An ``error_later`` filter is installed as a real ``always`` filter whose +# module field is a _WinProbe, immediately followed by a never-matching filter +# whose module field is a _LossProbe. The warnings module calls ``.match()`` on +# whatever object sits in the message and module fields of every filter it +# visits (Python/_warnings.c:check_matched, and ``mod.match(module)`` in the +# pure-Python warn_explicit), and stops at the first filter that matches. So +# for each warning: +# +# - the win probe sees the exact module name the stdlib matches against; +# - if its filter wins, the search stops and the warning is shown next, in +# the same thread, before anything else can run; +# - if its filter loses on a later field (category and line are checked after +# the module in C), the search goes on to the loss probe, which withdraws it. +# +# The verdict is handed over through a thread-local. The win probe cannot +# know where the warning will be shown: straight to pytest's recording, through +# a test's showwarning hook that forwards it there (pytest-dev/pytest#15139), or +# somewhere else entirely (pytest.warns, a test's own catch_warnings, +# logging.captureWarnings). So the probe only leaves the verdict pending, and a +# wrapper around ``warnings._showwarnmsg``, which the stdlib calls for every +# shown warning, makes it current for exactly that call. Whatever reaches +# pytest's recording log or showwarning wrapper during the call consumes it; +# when the call returns it is gone, so nothing is left for the next warning. + +_verdict = threading.local() + +#: Python 3.14+: filters and the record log live in a context variable. +_CONTEXT_AWARE: Final = bool(getattr(sys.flags, "context_aware_warnings", False)) + + +def _take_verdict() -> str | None: + """The module name, if an ``error_later`` filter won the warning being shown; once.""" + module: str | None = getattr(_verdict, "current", None) + _verdict.current = None + return module + + +def _scope_verdicts_to_show_calls() -> None: + show = warnings._showwarnmsg # type: ignore[attr-defined] + if getattr(show, "_pytest_scopes_verdicts", False): + return + + def show_with_verdict(msg: warnings.WarningMessage) -> None: + # Restored rather than cleared: a hook may warn again before forwarding. + outer = getattr(_verdict, "current", None) + _verdict.current = getattr(_verdict, "pending", None) + _verdict.pending = None + try: + show(msg) + finally: + _verdict.current = outer + + show_with_verdict._pytest_scopes_verdicts = True # type: ignore[attr-defined] + warnings._showwarnmsg = show_with_verdict # type: ignore[attr-defined] + + +class LateWarningLog(list[warnings.WarningMessage]): + """A ``catch_warnings(record=True)`` log that knows which entries erred later. + + It records through the plain ``list.append`` until an ``error_later`` + filter is installed in its context; only then does it route through + :meth:`_record`, so sessions without such a filter pay nothing. + """ + + def __init__(self) -> None: + super().__init__() + self._late: dict[int, str] = {} + + def _record(self, warning_message: warnings.WarningMessage) -> None: + module = _take_verdict() + if module is not None: + self._late[id(warning_message)] = module + self.append(warning_message) + + def is_late(self, warning_message: warnings.WarningMessage) -> bool: + return id(warning_message) in self._late + + +class _ContextSink: + """``_Context.log`` stand-in: 3.14 context-aware recording calls ``log.append``.""" + + __slots__ = ("append", "log") + + def __init__(self, log: LateWarningLog) -> None: + self.log = log + self.append = log._record + + +def _route_recording_through_verdicts() -> None: + if _CONTEXT_AWARE: + context = warnings._get_context() # type: ignore[attr-defined] + if isinstance(context.log, LateWarningLog): + context.log = _ContextSink(context.log) + else: + log = getattr(warnings._showwarnmsg_impl, "__self__", None) # type: ignore[attr-defined] + if isinstance(log, LateWarningLog): + warnings._showwarnmsg_impl = log._record # type: ignore[attr-defined] + + +class _WinProbe: + __slots__ = ("_regex", "pattern") + + def __init__(self, module: str) -> None: + #: Mirrors ``re.Pattern.pattern`` for code that inspects ``warnings.filters``. + self.pattern = module + self._regex = re.compile(module) if module else None + + def match(self, module: str) -> object: + matched = True if self._regex is None else self._regex.match(module) + if matched: + _verdict.pending = module + return matched + + def __repr__(self) -> str: + return f"" + + +class _LossProbe: + __slots__ = () + pattern = None + + def match(self, module: str) -> None: + _verdict.pending = None + + def __repr__(self) -> str: + return "" + + +def _install_error_later( + message: str, category: type[Warning], module: str, lineno: int +) -> None: + # Not warnings.filterwarnings(): it insists on str patterns and compiles them. + filters = cast( + "list[Any]", + warnings._get_filters() + if hasattr(warnings, "_get_filters") + else warnings.filters, + ) + regex = re.compile(message, re.IGNORECASE) if message else None + with getattr(warnings, "_lock", contextlib.nullcontext()): + filters.insert(0, ("always", None, Warning, _LossProbe(), 0)) + filters.insert(0, ("always", regex, category, _WinProbe(module), lineno)) + warnings._filters_mutated() # type: ignore[attr-defined] + _route_recording_through_verdicts() + _scope_verdicts_to_show_calls() + + +def install_warning_filter(filter_: WarningFilter) -> None: + """Apply a parsed filter to the :mod:`warnings` module.""" + action, message, category, module, lineno = filter_ + if action == ERROR_LATER_ACTION: + _install_error_later(message, category, module, lineno) + else: + warnings.filterwarnings(action, message, category, module, lineno) + + +@contextlib.contextmanager +def _keeping_showwarnmsg() -> Iterator[None]: + # catch_warnings() restores showwarning, but not the _showwarnmsg that an + # error_later filter wraps. + show = warnings._showwarnmsg # type: ignore[attr-defined] + try: + yield + finally: + warnings._showwarnmsg = show # type: ignore[attr-defined] + + +@contextlib.contextmanager +def catching_warnings() -> Iterator[None]: + """``catch_warnings()``, for contexts that may install ``error_later`` filters.""" + with _keeping_showwarnmsg(), warnings.catch_warnings(): + yield + + +@contextlib.contextmanager +def recording_warnings() -> Iterator[LateWarningLog]: + """``catch_warnings(record=True)``, recording into a :class:`LateWarningLog`.""" + with _keeping_showwarnmsg(), warnings.catch_warnings(record=True): + log = LateWarningLog() + if _CONTEXT_AWARE: + warnings._get_context().log = log # type: ignore[attr-defined] + else: + warnings._showwarnmsg_impl = log.append # type: ignore[attr-defined] + yield log + + +def collect_or_show( + state: LateWarningState, showwarning: Callable[..., object] +) -> Callable[..., None]: + """``warnings.showwarning`` for contexts that do not record.""" + + def show( + message: Warning | str, + category: type[Warning], + filename: str, + lineno: int, + file: TextIO | None = None, + line: str | None = None, + ) -> None: + if _take_verdict() is not None and not state.closed: + warning_message = warnings.WarningMessage( + message, category, filename, lineno, file, line + ) + state.collected.append(to_late_warning(warning_message, nodeid="")) + else: + showwarning(message, category, filename, lineno, file, line) + + return show + + +def to_late_warning( + warning_message: warnings.WarningMessage, nodeid: str +) -> LateWarning: + return LateWarning( + message=str(warning_message.message), + category=warning_message.category.__name__, + filename=warning_message.filename, + lineno=warning_message.lineno, + nodeid=nodeid, + ) + + +def select_late_warnings( + log: LateWarningLog, start: int, nodeid: str +) -> list[LateWarning]: + """The warnings of ``log[start:]`` that an ``error_later`` filter won.""" + if not log._late: + return [] + return [ + to_late_warning(warning_message, nodeid) + for warning_message in log[start:] + if log.is_late(warning_message) + ] diff --git a/src/_pytest/warnings.py b/src/_pytest/warnings.py index 2c721d6e3c9..2e394399cc4 100644 --- a/src/_pytest/warnings.py +++ b/src/_pytest/warnings.py @@ -3,15 +3,31 @@ from collections.abc import Generator from contextlib import contextmanager +import dataclasses +from typing import Any from typing import Literal import warnings +from _pytest._code.code import ExceptionRepr from _pytest.config import Config +from _pytest.config import ExitCode from _pytest.config import parse_warning_filter +from _pytest.config import UsageError from _pytest.main import Session from _pytest.nodes import Item +from _pytest.outcomes import fail +from _pytest.reports import TestReport +from _pytest.runner import CallInfo +from _pytest.stash import StashKey from _pytest.terminal import TerminalReporter from _pytest.tracemalloc import tracemalloc_message +from _pytest.warning_late_error import install_warning_filter +from _pytest.warning_late_error import LATE_WARNING_ERRORS_STAT +from _pytest.warning_late_error import late_warning_state_key +from _pytest.warning_late_error import LateWarning +from _pytest.warning_late_error import LateWarningLog +from _pytest.warning_late_error import LateWarningState +from _pytest.warning_late_error import select_late_warnings import pytest @@ -33,14 +49,27 @@ def catch_warnings_for_item( with config._catch_configured_warnings(record=record) as log: # apply filters from "filterwarnings" marks nodeid = "" if item is None else item.nodeid + state = config.stash.setdefault(late_warning_state_key, LateWarningState()) if item is not None: for mark in item.iter_markers(name="filterwarnings"): for arg in mark.args: - warnings.filterwarnings(*parse_warning_filter(arg, escape=False)) + parsed = parse_warning_filter(arg, escape=False) + install_warning_filter(parsed) + # record=True means log is not None; mypy can't infer that. + recording = _Recording(log=log, nodeid=nodeid) if log is not None else None + recordings = config.stash.setdefault(_recordings_key, []) + if recording is not None: + recordings.append(recording) try: yield finally: + if recording is not None: + recordings.remove(recording) + # Anything not already drained by a runtest phase boundary can + # only be reported at the end of the session. + state.collected.extend(_drain(config, recording)) + if record: # mypy can't infer that record=True means log is not None; help it. assert log is not None @@ -56,6 +85,115 @@ def catch_warnings_for_item( ) +@dataclasses.dataclass +class _Recording: + """A live ``catch_warnings(record=True)`` log and how much of it was drained.""" + + log: LateWarningLog + nodeid: str + cursor: int = 0 + + +#: Active recordings, innermost last. +_recordings_key: StashKey[list[_Recording]] = StashKey() + +#: Late warnings of a runtest phase that raised, keyed by phase, waiting for +#: that phase's report so they can be attached to its failure. +_phase_late_warnings_key: StashKey[dict[str, list[LateWarning]]] = StashKey() + + +def _drain(config: Config, recording: _Recording) -> list[LateWarning]: + """Take the warnings recorded since the last drain that must error later.""" + late = select_late_warnings(recording.log, recording.cursor, recording.nodeid) + recording.cursor = len(recording.log) + return late + + +def _describe(late: list[LateWarning]) -> str: + plural = "s" if len(late) > 1 else "" + lines = "\n".join(w.format() for w in late) + return f"{len(late)} warning{plural} matched an 'error_later' filter:\n{lines}" + + +def _late_warnings_phase(item: Item, when: str) -> Generator[None, object, object]: + """Wrap a runtest phase so its late warnings are reported against it. + + Draining in a ``finally``-style wrapper rather than in a ``trylast`` hook + makes sure a phase that raised is drained too; otherwise its warnings would + leak into the next phase and be reported there a second time. + """ + __tracebackhide__ = True + config = item.config + try: + result = yield + except BaseException: + if late := _drain_innermost(config): + if _error_later_report_mode(config) == "session": + config.stash[late_warning_state_key].collected.extend(late) + else: + # The phase already failed; pytest_runtest_makereport adds + # them to that failure instead of failing a second time. + item.stash.setdefault(_phase_late_warnings_key, {})[when] = late + raise + if late := _drain_innermost(config): + if _error_later_report_mode(config) == "session": + config.stash[late_warning_state_key].collected.extend(late) + else: + # The warning's own location is in the message; the frames between + # here and the emitting code are pytest's, so there is no traceback + # worth showing. + fail(_describe(late), pytrace=False) + return result + + +def _drain_innermost(config: Config) -> list[LateWarning]: + recordings = config.stash.get(_recordings_key, None) + if not recordings: + return [] + return _drain(config, recordings[-1]) + + +@pytest.hookimpl(wrapper=True, tryfirst=True) +def pytest_runtest_setup(item: Item) -> Generator[None, object, object]: + return (yield from _late_warnings_phase(item, "setup")) + + +@pytest.hookimpl(wrapper=True, tryfirst=True) +def pytest_runtest_call(item: Item) -> Generator[None, object, object]: + return (yield from _late_warnings_phase(item, "call")) + + +@pytest.hookimpl(wrapper=True, tryfirst=True) +def pytest_runtest_teardown(item: Item) -> Generator[None, object, object]: + return (yield from _late_warnings_phase(item, "teardown")) + + +@pytest.hookimpl(wrapper=True, tryfirst=True) +def pytest_runtest_makereport( + item: Item, call: CallInfo[None] +) -> Generator[None, TestReport, TestReport]: + report = yield + late = item.stash.get(_phase_late_warnings_key, {}).pop(call.when, None) + if late: + if report.failed: + _add_to_failure(report, _describe(late)) + else: + # Skipped or xfailed: there is no failure to add them to, and + # turning the outcome into one would defeat the marker. + item.config.stash[late_warning_state_key].collected.extend(late) + return report + + +def _add_to_failure(report: TestReport, text: str) -> None: + longrepr = report.longrepr + if isinstance(longrepr, ExceptionRepr): + longrepr.addsection(LATE_WARNING_ERRORS_STAT, text) + elif isinstance(longrepr, str): + report.longrepr = f"{longrepr}\n\n{text}" + else: + report.sections.append((LATE_WARNING_ERRORS_STAT, text)) + + def warning_record_to_str(warning_message: warnings.WarningMessage) -> str: """Convert a warnings.WarningMessage to a string.""" return warnings.formatwarning( @@ -89,19 +227,73 @@ def pytest_terminal_summary( terminalreporter: TerminalReporter, ) -> Generator[None]: config = terminalreporter.config - with catch_warnings_for_item( - config=config, ihook=config.hook, when="config", item=None - ): - return (yield) + try: + with catch_warnings_for_item( + config=config, ihook=config.hook, when="config", item=None + ): + return (yield) + finally: + if terminalreporter._session is not None: + _settle_late_warnings(terminalreporter._session) + _write_late_warnings_section(terminalreporter) @pytest.hookimpl(wrapper=True) def pytest_sessionfinish(session: Session) -> Generator[None]: config = session.config - with catch_warnings_for_item( - config=config, ihook=config.hook, when="config", item=None - ): - return (yield) + state = config.stash.setdefault(late_warning_state_key, LateWarningState()) + workeroutput = getattr(config, "workeroutput", None) + if workeroutput is not None: + # A pytest-xdist worker: its exit status and terminal go nowhere, so + # hand everything to the controller. xdist sends workeroutput once the + # inner sessionfinish hooks are done, so this has to happen first. + workeroutput[_XDIST_WORKEROUTPUT_KEY] = [ + dataclasses.asdict(late) for late in state.collected + ] + try: + # The context drains on exit, so it has to be closed before settling: + # otherwise warnings from other pytest_sessionfinish hooks are lost. + with catch_warnings_for_item( + config=config, ihook=config.hook, when="config", item=None + ): + return (yield) + finally: + _settle_late_warnings(session) + # From here on there is nothing left to report into, so warnings are + # shown the ordinary way instead of being collected and dropped. + state.closed = True + + +def _settle_late_warnings(session: Session) -> None: + """Count the late warnings no test phase could fail for, and fail the run. + + Under ``session`` that is every one of them; under ``test`` it is the ones + emitted outside a test phase, such as during collection, and the ones of a + test that was skipped. + """ + config = session.config + state = config.stash.get(late_warning_state_key, None) + if state is None or not state.collected: + return + terminalreporter: TerminalReporter | None = config.pluginmanager.getplugin( + "terminalreporter" + ) + if terminalreporter is not None: + counted = len(terminalreporter.stats.get(LATE_WARNING_ERRORS_STAT, [])) + if new := state.collected[counted:]: + terminalreporter._add_stats(LATE_WARNING_ERRORS_STAT, new) + if session.exitstatus == ExitCode.OK: + session.exitstatus = ExitCode.LATE_WARNING_ERROR + + +def _write_late_warnings_section(terminalreporter: TerminalReporter) -> None: + state = terminalreporter.config.stash.get(late_warning_state_key, None) + if state is None or not state.collected[state.reported :]: + return + terminalreporter.write_sep("=", LATE_WARNING_ERRORS_STAT, red=True, bold=True) + for late in state.collected[state.reported :]: + terminalreporter.write_line(late.format(with_nodeid=True)) + state.reported = len(state.collected) @pytest.hookimpl(wrapper=True) @@ -114,9 +306,50 @@ def pytest_load_initial_conftests( return (yield) +LATE_ERROR_REPORT_MODES = ("test", "session") + + +def _error_later_report_mode(config: Config) -> str: + mode: str = config.getini("error_later_report") + if mode not in LATE_ERROR_REPORT_MODES: + raise UsageError( + f"Invalid error_later_report value {mode!r}, " + f"expected one of {', '.join(LATE_ERROR_REPORT_MODES)}" + ) + return mode + + def pytest_configure(config: Config) -> None: + # Fail early on a bad value rather than once a warning matches the filter. + _error_later_report_mode(config) + config.pluginmanager.register( + _XdistLateWarningsCollector(config), "error-later-xdist-collector" + ) config.addinivalue_line( "markers", "filterwarnings(warning): add a warning filter to the given test. " "see https://docs.pytest.org/en/stable/how-to/capture-warnings.html#pytest-mark-filterwarnings ", ) + + +_XDIST_WORKEROUTPUT_KEY = "pytest_error_later" + + +class _XdistLateWarningsCollector: + """Merge the late warnings of pytest-xdist workers into the controller's.""" + + def __init__(self, config: Config) -> None: + self.config = config + + # Optional: the hookspec only exists with xdist installed. Checking for + # the hook on config.hook is not enough, as pytest-cov implements it as + # optional too, which creates the hook caller without a spec. + @pytest.hookimpl(optionalhook=True) + def pytest_testnodedown(self, node: Any, error: object) -> None: + workeroutput = getattr(node, "workeroutput", None) or {} + state = self.config.stash.setdefault(late_warning_state_key, LateWarningState()) + for data in workeroutput.get(_XDIST_WORKEROUTPUT_KEY, []): + late = LateWarning(**data) + # Every worker collects and configures; report those warnings once. + if late.nodeid or late not in state.collected: + state.collected.append(late) diff --git a/testing/test_config.py b/testing/test_config.py index 282e66409f7..a4c5be3e0ff 100644 --- a/testing/test_config.py +++ b/testing/test_config.py @@ -3168,6 +3168,22 @@ def test_strtobool() -> None: ), ("error:some\\msg:::", True, ("error", "some\\\\msg", Warning, "", 0)), ("error:::mod\\foo:", True, ("error", "", Warning, "mod\\\\foo\\Z", 0)), + ("error_later", False, ("error_later", "", Warning, "", 0)), + ( + "error_later::DeprecationWarning", + False, + ("error_later", "", DeprecationWarning, "", 0), + ), + ( + "error_later:some msg:DeprecationWarning", + False, + ("error_later", "some msg", DeprecationWarning, "", 0), + ), + ( + "error_later::DeprecationWarning:pkg.mod:42", + True, + ("error_later", "", DeprecationWarning, "pkg\\.mod\\Z", 42), + ), ], ) def test_parse_warning_filter( diff --git a/testing/test_warnings.py b/testing/test_warnings.py index 0344cb453d8..7dc78057027 100644 --- a/testing/test_warnings.py +++ b/testing/test_warnings.py @@ -1170,3 +1170,722 @@ def test_pythonwarnings_not_duplicated(pytester: Pytester) -> None: warnings_list = config.known_args_namespace.pythonwarnings assert warnings_list is not None assert warnings_list == ["error"] + + +class TestErrorLaterWarnings: + """The ``error_later`` filter action.""" + + def _emitting_test(self, pytester: Pytester) -> None: + # In-process pytester runs inherit the filters of the outer session, + # which runs under ``filterwarnings = error``. + pytester.makeini( + """ + [pytest] + filterwarnings = + always::UserWarning + """ + ) + pytester.makepyfile( + """ + import warnings + + def test_emits(): + warnings.warn("too late", UserWarning) + print("body ran to completion") + + def test_clean(): + pass + """ + ) + + def test_test_mode_fails_only_the_emitting_test(self, pytester: Pytester) -> None: + self._emitting_test(pytester) + + result = pytester.runpytest("-W", "error_later::UserWarning", "-s") + + result.assert_outcomes(passed=1, failed=1, warnings=1) + result.stdout.fnmatch_lines( + [ + # The test body is not interrupted the way an "error" filter + # would interrupt it. + "*body ran to completion*", + "*1 warning matched an 'error_later' filter:", + "*test_test_mode_fails_only_the_emitting_test.py:*: UserWarning: too late", + ] + ) + assert result.ret == ExitCode.TESTS_FAILED + + def test_session_mode_reports_at_the_end(self, pytester: Pytester) -> None: + self._emitting_test(pytester) + + result = pytester.runpytest( + "-W", "error_later::UserWarning", "-o", "error_later_report=session" + ) + + result.assert_outcomes(passed=2, warnings=1) + result.stdout.fnmatch_lines( + [ + "*= late warning errors =*", + "*: UserWarning: too late", + ] + ) + assert result.ret == ExitCode.LATE_WARNING_ERROR + + def test_not_matched_by_a_different_category(self, pytester: Pytester) -> None: + self._emitting_test(pytester) + + result = pytester.runpytest("-W", "error_later::DeprecationWarning") + + result.assert_outcomes(passed=2, warnings=1) + assert result.ret == ExitCode.OK + + def test_higher_precedence_filter_wins(self, pytester: Pytester) -> None: + """A ``filterwarnings`` mark is applied last, so it beats the ini.""" + pytester.makeini( + """ + [pytest] + filterwarnings = + error_later::UserWarning + """ + ) + pytester.makepyfile( + """ + import warnings + import pytest + + def test_errors_late(): + warnings.warn("boom", UserWarning) + + @pytest.mark.filterwarnings("ignore::UserWarning") + def test_overridden(): + warnings.warn("boom", UserWarning) + """ + ) + + result = pytester.runpytest() + + result.assert_outcomes(passed=1, failed=1, warnings=1) + + def test_mark_can_apply_to_a_single_test(self, pytester: Pytester) -> None: + pytester.makeini( + """ + [pytest] + filterwarnings = + always::UserWarning + """ + ) + pytester.makepyfile( + """ + import warnings + import pytest + + @pytest.mark.filterwarnings("error_later::UserWarning") + def test_errors_late(): + warnings.warn("boom", UserWarning) + + def test_unaffected(): + warnings.warn("boom", UserWarning) + """ + ) + + result = pytester.runpytest() + + result.assert_outcomes(passed=1, failed=1, warnings=2) + + def test_collection_warnings_are_reported_at_the_end( + self, pytester: Pytester + ) -> None: + """Collection has no test phase to fail, so it reports at the end.""" + pytester.makeini( + """ + [pytest] + filterwarnings = + always::UserWarning + """ + ) + pytester.makepyfile( + """ + import warnings + warnings.warn("at import time", UserWarning) + + def test_pass(): + pass + """ + ) + + result = pytester.runpytest("-W", "error_later::UserWarning") + + result.assert_outcomes(passed=1, warnings=1) + result.stdout.fnmatch_lines( + [ + "*= late warning errors =*", + "*: UserWarning: at import time", + "*= 1 passed, 1 warning, 1 late warning error in *", + ] + ) + assert result.ret == ExitCode.LATE_WARNING_ERROR + + def test_module_field_matches_the_emitting_module(self, pytester: Pytester) -> None: + pytester.makepyfile( + pkg_a="import warnings\ndef f(): warnings.warn('from a', UserWarning)", + pkg_b="import warnings\ndef f(): warnings.warn('from b', UserWarning)", + test_mod="import pkg_a, pkg_b\ndef test_it():\n pkg_a.f()\n pkg_b.f()", + ) + result = pytester.runpytest( + "-W", "always::UserWarning", "-W", "error_later::UserWarning:pkg_a" + ) + + assert result.ret == ExitCode.TESTS_FAILED + result.stdout.fnmatch_lines( + [ + "*1 warning matched an 'error_later' filter:", + "*pkg_a.py:2: UserWarning: from a", + "*= warnings summary =*", + "*pkg_b.py:2: UserWarning: from b", + ] + ) + + def test_line_field_matches_the_emitting_line(self, pytester: Pytester) -> None: + pytester.makepyfile( + """ + import warnings + def test_it(): + warnings.warn("line 3", UserWarning) + warnings.warn("line 4", UserWarning) + """ + ) + result = pytester.runpytest( + "-W", "always::UserWarning", "-W", "error_later::UserWarning::4" + ) + + assert result.ret == ExitCode.TESTS_FAILED + result.stdout.fnmatch_lines( + ["*1 warning matched an 'error_later' filter:", "*:4: UserWarning: line 4"] + ) + + def test_module_of_code_without_a_name_is_string(self, pytester: Pytester) -> None: + pytester.makepyfile( + """ + import warnings + def test_it(): + exec("warnings.warn('nameless', UserWarning)", {"warnings": warnings}) + """ + ) + result = pytester.runpytest("-W", "error_later::UserWarning:") + + assert result.ret == ExitCode.TESTS_FAILED + result.stdout.fnmatch_lines(["*UserWarning: nameless"]) + + def test_filter_installed_by_the_test_takes_precedence( + self, pytester: Pytester + ) -> None: + pytester.makepyfile( + """ + import warnings + def test_it(): + warnings.simplefilter("always") + warnings.warn("shown, not late", UserWarning) + """ + ) + result = pytester.runpytest("-W", "error_later::UserWarning") + + assert result.ret == ExitCode.OK + result.stdout.fnmatch_lines(["*1 passed, 1 warning*"]) + + def test_filter_installed_by_a_fixture_takes_precedence( + self, pytester: Pytester + ) -> None: + pytester.makeconftest( + """ + import warnings, pytest + @pytest.fixture(autouse=True) + def quiet_noisy(): + warnings.filterwarnings("ignore", module="noisy") + """ + ) + pytester.makepyfile( + noisy="import warnings\ndef f(): warnings.warn('noise', UserWarning)", + test_mod="import noisy\ndef test_it():\n noisy.f()", + ) + result = pytester.runpytest("-W", "error_later::UserWarning") + + assert result.ret == ExitCode.OK + result.stdout.fnmatch_lines(["*1 passed in*"]) + + def test_pytest_warns_takes_the_warning(self, pytester: Pytester) -> None: + pytester.makepyfile( + """ + import warnings, pytest + def test_it(): + with pytest.warns(UserWarning): + warnings.warn("expected", UserWarning) + warnings.warn("other", DeprecationWarning) + """ + ) + result = pytester.runpytest( + "-W", "error_later::UserWarning", "-W", "always::DeprecationWarning" + ) + + assert result.ret == ExitCode.OK + result.stdout.fnmatch_lines(["*1 passed, 1 warning*"]) + + def test_showwarning_hook_that_forwards_keeps_the_verdict( + self, pytester: Pytester + ) -> None: + """#15139: numpy.testing.suppress_warnings forwards like this.""" + pytester.makepyfile( + """ + import warnings + def test_it(): + orig = warnings.showwarning + warnings.showwarning = lambda *a, **k: orig(*a, **k) + try: + warnings.warn("forwarded", UserWarning) + finally: + warnings.showwarning = orig + """ + ) + result = pytester.runpytest("-W", "error_later::UserWarning") + + result.assert_outcomes(failed=1, warnings=1) + result.stdout.fnmatch_lines( + ["*1 warning matched an 'error_later' filter:", "*UserWarning: forwarded"] + ) + + def test_showwarning_hook_that_discards_leaves_no_verdict( + self, pytester: Pytester + ) -> None: + """A verdict outliving its warning would tag the next one with the same text.""" + pytester.makepyfile( + """ + import warnings + def test_it(): + orig = warnings.showwarning + warnings.showwarning = lambda *a, **k: None + try: + warnings.warn("same", UserWarning) + finally: + warnings.showwarning = orig + with warnings.catch_warnings(): + warnings.simplefilter("always") + warnings.warn("same", RuntimeWarning) + """ + ) + result = pytester.runpytest("-W", "error_later::UserWarning") + + result.assert_outcomes(passed=1, warnings=1) + result.stdout.fnmatch_lines(["*RuntimeWarning: same"]) + + def test_showwarning_hook_that_warns_before_forwarding( + self, pytester: Pytester + ) -> None: + pytester.makepyfile( + """ + import warnings + def test_it(): + orig = warnings.showwarning + def hook(*args, **kwargs): + warnings.showwarning = orig + warnings.warn("from the hook", RuntimeWarning) + orig(*args, **kwargs) + warnings.showwarning = hook + try: + warnings.warn("outer", UserWarning) + finally: + warnings.showwarning = orig + """ + ) + result = pytester.runpytest( + "-W", "error_later::UserWarning", "-W", "always::RuntimeWarning" + ) + + result.assert_outcomes(failed=1, warnings=2) + result.stdout.fnmatch_lines( + ["*1 warning matched an 'error_later' filter:", "*UserWarning: outer"] + ) + + def test_showwarnmsg_is_restored_after_the_session( + self, pytester: Pytester + ) -> None: + pytester.makepyfile("def test_pass(): pass") + before = warnings._showwarnmsg # type: ignore[attr-defined] + + pytester.runpytest_inprocess("-W", "error_later::UserWarning") + + assert warnings._showwarnmsg is before # type: ignore[attr-defined] + + def test_invalid_report_mode_is_a_usage_error(self, pytester: Pytester) -> None: + pytester.makepyfile("def test_pass(): pass") + + result = pytester.runpytest("-o", "error_later_report=nonsense") + + assert result.ret == ExitCode.USAGE_ERROR + result.stderr.fnmatch_lines(["*Invalid error_later_report value 'nonsense'*"]) + + def test_message_matches_case_insensitively(self, pytester: Pytester) -> None: + """The message field is matched with re.I, as by the warnings module.""" + self._emitting_test(pytester) + + result = pytester.runpytest("-W", "error_later:TOO LATE:UserWarning") + + result.assert_outcomes(passed=1, failed=1, warnings=1) + + def test_cmdline_message_is_escaped(self, pytester: Pytester) -> None: + """-W takes a literal message, so regex characters match themselves.""" + pytester.makeini( + """ + [pytest] + filterwarnings = + always::UserWarning + """ + ) + pytester.makepyfile( + """ + import warnings + + def test_literal(): + warnings.warn("value (x+y) is deprecated", UserWarning) + + def test_not_a_regex_match(): + warnings.warn("value xxy is deprecated", UserWarning) + """ + ) + + result = pytester.runpytest("-W", "error_later:value (x+y):UserWarning") + + result.assert_outcomes(passed=1, failed=1, warnings=2) + result.stdout.fnmatch_lines(["FAILED *::test_literal - *"]) + + def test_later_filter_restricted_to_the_module_wins( + self, pytester: Pytester + ) -> None: + """Module-restricted filters take part in precedence like the others.""" + pytester.makeini( + """ + [pytest] + filterwarnings = + error_later::UserWarning + default::UserWarning:test_later_filter_restricted_to_the_module_wins + """ + ) + pytester.makepyfile( + """ + import warnings + + def test_emits(): + warnings.warn("boom", UserWarning) + """ + ) + + result = pytester.runpytest() + + result.assert_outcomes(passed=1, warnings=1) + + def test_later_filter_restricted_to_another_line_does_not_win( + self, pytester: Pytester + ) -> None: + pytester.makeini( + """ + [pytest] + filterwarnings = + error_later::UserWarning + default::UserWarning::999 + """ + ) + pytester.makepyfile( + """ + import warnings + + def test_emits(): + warnings.warn("boom", UserWarning) + """ + ) + + result = pytester.runpytest() + + result.assert_outcomes(failed=1, warnings=1) + + def test_failing_test_is_reported_once(self, pytester: Pytester) -> None: + """The warnings go into the failure, not into a second teardown error.""" + self._emitting_test(pytester) + pytester.makepyfile( + """ + import warnings + + def test_emits_and_fails(): + warnings.warn("too late", UserWarning) + assert False, "the test's own failure" + """ + ) + + result = pytester.runpytest("-W", "error_later::UserWarning") + + result.assert_outcomes(failed=1, errors=0, warnings=1) + result.stdout.fnmatch_lines( + [ + "E AssertionError: the test's own failure", + "*- late warning errors -*", + "1 warning matched an 'error_later' filter:", + "*: UserWarning: too late", + ] + ) + + def test_failing_setup_is_reported_once(self, pytester: Pytester) -> None: + pytester.makeini( + """ + [pytest] + filterwarnings = + always::UserWarning + """ + ) + pytester.makepyfile( + """ + import warnings + import pytest + + @pytest.fixture + def broken(): + warnings.warn("from the fixture", UserWarning) + raise RuntimeError("setup failed") + + def test_uses_broken(broken): + pass + """ + ) + + result = pytester.runpytest("-W", "error_later::UserWarning") + + result.assert_outcomes(errors=1, warnings=1) + result.stdout.fnmatch_lines( + ["E RuntimeError: setup failed", "*: UserWarning: from the fixture"] + ) + result.stdout.no_fnmatch_line("*= late warning errors =*") + + def test_each_phase_reports_its_own_warnings(self, pytester: Pytester) -> None: + pytester.makeini( + """ + [pytest] + filterwarnings = + always::UserWarning + """ + ) + pytester.makepyfile( + """ + import warnings + import pytest + + @pytest.fixture + def warns_in_teardown(): + yield + warnings.warn("from teardown", UserWarning) + + def test_fails(warns_in_teardown): + warnings.warn("from call", UserWarning) + assert False + """ + ) + + result = pytester.runpytest("-W", "error_later::UserWarning") + + result.assert_outcomes(failed=1, errors=1, warnings=2) + result.stdout.fnmatch_lines( + [ + "*_ ERROR at teardown of test_fails _*", + "*: UserWarning: from teardown", + "*_ test_fails _*", + "*- late warning errors -*", + "*: UserWarning: from call", + ] + ) + + def test_skipped_test_reports_at_the_end(self, pytester: Pytester) -> None: + """A skip is not a failure to add to, and must not swallow the warning.""" + pytester.makeini( + """ + [pytest] + filterwarnings = + always::UserWarning + """ + ) + pytester.makepyfile( + """ + import warnings + import pytest + + def test_skips(): + warnings.warn("before the skip", UserWarning) + pytest.skip("not today") + """ + ) + + result = pytester.runpytest("-W", "error_later::UserWarning") + + result.assert_outcomes(skipped=1, warnings=1) + result.stdout.fnmatch_lines( + ["*= late warning errors =*", "*(*::test_skips): UserWarning: before*"] + ) + assert result.ret == ExitCode.LATE_WARNING_ERROR + + def test_session_mode_summary_line_counts_late_errors( + self, pytester: Pytester + ) -> None: + self._emitting_test(pytester) + + result = pytester.runpytest( + "-W", "error_later::UserWarning", "-o", "error_later_report=session" + ) + + result.stdout.fnmatch_lines( + ["*= 2 passed, 1 warning, 1 late warning error in *"] + ) + + def test_configure_time_warning_reports_at_the_end( + self, pytester: Pytester + ) -> None: + pytester.makeconftest( + """ + import warnings + + def pytest_configure(config): + warnings.warn("from pytest_configure", UserWarning) + """ + ) + pytester.makepyfile("def test_pass(): pass") + + result = pytester.runpytest_subprocess("-W", "error_later::UserWarning") + + result.stdout.fnmatch_lines( + ["*= late warning errors =*", "*conftest.py:4: UserWarning: from*"] + ) + assert result.ret == ExitCode.LATE_WARNING_ERROR + + def test_issue_config_time_warning_reports_at_the_end( + self, pytester: Pytester + ) -> None: + pytester.makeconftest( + """ + def pytest_configure(config): + config.issue_config_time_warning( + UserWarning("via issue_config_time_warning"), stacklevel=2 + ) + """ + ) + pytester.makepyfile("def test_pass(): pass") + + result = pytester.runpytest_subprocess("-W", "error_later::UserWarning") + + result.stdout.fnmatch_lines( + ["*= late warning errors =*", "*: UserWarning: via issue_config*"] + ) + assert result.ret == ExitCode.LATE_WARNING_ERROR + + def test_plugin_import_warning_reports_at_the_end(self, pytester: Pytester) -> None: + pytester.makepyfile( + warning_plugin=""" + import warnings + warnings.warn("while importing the plugin", UserWarning) + """ + ) + pytester.makepyfile("def test_pass(): pass") + pytester.syspathinsert() + + result = pytester.runpytest_subprocess( + "-W", "error_later::UserWarning", "-p", "warning_plugin" + ) + + result.stdout.fnmatch_lines( + ["*= late warning errors =*", "*: UserWarning: while importing*"] + ) + assert result.ret == ExitCode.LATE_WARNING_ERROR + + def test_sessionfinish_warning_reports_at_the_end(self, pytester: Pytester) -> None: + pytester.makeconftest( + """ + import warnings + + def pytest_sessionfinish(session): + warnings.warn("from pytest_sessionfinish", UserWarning) + """ + ) + pytester.makepyfile("def test_pass(): pass") + + result = pytester.runpytest_subprocess("-W", "error_later::UserWarning") + + result.stdout.fnmatch_lines( + ["*= late warning errors =*", "*: UserWarning: from pytest_sessionfin*"] + ) + assert result.ret == ExitCode.LATE_WARNING_ERROR + + def test_unconfigure_warning_is_still_shown(self, pytester: Pytester) -> None: + """Too late to report, so it is shown as usual rather than dropped.""" + pytester.makeconftest( + """ + import warnings + + def pytest_unconfigure(config): + warnings.warn("from pytest_unconfigure", UserWarning) + """ + ) + pytester.makepyfile("def test_pass(): pass") + + result = pytester.runpytest_subprocess("-W", "error_later::UserWarning") + + result.stderr.fnmatch_lines(["*: UserWarning: from pytest_unconfigure"]) + assert result.ret == ExitCode.OK + + def test_optional_xdist_hook_without_xdist(self, pytester: Pytester) -> None: + # pytest-cov implements pytest_testnodedown as an optional hook, which + # creates the hook caller without a spec when xdist is not installed. + pytester.makeconftest( + """ + import pytest + + @pytest.hookimpl(optionalhook=True) + def pytest_testnodedown(node, error): + pass + """ + ) + pytester.makepyfile("def test_pass(): pass") + + result = pytester.runpytest("-p", "no:xdist") + + result.assert_outcomes(passed=1) + + @pytest.mark.parametrize("mode", ["test", "session"]) + def test_xdist_reports_on_the_controller( + self, pytester: Pytester, monkeypatch: pytest.MonkeyPatch, mode: str + ) -> None: + pytest.importorskip("xdist") + monkeypatch.delenv("PYTEST_DISABLE_PLUGIN_AUTOLOAD", raising=False) + pytester.makeini( + f""" + [pytest] + error_later_report = {mode} + filterwarnings = + always::UserWarning + """ + ) + pytester.makepyfile( + test_one=""" + import warnings + warnings.warn("at import time", UserWarning) + + def test_emits(): + warnings.warn("in the test", UserWarning) + """, + test_two="def test_pass(): pass", + ) + + result = pytester.runpytest_subprocess("-n2", "-W", "error_later::UserWarning") + + # Both workers collect both modules; the import-time one is reported once. + late_section = result.stdout.str().partition("= late warning errors =")[2] + assert late_section.count("UserWarning: at import time") == 1 + if mode == "session": + result.stdout.fnmatch_lines( + ["*(test_one.py::test_emits): UserWarning: in the test"] + ) + assert result.ret == ExitCode.LATE_WARNING_ERROR + else: + result.stdout.fnmatch_lines(["FAILED test_one.py::test_emits - *"]) + assert result.ret == ExitCode.TESTS_FAILED