Deprecate registering command line options with required=True

changelog/3042.deprecation.rst

@@ -0,0 +1,4 @@
+Registering command line options with ``required=True`` is now deprecated and emits a :class:`pytest.PytestRemovedIn11Warning`.
+
+Required options make argparse reject every invocation without them, including ``pytest --fixtures`` and ``pytest --markers``.
+See :ref:`required command line options` for how to check for a missing value instead.

doc/en/deprecations.rst

@@ -15,6 +15,25 @@ Below is a complete list of all pytest features which are considered deprecated.
 :class:`~pytest.PytestWarning` or subclasses, which can be filtered using :ref:`standard warning filters <warnings>`.
 
 
+.. _required-command-line-options:
+
+Registering command line options with ``required=True``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. deprecated:: 9.2
+
+Passing ``required=True`` to :meth:`Parser.addoption <pytest.Parser.addoption>` or
+:meth:`OptionGroup.addoption <pytest.OptionGroup.addoption>` is deprecated and
+will be removed in pytest 11.
+
+pytest was never designed for required options; they only worked because argparse
+supports them. argparse rejects every invocation that misses them, so
+``pytest --fixtures`` and ``pytest --markers`` fail too (:issue:`3042`).
+
+Register the option without ``required`` and check for the missing value where
+it is needed instead, see :ref:`required command line options`.
+
+
 .. _callspec2-renamed:
 
 ``_pytest.python.CallSpec2`` renamed to ``CallSpec``

doc/en/example/simple.rst

@@ -204,6 +204,96 @@ This completes the basic pattern.  However, one often rather wants to
 process command line options outside of the test and rather pass in
 different or more complex objects.
 
+.. _`required command line options`:
+
+Require a command line option
+-----------------------------
+
+.. regendoc:wipe
+
+Some tests can't run without a value from the command line, for example the
+address of a server to test against. Don't register such an option with
+``required=True``: argparse then rejects every invocation without it,
+including informational ones like ``pytest --fixtures`` and ``pytest --markers``.
+``required=True`` is :ref:`deprecated <required-command-line-options>`.
+
+Instead, register the option without ``required`` and check for the missing
+value where it is needed. There are two ways to do that.
+
+Fail the tests that need the value
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Check in the fixture that provides the value:
+
+.. code-block:: python
+
+    # content of conftest.py
+    import pytest
+
+
+    def pytest_addoption(parser):
+        parser.addoption("--server", help="address of the server to test against")
+
+
+    @pytest.fixture(scope="session")
+    def server(request):
+        address = request.config.getoption("server")
+        if address is None:
+            pytest.fail(
+                "--server is required to run tests using the server fixture", pytrace=False
+            )
+        return address
+
+Only the tests that use ``server`` fail, each with a one-line error; all other
+tests run as usual. ``pytest --fixtures``, ``pytest --collect-only`` and
+selecting only tests that don't use ``server`` (for example with ``-k``) keep
+working. This also works with :pypi:`pytest-xdist`.
+
+Abort the whole run with a usage error
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To refuse to start the test run, like ``required=True`` did, raise
+:class:`pytest.UsageError` from :hook:`pytest_runtestloop`:
+
+.. code-block:: python
+
+    # content of conftest.py
+    import pytest
+
+
+    def pytest_addoption(parser):
+        parser.addoption("--server", help="address of the server to test against")
+
+
+    @pytest.hookimpl(tryfirst=True)
+    def pytest_runtestloop(session):
+        config = session.config
+        if not config.option.collectonly and config.getoption("server") is None:
+            raise pytest.UsageError("--server is required")
+
+
+    @pytest.fixture(scope="session")
+    def server(request):
+        return request.config.getoption("server")
+
+pytest then exits with code ``4`` (:class:`~pytest.ExitCode` ``USAGE_ERROR``)
+and prints the message before any test runs, also with :pypi:`pytest-xdist`.
+
+:hook:`pytest_runtestloop` is the first hook that only runs when tests are
+about to run. Checking earlier breaks informational options:
+
+* :hook:`pytest_configure` also runs for ``--help``, ``--markers``,
+  ``--fixtures``, ``--collect-only`` and ``--cache-show``.
+* :hook:`pytest_cmdline_main` also runs for ``--help``, ``--fixtures``,
+  ``--collect-only`` and ``--cache-show``.
+* :hook:`pytest_sessionstart` and the collection hooks also run for
+  ``--fixtures``, ``--collect-only`` and ``--cache-show``.
+
+The check applies to every run, including runs that only select tests which
+don't need the value. To fail only the tests that need it, use the fixture
+check above.
+
+
 Dynamically adding command line options
 --------------------------------------------------------------
 

doc/en/reference/reference.rst

@@ -1317,6 +1317,9 @@ Custom warnings generated in some situations such as improper usage or deprecate
 .. autoclass:: pytest.PytestRemovedIn10Warning
   :show-inheritance:
 
+.. autoclass:: pytest.PytestRemovedIn11Warning
+  :show-inheritance:
+
 .. autoclass:: pytest.PytestUnknownMarkWarning
    :show-inheritance:
 

src/_pytest/config/argparsing.py

@@ -19,11 +19,13 @@
 from typing import TYPE_CHECKING
 from typing import TypeAlias
 from typing import Union
+import warnings
 
 from .exceptions import UsageError
 import _pytest._io
 from _pytest.compat import NOTSET
 from _pytest.deprecated import check_ispytest
+from _pytest.deprecated import REQUIRED_OPTION
 
 
 if TYPE_CHECKING:
@@ -203,7 +205,7 @@ def addoption(self, *opts: str, **attrs: Any) -> None:
         by passing a ``dest`` attribute, for example
         ``addoption("--long", dest="NAME", ...)``.
         """
-        self._anonymous.addoption(*opts, **attrs)
+        self._anonymous._addoption_checked(opts, attrs)
 
     def parse(
         self,
@@ -451,6 +453,9 @@ def addoption(self, *opts: str, **attrs: Any) -> None:
             Same attributes as the argparse library's :meth:`add_argument()
             <argparse.ArgumentParser.add_argument>` function accepts.
         """
+        self._addoption_checked(opts, attrs)
+
+    def _addoption_checked(self, opts: tuple[str, ...], attrs: dict[str, Any]) -> None:
         conflict = set(opts).intersection(
             name for opt in self.options for name in opt.names()
         )
@@ -483,6 +488,10 @@ def _addoption_inner(
                 if len(opt) >= 2 and opt[0] == "-" and opt[1].islower():
                     raise ValueError("lowercase short options are reserved")
 
+        if attrs.get("required"):
+            # Points at the caller of Parser.addoption()/OptionGroup.addoption().
+            warnings.warn(REQUIRED_OPTION.format(names=", ".join(opts)), stacklevel=4)
+
         action = self._arggroup.add_argument(*opts, **attrs)
         option = Argument(action)
         self.options.append(option)

src/_pytest/deprecated.py

@@ -17,6 +17,7 @@
 
 from _pytest.warning_types import PytestDeprecationWarning
 from _pytest.warning_types import PytestRemovedIn10Warning
+from _pytest.warning_types import PytestRemovedIn11Warning
 from _pytest.warning_types import UnformattedWarning
 
 
@@ -106,6 +107,14 @@
     "See https://docs.pytest.org/en/stable/deprecations.html#the-pastebin-option",
 )
 
+REQUIRED_OPTION = UnformattedWarning(
+    PytestRemovedIn11Warning,
+    "Registering command line options with required=True is deprecated: {names}\n"
+    "Required options break informational modes like --fixtures and --markers.\n"
+    "Check for the missing value where it is used and raise pytest.UsageError or call pytest.fail() instead.\n"
+    "See https://docs.pytest.org/en/stable/deprecations.html#required-command-line-options",
+)
+
 INI_STRING_TYPE_NON_STR_VALUE = PytestRemovedIn10Warning(
     "Passing a value that is not a string to a 'string'-typed ini option is deprecated.\n"
     "In a future version this will raise a TypeError, matching the behavior of the "

src/_pytest/warning_types.py

@@ -56,6 +56,12 @@ class PytestRemovedIn10Warning(PytestDeprecationWarning):
     __module__ = "pytest"
 
 
+class PytestRemovedIn11Warning(PytestDeprecationWarning):
+    """Warning class for features that will be removed in pytest 11."""
+
+    __module__ = "pytest"
+
+
 @final
 class PytestExperimentalApiWarning(PytestWarning, FutureWarning):
     """Warning category used to denote experiments in pytest.

src/pytest/__init__.py

@@ -88,6 +88,7 @@
 from _pytest.warning_types import PytestExperimentalApiWarning
 from _pytest.warning_types import PytestFDWarning
 from _pytest.warning_types import PytestRemovedIn10Warning
+from _pytest.warning_types import PytestRemovedIn11Warning
 from _pytest.warning_types import PytestReturnNotNoneWarning
 from _pytest.warning_types import PytestUnhandledThreadExceptionWarning
 from _pytest.warning_types import PytestUnknownMarkWarning
@@ -141,6 +142,7 @@
     "PytestFDWarning",
     "PytestPluginManager",
     "PytestRemovedIn10Warning",
+    "PytestRemovedIn11Warning",
     "PytestReturnNotNoneWarning",
     "PytestUnhandledThreadExceptionWarning",
     "PytestUnknownMarkWarning",

testing/deprecated_test.py

@@ -401,3 +401,36 @@ def raise_and_count() -> int:
 
     # The traceback of a shared instance would grow with every raise.
     assert second <= first
+
+
+@pytest.mark.parametrize("use_group", [False, True])
+def test_required_option_deprecated(pytester: Pytester, use_group: bool) -> None:
+    target = 'parser.getgroup("custom")' if use_group else "parser"
+    pytester.makeconftest(
+        f"""
+        def pytest_addoption(parser):
+            {target}.addoption("--server", required=True)
+        """
+    )
+    pytester.makepyfile("def test_it(): pass")
+    result = pytester.runpytest("--server", "x")
+    result.assert_outcomes(passed=1, warnings=1)
+    result.stdout.fnmatch_lines(
+        [
+            "conftest.py:2",
+            "*PytestRemovedIn11Warning: Registering command line options with "
+            "required=True is deprecated: --server",
+        ]
+    )
+
+
+def test_required_option_not_required_no_warning(pytester: Pytester) -> None:
+    pytester.makeconftest(
+        """
+        def pytest_addoption(parser):
+            parser.addoption("--server", required=False)
+        """
+    )
+    pytester.makepyfile("def test_it(): pass")
+    result = pytester.runpytest("-Werror::pytest.PytestRemovedIn11Warning")
+    result.assert_outcomes(passed=1)

testing/test_conftest.py

@@ -1069,6 +1069,7 @@ def test_config():
     assert result.ret == ExitCode.OK
 
 
+@pytest.mark.filterwarnings("ignore::pytest.PytestRemovedIn11Warning")
 def test_required_option_help(pytester: Pytester) -> None:
     pytester.makeconftest("assert 0")
     x = pytester.mkdir("x")