Proposal: marker-driven loop/rerun protocol (deep hypothesis integration, pytest-repeat, retries) by RonnyPfannschmidt · Pull Request #14779 · pytest-dev/pytest

RonnyPfannschmidt · 2026-07-24T21:49:42Z

What

A design document only — no implementation — proposing a first-class test-loop protocol: a marker-carried loop controller (@pytest.mark.loop(controller)) drives repeated executions of one test item with

fresh function-scoped fixtures per iteration (setup/teardown cycled through the normal SetupState/FixtureDef paths — no new reset API),
intermediate teardown of only the item frame (nextitem is item sentinel + a new SetupState.teardown_item_frame), with the final teardown honoring the real nextitem,
per-iteration TestReports carrying a structured loop key (TestReport.loop), usable in junitxml (test_foo[param][key] testcases via (nodeid, workernode, loop_key) keying), the terminal, and machine-readable outputs,
deferrable reporting (report="live" | "defer" | "discard") so an engine like hypothesis can run examples silently and report only the shrunk failure.

Why

Anchored to the long-standing threads:

Deep integration between Hypothesis and py.test is currently impossible #916 — deep hypothesis integration is currently impossible (fixtures set up once for all examples)
RFC. Support for resetting function-scoped fixtures #12596 / Draft. Support for resetting function-scoped fixtures #12597 — RFC for resetting function-scoped fixtures; this proposal builds on its mechanics but replaces the "report the item as a single unit" stance with per-iteration keyed reports
Allow hooks to retry a single test case multiple times with fresh fixtures #12939 — retry a test with fresh fixtures (growing timeouts)
Finer grain fixture scope than function #9699 — finer-grain fixture scope (related, deferred as a follow-up item scope)
pytest function scoped fixtures should run once per example, not once per test HypothesisWorks/hypothesis#377

The document includes controller sketches for hypothesis (generate/shrink/minimal-replay), pytest-repeat, and #12939-style retries, the downstream compatibility contract for report consumers, and a list of open questions.

cc @Zac-HD — would value your review of the controller API and the hypothesis sketch in particular: does run_iteration(key, funcargs=..., report=...) plus provides= give the engine everything it needs (generate + shrink + replay-minimal-with-pdb)?

🤖 Generated with Claude Code

A design document (no implementation) for a first-class test-loop protocol: a marker-carried loop controller drives repeated executions of one item with fresh function-scoped fixtures per iteration, per-iteration TestReports carrying structured loop keys (junitxml/xunit-usable), and deferrable reporting for deep hypothesis integration. Anchored to pytest-dev#916, pytest-dev#12596/pytest-dev#12597, pytest-dev#12939, pytest-dev#9699 and HypothesisWorks/hypothesis#377. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

nicoddemus

Interesting proposal, didn't read everything but did note some comments along the way.

nicoddemus · 2026-10-17T11:49:33Z

+`LoopContext`, `IterationResult` and the `LoopController` typing protocol are
+exported as `pytest.LoopContext` etc.; the implementation lives in a new


All these new classes are somewhat advanced usage, perhaps we should use a package to hold them, perhaps pytest.loop (pytest.loop.LoopContext)?

nicoddemus · 2026-10-17T11:51:59Z

+    item: Item  # the item being looped
+    iteration_count: int  # iterations executed so far
+
+    def run_iteration(


nicoddemus · 2026-10-17T11:52:03Z

+        report: Literal["live", "defer", "discard"] = "live",
+    ) -> IterationResult: ...
+
+    def emit(self, result: IterationResult) -> None: ...


nicoddemus · 2026-10-17T11:54:48Z

+    when_failed: Literal["setup", "call", "teardown"] | None
+    duration: float
+
+    def emit(self) -> None: ...  # convenience for LoopContext.emit


This means IterationResult will have a an internal reference back to LoopContext?

I would remove this here from now, because it is easier for the controller to just call context.emit(result).

nicoddemus · 2026-10-17T11:54:59Z

+
+```python
+class LoopController(Protocol):
+    def __call__(self, loop: LoopContext) -> None: ...


Suggested change

def __call__(self, loop: LoopContext) -> None: ...

def __call__(self, context: LoopContext) -> None: ...

nicoddemus · 2026-10-17T11:57:52Z

+    excinfo: ExceptionInfo[BaseException] | None
+    when_failed: Literal["setup", "call", "teardown"] | None


Given those two are tied together:

@dataclass(frozen=True) class FailureDetails: when: Literal["setup", "call", "teardown"] excinfo: ExceptionInfo[BaseException]

Then:

Suggested change

excinfo: ExceptionInfo[BaseException] | None

when_failed: Literal["setup", "call", "teardown"] | None

failure: FailureDetails | None

nicoddemus · 2026-10-17T11:58:52Z

+    reports: tuple[TestReport, ...]  # setup[, call], teardown
+    excinfo: ExceptionInfo[BaseException] | None
+    when_failed: Literal["setup", "call", "teardown"] | None
+    duration: float


Can't we get duration from reports? If so we should make this a read-only property.

nicoddemus · 2026-10-17T11:59:54Z

+`LoopContext`, `IterationResult` and the `LoopController` typing protocol are
+exported as `pytest.LoopContext` etc.; the implementation lives in a new
+`src/_pytest/loop.py`.
+


We should include an example of how to implement LoopController that makes use of those classes here. We can use a simple example, like pytest-rerunfailures, with "repeat a test 3 times until it passes, otherwise reports the last failure".

pytest-rerunfailures itself reports a failed test as "R" (rerun), until it reaches the maximum failure count which then turns it into a proper error. For example, rerun=5, if a test fails twice and then passes on the 3rd time, it is reported as:

test_foo RERUN test_foo RERUN test_foo PASSED

How will this be handled in this proposal?

nicoddemus · 2026-10-17T12:15:16Z

+    ) -> IterationResult: ...
+
+    def emit(self, result: IterationResult) -> None: ...
+


Suggested change

@property

def config(self) -> pytest.Config: ... # Convenience for item.config.

nicoddemus · 2026-10-17T12:17:28Z

+- `report="discard"` — never reportable; only counted in the loop summary
+  (hypothesis's 99 silent passing examples).


Why having this option at all? I mean, the user can just ignore the report objects... the objective is to save memory? If so we can be explicit about the intent here.

RonnyPfannschmidt requested a review from Zac-HD July 24, 2026 21:50

RonnyPfannschmidt force-pushed the proposal/loop-protocol branch from f856c35 to 4a9be8f Compare September 15, 2026 11:34

RonnyPfannschmidt force-pushed the proposal/loop-protocol branch from 4a9be8f to d974ca3 Compare September 23, 2026 07:00

RonnyPfannschmidt force-pushed the proposal/loop-protocol branch 2 times, most recently from ef0e85d to 7028175 Compare October 11, 2026 07:26

RonnyPfannschmidt force-pushed the proposal/loop-protocol branch from 7028175 to 70b7ad4 Compare October 14, 2026 20:28

nicoddemus reviewed Oct 17, 2026

View reviewed changes

Navigation Menu

Repository navigation

Proposal: marker-driven loop/rerun protocol (deep hypothesis integration, pytest-repeat, retries) - #14779

Proposal: marker-driven loop/rerun protocol (deep hypothesis integration, pytest-repeat, retries)#14779
RonnyPfannschmidt wants to merge 1 commit into
pytest-dev:mainfrom
RonnyPfannschmidt:proposal/loop-protocol

RonnyPfannschmidt commented Jul 24, 2026

Uh oh!

nicoddemus left a comment

Uh oh!

nicoddemus Oct 17, 2026

Uh oh!

nicoddemus Oct 17, 2026

Uh oh!

nicoddemus Oct 17, 2026

Uh oh!

nicoddemus Oct 17, 2026

Uh oh!

nicoddemus Oct 17, 2026

Uh oh!

nicoddemus Oct 17, 2026

Uh oh!

nicoddemus Oct 17, 2026 •

edited

Loading

Uh oh!

nicoddemus Oct 17, 2026

Uh oh!

nicoddemus Oct 17, 2026

Uh oh!

nicoddemus Oct 17, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		`LoopContext`, `IterationResult` and the `LoopController` typing protocol are
		exported as `pytest.LoopContext` etc.; the implementation lives in a new

	def __call__(self, loop: LoopContext) -> None: ...
	def __call__(self, context: LoopContext) -> None: ...

		excinfo: ExceptionInfo[BaseException] \| None
		when_failed: Literal["setup", "call", "teardown"] \| None

	excinfo: ExceptionInfo[BaseException] \| None
	when_failed: Literal["setup", "call", "teardown"] \| None
	failure: FailureDetails \| None

		) -> IterationResult: ...

		def emit(self, result: IterationResult) -> None: ...


	@property
	def config(self) -> pytest.Config: ... # Convenience for item.config.

		- `report="discard"` — never reportable; only counted in the loop summary
		(hypothesis's 99 silent passing examples).

Conversation

RonnyPfannschmidt commented Jul 24, 2026

What

Why

Uh oh!

nicoddemus left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nicoddemus Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nicoddemus Oct 17, 2026 •

edited

Loading