Skip to content

Commit 976fc49

Browse files
feat: per-endpoint circuit breaker option (#22)
* feat: per-endpoint circuit breaker option Add `per_endpoint_circuit_breaker: bool = False` to `HyperpingClient` and `AsyncHyperpingClient`. When enabled, the client maintains an independent `CircuitBreaker` per request path (query string and fragment stripped from the key) so a single flaky endpoint no longer blocks traffic to healthy ones. Default `False` preserves the original single-shared-breaker behaviour and all existing breaker tests. Per-path state is queryable via `client.circuit_breaker_state_for(path)`; the existing `client.circuit_breaker` property still returns the shared breaker. The per-path dict is protected by its own `threading.Lock`; the per-path `CircuitBreaker` instances retain their own internal lock. Tests cover: isolation between paths, state-query API, query-string stripping, default-mode unchanged, async parity, and 50-thread concurrent access. * refactor: bucket per-endpoint breakers by Endpoint prefix Address review feedback on the initial per-endpoint breaker implementation. The previous version keyed the breaker dict on the literal request path. For paths that embed resource UUIDs (`/v1/monitors/{uuid}`, etc.) this meant every resource ended up with its own breaker, the dict could grow unbounded in long-lived clients, and the README's `circuit_breaker_state_for(Endpoint.MONITORS)` example only saw list-call state, not per-resource calls. Changes: - Default breaker key now collapses sub-resource paths under their matching `Endpoint` prefix (`/v1/monitors`, `/v3/incidents`, ...). The breaker set is bounded by the number of Endpoint values, not by resource cardinality. - New `breaker_key_fn: Callable[[str], str] | None` lets callers opt into a different scheme (per-UUID, per-verb, etc.) and explicitly own bounding. - OPEN-state error message in per-endpoint mode now includes the breaker key ("Circuit breaker OPEN for '/v1/monitors' - ..."), so the failure points at which endpoint tripped. - `circuit_breaker_state_for(path)` now falls back to the shared breaker's state in default mode instead of raising, so the call is always safe to make and toggling the flag at construction time doesn't change the API surface callers see. - `_breaker_key` rewritten with `urllib.parse.urlsplit` and the dict-lock comment explains the `threading.Lock` choice for the async client. Tests added for endpoint canonicalisation, custom `breaker_key_fn`, default- mode `circuit_breaker_state_for` returning shared state, and the OPEN error message including the endpoint key.
1 parent 03d4940 commit 976fc49

6 files changed

Lines changed: 632 additions & 28 deletions

File tree

‎BACKLOG.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ Agents: Security, Architecture, Code Quality, Refactor/Dead Code.
4646

4747
- [x] **M1** Async client (`AsyncHyperpingClient`) — shipped in PR #13 (feature/sdk-py-03-async-client).
4848
- [x] **M2** Pagination (`page` param, `hasNextPage` auto-pagination via `collect_all_pages` / `collect_all_pages_async`) — shipped in PR #12 (merged) and PR #13.
49-
- [ ] **M3** Per-endpoint circuit breaker (`per_endpoint_circuit_breaker: bool = False` option). **Deferred — moderate scope, add in follow-up.**
49+
- [x] **M3** Per-endpoint circuit breaker (`per_endpoint_circuit_breaker: bool = False` option) on `HyperpingClient` and `AsyncHyperpingClient`. Per-path state via `circuit_breaker_state_for(path)`. Default off; existing single-shared-breaker behaviour preserved.
5050
- [x] **M4** `MonitorCreate` now has `@model_validator(mode="after")` that raises `ValueError` if DNS fields are set on non-DNS monitors.
5151
- [x] **M5** `MonitorListResponse` is in `__all__` — retained but documented as not returned by any client method. Will be used once pagination lands.
5252
- [x] **M6** `APIErrorResponse` removed from `__all__` (documented as intentionally internal in comment).
@@ -106,7 +106,7 @@ The following items require either a semver bump, a separate PR, or manual work:
106106

107107
- ~~**M1** Async client~~ — shipped
108108
- ~~**M2** Pagination~~ — shipped
109-
- **M3** Per-endpoint circuit breaker option
109+
- ~~**M3** Per-endpoint circuit breaker option~~ — shipped
110110
- **M11** URL validation for HTTP-protocol monitors (cross-field, needs discriminated union work)
111111
- **M12** DateTime coercion (breaking change — v0.2.0)
112112
- **H11** Pin all GitHub Actions `uses:` to 40-char commit SHAs (requires per-tag SHA lookup)

‎CHANGELOG.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,20 @@ All notable changes to this project will be documented in this file.
55
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
66
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
77

8+
## [Unreleased]
9+
10+
### Added
11+
12+
- Per-endpoint circuit breaker option (`per_endpoint_circuit_breaker: bool = False`) on
13+
`HyperpingClient` and `AsyncHyperpingClient`. When enabled, each `Endpoint` gets its own
14+
breaker state so a single flaky endpoint no longer blocks traffic to healthy ones.
15+
Sub-resource paths (e.g. `/v1/monitors/{uuid}`, `/v1/monitors/{uuid}/reports`) are
16+
bucketed under their parent `Endpoint` prefix so the breaker set stays bounded; pass a
17+
custom `breaker_key_fn` to change that. The OPEN-state error message now identifies
18+
which endpoint tripped. State for a given path is readable via
19+
`client.circuit_breaker_state_for(path)` in either mode. Default behaviour is
20+
unchanged. See README for details.
21+
822
## [1.5.0] - 2026-04-20
923

1024
### Added

‎README.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -252,6 +252,48 @@ client = HyperpingClient(
252252
)
253253
```
254254

255+
### Per-endpoint circuit breaker
256+
257+
By default a single shared circuit breaker covers every request. If one endpoint flakes, every other endpoint is also blocked. Enable `per_endpoint_circuit_breaker=True` to keep one breaker per *endpoint* so a failing endpoint does not punish healthy ones:
258+
259+
```python
260+
client = HyperpingClient(
261+
api_key="sk_...",
262+
per_endpoint_circuit_breaker=True,
263+
)
264+
265+
# Inspect state for an endpoint. The breaker key is canonicalised to the
266+
# matching `Endpoint` prefix, so all sub-resource paths share a bucket:
267+
from hyperping import CircuitState, Endpoint
268+
269+
state = client.circuit_breaker_state_for(str(Endpoint.MONITORS))
270+
# /v1/monitors, /v1/monitors/mon_abc and /v1/monitors/mon_abc/reports all
271+
# report the same state — they share the `/v1/monitors` breaker.
272+
assert client.circuit_breaker_state_for(f"{Endpoint.MONITORS}/mon_abc") == state
273+
assert state in {CircuitState.CLOSED, CircuitState.HALF_OPEN, CircuitState.OPEN}
274+
```
275+
276+
If you need different bucketing (e.g. one breaker per resource UUID, or a single breaker per HTTP verb), pass a `breaker_key_fn`:
277+
278+
```python
279+
def per_resource(path: str) -> str:
280+
# one breaker per literal request path
281+
return path.split("?", 1)[0]
282+
283+
client = HyperpingClient(
284+
api_key="sk_...",
285+
per_endpoint_circuit_breaker=True,
286+
breaker_key_fn=per_resource,
287+
)
288+
```
289+
290+
| Option | Type | Default | Description |
291+
| --- | --- | --- | --- |
292+
| `per_endpoint_circuit_breaker` | `bool` | `False` | When `True`, maintain a separate circuit breaker keyed by request endpoint instead of using one shared breaker. The same `circuit_breaker_config` applies to every per-endpoint breaker. The shared breaker remains accessible via `client.circuit_breaker`. |
293+
| `breaker_key_fn` | `Callable[[str], str] \| None` | `None` | Override the default endpoint-prefix bucketing. Receives the request path and returns the breaker key. Default behaviour collapses every path under the matching `Endpoint` prefix so the breaker set stays bounded (one per `Endpoint`); a custom function takes responsibility for keeping the key set bounded. Ignored unless `per_endpoint_circuit_breaker=True`. |
294+
295+
State for any path is readable via `client.circuit_breaker_state_for(path)`. In the default (single-breaker) mode this returns the shared breaker's state for any path, so the call is always safe regardless of the flag. The same options and method are available on `AsyncHyperpingClient`.
296+
255297
## Type Safety
256298

257299
This package ships a `py.typed` marker (PEP 561) and is fully typed. Works out of the box with mypy and pyright.

‎src/hyperping/_async_client.py‎

Lines changed: 96 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,10 @@
1414
import asyncio
1515
import logging
1616
import random
17+
import threading
18+
from collections.abc import Callable
1719
from typing import Any
20+
from urllib.parse import urlsplit
1821

1922
import httpx
2023
from pydantic import SecretStr
@@ -28,10 +31,11 @@
2831
from hyperping._circuit_breaker import (
2932
CircuitBreaker,
3033
CircuitBreakerConfig,
34+
CircuitState,
3135
)
3236
from hyperping._internals import DEFAULT_USER_AGENT, RETRY_AFTER_MAX, sanitize_for_log
3337
from hyperping.client import DEFAULT_RETRY_CONFIG, RetryConfig
34-
from hyperping.endpoints import API_BASE
38+
from hyperping.endpoints import API_BASE, Endpoint
3539
from hyperping.exceptions import (
3640
HyperpingAPIError,
3741
HyperpingAuthError,
@@ -73,6 +77,8 @@ def __init__(
7377
retry_config: RetryConfig | None = None,
7478
circuit_breaker_config: CircuitBreakerConfig | None = None,
7579
user_agent: str | None = None,
80+
per_endpoint_circuit_breaker: bool = False,
81+
breaker_key_fn: Callable[[str], str] | None = None,
7682
) -> None:
7783
"""Initialize the async Hyperping API client.
7884
@@ -82,8 +88,19 @@ def __init__(
8288
base_url: Override the default API base URL.
8389
timeout: HTTP request timeout in seconds.
8490
retry_config: Retry behaviour configuration.
85-
circuit_breaker_config: Circuit breaker configuration.
91+
circuit_breaker_config: Circuit breaker configuration. When
92+
``per_endpoint_circuit_breaker`` is ``True`` the same config is
93+
applied to each per-endpoint breaker.
8694
user_agent: Custom ``User-Agent`` header value.
95+
per_endpoint_circuit_breaker: When ``True``, maintain an
96+
independent breaker per :class:`~hyperping.endpoints.Endpoint`
97+
prefix (sub-resources inherit the parent endpoint's breaker,
98+
so the breaker set stays bounded). Default ``False``
99+
preserves the original single-shared-breaker behaviour.
100+
breaker_key_fn: Override the default endpoint-prefix bucketing.
101+
Receives the request path and must return the breaker key.
102+
Ignored unless ``per_endpoint_circuit_breaker`` is ``True``.
103+
Caller is responsible for keeping the key set bounded.
87104
"""
88105
raw_key = api_key.get_secret_value() if isinstance(api_key, SecretStr) else api_key
89106
if not raw_key or not raw_key.strip():
@@ -92,7 +109,12 @@ def __init__(
92109
self.base_url = (base_url or self.DEFAULT_BASE_URL).rstrip("/")
93110
self.timeout = timeout
94111
self.retry_config = retry_config or DEFAULT_RETRY_CONFIG
112+
self._circuit_breaker_config = circuit_breaker_config
95113
self._circuit_breaker = CircuitBreaker(circuit_breaker_config)
114+
self._per_endpoint_circuit_breaker = per_endpoint_circuit_breaker
115+
self._breaker_key_fn = breaker_key_fn
116+
self._endpoint_breakers: dict[str, CircuitBreaker] = {}
117+
self._endpoint_breakers_lock = threading.Lock()
96118

97119
self._client = httpx.AsyncClient(
98120
base_url=self.base_url,
@@ -120,9 +142,74 @@ async def __aexit__(self, *args: Any) -> None:
120142

121143
@property
122144
def circuit_breaker(self) -> CircuitBreaker:
123-
"""Access the circuit breaker state (for monitoring)."""
145+
"""Access the (shared) circuit breaker state (for monitoring).
146+
147+
In per-endpoint mode this returns the original shared breaker, kept
148+
for backward compatibility; the per-path breakers are exposed via
149+
:meth:`circuit_breaker_state_for`.
150+
"""
124151
return self._circuit_breaker
125152

153+
def _resolve_breaker_key(self, path: str) -> str:
154+
"""Map a request path to its circuit-breaker key.
155+
156+
Default bucketing strips query/fragment and collapses the path under
157+
the longest matching :class:`Endpoint` prefix; a custom
158+
``breaker_key_fn`` wins outright.
159+
"""
160+
if self._breaker_key_fn is not None:
161+
return self._breaker_key_fn(path)
162+
pure = urlsplit(path).path
163+
for ep in Endpoint:
164+
ep_value = ep.value
165+
if pure == ep_value or pure.startswith(ep_value + "/"):
166+
return ep_value
167+
return pure
168+
169+
def _breaker_for(self, path: str) -> CircuitBreaker:
170+
"""Return the breaker that governs ``path`` (shared, or per-endpoint)."""
171+
if not self._per_endpoint_circuit_breaker:
172+
return self._circuit_breaker
173+
key = self._resolve_breaker_key(path)
174+
# threading.Lock here is intentional: see HyperpingClient._breaker_for
175+
# for the rationale (works under both pure-asyncio and mixed-thread use).
176+
with self._endpoint_breakers_lock:
177+
breaker = self._endpoint_breakers.get(key)
178+
if breaker is None:
179+
breaker = CircuitBreaker(self._circuit_breaker_config)
180+
self._endpoint_breakers[key] = breaker
181+
return breaker
182+
183+
def circuit_breaker_state_for(self, path: str) -> CircuitState:
184+
"""Return the circuit state of the breaker governing ``path``.
185+
186+
In per-endpoint mode the path is canonicalised the same way as during
187+
a request; untouched buckets report :attr:`CircuitState.CLOSED`
188+
without allocating a breaker. In default mode the shared breaker's
189+
state is returned for any path.
190+
"""
191+
if not self._per_endpoint_circuit_breaker:
192+
return self._circuit_breaker.state
193+
key = self._resolve_breaker_key(path)
194+
with self._endpoint_breakers_lock:
195+
breaker = self._endpoint_breakers.get(key)
196+
return breaker.state if breaker is not None else CircuitState.CLOSED
197+
198+
def _circuit_open_message(self, breaker: CircuitBreaker, path: str) -> str:
199+
"""Build the error message raised when a request is rejected by an OPEN breaker."""
200+
if self._per_endpoint_circuit_breaker:
201+
key = self._resolve_breaker_key(path)
202+
return (
203+
f"Circuit breaker OPEN for {key!r} - API calls to this endpoint suspended. "
204+
f"Consecutive failures: {breaker.failure_count}. "
205+
f"Will recover after {breaker.recovery_timeout}s."
206+
)
207+
return (
208+
f"Circuit breaker OPEN - API calls suspended. "
209+
f"Consecutive failures: {breaker.failure_count}. "
210+
f"Will recover after {breaker.recovery_timeout}s."
211+
)
212+
126213
# ==================== Error Handling ====================
127214

128215
def _parse_error_body(self, response: httpx.Response) -> dict[str, Any]:
@@ -236,7 +323,7 @@ async def _execute_single_attempt(
236323
if response.status_code >= 400:
237324
return response
238325

239-
self._circuit_breaker.record_success()
326+
self._breaker_for(path).record_success()
240327
if response.status_code == 204:
241328
return {}
242329
return response.json() # type: ignore[no-any-return]
@@ -262,13 +349,9 @@ async def _request(
262349
Raises:
263350
HyperpingAPIError: On API errors after retries exhausted
264351
"""
265-
if not self._circuit_breaker.call_allowed():
266-
cb = self._circuit_breaker
267-
raise HyperpingAPIError(
268-
f"Circuit breaker OPEN - API calls suspended. "
269-
f"Consecutive failures: {cb.failure_count}. "
270-
f"Will recover after {cb.recovery_timeout}s."
271-
)
352+
breaker = self._breaker_for(path)
353+
if not breaker.call_allowed():
354+
raise HyperpingAPIError(self._circuit_open_message(breaker, path))
272355

273356
last_exception: Exception | None = None
274357
delay = self.retry_config.initial_delay
@@ -299,7 +382,7 @@ async def _request(
299382
continue
300383

301384
if response.status_code >= 500:
302-
self._circuit_breaker.record_failure()
385+
breaker.record_failure()
303386
self._handle_response_error(response)
304387

305388
except (httpx.TimeoutException, httpx.RequestError) as e:
@@ -320,7 +403,7 @@ async def _request(
320403
self.retry_config.max_delay,
321404
)
322405
continue
323-
self._circuit_breaker.record_failure()
406+
breaker.record_failure()
324407
if isinstance(e, httpx.TimeoutException):
325408
raise HyperpingAPIError(f"Request timeout after {max_attempts} attempts") from e
326409
raise HyperpingAPIError(f"Request failed: {e}") from e

0 commit comments

Comments
 (0)