diff --git a/AGENTS.md b/AGENTS.md index 433bac8df2..fb6361ee58 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -395,8 +395,13 @@ Decide where state lives before writing code: Classes that use static caching need a `flushState()` method for test cleanup — see "Static state and test cleanup" under Writing Tests. - **Release idle pooled connections before unrelated long waits** — Holding connections while waiting for external services or child coroutines can exhaust pools and limit concurrency. Release them at the appropriate shared boundary, preserving active operations and session ownership; consider reacquisition costs rather than adding release calls indiscriminately. For database sessions, reuse `ConnectionResolver::releaseIdleConnections()`, which preserves transactions and explicit pins. See `Mailer::sendSymfonyMessage()` and `ConcurrentImportRunner::wait()` for examples. -- **Treat coroutine cancellation separately from errors** — Code that owns coroutine cancellation must pass `CanceledException` through unchanged and clean up the children or resources it owns. Add this only where cancellation can actually happen and mishandling it causes a real bug—not to every `catch (Throwable)`. -- **Contain cancellation in direct engine coroutines** — A callback passed directly to `Engine\Coroutine` must catch cancellation after cleanup so it cannot terminate the worker. The higher-level `Coroutine` class already provides this protection. + +### Coroutine cancellation + +- **Treat coroutine cancellation separately from errors** — Where cancellation needs different handling and the operation runs callbacks or listeners—including `releaseIdleConnections()`—use `Coroutine::causedByCancellation($exception)`. Cleanup errors can carry cancellation in their exception chain. Preserve the original exception, perform owned cleanup, and propagate cancellation instead of retrying or continuing normal work. Report genuine cleanup errors without swallowing cancellation. Add this only where cancellation can actually happen and mishandling it causes a real bug. Keep plain checks for native-only operations. +- **Preserve cancellation ownership** — If shared work can safely be retried, allow another waiter to retry after its owner is canceled. Otherwise, share the failure behind `ForeignCancellationException`, or `ChildCancellationException` for an independently canceled child, preserving the original cause. The canceled owner propagates its own exception unchanged; waiters must not mistake it for their own cancellation. +- **Contain cancellation in direct engine coroutines** — Callbacks passed directly to `Engine\Coroutine` must catch cancellation after cleanup so it cannot terminate the worker. The higher-level `Coroutine` class already provides this protection. +- **Test cancellation through cleanup** — When adding or changing cancellation-sensitive retry, child-management or shared-result handling, extend relevant regression tests with throwing cleanup during cancellation. Verify the resulting behavior, not just the exception type. ## When to Stop and Report diff --git a/docs/todo.md b/docs/todo.md index 355d9ad90a..f3cd082504 100644 --- a/docs/todo.md +++ b/docs/todo.md @@ -30,6 +30,7 @@ - Audit unmatched PHPStan inline ignores and global patterns with `reportUnmatchedIgnoredErrors` enabled — currently 196 unmatched inline ignores across 99 files plus 5 unmatched global patterns. Remove only suppressions that no longer match after tracing the underlying code; do not replace correct source with runtime branches or wider types merely to keep static analysis green. Decide as part of the work whether `phpstan.neon.dist` should then set `reportUnmatchedIgnoredErrors: true` permanently, since leaving it `false` lets the suppressions rot again. - Add PHPStan Eloquent extensions that preserve `Eloquent\Builder` for non-passthrough methods forwarded to `Query\Builder`, and expose model named scopes on Eloquent builders and relations. The query-builder mixin currently gives fluent calls the wrong builder type, while named scopes are treated as nonexistent methods; these gaps force scopes to split mutation from return and leave `HasDatabaseNotifications` with `method.notFound` suppressions. This will be the repository's first PHPStan extension, so use Larastan as prior art and wire the extensions into `phpstan.neon.dist` without maintaining duplicate query-method or scope lists in `@method` annotations. - Investigate static analysis that turns the coroutine-safety and worker-lifetime rules in `AGENTS.md` into automated checks, so mistakes no longer depend on agent steering and review alone. Decide which tool fits each check: PHPStan rules for detection with type information, Rector in dry-run mode, or another linter. Start with property writes outside the constructor on container-resolved classes (auto-singletons and singletons) that are not `Transient`, scoped, or execution-scoped, since per-request data stored there leaks to concurrent requests in the same worker. Intentional worker-lifetime state such as manager driver caches, registries, and resolved configuration needs an explicit way to opt out, so measure false positives across `src/` before choosing that mechanism. Then evaluate other patterns: static property writes outside boot code and `flushState()`; static caches keyed by request-derived values; request state captured in constructors of classes that are not execution-scoped; `Config::set()` during request handling; child coroutines started inside `DB::transaction()`, which borrow their own pooled connection and therefore run outside the transaction; and raw `sleep()`/`usleep()` in source instead of `Sleep`. Ship the useful rules so applications and private packages can enable the same checks. +- Investigate a focused PHPStan rule for cancellation checks that miss cleanup exception chains. Detect bare `CanceledException` checks around callbacks or listeners, including `releaseIdleConnections()`, where `Coroutine::causedByCancellation()` is needed to prevent retries or continued work after cancellation. Use the cancellation regression cases to assess detection and false positives; preserve valid native-only checks, cleanup-error reporting and foreign-cancellation boundaries. Integrate useful checks with the coroutine-safety analysis above rather than attempting to infer every ownership decision automatically. - Audit typed input accessors across `InteractsWithData`, Support `DataObject`, and Hypervel Data. Define the accepted integer, float, and boolean forms for each public contract. Extract a neutral Support conversion primitive only if `InteractsWithData` and `DataObject` converge on exactly the same strict semantics, and decide separately whether Hypervel Data should adopt those semantics rather than changing its tested permissive casts as a side effect. ## Testing diff --git a/src/ai/src/Gateway/TextGenerationLoop.php b/src/ai/src/Gateway/TextGenerationLoop.php index cac7afe867..9ce51b3aca 100644 --- a/src/ai/src/Gateway/TextGenerationLoop.php +++ b/src/ai/src/Gateway/TextGenerationLoop.php @@ -65,6 +65,7 @@ use Hypervel\Ai\Tools\Request; use Hypervel\Ai\Tools\ToolNameResolver; use Hypervel\Context\CoroutineContext; +use Hypervel\Coroutine\Coroutine; use Hypervel\Coroutine\WaitConcurrent; use Hypervel\Database\ConnectionResolver; use Hypervel\Engine\Channel; @@ -1163,9 +1164,11 @@ protected function resolveApprovalResults(array $approval, array $messages, arra try { $result = $this->executeTool($tool, $arguments, $toolCall->id, $context); - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $exception) { + if (Coroutine::causedByCancellation($exception)) { + throw $exception; + } + $failed = true; $result = 'The tool call failed: ' . $exception->getMessage(); } diff --git a/src/ai/src/Middleware/RememberConversation.php b/src/ai/src/Middleware/RememberConversation.php index 9466e1d643..7adadd05c8 100644 --- a/src/ai/src/Middleware/RememberConversation.php +++ b/src/ai/src/Middleware/RememberConversation.php @@ -22,10 +22,10 @@ use Hypervel\Ai\Storage\StoredTurn; use Hypervel\Ai\Support\ConversationSelection; use Hypervel\Ai\Support\PendingConversationTitle; +use Hypervel\Coroutine\Coroutine; use Hypervel\Support\ClassMetadataCache; use Hypervel\Support\Facades\DB; use Hypervel\Support\Str; -use Swoole\Coroutine\CanceledException; use Throwable; use WeakReference; @@ -207,10 +207,14 @@ protected function rememberTurn(AgentPrompt $prompt, AgentResponse $response, ?s $cancellation = null; if ($newConversation) { - if (! $exception instanceof CanceledException && ! $exception instanceof StreamAbandonedException) { + if (($exception === null || ! Coroutine::causedByCancellation($exception)) && ! $exception instanceof StreamAbandonedException) { try { $title = $prompt->pendingConversationTitle()?->value() ?? $this->generateTitle($prompt->prompt); - } catch (CanceledException $caught) { + } catch (Throwable $caught) { + if (! Coroutine::causedByCancellation($caught)) { + throw $caught; + } + $cancellation = $caught; } } @@ -367,9 +371,11 @@ protected function generateTitle(string $prompt): string ); return Str::limit($response->text, 100); - } catch (CanceledException $exception) { - throw $exception; - } catch (Throwable) { + } catch (Throwable $exception) { + if (Coroutine::causedByCancellation($exception)) { + throw $exception; + } + return Str::limit($prompt, 100, preserveWords: true); } } diff --git a/src/ai/src/Responses/StreamableAgentResponse.php b/src/ai/src/Responses/StreamableAgentResponse.php index 953acfc5b8..ae63ec13ae 100644 --- a/src/ai/src/Responses/StreamableAgentResponse.php +++ b/src/ai/src/Responses/StreamableAgentResponse.php @@ -25,7 +25,6 @@ use Hypervel\Http\Request; use Hypervel\Support\Collection; use IteratorAggregate; -use Swoole\Coroutine\CanceledException; use Symfony\Component\HttpFoundation\Response; use Throwable; use Traversable; @@ -146,16 +145,20 @@ public function broadcast(Channel|array $channels, bool $now = true, array $exce $send($event); } }); - } catch (CanceledException $exception) { - $batcher?->cancel(); - - throw $exception; } catch (Throwable $exception) { + if (Coroutine::causedByCancellation($exception)) { + $batcher?->cancel(); + + throw $exception; + } + try { $batcher?->finish(); - } catch (CanceledException $cancellation) { - throw $cancellation; } catch (Throwable $senderFailure) { + if (Coroutine::causedByCancellation($senderFailure)) { + throw $senderFailure; + } + if ($senderFailure !== $exception) { report($senderFailure); } diff --git a/src/ai/src/Streaming/BroadcastBatcher.php b/src/ai/src/Streaming/BroadcastBatcher.php index 387a759b7b..81a47cf643 100644 --- a/src/ai/src/Streaming/BroadcastBatcher.php +++ b/src/ai/src/Streaming/BroadcastBatcher.php @@ -8,6 +8,7 @@ use Hypervel\Ai\Streaming\Events\ReasoningDelta; use Hypervel\Ai\Streaming\Events\StreamEvent; use Hypervel\Ai\Streaming\Events\TextDelta; +use Hypervel\Coroutine\Coroutine; use Hypervel\Coroutine\Exceptions\ChildCancellationException; use Hypervel\Coroutine\WaitConcurrent; use Hypervel\Database\ConnectionResolver; @@ -52,10 +53,10 @@ public function __construct( try { $work = fn () => $this->drain($send); $runner === null ? $work() : $runner($work); - } catch (CanceledException $exception) { - $this->failure = new ChildCancellationException('The broadcast sender was canceled while its owner remained active.', previous: $exception); } catch (Throwable $exception) { - $this->failure = $exception; + $this->failure = Coroutine::causedByCancellation($exception) + ? new ChildCancellationException('The broadcast sender was canceled while its owner remained active.', previous: $exception) + : $exception; } finally { // Wake a producer blocked on a full channel if the sender fails. $this->channel->close(); @@ -98,12 +99,12 @@ public function finish(): void try { // Completion callbacks may have borrowed the slot needed to send the final batches. ConnectionResolver::releaseIdleConnections(); - } catch (CanceledException $exception) { - $this->cancel(); - - throw $exception; } catch (Throwable $exception) { - $this->concurrent->wait(); + if (Coroutine::causedByCancellation($exception)) { + $this->cancel(); + } else { + $this->concurrent->wait(); + } throw $exception; } diff --git a/src/ai/src/Tools/AgentTool.php b/src/ai/src/Tools/AgentTool.php index ee7bf72414..4ac59e5ad7 100644 --- a/src/ai/src/Tools/AgentTool.php +++ b/src/ai/src/Tools/AgentTool.php @@ -15,8 +15,8 @@ use Hypervel\Ai\Streaming\Events\StreamEvent; use Hypervel\Contracts\Container\Transient; use Hypervel\Contracts\JsonSchema\JsonSchema; +use Hypervel\Coroutine\Coroutine; use Stringable; -use Swoole\Coroutine\CanceledException; use Throwable; class AgentTool implements Approvable, Tool, Transient @@ -60,9 +60,11 @@ public function handle(Request $request): string { try { return $this->agent->prompt((string) $request['task'])->text; - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $throwable) { + if (Coroutine::causedByCancellation($throwable)) { + throw $throwable; + } + // Degraded to a result so the parent run survives; the failure surfaces as the sub-agent's own AgentFailed... return 'Agent failed: ' . $throwable->getMessage(); } @@ -85,9 +87,11 @@ public function stream(Request $request): Generator yield from $stream; return (string) $stream->text; - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $throwable) { + if (Coroutine::causedByCancellation($throwable)) { + throw $throwable; + } + return 'Agent failed: ' . $throwable->getMessage(); } } diff --git a/src/ai/src/Tools/Filesystem/CopyFile.php b/src/ai/src/Tools/Filesystem/CopyFile.php index f982b0e53b..2927e9442e 100644 --- a/src/ai/src/Tools/Filesystem/CopyFile.php +++ b/src/ai/src/Tools/Filesystem/CopyFile.php @@ -7,7 +7,7 @@ use Hypervel\Ai\Attributes\Strict; use Hypervel\Ai\Tools\Request; use Hypervel\Contracts\JsonSchema\JsonSchema; -use Swoole\Coroutine\CanceledException; +use Hypervel\Coroutine\Coroutine; use Throwable; #[Strict] @@ -37,9 +37,11 @@ public function handle(Request $request): string } $copied = $disk->copy($from, $to); - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $throwable) { + if (Coroutine::causedByCancellation($throwable)) { + throw $throwable; + } + return "Unable to copy [{$from}] to [{$to}]: {$throwable->getMessage()}"; } diff --git a/src/ai/src/Tools/Filesystem/GetFileMetadata.php b/src/ai/src/Tools/Filesystem/GetFileMetadata.php index 2c707d7d06..52d76390ef 100644 --- a/src/ai/src/Tools/Filesystem/GetFileMetadata.php +++ b/src/ai/src/Tools/Filesystem/GetFileMetadata.php @@ -7,7 +7,7 @@ use Hypervel\Ai\Attributes\Strict; use Hypervel\Ai\Tools\Request; use Hypervel\Contracts\JsonSchema\JsonSchema; -use Swoole\Coroutine\CanceledException; +use Hypervel\Coroutine\Coroutine; use Throwable; #[Strict] @@ -32,9 +32,11 @@ public function handle(Request $request): string try { $size = $disk->size($path); - } catch (CanceledException $exception) { - throw $exception; - } catch (Throwable) { + } catch (Throwable $exception) { + if (Coroutine::causedByCancellation($exception)) { + throw $exception; + } + return "File [{$path}] does not exist."; } diff --git a/src/ai/src/Tools/Filesystem/GetFileUrl.php b/src/ai/src/Tools/Filesystem/GetFileUrl.php index a47b48df55..de88949cbe 100644 --- a/src/ai/src/Tools/Filesystem/GetFileUrl.php +++ b/src/ai/src/Tools/Filesystem/GetFileUrl.php @@ -7,7 +7,7 @@ use Hypervel\Ai\Attributes\Strict; use Hypervel\Ai\Tools\Request; use Hypervel\Contracts\JsonSchema\JsonSchema; -use Swoole\Coroutine\CanceledException; +use Hypervel\Coroutine\Coroutine; use Throwable; #[Strict] @@ -43,9 +43,11 @@ public function handle(Request $request): string ? $disk->temporaryUrl($path, now()->addMinutes($minutes)) // @phpstan-ignore method.notFound : $disk->url($path); - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $throwable) { + if (Coroutine::causedByCancellation($throwable)) { + throw $throwable; + } + return "Unable to generate a URL for [{$path}]: {$throwable->getMessage()}"; } } diff --git a/src/ai/src/Tools/Filesystem/MoveFile.php b/src/ai/src/Tools/Filesystem/MoveFile.php index cfb149bc68..eadf8f3252 100644 --- a/src/ai/src/Tools/Filesystem/MoveFile.php +++ b/src/ai/src/Tools/Filesystem/MoveFile.php @@ -7,7 +7,7 @@ use Hypervel\Ai\Attributes\Strict; use Hypervel\Ai\Tools\Request; use Hypervel\Contracts\JsonSchema\JsonSchema; -use Swoole\Coroutine\CanceledException; +use Hypervel\Coroutine\Coroutine; use Throwable; #[Strict] @@ -37,9 +37,11 @@ public function handle(Request $request): string } $moved = $disk->move($from, $to); - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $throwable) { + if (Coroutine::causedByCancellation($throwable)) { + throw $throwable; + } + return "Unable to move [{$from}] to [{$to}]: {$throwable->getMessage()}"; } diff --git a/src/ai/src/Tools/Filesystem/ReadFile.php b/src/ai/src/Tools/Filesystem/ReadFile.php index 76b9002b9d..6943ad7fc8 100644 --- a/src/ai/src/Tools/Filesystem/ReadFile.php +++ b/src/ai/src/Tools/Filesystem/ReadFile.php @@ -7,7 +7,7 @@ use Hypervel\Ai\Attributes\Strict; use Hypervel\Ai\Tools\Request; use Hypervel\Contracts\JsonSchema\JsonSchema; -use Swoole\Coroutine\CanceledException; +use Hypervel\Coroutine\Coroutine; use Throwable; #[Strict] @@ -37,9 +37,11 @@ public function handle(Request $request): string try { $size = $disk->size($path); - } catch (CanceledException $exception) { - throw $exception; - } catch (Throwable) { + } catch (Throwable $exception) { + if (Coroutine::causedByCancellation($exception)) { + throw $exception; + } + return "File [{$path}] does not exist."; } diff --git a/src/cache/src/FailoverStore.php b/src/cache/src/FailoverStore.php index fe98c420d5..943e8f8c62 100644 --- a/src/cache/src/FailoverStore.php +++ b/src/cache/src/FailoverStore.php @@ -15,8 +15,8 @@ use Hypervel\Contracts\Cache\RawReadable; use Hypervel\Contracts\Cache\Repository as RepositoryContract; use Hypervel\Contracts\Events\Dispatcher; +use Hypervel\Coroutine\Coroutine; use RuntimeException; -use Swoole\Coroutine\CanceledException; use Throwable; use UnitEnum; use WeakReference; @@ -296,9 +296,11 @@ public function flushLocks(): bool if (! $store->flushLocks()) { $result = false; } - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $throwable) { + if (Coroutine::causedByCancellation($throwable)) { + throw $throwable; + } + $exception ??= $throwable; } } @@ -361,9 +363,11 @@ protected function attemptOnAllStores(Closure|string $method, array $arguments = return $method instanceof Closure ? $method($repository) : $repository->{$method}(...$arguments); - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $exception) { + if (Coroutine::causedByCancellation($exception)) { + throw $exception; + } + $lastException = $exception; $failedCaches[] = $store; @@ -405,9 +409,11 @@ protected function attemptOnEveryStore(string $method, array $arguments): array foreach ($this->stores as $position => $store) { try { $results[] = $this->store($store)->{$method}(...$arguments); - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $exception) { + if (Coroutine::causedByCancellation($exception)) { + throw $exception; + } + $lastException = $exception; $failedCaches[] = $store; diff --git a/src/cache/src/StackStore.php b/src/cache/src/StackStore.php index 37194a26ff..1d51ad6208 100644 --- a/src/cache/src/StackStore.php +++ b/src/cache/src/StackStore.php @@ -12,9 +12,9 @@ use Hypervel\Contracts\Cache\LockProvider; use Hypervel\Contracts\Cache\RawReadable; use Hypervel\Contracts\Cache\Store; +use Hypervel\Coroutine\Coroutine; use Hypervel\Support\CarbonImmutable; use InvalidArgumentException; -use Swoole\Coroutine\CanceledException; use Throwable; use UnitEnum; @@ -149,13 +149,15 @@ static function (StackStoreProxy $store) use ($values): void { foreach (array_keys($values) as $key) { try { $store->forget((string) $key); - } catch (CanceledException $exception) { - // Finish this layer's cleanup, as callStores() does across layers. - $cancellation ??= $exception; - } catch (Throwable) { - // An ordinary failure usually means the layer is unreachable, so stop - // instead of paying a timeout per key. callStores() discards these. - break; + } catch (Throwable $exception) { + if (Coroutine::causedByCancellation($exception)) { + // Finish this layer's cleanup, as callStores() does across layers. + $cancellation ??= $exception; + } else { + // An ordinary failure usually means the layer is unreachable, so stop + // instead of paying a timeout per key. callStores() discards these. + break; + } } } @@ -677,17 +679,18 @@ protected function callStores(Closure $handler, ?Closure $rollback = null, bool return true; } - $cancellation = $exception instanceof CanceledException ? $exception : null; + $cancellation = $exception !== null && Coroutine::causedByCancellation($exception) ? $exception : null; if ($rollback !== null) { // Clear deepest-first so a concurrent read cannot back-fill a cleared upper layer. foreach (array_reverse($completed) as $store) { try { $rollback($store); - } catch (CanceledException $cleanupCancellation) { - $cancellation ??= $cleanupCancellation; - } catch (Throwable) { - // Preserve the write failure that made compensation necessary. + } catch (Throwable $cleanupException) { + // Preserve the write failure unless cleanup was canceled. + if (Coroutine::causedByCancellation($cleanupException)) { + $cancellation ??= $cleanupException; + } } } } @@ -716,9 +719,11 @@ protected function callAllStores(Closure $handler): bool if (! $handler($store)) { $result = false; } - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $throwable) { + if (Coroutine::causedByCancellation($throwable)) { + throw $throwable; + } + $exception ??= $throwable; } } diff --git a/src/connection-pool/src/Connection.php b/src/connection-pool/src/Connection.php index 75f017aa00..6666fdb210 100644 --- a/src/connection-pool/src/Connection.php +++ b/src/connection-pool/src/Connection.php @@ -10,6 +10,7 @@ use Hypervel\Contracts\Container\Container; use Hypervel\Contracts\Events\Dispatcher; use Hypervel\Contracts\Log\StdoutLoggerInterface; +use Hypervel\Coroutine\Coroutine; use Swoole\Coroutine\CanceledException; use Throwable; @@ -48,7 +49,7 @@ public function release(): void try { $this->pool->release($this); } catch (Throwable $exception) { - if (! $failure instanceof CanceledException) { + if ($failure === null || ! Coroutine::causedByCancellation($failure)) { throw $exception; } @@ -86,9 +87,11 @@ private function dispatchReleasingEvent(): void $dispatcher->dispatch(new ConnectionReleasing($this)); } } - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $exception) { + if (Coroutine::causedByCancellation($exception)) { + throw $exception; + } + $this->logger?->error((string) $exception); } } diff --git a/src/console/src/Application.php b/src/console/src/Application.php index 1a3ef3cd91..d93ff3ef50 100644 --- a/src/console/src/Application.php +++ b/src/console/src/Application.php @@ -19,7 +19,6 @@ use Hypervel\Support\ProcessUtils; use Override; use ReflectionClass; -use Swoole\Coroutine\CanceledException; use Symfony\Component\Console\Application as SymfonyApplication; use Symfony\Component\Console\Attribute\AsCommand; use Symfony\Component\Console\Command\Command as SymfonyCommand; @@ -404,16 +403,16 @@ protected function doRunCommand(SymfonyCommand $command, InputInterface $input, $exception = $throwable; } - if (! $exception instanceof CanceledException + if (($exception === null || ! Coroutine::causedByCancellation($exception)) && $container->resolvedScoped(DeferredCallbackCollection::class) ) { try { $container->make(DeferredCallbackCollection::class) ->invokeWhen(fn (DeferredCallback $callback): bool => ($exception === null && $exitCode === SymfonyCommand::SUCCESS) || $callback->always); - } catch (CanceledException $cancellation) { - $exception = $cancellation; } catch (Throwable $throwable) { - $exception ??= $throwable; + if ($exception === null || Coroutine::causedByCancellation($throwable)) { + $exception = $throwable; + } } } } finally { diff --git a/src/console/src/Command.php b/src/console/src/Command.php index b97c371409..f8c792aac9 100644 --- a/src/console/src/Command.php +++ b/src/console/src/Command.php @@ -24,7 +24,6 @@ use Hypervel\Support\Traits\Macroable; use ReflectionClass; use RuntimeException; -use Swoole\Coroutine\CanceledException; use Swoole\ExitException; use Symfony\Component\Console\Command\Command as SymfonyCommand; use Symfony\Component\Console\Input\InputInterface; @@ -325,37 +324,33 @@ protected function execute(InputInterface $input, OutputInterface $output): int $this->normalizeExitCode($this->exitCode), )); } - } catch (CanceledException $cancellation) { - if (! $exception instanceof CanceledException) { - $exception = $cancellation; - } } catch (Throwable $throwable) { - $exception ??= $throwable; + if ($exception === null || (Coroutine::causedByCancellation($throwable) && ! Coroutine::causedByCancellation($exception))) { + $exception = $throwable; + } } if ($commandMutex !== null) { try { $commandMutex->forget($this); - } catch (CanceledException $cancellation) { - if (! $exception instanceof CanceledException) { - $exception = $cancellation; - } } catch (Throwable $throwable) { - $exception ??= $throwable; + if ($exception === null || (Coroutine::causedByCancellation($throwable) && ! Coroutine::causedByCancellation($exception))) { + $exception = $throwable; + } } } - if ($runsInOwnCoroutine && ! $exception instanceof CanceledException) { + if ($runsInOwnCoroutine && ($exception === null || ! Coroutine::causedByCancellation($exception))) { $container = Container::getInstance(); if ($container->resolvedScoped(DeferredCallbackCollection::class)) { try { $container->make(DeferredCallbackCollection::class) ->invokeWhen(fn (DeferredCallback $callback): bool => ($exception === null && $this->normalizeExitCode($this->exitCode) === self::SUCCESS) || $callback->always); - } catch (CanceledException $cancellation) { - $exception = $cancellation; } catch (Throwable $throwable) { - $exception ??= $throwable; + if ($exception === null || Coroutine::causedByCancellation($throwable)) { + $exception = $throwable; + } } } } diff --git a/src/console/src/Commands/ScheduleRunCommand.php b/src/console/src/Commands/ScheduleRunCommand.php index b62ba36939..2aef616a8d 100644 --- a/src/console/src/Commands/ScheduleRunCommand.php +++ b/src/console/src/Commands/ScheduleRunCommand.php @@ -21,6 +21,7 @@ use Hypervel\Contracts\Debug\ExceptionHandler; use Hypervel\Contracts\Events\Dispatcher; use Hypervel\Coroutine\Concurrent; +use Hypervel\Coroutine\Coroutine; use Hypervel\Coroutine\Waiter; use Hypervel\Log\Context\Repository as ContextRepository; use Hypervel\Support\CarbonImmutable; @@ -31,7 +32,6 @@ use Hypervel\Support\InteractsWithTime; use Hypervel\Support\Sleep; use RuntimeException; -use Swoole\Coroutine\CanceledException; use Symfony\Component\Console\Attribute\AsCommand; use Throwable; @@ -361,15 +361,15 @@ private function runTask(Closure $callback): void // Nested commands leave deferred work to this boundary. Drain after // task listeners so callbacks cannot run early. - if (! $exception instanceof CanceledException) { + if ($exception === null || ! Coroutine::causedByCancellation($exception)) { try { $this->invokeDeferredCallbacks( $exception === null && ! CoroutineContext::get(self::TASK_FAILED_CONTEXT_KEY, false) ); - } catch (CanceledException $cancellation) { - $exception = $cancellation; } catch (Throwable $throwable) { - $exception ??= $throwable; + if ($exception === null || Coroutine::causedByCancellation($throwable)) { + $exception = $throwable; + } } } @@ -521,9 +521,11 @@ protected function runEvent(Event $event): void "Scheduled command [{$command}] failed with exit code [{$exitCode}]." ); } - } catch (CanceledException $e) { - throw $e; } catch (Throwable $e) { + if (Coroutine::causedByCancellation($e)) { + throw $e; + } + $failed = true; // runEvent reports ordinary failures instead of throwing them to // runTask; keep that outcome local to this task's deferred work. diff --git a/src/console/src/Scheduling/CallbackEvent.php b/src/console/src/Scheduling/CallbackEvent.php index a14437d61b..0a398cef22 100644 --- a/src/console/src/Scheduling/CallbackEvent.php +++ b/src/console/src/Scheduling/CallbackEvent.php @@ -7,11 +7,11 @@ use DateTimeZone; use Hypervel\Context\CoroutineContext; use Hypervel\Contracts\Container\Container; +use Hypervel\Coroutine\Coroutine; use Hypervel\Support\Reflector; use InvalidArgumentException; use LogicException; use RuntimeException; -use Swoole\Coroutine\CanceledException; use Throwable; class CallbackEvent extends Event @@ -79,6 +79,10 @@ public function run(Container $container): mixed /** @var array{result: mixed, exception: ?Throwable} $state */ $state = CoroutineContext::get($this->callbackContextKey()); + if ($exception !== null && Coroutine::causedByCancellation($exception)) { + throw $exception; + } + if ($state['exception'] !== null) { throw $state['exception']; } @@ -127,10 +131,12 @@ protected function execute(Container $container): int ]); return $result === false ? 1 : 0; - } catch (CanceledException $e) { - // Cancellation must unwind the mutex without running failure callbacks. - throw $e; } catch (Throwable $e) { + if (Coroutine::causedByCancellation($e)) { + // Cancellation must unwind the mutex without running failure callbacks. + throw $e; + } + CoroutineContext::set($this->callbackContextKey(), [ 'result' => null, 'exception' => $e, diff --git a/src/console/src/Scheduling/Event.php b/src/console/src/Scheduling/Event.php index a81c81b9b1..dbf9545da9 100644 --- a/src/console/src/Scheduling/Event.php +++ b/src/console/src/Scheduling/Event.php @@ -19,6 +19,7 @@ use Hypervel\Contracts\Debug\ExceptionHandler; use Hypervel\Contracts\Foundation\Application as ApplicationContract; use Hypervel\Contracts\Mail\Mailer; +use Hypervel\Coroutine\Coroutine; use Hypervel\Filesystem\Filesystem; use Hypervel\Log\Context\Repository; use Hypervel\Support\Arr; @@ -171,25 +172,31 @@ public function run(Container $container): mixed $exception = $throwable; } - if ($exitCode === null) { + if ($exitCode !== null) { try { - $this->removeMutex(); + $this->writeOutput($container); } catch (Throwable $throwable) { - $exception ??= $throwable; + $exception = $throwable; } + } - CoroutineContext::forget($this->processContextKey()); - } else { + if ($exitCode === null || ($exception !== null && Coroutine::causedByCancellation($exception))) { try { - $this->writeOutput($container); + $this->removeMutex(); } catch (Throwable $throwable) { - $exception = $throwable; + if (Coroutine::causedByCancellation($throwable) && ! Coroutine::causedByCancellation($exception)) { + $exception = $throwable; + } } + CoroutineContext::forget($this->processContextKey()); + } else { try { $this->finish($container, $exitCode); } catch (Throwable $throwable) { - $exception ??= $throwable; + if ($exception === null || (Coroutine::causedByCancellation($throwable) && ! Coroutine::causedByCancellation($exception))) { + $exception = $throwable; + } } } @@ -308,13 +315,17 @@ public function finish(Container $container, int $exitCode): void try { $this->callAfterCallbacks($container); } catch (Throwable $throwable) { - $exception ??= $throwable; + if ($exception === null || (Coroutine::causedByCancellation($throwable) && ! Coroutine::causedByCancellation($exception))) { + $exception = $throwable; + } } try { $this->removeMutex(); } catch (Throwable $throwable) { - $exception ??= $throwable; + if ($exception === null || (Coroutine::causedByCancellation($throwable) && ! Coroutine::causedByCancellation($exception))) { + $exception = $throwable; + } } CoroutineContext::forget($this->processContextKey()); diff --git a/src/container/composer.json b/src/container/composer.json index f55e4e2595..dad1dc6339 100644 --- a/src/container/composer.json +++ b/src/container/composer.json @@ -28,6 +28,7 @@ "hypervel/collections": "^0.4", "hypervel/context": "^0.4", "hypervel/contracts": "^0.4", + "hypervel/coroutine": "^0.4", "hypervel/reflection": "^0.4", "psr/container": "^2.0.1" }, diff --git a/src/container/src/SharedResolution.php b/src/container/src/SharedResolution.php index e9dddc62ca..82b07b3971 100644 --- a/src/container/src/SharedResolution.php +++ b/src/container/src/SharedResolution.php @@ -5,6 +5,8 @@ namespace Hypervel\Container; use Hypervel\Contracts\Container\BindingResolutionException; +use Hypervel\Coroutine\Coroutine; +use Hypervel\Coroutine\Exceptions\ForeignCancellationException; use Swoole\Coroutine\CanceledException; use Swoole\Coroutine\Channel; use Throwable; @@ -47,8 +49,11 @@ public function fail(Throwable $failure): void { // The owner's cancellation belongs to the owner. Waiters see an ordinary // failure, so their own requests and jobs do not stop as if canceled. - $this->failure = $failure instanceof CanceledException - ? new BindingResolutionException('Shared container resolution was canceled before completion.', previous: $failure) + $this->failure = Coroutine::causedByCancellation($failure) + ? new BindingResolutionException('Shared container resolution was canceled before completion.', previous: new ForeignCancellationException( + 'The coroutine resolving the shared service was canceled.', + previous: $failure, + )) : $failure; $this->settled = true; $this->signal->close(); diff --git a/src/coroutine/src/Coroutine.php b/src/coroutine/src/Coroutine.php index a32351e67f..2e78fdc430 100644 --- a/src/coroutine/src/Coroutine.php +++ b/src/coroutine/src/Coroutine.php @@ -8,6 +8,7 @@ use Hypervel\Container\Container; use Hypervel\Context\CoroutineContext; use Hypervel\Contracts\Debug\ExceptionHandler as ExceptionHandlerContract; +use Hypervel\Coroutine\Exceptions\ForeignCancellationException; use Hypervel\Engine\Coroutine as Co; use Hypervel\Engine\Exceptions\CoroutineDestroyedException; use Hypervel\Engine\Exceptions\RunningInNonCoroutineException; @@ -18,7 +19,7 @@ class Coroutine { public const string DETACHED_CONTEXT_KEY = '__coroutine.detached'; - protected static bool $enableReportException = true; + protected static bool $reportExceptions = true; /** * @var array @@ -84,17 +85,6 @@ public static function parentId(?int $coroutineId = null): int return Co::pid($coroutineId); } - /** - * Alias of Coroutine::parentId(). - * - * @throws RunningInNonCoroutineException When running in non-coroutine context - * @throws CoroutineDestroyedException When the coroutine has been destroyed - */ - public static function pid(?int $coroutineId = null): int - { - return Co::pid($coroutineId); - } - /** * Create a new coroutine. */ @@ -253,23 +243,32 @@ public static function list(): iterable } /** - * Enable or disable exception reporting in coroutines. - * - * Boot-only. The flag persists in a static property for the worker lifetime - * and applies to every coroutine spawned across the process. + * Determine if the given exception was caused by cancellation, including errors raised during cleanup. */ - public static function enableReportException(bool $enableReportException): void + public static function causedByCancellation(Throwable $exception): bool { - static::$enableReportException = $enableReportException; + do { + if ($exception instanceof ForeignCancellationException) { + return false; + } + + if ($exception instanceof CanceledException) { + return true; + } + } while (($exception = $exception->getPrevious()) !== null); + + return false; } /** - * Flush all static state. + * Enable or disable exception reporting in coroutines. + * + * Boot-only. The flag persists in a static property for the worker lifetime + * and applies to every coroutine spawned across the process. */ - public static function flushState(): void + public static function reportExceptions(bool $report = true): void { - static::$enableReportException = true; - static::$afterCreatedCallbacks = []; + static::$reportExceptions = $report; } /** @@ -283,7 +282,7 @@ protected static function printLog(Throwable $throwable): void throw $throwable; } - if (! static::$enableReportException) { + if (! static::$reportExceptions) { return; } @@ -320,4 +319,13 @@ protected static function reportUncaught(Throwable $throwable): void } catch (CanceledException) { } } + + /** + * Flush all static state. + */ + public static function flushState(): void + { + static::$reportExceptions = true; + static::$afterCreatedCallbacks = []; + } } diff --git a/src/coroutine/src/Exceptions/ChildCancellationException.php b/src/coroutine/src/Exceptions/ChildCancellationException.php index 3c5d31bf6e..4a880079d8 100644 --- a/src/coroutine/src/Exceptions/ChildCancellationException.php +++ b/src/coroutine/src/Exceptions/ChildCancellationException.php @@ -4,11 +4,9 @@ namespace Hypervel\Coroutine\Exceptions; -use RuntimeException; - /** * An independently canceled child whose owner remains active. */ -class ChildCancellationException extends RuntimeException +class ChildCancellationException extends ForeignCancellationException { } diff --git a/src/coroutine/src/Exceptions/ForeignCancellationException.php b/src/coroutine/src/Exceptions/ForeignCancellationException.php new file mode 100644 index 0000000000..164ccf9353 --- /dev/null +++ b/src/coroutine/src/Exceptions/ForeignCancellationException.php @@ -0,0 +1,14 @@ +getThrowable(); - if ($exception instanceof CanceledException) { + if (Coroutine::causedByCancellation($exception)) { throw new ChildCancellationException( 'A child coroutine managed by Waiter was canceled while its owner remained active.', previous: $exception, diff --git a/src/database/src/Pool/PooledConnection.php b/src/database/src/Pool/PooledConnection.php index 6b3657de14..5ee4bdc216 100644 --- a/src/database/src/Pool/PooledConnection.php +++ b/src/database/src/Pool/PooledConnection.php @@ -9,6 +9,7 @@ use Hypervel\Contracts\Container\Container; use Hypervel\Contracts\Events\Dispatcher; use Hypervel\Contracts\Log\StdoutLoggerInterface; +use Hypervel\Coroutine\Coroutine; use Hypervel\Coroutine\Waiter; use Hypervel\Database\Connection; use Hypervel\Database\ConnectionName; @@ -462,18 +463,19 @@ public function release(): void // Recorded after the listeners, which may still change a shared connection's grammar. $this->recordDateFormat(); - } catch (CanceledException $cancellation) { - $cancellationFailure = $cancellation; - $this->markInvalid(); } catch (Throwable $exception) { $this->markInvalid(); - try { - $this->logger->error('Release connection failed: ' . $exception); - } catch (CanceledException $loggingCancellation) { - $cancellationFailure = $loggingCancellation; - } catch (Throwable $loggingException) { - $ordinaryFailure = $loggingException; + if (Coroutine::causedByCancellation($exception)) { + $cancellationFailure = $exception; + } else { + try { + $this->logger->error('Release connection failed: ' . $exception); + } catch (CanceledException $loggingCancellation) { + $cancellationFailure = $loggingCancellation; + } catch (Throwable $loggingException) { + $ordinaryFailure = $loggingException; + } } } finally { $this->lease?->detach(); diff --git a/src/docs/coroutines.md b/src/docs/coroutines.md index 5a364cda5e..ffcc1fda2e 100644 --- a/src/docs/coroutines.md +++ b/src/docs/coroutines.md @@ -104,14 +104,12 @@ When code is running outside a coroutine, `Coroutine::id()` returns `-1`. Inside ### Parent Coroutine IDs -You may retrieve the parent coroutine ID using the `parentId` method or its `pid` alias: +You may retrieve the parent coroutine ID using the `parentId` method: ```php use Hypervel\Coroutine\Coroutine; $parentId = Coroutine::parentId(); - -$parentId = Coroutine::pid(); ``` When the current coroutine is a top-level coroutine, the parent ID is `0`. You may also pass a coroutine ID to inspect the parent of another coroutine: @@ -358,20 +356,24 @@ Hypervel uses `Swoole\Coroutine\CanceledException` as a terminal control-flow si Most application code does not need special cancellation handling. If low-level application or package code catches `Throwable` to report, retry, wrap, or replace an operation failure, it should let cancellation escape first: ```php -use Swoole\Coroutine\CanceledException; +use Hypervel\Coroutine\Coroutine; use Throwable; try { $result = performOperation(); -} catch (CanceledException $exception) { - throw $exception; } catch (Throwable $exception) { + if (Coroutine::causedByCancellation($exception)) { + throw $exception; + } + report($exception); return null; } ``` +`Coroutine::causedByCancellation` also recognizes cancellation when a `finally` block throws another exception while unwinding it. Rethrowing preserves any cleanup error for the caller to report. A cancellation wrapped in `Hypervel\Coroutine\Exceptions\ForeignCancellationException` came from another coroutine, such as the owner of a shared operation this code was waiting for. If the chain reaches that exception before it reaches a cancellation, `causedByCancellation` returns `false`, so you can handle it as an ordinary failure. `ChildCancellationException` extends it for independently canceled children. + Cleanup that does not wait should remain in `finally` as usual. Code that owns child coroutines should use `wait`, `parallel`, or `WaitConcurrent` so Hypervel can cancel active children when their parent is canceled. @@ -379,12 +381,12 @@ Cleanup that does not wait should remain in `finally` as usual. Code that owns c Hypervel catches unhandled exceptions thrown inside `Coroutine::create`, `go`, `co`, or `Coroutine::fork` and reports them through the application's exception handler when one is available. -You may disable this automatic reporting for the entire worker process using `enableReportException`: +You may disable this automatic reporting for the entire worker process using `reportExceptions`: ```php use Hypervel\Coroutine\Coroutine; -Coroutine::enableReportException(false); +Coroutine::reportExceptions(false); ``` > [!WARNING] @@ -544,7 +546,7 @@ Copied values follow the same rules as [`go` and `Coroutine::fork`](#copying-cor If the closure throws an exception, `wait` rethrows it in the waiting coroutine after the child's deferred callbacks have finished. -If the child is canceled independently while its waiting parent remains active, `wait` throws `Hypervel\Coroutine\Exceptions\ChildCancellationException`. The native cancellation is available as the previous exception. +If the child is canceled independently while its waiting parent remains active, `wait` throws `Hypervel\Coroutine\Exceptions\ChildCancellationException`. Its previous exception is the native cancellation, or the cleanup error if the child's cleanup threw while unwinding cancellation. If the timeout is reached, Hypervel cancels the child by throwing `Swoole\Coroutine\CanceledException` inside it. Hypervel then gives the child up to 10 seconds to finish and run its deferred callbacks before throwing `Hypervel\Coroutine\Exceptions\WaitTimeoutException` in the waiting coroutine. @@ -692,7 +694,7 @@ try { } ``` -An independently canceled child is recorded as `Hypervel\Coroutine\Exceptions\ChildCancellationException`, with the native cancellation available as its previous exception. +An independently canceled child is recorded as `Hypervel\Coroutine\Exceptions\ChildCancellationException`. Its previous exception is the native cancellation, or the cleanup error if the child's cleanup threw while unwinding cancellation. ### Limiting Parallel Work diff --git a/src/docs/grpc.md b/src/docs/grpc.md index f43133ee2a..61495e44c9 100644 --- a/src/docs/grpc.md +++ b/src/docs/grpc.md @@ -277,6 +277,8 @@ $previousAttempts = $call->previousAttempts(); The `deadline` method returns a `Hypervel\Support\CarbonImmutable` instance or `null` when the client did not set a deadline. The `timeRemaining` method returns the remaining number of seconds, while `deadlineExceeded` determines whether the deadline has passed. +If the client's deadline passes while the service is waiting, for example on a query, an HTTP request or a sleep, Hypervel interrupts the wait by throwing `Swoole\Coroutine\CanceledException` and responds with `DeadlineExceeded`. Place cleanup in `finally` blocks and follow the [coroutine cancellation guidance](/docs/{{version}}/coroutines#coroutine-cancellation) when catching errors. If cleanup throws while the cancellation unwinds, the exception is reported and the client still receives `DeadlineExceeded`. + The `previousAttempts` method returns the number of completed retry attempts reported by the client. This can be useful for logging or idempotency handling. > [!NOTE] diff --git a/src/docs/porting-from-laravel.md b/src/docs/porting-from-laravel.md index 0b2a0df657..00df1e9722 100644 --- a/src/docs/porting-from-laravel.md +++ b/src/docs/porting-from-laravel.md @@ -865,7 +865,7 @@ Hypervel stores job batches in a relational database. Laravel's DynamoDB batch r Custom queue drivers whose job classes implement `Hypervel\Contracts\Queue\Job` directly must also implement `Hypervel\Contracts\Queue\JobLifecycleCallbacks` to run jobs that use `CountCrashesAsExceptions`; extending `Hypervel\Queue\Jobs\Job` provides it. See [counting worker crashes](/docs/{{version}}/queues#counting-worker-crashes) for the callback requirements. -With `Worker::$killOnTimeout = false`, a timeout interrupts job code with `Swoole\Coroutine\CanceledException`; catching `TimeoutExceededException` inside the job does not catch it. Use `finally` for cleanup and rethrow cancellation if caught. Failure callbacks and worker reporting still receive `TimeoutExceededException`. See [worker timeouts](/docs/{{version}}/queues#worker-timeouts). +Timeouts unwind yielding job code with `Swoole\Coroutine\CanceledException`, starting its `catch` and `finally` blocks. Use `finally` for cleanup and rethrow cancellation from broad `catch (Exception)` or `catch (Throwable)` handlers. The worker gives canceled code a bounded wait; if the job is still unwinding when that wait ends, it terminates without waiting for cleanup or running the job's failure handling. `Worker::$killOnTimeout = false` instead waits for the job to unwind while other jobs continue. Failure callbacks receive `TimeoutExceededException`. See [worker timeouts](/docs/{{version}}/queues#worker-timeouts) for recovery behavior and limits. If you extend Horizon's `RedisQueue`, do not rely on `push()` calling your `pushRaw()` override. Override the protected `preparePayloadForPush()` method to prepare normal pushes, or `handlePayloadPushed()` to run code after Redis accepts the job. Direct calls to `pushRaw()` remain supported. diff --git a/src/docs/queues.md b/src/docs/queues.md index 3a1407f3e7..772b87129c 100644 --- a/src/docs/queues.md +++ b/src/docs/queues.md @@ -2030,7 +2030,7 @@ php artisan queue:work --timeout=30 If the job exceeds its maximum attempts by continually timing out, it will be marked as failed. -Hypervel checks for timed out jobs using a coroutine timer. You may configure how often the worker checks running jobs using the `--monitor-interval` option: +Hypervel checks for timed out jobs using a coroutine timer. With PCNTL enabled, the worker can also recover when PHP code blocks that timer. See [worker timeouts](#worker-timeouts) for cancellation and recovery behavior. You may configure how often the worker checks running jobs using the `--monitor-interval` option: ```shell php artisan queue:work --monitor-interval=1 @@ -2056,7 +2056,7 @@ class ProcessPodcast implements ShouldQueue Sometimes, IO blocking processes such as sockets or outgoing HTTP connections may not respect your specified timeout. Therefore, when using these features, you should always attempt to specify a timeout using their APIs as well. For example, when using [Guzzle](https://docs.guzzlephp.org), you should always specify a connection and request timeout value. > [!WARNING] -> A job's "timeout" value should always be less than its ["retry after"](#job-expiration) value. Otherwise, the job may be re-attempted before it has actually finished executing or timed out. The `--timeout` option has no effect when the `queue:work` command is invoked with the `--once` option. The [PCNTL](https://www.php.net/manual/en/book.pcntl.php) PHP extension is recommended so workers can respond to process signals gracefully. +> A job's "timeout" value should always be less than its ["retry after"](#job-expiration) value, allowing time for detection and cleanup. Otherwise, the job may be re-attempted before it has actually finished executing or timed out. The `--timeout` option has no effect when the `queue:work` command is invoked with the `--once` option. The [PCNTL](https://www.php.net/manual/en/book.pcntl.php) PHP extension enables graceful process signals and timeout recovery when PHP code blocks the event loop. #### Failing on Timeout @@ -2079,7 +2079,7 @@ class ProcessPodcast implements ShouldQueue ``` > [!NOTE] -> By default, when a job times out, it consumes one attempt and is released back to the queue (if retries are allowed). However, if you configure the job to fail on timeout, it will not be retried, regardless of the value set for tries. +> A timeout consumes an attempt. By default, a job that remains reserved becomes available again after its reservation expires. `FailOnTimeout` marks the job as failed instead, regardless of the remaining attempts. If the worker must terminate before it can safely run failure handling, the reservation recovers normally; see [worker timeouts](#worker-timeouts). ### SQS FIFO and Fair Queues @@ -2309,6 +2309,8 @@ $this->release(10); $this->release(now()->plus(seconds: 10)); ``` +After releasing a job, return from its handler. If a worker-processed job throws after release, the worker reports the exception but does not fail or release that attempt again. Exceptions thrown after release do not count toward `maxExceptions`. The next delivery checks the job's attempt limit as usual. + #### Manually Failing a Job @@ -3075,7 +3077,13 @@ Hypervel monitors running jobs using a coroutine timer. The `--monitor-interval` php artisan queue:work --monitor-interval=1 ``` -By default, a timeout stops the worker process, including its other running jobs. To cancel only the timed-out job, set the following property during application boot: +By default, a timeout stops the worker process, including its other running jobs. The worker stops accepting new jobs, notifies the timed-out job's [`Interruptible` handler](#reacting-to-worker-signals), and cancels its coroutine with `Swoole\Coroutine\CanceledException`. It waits up to `max(1, monitorInterval) + 1` seconds (two seconds by default) for the canceled code to finish before applying the job's failure settings. While the event loop remains responsive, yielding failure callbacks may then run to completion; this cancellation wait does not limit the job's `failed` method. + +Use `finally` for cleanup and rethrow cancellation if caught. `CanceledException` extends `Exception`, so a broad `catch (Exception)` also catches it. If job code catches cancellation and completes, releases, or fails the attempt during cleanup, the worker preserves that outcome. If the coroutine remains active after the wait, the worker terminates without running failure handling concurrently with it. A remaining reservation becomes available again normally, but `FailOnTimeout`, ordinary exception counting, and failure callbacks cannot run on that path. + +When PHP code blocks the event loop, the PCNTL alarm suspends the blocking coroutine so the timeout monitor can run failure handling before terminating the worker. Recovery, including the job's `failed` method, has `max(1, monitorInterval) + 1` seconds to finish (two seconds by default). A second alarm terminates the process when that deadline expires, even if recovery blocks again. Native calls that block signal handling cannot be interrupted this way. Other jobs interrupted by process termination recover through their queue reservations. + +To cancel only the timed-out job and keep the worker running, set the following property during application boot: ```php use Hypervel\Queue\Worker; @@ -3083,10 +3091,10 @@ use Hypervel\Queue\Worker; Worker::$killOnTimeout = false; ``` -In this mode, the job receives a `Swoole\Coroutine\CanceledException`, allowing its `finally` blocks to release resources. The worker then handles the timeout using the job's normal retry, backoff, and failure settings, reporting a `Hypervel\Queue\TimeoutExceededException`. Other jobs continue running. Do not swallow cancellation: a job that continues running keeps its concurrency slot. Cancellation requires the job to yield; CPU-bound code cannot be interrupted by the coroutine timer. +In this mode, the job also receives `Swoole\Coroutine\CanceledException`. Once its code unwinds, the worker handles the timeout using the job's normal retry, backoff, and failure settings, including `FailOnTimeout`, and reports a `Hypervel\Queue\TimeoutExceededException`. Other jobs continue running. Once job execution finishes, the worker's failure handling and completion listeners finish without further timeout cancellation. A job that swallows cancellation and continues running keeps its concurrency slot. This mode does not use the PCNTL recovery alarm: cancellation requires the job to yield, so CPU-bound code cannot be interrupted by the coroutine timer. > [!WARNING] -> The `--timeout` value should always be at least several seconds shorter than your `retry_after` configuration value. This will ensure that a worker processing a frozen job is always terminated before the job is retried. If your `--timeout` option is longer than your `retry_after` configuration value, your jobs may be processed twice. +> Keep `--timeout` comfortably shorter than `retry_after`, with room for the monitor interval, cancellation, and failure cleanup. A timeout is not a guarantee that cleanup or process termination finishes before the reservation expires. Jobs may be delivered again after a worker stops, so make jobs safe to retry and set timeouts on external calls as well. ### Pausing and Resuming Queue Workers @@ -4146,7 +4154,7 @@ When an interrupting signal is delivered to running jobs, Hypervel dispatches a Queue workers also dispatch a `WorkerStopping` event before they stop. You may register a listener using `Queue::stopping` in the `boot` method of a service provider. Its `connectionName` and `queue` properties identify the worker, while `terminatesImmediately` is `true` when the process will be terminated as soon as the listeners return. In that case, listeners should not start cleanup that must finish after the listener returns. -When a job times out, the worker normally ends its own process with `SIGKILL`, so a process supervisor sees the signal rather than an exit status. If you need to control how the worker terminates, you may register a callback using `Hypervel\Queue\Worker::killUsing` in a service provider's `boot` method. The callback receives the worker's exit status after the `WorkerStopping` event is dispatched. If the callback returns, the worker is terminated as usual. +When a job times out, the worker normally ends its own process with `SIGKILL`, so a process supervisor sees the signal rather than an exit status. If you need to control how the worker terminates, you may register a callback using `Hypervel\Queue\Worker::killUsing` in a service provider's `boot` method. The callback receives the worker's exit status after the `WorkerStopping` event is dispatched. If the callback returns, the worker is terminated as usual. Exceptions from stopping listeners or the callback are reported and do not prevent termination. During [blocked-loop recovery](#worker-timeouts), emergency termination can occur before stopping listeners or the callback finish. ## Credits diff --git a/src/events/composer.json b/src/events/composer.json index 6f2e8c0938..80cc87861c 100644 --- a/src/events/composer.json +++ b/src/events/composer.json @@ -30,6 +30,7 @@ "hypervel/container": "^0.4", "hypervel/context": "^0.4", "hypervel/contracts": "^0.4", + "hypervel/coroutine": "^0.4", "hypervel/macroable": "^0.4", "hypervel/queue": "^0.4", "hypervel/reflection": "^0.4", diff --git a/src/events/src/Dispatcher.php b/src/events/src/Dispatcher.php index 6e15cd8f4a..2aae255214 100755 --- a/src/events/src/Dispatcher.php +++ b/src/events/src/Dispatcher.php @@ -26,6 +26,7 @@ use Hypervel\Contracts\Queue\ShouldQueue; use Hypervel\Contracts\Queue\ShouldQueueAfterCommit; use Hypervel\Contracts\Queue\ShouldQueueViaOutbox; +use Hypervel\Coroutine\Coroutine; use Hypervel\Database\DatabaseTransactionsManager; use Hypervel\Queue\Attributes\Backoff; use Hypervel\Queue\Attributes\Connection; @@ -49,7 +50,6 @@ use LogicException; use ReflectionClass; use ReflectionException; -use Swoole\Coroutine\CanceledException; use Throwable; use UnitEnum; @@ -505,14 +505,18 @@ protected function invokeListenersAndObservers(string $event, array $payload, bo { try { $result = $this->invokeListeners($event, $payload, $halt); - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $exception) { + if (Coroutine::causedByCancellation($exception)) { + throw $exception; + } + try { $this->invokeObservers($event, $payload); - } catch (CanceledException $cancellation) { - throw $cancellation; - } catch (Throwable) { + } catch (Throwable $observerException) { + if (Coroutine::causedByCancellation($observerException)) { + throw $observerException; + } + // An observer failure must not replace the active listener failure. } @@ -539,9 +543,11 @@ protected function invokeObservers(string $event, array $payload): void foreach ($this->getObservers($event) as $observer) { try { $observer($event, $payload); - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $exception) { + if (Coroutine::causedByCancellation($exception)) { + throw $exception; + } + $firstException ??= $exception; } } diff --git a/src/foundation/src/helpers.php b/src/foundation/src/helpers.php index a5a43c13dc..f9823da9ac 100644 --- a/src/foundation/src/helpers.php +++ b/src/foundation/src/helpers.php @@ -27,6 +27,7 @@ use Hypervel\Contracts\View\Factory as ViewFactory; use Hypervel\Contracts\View\View as ViewContract; use Hypervel\Cookie\CookieJar; +use Hypervel\Coroutine\Coroutine; use Hypervel\Foundation\Application; use Hypervel\Foundation\Bus\PendingClosureDispatch; use Hypervel\Foundation\Bus\PendingDispatch; @@ -51,7 +52,6 @@ use League\Uri\Contracts\UriInterface; use Psr\Log\LoggerInterface; use Psr\Log\LogLevel; -use Swoole\Coroutine\CanceledException; use Symfony\Component\HttpFoundation\Cookie; use Symfony\Component\HttpFoundation\Response; use Symfony\Component\HttpKernel\Exception\HttpException; @@ -789,9 +789,11 @@ function rescue(callable $callback, $rescue = null, $report = true) { try { return $callback(); - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $e) { + if (Coroutine::causedByCancellation($e)) { + throw $e; + } + if (value($report, $e)) { report($e); } diff --git a/src/grpc/composer.json b/src/grpc/composer.json index 47727fbdf1..c9cbfc5c8c 100644 --- a/src/grpc/composer.json +++ b/src/grpc/composer.json @@ -42,6 +42,7 @@ "hypervel/context": "^0.4", "hypervel/contracts": "^0.4", "hypervel/coordinator": "^0.4", + "hypervel/coroutine": "^0.4", "hypervel/database": "^0.4", "hypervel/engine": "^0.4", "hypervel/foundation": "^0.4", diff --git a/src/grpc/src/Client/BaseClient.php b/src/grpc/src/Client/BaseClient.php index 43b405adcb..4b4f9721b5 100644 --- a/src/grpc/src/Client/BaseClient.php +++ b/src/grpc/src/Client/BaseClient.php @@ -9,6 +9,7 @@ use Google\Protobuf\Internal\Message; use Hypervel\Container\Container; use Hypervel\Contracts\Engine\Http\V2\ClientFactoryInterface; +use Hypervel\Coroutine\Coroutine; use Hypervel\Grpc\ClientGrpcOperation; use Hypervel\Grpc\Compression; use Hypervel\Grpc\Exceptions\RpcException; @@ -501,9 +502,11 @@ private function startInitialAttempt( $connection->start($requestFactory, $state, $deadline); return [$state, $connection]; - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $throwable) { + if (Coroutine::causedByCancellation($throwable)) { + throw $throwable; + } + $operationHandle?->finish(new GrpcOperationResult( $throwable instanceof RpcException ? $throwable->status() : null, $throwable, diff --git a/src/grpc/src/Client/Call.php b/src/grpc/src/Client/Call.php index 5f0343b1af..7a0b34781e 100644 --- a/src/grpc/src/Client/Call.php +++ b/src/grpc/src/Client/Call.php @@ -6,6 +6,7 @@ use Closure; use Google\Protobuf\Internal\Message; +use Hypervel\Coroutine\Coroutine; use Hypervel\Database\ConnectionResolver; use Hypervel\Engine\Channel; use Hypervel\Grpc\Compression; @@ -308,9 +309,11 @@ protected function waitForUnaryResponse(): Message } $this->unaryResponse = $this->deserializePendingPayload(); - } catch (CanceledException $throwable) { - throw $throwable; } catch (Throwable $throwable) { + if (Coroutine::causedByCancellation($throwable)) { + throw $throwable; + } + if ($throwable instanceof ProtocolException) { $this->storeFailure($throwable); } @@ -677,13 +680,15 @@ private function retry(StreamState $state, Status $status): bool ++$this->attempts; return true; - } catch (CanceledException $throwable) { - if ($backoffCheckpoint !== null) { - $this->retryBackoff?->restore($backoffCheckpoint); + } catch (Throwable $throwable) { + if (Coroutine::causedByCancellation($throwable)) { + if ($backoffCheckpoint !== null) { + $this->retryBackoff?->restore($backoffCheckpoint); + } + + throw $throwable; } - throw $throwable; - } catch (Throwable $throwable) { if ($this->isLogicallyCanceled()) { if ($backoffCheckpoint !== null) { $this->retryBackoff?->restore($backoffCheckpoint); @@ -828,7 +833,7 @@ private function throwFailedCompletion(): bool */ private function storeFailure(Throwable $failure): void { - if ($failure instanceof CanceledException) { + if (Coroutine::causedByCancellation($failure)) { throw $failure; } diff --git a/src/grpc/src/Client/Connection.php b/src/grpc/src/Client/Connection.php index b0bd5716c3..6990e9939f 100644 --- a/src/grpc/src/Client/Connection.php +++ b/src/grpc/src/Client/Connection.php @@ -8,6 +8,7 @@ use Hypervel\Contracts\Engine\Http\V2\ClientFactoryInterface; use Hypervel\Contracts\Engine\Http\V2\ClientInterface; use Hypervel\Contracts\Engine\Http\V2\ResponseInterface; +use Hypervel\Coroutine\Exceptions\ForeignCancellationException; use Hypervel\Database\ConnectionResolver; use Hypervel\Engine\Channel; use Hypervel\Engine\Coroutine; @@ -498,7 +499,7 @@ private function receive(): void if (! $this->isClosed()) { try { $this->failConnection($this->connectionException( - $exception, + new ForeignCancellationException($exception->getMessage(), previous: $exception), 'The gRPC response receiver was canceled.', )); } catch (CanceledException) { @@ -815,7 +816,10 @@ private function terminateAfterCancellation( string $fallback, ): never { try { - $this->terminateWhileLocked($this->connectionException($cancellation, $fallback)); + $this->terminateWhileLocked($this->connectionException( + new ForeignCancellationException($cancellation->getMessage(), previous: $cancellation), + $fallback, + )); } catch (CanceledException) { // The caller's original cancellation remains the primary outcome. } diff --git a/src/grpc/src/GrpcOperationHandle.php b/src/grpc/src/GrpcOperationHandle.php index b058f343d5..c4d1f43cbc 100644 --- a/src/grpc/src/GrpcOperationHandle.php +++ b/src/grpc/src/GrpcOperationHandle.php @@ -4,8 +4,8 @@ namespace Hypervel\Grpc; +use Hypervel\Coroutine\Coroutine; use Hypervel\Grpc\Contracts\GrpcOperationObserver; -use Swoole\Coroutine\CanceledException; use Throwable; /** @@ -49,9 +49,11 @@ public function finish(GrpcOperationResult $result): void foreach ($this->observers as [$observer, $token]) { try { $observer->finished($this->operation, $token, $result); - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $throwable) { + if (Coroutine::causedByCancellation($throwable)) { + throw $throwable; + } + $observerFailure ??= $throwable; } } diff --git a/src/grpc/src/GrpcOperationRunner.php b/src/grpc/src/GrpcOperationRunner.php index 2f072528f5..c611020ec5 100644 --- a/src/grpc/src/GrpcOperationRunner.php +++ b/src/grpc/src/GrpcOperationRunner.php @@ -4,8 +4,8 @@ namespace Hypervel\Grpc; +use Hypervel\Coroutine\Coroutine; use Hypervel\Grpc\Contracts\GrpcOperationObserver; -use Swoole\Coroutine\CanceledException; use Throwable; class GrpcOperationRunner @@ -43,9 +43,11 @@ public function start(GrpcOperation $operation): GrpcOperationHandle foreach ($this->observers as $observer) { $startedObservers[] = [$observer, $observer->starting($operation)]; } - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $throwable) { + if (Coroutine::causedByCancellation($throwable)) { + throw $throwable; + } + (new GrpcOperationHandle($operation, $startedObservers))->finish( new GrpcOperationResult(null, $throwable, 0), ); diff --git a/src/grpc/src/GrpcServiceProvider.php b/src/grpc/src/GrpcServiceProvider.php index 811bfd55d7..946e87fe06 100644 --- a/src/grpc/src/GrpcServiceProvider.php +++ b/src/grpc/src/GrpcServiceProvider.php @@ -116,6 +116,7 @@ private function registerServerServices(array $server): void $this->app->singleton(HandleCall::class, fn ($app) => new HandleCall( $app->make(CallContextStore::class), $app->make(Timer::class), + $app->make(ExceptionMapper::class), $server['max_receive_message_size'], $server['compression'], )); diff --git a/src/grpc/src/Protocol/MessageSerializer.php b/src/grpc/src/Protocol/MessageSerializer.php index ede07d26a2..7e703a718a 100644 --- a/src/grpc/src/Protocol/MessageSerializer.php +++ b/src/grpc/src/Protocol/MessageSerializer.php @@ -5,6 +5,7 @@ namespace Hypervel\Grpc\Protocol; use Google\Protobuf\Internal\Message; +use Hypervel\Coroutine\Coroutine; use Hypervel\Grpc\Exceptions\ProtocolException; use InvalidArgumentException; use Swoole\Coroutine\CanceledException; @@ -93,9 +94,11 @@ public static function deserialize(array|callable $deserialize, string $payload) try { $message = $deserialize($payload); - } catch (CanceledException $throwable) { - throw $throwable; } catch (Throwable $throwable) { + if (Coroutine::causedByCancellation($throwable)) { + throw $throwable; + } + throw new ProtocolException( 'Unable to deserialize the gRPC message.', previous: $throwable, diff --git a/src/grpc/src/Server/Middleware/HandleCall.php b/src/grpc/src/Server/Middleware/HandleCall.php index c9526ac5e8..70d9d3232d 100644 --- a/src/grpc/src/Server/Middleware/HandleCall.php +++ b/src/grpc/src/Server/Middleware/HandleCall.php @@ -7,6 +7,7 @@ use Closure; use Google\Protobuf\Internal\Message; use Hypervel\Coordinator\Timer; +use Hypervel\Coroutine\Coroutine as FrameworkCoroutine; use Hypervel\Engine\Coroutine; use Hypervel\Grpc\Compression; use Hypervel\Grpc\Exceptions\ProtocolException; @@ -17,6 +18,7 @@ use Hypervel\Grpc\Protocol\MetadataCodec; use Hypervel\Grpc\Protocol\Timeout; use Hypervel\Grpc\Server\CallContextStore; +use Hypervel\Grpc\Server\ExceptionMapper; use Hypervel\Grpc\Server\GrpcStreamedResponse; use Hypervel\Grpc\Server\ResponseFactory; use Hypervel\Grpc\Server\ServerCallContext; @@ -27,6 +29,7 @@ use InvalidArgumentException; use LogicException; use Swoole\Coroutine\CanceledException; +use Throwable; /** * Decode and scope one supported inbound gRPC call. @@ -35,9 +38,13 @@ */ class HandleCall { + /** + * Create the inbound call handler. + */ public function __construct( private readonly CallContextStore $contexts, private readonly Timer $timer, + private readonly ExceptionMapper $exceptions, private readonly int $maxReceiveMessageSize, private readonly Compression $preferredCompression, ) { @@ -128,11 +135,15 @@ static function (bool $closing) use ($handlerId): void { try { $response = $next($request); - } catch (CanceledException $exception) { - if (! $context->deadlineExceeded()) { + } catch (Throwable $exception) { + if (! FrameworkCoroutine::causedByCancellation($exception) || ! $context->deadlineExceeded()) { throw $exception; } + if (! $exception instanceof CanceledException) { + $this->exceptions->report($exception); + } + return $this->deadlineFailure(); } diff --git a/src/grpc/src/Server/ResponseFactory.php b/src/grpc/src/Server/ResponseFactory.php index 0b3c4f83c8..d3e6d1c1c6 100644 --- a/src/grpc/src/Server/ResponseFactory.php +++ b/src/grpc/src/Server/ResponseFactory.php @@ -7,6 +7,7 @@ use Composer\InstalledVersions; use Generator; use Google\Protobuf\Internal\Message; +use Hypervel\Coroutine\Coroutine; use Hypervel\Grpc\Compression; use Hypervel\Grpc\Exceptions\RpcException; use Hypervel\Grpc\Metadata; @@ -575,8 +576,12 @@ private function mapStreamFailure( Throwable $throwable, ServerCallContext $context, ): RpcException { - if ($throwable instanceof CanceledException) { + if (Coroutine::causedByCancellation($throwable)) { if ($context->deadlineExceeded()) { + if (! $throwable instanceof CanceledException) { + $this->exceptions->report($throwable); + } + return $this->deadlineFailure(); } diff --git a/src/http/src/Client/PendingRequest.php b/src/http/src/Client/PendingRequest.php index c12e724a05..b0a5dc2b88 100644 --- a/src/http/src/Client/PendingRequest.php +++ b/src/http/src/Client/PendingRequest.php @@ -24,6 +24,7 @@ use GuzzleHttp\Utils; use Hypervel\Contracts\Container\Transient; use Hypervel\Contracts\Support\Arrayable; +use Hypervel\Coroutine\Coroutine; use Hypervel\Database\ConnectionResolver; use Hypervel\Http\Client\Destinations\CurlCapabilities; use Hypervel\Http\Client\Destinations\DestinationPolicy; @@ -49,7 +50,6 @@ use Psr\Http\Message\StreamInterface; use RuntimeException; use SensitiveParameter; -use Swoole\Coroutine\CanceledException; use Symfony\Component\VarDumper\VarDumper; use Throwable; use UnitEnum; @@ -1103,8 +1103,8 @@ function (&$response) use ($attempt, &$shouldRetry) { ); } catch (TransferException $e) { // Guzzle wraps failures raised by transport callbacks. - if ($e->getPrevious() instanceof CanceledException) { - throw $e->getPrevious(); + if (($previous = $e->getPrevious()) !== null && Coroutine::causedByCancellation($previous)) { + throw $previous; } if (($response = $this->responseFromException($e)) !== null) { @@ -1270,11 +1270,11 @@ protected function makePromise(string $method, string $url, array $options = [], return $this->runAfterResponseCallbacks($response); }) ->otherwise(function (Throwable $e) { - if ($e instanceof TransferException && $e->getPrevious() instanceof CanceledException) { - throw $e->getPrevious(); + if ($e instanceof TransferException && ($previous = $e->getPrevious()) !== null && Coroutine::causedByCancellation($previous)) { + throw $previous; } - if ($e instanceof CanceledException) { + if (Coroutine::causedByCancellation($e)) { throw $e; } @@ -1341,9 +1341,11 @@ protected function handlePromiseResponse( $this, $this->request?->toPsrRequest()->getMethod() ) : true); - } catch (CanceledException $exception) { - throw $exception; } catch (Exception $exception) { + if (Coroutine::causedByCancellation($exception)) { + throw $exception; + } + return $exception; } @@ -1359,9 +1361,11 @@ protected function handlePromiseResponse( && ($this->throwIfCallback === null || call_user_func($this->throwIfCallback, $response))) { try { $response->throw($this->throwCallback); - } catch (CanceledException $exception) { - throw $exception; } catch (Exception $exception) { + if (Coroutine::causedByCancellation($exception)) { + throw $exception; + } + return $exception; } } diff --git a/src/image/src/ImageSource.php b/src/image/src/ImageSource.php index 517f113873..2525681707 100644 --- a/src/image/src/ImageSource.php +++ b/src/image/src/ImageSource.php @@ -5,8 +5,9 @@ namespace Hypervel\Image; use Closure; +use Hypervel\Coroutine\Coroutine; +use Hypervel\Coroutine\Exceptions\ForeignCancellationException; use Hypervel\Coroutine\Locker; -use Swoole\Coroutine\CanceledException; use Throwable; /** @@ -66,13 +67,18 @@ public function contents(): string } $this->contents = $contents; - } catch (CanceledException $exception) { - // The owner's cancellation belongs to the owner. Every other reader sees an - // ordinary failure, since the resolver cannot be repeated after a partial read. - $this->exception = new ImageException('Image source resolution was canceled.', previous: $exception); - - throw $exception; } catch (Throwable $exception) { + if (Coroutine::causedByCancellation($exception)) { + // The resolver cannot be repeated after a partial read. Other readers + // receive its failure without inheriting the owner's cancellation. + $this->exception = new ImageException('Image source resolution was canceled.', previous: new ForeignCancellationException( + 'The coroutine resolving the image source was canceled.', + previous: $exception, + )); + + throw $exception; + } + $this->exception = $exception; } finally { $this->resolver = null; diff --git a/src/queue/README.md b/src/queue/README.md index 4e5fcf9641..63ccf48feb 100644 --- a/src/queue/README.md +++ b/src/queue/README.md @@ -9,7 +9,7 @@ Documentation: https://hypervel.org/docs/queues - Laravel's DynamoDB failed-job provider is not provided. Use database or file storage for failed jobs. - Custom job implementations must support `JobLifecycleCallbacks` to use `CountCrashesAsExceptions`, so cleanup completes before a concurrent retry starts. The base `Jobs\Job` already implements it; the `Job` contract is unchanged. See [counting worker crashes](https://hypervel.org/docs/queues#counting-worker-crashes). -- With `Worker::$killOnTimeout = false`, job code receives Swoole's `CanceledException`, not `TimeoutExceededException`. Use `finally` for cleanup; the worker reports the timeout as `TimeoutExceededException` and applies the normal retry and failure settings. +- Timeouts unwind yielding job code with Swoole's `CanceledException`, starting its `catch` and `finally` blocks. Use `finally` for cleanup and rethrow cancellation if caught. The worker stops after timeout handling by default, with a bounded wait for cleanup; `Worker::$killOnTimeout = false` keeps other jobs running. See [worker timeouts](https://hypervel.org/docs/queues#worker-timeouts) for cleanup and blocked-loop recovery. - The protected `enqueueUsing()` callback receives the queue that owns the operation as its first argument. This lets deferred work borrow a fresh pooled queue connection after a database transaction commits instead of retaining a connection for the lifetime of the transaction. - Positive per-message delays on SQS FIFO queues throw `LogicException`. Laravel silently omits the delay and sends the job immediately. - `RateLimited::releaseAfter(0)` requests an immediate retry. Laravel treats zero as absent and uses the limiter's computed retry delay. diff --git a/src/queue/src/AwsCredentialCache.php b/src/queue/src/AwsCredentialCache.php index 7b5e5c5b5d..35af2329a2 100644 --- a/src/queue/src/AwsCredentialCache.php +++ b/src/queue/src/AwsCredentialCache.php @@ -12,7 +12,7 @@ use Hypervel\Contracts\Cache\Lock; use Hypervel\Contracts\Cache\LockProvider; use Hypervel\Contracts\Cache\Repository; -use Swoole\Coroutine\CanceledException; +use Hypervel\Coroutine\Coroutine; use Throwable; class AwsCredentialCache @@ -80,11 +80,13 @@ public function resolve(string $key, callable $provider): PromiseInterface $lock = $store->lock($key . ':refresh', static::LOCK_SECONDS); $lock->block(static::LOCK_WAIT_SECONDS); - } catch (CanceledException $e) { - // The lock may have been acquired before the wait was canceled; - // releasing an unowned lock is a no-op. - $this->releaseAfterFailure($lock, $e); - } catch (Throwable) { + } catch (Throwable $e) { + if (Coroutine::causedByCancellation($e)) { + // The lock may have been acquired before the wait was canceled; + // releasing an unowned lock is a no-op. + $this->releaseAfterFailure($lock, $e); + } + continue; } @@ -121,9 +123,11 @@ protected function freshCredentials(string $key): ?CredentialsInterface && $credentials->getExpiration() - time() > static::REFRESH_WINDOW) { return $credentials; } - } catch (CanceledException $e) { - throw $e; - } catch (Throwable) { + } catch (Throwable $e) { + if (Coroutine::causedByCancellation($e)) { + throw $e; + } + // An unavailable store falls back to the next store or a direct fetch. } } @@ -184,9 +188,11 @@ protected function put(string $key, CredentialsInterface $credentials, int $ttl) foreach ($this->repositories() as $repository) { try { $repository()->put($key, $credentials, $ttl); - } catch (CanceledException $e) { - throw $e; - } catch (Throwable) { + } catch (Throwable $e) { + if (Coroutine::causedByCancellation($e)) { + throw $e; + } + // An unavailable store must not prevent credential resolution. } } @@ -200,9 +206,11 @@ protected function forget(string $key): void foreach ($this->repositories() as $repository) { try { $repository()->forget($key); - } catch (CanceledException $e) { - throw $e; - } catch (Throwable) { + } catch (Throwable $e) { + if (Coroutine::causedByCancellation($e)) { + throw $e; + } + // An unavailable store must not prevent credential resolution. } } @@ -225,9 +233,11 @@ protected function release(?Lock $lock): void { try { $lock?->release(); - } catch (CanceledException $e) { - throw $e; - } catch (Throwable) { + } catch (Throwable $e) { + if (Coroutine::causedByCancellation($e)) { + throw $e; + } + // The lock expires on its own if it cannot be released. } } @@ -242,11 +252,11 @@ protected function releaseAfterFailure(?Lock $lock, Throwable $failure): never { try { $lock?->release(); - } catch (CanceledException $cleanupCancellation) { - if (! $failure instanceof CanceledException) { - throw $cleanupCancellation; + } catch (Throwable $cleanupException) { + if (Coroutine::causedByCancellation($cleanupException) && ! Coroutine::causedByCancellation($failure)) { + throw $cleanupException; } - } catch (Throwable) { + // Preserve the primary failure. } diff --git a/src/queue/src/CoroutineQueue.php b/src/queue/src/CoroutineQueue.php index 61c084161f..c093c65f99 100644 --- a/src/queue/src/CoroutineQueue.php +++ b/src/queue/src/CoroutineQueue.php @@ -137,8 +137,8 @@ function (bool $isClosing = false) use ($payload, $queue, $snapshot): void { protected function executePayload(string $payload, ?string $queue = null): int { $this->scheduleExecution(function () use ($payload, $queue): void { - // Report failures while the job's context is still active, then - // restore the dispatching coroutine. + // Only native cancellation is silent. Report cleanup errors through the + // configured callback while the job's log context is active. $this->withLogContext(null, function () use ($payload, $queue): void { try { $this->processPayload($payload, $queue); diff --git a/src/queue/src/FailoverQueue.php b/src/queue/src/FailoverQueue.php index f0ccad5d81..4d233abedf 100644 --- a/src/queue/src/FailoverQueue.php +++ b/src/queue/src/FailoverQueue.php @@ -12,13 +12,13 @@ use Hypervel\Contracts\Queue\IndexAwareQueue; use Hypervel\Contracts\Queue\Job as JobContract; use Hypervel\Contracts\Queue\Queue as QueueContract; +use Hypervel\Coroutine\Coroutine; use Hypervel\Database\DatabaseTransactionRecord; use Hypervel\Database\DatabaseTransactionsManager; use Hypervel\Log\Context\Repository as ContextRepository; use Hypervel\Queue\Events\QueueFailedOver; use Hypervel\Support\Collection; use RuntimeException; -use Swoole\Coroutine\CanceledException; use Throwable; use UnitEnum; use WeakReference; @@ -296,9 +296,11 @@ protected function attemptOnConnections( $forwarded, fn () => $this->manager->connection($connection)->{$method}(...$arguments), ); - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $e) { + if (Coroutine::causedByCancellation($e)) { + throw $e; + } + $lastException = $e; $failedQueues[] = $connection; diff --git a/src/queue/src/Jobs/Job.php b/src/queue/src/Jobs/Job.php index f5bbb0593d..e6c5b3bac2 100644 --- a/src/queue/src/Jobs/Job.php +++ b/src/queue/src/Jobs/Job.php @@ -11,6 +11,7 @@ use Hypervel\Contracts\Events\Dispatcher; use Hypervel\Contracts\Queue\Job as JobContract; use Hypervel\Contracts\Queue\JobLifecycleCallbacks; +use Hypervel\Coroutine\Coroutine; use Hypervel\ObjectPool\Lease; use Hypervel\Queue\Events\JobFailed; use Hypervel\Queue\InvalidPayloadException; @@ -19,7 +20,6 @@ use Hypervel\Support\InteractsWithTime; use JsonException; use RuntimeException; -use Swoole\Coroutine\CanceledException; use Throwable; abstract class Job implements JobContract, JobLifecycleCallbacks @@ -324,10 +324,10 @@ public function fail(?Throwable $e = null): void try { $batchRepository->rollBack(); - } catch (CanceledException $exception) { - throw $exception; - } catch (Throwable) { - // ... + } catch (Throwable $exception) { + if (Coroutine::causedByCancellation($exception)) { + throw $exception; + } } } @@ -351,8 +351,8 @@ public function fail(?Throwable $e = null): void if ($this->payloadException === null) { $this->failed($e); } - } catch (CanceledException $exception) { - $canceled = true; + } catch (Throwable $exception) { + $canceled = Coroutine::causedByCancellation($exception); throw $exception; } finally { diff --git a/src/queue/src/Middleware/FailOnException.php b/src/queue/src/Middleware/FailOnException.php index dffda04e4d..28c6528ad9 100644 --- a/src/queue/src/Middleware/FailOnException.php +++ b/src/queue/src/Middleware/FailOnException.php @@ -5,7 +5,7 @@ namespace Hypervel\Queue\Middleware; use Closure; -use Swoole\Coroutine\CanceledException; +use Hypervel\Coroutine\Coroutine; use Throwable; class FailOnException @@ -60,10 +60,13 @@ public function handle(mixed $job, callable $next): mixed { try { return $next($job); - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $e) { - if (call_user_func($this->callback, $e, $job) === true) { + if (Coroutine::causedByCancellation($e)) { + throw $e; + } + + if (call_user_func($this->callback, $e, $job) === true + && ! ($job->job ?? null)?->isReleased()) { $job->fail($e); } diff --git a/src/queue/src/Middleware/ThrottlesExceptions.php b/src/queue/src/Middleware/ThrottlesExceptions.php index 237bac9464..fa352698fa 100644 --- a/src/queue/src/Middleware/ThrottlesExceptions.php +++ b/src/queue/src/Middleware/ThrottlesExceptions.php @@ -6,9 +6,9 @@ use Closure; use Hypervel\Container\Container; +use Hypervel\Coroutine\Coroutine; use Hypervel\RateLimiter\Limit; use Hypervel\RateLimiter\RateLimiter; -use Swoole\Coroutine\CanceledException; use Throwable; use UnitEnum; @@ -108,9 +108,14 @@ public function handle(mixed $job, callable $next): mixed $next($job); $limiter->clear($policy); - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $throwable) { + $queueJob = $job->job ?? null; + + if (Coroutine::causedByCancellation($throwable) + || $queueJob?->hasFailed() || $queueJob?->isDeletedOrReleased()) { + throw $throwable; + } + if ($this->whenCallback && ! call_user_func($this->whenCallback, $throwable, $limiter)) { throw $throwable; } diff --git a/src/queue/src/SyncQueue.php b/src/queue/src/SyncQueue.php index 900a3bc690..a38a118aaa 100644 --- a/src/queue/src/SyncQueue.php +++ b/src/queue/src/SyncQueue.php @@ -11,6 +11,7 @@ use Hypervel\Contracts\Events\Dispatcher as EventDispatcher; use Hypervel\Contracts\Queue\Job as JobContract; use Hypervel\Contracts\Queue\Queue as QueueContract; +use Hypervel\Coroutine\Coroutine; use Hypervel\Database\DatabaseTransactionsManager; use Hypervel\Queue\Events\JobAttempted; use Hypervel\Queue\Events\JobExceptionOccurred; @@ -18,7 +19,6 @@ use Hypervel\Queue\Events\JobProcessing; use Hypervel\Queue\Jobs\SyncJob; use Hypervel\Support\Collection; -use Swoole\Coroutine\CanceledException; use Throwable; use UnitEnum; @@ -249,17 +249,19 @@ protected function processPayload(string $payload, ?string $queue = null): int } $this->raiseAfterJobEvent($queueJob, $duration); - } catch (CanceledException $exception) { - $canceled = true; - - throw $exception; } catch (Throwable $e) { + if (Coroutine::causedByCancellation($e)) { + $canceled = true; + + throw $e; + } + $exceptionOccurred = $e; try { $this->handleException($queueJob, $e); - } catch (CanceledException $exception) { - $canceled = true; + } catch (Throwable $exception) { + $canceled = Coroutine::causedByCancellation($exception); throw $exception; } diff --git a/src/queue/src/Worker.php b/src/queue/src/Worker.php index 55fc440bc4..4acbad0fde 100644 --- a/src/queue/src/Worker.php +++ b/src/queue/src/Worker.php @@ -15,8 +15,10 @@ use Hypervel\Contracts\Queue\JobLifecycleCallbacks; use Hypervel\Contracts\Queue\Queue as QueueContract; use Hypervel\Coordinator\Timer; +use Hypervel\Coroutine\Coroutine as FrameworkCoroutine; use Hypervel\Coroutine\WaitConcurrent; use Hypervel\Coroutine\Waiter; +use Hypervel\Coroutine\WaitGroup; use Hypervel\Database\DetectsLostConnections; use Hypervel\Database\LostConnectionException; use Hypervel\Database\QueryException; @@ -108,24 +110,56 @@ class Worker /** * The timeout job IDs. * - * @var array + * @var array */ protected array $timeoutJobIds = []; + /** + * Notifications that may still publish a timed-out attempt's outcome. + * + * @var array + */ + protected array $timeoutNotifications = []; + /** * The job monitor's ID. */ protected ?int $monitorId = null; /** - * The timer that owns the timeout monitor. + * The independent event-loop liveness timer. */ - protected Timer $timer; + protected ?int $livenessTimerId = null; + + /** + * The last time the event loop made progress. + */ + protected float $lastLoopProgressAt = 0; + + /** + * Indicates whether this daemon owns the timeout alarm. + */ + protected bool $ownsTimeoutAlarm = false; + + /** + * The deadline currently armed with PCNTL. + */ + protected ?float $timeoutAlarmAt = null; + + /** + * Indicates that blocked-loop recovery has committed to terminating the worker. + */ + protected bool $recoveringBlockedLoop = false; + + /** + * The coroutine frozen by emergency recovery. + */ + protected ?int $parkedCoroutineId = null; /** - * Indicates if the job monitor is checking for timeout jobs. + * The timer that owns the timeout monitor. */ - protected bool $monitorLocked = false; + protected Timer $timer; /** * The number of jobs completed by the worker. @@ -301,10 +335,17 @@ function () use ($connectionName, $queue, $options): ?int { $popWaiter = $this->createPopWaiter(); $concurrent = new WaitConcurrent($options->concurrency); - $this->monitorTimeoutJobs($options, $connectionName, $queue); - try { + $this->startTimeoutAlarm($options); + $this->monitorTimeoutJobs($options, $connectionName, $queue); + while (true) { + if ($this->recoveringBlockedLoop || (static::$killOnTimeout && $this->hasTimeoutJobs())) { + $this->sleep(self::SHUTDOWN_WAIT_SECONDS); + + continue; + } + $this->drainPendingSignals($lifecycleWaiter); // Before reserving any jobs, we will make sure this queue is not paused and @@ -391,11 +432,24 @@ function () use ($connectionName, $queue, $options): ?JobContract { return $job; }, )); + if ($this->recoveringBlockedLoop || (static::$killOnTimeout && $this->hasTimeoutJobs())) { + // Leave an already reserved job for its normal reservation expiry. + continue; + } + if ($job) { ++$jobsAdmitted; $concurrent->create(function () use ($job, $connectionName, $options): void { try { $this->runJob($job, $connectionName, $options); + } catch (Throwable $exception) { + if (! FrameworkCoroutine::causedByCancellation($exception)) { + throw $exception; + } + + if (! $exception instanceof CanceledException && static::$reportJobExceptions) { + $this->withCoroutineContext($options, fn () => $this->reportException($exception)); + } } finally { ++$this->jobsProcessed; $this->lastJobProcessedAt = $this->currentTime(); @@ -439,6 +493,18 @@ function () use ($connectionName, $queue, $options): ?JobContract { try { $concurrent->cancel(); } finally { + if ($this->ownsTimeoutAlarm && ! $this->recoveringBlockedLoop) { + pcntl_alarm(0); + pcntl_signal(SIGALRM, SIG_DFL); + $this->ownsTimeoutAlarm = false; + $this->timeoutAlarmAt = null; + } + + if ($this->livenessTimerId !== null) { + $this->timer->clear($this->livenessTimerId); + $this->livenessTimerId = null; + } + if ($this->monitorId !== null) { $this->timer->clear($this->monitorId); $this->monitorId = null; @@ -464,7 +530,15 @@ protected function waitForRunningJobs(WaitConcurrent $concurrent): void do { $this->drainPendingSignals($lifecycleWaiter); - } while (! $concurrent->wait(self::SHUTDOWN_WAIT_SECONDS)); + $finished = $concurrent->wait(self::SHUTDOWN_WAIT_SECONDS); + + // A timeout can arrive while a normal shutdown waits for these jobs. + // Keep the monitor alive until it publishes the timeout outcome and exits. + if ($finished && static::$killOnTimeout && $this->hasTimeoutJobs()) { + $this->sleep(self::SHUTDOWN_WAIT_SECONDS); + $finished = false; + } + } while (! $finished); $this->drainPendingSignals($lifecycleWaiter); } @@ -531,59 +605,212 @@ protected function monitorTimeoutJobs( $this->monitorId = $this->timer->tick($options->monitorInterval, function () use ($options, $connectionName, $queue): void { $this->withCoroutineContext($options, function () use ($options, $connectionName, $queue): void { - if ($this->monitorLocked) { - return; + $this->terminateTimeoutJobs($options); + + if (static::$killOnTimeout && $this->hasTimeoutJobs()) { + $this->shouldQuit = true; + $this->kill( + $connectionName, + $queue, + static::$timedOutExitCode ?? static::EXIT_ERROR, + $options, + WorkerStopReason::TimedOut, + ); } + }); + }); + } - $this->monitorLocked = true; + /** + * Allow a full monitor interval plus whole-second alarm granularity. + */ + protected function timeoutGracePeriod(WorkerOptions $options): int + { + return max(1, $options->monitorInterval) + 1; + } - try { - $this->terminateTimeoutJobs($options); + /** + * Recover kill-mode timeouts when PHP code prevents the monitor from running. + */ + protected function startTimeoutAlarm(WorkerOptions $options): void + { + if (! static::$killOnTimeout || ! $this->supportsAsyncSignals()) { + return; + } - if (static::$killOnTimeout && $this->hasTimeoutJobs()) { - $this->shouldQuit = true; - $this->kill( - $connectionName, - $queue, - static::$timedOutExitCode ?? static::EXIT_ERROR, - $options, - WorkerStopReason::TimedOut, - ); - } - } finally { - $this->monitorLocked = false; - } - }); + $this->ownsTimeoutAlarm = true; + $this->lastLoopProgressAt = $this->currentTime(); + pcntl_signal(SIGALRM, fn () => $this->handleTimeoutAlarm($options)); + + if (! pcntl_sigprocmask(SIG_UNBLOCK, [SIGALRM])) { + throw new RuntimeException('Unable to unblock the queue worker timeout alarm.'); + } + + // The timeout monitor can itself wait on yielding failure callbacks. + // Track the loop independently so those callbacks retain time to finish. + $this->livenessTimerId = $this->timer->tick(max(1, $options->monitorInterval), function (): void { + $this->lastLoopProgressAt = $this->currentTime(); }); } + /** + * Arm an earlier timeout check without scanning jobs on the admission path. + */ + protected function armTimeoutAlarm(float $deadline): void + { + if (! $this->ownsTimeoutAlarm || $this->recoveringBlockedLoop + || ($this->timeoutAlarmAt !== null && $this->timeoutAlarmAt <= $deadline)) { + return; + } + + $this->timeoutAlarmAt = $deadline; + pcntl_alarm(max(1, (int) ceil($deadline - $this->currentTime()))); + } + + /** + * Give the normal timeout handler a bounded chance to recover a blocked loop. + */ + protected function handleTimeoutAlarm(WorkerOptions $options): void + { + $this->timeoutAlarmAt = null; + $deadline = null; + + foreach ($this->runningJobs as $job) { + if ($job['expires_at'] !== null) { + $deadline = $deadline === null ? $job['expires_at'] : min($deadline, $job['expires_at']); + } + } + + foreach ($this->timeoutJobIds as $expiresAt) { + $deadline = $deadline === null ? $expiresAt : min($deadline, $expiresAt); + } + + if ($deadline === null) { + return; + } + + $grace = $this->timeoutGracePeriod($options); + $now = $this->currentTime(); + if ($now < $deadline + $grace || $now < $this->lastLoopProgressAt + $grace) { + $this->armTimeoutAlarm(max($deadline + $grace, $this->lastLoopProgressAt + $grace)); + + return; + } + + $this->recoveringBlockedLoop = true; + $this->parkedCoroutineId = Coroutine::id(); + + // PHP cannot dispatch another signal callback while this one is parked. + // The kernel's default action still terminates a second stall or blocked cleanup. + pcntl_signal(SIGALRM, SIG_DFL); + pcntl_alarm($grace); + + try { + if (Coroutine::id() > 0) { + Coroutine::yield(); + } + } finally { + // Never return to the interrupted job if it is resumed or canceled. + $this->terminateProcess(static::$timedOutExitCode ?? static::EXIT_ERROR); + } + } + /** * Scan the running jobs and terminate the timeout jobs. */ protected function terminateTimeoutJobs(WorkerOptions $options): void { $currentTime = $this->currentTime(); + $expiredJobs = []; + foreach ($this->runningJobs as $jobId => $job) { - // Timeout handling can yield while a later job in this snapshot finishes. - if (! isset($this->runningJobs[$jobId])) { + // Earlier timeout handling may yield while another job finishes. + if (! isset($this->runningJobs[$jobId]) || isset($this->timeoutJobIds[$jobId])) { continue; } if ($job['expires_at'] !== null && $job['expires_at'] <= $currentTime) { - $this->timeoutJobIds[$jobId] = true; - unset($this->runningJobs[$jobId]); - - try { - $this->handleTimeoutJob($job['job'], $options, $job['processing_marker']); - } finally { - // A completed body may still own a coroutine running deferred cleanup. - if (! static::$killOnTimeout && isset($this->timeoutJobIds[$jobId])) { - // @TODO: Pass TimeoutExceededException as the cancellation reason if Swoole adds cancellation reasons. - Coroutine::cancelById($job['coroutine_id'], throwException: true); - } + $this->timeoutJobIds[$jobId] = $job['expires_at']; + + if (! static::$killOnTimeout) { + $this->interruptTimeoutJob($jobId, $job); + + continue; } + + $job['parked'] = $job['coroutine_id'] === $this->parkedCoroutineId; + $expiredJobs[$jobId] = $job; + } + } + + // A parked attempt cannot resume. Handle it before spending its kernel + // deadline waiting for other attempts to unwind. + foreach ($expiredJobs as $jobId => $job) { + if ($job['parked']) { + $this->notifyTimeoutJob($job['job']); + $this->handleTimeoutJob($job['job'], $options, $job['processing_marker']); + unset($expiredJobs[$jobId]); + } + } + + $coroutineIds = []; + + foreach ($expiredJobs as $jobId => $job) { + $this->interruptTimeoutJob($jobId, $job); + + if ($job['coroutine_id'] !== $this->parkedCoroutineId && Coroutine::exists($job['coroutine_id'])) { + $coroutineIds[] = $job['coroutine_id']; } } + + if ($coroutineIds !== []) { + Coroutine::join($coroutineIds, $this->timeoutGracePeriod($options)); + } + + foreach ($expiredJobs as $job) { + // Cancellation can leave finally blocks running. Never race their + // release or failure operations after an unsuccessful wait. + if (Coroutine::exists($job['coroutine_id'])) { + $this->raiseTimedOutJobEvent($job['job'], $options); + } else { + $this->handleTimeoutJob($job['job'], $options, $job['processing_marker']); + } + } + } + + /** + * Interrupt an expired body without canceling a completed body's deferred cleanup. + */ + protected function interruptTimeoutJob(string $jobId, array $job): void + { + // Create the barrier before notification can yield, so a body finishing meanwhile waits for it. + $notification = static::$killOnTimeout ? null : ($this->timeoutNotifications[$jobId] = new WaitGroup(1)); + + try { + $this->notifyTimeoutJob($job['job']); + } finally { + unset($this->timeoutNotifications[$jobId]); + $notification?->done(); + + if (isset($this->runningJobs[$jobId]) + && $job['coroutine_id'] !== $this->parkedCoroutineId + && Coroutine::exists($job['coroutine_id'])) { + // @TODO: Pass TimeoutExceededException as the cancellation reason if Swoole adds cancellation reasons. + Coroutine::cancelById($job['coroutine_id'], throwException: true); + } + } + } + + /** + * Notify a job before cancellation unwinds its running command. + */ + protected function notifyTimeoutJob(JobContract $job): void + { + try { + $this->notifyJobOfSignal($job, SIGALRM); + } catch (Throwable $exception) { + $this->reportTimeoutException($exception); + } } /** @@ -595,46 +822,91 @@ protected function hasTimeoutJobs(): bool } /** - * Handle a job that exceeded its timeout. + * Handle a timed-out job whose execution has ended or is frozen. */ protected function handleTimeoutJob(JobContract $job, WorkerOptions $options, ?JobProcessingMarker $processingMarker = null): void { try { - $this->notifyJobOfSignal($job, SIGALRM); + if (! $job->hasFailed() && ! $job->isDeletedOrReleased()) { + $this->markJobAsFailedIfWillExceedMaxAttempts( + $job->getConnectionName(), + $job, + (int) $options->maxTries, + $e = $this->timeoutExceededException($job) + ); + + $this->markJobAsFailedIfWillExceedMaxExceptions( + $job->getConnectionName(), + $job, + $e + ); + + $this->markJobAsFailedIfItShouldFailOnTimeout( + $job->getConnectionName(), + $job, + $e + ); + } } catch (Throwable $exception) { - $this->exceptions->report($exception); + $this->reportTimeoutException($exception); } - $this->markJobAsFailedIfWillExceedMaxAttempts( - $job->getConnectionName(), - $job, - (int) $options->maxTries, - $e = $this->timeoutExceededException($job) - ); + try { + $processingMarker?->clear(); + } catch (Throwable $exception) { + $this->reportTimeoutException($exception); + } - // Non-killing timeouts are counted by the job's normal exception handler after unwind. - if (static::$killOnTimeout) { - $this->markJobAsFailedIfWillExceedMaxExceptions( - $job->getConnectionName(), - $job, - $e - ); + $this->raiseTimedOutJobEvent($job, $options); + } - $processingMarker?->clear(); + /** + * Raise the event for an attempt that exceeded its timeout. + */ + protected function raiseTimedOutJobEvent(JobContract $job, WorkerOptions $options): void + { + try { + if ($this->events->hasListeners(JobTimedOut::class)) { + $this->events->dispatch(new JobTimedOut( + $job->getConnectionName(), + $job, + $this->timeoutForJob($job, $options) + )); + } + } catch (Throwable $exception) { + $this->reportTimeoutException($exception); } + } - $this->markJobAsFailedIfItShouldFailOnTimeout( - $job->getConnectionName(), - $job, - $e - ); + /** + * Report a timeout-handling error without abandoning other expired attempts. + */ + protected function reportTimeoutException(Throwable $exception): void + { + if (FrameworkCoroutine::causedByCancellation($exception)) { + throw $exception; + } - if ($this->events->hasListeners(JobTimedOut::class)) { - $this->events->dispatch(new JobTimedOut( - $job->getConnectionName(), - $job, - $this->timeoutForJob($job, $options) - )); + $this->reportException($exception); + } + + /** + * Report an exception, writing it to the PHP error log if reporting fails. + */ + protected function reportException(Throwable $exception): void + { + try { + $this->exceptions->report($exception); + } catch (Throwable $reportingFailure) { + // A failed logger can rethrow the error we were already handling. + if ($reportingFailure !== $exception && FrameworkCoroutine::causedByCancellation($reportingFailure)) { + throw $reportingFailure; + } + + try { + error_log((string) $exception . ($reportingFailure === $exception ? '' : PHP_EOL . (string) $reportingFailure)); + } catch (Throwable) { + } } } @@ -779,10 +1051,12 @@ protected function getNextJob(QueueContract $connection, string $queue): ?JobCon return $job; } } - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $e) { - $this->exceptions->report($e); + if (FrameworkCoroutine::causedByCancellation($e)) { + throw $e; + } + + $this->reportException($e); $this->stopWorkerIfLostConnection($e); @@ -839,11 +1113,13 @@ protected function runJob(JobContract $job, string $connectionName, WorkerOption return $this->withCoroutineContext($options, function () use ($job, $connectionName, $options) { try { $this->process($connectionName, $job, $options); - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $e) { + if (FrameworkCoroutine::causedByCancellation($e)) { + throw $e; + } + if (static::$reportJobExceptions) { - $this->exceptions->report($e); + $this->reportException($e); } $this->stopWorkerIfLostConnection($e); @@ -964,11 +1240,34 @@ public function process(string $connectionName, JobContract $job, WorkerOptions $job->fire(); } finally { $duration = round(($this->currentTime() - $startedAt) * 1000, 2); + + // Non-kill timeouts interrupt execution, not the worker's subsequent + // failure handling or completion listeners. + if (! static::$killOnTimeout) { + unset($this->runningJobs[$runningJobId]); + + // The notification may still release or delete this attempt. + if ($notification = $this->timeoutNotifications[$runningJobId] ?? null) { + try { + $notification->wait(); + } catch (CanceledException $exception) { + $canceled = true; + + throw $exception; + } + } + } } - // The monitor has already handled this timeout; do not report normal completion. + // A declared timeout must not also report normal completion. if (isset($this->timeoutJobIds[$runningJobId])) { $exceptionOccurred = $this->timeoutExceededException($job); + $canceled = static::$killOnTimeout; + + if (! static::$killOnTimeout && ! $job->isDeletedOrReleased() && ! $job->hasFailed()) { + // Catching cancellation does not replace the timeout's retry or failure policy. + throw $exceptionOccurred; + } return; } @@ -982,26 +1281,46 @@ public function process(string $connectionName, JobContract $job, WorkerOptions )); } } catch (Throwable $e) { - if ($e instanceof CanceledException) { - if (static::$killOnTimeout || $runningJobId === null || ! isset($this->timeoutJobIds[$runningJobId])) { + if ($canceled) { + throw $e; + } + + if ($runningJobId !== null && isset($this->timeoutJobIds[$runningJobId])) { + if (static::$killOnTimeout) { $canceled = true; throw $e; } + if (! $e instanceof CanceledException && ! $e instanceof TimeoutExceededException && static::$reportJobExceptions) { + try { + $this->reportException($e); + } catch (Throwable $exception) { + $canceled = true; + + throw $exception; + } + } + $e = $this->timeoutExceededException($job); + } elseif (FrameworkCoroutine::causedByCancellation($e)) { + $canceled = true; + + throw $e; } $exceptionOccurred = $e; try { $this->handleJobException($connectionName, $job, $options, $e); - } catch (CanceledException $exception) { - $canceled = true; + } catch (Throwable $exception) { + $canceled = FrameworkCoroutine::causedByCancellation($exception); throw $exception; } } finally { + $timedOut = $runningJobId !== null && isset($this->timeoutJobIds[$runningJobId]); + if ($runningJobId) { unset($this->runningJobs[$runningJobId]); @@ -1013,6 +1332,10 @@ public function process(string $connectionName, JobContract $job, WorkerOptions if (! $canceled) { $processingMarker?->clear(); + if ($timedOut && ! static::$killOnTimeout) { + $this->raiseTimedOutJobEvent($job, $options); + } + if ($this->events->hasListeners(JobAttempted::class)) { $this->events->dispatch(new JobAttempted( $connectionName, @@ -1023,7 +1346,7 @@ public function process(string $connectionName, JobContract $job, WorkerOptions } if ($invalidPayloadException !== null && static::$reportJobExceptions) { - $this->exceptions->report($invalidPayloadException); + $this->reportException($invalidPayloadException); } } } @@ -1091,6 +1414,10 @@ protected function registerCoroutineJob(JobContract $job, WorkerOptions $options : null, ]; + if ($this->runningJobs[$jobId]['expires_at'] !== null) { + $this->armTimeoutAlarm($this->runningJobs[$jobId]['expires_at'] + $this->timeoutGracePeriod($options)); + } + return $jobId; } @@ -1107,7 +1434,9 @@ protected function handleJobException(string $connectionName, JobContract $job, // First, we will go ahead and mark the job as failed if it will exceed the maximum // attempts it is allowed to run the next time we process it. If so we will just // go ahead and mark it as failed now so we do not have to release this again. - if (! $job->hasFailed()) { + // A released attempt no longer owns the reservation; the next delivery + // enforces its limits without publishing a second outcome here. + if (! $job->hasFailed() && ! $job->isDeletedOrReleased()) { $this->markJobAsFailedIfWillExceedMaxAttempts( $connectionName, $job, @@ -1126,6 +1455,10 @@ protected function handleJobException(string $connectionName, JobContract $job, $job, $e ); + + if ($e instanceof TimeoutExceededException) { + $this->markJobAsFailedIfItShouldFailOnTimeout($connectionName, $job, $e); + } } $this->raiseExceptionOccurredJobEvent( @@ -1133,8 +1466,8 @@ protected function handleJobException(string $connectionName, JobContract $job, $job, $e ); - } catch (CanceledException $exception) { - $canceled = true; + } catch (Throwable $exception) { + $canceled = FrameworkCoroutine::causedByCancellation($exception); throw $exception; } finally { @@ -1577,25 +1910,34 @@ public function kill( ?WorkerOptions $options = null, ?WorkerStopReason $reason = null, ): never { - if ($this->events->hasListeners(WorkerStopping::class)) { - $this->events->dispatch(new WorkerStopping( - $connectionName, - $queue, - $status, - $options, - $reason, - $this->jobsProcessed, - $this->lastJobProcessedAt, - $this->currentMemoryUsage(), - terminatesImmediately: true, - )); - } - - if (static::$killCallback) { - call_user_func(static::$killCallback, $status); + try { + if ($this->events->hasListeners(WorkerStopping::class)) { + $this->events->dispatch(new WorkerStopping( + $connectionName, + $queue, + $status, + $options, + $reason, + $this->jobsProcessed, + $this->lastJobProcessedAt, + $this->currentMemoryUsage(), + terminatesImmediately: true, + )); + } + } catch (Throwable $exception) { + $this->reportException($exception); + } finally { + try { + if (static::$killCallback) { + call_user_func(static::$killCallback, $status); + } + } catch (Throwable $exception) { + $this->reportException($exception); + } finally { + // Neither shutdown hooks nor their error reporting may keep the worker alive. + $this->terminateProcess($status); + } } - - $this->terminateProcess($status); } /** diff --git a/src/saloon/src/Http/Auth/TokenAcquisition.php b/src/saloon/src/Http/Auth/TokenAcquisition.php index 2c66060182..db1b441220 100644 --- a/src/saloon/src/Http/Auth/TokenAcquisition.php +++ b/src/saloon/src/Http/Auth/TokenAcquisition.php @@ -5,10 +5,10 @@ namespace Hypervel\Saloon\Http\Auth; use Closure; +use Hypervel\Coroutine\Coroutine; use Hypervel\Coroutine\Locker; use Hypervel\Database\ConnectionResolver; use Hypervel\Saloon\Data\TokenAcquisitionResult; -use Swoole\Coroutine\CanceledException; use Throwable; class TokenAcquisition @@ -50,10 +50,10 @@ public function run(string $key, Closure $acquire): mixed $result->completed = true; return $result->value; - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $exception) { - $result->error = $exception; + if (! Coroutine::causedByCancellation($exception)) { + $result->error = $exception; + } throw $exception; } finally { diff --git a/src/saloon/src/Http/Pool.php b/src/saloon/src/Http/Pool.php index f508752610..94ee36abce 100644 --- a/src/saloon/src/Http/Pool.php +++ b/src/saloon/src/Http/Pool.php @@ -12,7 +12,6 @@ use Hypervel\Saloon\Exceptions\InvalidPoolItemException; use Hypervel\Saloon\Exceptions\PoolException; use InvalidArgumentException; -use Swoole\Coroutine\CanceledException; use Throwable; use function Hypervel\Coroutine\run; @@ -217,28 +216,29 @@ protected function orchestrate(bool $collectResponses): array try { // Throwable responses are failures, so the exception handler receives every failed request. $response = $this->connector->send($request)->throw(); - } catch (CanceledException $exception) { - $failures[$inputPosition] = [$key, new ChildCancellationException( - 'A child coroutine running a Saloon pool request was canceled while its owner remained active.', - previous: $exception, - )]; - - return; } catch (Throwable $exception) { + if (Coroutine::causedByCancellation($exception)) { + $failures[$inputPosition] = [$key, new ChildCancellationException( + 'A child coroutine running a Saloon pool request was canceled while its owner remained active.', + previous: $exception, + )]; + + return; + } + if ($this->exceptionHandler === null) { $failures[$inputPosition] = [$key, $exception]; } else { try { ($this->exceptionHandler)($exception, $key); - } catch (CanceledException $callbackException) { - $failures[$inputPosition] = [$key, $exception]; - $callbackFailures[$inputPosition] = [$key, new ChildCancellationException( - 'A child coroutine running a Saloon pool callback was canceled while its owner remained active.', - previous: $callbackException, - )]; } catch (Throwable $callbackException) { $failures[$inputPosition] = [$key, $exception]; - $callbackFailures[$inputPosition] = [$key, $callbackException]; + $callbackFailures[$inputPosition] = [$key, Coroutine::causedByCancellation($callbackException) + ? new ChildCancellationException( + 'A child coroutine running a Saloon pool callback was canceled while its owner remained active.', + previous: $callbackException, + ) + : $callbackException]; } } @@ -251,30 +251,35 @@ protected function orchestrate(bool $collectResponses): array try { $this->responseHandler?->__invoke($response, $key); - } catch (CanceledException $callbackException) { - $callbackFailures[$inputPosition] = [$key, new ChildCancellationException( - 'A child coroutine running a Saloon pool callback was canceled while its owner remained active.', - previous: $callbackException, - )]; } catch (Throwable $callbackException) { - $callbackFailures[$inputPosition] = [$key, $callbackException]; + $callbackFailures[$inputPosition] = [$key, Coroutine::causedByCancellation($callbackException) + ? new ChildCancellationException( + 'A child coroutine running a Saloon pool callback was canceled while its owner remained active.', + previous: $callbackException, + ) + : $callbackException]; } }); } - } catch (CanceledException $exception) { - $concurrent->cancel(); - throw $exception; } catch (Throwable $exception) { + if (Coroutine::causedByCancellation($exception)) { + $concurrent->cancel(); + + throw $exception; + } + $orchestrationFailure = $exception; } try { ConnectionResolver::releaseIdleConnections(); - } catch (CanceledException $exception) { - $concurrent->cancel(); - - throw $exception; } catch (Throwable $exception) { + if (Coroutine::causedByCancellation($exception)) { + $concurrent->cancel(); + + throw $exception; + } + $orchestrationFailure ??= $exception; } diff --git a/src/saloon/src/Pagination/Paginator.php b/src/saloon/src/Pagination/Paginator.php index 2832cdd8e0..94fc72f8cc 100644 --- a/src/saloon/src/Pagination/Paginator.php +++ b/src/saloon/src/Pagination/Paginator.php @@ -5,6 +5,7 @@ namespace Hypervel\Saloon\Pagination; use Countable; +use Hypervel\Coroutine\Coroutine; use Hypervel\Saloon\Exceptions\PoolException; use Hypervel\Saloon\Http\Connector; use Hypervel\Saloon\Http\Request; @@ -16,7 +17,6 @@ use InvalidArgumentException; use Iterator; use LogicException; -use Swoole\Coroutine\CanceledException; use Throwable; /** @@ -285,9 +285,11 @@ public function pool( if ($responseHandler !== null) { try { $responseHandler($firstResponse, $firstKey); - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $exception) { + if (Coroutine::causedByCancellation($exception)) { + throw $exception; + } + $initialCallbackFailure = $exception; } } diff --git a/src/scout/src/Console/ConcurrentImportRunner.php b/src/scout/src/Console/ConcurrentImportRunner.php index ea3d331ce6..5f2899cfb3 100644 --- a/src/scout/src/Console/ConcurrentImportRunner.php +++ b/src/scout/src/Console/ConcurrentImportRunner.php @@ -4,10 +4,10 @@ namespace Hypervel\Scout\Console; +use Hypervel\Coroutine\Coroutine; use Hypervel\Coroutine\Exceptions\ChildCancellationException; use Hypervel\Coroutine\WaitConcurrent; use Hypervel\Database\ConnectionResolver; -use Swoole\Coroutine\CanceledException; use Throwable; class ConcurrentImportRunner @@ -46,13 +46,13 @@ public function create(callable $operation): void // Return idle sessions before another child can take this concurrency slot. ConnectionResolver::releaseIdleConnections(); } - } catch (CanceledException $exception) { - $this->failure ??= new ChildCancellationException( - 'A child coroutine running a Scout import was canceled while its owner remained active.', - previous: $exception, - ); } catch (Throwable $exception) { - $this->failure ??= $exception; + $this->failure ??= Coroutine::causedByCancellation($exception) + ? new ChildCancellationException( + 'A child coroutine running a Scout import was canceled while its owner remained active.', + previous: $exception, + ) + : $exception; } }, [ConnectionResolver::DEFAULT_CONNECTION_CONTEXT_KEY]); @@ -71,12 +71,12 @@ public function wait(): void try { // Keep the parent's session while producing chunks, but not while joining children. ConnectionResolver::releaseIdleConnections(); - } catch (CanceledException $exception) { - $this->concurrent->cancel(); - - throw $exception; } catch (Throwable $exception) { - $this->concurrent->wait(); + if (Coroutine::causedByCancellation($exception)) { + $this->concurrent->cancel(); + } else { + $this->concurrent->wait(); + } throw $exception; } diff --git a/src/scout/src/EngineOperationRunner.php b/src/scout/src/EngineOperationRunner.php index 5864d82ded..fd328996df 100644 --- a/src/scout/src/EngineOperationRunner.php +++ b/src/scout/src/EngineOperationRunner.php @@ -5,8 +5,8 @@ namespace Hypervel\Scout; use Closure; +use Hypervel\Coroutine\Coroutine; use Hypervel\Scout\Contracts\EngineOperationObserver; -use Swoole\Coroutine\CanceledException; use Throwable; class EngineOperationRunner @@ -60,9 +60,11 @@ public function run(EngineOperation $operation, Closure $callback): mixed } $result = $callback(); - } catch (CanceledException $throwable) { - throw $throwable; } catch (Throwable $throwable) { + if (Coroutine::causedByCancellation($throwable)) { + throw $throwable; + } + $operationException = $throwable; } @@ -71,9 +73,11 @@ public function run(EngineOperation $operation, Closure $callback): mixed foreach ($startedObservers as [$observer, $token]) { try { $observer->finished($operation, $token, $operationException); - } catch (CanceledException $throwable) { - throw $throwable; } catch (Throwable $throwable) { + if (Coroutine::causedByCancellation($throwable)) { + throw $throwable; + } + $observerException ??= $throwable; } } diff --git a/src/support/src/helpers.php b/src/support/src/helpers.php index 41519bb6a5..ea54cf4a4a 100644 --- a/src/support/src/helpers.php +++ b/src/support/src/helpers.php @@ -5,6 +5,7 @@ use Carbon\CarbonInterval; use Hypervel\Contracts\Support\DeferringDisplayableValue; use Hypervel\Contracts\Support\Htmlable; +use Hypervel\Coroutine\Coroutine; use Hypervel\Database\ConnectionResolver; use Hypervel\Database\Eloquent\Model; use Hypervel\Support\Arr; @@ -17,7 +18,6 @@ use Hypervel\Support\Sleep; use Hypervel\Support\Str; use Hypervel\Support\Stringable as SupportStringable; -use Swoole\Coroutine\CanceledException; if (! function_exists('append_config')) { /** @@ -337,9 +337,11 @@ function retry($times, callable $callback, $sleepMilliseconds = 0, $when = null) try { return $callback($attempts); - } catch (CanceledException $exception) { - throw $exception; } catch (Throwable $e) { + if (Coroutine::causedByCancellation($e)) { + throw $e; + } + if ($times < 1 || ($when && ! $when($e))) { throw $e; } diff --git a/tests/Ai/BroadcastBatchingTest.php b/tests/Ai/BroadcastBatchingTest.php index de61638410..8630012af4 100644 --- a/tests/Ai/BroadcastBatchingTest.php +++ b/tests/Ai/BroadcastBatchingTest.php @@ -15,14 +15,19 @@ use Hypervel\Broadcasting\Channel as BroadcastChannel; use Hypervel\Context\CoroutineContext; use Hypervel\Coroutine\Coroutine; +use Hypervel\Coroutine\Exceptions\ChildCancellationException; +use Hypervel\Database\Pool\ConnectionLease; use Hypervel\Engine\Channel; use Hypervel\Engine\Coroutine as EngineCoroutine; use Hypervel\Support\Facades\Context; use Hypervel\Support\Facades\Event; use Hypervel\Support\Facades\Exceptions; +use Mockery as m; use PHPUnit\Framework\Attributes\DataProvider; +use PHPUnit\Framework\Attributes\TestWith; use RuntimeException; use Swoole\Coroutine\CanceledException; +use Throwable; class BroadcastBatchingTest extends TestCase { @@ -158,12 +163,13 @@ public static function senderFailures(): array } #[DataProvider('cancellationModes')] - public function testCancelingAProducerBlockedOnItsBacklogCancelsTheSender(bool $throwException): void + public function testCancelingAProducerBlockedOnItsBacklogCancelsTheSender(bool $throwException, bool $cleanupThrows): void { config(['broadcasting.default' => 'null', 'ai.broadcasting.batching.max_bytes' => 1]); $waiting = new Channel(1); $active = false; $caught = null; + $cleanupException = new RuntimeException('Producer cleanup failed.'); Event::listen(AnonymousEvent::class, static function () use ($waiting, &$active): void { $active = true; try { @@ -172,15 +178,21 @@ public function testCancelingAProducerBlockedOnItsBacklogCancelsTheSender(bool $ $active = false; } }); - $parent = EngineCoroutine::create(static function () use (&$caught): void { - $response = new StreamableAgentResponse('invocation', static function (): Generator { - for ($index = 0; $index < 256; ++$index) { - yield new TextDelta((string) $index, 'message', 'x', 1); + $parent = EngineCoroutine::create(static function () use (&$caught, $cleanupThrows, $cleanupException): void { + $response = new StreamableAgentResponse('invocation', static function () use ($cleanupThrows, $cleanupException): Generator { + try { + for ($index = 0; $index < 256; ++$index) { + yield new TextDelta((string) $index, 'message', 'x', 1); + } + } finally { + if ($cleanupThrows) { + throw $cleanupException; + } } }); try { $response->broadcast(new BroadcastChannel('conversation')); - } catch (CanceledException $exception) { + } catch (Throwable $exception) { $caught = $exception; } }); @@ -189,7 +201,12 @@ public function testCancelingAProducerBlockedOnItsBacklogCancelsTheSender(bool $ $this->assertTrue($active); $this->assertTrue(EngineCoroutine::cancelById($parent->getId(), $throwException)); Coroutine::join([$parent->getId()], 1); - $this->assertInstanceOf(CanceledException::class, $caught); + if ($cleanupThrows) { + $this->assertSame($cleanupException, $caught); + $this->assertInstanceOf(CanceledException::class, $caught->getPrevious()); + } else { + $this->assertInstanceOf(CanceledException::class, $caught); + } $this->assertFalse($active); } finally { $waiting->close(); @@ -201,11 +218,97 @@ public function testCancelingAProducerBlockedOnItsBacklogCancelsTheSender(bool $ } /** - * Cover both native coroutine cancellation modes. + * Cover both native cancellation modes with normal and throwing cleanup. */ public static function cancellationModes(): array { - return ['throw' => [true], 'return false' => [false]]; + return [ + 'throw' => [true, false], + 'return false' => [false, false], + 'throw with cleanup failure' => [true, true], + 'return false with cleanup failure' => [false, true], + ]; + } + + #[TestWith([false, false])] + #[TestWith([false, true])] + #[TestWith([true, false])] + #[TestWith([true, true])] + public function testCancellationWhileFinishingBroadcastsCancelsTheSender(bool $cleanupThrows, bool $producerFails): void + { + config(['broadcasting.default' => 'null', 'ai.broadcasting.batching.max_bytes' => 1]); + $failure = $cleanupThrows ? new RuntimeException('Release cleanup failed.', previous: new CanceledException) : new CanceledException; + $lease = m::mock(ConnectionLease::class); + $lease->shouldReceive('releaseIfIdle')->once()->andThrow($failure); + $waiting = new Channel(1); + $senderCanceled = false; + $senderId = null; + Event::listen(AnonymousEvent::class, static function () use ($waiting, &$senderCanceled, &$senderId): void { + $senderId = Coroutine::id(); + + try { + $waiting->pop(1); + } catch (CanceledException $exception) { + $senderCanceled = true; + throw $exception; + } + }); + $response = new StreamableAgentResponse('invocation', static function () use ($producerFails): Generator { + yield new TextDelta('text', 'message', 'Hello', 1); + if ($producerFails) { + throw new RuntimeException('Producer failed.'); + } + }); + $caught = null; + CoroutineContext::set('__ai.test.connection_lease', $lease); + + try { + $response->broadcast(new BroadcastChannel('conversation')); + } catch (Throwable $exception) { + $caught = $exception; + } finally { + CoroutineContext::forget('__ai.test.connection_lease'); + $waiting->close(); + if ($senderId !== null) { + Coroutine::join([$senderId], 1); + } + } + + $this->assertSame($failure, $caught); + $this->assertTrue($senderCanceled); + } + + #[TestWith([false])] + #[TestWith([true])] + public function testIndependentSenderCancellationIsAChildFailure(bool $cleanupThrows): void + { + config(['broadcasting.default' => 'null', 'ai.broadcasting.batching.max_bytes' => 1]); + $cleanupException = new RuntimeException('Sender cleanup failed.'); + Event::listen(AnonymousEvent::class, static function () use ($cleanupThrows, $cleanupException): void { + try { + EngineCoroutine::cancelById(Coroutine::id(), throwException: true); + } finally { + if ($cleanupThrows) { + Coroutine::sleep(0.001); + throw $cleanupException; + } + } + }); + $response = new StreamableAgentResponse('invocation', static fn (): array => [new TextDelta('text', 'message', 'Hello', 1)]); + + try { + $response->broadcast(new BroadcastChannel('conversation')); + $this->fail('The sender cancellation must reach the caller.'); + } catch (ChildCancellationException $exception) { + $this->assertFalse(Coroutine::causedByCancellation($exception)); + + if ($cleanupThrows) { + $this->assertSame($cleanupException, $exception->getPrevious()); + $this->assertInstanceOf(CanceledException::class, $cleanupException->getPrevious()); + } else { + $this->assertInstanceOf(CanceledException::class, $exception->getPrevious()); + } + } } public function testSenderFailureReachesTheCallerAndDisposesTheProducer(): void diff --git a/tests/Ai/Feature/FilesystemToolsTest.php b/tests/Ai/Feature/FilesystemToolsTest.php index d8029a6c64..5e27092bfe 100644 --- a/tests/Ai/Feature/FilesystemToolsTest.php +++ b/tests/Ai/Feature/FilesystemToolsTest.php @@ -33,7 +33,9 @@ use Hypervel\Tests\Ai\TestCase; use Mockery as m; use PHPUnit\Framework\Attributes\DataProvider; +use RuntimeException; use Swoole\Coroutine\CanceledException; +use Throwable; use function Hypervel\Ai\agent; use function Hypervel\Tests\Ai\Fixtures\fakeOpenAiResponse; @@ -62,17 +64,28 @@ public function testResolvedFilesystemToolsDoNotShareApprovalRequirements(): voi } #[DataProvider('cancellationPaths')] - public function testFilesystemCancellationIsNotConvertedToAToolResult(string $toolClass, string $method): void + public function testFilesystemCancellationIsNotConvertedToAToolResult(string $toolClass, string $method, bool $cleanupThrows): void { $cancellation = new CanceledException('Download canceled.'); + if ($cleanupThrows) { + $cancellation = new RuntimeException('Filesystem cleanup failed.', previous: $cancellation); + } $disk = m::mock(FilesystemAdapter::class); $disk->shouldReceive('fileExists')->andReturn(true); $disk->shouldReceive('size')->andReturn(10)->byDefault(); + $disk->shouldReceive('lastModified')->andReturn(1)->byDefault(); + $disk->shouldReceive('mimeType')->andReturn('text/plain')->byDefault(); + $disk->shouldReceive('getVisibility')->andReturn('public')->byDefault(); $disk->shouldReceive($method)->once()->andThrow($cancellation); + $caught = null; - $this->expectExceptionObject($cancellation); + try { + (new $toolClass($disk))->handle(new Request(['path' => 'file.txt', 'from' => 'file.txt', 'to' => 'copy.txt'])); + } catch (Throwable $exception) { + $caught = $exception; + } - (new $toolClass($disk))->handle(new Request(['path' => 'file.txt', 'from' => 'file.txt', 'to' => 'copy.txt'])); + $this->assertSame($cancellation, $caught); } /** @@ -80,14 +93,24 @@ public function testFilesystemCancellationIsNotConvertedToAToolResult(string $to */ public static function cancellationPaths(): array { - return [ + $paths = [ 'copy' => [CopyFile::class, 'copy'], 'move' => [MoveFile::class, 'move'], 'read metadata' => [ReadFile::class, 'size'], 'metadata size' => [GetFileMetadata::class, 'size'], - 'optional metadata' => [GetFileMetadata::class, 'lastModified'], + 'metadata modification time' => [GetFileMetadata::class, 'lastModified'], + 'metadata MIME type' => [GetFileMetadata::class, 'mimeType'], + 'metadata visibility' => [GetFileMetadata::class, 'getVisibility'], 'URL' => [GetFileUrl::class, 'url'], ]; + $cases = []; + + foreach ($paths as $name => $arguments) { + $cases[$name] = [...$arguments, false]; + $cases[$name . ' with cleanup failure'] = [...$arguments, true]; + } + + return $cases; } #[DataProvider('invalidSourcePaths')] diff --git a/tests/Ai/Feature/RememberConversationTest.php b/tests/Ai/Feature/RememberConversationTest.php index 05b71f7fa8..dcf0163926 100644 --- a/tests/Ai/Feature/RememberConversationTest.php +++ b/tests/Ai/Feature/RememberConversationTest.php @@ -49,6 +49,7 @@ use Hypervel\Tests\Ai\TestCase; use Mockery as m; use PHPUnit\Framework\Attributes\DataProvider; +use PHPUnit\Framework\Attributes\TestWith; use RuntimeException; use Swoole\Coroutine\CanceledException; use Throwable; @@ -561,7 +562,7 @@ public static function executionModes(): array } #[DataProvider('stoppedTitleModes')] - public function testRecordableStoppedStreamsDoNotWaitForOrStartTitleRequests(bool $concurrent, bool $abandoned): void + public function testRecordableStoppedStreamsDoNotWaitForOrStartTitleRequests(bool $concurrent, bool $abandoned, bool $cleanupThrows): void { config(['ai.conversations.generate_title' => true, 'ai.conversations.concurrent_title' => $concurrent]); $prompt = $this->prompt(); @@ -570,6 +571,9 @@ public function testRecordableStoppedStreamsDoNotWaitForOrStartTitleRequests(boo $active = false; $finished = false; $failure = new CanceledException('Stream consumer canceled'); + if ($cleanupThrows) { + $failure = new RuntimeException('Stream cleanup failed.', previous: $failure); + } $middleware = m::mock(RememberConversation::class, [new DatabaseConversationStore, $prompt->provider()]) ->makePartial()->shouldAllowMockingProtectedMethods(); @@ -630,9 +634,10 @@ public function testRecordableStoppedStreamsDoNotWaitForOrStartTitleRequests(boo public static function stoppedTitleModes(): array { return [ - 'serial abandonment' => [false, true], - 'concurrent abandonment' => [true, true], - 'serial cancellation' => [false, false], + 'serial abandonment' => [false, true, false], + 'concurrent abandonment' => [true, true, false], + 'serial cancellation' => [false, false, false], + 'serial cancellation with cleanup failure' => [false, false, true], ]; } @@ -867,11 +872,16 @@ public static function lateTitleOutcomes(): array return ['failed turn' => [true], 'completed turn' => [false]]; } - public function testTitleCancellationSavesTheCompletedTurnBeforeRethrowing(): void + #[TestWith([false])] + #[TestWith([true])] + public function testTitleCancellationSavesTheCompletedTurnBeforeRethrowing(bool $cleanupThrows): void { config(['ai.conversations.generate_title' => true]); $prompt = $this->prompt(); $failure = new CanceledException('Title request canceled'); + if ($cleanupThrows) { + $failure = new RuntimeException('Title cleanup failed.', previous: $failure); + } $loop = m::mock(TextGenerationLoop::class); $loop->shouldReceive('generate')->once()->andThrow($failure); $prompt->provider()->shouldReceive('textGenerationLoop')->once()->andReturn($loop); diff --git a/tests/Ai/Unit/Gateway/TextGenerationLoopTest.php b/tests/Ai/Unit/Gateway/TextGenerationLoopTest.php index 74e996b936..1503b9f9d8 100644 --- a/tests/Ai/Unit/Gateway/TextGenerationLoopTest.php +++ b/tests/Ai/Unit/Gateway/TextGenerationLoopTest.php @@ -68,7 +68,10 @@ use LogicException; use Mockery as m; use PHPUnit\Framework\Attributes\DataProvider; +use PHPUnit\Framework\Attributes\TestWith; +use RuntimeException; use Swoole\Coroutine\CanceledException; +use Throwable; class TextGenerationLoopTest extends TestCase { @@ -1650,16 +1653,21 @@ public function testAnUnavailableClaimPreventsToolExecution(): void ); } - public function testCancellationIsNotRecordedAsACompletedToolFailure(): void + #[TestWith([false])] + #[TestWith([true])] + public function testCancellationIsNotRecordedAsACompletedToolFailure(bool $cleanupThrows): void { $provider = textGenerationLoopProvider(); $gateway = m::mock(StepTextGateway::class); - $gateway->shouldReceive('generateTextStep')->andReturn(new StepResponse('Finished', [], FinishReason::Stop, new TextUsage, new Meta)); + $gateway->shouldNotReceive('generateTextStep'); $store = m::mock(ClaimsPendingApprovals::class); $claim = new ApprovalClaim('message', 'token'); - $store->shouldReceive('claimPendingApprovals')->once()->with('conversation', ['call'])->andReturn($claim); + $store->shouldReceive('claimPendingApprovals')->once()->with('conversation', ['call', 'next'])->andReturn($claim); $store->shouldNotReceive('recordApprovalResult'); $exception = new CanceledException('Canceled while executing the approved tool.'); + if ($cleanupThrows) { + $exception = new RuntimeException('Tool cleanup failed.', previous: $exception); + } $tool = $this->approvalTool(); $tool->shouldReceive('handle')->once()->andThrow($exception); $context = new RunContext('invocation', m::mock(Agent::class), $provider, 'model', new Dispatcher, approvalStore: $store, conversationId: 'conversation'); @@ -1669,13 +1677,13 @@ public function testCancellationIsNotRecordedAsACompletedToolFailure(): void $provider, 'model', '', - messages: [new AssistantMessage('', collect([new ToolCall('call', 'tool', [])]))], + messages: [new AssistantMessage('', collect([new ToolCall('call', 'tool', []), new ToolCall('next', 'tool', [])]))], tools: [$tool], - approval: ['call' => Decision::approve()], + approval: ['*' => Decision::approve()], context: $context, ); $this->fail('Cancellation was swallowed.'); - } catch (CanceledException $caught) { + } catch (Throwable $caught) { $this->assertSame($exception, $caught); } diff --git a/tests/Ai/Unit/Tools/AgentToolTest.php b/tests/Ai/Unit/Tools/AgentToolTest.php index 08a531a716..ddae4be6fa 100644 --- a/tests/Ai/Unit/Tools/AgentToolTest.php +++ b/tests/Ai/Unit/Tools/AgentToolTest.php @@ -20,6 +20,7 @@ use PHPUnit\Framework\Attributes\DataProvider; use RuntimeException; use Swoole\Coroutine\CanceledException; +use Throwable; class AgentToolTest extends TestCase { @@ -73,9 +74,12 @@ public function testAFailingSubAgentSurfacesItsErrorAsTheToolResultOnTheStreamin } #[DataProvider('deliveryModes')] - public function testCancellationEscapesTheSubAgent(bool $streaming): void + public function testCancellationEscapesTheSubAgent(bool $streaming, bool $cleanupThrows): void { $exception = new CanceledException('Sub-agent canceled.'); + if ($cleanupThrows) { + $exception = new RuntimeException('Sub-agent cleanup failed.', previous: $exception); + } $agent = m::mock(Agent::class); if ($streaming) { @@ -94,16 +98,21 @@ public function testCancellationEscapesTheSubAgent(bool $streaming): void try { $streaming ? iterator_to_array($tool->stream($request)) : $tool->handle($request); $this->fail('Cancellation must reach the parent agent.'); - } catch (CanceledException $caught) { + } catch (Throwable $caught) { $this->assertSame($exception, $caught); } } /** - * Provide synchronous and streaming delegation. + * Provide synchronous and streaming cancellation with normal and throwing cleanup. */ public static function deliveryModes(): array { - return ['prompt' => [false], 'stream' => [true]]; + return [ + 'prompt' => [false, false], + 'stream' => [true, false], + 'prompt cleanup failure' => [false, true], + 'stream cleanup failure' => [true, true], + ]; } } diff --git a/tests/Cache/CacheFailoverStoreTest.php b/tests/Cache/CacheFailoverStoreTest.php index a40fbef40e..aa7d776769 100644 --- a/tests/Cache/CacheFailoverStoreTest.php +++ b/tests/Cache/CacheFailoverStoreTest.php @@ -24,9 +24,11 @@ use Hypervel\Tests\TestCase; use Mockery as m; use PHPUnit\Framework\Attributes\DataProvider; +use PHPUnit\Framework\Attributes\TestWith; use RuntimeException; use stdClass; use Swoole\Coroutine\CanceledException; +use Throwable; class CacheFailoverStoreTest extends TestCase { @@ -308,9 +310,14 @@ public function testFlushLocksAttemptsEveryStoreAndPreservesTheEarliestFailure() $store->flushLocks(); } - public function testFlushLocksStopsAtCancellation(): void + #[TestWith([false])] + #[TestWith([true])] + public function testFlushLocksStopsAtCancellation(bool $cleanupThrows): void { $cancellation = new CanceledException('flush canceled'); + if ($cleanupThrows) { + $cancellation = new RuntimeException('Lock cleanup failed.', previous: $cancellation); + } $first = $this->lockStore(); $first->shouldReceive('supportsFlushingLocks')->once()->andReturnTrue(); $first->shouldReceive('flushLocks')->once()->andThrow($cancellation); @@ -325,14 +332,19 @@ public function testFlushLocksStopsAtCancellation(): void try { $store->flushLocks(); $this->fail('Flushing the lock stores was expected to be canceled.'); - } catch (CanceledException $exception) { + } catch (Throwable $exception) { $this->assertSame($cancellation, $exception); } } - public function testFirstSuccessOperationDoesNotFailOverAfterCancellation(): void + #[TestWith([false])] + #[TestWith([true])] + public function testFirstSuccessOperationDoesNotFailOverAfterCancellation(bool $cleanupThrows): void { $cancellation = new CanceledException('read canceled'); + if ($cleanupThrows) { + $cancellation = new RuntimeException('Read cleanup failed.', previous: $cancellation); + } $first = m::mock(Store::class); $first->shouldReceive('get')->once()->with('key')->andThrow(new RuntimeException('first failure')); $second = m::mock(Store::class); @@ -355,16 +367,21 @@ function (CacheFailedOver $event) use (&$failedStores): void { try { $store->get('key'); $this->fail('Reading from the failover store was expected to be canceled.'); - } catch (CanceledException $exception) { + } catch (Throwable $exception) { $this->assertSame($cancellation, $exception); } $this->assertSame(['first'], $failedStores); } - public function testEveryStoreOperationDoesNotContinueAfterCancellation(): void + #[TestWith([false])] + #[TestWith([true])] + public function testEveryStoreOperationDoesNotContinueAfterCancellation(bool $cleanupThrows): void { $cancellation = new CanceledException('forget canceled'); + if ($cleanupThrows) { + $cancellation = new RuntimeException('Forget cleanup failed.', previous: $cancellation); + } $first = m::mock(Store::class); $first->shouldReceive('forget')->once()->with('key')->andReturnTrue(); $second = m::mock(Store::class); @@ -380,7 +397,7 @@ public function testEveryStoreOperationDoesNotContinueAfterCancellation(): void try { $store->forget('key'); $this->fail('Forgetting from every failover store was expected to be canceled.'); - } catch (CanceledException $exception) { + } catch (Throwable $exception) { $this->assertSame($cancellation, $exception); } } diff --git a/tests/Cache/CacheStackStoreTest.php b/tests/Cache/CacheStackStoreTest.php index dd21b29705..f5f23204e5 100644 --- a/tests/Cache/CacheStackStoreTest.php +++ b/tests/Cache/CacheStackStoreTest.php @@ -21,8 +21,10 @@ use Mockery as m; use Mockery\MockInterface; use PHPUnit\Framework\Attributes\DataProvider; +use PHPUnit\Framework\Attributes\TestWith; use RuntimeException; use Swoole\Coroutine\CanceledException; +use Throwable; use function Hypervel\Support\defer; @@ -488,11 +490,16 @@ public function put(string $key, mixed $value, int $seconds): bool $this->assertSame('old', $stack->get('foo')); } - public function testPutPreservesOriginalCancellationAndCompletesCompensation(): void + #[TestWith([false])] + #[TestWith([true])] + public function testPutPreservesOriginalCancellationAndCompletesCompensation(bool $cleanupThrows): void { $top = m::mock(ArrayStore::class); $bottom = m::mock(ArrayStore::class); $cancellation = new CanceledException('write canceled'); + if ($cleanupThrows) { + $cancellation = new RuntimeException('Write cleanup failed.', previous: $cancellation); + } $cleanupCancellation = new CanceledException('cleanup canceled'); $top->shouldReceive('put')->once()->andReturnTrue(); @@ -503,16 +510,21 @@ public function testPutPreservesOriginalCancellationAndCompletesCompensation(): try { (new StackStore([$top, $bottom]))->put('foo', 'bar', 60); $this->fail('Expected the write cancellation to be thrown.'); - } catch (CanceledException $exception) { + } catch (Throwable $exception) { $this->assertSame($cancellation, $exception); } } - public function testPutCleanupCancellationSupersedesOrdinaryWriteFailure(): void + #[TestWith([false])] + #[TestWith([true])] + public function testPutCleanupCancellationSupersedesOrdinaryWriteFailure(bool $cleanupThrows): void { $top = m::mock(ArrayStore::class); $bottom = m::mock(ArrayStore::class); $cleanupCancellation = new CanceledException('cleanup canceled'); + if ($cleanupThrows) { + $cleanupCancellation = new RuntimeException('Release cleanup failed.', previous: $cleanupCancellation); + } $top->shouldReceive('put')->once()->andReturnTrue(); $bottom->shouldReceive('put')->once()->andThrow(new RuntimeException('write failed')); @@ -522,16 +534,21 @@ public function testPutCleanupCancellationSupersedesOrdinaryWriteFailure(): void try { (new StackStore([$top, $bottom]))->put('foo', 'bar', 60); $this->fail('Expected the cleanup cancellation to be thrown.'); - } catch (CanceledException $exception) { + } catch (Throwable $exception) { $this->assertSame($cleanupCancellation, $exception); } } - public function testPutCleanupCancellationSupersedesFalseWriteResult(): void + #[TestWith([false])] + #[TestWith([true])] + public function testPutCleanupCancellationSupersedesFalseWriteResult(bool $cleanupThrows): void { $top = m::mock(ArrayStore::class); $bottom = m::mock(ArrayStore::class); $cleanupCancellation = new CanceledException('cleanup canceled'); + if ($cleanupThrows) { + $cleanupCancellation = new RuntimeException('Release cleanup failed.', previous: $cleanupCancellation); + } $top->shouldReceive('put')->once()->andReturnTrue(); $bottom->shouldReceive('put')->once()->andReturnFalse(); @@ -540,7 +557,7 @@ public function testPutCleanupCancellationSupersedesFalseWriteResult(): void try { (new StackStore([$top, $bottom]))->put('foo', 'bar', 60); $this->fail('Expected the cleanup cancellation to be thrown.'); - } catch (CanceledException $exception) { + } catch (Throwable $exception) { $this->assertSame($cleanupCancellation, $exception); } } @@ -751,11 +768,16 @@ public function putMany(array $values, int $seconds): bool $this->assertSame([null, null], [$bottom->get('first'), $bottom->get('second')]); } - public function testPutManyCleanupContinuesAfterCancellationStopsAtOrdinaryFailureAndKeepsCancellation(): void + #[TestWith([false])] + #[TestWith([true])] + public function testPutManyCleanupContinuesAfterCancellationStopsAtOrdinaryFailureAndKeepsCancellation(bool $cleanupThrows): void { $top = m::mock(ArrayStore::class); $bottom = m::mock(ArrayStore::class); $cleanupCancellation = new CanceledException('cleanup canceled'); + if ($cleanupThrows) { + $cleanupCancellation = new RuntimeException('Release cleanup failed.', previous: $cleanupCancellation); + } $top->shouldReceive('putMany')->once()->andReturnTrue(); $bottom->shouldReceive('putMany')->once()->andThrow(new RuntimeException('write failed')); @@ -769,7 +791,7 @@ public function testPutManyCleanupContinuesAfterCancellationStopsAtOrdinaryFailu try { (new StackStore([$top, $bottom]))->putMany(['first' => 'one', 'second' => 'two', 'third' => 'three'], 60); $this->fail('Expected the cleanup cancellation to be thrown.'); - } catch (CanceledException $exception) { + } catch (Throwable $exception) { $this->assertSame($cleanupCancellation, $exception); } } @@ -957,12 +979,17 @@ public function testForgetAttemptsEveryLayerAndPreservesTheEarliestFailure(): vo } } - public function testForgetCancellationSupersedesEarlierFailureAndStopsFurtherLayers(): void + #[TestWith([false])] + #[TestWith([true])] + public function testForgetCancellationSupersedesEarlierFailureAndStopsFurtherLayers(bool $cleanupThrows): void { $top = m::mock(ArrayStore::class); $middle = m::mock(ArrayStore::class); $bottom = m::mock(ArrayStore::class); $cancellation = new CanceledException('forget canceled'); + if ($cleanupThrows) { + $cancellation = new RuntimeException('Forget cleanup failed.', previous: $cancellation); + } $top->shouldReceive('forget')->once()->with('foo')->andThrow(new RuntimeException('first failure')); $middle->shouldReceive('forget')->once()->with('foo')->andThrow($cancellation); @@ -971,7 +998,7 @@ public function testForgetCancellationSupersedesEarlierFailureAndStopsFurtherLay try { (new StackStore([$top, $middle, $bottom]))->forget('foo'); $this->fail('Expected the forget cancellation to be thrown.'); - } catch (CanceledException $exception) { + } catch (Throwable $exception) { $this->assertSame($cancellation, $exception); } } diff --git a/tests/ConnectionPool/ConnectionTest.php b/tests/ConnectionPool/ConnectionTest.php index b707e478d5..efb5f6a02c 100644 --- a/tests/ConnectionPool/ConnectionTest.php +++ b/tests/ConnectionPool/ConnectionTest.php @@ -19,6 +19,7 @@ use Hypervel\Tests\TestCase; use Mockery as m; use PHPUnit\Framework\Attributes\DataProvider; +use PHPUnit\Framework\Attributes\TestWith; use RuntimeException; use Swoole\Coroutine\CanceledException; use Throwable; @@ -143,15 +144,18 @@ public function testReleaseUsesTheCurrentEventDispatcher(): void $fake->assertDispatched(ConnectionReleasing::class, fn (ConnectionReleasing $event): bool => $event->connection === $connection); } - public function testReleaseListenerCancellationStillReturnsTheConnectionOnce(): void + #[TestWith([false])] + #[TestWith([true])] + public function testReleaseListenerCancellationStillReturnsTheConnectionOnce(bool $cleanupThrows): void { $cancellation = new CanceledException('listener canceled'); + $failure = $cleanupThrows ? new RuntimeException('Listener cleanup failed.', previous: $cancellation) : $cancellation; $container = m::mock(ContainerContract::class); $container->shouldReceive('has')->with(StdoutLoggerInterface::class)->once()->andReturnFalse(); $container->shouldReceive('bound')->with('events')->andReturnTrue(); $container->shouldReceive('make')->with('events')->andReturn($dispatcher = m::mock(Dispatcher::class)); $dispatcher->shouldReceive('hasListeners')->once()->with(ConnectionReleasing::class)->andReturnTrue(); - $dispatcher->shouldReceive('dispatch')->once()->andThrow($cancellation); + $dispatcher->shouldReceive('dispatch')->once()->andThrow($failure); $connection = new ActiveConnectionStub($container, $pool = m::mock(ConnectionPool::class)); $pool->shouldReceive('getOptions')->once()->andReturn(PoolOptions::fromArray(['events' => [ConnectionReleasing::class]])); $pool->shouldReceive('release')->once()->with($connection); @@ -159,21 +163,24 @@ public function testReleaseListenerCancellationStillReturnsTheConnectionOnce(): try { $connection->release(); $this->fail('The release listener was expected to be canceled.'); - } catch (CanceledException $exception) { - $this->assertSame($cancellation, $exception); + } catch (Throwable $exception) { + $this->assertSame($failure, $exception); } } - public function testReleaseListenerCancellationRemainsPrimaryOverCleanupCancellation(): void + #[TestWith([false])] + #[TestWith([true])] + public function testReleaseListenerCancellationRemainsPrimaryOverCleanupCancellation(bool $cleanupThrows): void { $cancellation = new CanceledException('listener canceled'); + $failure = $cleanupThrows ? new RuntimeException('Listener cleanup failed.', previous: $cancellation) : $cancellation; $cleanupCancellation = new CanceledException('release canceled'); $container = m::mock(ContainerContract::class); $container->shouldReceive('has')->with(StdoutLoggerInterface::class)->once()->andReturnFalse(); $container->shouldReceive('bound')->with('events')->andReturnTrue(); $container->shouldReceive('make')->with('events')->andReturn($dispatcher = m::mock(Dispatcher::class)); $dispatcher->shouldReceive('hasListeners')->once()->with(ConnectionReleasing::class)->andReturnTrue(); - $dispatcher->shouldReceive('dispatch')->once()->andThrow($cancellation); + $dispatcher->shouldReceive('dispatch')->once()->andThrow($failure); $connection = new ActiveConnectionStub($container, $pool = m::mock(ConnectionPool::class)); $pool->shouldReceive('getOptions')->once()->andReturn(PoolOptions::fromArray(['events' => [ConnectionReleasing::class]])); $pool->shouldReceive('release')->once()->with($connection)->andThrow($cleanupCancellation); @@ -181,8 +188,8 @@ public function testReleaseListenerCancellationRemainsPrimaryOverCleanupCancella try { $connection->release(); $this->fail('The release listener was expected to be canceled.'); - } catch (CanceledException $exception) { - $this->assertSame($cancellation, $exception); + } catch (Throwable $exception) { + $this->assertSame($failure, $exception); } } diff --git a/tests/Console/CommandMutexTest.php b/tests/Console/CommandMutexTest.php index 7fa51197c6..424eb8956c 100644 --- a/tests/Console/CommandMutexTest.php +++ b/tests/Console/CommandMutexTest.php @@ -19,12 +19,14 @@ use Hypervel\Tests\Console\Fixtures\FakeCacheFactory; use Mockery as m; use Mockery\MockInterface; +use PHPUnit\Framework\Attributes\TestWith; use RuntimeException; use Swoole\Coroutine\CanceledException; use Symfony\Component\Console\Input\ArrayInput; use Symfony\Component\Console\Input\InputInterface; use Symfony\Component\Console\Output\NullOutput; use Symfony\Component\Console\Output\OutputInterface; +use Throwable; class CommandMutexTest extends TestCase { @@ -216,7 +218,9 @@ public function __invoke(): void } } - public function testAfterExecuteCancellationSupersedesAnOrdinaryCommandFailureAndStillReleasesTheMutex(): void + #[TestWith([false])] + #[TestWith([true])] + public function testAfterExecuteCancellationSupersedesAnOrdinaryCommandFailureAndStillReleasesTheMutex(bool $cleanupThrows): void { $command = new class extends Command implements Isolatable { public function __invoke(): void @@ -226,24 +230,35 @@ public function __invoke(): void }; $command->setHypervel($this->app); $cancellation = new CanceledException('canceled'); + $cleanup = $cleanupThrows ? new RuntimeException('Observer cleanup failed.') : null; $commandMutex = $this->mockCommandMutex(); $commandMutex->shouldReceive('create')->once()->andReturnTrue(); $commandMutex->shouldReceive('forget')->once()->andReturnTrue(); $this->app->make(Dispatcher::class)->listen( AfterExecute::class, - fn () => throw $cancellation, + static function () use ($cancellation, $cleanup): never { + try { + throw $cancellation; + } finally { + if ($cleanup !== null) { + throw $cleanup; + } + } + }, ); try { $command->run(new ArrayInput(['--isolated' => true]), new NullOutput); $this->fail('Expected the after-execute observer to preserve cancellation.'); - } catch (CanceledException $exception) { - $this->assertSame($cancellation, $exception); + } catch (Throwable $exception) { + $this->assertSame($cleanup ?? $cancellation, $exception); } } - public function testMutexCancellationSupersedesAnOrdinaryCommandFailure(): void + #[TestWith([false])] + #[TestWith([true])] + public function testMutexCancellationSupersedesAnOrdinaryCommandFailure(bool $cleanupThrows): void { $command = new class extends Command implements Isolatable { public function __invoke(): void @@ -253,16 +268,25 @@ public function __invoke(): void }; $command->setHypervel($this->app); $cancellation = new CanceledException('canceled'); + $cleanup = $cleanupThrows ? new RuntimeException('Mutex cleanup failed.') : null; $commandMutex = $this->mockCommandMutex(); $commandMutex->shouldReceive('create')->once()->andReturnTrue(); - $commandMutex->shouldReceive('forget')->once()->andThrow($cancellation); + $commandMutex->shouldReceive('forget')->once()->andReturnUsing(static function () use ($cancellation, $cleanup): never { + try { + throw $cancellation; + } finally { + if ($cleanup !== null) { + throw $cleanup; + } + } + }); try { $command->run(new ArrayInput(['--isolated' => true]), new NullOutput); $this->fail('Expected mutex release to preserve cancellation.'); - } catch (CanceledException $exception) { - $this->assertSame($cancellation, $exception); + } catch (Throwable $exception) { + $this->assertSame($cleanup ?? $cancellation, $exception); } } diff --git a/tests/Console/ConsoleApplicationDeferredCallbacksTest.php b/tests/Console/ConsoleApplicationDeferredCallbacksTest.php index ddf98a5058..673be77d63 100644 --- a/tests/Console/ConsoleApplicationDeferredCallbacksTest.php +++ b/tests/Console/ConsoleApplicationDeferredCallbacksTest.php @@ -16,6 +16,7 @@ use Hypervel\Support\Defer\DeferredCallbackCollection; use Hypervel\Testbench\TestCase; use Mockery as m; +use PHPUnit\Framework\Attributes\TestWith; use RuntimeException; use Swoole\Coroutine\CanceledException; use Symfony\Component\Console\Command\Command as SymfonyCommand; @@ -330,49 +331,67 @@ public function testApplicationOwnerRunsAlwaysCallbacksAndPreservesCommandFailur $this->assertSame(['always', 'recovery'], $calls); } - public function testApplicationOwnerCancellationSkipsDeferredCallbacks(): void + #[TestWith([false])] + #[TestWith([true])] + public function testApplicationOwnerCancellationSkipsDeferredCallbacks(bool $cleanupThrows): void { $application = $this->createConsoleApplication(); $cancellation = new CanceledException; + $cleanup = $cleanupThrows ? new RuntimeException('Cleanup failed.') : null; $calls = []; - $application->addCommand(new PlainDeferredCommand('test:application-cancellation', function () use (&$calls, $cancellation): never { + $application->addCommand(new PlainDeferredCommand('test:application-cancellation', function () use (&$calls, $cancellation, $cleanup): never { defer(function () use (&$calls): void { $calls[] = 'always'; }, always: true); - throw $cancellation; + try { + throw $cancellation; + } finally { + if ($cleanup !== null) { + throw $cleanup; + } + } })); try { $application->call('test:application-cancellation'); $this->fail('Expected the command to be cancelled.'); - } catch (CanceledException $exception) { - $this->assertSame($cancellation, $exception); + } catch (Throwable $exception) { + $this->assertSame($cleanup ?? $cancellation, $exception); } $this->assertSame([], $calls); } - public function testCommandOwnerCancellationSkipsDeferredCallbacks(): void + #[TestWith([false])] + #[TestWith([true])] + public function testCommandOwnerCancellationSkipsDeferredCallbacks(bool $cleanupThrows): void { $application = $this->createConsoleApplication(); $cancellation = new CanceledException; + $cleanup = $cleanupThrows ? new RuntimeException('Cleanup failed.') : null; $calls = []; - $application->addCommand(new DeferredCommand('test:command-cancellation', function () use (&$calls, $cancellation): never { + $application->addCommand(new DeferredCommand('test:command-cancellation', function () use (&$calls, $cancellation, $cleanup): never { defer(function () use (&$calls): void { $calls[] = 'always'; }, always: true); - throw $cancellation; + try { + throw $cancellation; + } finally { + if ($cleanup !== null) { + throw $cleanup; + } + } })); try { $application->call('test:command-cancellation'); $this->fail('Expected the command to be cancelled.'); - } catch (CanceledException $exception) { - $this->assertSame($cancellation, $exception); + } catch (Throwable $exception) { + $this->assertSame($cleanup ?? $cancellation, $exception); } $this->assertSame([], $calls); @@ -426,14 +445,17 @@ public function testCommandOwnerPreservesEarlierFailureWhenDrainAlsoFails(): voi } } - public function testApplicationOwnerCancellationDuringTheDrainSupersedesAnEarlierFailure(): void + #[TestWith([false])] + #[TestWith([true])] + public function testApplicationOwnerCancellationDuringTheDrainSupersedesAnEarlierFailure(bool $cleanupThrows): void { $application = $this->createConsoleApplication(); $commandFailure = new RuntimeException('command failed'); $cancellation = new CanceledException('canceled'); + $cleanup = $cleanupThrows ? new RuntimeException('Drain cleanup failed.') : null; $this->app->scoped( DeferredCallbackCollection::class, - fn (): ThrowingDeferredCallbackCollection => new ThrowingDeferredCallbackCollection($cancellation), + fn (): ThrowingDeferredCallbackCollection => new ThrowingDeferredCallbackCollection($cancellation, $cleanup), ); $application->addCommand(new DeferredCommand('test:application-drain-cancellation', function () use ($commandFailure): never { @@ -445,19 +467,22 @@ public function testApplicationOwnerCancellationDuringTheDrainSupersedesAnEarlie try { $application->call('test:application-drain-cancellation'); $this->fail('Expected the deferred callback drain to preserve cancellation.'); - } catch (CanceledException $exception) { - $this->assertSame($cancellation, $exception); + } catch (Throwable $exception) { + $this->assertSame($cleanup ?? $cancellation, $exception); } } - public function testCommandOwnerCancellationDuringTheDrainSupersedesAnEarlierFailure(): void + #[TestWith([false])] + #[TestWith([true])] + public function testCommandOwnerCancellationDuringTheDrainSupersedesAnEarlierFailure(bool $cleanupThrows): void { $application = $this->createConsoleApplication(); $commandFailure = new RuntimeException('command failed'); $cancellation = new CanceledException('canceled'); + $cleanup = $cleanupThrows ? new RuntimeException('Drain cleanup failed.') : null; $this->app->scoped( DeferredCallbackCollection::class, - fn (): ThrowingDeferredCallbackCollection => new ThrowingDeferredCallbackCollection($cancellation), + fn (): ThrowingDeferredCallbackCollection => new ThrowingDeferredCallbackCollection($cancellation, $cleanup), ); $application->addCommand(new DeferredCommand('test:command-drain-cancellation', function () use ($commandFailure): never { @@ -469,8 +494,8 @@ public function testCommandOwnerCancellationDuringTheDrainSupersedesAnEarlierFai try { $application->call('test:command-drain-cancellation'); $this->fail('Expected the deferred callback drain to preserve cancellation.'); - } catch (CanceledException $exception) { - $this->assertSame($cancellation, $exception); + } catch (Throwable $exception) { + $this->assertSame($cleanup ?? $cancellation, $exception); } } @@ -539,7 +564,7 @@ class ThrowingDeferredCallbackCollection extends DeferredCallbackCollection /** * Create a new callback collection. */ - public function __construct(private readonly Throwable $exception) + public function __construct(private readonly Throwable $exception, private readonly ?Throwable $cleanupException = null) { } @@ -548,6 +573,12 @@ public function __construct(private readonly Throwable $exception) */ public function invokeWhen(?Closure $when = null): void { - throw $this->exception; + try { + throw $this->exception; + } finally { + if ($this->cleanupException !== null) { + throw $this->cleanupException; + } + } } } diff --git a/tests/Console/Scheduling/CallbackEventTest.php b/tests/Console/Scheduling/CallbackEventTest.php index 3e568f6229..368ae3ad83 100644 --- a/tests/Console/Scheduling/CallbackEventTest.php +++ b/tests/Console/Scheduling/CallbackEventTest.php @@ -13,7 +13,10 @@ use LogicException; use Mockery as m; use Mockery\MockInterface; +use PHPUnit\Framework\Attributes\TestWith; use RuntimeException; +use Swoole\Coroutine\CanceledException; +use Throwable; use function Hypervel\Coroutine\parallel; @@ -253,6 +256,31 @@ public function testCallbackFailureRemainsPrimaryWhenAfterCallbackFails(): void } } + #[TestWith([false])] + #[TestWith([true])] + public function testAfterCallbackCancellationSupersedesCallbackFailure(bool $cleanupThrows): void + { + $cancellation = new CanceledException('After callback canceled.'); + $cleanup = $cleanupThrows ? new RuntimeException('After callback cleanup failed.') : null; + $event = new CallbackEvent($this->mutex, static fn () => throw new RuntimeException('Callback failed.')); + $event->after(static function () use ($cancellation, $cleanup): never { + try { + throw $cancellation; + } finally { + if ($cleanup !== null) { + throw $cleanup; + } + } + }); + + try { + $event->run($this->app); + $this->fail('Expected cancellation to supersede the callback failure.'); + } catch (Throwable $exception) { + $this->assertSame($cleanup ?? $cancellation, $exception); + } + } + public function testConcurrentRunsKeepResultsAndExceptionsIsolated(): void { $firstReady = new Channel(1); diff --git a/tests/Console/Scheduling/EventTest.php b/tests/Console/Scheduling/EventTest.php index aa9c49305a..960921df09 100644 --- a/tests/Console/Scheduling/EventTest.php +++ b/tests/Console/Scheduling/EventTest.php @@ -24,7 +24,9 @@ use Hypervel\Tests\Console\Fixtures\FakeEventMutex; use Hypervel\Tests\TestCase; use Mockery as m; +use PHPUnit\Framework\Attributes\TestWith; use RuntimeException; +use Swoole\Coroutine\CanceledException; use Symfony\Component\Mime\Email; use Symfony\Component\Process\Process; use Throwable; @@ -341,6 +343,52 @@ public function testAfterCallbackFailureRemainsPrimaryWhenMutexCleanupFails(): v $event->releaseMutexOnTerminationSignal(); } + #[TestWith(['start', false])] + #[TestWith(['start', true])] + #[TestWith(['after', false])] + #[TestWith(['after', true])] + #[TestWith(['output', false])] + #[TestWith(['output', true])] + public function testCancellationDuringCleanupSupersedesAnEarlierFailure(string $failurePhase, bool $cleanupThrows): void + { + $cancellation = new CanceledException('Cleanup canceled.'); + $cleanup = $cleanupThrows ? new RuntimeException('Cleanup failed.') : null; + $cancel = static function () use ($cancellation, $cleanup): never { + try { + throw $cancellation; + } finally { + if ($cleanup !== null) { + throw $cleanup; + } + } + }; + $mutex = m::mock(EventMutex::class); + $mutex->shouldReceive('create')->once()->andReturnTrue(); + $event = new EventTestExecutableEvent($mutex); + $event->withoutOverlapping(); + + if ($failurePhase === 'output') { + $event->sendOutputTo('test.log')->after($cancel); + $filesystem = m::mock(Filesystem::class); + $filesystem->shouldReceive('isFile')->once()->with('test.log')->andReturnTrue(); + $filesystem->shouldReceive('put')->once()->with('test.log', 'output')->andReturnFalse(); + $this->container->instance(Filesystem::class, $filesystem); + $mutex->shouldReceive('forget')->once(); + } else { + $event->{$failurePhase === 'start' ? 'before' : 'after'}(static fn () => throw new RuntimeException('Earlier failure.')); + $mutex->shouldReceive('forget')->once()->andReturnUsing($cancel); + } + + try { + $event->run($this->container); + $this->fail('Expected cancellation to supersede the earlier failure.'); + } catch (Throwable $exception) { + $this->assertSame($cleanup ?? $cancellation, $exception); + } + + $event->releaseMutexOnTerminationSignal(); + } + public function testExitPublicationFailureStillRunsCallbacksAndReleasesMutex(): void { $afterCalled = false; @@ -550,6 +598,49 @@ public function testProcessIsReleasedAfterOutputFailure(): void $this->assertFalse($event->hasRetainedProcess()); } + #[TestWith([false])] + #[TestWith([true])] + public function testOutputCancellationReleasesTheMutexAndProcessWithoutRunningAfterCallbacks(bool $cleanupThrows): void + { + $cancellation = new CanceledException('Output write canceled.'); + $cleanup = $cleanupThrows ? new RuntimeException('Output cleanup failed.') : null; + $afterCalled = false; + $process = m::mock(Process::class); + $process->shouldReceive('getOutput')->once()->andReturn('process output'); + $mutex = m::mock(EventMutex::class); + $mutex->shouldReceive('create')->once()->andReturnTrue(); + $mutex->shouldReceive('forget')->once(); + $event = new EventTestProcessEvent($mutex, $process); + $event->withoutOverlapping()->sendOutputTo('test.log'); + $event->after(static function () use (&$afterCalled): void { + $afterCalled = true; + }); + $filesystem = m::mock(Filesystem::class); + $filesystem->shouldReceive('put')->once()->with('test.log', 'process output')->andReturnUsing( + static function () use ($cancellation, $cleanup): never { + try { + throw $cancellation; + } finally { + if ($cleanup !== null) { + throw $cleanup; + } + } + } + ); + $this->container->instance(Filesystem::class, $filesystem); + + try { + $event->run($this->container); + $this->fail('Expected output cancellation to escape.'); + } catch (Throwable $exception) { + $this->assertSame($cleanup ?? $cancellation, $exception); + } + + $this->assertFalse($afterCalled); + $this->assertFalse($event->hasRetainedProcess()); + $event->releaseMutexOnTerminationSignal(); + } + public function testProcessIsReleasedAfterAfterCallbackFailure(): void { $event = new EventTestProcessEvent(m::mock(EventMutex::class), m::mock(Process::class)); diff --git a/tests/Console/Scheduling/ScheduleRunCommandTest.php b/tests/Console/Scheduling/ScheduleRunCommandTest.php index 269a945547..3f168ea8a1 100644 --- a/tests/Console/Scheduling/ScheduleRunCommandTest.php +++ b/tests/Console/Scheduling/ScheduleRunCommandTest.php @@ -39,6 +39,7 @@ use Hypervel\Testbench\TestCase; use Mockery as m; use PHPUnit\Framework\Attributes\DataProvider; +use PHPUnit\Framework\Attributes\TestWith; use ReflectionMethod; use ReflectionProperty; use RuntimeException; @@ -190,20 +191,29 @@ public static function deferredTaskOutcomes(): array ]; } - public function testCanceledCallbackReleasesItsMutexWithoutRunningFailureOrDeferredCallbacks(): void + #[TestWith([false])] + #[TestWith([true])] + public function testCanceledCallbackReleasesItsMutexWithoutRunningFailureOrDeferredCallbacks(bool $cleanupThrows): void { $calls = []; $cancellation = new CanceledException('Task canceled'); + $cleanup = $cleanupThrows ? new RuntimeException('Task cleanup failed.') : null; $mutex = m::mock(EventMutex::class); $mutex->shouldReceive('exists')->once()->andReturnFalse(); $mutex->shouldReceive('create')->once()->andReturnTrue(); $mutex->shouldReceive('forget')->once(); - $event = new CallbackEvent($mutex, function () use (&$calls, $cancellation): void { + $event = new CallbackEvent($mutex, function () use (&$calls, $cancellation, $cleanup): void { defer(function () use (&$calls): void { $calls[] = 'always'; })->always(); - throw $cancellation; + try { + throw $cancellation; + } finally { + if ($cleanup !== null) { + throw $cleanup; + } + } }); $event->name('canceled')->withoutOverlapping()->onFailure(function () use (&$calls): void { $calls[] = 'failure'; @@ -214,7 +224,7 @@ public function testCanceledCallbackReleasesItsMutexWithoutRunningFailureOrDefer $this->invokeRunEvents($this->makeCommand(), [$event]); $this->fail('Expected the owned task to report cancellation.'); } catch (ChildCancellationException $exception) { - $this->assertSame($cancellation, $exception->getPrevious()); + $this->assertSame($cleanup ?? $cancellation, $exception->getPrevious()); } $this->assertSame([], $calls); @@ -250,13 +260,22 @@ public function testStartingListenerFailureRunsOnlyAlwaysCallbacksAndPreservesTh $this->assertSame(['always'], $calls); } - public function testCancellationDuringDeferredWorkStopsTheRemainingCallbacks(): void + #[TestWith([false])] + #[TestWith([true])] + public function testCancellationDuringDeferredWorkStopsTheRemainingCallbacks(bool $cleanupThrows): void { $calls = []; $cancellation = new CanceledException('Deferred work canceled'); - $event = new CallbackEvent(m::mock(EventMutex::class), function () use (&$calls, $cancellation): void { - defer(function () use ($cancellation): void { - throw $cancellation; + $cleanup = $cleanupThrows ? new RuntimeException('Deferred cleanup failed.') : null; + $event = new CallbackEvent(m::mock(EventMutex::class), function () use (&$calls, $cancellation, $cleanup): void { + defer(function () use ($cancellation, $cleanup): void { + try { + throw $cancellation; + } finally { + if ($cleanup !== null) { + throw $cleanup; + } + } }); defer(function () use (&$calls): void { $calls[] = 'always'; @@ -267,7 +286,43 @@ public function testCancellationDuringDeferredWorkStopsTheRemainingCallbacks(): $this->invokeRunEvents($this->makeCommand(), [$event]); $this->fail('Expected cancellation from deferred work.'); } catch (ChildCancellationException $exception) { - $this->assertSame($cancellation, $exception->getPrevious()); + $this->assertSame($cleanup ?? $cancellation, $exception->getPrevious()); + } + + $this->assertSame([], $calls); + } + + #[TestWith([false])] + #[TestWith([true])] + public function testDeferredCancellationSupersedesAnEarlierTaskFailure(bool $cleanupThrows): void + { + $cancellation = new CanceledException('Deferred work canceled'); + $cleanup = $cleanupThrows ? new RuntimeException('Deferred cleanup failed.') : null; + $calls = []; + $this->dispatcher = $this->app->make(Dispatcher::class); + $this->dispatcher->listen(ScheduledTaskStarting::class, static function () use (&$calls, $cancellation, $cleanup): never { + defer(static function () use ($cancellation, $cleanup): never { + try { + throw $cancellation; + } finally { + if ($cleanup !== null) { + throw $cleanup; + } + } + })->always(); + defer(static function () use (&$calls): void { + $calls[] = 'later callback'; + })->always(); + + throw new RuntimeException('Starting listener failed.'); + }); + $event = new CallbackEvent(m::mock(EventMutex::class), static fn (): bool => true); + + try { + $this->invokeRunEvents($this->makeCommand(), [$event]); + $this->fail('Expected cancellation to supersede the task failure.'); + } catch (ChildCancellationException $exception) { + $this->assertSame($cleanup ?? $cancellation, $exception->getPrevious()); } $this->assertSame([], $calls); diff --git a/tests/Container/CoroutineSafetyTest.php b/tests/Container/CoroutineSafetyTest.php index 2ca5fbdcda..fb44ec2e9f 100644 --- a/tests/Container/CoroutineSafetyTest.php +++ b/tests/Container/CoroutineSafetyTest.php @@ -11,6 +11,7 @@ use Hypervel\Contracts\Container\CircularDependencyException; use Hypervel\Contracts\Container\Transient; use Hypervel\Coroutine\Coroutine; +use Hypervel\Coroutine\Exceptions\ForeignCancellationException; use Hypervel\Tests\TestCase; use PHPUnit\Framework\Attributes\DataProvider; use PHPUnit\Framework\Attributes\TestWith; @@ -776,7 +777,9 @@ public function testCanceledWaiterRemovesCoordinatorEdge(bool $throwException): $this->assertInstanceOf(stdClass::class, $ownerFinished->pop(1)); } - public function testOwnerCancellationReachesOnlyTheOwnerAndTheNextResolutionRetries(): void + #[TestWith([false])] + #[TestWith([true])] + public function testOwnerCancellationReachesOnlyTheOwnerAndTheNextResolutionRetries(bool $cleanupThrows): void { $container = new CoroutineInspectingContainer; $ownerEntered = new Channel(1); @@ -785,10 +788,19 @@ public function testOwnerCancellationReachesOnlyTheOwnerAndTheNextResolutionRetr $waiterFinished = new Channel(1); $attempts = 0; - $container->singleton('service', function () use ($ownerEntered, $releaseOwner, &$attempts) { + $cleanupException = new RuntimeException('Resolution cleanup failed.'); + $container->singleton('service', function () use ($ownerEntered, $releaseOwner, &$attempts, $cleanupThrows, $cleanupException) { if (++$attempts === 1) { $ownerEntered->push(true); - $releaseOwner->pop(); + try { + $releaseOwner->pop(); + } finally { + if ($cleanupThrows) { + usleep(1000); + + throw $cleanupException; + } + } } return new stdClass; @@ -823,10 +835,22 @@ public function testOwnerCancellationReachesOnlyTheOwnerAndTheNextResolutionRetr $ownerException = $ownerFinished->pop(1); $waiterException = $waiterFinished->pop(1); + SwooleCoroutine::join([$ownerId, $waiterId], 1); + $this->assertFalse(SwooleCoroutine::exists($ownerId)); + $this->assertFalse(SwooleCoroutine::exists($waiterId)); + + if ($cleanupThrows) { + $this->assertSame($cleanupException, $ownerException); + $this->assertInstanceOf(CanceledException::class, $ownerException->getPrevious()); + } else { + $this->assertInstanceOf(CanceledException::class, $ownerException); + } - $this->assertInstanceOf(CanceledException::class, $ownerException); + $this->assertTrue(Coroutine::causedByCancellation($ownerException)); $this->assertInstanceOf(BindingResolutionException::class, $waiterException); - $this->assertSame($ownerException, $waiterException->getPrevious()); + $this->assertInstanceOf(ForeignCancellationException::class, $waiterException->getPrevious()); + $this->assertSame($ownerException, $waiterException->getPrevious()->getPrevious()); + $this->assertFalse(Coroutine::causedByCancellation($waiterException)); } finally { $releaseOwner->push(true, 0.001); $this->cancelSurvivingCoroutines([$ownerId, $waiterId]); diff --git a/tests/Coroutine/CoroutineTest.php b/tests/Coroutine/CoroutineTest.php index f66c5336f7..6231ed03c5 100644 --- a/tests/Coroutine/CoroutineTest.php +++ b/tests/Coroutine/CoroutineTest.php @@ -10,6 +10,8 @@ use Hypervel\Context\CoroutineContext; use Hypervel\Contracts\Debug\ExceptionHandler as ExceptionHandlerContract; use Hypervel\Coroutine\Coroutine; +use Hypervel\Coroutine\Exceptions\ChildCancellationException; +use Hypervel\Coroutine\Exceptions\ForeignCancellationException; use Hypervel\Engine\Channel; use Hypervel\Engine\Coroutine as EngineCoroutine; use Hypervel\Engine\Exceptions\CoroutineDestroyedException; @@ -18,14 +20,46 @@ use Hypervel\Tests\Context\Fixtures\ThrowingReplicableContext; use Hypervel\Tests\TestCase; use Mockery as m; +use PHPUnit\Framework\Attributes\DataProvider; use PHPUnit\Framework\Attributes\RunInSeparateProcess; +use PHPUnit\Framework\Attributes\TestWith; use RuntimeException; +use Swoole\Coroutine\CanceledException; use Throwable; use function Hypervel\Coroutine\go; class CoroutineTest extends TestCase { + #[DataProvider('cancellationExceptions')] + public function testCancellationClassificationRespectsCoroutineBoundaries(Throwable $exception, bool $expected): void + { + $this->assertSame($expected, Coroutine::causedByCancellation($exception)); + } + + /** + * Distinguish operation cancellation from another coroutine's failure. + */ + public static function cancellationExceptions(): array + { + $cancellation = new CanceledException; + $cleanup = new RuntimeException('Cleanup failed.', previous: $cancellation); + $child = new ChildCancellationException('The child stopped.', previous: $cleanup); + $foreign = new ForeignCancellationException('The shared operation stopped.', previous: $cleanup); + + return [ + 'direct cancellation' => [$cancellation, true], + 'cleanup replaced cancellation' => [$cleanup, true], + 'nested cleanup' => [new RuntimeException(previous: $cleanup), true], + 'independent child' => [$child, false], + 'wrapped independent child' => [new RuntimeException(previous: $child), false], + 'grandchild boundary' => [new ChildCancellationException(previous: $child), false], + 'shared operation' => [$foreign, false], + 'wrapped shared operation' => [new RuntimeException(previous: $foreign), false], + 'ordinary error' => [new RuntimeException, false], + ]; + } + public function testCreateAndForkReturnPositiveCoroutineIds(): void { $this->assertGreaterThan(0, Coroutine::create(static fn (): null => null)); @@ -263,7 +297,9 @@ public function testForksDuringDetachedStartupOnlyDetachWhenRequested(): void $this->assertFalse(CoroutineContext::has(Coroutine::DETACHED_CONTEXT_KEY)); } - public function testFlushStateRestoresExceptionReporting(): void + #[TestWith([false])] + #[TestWith([true])] + public function testExceptionReportingCanBeRestored(bool $flushState): void { try { $container = new Container; @@ -273,11 +309,19 @@ public function testFlushStateRestoresExceptionReporting(): void $handler->shouldReceive('report')->once(); - Coroutine::enableReportException(false); - Coroutine::flushState(); + Coroutine::reportExceptions(false); + Coroutine::create(function () { + throw new Exception('Reporting is disabled.'); + }); + + if ($flushState) { + Coroutine::flushState(); + } else { + Coroutine::reportExceptions(); + } Coroutine::create(function () { - throw new Exception('Should be reported after flushState.'); + throw new Exception('Reporting is enabled again.'); }); } finally { Coroutine::flushState(); diff --git a/tests/Coroutine/ParallelTest.php b/tests/Coroutine/ParallelTest.php index ddd00d7fec..28a184e604 100644 --- a/tests/Coroutine/ParallelTest.php +++ b/tests/Coroutine/ParallelTest.php @@ -19,6 +19,7 @@ use Hypervel\Tests\Context\Fixtures\ThrowingReplicableContext; use Hypervel\Tests\TestCase; use Mockery as m; +use PHPUnit\Framework\Attributes\DataProvider; use RuntimeException; use Swoole\Coroutine as SwooleCoroutine; use Swoole\Coroutine\CanceledException; @@ -238,17 +239,19 @@ public function testParallelThrowsAfterChildCoroutinesExit(): void $this->assertFalse(Coroutine::exists($childCoroutineId)); } - public function testIndependentChildCancellationIsStoredAsAChildFailure(): void + #[DataProvider('cleanupFailures')] + public function testIndependentChildCancellationIsStoredAsAChildFailure(bool $cleanupThrows): void { $parallel = new Parallel; $childStarted = new Channel(1); $blocker = new Channel(1); $childCoroutineId = null; $childCancellation = null; + $cleanupException = new RuntimeException('Child cleanup failed.'); $outcome = null; $runnerFailure = null; - $parallel->add(static function () use ($childStarted, $blocker, &$childCoroutineId, &$childCancellation): void { + $parallel->add(static function () use ($childStarted, $blocker, &$childCoroutineId, &$childCancellation, $cleanupThrows, $cleanupException): void { $childCoroutineId = Coroutine::id(); $childStarted->push(true); @@ -257,6 +260,12 @@ public function testIndependentChildCancellationIsStoredAsAChildFailure(): void } catch (CanceledException $exception) { $childCancellation = $exception; throw $exception; + } finally { + if ($cleanupThrows) { + usleep(1000); + + throw $cleanupException; + } } }, 'canceled'); @@ -277,7 +286,16 @@ public function testIndependentChildCancellationIsStoredAsAChildFailure(): void $this->assertNull($runnerFailure); $this->assertInstanceOf(ChildCancellationException::class, $outcome); $this->assertSame('A child coroutine managed by Parallel was canceled while its owner remained active.', $outcome->getMessage()); - $this->assertSame($childCancellation, $outcome->getPrevious()); + $this->assertSame($cleanupThrows ? $cleanupException : $childCancellation, $outcome->getPrevious()); + $this->assertFalse(Coroutine::causedByCancellation($outcome)); + } + + /** + * Exercise cancellation with and without a failing cleanup. + */ + public static function cleanupFailures(): array + { + return ['normal cleanup' => [false], 'throwing cleanup' => [true]]; } public function testOwnerCancellationCancelsEveryLiveChildAndEscapesExactly(): void diff --git a/tests/Coroutine/WaiterTest.php b/tests/Coroutine/WaiterTest.php index 7090826bdd..48ca55e5f6 100644 --- a/tests/Coroutine/WaiterTest.php +++ b/tests/Coroutine/WaiterTest.php @@ -20,6 +20,7 @@ use Hypervel\Tests\TestCase; use Mockery as m; use PHPUnit\Framework\Attributes\DataProvider; +use PHPUnit\Framework\Attributes\TestWith; use ReflectionProperty; use RuntimeException; use Swoole\Coroutine\CanceledException; @@ -392,19 +393,24 @@ public function testStrictParentCancellationPropagatesOnlyAfterTheChildFinishesI $this->assertFalse($childAlive); } - public function testIndependentChildCancellationIsWrappedForTheActiveOwner(): void + #[TestWith([false])] + #[TestWith([true])] + public function testIndependentChildCancellationIsWrappedForTheActiveOwner(bool $cleanupThrows): void { $waiter = new Waiter; $childStarted = new Channel(1); $childBlocker = new Channel(1); $childCoroutineId = null; $childCancellation = null; + $cleanupException = new RuntimeException('Child cleanup failed.'); $outcome = null; $parent = EngineCoroutine::create(function () use ( $waiter, $childStarted, $childBlocker, + $cleanupThrows, + $cleanupException, &$childCoroutineId, &$childCancellation, &$outcome, @@ -413,6 +419,8 @@ public function testIndependentChildCancellationIsWrappedForTheActiveOwner(): vo $outcome = $waiter->wait(function () use ( $childStarted, $childBlocker, + $cleanupThrows, + $cleanupException, &$childCoroutineId, &$childCancellation, ): void { @@ -424,6 +432,11 @@ public function testIndependentChildCancellationIsWrappedForTheActiveOwner(): vo } catch (CanceledException $exception) { $childCancellation = $exception; throw $exception; + } finally { + if ($cleanupThrows) { + Coroutine::sleep(0.001); + throw $cleanupException; + } } }); } catch (Throwable $exception) { @@ -431,14 +444,24 @@ public function testIndependentChildCancellationIsWrappedForTheActiveOwner(): vo } }); - $this->assertTrue($childStarted->pop()); - $this->assertIsInt($childCoroutineId); - $this->assertTrue(EngineCoroutine::cancelById($childCoroutineId, throwException: true)); - Coroutine::join([$parent->getId()]); + try { + $this->assertTrue($childStarted->pop(1)); + $this->assertIsInt($childCoroutineId); + $this->assertTrue(EngineCoroutine::cancelById($childCoroutineId, throwException: true)); + Coroutine::join([$parent->getId()], 1); + } finally { + $childBlocker->close(); + Coroutine::join([$parent->getId()], 1); + } $this->assertInstanceOf(ChildCancellationException::class, $outcome); $this->assertSame('A child coroutine managed by Waiter was canceled while its owner remained active.', $outcome->getMessage()); - $this->assertSame($childCancellation, $outcome->getPrevious()); + $this->assertSame($cleanupThrows ? $cleanupException : $childCancellation, $outcome->getPrevious()); + $this->assertFalse(Coroutine::causedByCancellation($outcome)); + + if ($cleanupThrows) { + $this->assertSame($childCancellation, $cleanupException->getPrevious()); + } } public function testWaitReturnsAfterDeferredWorkCompletes(): void diff --git a/tests/Events/EventsDispatcherTest.php b/tests/Events/EventsDispatcherTest.php index 057f1212a4..99c26ba787 100755 --- a/tests/Events/EventsDispatcherTest.php +++ b/tests/Events/EventsDispatcherTest.php @@ -10,6 +10,7 @@ use Hypervel\Container\Container; use Hypervel\Contracts\Events\ShouldDispatchAfterCommit; use Hypervel\Contracts\Queue\ShouldQueue; +use Hypervel\Coroutine\Exceptions\ChildCancellationException; use Hypervel\Database\DatabaseTransactionsManager; use Hypervel\Events\Dispatcher; use Hypervel\Foundation\Events\Dispatchable; @@ -23,6 +24,7 @@ use Hypervel\Tests\Events\Fixtures\UnlistenedEvent; use Hypervel\Tests\Events\Fixtures\UnlistenedStringEvent; use Hypervel\Tests\TestCase; +use PHPUnit\Framework\Attributes\TestWith; use ReflectionClass; use RuntimeException; use Swoole\Coroutine\CanceledException; @@ -1217,13 +1219,16 @@ public function testObserverRunsWhenActiveListenerThrows(): void $this->assertTrue($observerInvoked); } - public function testListenerCancellationSkipsObservers(): void + #[TestWith([false])] + #[TestWith([true])] + public function testListenerCancellationSkipsObservers(bool $cleanupThrows): void { $dispatcher = new Dispatcher; $cancellation = new CanceledException('canceled'); + $failure = $cleanupThrows ? new RuntimeException('Cleanup failed.', previous: $cancellation) : $cancellation; $observerInvoked = false; - $dispatcher->listen('foo', fn () => throw $cancellation); + $dispatcher->listen('foo', fn () => throw $failure); $dispatcher->observe('foo', function () use (&$observerInvoked): void { $observerInvoked = true; }); @@ -1231,17 +1236,20 @@ public function testListenerCancellationSkipsObservers(): void try { $dispatcher->dispatch('foo'); $this->fail('Expected listener cancellation to escape.'); - } catch (CanceledException $exception) { - $this->assertSame($cancellation, $exception); + } catch (Exception $exception) { + $this->assertSame($failure, $exception); } $this->assertFalse($observerInvoked); } - public function testObserverCancellationSupersedesEarlierFailuresAndStopsLaterObservers(): void + #[TestWith([false])] + #[TestWith([true])] + public function testObserverCancellationSupersedesEarlierFailuresAndStopsLaterObservers(bool $cleanupThrows): void { $dispatcher = new Dispatcher; $cancellation = new CanceledException('canceled'); + $failure = $cleanupThrows ? new RuntimeException('Cleanup failed.', previous: $cancellation) : $cancellation; $order = []; $dispatcher->listen('foo', function () use (&$order): void { @@ -1254,10 +1262,10 @@ public function testObserverCancellationSupersedesEarlierFailuresAndStopsLaterOb throw new RuntimeException('observer failed'); }); - $dispatcher->observe('foo', function () use (&$order, $cancellation): void { + $dispatcher->observe('foo', function () use (&$order, $failure): void { $order[] = 'cancelling observer'; - throw $cancellation; + throw $failure; }); $dispatcher->observe('foo', function () use (&$order): void { $order[] = 'later observer'; @@ -1266,13 +1274,38 @@ public function testObserverCancellationSupersedesEarlierFailuresAndStopsLaterOb try { $dispatcher->dispatch('foo'); $this->fail('Expected observer cancellation to escape.'); - } catch (CanceledException $exception) { - $this->assertSame($cancellation, $exception); + } catch (Exception $exception) { + $this->assertSame($failure, $exception); } $this->assertSame(['listener', 'first observer', 'cancelling observer'], $order); } + public function testIndependentChildCancellationKeepsOrdinaryObserverFailureRules(): void + { + $dispatcher = new Dispatcher; + $listenerFailure = new ChildCancellationException(previous: new CanceledException); + $observerFailure = new ChildCancellationException(previous: new CanceledException); + $order = []; + $dispatcher->listen('foo', static fn () => throw $listenerFailure); + $dispatcher->observe('foo', static function () use (&$order, $observerFailure): never { + $order[] = 'first'; + throw $observerFailure; + }); + $dispatcher->observe('foo', static function () use (&$order): void { + $order[] = 'second'; + }); + + try { + $dispatcher->dispatch('foo'); + $this->fail('Expected the original listener failure.'); + } catch (ChildCancellationException $exception) { + $this->assertSame($listenerFailure, $exception); + } + + $this->assertSame(['first', 'second'], $order); + } + public function testLaterObserverRunsWhenEarlierObserverThrows(): void { $dispatcher = new Dispatcher; diff --git a/tests/Grpc/BaseClientTest.php b/tests/Grpc/BaseClientTest.php index fa73fa3d97..aff6826a4c 100644 --- a/tests/Grpc/BaseClientTest.php +++ b/tests/Grpc/BaseClientTest.php @@ -8,8 +8,10 @@ use Google\Protobuf\Internal\Message; use Google\Protobuf\StringValue; use Hypervel\Container\Container; +use Hypervel\Context\CoroutineContext; use Hypervel\Contracts\Engine\Http\V2\ClientFactoryInterface; use Hypervel\Contracts\Engine\Http\V2\ClientInterface; +use Hypervel\Database\Pool\ConnectionLease; use Hypervel\Engine\Http\V2\Response; use Hypervel\Grpc\Client\BaseClient; use Hypervel\Grpc\Client\BidiStreamingCall; @@ -35,9 +37,12 @@ use Hypervel\Tests\TestCase; use InvalidArgumentException; use LogicException; +use Mockery as m; use PHPUnit\Framework\Attributes\DataProvider; +use PHPUnit\Framework\Attributes\TestWith; use ReflectionClass; use ReflectionMethod; +use RuntimeException; use Swoole\Coroutine\CanceledException; use Throwable; @@ -294,6 +299,41 @@ static function ( } } + #[TestWith([false])] + #[TestWith([true])] + public function testCancellationWhileReleasingAnIdleConnectionDoesNotFinishTheOperation(bool $cleanupThrows): void + { + $engineClient = new ClientCallClient; + $this->bindFactory(new ClientCallClientFactory($engineClient)); + $finished = false; + Container::getInstance()->make(GrpcOperationRunner::class)->observe( + new BaseClientGrpcOperationObserverStub( + static fn (): null => null, + static function () use (&$finished): void { + $finished = true; + }, + ), + ); + $client = $this->newClient(); + $cancellation = new CanceledException; + $failure = $cleanupThrows ? new RuntimeException('Release cleanup failed.', previous: $cancellation) : $cancellation; + $lease = m::mock(ConnectionLease::class); + $lease->shouldReceive('releaseIfIdle')->once()->andThrow($failure); + CoroutineContext::set('__grpc.test.connection_lease', $lease); + + try { + $client->unary('/testing.Service/Unary', new StringValue, [StringValue::class, 'decode']); + $this->fail('Expected cancellation before the transport starts.'); + } catch (Throwable $exception) { + $this->assertSame($failure, $exception); + } finally { + CoroutineContext::forget('__grpc.test.connection_lease'); + } + + $this->assertFalse($finished); + $this->assertSame([], $engineClient->sentRequests); + } + public function testPreTransportFailureFinishesTheLogicalOperationWithoutAnAttempt(): void { $engineClient = new ClientCallClient; diff --git a/tests/Grpc/ConnectionTest.php b/tests/Grpc/ConnectionTest.php index eb9cb24a75..84257bf097 100644 --- a/tests/Grpc/ConnectionTest.php +++ b/tests/Grpc/ConnectionTest.php @@ -9,6 +9,8 @@ use Hypervel\Contracts\Engine\Http\V2\ClientInterface; use Hypervel\Contracts\Engine\Http\V2\RequestInterface; use Hypervel\Contracts\Engine\Http\V2\ResponseInterface; +use Hypervel\Coroutine\Coroutine as FrameworkCoroutine; +use Hypervel\Coroutine\Exceptions\ForeignCancellationException; use Hypervel\Engine\Channel; use Hypervel\Engine\Coroutine; use Hypervel\Engine\Exceptions\HttpClientException; @@ -477,10 +479,13 @@ public function testNativeSendCancellationTerminatesTheConnectionAndPreservesThe { $cancellation = new CanceledException; $client = new ConnectionTestClient; + $connection = $this->connection(new ConnectionTestClientFactory($client)); + $active = $this->state(); + $this->start($connection, $this->request(), $active); + $client->sendUsing(static function () use ($cancellation): int { throw $cancellation; }); - $connection = $this->connection(new ConnectionTestClientFactory($client)); $state = $this->state(); try { @@ -492,11 +497,15 @@ public function testNativeSendCancellationTerminatesTheConnectionAndPreservesThe $this->assertTrue($connection->isClosed()); - try { - $state->status(); - $this->fail('Expected the uncertain stream to fail with its connection.'); - } catch (ConnectionException $exception) { - $this->assertSame($cancellation, $exception->getPrevious()); + foreach ([$active, $state] as $failedState) { + try { + $failedState->status(); + $this->fail('Expected the stream to fail with its connection.'); + } catch (ConnectionException $exception) { + $this->assertInstanceOf(ForeignCancellationException::class, $exception->getPrevious()); + $this->assertSame($cancellation, $exception->getPrevious()->getPrevious()); + $this->assertFalse(FrameworkCoroutine::causedByCancellation($exception)); + } } } @@ -558,7 +567,9 @@ public function testNativeWriteCancellationTerminatesTheConnectionAndPreservesTh $state->status(); $this->fail('Expected the uncertain stream to fail with its connection.'); } catch (ConnectionException $exception) { - $this->assertSame($cancellation, $exception->getPrevious()); + $this->assertInstanceOf(ForeignCancellationException::class, $exception->getPrevious()); + $this->assertSame($cancellation, $exception->getPrevious()->getPrevious()); + $this->assertFalse(FrameworkCoroutine::causedByCancellation($exception)); } } diff --git a/tests/Grpc/GrpcOperationRunnerTest.php b/tests/Grpc/GrpcOperationRunnerTest.php index 0a02343af5..2ff913173b 100644 --- a/tests/Grpc/GrpcOperationRunnerTest.php +++ b/tests/Grpc/GrpcOperationRunnerTest.php @@ -16,8 +16,10 @@ use Hypervel\Grpc\Status; use Hypervel\Grpc\StatusCode; use Hypervel\Tests\TestCase; +use PHPUnit\Framework\Attributes\TestWith; use RuntimeException; use Swoole\Coroutine\CanceledException; +use Throwable; class GrpcOperationRunnerTest extends TestCase { @@ -191,10 +193,13 @@ function () use (&$finished): never { } } - public function testStartingCancellationSkipsPreviouslyStartedCompletions(): void + #[TestWith([false])] + #[TestWith([true])] + public function testStartingCancellationSkipsPreviouslyStartedCompletions(bool $cleanupThrows): void { $runner = new GrpcOperationRunner; $cancellation = new CanceledException; + $failure = $cleanupThrows ? new RuntimeException('Observer cleanup failed.', previous: $cancellation) : $cancellation; $finished = false; $runner->observe($this->observer( @@ -204,31 +209,34 @@ function () use (&$finished): void { }, )); $runner->observe($this->observer( - function () use ($cancellation): never { - throw $cancellation; + function () use ($failure): never { + throw $failure; }, )); try { $runner->start($this->clientOperation()); $this->fail('Expected observer startup to be cancelled.'); - } catch (CanceledException $exception) { - $this->assertSame($cancellation, $exception); + } catch (Throwable $exception) { + $this->assertSame($failure, $exception); } $this->assertFalse($finished); } - public function testCompletionCancellationStopsLaterObserversAndSupersedesARealFailure(): void + #[TestWith([false])] + #[TestWith([true])] + public function testCompletionCancellationStopsLaterObserversAndSupersedesARealFailure(bool $cleanupThrows): void { $runner = new GrpcOperationRunner; $cancellation = new CanceledException; + $failure = $cleanupThrows ? new RuntimeException('Observer cleanup failed.', previous: $cancellation) : $cancellation; $laterFinished = false; $runner->observe($this->observer( fn (): null => null, - function () use ($cancellation): never { - throw $cancellation; + function () use ($failure): never { + throw $failure; }, )); $runner->observe($this->observer( @@ -246,8 +254,8 @@ function () use (&$laterFinished): void { 1, )); $this->fail('Expected observer completion to be cancelled.'); - } catch (CanceledException $exception) { - $this->assertSame($cancellation, $exception); + } catch (Throwable $exception) { + $this->assertSame($failure, $exception); } $this->assertTrue($handle->isFinished()); diff --git a/tests/Grpc/HandleCallTest.php b/tests/Grpc/HandleCallTest.php index f7a48a8f9d..93edf8d496 100644 --- a/tests/Grpc/HandleCallTest.php +++ b/tests/Grpc/HandleCallTest.php @@ -15,6 +15,7 @@ use Hypervel\Grpc\Protocol\FrameEncoder; use Hypervel\Grpc\Protocol\MessageSerializer; use Hypervel\Grpc\Server\CallContextStore; +use Hypervel\Grpc\Server\ExceptionMapper; use Hypervel\Grpc\Server\GrpcStreamedResponse; use Hypervel\Grpc\Server\Middleware\HandleCall; use Hypervel\Grpc\Server\ResponseFactory; @@ -24,14 +25,29 @@ use Hypervel\Tests\TestCase; use LogicException; use Mockery as m; +use PHPUnit\Framework\Attributes\TestWith; +use RuntimeException; use Swoole\Coroutine\CanceledException; +use Throwable; class HandleCallTest extends TestCase { + private ExceptionMapper $exceptions; + + /** + * Set up the exception mapper. + */ + protected function setUp(): void + { + parent::setUp(); + + $this->exceptions = m::mock(ExceptionMapper::class); + } + public function testDecodesTheRequestAndScopesItsCallContext(): void { $contexts = new CallContextStore; - $handler = new HandleCall($contexts, new Timer, 1024, Compression::Gzip); + $handler = new HandleCall($contexts, new Timer, $this->exceptions, 1024, Compression::Gzip); $message = (new Any)->setTypeUrl('testing.Request')->setValue('payload'); $request = $this->request($message, server: [ 'REMOTE_ADDR' => '2001:db8::1', @@ -68,7 +84,7 @@ public function testDecodesTheRequestAndScopesItsCallContext(): void public function testDecodesGzipRequestMessages(): void { $contexts = new CallContextStore; - $handler = new HandleCall($contexts, new Timer, 1024, Compression::Identity); + $handler = new HandleCall($contexts, new Timer, $this->exceptions, 1024, Compression::Identity); $message = (new Any)->setValue(str_repeat('compressible', 20)); $request = $this->request($message, Compression::Gzip, [ 'HTTP_GRPC_ENCODING' => 'gzip', @@ -89,7 +105,7 @@ public function testRequiresExactlyOneRequestFrame(): void 'multiple' => $this->body(new GPBEmpty) . $this->body(new GPBEmpty), ] as $case => $body) { $contexts = new CallContextStore; - $handler = new HandleCall($contexts, new Timer, 1024, Compression::Identity); + $handler = new HandleCall($contexts, new Timer, $this->exceptions, 1024, Compression::Identity); $request = $this->request(new GPBEmpty, body: $body); try { @@ -106,7 +122,7 @@ public function testRequiresExactlyOneRequestFrame(): void public function testRejectsUnsupportedRequestCompression(): void { - $handler = new HandleCall(new CallContextStore, new Timer, 1024, Compression::Identity); + $handler = new HandleCall(new CallContextStore, new Timer, $this->exceptions, 1024, Compression::Identity); $request = $this->request(new GPBEmpty, server: [ 'HTTP_GRPC_ENCODING' => 'snappy', ]); @@ -139,7 +155,7 @@ public function testRejectsMalformedTimeoutAndPreviousAttemptHeaders(): void 'The grpc-previous-rpc-attempts request header is too large.', ], ] as $case => [$name, $value, $message]) { - $handler = new HandleCall(new CallContextStore, new Timer, 1024, Compression::Identity); + $handler = new HandleCall(new CallContextStore, new Timer, $this->exceptions, 1024, Compression::Identity); $request = $this->request(new GPBEmpty, server: [$name => $value]); try { @@ -160,7 +176,7 @@ public function testRegistersAndClearsOneDeadlineTimer(): void ->andReturn(41); $timer->shouldReceive('clear')->once()->with(41); $contexts = new CallContextStore; - $handler = new HandleCall($contexts, $timer, 1024, Compression::Identity); + $handler = new HandleCall($contexts, $timer, $this->exceptions, 1024, Compression::Identity); $request = $this->request(new GPBEmpty, server: ['HTTP_GRPC_TIMEOUT' => '1S']); $result = $handler->handle($request, function () use ($contexts): string { @@ -180,7 +196,7 @@ public function testForgetsTheCallContextWhenDeadlineTimerRegistrationFails(): v $timer->shouldReceive('after')->once()->andThrow($failure); $timer->shouldNotReceive('clear'); $contexts = new CallContextStore; - $handler = new HandleCall($contexts, $timer, 1024, Compression::Identity); + $handler = new HandleCall($contexts, $timer, $this->exceptions, 1024, Compression::Identity); $request = $this->request(new GPBEmpty, server: ['HTTP_GRPC_TIMEOUT' => '1S']); try { @@ -199,7 +215,7 @@ public function testRechecksTheDeadlineAfterNonYieldingServiceWork(): void $timer = m::mock(Timer::class); $timer->shouldReceive('after')->once()->andReturn(42); $timer->shouldReceive('clear')->once()->with(42); - $handler = new HandleCall(new CallContextStore, $timer, 1024, Compression::Identity); + $handler = new HandleCall(new CallContextStore, $timer, $this->exceptions, 1024, Compression::Identity); // Setup counts against the deadline, so leave headroom under suite load while the service work still outlasts it. $request = $this->request(new GPBEmpty, server: ['HTTP_GRPC_TIMEOUT' => '100m']); @@ -213,20 +229,58 @@ public function testRechecksTheDeadlineAfterNonYieldingServiceWork(): void $this->assertSame(StatusCode::DeadlineExceeded, $result->status()->code()); } - public function testRethrowsCancellationThatWasNotCausedByItsDeadline(): void + #[TestWith([false])] + #[TestWith([true])] + public function testDeadlineCancellationReportsCleanupFailuresAndKeepsDeadlineStatus(bool $cleanupThrows): void + { + $cleanupException = new RuntimeException('Service cleanup failed.'); + if ($cleanupThrows) { + $this->exceptions->shouldReceive('report')->once()->with($cleanupException); + } else { + $this->exceptions->shouldNotReceive('report'); + } + $handler = new HandleCall(new CallContextStore, new Timer, $this->exceptions, 1024, Compression::Identity); + $request = $this->request(new GPBEmpty, server: ['HTTP_GRPC_TIMEOUT' => '100m']); + $cancellation = null; + + $result = $handler->handle($request, static function () use ($cleanupThrows, $cleanupException, &$cancellation): void { + try { + usleep(300_000); + } catch (CanceledException $exception) { + $cancellation = $exception; + + throw $exception; + } finally { + if ($cleanupThrows) { + usleep(1000); + + throw $cleanupException; + } + } + }); + + $this->assertInstanceOf(CanceledException::class, $cancellation); + $this->assertInstanceOf(RpcException::class, $result); + $this->assertSame(StatusCode::DeadlineExceeded, $result->status()->code()); + } + + #[TestWith([false])] + #[TestWith([true])] + public function testRethrowsCancellationThatWasNotCausedByItsDeadline(bool $cleanupThrows): void { $contexts = new CallContextStore; - $handler = new HandleCall($contexts, new Timer, 1024, Compression::Identity); + $handler = new HandleCall($contexts, new Timer, $this->exceptions, 1024, Compression::Identity); $request = $this->request(new GPBEmpty); $cancellation = new CanceledException; + $failure = $cleanupThrows ? new RuntimeException('Service cleanup failed.', previous: $cancellation) : $cancellation; try { - $handler->handle($request, static function () use ($cancellation): never { - throw $cancellation; + $handler->handle($request, static function () use ($failure): never { + throw $failure; }); $this->fail('Expected unrelated cancellation to escape.'); - } catch (CanceledException $exception) { - $this->assertSame($cancellation, $exception); + } catch (Throwable $exception) { + $this->assertSame($failure, $exception); } $this->expectException(LogicException::class); @@ -236,7 +290,7 @@ public function testRethrowsCancellationThatWasNotCausedByItsDeadline(): void public function testTransfersCleanupToAStreamedResponse(): void { $contexts = new CallContextStore; - $handler = new HandleCall($contexts, new Timer, 1024, Compression::Identity); + $handler = new HandleCall($contexts, new Timer, $this->exceptions, 1024, Compression::Identity); $request = $this->request(new GPBEmpty); $response = new GrpcStreamedResponse( ['frame'], diff --git a/tests/Grpc/ResponseFactoryTest.php b/tests/Grpc/ResponseFactoryTest.php index b57fa1b42e..c3e2158150 100644 --- a/tests/Grpc/ResponseFactoryTest.php +++ b/tests/Grpc/ResponseFactoryTest.php @@ -27,6 +27,7 @@ use Iterator; use Mockery as m; use PHPUnit\Framework\Attributes\DataProvider; +use PHPUnit\Framework\Attributes\TestWith; use RuntimeException; use Swoole\Coroutine\CanceledException; use Symfony\Component\HttpFoundation\Response; @@ -230,19 +231,53 @@ public function testMapsMidStreamExpectedFailureToFinalTrailers(): void $this->assertSame('-1', $trailers['grpc-retry-pushback-ms']); } - public function testPreservesCancellationWhileAdvancingAStream(): void + #[TestWith([false, false])] + #[TestWith([false, true])] + #[TestWith([true, false])] + #[TestWith([true, true])] + public function testPreservesCancellationWhileAdvancingAStream(bool $deadlineExpired, bool $cleanupThrows): void { - [$factory] = $this->factory(); + $handler = m::mock(ExceptionHandler::class); $cancellation = new CanceledException; - $messages = static function () use ($cancellation): iterable { + $cleanupException = new RuntimeException('Stream cleanup failed.'); + if ($deadlineExpired && $cleanupThrows) { + $handler->shouldReceive('report')->once()->with($cleanupException); + } else { + $handler->shouldNotReceive('report'); + } + [$factory, $contexts] = $this->factory($handler); + $now = 0; + if ($deadlineExpired) { + $contexts->set(new ServerCallContext( + Metadata::make(), + 'testing.Service', + 'Call', + '127.0.0.1:50051', + null, + Deadline::usingClock(1, static function () use (&$now): int { return $now; }), + 0, + )); + } + $messages = static function () use ($cancellation, $cleanupThrows, $cleanupException, &$now): iterable { yield new GPBEmpty; - throw $cancellation; + $now = 1; + try { + throw $cancellation; + } finally { + if ($cleanupThrows) { + throw $cleanupException; + } + } }; $response = $factory->make($this->request(serverStreaming: true), $messages()); - $this->expectExceptionObject($cancellation); + if (! $deadlineExpired) { + $this->expectExceptionObject($cleanupThrows ? $cleanupException : $cancellation); + } $response->streamTo(static fn (): bool => true); + + $this->assertSame((string) StatusCode::DeadlineExceeded->value, $response->trailers()['grpc-status']); } public function testReportsInvalidUnaryAndStreamedServiceValues(): void diff --git a/tests/Grpc/ServerTest.php b/tests/Grpc/ServerTest.php index 3fa79225c3..3d580fa77c 100644 --- a/tests/Grpc/ServerTest.php +++ b/tests/Grpc/ServerTest.php @@ -398,6 +398,7 @@ private function environment( $handler = new HandleCall( $contexts, new Timer, + $exceptions, 4 * 1024 * 1024, Compression::Identity, ); diff --git a/tests/Grpc/UnaryCallTest.php b/tests/Grpc/UnaryCallTest.php index 283db21224..9c68b7a5b2 100644 --- a/tests/Grpc/UnaryCallTest.php +++ b/tests/Grpc/UnaryCallTest.php @@ -27,6 +27,7 @@ use Hypervel\Grpc\Protocol\ServiceMethod; use Hypervel\Grpc\StatusCode; use Hypervel\Tests\TestCase; +use PHPUnit\Framework\Attributes\TestWith; use Random\Engine\Mt19937; use Random\Randomizer; use RuntimeException; @@ -72,21 +73,37 @@ static function (string $payload) use (&$deserializations): Message { $this->assertSame($results['first'], $call->wait()); } - public function testCanceledUnaryDeserializationCanBeResumedByAnotherWaiter(): void + #[TestWith([false])] + #[TestWith([true])] + public function testCanceledUnaryDeserializationCanBeResumedByAnotherWaiter(bool $cleanupThrows): void { $deadline = Deadline::fromTimeout(null); $state = $this->state($deadline); $this->respondSuccessfully($state, 'hello'); - $cancellation = new CanceledException; + $cancellation = null; + $cleanupException = new RuntimeException('Deserializer cleanup failed.'); + $blocker = new Channel(1); $deserializations = 0; $call = $this->call( $state, $deadline, - static function (string $payload) use (&$deserializations, $cancellation): Message { + static function (string $payload) use (&$deserializations, &$cancellation, $blocker, $cleanupThrows, $cleanupException): Message { ++$deserializations; if ($deserializations === 1) { - throw $cancellation; + try { + $blocker->pop(); + } catch (CanceledException $exception) { + $cancellation = $exception; + + throw $exception; + } finally { + if ($cleanupThrows) { + usleep(1000); + + throw $cleanupException; + } + } } $message = new StringValue; @@ -96,15 +113,26 @@ static function (string $payload) use (&$deserializations, $cancellation): Messa }, ); + $failure = null; + $waiter = EngineCoroutine::create(static function () use ($call, &$failure): void { + try { + $call->wait(); + } catch (Throwable $exception) { + $failure = $exception; + } + }); + try { - $call->wait(); - $this->fail('Expected cancellation to propagate.'); - } catch (CanceledException $exception) { - $this->assertSame($cancellation, $exception); + $this->assertTrue(EngineCoroutine::cancelById($waiter->getId(), throwException: true)); + EngineCoroutine::join([$waiter->getId()], 1); + $this->assertInstanceOf(CanceledException::class, $cancellation); + $this->assertSame($cleanupThrows ? $cleanupException : $cancellation, $failure); + $this->assertSame('hello', $call->wait()->getValue()); + $this->assertSame(2, $deserializations); + } finally { + $blocker->close(); + EngineCoroutine::join([$waiter->getId()], 1); } - - $this->assertSame('hello', $call->wait()->getValue()); - $this->assertSame(2, $deserializations); } public function testCancellingOneMetadataWaiterDoesNotFailTheCall(): void @@ -437,38 +465,55 @@ public function testWaiterCancellationDuringRetryBackoffRestoresTheBackoffSequen } } - public function testCanceledRetryDoesNotPreventALaterRetry(): void + #[TestWith([false])] + #[TestWith([true])] + public function testCanceledRetryDoesNotPreventALaterRetry(bool $cleanupThrows): void { $deadline = Deadline::fromTimeout(null); $first = $this->state($deadline); - $first->handle($this->trailersOnly( - status: StatusCode::Unavailable, - headers: ['grpc-retry-pushback-ms' => '0'], - )); + $first->handle($this->trailersOnly(status: StatusCode::Unavailable)); + $policy = new RetryPolicy(maxAttempts: 2, initialBackoff: 0.001, maxBackoff: 0.001); + $backoff = new RetryBackoff($policy, new Randomizer(new Mt19937(1234))); $factoryStarted = new Channel(2); $releaseFactory = new Channel(1); $waiterCompleted = new Channel(1); $cancellation = null; $failure = null; $attempts = 0; + $cleanupException = new RuntimeException('Retry preparation cleanup failed.'); $call = $this->call( $first, $deadline, - retryPolicy: new RetryPolicy(maxAttempts: 2), + retryPolicy: $policy, attemptFactory: function () use ( $deadline, $factoryStarted, $releaseFactory, &$attempts, + $cleanupThrows, + $cleanupException, ): StreamState { - $factoryStarted->push(true); - $releaseFactory->pop(); + try { + $factoryStarted->push(true); + $releaseFactory->pop(); + } catch (CanceledException $exception) { + try { + throw $exception; + } finally { + if ($cleanupThrows) { + usleep(1000); + + throw $cleanupException; + } + } + } ++$attempts; $state = $this->state($deadline); $this->respondSuccessfully($state, 'retried'); return $state; }, + retryBackoff: $backoff, ); $waiter = EngineCoroutine::create(static function () use ( $call, @@ -491,8 +536,15 @@ public function testCanceledRetryDoesNotPreventALaterRetry(): void $this->assertTrue($factoryStarted->pop(1)); $this->assertTrue(EngineCoroutine::cancelById($waiter->getId(), throwException: true)); $this->assertTrue($waiterCompleted->pop(1)); - $this->assertInstanceOf(CanceledException::class, $cancellation); - $this->assertNull($failure); + EngineCoroutine::join([$waiter->getId()], 1); + if ($cleanupThrows) { + $this->assertSame($cleanupException, $failure); + $this->assertInstanceOf(CanceledException::class, $failure->getPrevious()); + } else { + $this->assertInstanceOf(CanceledException::class, $cancellation); + $this->assertNull($failure); + } + $this->assertSame(0, $backoff->checkpoint()); $this->assertSame(0, $attempts); $this->assertTrue($releaseFactory->push(true)); diff --git a/tests/Http/HttpClientTest.php b/tests/Http/HttpClientTest.php index 2a5e294e75..c8673b6df0 100644 --- a/tests/Http/HttpClientTest.php +++ b/tests/Http/HttpClientTest.php @@ -4608,16 +4608,19 @@ public function testRequestExceptionIsThrownWhenRetriesExhaustedWithBackoffArray $this->factory->assertSentCount(2); } - public function testSynchronousRequestCancellationIsNotRetriedOrPassedToRetryPolicy(): void + #[TestWith([false])] + #[TestWith([true])] + public function testSynchronousRequestCancellationIsNotRetriedOrPassedToRetryPolicy(bool $cleanupThrows): void { $cancellation = new CanceledException('request canceled'); + $failure = $cleanupThrows ? new RuntimeException('Cleanup failed.', previous: $cancellation) : $cancellation; $attempts = 0; $policyCalled = false; - $this->factory->fake(function () use (&$attempts, $cancellation) { + $this->factory->fake(function () use (&$attempts, $failure) { ++$attempts; - return Create::rejectionFor($cancellation); + return Create::rejectionFor($failure); }); try { @@ -4629,8 +4632,8 @@ public function testSynchronousRequestCancellationIsNotRetriedOrPassedToRetryPol }) ->get('http://foo.com/get'); $this->fail('Expected the request cancellation to be thrown.'); - } catch (CanceledException $exception) { - $this->assertSame($cancellation, $exception); + } catch (Throwable $exception) { + $this->assertSame($failure, $exception); } $this->assertSame(1, $attempts); @@ -4638,16 +4641,19 @@ public function testSynchronousRequestCancellationIsNotRetriedOrPassedToRetryPol $this->factory->assertSentCount(1); } - public function testAsynchronousRequestCancellationIsNotRetriedOrPassedToRetryPolicy(): void + #[TestWith([false])] + #[TestWith([true])] + public function testAsynchronousRequestCancellationIsNotRetriedOrPassedToRetryPolicy(bool $cleanupThrows): void { $cancellation = new CanceledException('request canceled'); + $failure = $cleanupThrows ? new RuntimeException('Cleanup failed.', previous: $cancellation) : $cancellation; $attempts = 0; $policyCalled = false; - $this->factory->fake(function () use (&$attempts, $cancellation) { + $this->factory->fake(function () use (&$attempts, $failure) { ++$attempts; - return Create::rejectionFor($cancellation); + return Create::rejectionFor($failure); }); $promise = $this->factory @@ -4662,8 +4668,8 @@ public function testAsynchronousRequestCancellationIsNotRetriedOrPassedToRetryPo try { $promise->wait(); $this->fail('Expected the asynchronous request cancellation to be thrown.'); - } catch (CanceledException $exception) { - $this->assertSame($cancellation, $exception); + } catch (Throwable $exception) { + $this->assertSame($failure, $exception); } $this->assertSame(1, $attempts); @@ -4671,50 +4677,91 @@ public function testAsynchronousRequestCancellationIsNotRetriedOrPassedToRetryPo $this->factory->assertSentCount(1); } - public function testAsynchronousRetryCallbackCancellationRejectsPromise(): void + #[TestWith([false])] + #[TestWith([true])] + public function testAsynchronousRetryCallbackCancellationRejectsPromise(bool $cleanupThrows): void { $cancellation = new CanceledException('retry callback canceled'); + $failure = $cleanupThrows ? new RuntimeException('Cleanup failed.', previous: $cancellation) : $cancellation; $this->factory->fake([ '*' => $this->factory::response(['error'], 500), ]); $promise = $this->factory ->async() - ->retry(3, when: fn () => throw $cancellation) + ->retry(3, when: fn () => throw $failure) ->get('http://foo.com/get'); try { $promise->wait(); $this->fail('Expected the retry callback cancellation to be thrown.'); - } catch (CanceledException $exception) { - $this->assertSame($cancellation, $exception); + } catch (Throwable $exception) { + $this->assertSame($failure, $exception); } $this->factory->assertSentCount(1); } - public function testAsynchronousThrowCallbackCancellationRejectsPromise(): void + #[TestWith([false])] + #[TestWith([true])] + public function testAsynchronousThrowCallbackCancellationRejectsPromise(bool $cleanupThrows): void { $cancellation = new CanceledException('throw callback canceled'); + $failure = $cleanupThrows ? new RuntimeException('Cleanup failed.', previous: $cancellation) : $cancellation; $this->factory->fake([ '*' => $this->factory::response(['error'], 500), ]); $promise = $this->factory ->async() - ->throw(fn () => throw $cancellation) + ->throw(fn () => throw $failure) ->get('http://foo.com/get'); try { $promise->wait(); $this->fail('Expected the throw callback cancellation to be thrown.'); - } catch (CanceledException $exception) { - $this->assertSame($cancellation, $exception); + } catch (Throwable $exception) { + $this->assertSame($failure, $exception); } $this->factory->assertSentCount(1); } + #[TestWith([false, 200])] + #[TestWith([false, 500])] + #[TestWith([true, 200])] + #[TestWith([true, 500])] + public function testResponseBearingTransportFailurePreservesChainedCancellation(bool $async, int $status): void + { + $failure = new RuntimeException('Sink cleanup failed.', previous: new CanceledException('Request canceled.')); + $attempts = 0; + $policyCalled = false; + $sink = m::mock(StreamInterface::class); + $sink->shouldReceive('isWritable')->andReturnTrue(); + $sink->shouldReceive('write')->andReturnUsing(function () use (&$attempts, $failure): never { + ++$attempts; + + throw $failure; + }); + $this->factory->fake(['*' => $this->factory::response('Response body', $status)]); + $request = $this->factory->withOptions(['sink' => $sink])->retry(2, 0, function () use (&$policyCalled): bool { + $policyCalled = true; + + return true; + }); + $caught = null; + + try { + $async ? $request->async()->get('https://example.com')->wait() : $request->get('https://example.com'); + } catch (Throwable $exception) { + $caught = $exception; + } + + $this->assertSame($failure, $caught); + $this->assertSame(1, $attempts); + $this->assertFalse($policyCalled); + } + public function testRequestExceptionIsThrownWithoutRetriesIfRetryNotNecessary(): void { $this->factory->fake([ diff --git a/tests/Image/CoroutineSafetyTest.php b/tests/Image/CoroutineSafetyTest.php index ccabfdf8b5..6eed8d2f5a 100644 --- a/tests/Image/CoroutineSafetyTest.php +++ b/tests/Image/CoroutineSafetyTest.php @@ -9,6 +9,7 @@ use Hypervel\Contracts\Image\Driver; use Hypervel\Contracts\Image\Transformation; use Hypervel\Coroutine\Coroutine; +use Hypervel\Coroutine\Exceptions\ForeignCancellationException; use Hypervel\Engine\Channel; use Hypervel\Engine\Coroutine as EngineCoroutine; use Hypervel\Image\Image; @@ -16,6 +17,7 @@ use Hypervel\Image\ImageManager; use Hypervel\Image\ImagePipeline; use Hypervel\Tests\TestCase; +use PHPUnit\Framework\Attributes\TestWith; use RuntimeException; use Swoole\Coroutine\CanceledException; use Throwable; @@ -86,7 +88,9 @@ public function testClonedImagesReceiveTheOriginalTerminalSourceException(): voi $this->assertSame(1, $resolutionCalls); } - public function testOwnerCancellationReachesOnlyTheOwnerAndOthersGetTheCachedFailure(): void + #[TestWith([false])] + #[TestWith([true])] + public function testOwnerCancellationReachesOnlyTheOwnerAndOthersGetTheCachedFailure(bool $cleanupThrows): void { $container = new Container; $container->instance('config', new Repository([ @@ -99,9 +103,18 @@ public function testOwnerCancellationReachesOnlyTheOwnerAndOthersGetTheCachedFai $resolutionCalls = 0; $release = new Channel(1); - $image = new Image(function () use (&$resolutionCalls, $release): string { + $cleanupException = new RuntimeException('Image source cleanup failed.'); + $image = new Image(function () use (&$resolutionCalls, $release, $cleanupThrows, $cleanupException): string { ++$resolutionCalls; - $release->pop(); + try { + $release->pop(); + } finally { + if ($cleanupThrows) { + usleep(1000); + + throw $cleanupException; + } + } return 'shared image'; }); @@ -129,11 +142,22 @@ public function testOwnerCancellationReachesOnlyTheOwnerAndOthersGetTheCachedFai $first = $failures->pop(1); $second = $failures->pop(1); + Coroutine::join([$ownerId, $waiterId], 1); $results = [$first[0] => $first[1], $second[0] => $second[1]]; - $this->assertInstanceOf(CanceledException::class, $results['owner']); + if ($cleanupThrows) { + $this->assertInstanceOf(ImageException::class, $results['owner']); + $this->assertSame($cleanupException, $results['owner']->getPrevious()); + $this->assertInstanceOf(CanceledException::class, $cleanupException->getPrevious()); + } else { + $this->assertInstanceOf(CanceledException::class, $results['owner']); + } + + $this->assertTrue(Coroutine::causedByCancellation($results['owner'])); $this->assertInstanceOf(ImageException::class, $results['waiter']); - $this->assertSame($results['owner'], $results['waiter']->getPrevious()); + $this->assertInstanceOf(ForeignCancellationException::class, $results['waiter']->getPrevious()); + $this->assertSame($cleanupThrows ? $cleanupException : $results['owner'], $results['waiter']->getPrevious()->getPrevious()); + $this->assertFalse(Coroutine::causedByCancellation($results['waiter'])); } finally { $release->push(true, 0.001); diff --git a/tests/Integration/Database/PooledConnectionTest.php b/tests/Integration/Database/PooledConnectionTest.php index e3d2b1cbfa..99106080f4 100644 --- a/tests/Integration/Database/PooledConnectionTest.php +++ b/tests/Integration/Database/PooledConnectionTest.php @@ -33,6 +33,7 @@ use Mockery as m; use PDO; use PHPUnit\Framework\Attributes\DataProvider; +use PHPUnit\Framework\Attributes\TestWith; use ReflectionProperty; use RuntimeException; use Swoole\Coroutine\CanceledException; @@ -1133,15 +1134,18 @@ public function testRollbackCancellationStillReturnsTheConnectionAndEscapesExact $pool->close(); } - public function testReleaseListenerCancellationStillReturnsTheConnectionAndEscapesExactly(): void + #[TestWith([false])] + #[TestWith([true])] + public function testReleaseListenerCancellationStillReturnsTheConnectionAndEscapesExactly(bool $cleanupThrows): void { $this->app->make('config')->set('database.connections.pool_test.pool.events', [ ConnectionReleasing::class, ]); $cancellation = new CanceledException('Release listener was canceled.'); + $failure = $cleanupThrows ? new RuntimeException('Release listener cleanup failed.', previous: $cancellation) : $cancellation; $this->app->make(Dispatcher::class)->listen( ConnectionReleasing::class, - static fn () => throw $cancellation + static fn () => throw $failure ); $pool = new DatabasePool($this->app, 'pool_test'); @@ -1151,8 +1155,8 @@ public function testReleaseListenerCancellationStillReturnsTheConnectionAndEscap try { $pooledConnection->release(); $this->fail('Expected release listener cancellation to propagate.'); - } catch (CanceledException $throwable) { - $this->assertSame($cancellation, $throwable); + } catch (Throwable $throwable) { + $this->assertSame($failure, $throwable); } $this->assertTrue($this->isInvalid($pooledConnection)); diff --git a/tests/Integration/Foundation/FoundationHelpersTest.php b/tests/Integration/Foundation/FoundationHelpersTest.php index 0b35a20e5a..27d8033e4b 100644 --- a/tests/Integration/Foundation/FoundationHelpersTest.php +++ b/tests/Integration/Foundation/FoundationHelpersTest.php @@ -10,7 +10,9 @@ use Hypervel\Contracts\Debug\ExceptionHandler; use Hypervel\Testbench\TestCase; use Mockery\MockInterface; +use PHPUnit\Framework\Attributes\TestWith; use Psr\Log\LogLevel; +use RuntimeException; use Swoole\Coroutine\CanceledException; use Throwable; @@ -83,15 +85,18 @@ public function test(int $a) ); } - public function testRescuePreservesCancellation(): void + #[TestWith([false])] + #[TestWith([true])] + public function testRescuePreservesCancellation(bool $cleanupThrows): void { $cancellation = new CanceledException('canceled'); + $failure = $cleanupThrows ? new RuntimeException('Cleanup failed.', previous: $cancellation) : $cancellation; $reported = false; $rescued = false; try { rescue( - fn () => throw $cancellation, + fn () => throw $failure, function () use (&$rescued): void { $rescued = true; }, @@ -103,8 +108,8 @@ function () use (&$reported): bool { ); $this->fail('The cancellation was not preserved.'); - } catch (CanceledException $exception) { - $this->assertSame($cancellation, $exception); + } catch (Throwable $exception) { + $this->assertSame($failure, $exception); } $this->assertFalse($reported); diff --git a/tests/Integration/Queue/FailOnExceptionMiddlewareTest.php b/tests/Integration/Queue/FailOnExceptionMiddlewareTest.php index 1f96409851..c58bf23a1f 100644 --- a/tests/Integration/Queue/FailOnExceptionMiddlewareTest.php +++ b/tests/Integration/Queue/FailOnExceptionMiddlewareTest.php @@ -7,6 +7,7 @@ use Hypervel\Bus\Dispatcher; use Hypervel\Bus\Queueable; use Hypervel\Contracts\Queue\ShouldQueue; +use Hypervel\Coroutine\Exceptions\ChildCancellationException; use Hypervel\Engine\Channel; use Hypervel\Engine\Coroutine as EngineCoroutine; use Hypervel\Foundation\Bus\Dispatchable; @@ -111,7 +112,9 @@ public function testCanTestAgainstJobProperties(mixed $value, bool $expectedToFa $expectedToFail ? $job->assertFailed() : $job->assertNotFailed(); } - public function testCancellationBypassesTheFailurePredicateAndJobFailure(): void + #[TestWith([false])] + #[TestWith([true])] + public function testCancellationBypassesTheFailurePredicateAndJobFailure(bool $cleanupThrows): void { $predicateCalled = false; $middleware = new FailOnException( @@ -124,19 +127,79 @@ static function () use (&$predicateCalled): bool { $job = m::mock(); $job->shouldNotReceive('fail'); $gate = $this->armCurrentCoroutineCancellation(); + $cleanupException = new LogicException('Cleanup failed.'); try { - $middleware->handle($job, static function () use ($gate): never { - $gate->push(true); - - throw new LogicException('Cancellation was not delivered.'); + $middleware->handle($job, static function () use ($gate, $cleanupThrows, $cleanupException): never { + try { + $gate->push(true); + + throw new LogicException('Cancellation was not delivered.'); + } finally { + if ($cleanupThrows) { + throw $cleanupException; + } + } }); $this->fail('Expected cancellation to escape the middleware.'); - } catch (CanceledException) { + } catch (Throwable $exception) { + if ($cleanupThrows) { + $this->assertSame($cleanupException, $exception); + $this->assertInstanceOf(CanceledException::class, $exception->getPrevious()); + } else { + $this->assertInstanceOf(CanceledException::class, $exception); + } + $this->assertFalse($predicateCalled); } } + public function testIndependentChildCancellationAppliesTheFailurePolicy(): void + { + $job = (new FailOnExceptionMiddlewareTestJob(LogicException::class))->withFakeQueueInteractions(); + $failure = new ChildCancellationException(previous: new CanceledException); + $middleware = new FailOnException([ChildCancellationException::class]); + + try { + $middleware->handle($job, static fn () => throw $failure); + $this->fail('Expected the child failure.'); + } catch (ChildCancellationException $exception) { + $this->assertSame($failure, $exception); + } + + $job->assertFailedWith($failure)->assertNotReleased(); + } + + #[TestWith([false])] + #[TestWith([true])] + public function testReleasedJobIsNotFailedEvenWhenThePredicateReleasesIt(bool $releaseInPredicate): void + { + $job = (new FailOnExceptionMiddlewareTestJob(LogicException::class))->withFakeQueueInteractions(); + $exception = new LogicException('Job failed after release.'); + $middleware = new FailOnException(static function () use ($job, $releaseInPredicate): bool { + if ($releaseInPredicate) { + $job->release(30); + } + + return true; + }); + + try { + $middleware->handle($job, static function () use ($job, $releaseInPredicate, $exception): never { + if (! $releaseInPredicate) { + $job->release(30); + } + + throw $exception; + }); + $this->fail('Expected the original exception.'); + } catch (LogicException $caught) { + $this->assertSame($exception, $caught); + } + + $job->assertReleased(30)->assertNotFailed(); + } + /** * Arm exact cancellation of the current coroutine at a controlled channel handoff. */ diff --git a/tests/Integration/Queue/ThrottlesExceptionsTest.php b/tests/Integration/Queue/ThrottlesExceptionsTest.php index e344c4deb4..9ec1e948c6 100644 --- a/tests/Integration/Queue/ThrottlesExceptionsTest.php +++ b/tests/Integration/Queue/ThrottlesExceptionsTest.php @@ -8,11 +8,13 @@ use Hypervel\Bus\Dispatcher; use Hypervel\Bus\Queueable; use Hypervel\Contracts\Debug\ExceptionHandler; +use Hypervel\Coroutine\Exceptions\ChildCancellationException; use Hypervel\Engine\Channel; use Hypervel\Engine\Coroutine as EngineCoroutine; use Hypervel\Queue\CallQueuedHandler; use Hypervel\Queue\InteractsWithQueue; use Hypervel\Queue\Jobs\FakeJob; +use Hypervel\Queue\Middleware\FailOnException; use Hypervel\Queue\Middleware\ThrottlesExceptions; use Hypervel\RateLimiter\Limit; use Hypervel\RateLimiter\Limiter; @@ -21,8 +23,10 @@ use Hypervel\Support\CarbonImmutable; use Hypervel\Testbench\TestCase; use Mockery as m; +use PHPUnit\Framework\Attributes\TestWith; use RuntimeException; use Swoole\Coroutine\CanceledException; +use Throwable; class ThrottlesExceptionsTest extends TestCase { @@ -405,7 +409,9 @@ public function release(int $delay): static $this->assertSame(63, $job->releasedAfter); } - public function testCancellationBypassesFailurePolicyAndRateLimitAccounting(): void + #[TestWith([false])] + #[TestWith([true])] + public function testCancellationBypassesFailurePolicyAndRateLimitAccounting(bool $cleanupThrows): void { $limiter = m::mock(Limiter::class); $limiter->expects('inspect') @@ -443,19 +449,85 @@ public function testCancellationBypassesFailurePolicyAndRateLimitAccounting(): v $job = m::mock(); $job->shouldNotReceive('release', 'delete', 'fail'); $gate = $this->armCurrentCoroutineCancellation(); + $cleanupException = new RuntimeException('Cleanup failed.'); try { - $middleware->handle($job, static function () use ($gate): never { - $gate->push(true); - - throw new RuntimeException('Cancellation was not delivered.'); + $middleware->handle($job, static function () use ($gate, $cleanupThrows, $cleanupException): never { + try { + $gate->push(true); + + throw new RuntimeException('Cancellation was not delivered.'); + } finally { + if ($cleanupThrows) { + throw $cleanupException; + } + } }); $this->fail('Expected cancellation to escape the middleware.'); - } catch (CanceledException) { + } catch (Throwable $exception) { + if ($cleanupThrows) { + $this->assertSame($cleanupException, $exception); + $this->assertInstanceOf(CanceledException::class, $exception->getPrevious()); + } else { + $this->assertInstanceOf(CanceledException::class, $exception); + } + $this->assertFalse($callbacksCalled); } } + public function testIndependentChildCancellationIsCountedAndReleased(): void + { + $job = (new CircuitBreakerTestJob)->withFakeQueueInteractions(); + $failure = new ChildCancellationException(previous: new CanceledException); + $limiter = m::mock(Limiter::class); + $limiter->expects('inspect')->with(m::type(Limit::class))->andReturn(new LimitResult(true, 10, 10, 0, 0)); + $limiter->expects('consume')->with(m::type(Limit::class))->andReturn(new LimitResult(true, 10, 9, 0, 0)); + $limiter->shouldNotReceive('clear'); + $rateLimiter = m::mock(RateLimiter::class); + $rateLimiter->expects('store')->with(null)->andReturn($limiter); + $this->app->instance(RateLimiter::class, $rateLimiter); + $this->spy(ExceptionHandler::class)->expects('report')->with($failure, []); + + (new ThrottlesExceptions)->backoff(1)->report()->handle($job, static fn () => throw $failure); + + $job->assertReleased(60)->assertNotFailed(); + } + + #[TestWith(['release'])] + #[TestWith(['delete'])] + #[TestWith(['fail'])] + public function testInnerTerminalActionIsNotReplacedByAnotherRelease(string $action): void + { + $job = (new CircuitBreakerTestJob)->withFakeQueueInteractions(); + $exception = new RuntimeException('Inner pipeline failed.'); + $released = 0; + $failed = 0; + $job->job->beforeReleasing(static function () use (&$released): void { ++$released; }); + $job->job->beforeFailing(static function () use (&$failed): void { ++$failed; }); + + try { + (new ThrottlesExceptions)->handle($job, static function () use ($job, $action, $exception): never { + if ($action === 'fail') { + (new FailOnException([RuntimeException::class]))->handle($job, static function () use ($exception): never { + throw $exception; + }); + } else { + $job->{$action}(); + } + + throw $exception; + }); + $this->fail('Expected the original exception.'); + } catch (RuntimeException $caught) { + $this->assertSame($exception, $caught); + } + + $this->assertSame($action === 'release' ? 1 : 0, $released); + $this->assertSame($action === 'fail' ? 1 : 0, $failed); + $this->assertSame($action === 'delete', $job->job->isDeleted()); + } + public function testReportingExceptions(): void { $this->spy(ExceptionHandler::class) diff --git a/tests/Queue/FailoverQueueTest.php b/tests/Queue/FailoverQueueTest.php index 95e6ad2ab3..59abc64927 100644 --- a/tests/Queue/FailoverQueueTest.php +++ b/tests/Queue/FailoverQueueTest.php @@ -37,6 +37,7 @@ use Hypervel\Tests\TestCase; use Mockery as m; use PHPUnit\Framework\Attributes\DataProvider; +use PHPUnit\Framework\Attributes\TestWith; use RuntimeException; use Swoole\Coroutine\CanceledException; use Throwable; @@ -154,12 +155,17 @@ public function testReplacementQueueReportsItsOwnFirstFailure(): void $this->assertSame('job-id', $replacement->push('job')); } - public function testCancellationDoesNotFailOverOrEmitAnEvent(): void + #[TestWith([false])] + #[TestWith([true])] + public function testCancellationDoesNotFailOverOrEmitAnEvent(bool $cleanupThrows): void { $manager = m::mock(QueueManager::class); $events = m::mock(DispatcherContract::class); $primary = m::mock(Queue::class); $cancellation = new CanceledException; + if ($cleanupThrows) { + $cancellation = new RuntimeException('Queue cleanup failed.', previous: $cancellation); + } $failover = new FailoverQueue($manager, $events, ['primary', 'secondary']); $manager->shouldReceive('connection')->once()->with('primary')->andReturn($primary); @@ -170,18 +176,23 @@ public function testCancellationDoesNotFailOverOrEmitAnEvent(): void try { $failover->push('job'); $this->fail('Expected cancellation to escape the failover queue.'); - } catch (CanceledException $exception) { + } catch (Throwable $exception) { $this->assertSame($cancellation, $exception); } } - public function testCancellationAfterAnOrdinaryFailureRetainsFailureSuppressionState(): void + #[TestWith([false])] + #[TestWith([true])] + public function testCancellationAfterAnOrdinaryFailureRetainsFailureSuppressionState(bool $cleanupThrows): void { $events = new FailoverQueueFakeDispatcher; $manager = m::mock(QueueManager::class); $primary = m::mock(Queue::class); $secondary = m::mock(Queue::class); $cancellation = new CanceledException; + if ($cleanupThrows) { + $cancellation = new RuntimeException('Queue cleanup failed.', previous: $cancellation); + } $secondaryAttempts = 0; $failover = new FailoverQueue($manager, $events, ['primary', 'secondary', 'tertiary']); @@ -202,7 +213,7 @@ static function () use (&$secondaryAttempts, $cancellation): string { try { $failover->push('first'); $this->fail('Expected cancellation to stop the first failover attempt.'); - } catch (CanceledException $exception) { + } catch (Throwable $exception) { $this->assertSame($cancellation, $exception); } diff --git a/tests/Queue/Fixtures/timeout-worker.php b/tests/Queue/Fixtures/timeout-worker.php new file mode 100644 index 0000000000..f0019b8206 --- /dev/null +++ b/tests/Queue/Fixtures/timeout-worker.php @@ -0,0 +1,196 @@ +attempts(), "\n"; + + if ($job->attempts() > 1) { + $job->delete(); + echo "retried\n"; + + return; + } + + if ($data['mode'] === 'suspended') { + Coroutine::yield(); + + throw new RuntimeException('The suspended attempt was unexpectedly resumed.'); + } + + if ($data['mode'] === 'released-timeout') { + $job->release(30); + echo "released\n"; + Coroutine::sleep(30); + + return; + } + + if ($data['mode'] === 'catch-release') { + try { + Coroutine::sleep(30); + } catch (Exception) { + $job->release(30); + echo "released\n"; + } + + return; + } + + if (in_array($data['mode'], ['unwind', 'unwind-stall'], true)) { + try { + Coroutine::sleep(30); + } finally { + echo "unwind-start\n"; + Coroutine::sleep($data['mode'] === 'unwind-stall' ? 30 : 0.02); + echo "unwind-finished\n"; + } + + return; + } + + if (in_array($data['mode'], ['yielding', 'blocked-cleanup'], true)) { + Coroutine::sleep(30); + } else { + while (true); + } + } + + /** + * Require coroutine-aware failure cleanup before recording completion. + */ + public function failed(array $data, Throwable $exception, string $uuid, Job $job): void + { + echo "failed-start\n"; + Coroutine::sleep($data['mode'] === 'yielding' ? 4 : 0.02); + echo "failed-finished\n"; + } +} + +class LateParkTimeoutWorkerJob implements Interruptible, ShouldQueue +{ + use InteractsWithQueue; + + protected ?Channel $gate = null; + + /** + * Block the loop only after the monitor has declared the timeout. + */ + public function handle(): void + { + echo 'attempt:', $this->job->attempts(), "\n"; + $this->gate = new Channel(1); + $this->gate->pop(30); + + while (true); + } + + /** + * Resume the attempt while its timeout notification is still on the stack. + */ + public function interrupted(int $signal): void + { + $this->gate->push(true); + } +} + +Runtime::enableCoroutine(SWOOLE_HOOK_ALL); + +Coroutine\run(static function () use ($argv): void { + [$script, $databasePath, $mode] = $argv; + $container = new Container; + Container::setInstance($container); + $container->instance(CallQueuedHandler::class, new CallQueuedHandler(new BusDispatcher($container), $container)); + $database = new Manager($container); + $database->addConnection(['driver' => 'sqlite', 'database' => $databasePath]); + $connection = $database->getConnection(); + $connection->statement('CREATE TABLE IF NOT EXISTS jobs (id INTEGER PRIMARY KEY AUTOINCREMENT, queue TEXT, payload TEXT, attempts INTEGER, reserved_at INTEGER, available_at INTEGER, created_at INTEGER)'); + $connection->statement('CREATE TABLE IF NOT EXISTS failed_jobs (id INTEGER PRIMARY KEY AUTOINCREMENT, uuid TEXT, connection TEXT, queue TEXT, payload TEXT, exception TEXT, failed_at TEXT)'); + $container->make('config')->set('queue.failed', ['driver' => 'database-uuids', 'database' => 'default']); + $events = new Dispatcher($container); + $container->instance('events', $events); + $container->instance(DispatcherContract::class, $events); + $resolver = $container->make('db.resolver'); + $queue = (new DatabaseQueue($resolver, 'default', 'jobs', retryAfter: 10)) + ->setContainer($container) + ->setConnectionName('database'); + $failures = new DatabaseUuidFailedJobProvider($resolver, 'default', 'failed_jobs'); + + $events->listen(JobFailed::class, static function (JobFailed $event) use ($failures): void { + $failures->log($event->connectionName, $event->job->getQueue(), $event->job->getRawBody(), $event->exception); + echo "failure-recorded\n"; + }); + $events->listen(JobTimedOut::class, static function () use ($mode): void { + echo "timed-out\n"; + + if (in_array($mode, ['reblock', 'blocked-cleanup'], true)) { + while (true); + } + }); + $events->listen(WorkerStopping::class, static function (): void { + echo "stopping\n"; + }); + Worker::killUsing(static function (): void { + echo "kill-callback\n"; + }); + Worker::$pausable = false; + Worker::$restartable = false; + + if ($mode !== 'finish') { + $queue->pushRaw(json_encode([ + 'uuid' => 'timeout-worker-job', + 'job' => $mode === 'late-park' ? CallQueuedHandler::class . '@call' : TimeoutWorkerJob::class . '@fire', + 'data' => $mode === 'late-park' + ? ['commandName' => LateParkTimeoutWorkerJob::class, 'command' => serialize(new LateParkTimeoutWorkerJob)] + : ['mode' => $mode], + 'maxTries' => 3, + 'timeout' => 1, + 'failOnTimeout' => $mode !== 'retry', + ], JSON_THROW_ON_ERROR)); + } + + $manager = m::mock(Factory::class); + $manager->shouldReceive('connection')->andReturn($queue); + $exceptions = m::mock(ExceptionHandler::class); + $exceptions->shouldReceive('report')->andReturnUsing(static function (Throwable $exception): void { + fwrite(STDERR, (string) $exception); + }); + $worker = new Worker($manager, $events, $exceptions, static fn (): bool => false); + $worker->daemon('database', 'default', new WorkerOptions(memory: 0, sleep: 0.01, maxTries: 3, maxJobs: 1)); + echo "finished\n"; +}); diff --git a/tests/Queue/QueueBackgroundQueueTest.php b/tests/Queue/QueueBackgroundQueueTest.php index 89ffd19098..e229e1d8c4 100644 --- a/tests/Queue/QueueBackgroundQueueTest.php +++ b/tests/Queue/QueueBackgroundQueueTest.php @@ -34,6 +34,7 @@ use Mockery as m; use PHPUnit\Framework\Attributes\TestWith; use RuntimeException; +use Swoole\Coroutine\CanceledException; use Throwable; use function Hypervel\Coroutine\run; @@ -171,16 +172,21 @@ public function testFailedJobGetsHandledWhenAnExceptionIsThrown(): void $this->assertTrue($_SERVER['__background.failed']); } - public function testCancellationDoesNotInvokeBackgroundExceptionCallback(): void + #[TestWith([false])] + #[TestWith([true])] + public function testCancellationReportsOnlyCleanupErrorsThroughTheExceptionCallback(bool $cleanupThrows): void { CancelingBackgroundQueueTestHandler::$failed = false; - $result = null; + $reported = []; $background = new BackgroundQueue; - $background->setExceptionCallback(static function (Throwable $exception) use (&$result): void { - $result = $exception; + $background->setExceptionCallback(static function (Throwable $exception) use (&$reported): void { + $reported[] = $exception; }); $background->setConnectionName('background'); $container = $this->getContainer(); + $handler = new CancelingBackgroundQueueTestHandler; + $handler->cleanupException = $cleanupThrows ? new RuntimeException('Job cleanup failed.') : null; + $container->instance(CancelingBackgroundQueueTestHandler::class, $handler); $events = m::mock(Dispatcher::class); $events->shouldReceive('hasListeners')->once()->with(JobProcessing::class)->andReturnTrue(); $events->shouldReceive('dispatch')->once(); @@ -191,7 +197,10 @@ public function testCancellationDoesNotInvokeBackgroundExceptionCallback(): void try { run(fn () => $background->push(CancelingBackgroundQueueTestHandler::class)); - $this->assertNull($result); + $this->assertSame($cleanupThrows ? [$handler->cleanupException] : [], $reported); + if ($cleanupThrows) { + $this->assertInstanceOf(CanceledException::class, $handler->cleanupException->getPrevious()); + } $this->assertFalse(CancelingBackgroundQueueTestHandler::$failed); } finally { CancelingBackgroundQueueTestHandler::$failed = false; @@ -730,6 +739,8 @@ class CancelingBackgroundQueueTestHandler { public static bool $failed = false; + public ?RuntimeException $cleanupException = null; + public function fire(): never { $gate = new Channel(1); @@ -740,9 +751,15 @@ public function fire(): never EngineCoroutine::cancelById($coroutineId, throwException: true); }); - $gate->push(true); + try { + $gate->push(true); - throw new RuntimeException('Cancellation was not delivered.'); + throw new RuntimeException('Cancellation was not delivered.'); + } finally { + if ($this->cleanupException !== null) { + throw $this->cleanupException; + } + } } public function failed(): void diff --git a/tests/Queue/QueueDatabaseQueueIntegrationTest.php b/tests/Queue/QueueDatabaseQueueIntegrationTest.php index 6ef93049f7..e7d56a9e1e 100644 --- a/tests/Queue/QueueDatabaseQueueIntegrationTest.php +++ b/tests/Queue/QueueDatabaseQueueIntegrationTest.php @@ -4,16 +4,25 @@ namespace Hypervel\Tests\Queue; +use Hypervel\Contracts\Debug\ExceptionHandler; use Hypervel\Contracts\Events\Dispatcher; +use Hypervel\Contracts\Queue\Factory; +use Hypervel\Contracts\Queue\Job; use Hypervel\Database\ConnectionInterface; use Hypervel\Foundation\Testing\RefreshDatabase; use Hypervel\Queue\DatabaseQueue; +use Hypervel\Queue\Events\JobFailed; use Hypervel\Queue\Events\JobQueued; use Hypervel\Queue\Events\JobQueueing; +use Hypervel\Queue\MaxAttemptsExceededException; use Hypervel\Queue\Queue; +use Hypervel\Queue\Worker; +use Hypervel\Queue\WorkerOptions; use Hypervel\Support\CarbonImmutable; use Hypervel\Support\Str; use Hypervel\Testbench\TestCase; +use RuntimeException; +use Throwable; class QueueDatabaseQueueIntegrationTest extends TestCase { @@ -109,6 +118,55 @@ public function testPoppedJobsIncrementAttempts(): void $this->assertEquals(1, $poppedJob->attempts(), 'The "attempts" attribute of the Job object was not updated by pop!'); } + public function testReleasedAttemptIsNotFailedAlongsideItsQueuedRetry(): void + { + $handler = new DatabaseQueueReleasedAttemptHandler; + $this->app->instance(DatabaseQueueReleasedAttemptHandler::class, $handler); + $failures = []; + $events = $this->app->make(Dispatcher::class); + $events->listen(JobFailed::class, static function (JobFailed $event) use (&$failures): void { + $failures[] = $event; + }); + $worker = new Worker( + $this->app->make(Factory::class), + $events, + $this->app->make(ExceptionHandler::class), + static fn (): bool => false, + ); + $this->queue->pushRaw(json_encode([ + 'uuid' => (string) Str::uuid(), + 'job' => DatabaseQueueReleasedAttemptHandler::class . '@fire', + 'data' => [], + 'maxTries' => 1, + ], JSON_THROW_ON_ERROR)); + $first = $this->queue->pop(); + + try { + $worker->process('default', $first, new WorkerOptions(maxTries: 1)); + $this->fail('Expected the error thrown after release.'); + } catch (RuntimeException $exception) { + $this->assertSame($handler->exception, $exception); + } + + $this->assertSame([], $handler->failures); + $this->assertSame([], $failures); + $this->assertSame(1, $this->queue->size()); + $retry = $this->queue->pop(); + $this->assertNotSame($first->getJobId(), $retry->getJobId()); + $this->assertSame(2, $retry->attempts()); + + try { + $worker->process('default', $retry, new WorkerOptions(maxTries: 1)); + $this->fail('Expected the retry to exhaust its attempt limit.'); + } catch (MaxAttemptsExceededException $exception) { + $this->assertSame([$exception], $handler->failures); + $this->assertCount(1, $failures); + $this->assertSame($exception, $failures[0]->exception); + } + + $this->assertSame(0, $this->queue->size()); + } + /** * Test that the queue can be cleared. */ @@ -279,3 +337,37 @@ public function testJobPayloadIsAvailableOnEvents(): void $this->assertSame((string) $uuid, $jobQueuedEvent->payload()['uuid']); } } + +class DatabaseQueueReleasedAttemptHandler +{ + public RuntimeException $exception; + + /** @var list */ + public array $failures = []; + + /** + * Prepare an identifiable error for the released attempt. + */ + public function __construct() + { + $this->exception = new RuntimeException('The handler threw after releasing its reservation.'); + } + + /** + * Publish a retry before an outer handler observes the error. + */ + public function fire(Job $job, array $data): never + { + $job->release(); + + throw $this->exception; + } + + /** + * Record actual failure callbacks independently of failure events. + */ + public function failed(array $data, Throwable $exception, string $uuid, Job $job): void + { + $this->failures[] = $exception; + } +} diff --git a/tests/Queue/QueueJobTest.php b/tests/Queue/QueueJobTest.php index 618b4d763c..68697e1e64 100644 --- a/tests/Queue/QueueJobTest.php +++ b/tests/Queue/QueueJobTest.php @@ -9,6 +9,7 @@ use Hypervel\Config\Repository as ConfigRepository; use Hypervel\Contracts\Container\Container; use Hypervel\Contracts\Events\Dispatcher; +use Hypervel\Coroutine\Coroutine; use Hypervel\Database\ConnectionInterface; use Hypervel\Database\ConnectionResolverInterface; use Hypervel\Engine\Channel; @@ -20,6 +21,7 @@ use Hypervel\Tests\TestCase; use Mockery as m; use PHPUnit\Framework\Attributes\DataProvider; +use PHPUnit\Framework\Attributes\TestWith; use RuntimeException; use Swoole\Coroutine\CanceledException; use Throwable; @@ -167,7 +169,9 @@ public function testInvalidPayloadExceptionFromUserFailedHookStillPropagates(): $this->assertTrue($job->isDeleted()); } - public function testBatchRollbackCancellationEscapesBeforeFurtherFailureCleanup(): void + #[TestWith([false])] + #[TestWith([true])] + public function testBatchRollbackCancellationEscapesBeforeFurtherFailureCleanup(bool $cleanupThrows): void { $job = new QueueJobPayloadStub(json_encode([ 'uuid' => 'job-uuid', @@ -176,13 +180,21 @@ public function testBatchRollbackCancellationEscapesBeforeFurtherFailureCleanup( ], JSON_THROW_ON_ERROR)); $timeout = TimeoutExceededException::forJob($job); $gate = $this->armCurrentCoroutineCancellation(); + $cleanupException = new RuntimeException('Rollback cleanup failed.'); $batchRepository = m::mock(BatchRepository::class); $batchRepository->shouldReceive('rollBack') ->once() - ->andReturnUsing(static function () use ($gate): never { - $gate->push(true); - - throw new RuntimeException('Cancellation was not delivered.'); + ->andReturnUsing(static function () use ($gate, $cleanupThrows, $cleanupException): never { + try { + $gate->push(true); + + throw new RuntimeException('Cancellation was not delivered.'); + } finally { + if ($cleanupThrows) { + Coroutine::sleep(0.001); + throw $cleanupException; + } + } }); $container = m::mock(Container::class); $container->shouldReceive('make')->once()->with(BatchRepository::class)->andReturn($batchRepository); @@ -192,13 +204,21 @@ public function testBatchRollbackCancellationEscapesBeforeFurtherFailureCleanup( try { $job->fail($timeout); $this->fail('Expected batch rollback cancellation to escape.'); - } catch (CanceledException) { + } catch (Throwable $exception) { + if ($cleanupThrows) { + $this->assertSame($cleanupException, $exception); + $this->assertInstanceOf(CanceledException::class, $exception->getPrevious()); + } else { + $this->assertInstanceOf(CanceledException::class, $exception); + } $this->assertTrue($job->hasFailed()); $this->assertFalse($job->isDeleted()); } } - public function testFailedHookCancellationSkipsTheFailedEvent(): void + #[TestWith([false])] + #[TestWith([true])] + public function testFailedHookCancellationSkipsTheFailedEvent(bool $cleanupThrows): void { $failure = new RuntimeException('Queue job failed.'); $job = new QueueJobPayloadStub(json_encode([ @@ -206,7 +226,8 @@ public function testFailedHookCancellationSkipsTheFailedEvent(): void 'job' => QueueJobCancelingFailedHookStub::class, 'data' => [], ], JSON_THROW_ON_ERROR)); - $handler = new QueueJobCancelingFailedHookStub($this->armCurrentCoroutineCancellation()); + $cleanupException = $cleanupThrows ? new RuntimeException('Failed-hook cleanup failed.') : null; + $handler = new QueueJobCancelingFailedHookStub($this->armCurrentCoroutineCancellation(), $cleanupException); $container = m::mock(Container::class); $container->shouldReceive('make') ->once() @@ -219,7 +240,13 @@ public function testFailedHookCancellationSkipsTheFailedEvent(): void try { $job->fail($failure); $this->fail('Expected failed hook cancellation to escape.'); - } catch (CanceledException) { + } catch (Throwable $exception) { + if ($cleanupThrows) { + $this->assertSame($cleanupException, $exception); + $this->assertInstanceOf(CanceledException::class, $exception->getPrevious()); + } else { + $this->assertInstanceOf(CanceledException::class, $exception); + } $this->assertTrue($job->hasFailed()); $this->assertTrue($job->isDeleted()); } @@ -366,15 +393,22 @@ public function failed(array $data, ?Throwable $e, string $uuid, Job $job): neve class QueueJobCancelingFailedHookStub { - public function __construct(protected Channel $gate) + public function __construct(protected Channel $gate, protected ?RuntimeException $cleanupException = null) { } public function failed(array $data, ?Throwable $e, string $uuid, Job $job): never { - $this->gate->push(true); - - throw new RuntimeException('Cancellation was not delivered.'); + try { + $this->gate->push(true); + + throw new RuntimeException('Cancellation was not delivered.'); + } finally { + if ($this->cleanupException !== null) { + Coroutine::sleep(0.001); + throw $this->cleanupException; + } + } } } diff --git a/tests/Queue/QueueSqsConnectorTest.php b/tests/Queue/QueueSqsConnectorTest.php index 0179d8a1d1..d797155dd4 100644 --- a/tests/Queue/QueueSqsConnectorTest.php +++ b/tests/Queue/QueueSqsConnectorTest.php @@ -10,8 +10,6 @@ use Aws\Credentials\InstanceProfileProvider; use Aws\Exception\CredentialsException; use Closure; -use DateInterval; -use DateTimeInterface; use GuzzleHttp\Promise\Create; use GuzzleHttp\Promise\PromiseInterface; use GuzzleHttp\Psr7\Response; @@ -30,6 +28,7 @@ use InvalidArgumentException; use Mockery as m; use PHPUnit\Framework\Attributes\DataProvider; +use PHPUnit\Framework\Attributes\TestWith; use Psr\Http\Message\RequestInterface; use ReflectionFunction; use RuntimeException; @@ -263,9 +262,14 @@ public function testCredentialResolutionFallsBackToADirectFetchWhenEveryCacheSto $this->assertSame('direct-key', $provider()->wait()->getAccessKeyId()); } - public function testCancellationWhileRecheckingCachedCredentialsReleasesTheRefreshLock(): void + #[TestWith([false])] + #[TestWith([true])] + public function testCancellationWhileRecheckingCachedCredentialsReleasesTheRefreshLock(bool $cleanupThrows): void { $cancellation = new CanceledException('canceled'); + if ($cleanupThrows) { + $cancellation = new RuntimeException('Read cleanup failed.', previous: $cancellation); + } $repository = new InterruptibleCredentialRepository(new ArrayStore); // The first read misses before locking; the second is the recheck under the lock. $repository->getFailures[2] = $cancellation; @@ -277,29 +281,36 @@ public function testCancellationWhileRecheckingCachedCredentialsReleasesTheRefre ); $this->fail('The cancellation should propagate.'); - } catch (CanceledException $exception) { + } catch (Throwable $exception) { $this->assertSame($cancellation, $exception); } $this->assertRefreshLockIsFree($repository); } - public function testCancellationWhileStoringCredentialsReleasesTheRefreshLock(): void + #[TestWith([false, true])] + #[TestWith([true, true])] + #[TestWith([false, false])] + #[TestWith([true, false])] + public function testCancellationWhileUpdatingTheCredentialCacheReleasesTheRefreshLock(bool $cleanupThrows, bool $cacheable): void { $cancellation = new CanceledException('canceled'); - $repository = new InterruptibleCredentialRepository(new ArrayStore); - $repository->putFailure = $cancellation; + if ($cleanupThrows) { + $cancellation = new RuntimeException('Cache cleanup failed.', previous: $cancellation); + } + $repository = m::mock(Repository::class, [new ArrayStore])->makePartial(); + $repository->shouldReceive($cacheable ? 'put' : 'forget')->once()->andThrow($cancellation); $promise = (new AwsCredentialCache(fn () => $repository))->resolve( 'credentials', - fn () => Create::promiseFor(new Credentials('fresh-key', 'fresh-secret', expires: time() + 3600)), + fn () => Create::promiseFor(new Credentials('fresh-key', 'fresh-secret', expires: $cacheable ? time() + 3600 : null)), ); try { $promise->wait(); $this->fail('The cancellation should propagate.'); - } catch (CanceledException $exception) { + } catch (Throwable $exception) { $this->assertSame($cancellation, $exception); } @@ -339,9 +350,14 @@ public static function failingCredentialProviders(): array ]; } - public function testRejectedCancellationSurvivesACanceledLockRelease(): void + #[TestWith([false])] + #[TestWith([true])] + public function testRejectedCancellationSurvivesACanceledLockRelease(bool $cleanupThrows): void { $cancellation = new CanceledException('canceled'); + if ($cleanupThrows) { + $cancellation = new RuntimeException('Provider cleanup failed.', previous: $cancellation); + } $store = new InterruptibleCredentialStore; $store->releaseFailure = new CanceledException('release canceled'); $repository = new Repository($store); @@ -355,11 +371,64 @@ public function testRejectedCancellationSurvivesACanceledLockRelease(): void $promise->wait(); $this->fail('The cancellation should propagate.'); - } catch (CanceledException $exception) { + } catch (Throwable $exception) { $this->assertSame($cancellation, $exception); } } + #[TestWith([false])] + #[TestWith([true])] + public function testCancellationWhileAcquiringTheRefreshLockDoesNotCallTheProvider(bool $cleanupThrows): void + { + $failure = $cleanupThrows ? new RuntimeException('Lock cleanup failed.', previous: new CanceledException) : new CanceledException; + $lock = m::mock(ArrayLock::class); + $lock->shouldReceive('block')->once()->andThrow($failure); + $lock->shouldReceive('release')->once()->andReturnFalse(); + $store = m::mock(ArrayStore::class)->makePartial(); + $store->shouldReceive('lock')->once()->andReturn($lock); + $repository = new Repository($store); + $providerCalled = false; + $caught = null; + + try { + (new AwsCredentialCache(fn () => $repository))->resolve('credentials', function () use (&$providerCalled): PromiseInterface { + $providerCalled = true; + + return Create::promiseFor(new Credentials('key', 'secret')); + }); + } catch (Throwable $exception) { + $caught = $exception; + } + + $this->assertSame($failure, $caught); + $this->assertFalse($providerCalled); + } + + #[TestWith([false])] + #[TestWith([true])] + public function testLockReleaseCancellationPropagatesAfterCredentialResolution(bool $providerFails): void + { + $failure = new RuntimeException('Lock cleanup failed.', previous: new CanceledException); + $store = new InterruptibleCredentialStore; + $store->releaseFailure = $failure; + $repository = new Repository($store); + $caught = null; + + try { + (new AwsCredentialCache(fn () => $repository))->resolve('credentials', static function () use ($providerFails): PromiseInterface { + if ($providerFails) { + throw new RuntimeException('Provider failed.'); + } + + return Create::promiseFor(new Credentials('key', 'secret', expires: time() + 3600)); + })->wait(); + } catch (Throwable $exception) { + $caught = $exception; + } + + $this->assertSame($failure, $caught); + } + public function testCredentialsCacheKeyIsScopedToThePodIdentityAndConnection(): void { $this->withoutContainerCredentials(function () { @@ -856,11 +925,6 @@ class InterruptibleCredentialRepository extends Repository */ public array $getFailures = []; - /** - * The failure thrown by put(), if any. - */ - public ?Throwable $putFailure = null; - /** * The number of get() calls made. */ @@ -877,18 +941,6 @@ public function get(array|UnitEnum|string $key, mixed $default = null): mixed return parent::get($key, $default); } - - /** - * Store an item in the cache, or throw the queued put failure. - */ - public function put(array|UnitEnum|string $key, mixed $value, DateInterval|DateTimeInterface|int|null $ttl = null): bool - { - if ($this->putFailure) { - throw $this->putFailure; - } - - return parent::put($key, $value, $ttl); - } } class InterruptibleCredentialStore extends ArrayStore diff --git a/tests/Queue/QueueSyncQueueTest.php b/tests/Queue/QueueSyncQueueTest.php index 05fdefb2be..0670a34695 100644 --- a/tests/Queue/QueueSyncQueueTest.php +++ b/tests/Queue/QueueSyncQueueTest.php @@ -16,6 +16,7 @@ use Hypervel\Contracts\Queue\ShouldBeUnique; use Hypervel\Contracts\Queue\ShouldQueue; use Hypervel\Contracts\Queue\ShouldQueueAfterCommit; +use Hypervel\Coroutine\Coroutine; use Hypervel\Database\DatabaseTransactionsManager; use Hypervel\Engine\Channel; use Hypervel\Engine\Coroutine as EngineCoroutine; @@ -33,8 +34,10 @@ use Hypervel\Tests\TestCase; use LogicException; use Mockery as m; +use PHPUnit\Framework\Attributes\TestWith; use RuntimeException; use Swoole\Coroutine\CanceledException; +use Throwable; class QueueSyncQueueTest extends TestCase { @@ -264,11 +267,15 @@ public function testAttemptedEventRunsAfterThrownSyncJob(): void } } - public function testCancellationEscapesWithoutFailedOrCompletionEvents(): void + #[TestWith([false])] + #[TestWith([true])] + public function testCancellationEscapesWithoutFailedOrCompletionEvents(bool $cleanupThrows): void { CancelingSyncQueueTestHandler::reset(); $gate = $this->armCurrentCoroutineCancellation(); CancelingSyncQueueTestHandler::$gate = $gate; + $cleanupException = $cleanupThrows ? new RuntimeException('Sync job cleanup failed.') : null; + CancelingSyncQueueTestHandler::$cleanupException = $cleanupException; $dispatched = []; $sync = new SyncQueue; $sync->setConnectionName('sync'); @@ -288,7 +295,13 @@ public function testCancellationEscapesWithoutFailedOrCompletionEvents(): void try { $sync->push(CancelingSyncQueueTestHandler::class); $this->fail('Expected cancellation to escape the sync queue.'); - } catch (CanceledException) { + } catch (Throwable $exception) { + if ($cleanupThrows) { + $this->assertSame($cleanupException, $exception); + $this->assertInstanceOf(CanceledException::class, $exception->getPrevious()); + } else { + $this->assertInstanceOf(CanceledException::class, $exception); + } $this->assertSame([JobProcessing::class], $dispatched); $this->assertFalse(CancelingSyncQueueTestHandler::$failed); } finally { @@ -296,6 +309,42 @@ public function testCancellationEscapesWithoutFailedOrCompletionEvents(): void } } + public function testWrappedCancellationDuringFailureObservationSkipsFailureAndCompletion(): void + { + $cleanupException = new RuntimeException('Failure observer cleanup failed.'); + $gate = $this->armCurrentCoroutineCancellation(); + $events = m::mock(EventDispatcher::class); + $events->shouldReceive('hasListeners')->with(JobProcessing::class)->andReturnFalse(); + $events->shouldReceive('hasListeners')->with(JobExceptionOccurred::class)->andReturnTrue(); + $events->shouldNotReceive('hasListeners')->with(JobAttempted::class); + $events->expects('dispatch')->with(m::type(JobExceptionOccurred::class))->andReturnUsing(static function () use ($gate, $cleanupException): never { + try { + $gate->push(true); + throw new RuntimeException('Cancellation was not delivered.'); + } finally { + Coroutine::sleep(0.001); + throw $cleanupException; + } + }); + $container = $this->getContainer(); + $container->instance('events', $events); + $job = m::mock(SyncJob::class); + $job->expects('fire')->andThrow(new RuntimeException('Job failed.')); + $job->shouldNotReceive('fail'); + $sync = m::mock(SyncQueue::class)->makePartial()->shouldAllowMockingProtectedMethods(); + $sync->shouldReceive('resolveJob')->andReturn($job); + $sync->setConnectionName('sync'); + $sync->setContainer($container); + + try { + $sync->pushRaw('{}'); + $this->fail('Expected the cleanup error to escape.'); + } catch (RuntimeException $exception) { + $this->assertSame($cleanupException, $exception); + $this->assertInstanceOf(CanceledException::class, $exception->getPrevious()); + } + } + public function testCancellationDuringSerializationIsNotWrapped(): void { $sync = new SyncQueue; @@ -469,8 +518,9 @@ private function armCurrentCoroutineCancellation(): Channel $coroutineId = EngineCoroutine::id(); EngineCoroutine::create(static function () use ($coroutineId, $gate): void { - $gate->pop(); - EngineCoroutine::cancelById($coroutineId, throwException: true); + if ($gate->pop(5)) { + EngineCoroutine::cancelById($coroutineId, throwException: true); + } }); return $gate; @@ -552,6 +602,8 @@ class CancelingSyncQueueTestHandler { public static ?Channel $gate = null; + public static ?RuntimeException $cleanupException = null; + public static bool $failed = false; /** @@ -559,9 +611,16 @@ class CancelingSyncQueueTestHandler */ public function fire(): never { - static::$gate?->push(true); + try { + static::$gate?->push(true); - throw new RuntimeException('Cancellation was not delivered.'); + throw new RuntimeException('Cancellation was not delivered.'); + } finally { + if (static::$cleanupException !== null) { + Coroutine::sleep(0.001); + throw static::$cleanupException; + } + } } /** @@ -578,6 +637,7 @@ public function failed(): void public static function reset(): void { static::$gate = null; + static::$cleanupException = null; static::$failed = false; } } diff --git a/tests/Queue/QueueWorkerTest.php b/tests/Queue/QueueWorkerTest.php index a79924e8a2..042f6d5ad5 100644 --- a/tests/Queue/QueueWorkerTest.php +++ b/tests/Queue/QueueWorkerTest.php @@ -13,6 +13,7 @@ use Hypervel\Cache\Events\ForgettingKey; use Hypervel\Cache\Repository; use Hypervel\Cache\WorkerArrayStore; +use Hypervel\Config\Repository as ConfigRepository; use Hypervel\Container\Container; use Hypervel\Context\CoroutineContext; use Hypervel\Contracts\Cache\Repository as CacheContract; @@ -28,6 +29,7 @@ use Hypervel\Coordinator\Constants; use Hypervel\Coordinator\Timer; use Hypervel\Coroutine\Coroutine; +use Hypervel\Coroutine\Exceptions\ChildCancellationException; use Hypervel\Coroutine\WaitConcurrent; use Hypervel\Coroutine\Waiter; use Hypervel\Database\LostConnectionException; @@ -35,6 +37,7 @@ use Hypervel\Engine\Channel; use Hypervel\Engine\Coroutine as EngineCoroutine; use Hypervel\Events\Dispatcher as EventsDispatcher; +use Hypervel\Filesystem\Filesystem; use Hypervel\Foundation\Exceptions\Handler; use Hypervel\Http\Request; use Hypervel\Queue\CallQueuedHandler; @@ -58,17 +61,24 @@ use Hypervel\Queue\Events\WorkerResuming; use Hypervel\Queue\Events\WorkerStarting; use Hypervel\Queue\Events\WorkerStopping; +use Hypervel\Queue\InteractsWithQueue; use Hypervel\Queue\InvalidPayloadException; use Hypervel\Queue\Jobs\SyncJob; use Hypervel\Queue\MaxAttemptsExceededException; +use Hypervel\Queue\Middleware\ThrottlesExceptions; use Hypervel\Queue\QueueManager; use Hypervel\Queue\TimeoutExceededException; use Hypervel\Queue\Worker; use Hypervel\Queue\WorkerOptions; use Hypervel\Queue\WorkerStopReason; +use Hypervel\RateLimiter\Limit; +use Hypervel\RateLimiter\Limiter; +use Hypervel\RateLimiter\LimitResult; +use Hypervel\RateLimiter\RateLimiter; use Hypervel\Support\CarbonImmutable; use Hypervel\Support\Testing\Fakes\EventFake; use Hypervel\Support\Testing\Fakes\ExceptionHandlerFake; +use Hypervel\Testing\ParallelTesting; use Hypervel\Tests\TestCase; use LogicException; use Mockery as m; @@ -95,6 +105,13 @@ class QueueWorkerTest extends TestCase protected ContainerContract $container; + /** @var list */ + protected array $registeredJobs = []; + + private ?string $errorLogDirectory = null; + + private string|false $previousErrorLog = false; + protected function setUp(): void { parent::setUp(); @@ -109,6 +126,27 @@ protected function setUp(): void $this->container->instance(ExceptionHandlerContract::class, $this->exceptionHandler); } + /** + * End synthetic running-job coroutines even when an assertion fails. + */ + protected function tearDownInCoroutine(): void + { + try { + foreach ($this->registeredJobs as [$gate, $coroutineId]) { + $gate->close(); + Coroutine::join([$coroutineId], 1); + } + } finally { + if ($this->errorLogDirectory !== null) { + if ($this->previousErrorLog !== false) { + ini_set('error_log', $this->previousErrorLog); + } + + (new Filesystem)->deleteDirectory($this->errorLogDirectory); + } + } + } + public function testJobCanBeFired(): void { $worker = $this->getWorker('default', ['queue' => [$job = new WorkerFakeJob]]); @@ -207,34 +245,49 @@ function (object $event) use (&$order, &$attemptedEvent): void { $this->assertFalse($attemptedEvent->successful()); } - #[TestWith([true])] - #[TestWith([false])] - public function testCancellationDuringJobExecutionEscapesWithoutFailureOrCompletion(bool $killOnTimeout): void + #[TestWith([true, false])] + #[TestWith([false, false])] + #[TestWith([false, true])] + public function testCancellationDuringJobExecutionEscapesWithoutFailureOrCompletion(bool $killOnTimeout, bool $cleanupThrows): void { Worker::$killOnTimeout = $killOnTimeout; Worker::$pausable = false; $gate = $this->armCurrentCoroutineCancellation(); - $job = new WorkerFakeJob(static function () use ($gate): never { - $gate->push(true); + $cleanupException = new RuntimeException('Job cleanup failed.'); + $job = new WorkerFakeJob(static function () use ($gate, $cleanupThrows, $cleanupException): never { + try { + $gate->push(true); - throw new RuntimeException('Cancellation was not delivered.'); + throw new RuntimeException('Cancellation was not delivered.'); + } finally { + if ($cleanupThrows) { + throw $cleanupException; + } + } }); $job->payload = ['countCrashesAsExceptions' => true]; $job->maxExceptions = 2; $worker = $this->getWorker('default', ['queue' => [$job]]); $worker->setCache($cache = new Repository(new WorkerArrayStore)); + $caught = null; try { $worker->runNextJob('default', 'queue', new WorkerOptions); - $this->fail('Expected job cancellation to escape the worker.'); - } catch (CanceledException) { - $this->assertTrue($job->fired); - $this->assertFalse($job->isDeleted()); - $this->assertFalse($job->isReleased()); - $this->assertFalse($job->hasFailed()); - $this->assertSame([], $worker->runningJobsForTest()); + } catch (Throwable $exception) { + $caught = $exception; } + if ($cleanupThrows) { + $this->assertSame($cleanupException, $caught); + $this->assertInstanceOf(CanceledException::class, $caught->getPrevious()); + } else { + $this->assertInstanceOf(CanceledException::class, $caught); + } + $this->assertTrue($job->fired); + $this->assertFalse($job->isDeleted()); + $this->assertFalse($job->isReleased()); + $this->assertFalse($job->hasFailed()); + $this->assertSame([], $worker->runningJobsForTest()); $this->exceptionHandler->assertNothingReported(); $this->events->assertNotDispatched(JobExceptionOccurred::class); $this->events->assertNotDispatched(JobProcessed::class); @@ -242,33 +295,50 @@ public function testCancellationDuringJobExecutionEscapesWithoutFailureOrComplet $this->assertTrue($cache->get('job-processing:' . $job->uuid)); } - public function testCancellationDuringFailureObservationStopsReleaseAndCompletion(): void + #[TestWith([false])] + #[TestWith([true])] + public function testCancellationDuringFailureObservationStopsReleaseAndCompletion(bool $cleanupThrows): void { $this->spyEvents(); $gate = $this->armCurrentCoroutineCancellation(); + $cleanupException = new RuntimeException('Observer cleanup failed.'); $this->events->shouldReceive('dispatch') ->with(m::type(JobExceptionOccurred::class)) ->once() - ->andReturnUsing(static function () use ($gate): never { - $gate->push(true); + ->andReturnUsing(static function () use ($gate, $cleanupThrows, $cleanupException): never { + try { + $gate->push(true); - throw new RuntimeException('Cancellation was not delivered.'); + throw new RuntimeException('Cancellation was not delivered.'); + } finally { + if ($cleanupThrows) { + Coroutine::sleep(0.001); + throw $cleanupException; + } + } }); $job = new WorkerFakeJob(static function (): never { throw new RuntimeException('Job failed.'); }); $worker = $this->getWorker('default', ['queue' => [$job]]); + $caught = null; try { $worker->runNextJob('default', 'queue', new WorkerOptions(maxTries: 0)); - $this->fail('Expected observer cancellation to escape the worker.'); - } catch (CanceledException) { - $this->assertFalse($job->isDeleted()); - $this->assertFalse($job->isReleased()); - $this->assertFalse($job->hasFailed()); - $this->assertSame([], $worker->runningJobsForTest()); + } catch (Throwable $exception) { + $caught = $exception; } + $this->assertFalse($job->isDeleted()); + $this->assertFalse($job->isReleased()); + $this->assertFalse($job->hasFailed()); + $this->assertSame([], $worker->runningJobsForTest()); + if ($cleanupThrows) { + $this->assertSame($cleanupException, $caught); + $this->assertInstanceOf(CanceledException::class, $cleanupException->getPrevious()); + } else { + $this->assertInstanceOf(CanceledException::class, $caught); + } $this->exceptionHandler->assertNothingReported(); $this->events->shouldNotHaveReceived('dispatch', [m::type(JobReleasedAfterException::class)]); $this->events->shouldNotHaveReceived('dispatch', [m::type(JobAttempted::class)]); @@ -534,7 +604,7 @@ public function testTimeoutDeadlinesUseTheMonotonicWorkerClock(): void $worker->currentTime = 100; $options = new WorkerOptions(timeout: 5); - $jobId = $worker->registerCoroutineJobForTest(new WorkerContractOnlyJob, $options); + $jobId = $this->registerRunningJob($worker, new WorkerContractOnlyJob, $options); $this->assertSame(105.0, $worker->runningJobExpiryForTest($jobId)); @@ -557,7 +627,7 @@ public function testTimeoutMonitorUsesTheDefaultErrorExitAndTimedOutReason(?int $options = new WorkerOptions(timeout: 5); $job = m::mock(WorkerContractOnlyJob::class)->makePartial(); $job->shouldReceive('timeout')->andReturn($jobTimeout); - $worker->registerCoroutineJobForTest($job, $options); + $this->registerRunningJob($worker, $job, $options); $worker->startMonitorForTest($options); $worker->currentTime = 100 + $expectedTimeout; @@ -584,9 +654,10 @@ public function testTimeoutMonitorUsesTheDefaultErrorExitAndTimedOutReason(?int ); } - #[TestWith([false])] - #[TestWith([true])] - public function testNonKillingTimeoutUnwindsAndRetriesOnlyTheExpiredJob(bool $listenerThrows): void + #[TestWith([false, false])] + #[TestWith([true, false])] + #[TestWith([false, true])] + public function testNonKillingTimeoutUnwindsAndRetriesOnlyTheExpiredJob(bool $listenerThrows, bool $catchCancellation): void { $this->spyEvents(); Worker::$killOnTimeout = false; @@ -596,13 +667,15 @@ public function testNonKillingTimeoutUnwindsAndRetriesOnlyTheExpiredJob(bool $li $otherGate = new Channel(1); $unwound = false; $cancellation = null; - $expired = m::mock(WorkerFakeJob::class, [static function () use ($expiredGate, &$unwound, &$cancellation): void { + $expired = m::mock(WorkerFakeJob::class, [static function () use ($expiredGate, &$unwound, &$cancellation, $catchCancellation): void { try { $expiredGate->pop(); } catch (CanceledException $exception) { $cancellation = $exception; - throw $exception; + if (! $catchCancellation) { + throw $exception; + } } finally { $unwound = true; } @@ -639,7 +712,7 @@ public function testNonKillingTimeoutUnwindsAndRetriesOnlyTheExpiredJob(bool $li } Coroutine::join([$coroutines[0]], 5); - $this->assertSame($listenerThrows ? $listenerException : null, $monitorException); + $this->assertNull($monitorException); $this->assertTrue($unwound); $this->assertInstanceOf(CanceledException::class, $cancellation); $this->assertTrue($expired->released); @@ -663,8 +736,11 @@ public function testNonKillingTimeoutUnwindsAndRetriesOnlyTheExpiredJob(bool $li static fn (object $event): bool => $event instanceof JobProcessed && $event->job === $other ))->once(); $this->events->shouldNotHaveReceived('dispatch', [m::type(WorkerStopping::class)]); - $this->exceptionHandler->assertReportedCount(1); + $this->exceptionHandler->assertReportedCount($listenerThrows ? 2 : 1); $this->exceptionHandler->assertReported(TimeoutExceededException::class); + if ($listenerThrows) { + $this->exceptionHandler->assertReported(static fn (RuntimeException $exception): bool => $exception === $listenerException); + } } finally { $expiredGate->close(); $otherGate->close(); @@ -672,6 +748,199 @@ public function testNonKillingTimeoutUnwindsAndRetriesOnlyTheExpiredJob(bool $li } } + #[TestWith([false, 'normal'])] + #[TestWith([true, 'normal'])] + #[TestWith([true, 'original'])] + #[TestWith([true, 'ordinary'])] + #[TestWith([true, 'cancellation'])] + public function testNonKillingTimeoutHandlesCleanupAndReportingFailures(bool $reportExceptions, string $reportingMode): void + { + Worker::$killOnTimeout = false; + Worker::$reportJobExceptions = $reportExceptions; + Worker::$pausable = false; + $timer = new QueueWorkerTimer; + $gate = new Channel(1); + $cleanupException = new RuntimeException('Job cleanup failed.'); + $reportingFailure = new RuntimeException('Reporting failed.'); + $caught = null; + $errorLog = null; + if ($reportingMode !== 'normal') { + $errorLog = $this->captureErrorLog(); + $this->exceptionHandler = m::mock(ExceptionHandlerContract::class); + $report = $this->exceptionHandler->expects('report')->with($cleanupException); + if ($reportingMode === 'cancellation') { + $report->andReturnUsing(function () use ($reportingFailure): never { + try { + $this->armCurrentCoroutineCancellation()->push(true); + + throw new RuntimeException('Cancellation was not delivered.'); + } finally { + throw $reportingFailure; + } + }); + } else { + $report->andThrow($reportingMode === 'original' ? $cleanupException : $reportingFailure); + $this->exceptionHandler->expects('report')->with(m::type(TimeoutExceededException::class)); + } + } + $command = new class { + use InteractsWithQueue; + }; + $limiter = m::mock(Limiter::class); + $limiter->expects('inspect')->with(m::type(Limit::class))->andReturn(new LimitResult(true, 10, 10, 0, 0)); + $limiter->shouldNotReceive('consume', 'clear'); + $rateLimiter = m::mock(RateLimiter::class); + $rateLimiter->expects('store')->with(null)->andReturn($limiter); + $this->container->instance(RateLimiter::class, $rateLimiter); + $job = m::mock(WorkerFakeJob::class, [static function () use ($command, $gate, $cleanupException): void { + (new ThrottlesExceptions)->handle($command, static function () use ($gate, $cleanupException): void { + try { + $gate->pop(5); + } finally { + Coroutine::sleep(0.001); + throw $cleanupException; + } + }); + }])->makePartial(); + $command->setJob($job); + $job->shouldReceive('timeout')->andReturn(5); + $job->shouldFailOnTimeout = true; + $job->maxExceptions = 3; + $worker = new KillTestWorker(...$this->workerDependencies('default', ['queue' => [$job]], timer: $timer)); + $worker->setCache(new Repository(new WorkerArrayStore)); + $worker->currentTime = 100; + $options = new WorkerOptions; + $worker->startMonitorForTest($options); + $coroutineId = Coroutine::create(static function () use ($worker, $options, &$caught): void { + try { + $worker->runNextJob('default', 'queue', $options); + } catch (Throwable $exception) { + $caught = $exception; + } + }); + + try { + $worker->currentTime = 105; + $timer->fire(1); + Coroutine::join([$coroutineId], 5); + + $this->assertFalse(Coroutine::exists($coroutineId)); + if ($reportingMode === 'cancellation') { + $this->assertSame($reportingFailure, $caught); + $this->assertInstanceOf(CanceledException::class, $caught->getPrevious()); + $this->assertFalse($job->failed); + $this->assertFalse($job->released); + $this->events->assertNotDispatched(JobAttempted::class); + $this->assertFileDoesNotExist($errorLog); + + return; + } + + $this->assertNull($caught); + $this->assertTrue($job->failed); + $this->assertFalse($job->released); + $this->assertInstanceOf(TimeoutExceededException::class, $job->failedWith); + $this->assertInstanceOf(CanceledException::class, $cleanupException->getPrevious()); + $this->events->assertDispatchedOnce(JobTimedOut::class); + $this->events->assertDispatchedOnce(JobAttempted::class); + $this->events->assertDispatched(JobAttempted::class, static fn (JobAttempted $event): bool => ! $event->successful()); + $this->events->assertNotDispatched(JobProcessed::class); + $this->assertFalse($worker->hasTimeoutJobsForTest()); + $this->assertSame([], $worker->runningJobsForTest()); + if ($reportingMode === 'normal') { + $this->exceptionHandler->assertReportedCount($reportExceptions ? 2 : 0); + if ($reportExceptions) { + $this->exceptionHandler->assertReported(static fn (RuntimeException $exception): bool => $exception === $cleanupException); + $this->exceptionHandler->assertReported(TimeoutExceededException::class); + } + } else { + $contents = file_get_contents($errorLog); + $this->assertSame(1, substr_count($contents, 'Job cleanup failed.')); + if ($reportingMode === 'ordinary') { + $this->assertStringContainsString('Reporting failed.', $contents); + } + } + } finally { + $gate->close(); + Coroutine::join([$coroutineId], 5); + } + } + + #[TestWith([false, false])] + #[TestWith([true, false])] + #[TestWith([true, true])] + public function testNonKillingTimeoutDoesNotInterruptPostExecutionListeners(bool $throws, bool $failOnTimeout): void + { + Worker::$killOnTimeout = false; + Worker::$pausable = false; + $events = new EventsDispatcher($this->container); + $this->events = $events; + $this->container->instance(EventDispatcher::class, $events); + $listenerStarted = new Channel(1); + $listenerGate = new Channel(1); + $listenerFinished = false; + $attempted = []; + $timedOut = []; + $failure = new RuntimeException('The job failed before its deadline.'); + $events->listen($throws ? JobExceptionOccurred::class : JobProcessed::class, static function () use ($listenerStarted, $listenerGate, &$listenerFinished): void { + $listenerStarted->push(true); + $listenerGate->pop(5); + $listenerFinished = true; + }); + $events->listen(JobAttempted::class, static function (JobAttempted $event) use (&$attempted): void { + $attempted[] = $event; + }); + $events->listen(JobTimedOut::class, static function (JobTimedOut $event) use (&$timedOut): void { + $timedOut[] = $event; + }); + $job = m::mock(WorkerFakeJob::class, [static function () use ($throws, $failure): void { + if ($throws) { + throw $failure; + } + }])->makePartial(); + $job->shouldReceive('timeout')->andReturn(5); + $job->shouldFailOnTimeout = $failOnTimeout; + $job->attempts = 1; + $job->maxExceptions = 3; + $job->payload = ['countCrashesAsExceptions' => true]; + $timer = new QueueWorkerTimer; + $worker = new KillTestWorker(...$this->workerDependencies('default', ['queue' => [$job]], timer: $timer)); + $worker->setCache($cache = new Repository(new WorkerArrayStore)); + $worker->currentTime = 100; + $options = new WorkerOptions(maxTries: 3, backoff: 7); + $worker->startMonitorForTest($options); + $coroutineId = Coroutine::create(static fn () => $worker->runNextJob('default', 'queue', $options)); + + try { + $this->assertTrue($listenerStarted->pop(1)); + $worker->currentTime = 106; + $timer->fire(1); + $this->assertTrue(Coroutine::exists($coroutineId)); + $this->assertFalse($listenerFinished); + $this->assertSame([], $timedOut); + $listenerGate->push(true); + Coroutine::join([$coroutineId], 5); + + $this->assertTrue($listenerFinished); + $this->assertFalse($job->failed); + $this->assertSame($throws, $job->released); + $this->assertSame($throws ? 7 : null, $job->releaseAfter); + $this->assertSame($throws ? 1 : null, $cache->get('job-exceptions:' . $job->uuid)); + $this->assertNull($cache->get('job-processing:' . $job->uuid)); + $this->assertCount(1, $attempted); + $this->assertSame($throws ? $failure : null, $attempted[0]->exception); + $this->assertSame(! $throws, $attempted[0]->successful()); + $this->assertSame([], $timedOut); + $this->exceptionHandler->assertReportedCount($throws ? 1 : 0); + if ($throws) { + $this->exceptionHandler->assertReported(static fn (RuntimeException $exception): bool => $exception === $failure); + } + } finally { + $listenerGate->close(); + Coroutine::join([$coroutineId], 5); + } + } + public function testTimeoutScanDoesNotCancelCompletedCleanupOrTimeoutAnotherFinishedJob(): void { $this->spyEvents(); @@ -682,13 +951,14 @@ public function testTimeoutScanDoesNotCancelCompletedCleanupOrTimeoutAnotherFini $cleanupGate = new Channel(1); $cleanupStarted = false; $cleanupFinished = false; - $first = m::mock(WorkerFakeJob::class, [static function () use ($firstGate, $cleanupGate, &$cleanupStarted, &$cleanupFinished): void { + $first = m::mock(WorkerFakeJob::class, [static function (WorkerFakeJob $job) use ($firstGate, $cleanupGate, &$cleanupStarted, &$cleanupFinished): void { Coroutine::defer(static function () use ($cleanupGate, &$cleanupStarted, &$cleanupFinished): void { $cleanupStarted = true; $cleanupGate->pop(); $cleanupFinished = true; }); $firstGate->pop(); + $job->delete(); }])->makePartial(); $first->shouldReceive('timeout')->andReturn(5); $second = m::mock(WorkerFakeJob::class, [static fn (): mixed => $secondGate->pop()])->makePartial(); @@ -697,12 +967,16 @@ public function testTimeoutScanDoesNotCancelCompletedCleanupOrTimeoutAnotherFini $worker->currentTime = 100; $options = new WorkerOptions; $worker->startMonitorForTest($options); - $this->events->shouldReceive('dispatch')->with(m::type(JobTimedOut::class))->once()->andReturnUsing( + $interruptible = m::mock(Interruptible::class); + $interruptible->expects('interrupted')->with(SIGALRM)->andReturnUsing( static function () use ($firstGate, $secondGate): void { $firstGate->push(true); $secondGate->push(true); } ); + $handler = m::mock(CallQueuedHandler::class); + $handler->expects('getRunningCommand')->andReturn($interruptible); + $first->resolvedJob = $handler; $coroutines = []; try { @@ -757,7 +1031,7 @@ public function testTimeoutMonitorRunsInAnOwnedCoroutineWithWorkerContextAndStop $worker->currentTime = 100; $options = new WorkerOptions(timeout: 5, monitorInterval: 0); $options->coroutineContext = ['queue.worker.test' => 'configured']; - $worker->registerCoroutineJobForTest(new WorkerContractOnlyJob, $options); + $this->registerRunningJob($worker, new WorkerContractOnlyJob, $options); $worker->currentTime = 105; $this->events->shouldReceive('dispatch')->andReturnUsing( @@ -799,7 +1073,7 @@ public function testTimeoutLifecycleEventsAreNotDispatchedWithoutListeners(): vo $worker = new KillTestWorker(...$this->workerDependencies(timer: $timer)); $worker->currentTime = 100; $options = new WorkerOptions(timeout: 5); - $worker->registerCoroutineJobForTest(new WorkerContractOnlyJob, $options); + $this->registerRunningJob($worker, new WorkerContractOnlyJob, $options); $worker->startMonitorForTest($options); $worker->currentTime = 105; @@ -815,13 +1089,109 @@ public function testTimeoutLifecycleEventsAreNotDispatchedWithoutListeners(): vo $this->events->shouldNotHaveReceived('dispatch'); } + #[TestWith(['failed callback', false])] + #[TestWith(['marker clear', false])] + #[TestWith(['failed callback', true])] + public function testTimeoutHandlingErrorsDoNotAbandonOtherAttempts(string $failurePhase, bool $reportingThrows): void + { + $this->container->instance('config', new ConfigRepository(['queue' => ['failed' => ['driver' => 'null']]])); + $this->events = new EventsDispatcher($this->container); + $this->container->instance(EventDispatcher::class, $this->events); + $failedJobs = []; + $timedOutJobs = []; + $this->events->listen(JobFailed::class, static function (JobFailed $event) use (&$failedJobs): void { + $failedJobs[] = $event->job; + }); + $this->events->listen(JobTimedOut::class, static function (JobTimedOut $event) use (&$timedOutJobs): void { + $timedOutJobs[] = $event->job; + }); + $error = new RuntimeException('Timeout handling failed.'); + $errorLog = null; + if ($reportingThrows) { + $errorLog = $this->captureErrorLog(); + $this->exceptionHandler = m::mock(ExceptionHandlerContract::class); + $this->exceptionHandler->expects('report')->with($error)->andThrow(new RuntimeException('Reporting failed.')); + } + $timer = new QueueWorkerTimer; + $worker = new KillTestWorker(...$this->workerDependencies(timer: $timer)); + $worker->currentTime = 100; + $cache = new Repository(new WorkerArrayStore); + $worker->setCache($cache); + if ($failurePhase === 'marker clear') { + $cache->setEventDispatcher($cacheEvents = new EventsDispatcher($this->container)); + $cacheEvents->listen(ForgettingKey::class, static function (ForgettingKey $event) use ($error): void { + if ($event->key === 'job-processing:first') { + throw $error; + } + }); + } + $options = new WorkerOptions(maxTries: 0); + $jobs = []; + $coroutines = []; + $gates = []; + $unexpected = []; + + try { + foreach (['first', 'second'] as $name) { + $gates[] = $gate = new Channel(1); + $job = m::mock(WorkerLifecycleJob::class, [ + $this->container, static fn (): mixed => $gate->pop(5), 5, + ])->makePartial(); + $job->shouldReceive('uuid')->andReturn($name); + $job->shouldReceive('getResolvedJob')->andReturnNull(); + $job->shouldReceive('shouldFailOnTimeout')->andReturn($name === 'second' || $failurePhase === 'failed callback'); + if ($name === 'first' && $failurePhase === 'failed callback') { + $job->shouldAllowMockingProtectedMethods()->expects('failed')->andThrow($error); + } + $jobs[] = $job; + $coroutines[] = Coroutine::create(static function () use ($worker, $job, $options, &$unexpected): void { + try { + $worker->process('default', $job, $options); + } catch (CanceledException) { + // The monitor owns these attempts after their bodies finish. + } catch (Throwable $exception) { + $unexpected[] = $exception; + } + }); + } + $worker->startMonitorForTest($options); + $worker->currentTime = 105; + + try { + $timer->fire(1); + $this->fail('Expected termination on this monitor tick.'); + } catch (WorkerKilledException $exception) { + $this->assertSame(Worker::EXIT_ERROR, $exception->status); + } + + $this->assertSame([], $unexpected); + $this->assertSame($failurePhase === 'failed callback' ? $jobs : [$jobs[1]], $failedJobs); + $this->assertSame($jobs, $timedOutJobs); + $this->assertSame($failurePhase === 'marker clear', $cache->has('job-processing:first')); + $this->assertFalse($cache->has('job-processing:second')); + if ($reportingThrows) { + $contents = file_get_contents($errorLog); + $this->assertStringContainsString('Timeout handling failed.', $contents); + $this->assertStringContainsString('Reporting failed.', $contents); + } else { + $this->exceptionHandler->assertReportedCount(1); + $this->exceptionHandler->assertReported(static fn (RuntimeException $reported): bool => $reported === $error); + } + } finally { + foreach ($gates as $gate) { + $gate->close(); + } + Coroutine::join($coroutines, 1); + } + } + public function testTimeoutMonitorUsesTheConfiguredExitCode(): void { $timer = new QueueWorkerTimer; $worker = new KillTestWorker(...$this->workerDependencies(timer: $timer)); $worker->currentTime = 100; $options = new WorkerOptions(timeout: 5); - $worker->registerCoroutineJobForTest(new WorkerContractOnlyJob, $options); + $this->registerRunningJob($worker, new WorkerContractOnlyJob, $options); $worker->startMonitorForTest($options); $worker->currentTime = 105; @@ -892,16 +1262,24 @@ public function testDaemonFailureCancelsAdmittedJobsWithoutFalseOutcomes(): void $this->events->assertNotDispatched(JobAttempted::class); } - public function testCancellationDuringGracefulShutdownCancelsRunningJobs(): void + #[TestWith([false, true])] + #[TestWith([true, true])] + #[TestWith([true, false])] + public function testCancellationDuringGracefulShutdownCancelsRunningJobs(bool $cleanupThrows, bool $reportExceptions): void { + Worker::$reportJobExceptions = $reportExceptions; $cancelOwner = $this->armCurrentCoroutineCancellation(); $blocker = new Channel(1); $jobUnwound = false; - $job = new WorkerFakeJob(static function () use ($blocker, &$jobUnwound): void { + $cleanupException = new RuntimeException('Job cleanup failed.'); + $job = new WorkerFakeJob(static function () use ($blocker, &$jobUnwound, $cleanupThrows, $cleanupException): void { try { $blocker->pop(); } finally { $jobUnwound = true; + if ($cleanupThrows) { + throw $cleanupException; + } } }); $worker = new ShutdownCancellationWorker( @@ -919,10 +1297,14 @@ public function testCancellationDuringGracefulShutdownCancelsRunningJobs(): void $blocker->close(); } + $this->exceptionHandler->assertReportedCount($cleanupThrows && $reportExceptions ? 1 : 0); + if ($cleanupThrows && $reportExceptions) { + $this->exceptionHandler->assertReported(static fn (RuntimeException $exception): bool => $exception === $cleanupException); + } $this->events->assertNotDispatched(WorkerStopping::class); } - public function testTimeoutMonitorUnlocksWhenScanningThrows(): void + public function testTimeoutMonitorScansAgainAfterAnEarlierScanThrows(): void { $timer = new QueueWorkerTimer; $worker = new MonitorFailureWorker( @@ -937,10 +1319,8 @@ public function testTimeoutMonitorUnlocksWhenScanningThrows(): void $this->assertSame('timeout scan failed', $exception->getMessage()); } - $this->assertFalse($worker->monitorIsLockedForTest()); $timer->fire(1); $this->assertSame(2, $worker->scanCount); - $this->assertFalse($worker->monitorIsLockedForTest()); } public function testKillDoesNotWaitForUnrelatedActiveJobs(): void @@ -1072,20 +1452,60 @@ public function testWorkerWaitsForConcurrencySlotWithoutPollingDelay(): void $this->assertNull($worker->sleptFor); } - public function testTimeoutJobsUseMinimumWaitWhenPollingIsDisabled(): void + public function testTimeoutWaitsWithoutPollingAndWinsOverNormalShutdown(): void { - $worker = $this->getWorker('default', ['queue' => []]); - $worker->currentTime = 0; - $options = new WorkerOptions(timeout: 1, sleep: 0, maxTime: 1); + $timer = new QueueWorkerTimer; + $gate = new Channel(1); + $started = new Channel(1); + $job = m::mock(WorkerFakeJob::class, [static function () use ($gate, $started): void { + $started->push(true); + $gate->pop(5); + }])->makePartial(); + $job->shouldReceive('timeout')->andReturn(1); + $worker = new class(...$this->workerDependencies('default', ['queue' => [$job]], timer: $timer)) extends KillTestWorker { + /** + * Keep real yielding waits while recording the pending-timeout delay. + */ + public function sleep(float|int $seconds): void + { + $this->sleptFor = $seconds; + Coroutine::sleep($seconds); + } + }; + $worker->currentTime = 100; + $options = new WorkerOptions(timeout: 1, sleep: 0, maxTries: 1, maxJobs: 1); + $job->attempts = 1; + $daemonFinished = false; + $daemon = Coroutine::create(static function () use ($worker, $options, &$daemonFinished): void { + try { + $worker->daemon('default', 'queue', $options); + $daemonFinished = true; + } catch (CanceledException) { + // A real kill ends the process; this test cleans up its fake daemon. + } + }); - $worker->registerCoroutineJobForTest(new WorkerContractOnlyJob, $options); - $worker->currentTime = 1; - $worker->terminateTimeoutJobsForTest($options); + try { + $this->assertTrue($started->pop(1)); + $worker->currentTime = 101; - $status = $worker->daemon('default', 'queue', $options); + try { + $timer->fire(1); + $this->fail('Expected timeout termination.'); + } catch (WorkerKilledException $exception) { + $this->assertSame(Worker::EXIT_ERROR, $exception->status); + } - $this->assertSame(Worker::EXIT_SUCCESS, $status); - $this->assertSame(1, $worker->sleptFor); + $this->assertFalse($daemonFinished); + $this->assertSame(0.1, $worker->sleptFor); + $this->assertInstanceOf(TimeoutExceededException::class, $job->failedWith); + $this->events->assertDispatchedOnce(WorkerStopping::class); + $this->events->assertDispatched(WorkerStopping::class, static fn (WorkerStopping $event): bool => $event->reason === WorkerStopReason::TimedOut); + } finally { + $gate->close(); + EngineCoroutine::cancelById($daemon, throwException: true); + Coroutine::join([$daemon], 1); + } } public function testDaemonStopsImmediatelyWhenQueueIsEmptyAtStartup(): void @@ -1363,10 +1783,18 @@ public function testWorkerPassesQueuePriorityIndexesToAwareConnections(): void ], $connection->pops); } - public function testExceptionIsReportedIfConnectionThrowsExceptionOnJobPop(): void + #[TestWith([false])] + #[TestWith([true])] + public function testExceptionIsReportedIfConnectionThrowsExceptionOnJobPop(bool $reportingThrows): void { + $e = new RuntimeException('Pop failed.'); + if ($reportingThrows) { + $this->captureErrorLog(); + $this->exceptionHandler = m::mock(ExceptionHandlerContract::class); + $this->exceptionHandler->expects('report')->with($e)->andThrow(new RuntimeException('Reporting failed.')); + } $worker = new InsomniacWorker( - new WorkerFakeManager('default', new BrokenQueueConnection('default', $e = new RuntimeException)), + new WorkerFakeManager('default', new BrokenQueueConnection('default', $e)), $this->events, $this->exceptionHandler, function () { @@ -1376,29 +1804,47 @@ function () { $worker->runNextJob('default', 'queue', $this->workerOptions()); - $this->exceptionHandler->assertReported(fn (RuntimeException $reported): bool => $reported === $e); + if (! $reportingThrows) { + $this->exceptionHandler->assertReported(fn (RuntimeException $reported): bool => $reported === $e); + } } - public function testCancellationDuringJobPopEscapesWithoutReportingOrSleeping(): void + #[TestWith([false])] + #[TestWith([true])] + public function testCancellationDuringJobPopEscapesWithoutReportingOrSleeping(bool $cleanupThrows): void { $gate = $this->armCurrentCoroutineCancellation(); $worker = $this->getWorker('default', ['queue' => []]); $worker->setName('cancel-pop'); - Worker::popUsing('cancel-pop', static function () use ($gate): never { - $gate->push(true); + $cleanupException = new RuntimeException('Pop cleanup failed.'); + Worker::popUsing('cancel-pop', static function () use ($gate, $cleanupThrows, $cleanupException): never { + try { + $gate->push(true); - throw new RuntimeException('Cancellation was not delivered.'); + throw new RuntimeException('Cancellation was not delivered.'); + } finally { + if ($cleanupThrows) { + throw $cleanupException; + } + } }); + $caught = null; try { $worker->runNextJob('default', 'queue', new WorkerOptions(sleep: 5)); - $this->fail('Expected pop cancellation to escape the worker.'); - } catch (CanceledException) { - $this->assertNull($worker->sleptFor); + } catch (Throwable $exception) { + $caught = $exception; } finally { Worker::popUsing('cancel-pop', null); } + if ($cleanupThrows) { + $this->assertSame($cleanupException, $caught); + $this->assertInstanceOf(CanceledException::class, $caught->getPrevious()); + } else { + $this->assertInstanceOf(CanceledException::class, $caught); + } + $this->assertNull($worker->sleptFor); $this->exceptionHandler->assertNothingReported(); $this->events->assertNotDispatched(JobPopped::class); } @@ -1428,6 +1874,91 @@ public function testJobIsReleasedOnException(): void $this->events->assertNotDispatched(JobProcessed::class); } + #[TestWith([false])] + #[TestWith([true])] + public function testIndependentChildCancellationUsesNormalJobFailureHandling(bool $lastAttempt): void + { + $cleanupException = new RuntimeException('Child cleanup failed.'); + $job = new WorkerFakeJob(static function () use ($cleanupException): void { + (new Waiter)->wait(static function () use ($cleanupException): void { + try { + EngineCoroutine::cancelById(Coroutine::id(), throwException: true); + } finally { + Coroutine::sleep(0.001); + throw $cleanupException; + } + }); + }); + $job->attempts = 1; + $worker = $this->getWorker('default', ['queue' => [$job]]); + + $worker->runNextJob('default', 'queue', new WorkerOptions(maxTries: $lastAttempt ? 1 : 3, backoff: 10)); + + $this->assertSame($lastAttempt, $job->failed); + $this->assertSame(! $lastAttempt, $job->released); + if ($lastAttempt) { + $this->assertInstanceOf(ChildCancellationException::class, $job->failedWith); + $this->assertSame($cleanupException, $job->failedWith->getPrevious()); + } else { + $this->assertSame(10, $job->releaseAfter); + } + $this->events->assertDispatchedOnce(JobExceptionOccurred::class); + $this->events->assertDispatchedOnce(JobAttempted::class); + $this->events->assertNotDispatched(JobProcessed::class); + $this->exceptionHandler->assertReportedCount(1); + $this->exceptionHandler->assertReported(static fn (ChildCancellationException $exception): bool => $exception->getPrevious() === $cleanupException); + } + + #[TestWith(['last attempt'])] + #[TestWith(['exception limit'])] + #[TestWith(['do not retry'])] + #[TestWith(['release error'])] + public function testReleasedAttemptReportsItsErrorWithoutAnotherOutcome(string $scenario): void + { + Worker::$pausable = false; + $exception = new RuntimeException('The attempt relinquished its reservation.'); + $job = new WorkerLifecycleJob($this->container, static function (WorkerLifecycleJob $job) use ($exception): never { + $job->release(30); + + throw $exception; + }); + $releases = 0; + $job->onReleased = static function () use (&$releases, $scenario, $exception): void { + ++$releases; + + if ($scenario === 'release error') { + throw $exception; + } + }; + if ($scenario === 'do not retry') { + $this->exceptionHandler = m::mock(ShouldntRetryExceptionHandler::class)->makePartial(); + $this->exceptionHandler->expects('report')->with($exception); + $this->exceptionHandler->shouldNotReceive('shouldStopRetries'); + } + $worker = $this->getWorker('default', ['queue' => [$job]]); + $cache = new Repository(new WorkerArrayStore); + $cache->put('job-exceptions:uuid', 2, 60); + $worker->setCache($cache); + + $worker->runNextJob('default', 'queue', new WorkerOptions( + maxTries: $scenario === 'last attempt' ? 1 : 0, + backoff: 60, + )); + + $this->assertSame(1, $releases); + $this->assertTrue($job->isReleased()); + $this->assertFalse($job->hasFailed()); + $this->assertFalse($job->isDeleted()); + $this->assertSame(2, $cache->get('job-exceptions:uuid')); + $this->events->assertDispatchedOnce(JobExceptionOccurred::class); + $this->events->assertNotDispatched(JobReleasedAfterException::class); + $this->events->assertNotDispatched(JobFailed::class); + if ($scenario !== 'do not retry') { + $this->exceptionHandler->assertReportedCount(1); + $this->exceptionHandler->assertReported(static fn (RuntimeException $reported): bool => $reported === $exception); + } + } + public function testJobAttemptedEventIsDispatched(): void { $e = new RuntimeException; @@ -1766,11 +2297,102 @@ public function testFailedMarkerCleanupDoesNotPreventTheWorkersReleaseRetry(): v $this->assertNull($cache->get('job-processing:uuid')); } - #[TestWith([false])] - #[TestWith([true])] - public function testReleaseWaitsForTimeoutMarkerCleanupAndRetriesFailedCleanup(bool $cleanupThrows): void + #[TestWith([false, 'release', null])] + #[TestWith([true, 'release', null])] + #[TestWith([false, 'delete', null])] + #[TestWith([true, 'delete', null])] + #[TestWith([false, 'release', true])] + #[TestWith([false, 'release', false])] + public function testTimeoutPoliciesWaitForTheInterruptionHandlerOutcome(bool $bodyThrows, string $outcome, ?bool $throwCancellation): void { - Worker::$killOnTimeout = true; + Worker::$killOnTimeout = false; + Worker::$pausable = false; + $timer = new QueueWorkerTimer; + $bodyGate = new Channel(1); + $notificationGate = new Channel(1); + $bodyFailure = new RuntimeException('Body failed during notification.'); + $caught = null; + $notificationStarted = false; + $job = m::mock(WorkerFakeJob::class, [static function () use ($bodyGate, $bodyThrows, $bodyFailure): void { + $bodyGate->pop(); + if ($bodyThrows) { + throw $bodyFailure; + } + }])->makePartial(); + $job->shouldReceive('timeout')->andReturn(5); + $job->shouldReceive($outcome)->once()->passthru(); + $job->shouldNotReceive($outcome === 'release' ? 'delete' : 'release'); + $job->shouldFailOnTimeout = true; + $job->maxExceptions = 2; + $interruptible = m::mock(Interruptible::class); + $interruptible->expects('interrupted')->with(SIGALRM)->andReturnUsing( + static function () use ($notificationGate, &$notificationStarted, $job, $outcome): void { + $notificationStarted = true; + $notificationGate->pop(); + $job->{$outcome}(); + } + ); + $handler = m::mock(CallQueuedHandler::class); + $handler->expects('getRunningCommand')->andReturn($interruptible); + $job->resolvedJob = $handler; + $worker = new KillTestWorker(...$this->workerDependencies('default', ['queue' => [$job]], timer: $timer)); + $worker->setCache($cache = new Repository(new WorkerArrayStore)); + $worker->currentTime = 100; + $options = new WorkerOptions(maxTries: 0); + $worker->startMonitorForTest($options); + $bodyId = Coroutine::create(static function () use ($worker, $options, &$caught): void { + try { + $worker->runNextJob('default', 'queue', $options); + } catch (Throwable $exception) { + $caught = $exception; + } + }); + $worker->currentTime = 105; + $monitorId = Coroutine::create(static fn () => $timer->fire(1)); + + try { + $this->assertTrue($notificationStarted); + $bodyGate->push(true); + $this->assertTrue(Coroutine::exists($bodyId)); + $this->assertFalse($job->isDeletedOrReleased()); + $this->assertFalse($job->hasFailed()); + $this->exceptionHandler->assertNothingReported(); + + if ($throwCancellation !== null) { + EngineCoroutine::cancelById($bodyId, throwException: $throwCancellation); + Coroutine::join([$bodyId], 5); + $this->assertInstanceOf(CanceledException::class, $caught); + $this->assertFalse($job->isDeletedOrReleased()); + $this->assertFalse($job->hasFailed()); + $this->events->assertNotDispatched(JobAttempted::class); + } + + $notificationGate->push(true); + Coroutine::join([$bodyId, $monitorId], 5); + $this->assertSame($outcome === 'release', $job->isReleased()); + $this->assertSame($outcome === 'delete', $job->isDeleted()); + $this->assertFalse($job->hasFailed()); + $this->assertNull($cache->get('job-exceptions:' . $job->uuid)); + $this->events->assertNotDispatched(JobReleasedAfterException::class); + $this->events->assertNotDispatched(JobFailed::class); + $this->exceptionHandler->assertReportedCount($bodyThrows ? 2 : 0); + if ($bodyThrows) { + $this->exceptionHandler->assertReported(static fn (RuntimeException $exception): bool => $exception === $bodyFailure); + $this->exceptionHandler->assertReported(TimeoutExceededException::class); + } + if ($throwCancellation === null) { + $this->assertNull($caught); + } + } finally { + $bodyGate->close(); + $notificationGate->close(); + Coroutine::join([$bodyId, $monitorId], 5); + } + } + + public function testInterruptedReleaseSerializesMarkerCleanupWithTheFinishingBody(): void + { + Worker::$killOnTimeout = false; $timer = new QueueWorkerTimer; $worker = new KillTestWorker(...$this->workerDependencies(timer: $timer)); $worker->currentTime = 100; @@ -1787,9 +2409,8 @@ public function testReleaseWaitsForTimeoutMarkerCleanupAndRetriesFailedCleanup(b $monitorException = null; $failures = []; $coroutines = []; - $failure = new RuntimeException('Marker deletion failed.'); - $cacheEvents->listen(ForgettingKey::class, static function (ForgettingKey $event) use (&$forgetCalls, $forgetEntered, $allowForget, $cleanupThrows, $failure): void { + $cacheEvents->listen(ForgettingKey::class, static function (ForgettingKey $event) use (&$forgetCalls, $forgetEntered, $allowForget): void { if ($event->key !== 'job-processing:uuid' || ++$forgetCalls !== 1) { return; } @@ -1798,9 +2419,6 @@ public function testReleaseWaitsForTimeoutMarkerCleanupAndRetriesFailedCleanup(b if (! $allowForget->pop(5)) { throw new RuntimeException('Marker deletion was not allowed to finish.'); } - if ($cleanupThrows) { - throw $failure; - } }); $next = new WorkerLifecycleJob($this->container, static function () use ($nextStarted, $nextFinish): void { @@ -1809,13 +2427,17 @@ public function testReleaseWaitsForTimeoutMarkerCleanupAndRetriesFailedCleanup(b throw new RuntimeException('The next attempt was not allowed to finish.'); } }); - $first = new WorkerLifecycleJob($this->container, static function (WorkerLifecycleJob $job) use ($releaseGate, $releaseStarted): void { + $first = m::mock(WorkerLifecycleJob::class, [$this->container, static function () use ($releaseGate, $releaseStarted): void { if (! $releaseGate->pop(5)) { throw new RuntimeException('The first attempt was not allowed to release.'); } $releaseStarted->push(true); - $job->release(); - }, timeout: 5); + }, 5])->makePartial(); + $interruptible = m::mock(Interruptible::class); + $interruptible->expects('interrupted')->with(SIGALRM)->andReturnUsing(static fn (): mixed => $first->release()); + $handler = m::mock(CallQueuedHandler::class); + $handler->expects('getRunningCommand')->andReturn($interruptible); + $first->shouldReceive('getResolvedJob')->andReturn($handler); $first->onReleased = static function () use (&$handedOff, &$coroutines, &$failures, $worker, $next): void { $handedOff = true; $coroutines[] = Coroutine::create(static function () use ($worker, $next, &$failures): void { @@ -1854,13 +2476,9 @@ public function testReleaseWaitsForTimeoutMarkerCleanupAndRetriesFailedCleanup(b Coroutine::join(array_slice($coroutines, 0, 2), 5); $this->assertTrue($handedOff); - $this->assertSame($cleanupThrows ? 2 : 1, $forgetCalls); - if ($cleanupThrows) { - $this->assertSame($failure, $monitorException); - } else { - $this->assertInstanceOf(WorkerKilledException::class, $monitorException); - } - $this->assertSame(1, $cache->get('job-exceptions:uuid')); + $this->assertSame(1, $forgetCalls); + $this->assertNull($monitorException); + $this->assertNull($cache->get('job-exceptions:uuid')); $this->assertTrue($cache->get('job-processing:uuid')); } finally { $releaseGate->push(true, 0.01); @@ -2134,25 +2752,57 @@ public function testFlushStateResetsWorkerStaticState(): void public function testWorkerCanBeKilledUsingCustomCallback(): void { - Worker::killUsing(function (int $status): never { - throw new RuntimeException("Killed with status [{$status}]."); - }); + $failure = new RuntimeException('The kill callback failed.'); + $callbackStatus = null; + Worker::killUsing(function (int $status) use ($failure, &$callbackStatus): never { + $callbackStatus = $status; - $this->expectException(RuntimeException::class); - $this->expectExceptionMessageIs('Killed with status [124].'); + throw $failure; + }); try { (new KillTestWorker(...$this->workerDependencies('default', ['queue' => []]))) ->kill('default', 'default', 124, new WorkerOptions, WorkerStopReason::TimedOut); - } finally { - Worker::killUsing(null); + } catch (WorkerKilledException $exception) { + $this->assertSame(124, $exception->status); + } - $this->events->assertDispatchedOnce(WorkerStopping::class); - $this->events->assertDispatched(WorkerStopping::class, function (WorkerStopping $event): bool { - return $event->status === 124 - && $event->reason === WorkerStopReason::TimedOut; - }); + $this->assertSame(124, $callbackStatus); + $this->exceptionHandler->assertReported(static fn (RuntimeException $exception): bool => $exception === $failure); + $this->events->assertDispatchedOnce(WorkerStopping::class); + $this->events->assertDispatched(WorkerStopping::class, function (WorkerStopping $event): bool { + return $event->status === 124 + && $event->reason === WorkerStopReason::TimedOut; + }); + } + + #[TestWith([false])] + #[TestWith([true])] + public function testStoppingListenerFailureCannotPreventTheKillCallbackOrTermination(bool $reportingThrows): void + { + $failure = new RuntimeException('The stopping listener failed.'); + $this->spyEvents(); + $this->events->shouldReceive('dispatch')->once()->with(m::type(WorkerStopping::class))->andThrow($failure); + $this->exceptionHandler = m::mock(ExceptionHandlerContract::class); + $report = $this->exceptionHandler->shouldReceive('report')->once()->with($failure); + if ($reportingThrows) { + $this->captureErrorLog(); + $report->andThrow(new RuntimeException('Reporting failed.')); + } + + $callbackStatus = null; + Worker::killUsing(function (int $status) use (&$callbackStatus): void { + $callbackStatus = $status; + }); + + try { + (new KillTestWorker(...$this->workerDependencies())) + ->kill('default', 'queue', 124, new WorkerOptions, WorkerStopReason::TimedOut); + } catch (WorkerKilledException $exception) { + $this->assertSame(124, $exception->status); } + + $this->assertSame(124, $callbackStatus); } public function testWorkerStartingIsDispatched(): void @@ -2218,8 +2868,13 @@ public function testWorkerStoppingIsDispatched(): void } #[DataProvider('lostDatabaseConnections')] - public function testWorkerStopsWithLostConnectionReason(Throwable $failure): void + public function testWorkerStopsWithLostConnectionReason(Throwable $failure, bool $reportingThrows = false): void { + if ($reportingThrows) { + $this->captureErrorLog(); + $this->exceptionHandler = m::mock(ExceptionHandlerContract::class); + $this->exceptionHandler->expects('report')->with($failure)->andThrow(new RuntimeException('Reporting failed.')); + } $workerOptions = new WorkerOptions(stopWhenEmpty: true); $worker = $this->getWorker('default', ['queue' => [ $job = new WorkerFakeJob(static fn (): never => throw $failure), @@ -2239,11 +2894,12 @@ public function testWorkerStopsWithLostConnectionReason(Throwable $failure): voi } /** - * @return iterable + * @return iterable */ public static function lostDatabaseConnections(): iterable { yield 'query' => [self::queryException('select 1', [], 'server has gone away')]; + yield 'query with failing reporter' => [self::queryException('select 1', [], 'server has gone away'), true]; yield 'no reconnector' => [new LostConnectionException('Lost connection and no reconnector available.')]; yield 'wrapped by the job' => [new RuntimeException('The import failed.', previous: self::queryException('select 1', [], 'server has gone away'))]; yield 'wrapped driver failure' => [new QueryException('test', 'select 1', [], new RuntimeException('The driver failed.', previous: new PDOException('server has gone away')))]; @@ -2704,6 +3360,20 @@ private function workerDependencies( ]; } + /** + * Capture fallback diagnostics until this test's coroutine is cleaned up. + */ + private function captureErrorLog(): string + { + $this->errorLogDirectory = ParallelTesting::tempDir('QueueWorkerTest'); + (new Filesystem)->deleteDirectory($this->errorLogDirectory); + mkdir($this->errorLogDirectory, 0777, true); + $path = $this->errorLogDirectory . '/php-error.log'; + $this->previousErrorLog = ini_set('error_log', $path); + + return $path; + } + /** * Replace the event fake with a spy for tests that intercept dispatcher calls. */ @@ -2739,12 +3409,33 @@ private function timeOutJob(InsomniacWorker $worker, Job $job): void { $options = $this->workerOptions(); $worker->currentTime = 100; - $jobId = $worker->registerCoroutineJobForTest($job, $options); + $jobId = $this->registerRunningJob($worker, $job, $options); $worker->currentTime = $worker->runningJobExpiryForTest($jobId); $worker->terminateTimeoutJobsForTest($options); } + /** + * Register a bounded child so timeout tests interrupt a job, not the test itself. + */ + private function registerRunningJob(InsomniacWorker $worker, Job $job, WorkerOptions $options): string + { + $gate = new Channel(1); + $jobId = null; + $coroutineId = Coroutine::create(static function () use ($worker, $job, $options, $gate, &$jobId): void { + $jobId = $worker->registerCoroutineJobForTest($job, $options); + + try { + $gate->pop(5); + } catch (CanceledException) { + // The test controls the monitor and inspects its outcome separately. + } + }); + $this->registeredJobs[] = [$gate, $coroutineId]; + + return $jobId; + } + private function workerJobWithRunningCommand(object $command): WorkerFakeJob { $handler = m::mock(CallQueuedHandler::class); @@ -2996,11 +3687,6 @@ public function startMonitorForTest( $this->monitorTimeoutJobs($options, $connectionName, $queue); } - public function monitorIsLockedForTest(): bool - { - return $this->monitorLocked; - } - protected function terminateTimeoutJobs(WorkerOptions $options): void { if (++$this->scanCount === 1) { diff --git a/tests/Queue/WorkerTimeoutTest.php b/tests/Queue/WorkerTimeoutTest.php new file mode 100644 index 0000000000..e7e7edaa54 --- /dev/null +++ b/tests/Queue/WorkerTimeoutTest.php @@ -0,0 +1,163 @@ +directory = ParallelTesting::tempDir('WorkerTimeoutTest'); + (new Filesystem)->deleteDirectory($this->directory); + mkdir($this->directory, 0777, true); + touch($this->directory . '/queue.sqlite'); + } + + /** + * Remove the worker's durable storage. + */ + protected function tearDown(): void + { + (new Filesystem)->deleteDirectory($this->directory); + + parent::tearDown(); + } + + #[DataProvider('timeoutModes')] + public function testTimeoutRunsFailureHandlingBeforeTerminating(string $mode, string $cleanupOutput): void + { + $process = $this->runWorker($mode); + + $this->assertSame(SIGKILL, $process->getTermSignal()); + $this->assertSame("attempt:1\n" . $cleanupOutput . "failed-start\nfailed-finished\nfailure-recorded\ntimed-out\nstopping\nkill-callback\n", $process->getOutput()); + + $database = new PDO('sqlite:' . $this->directory . '/queue.sqlite'); + $this->assertSame(0, (int) $database->query('SELECT COUNT(*) FROM jobs')->fetchColumn()); + $failures = $database->query('SELECT * FROM failed_jobs')->fetchAll(PDO::FETCH_ASSOC); + $this->assertCount(1, $failures); + $this->assertSame('timeout-worker-job', $failures[0]['uuid']); + $this->assertStringContainsString(TimeoutExceededException::class, $failures[0]['exception']); + } + + /** + * Cover blocked PHP and slow failure cleanup with a healthy event loop. + */ + public static function timeoutModes(): array + { + return [ + 'CPU-bound job' => ['fail', ''], + 'yielding failure cleanup outlives the alarm grace' => ['yielding', ''], + 'manually suspended job' => ['suspended', ''], + 'job cleanup finishes before failure handling' => ['unwind', "unwind-start\nunwind-finished\n"], + ]; + } + + public function testTimedOutJobCanBeRetriedAfterItsReservationExpires(): void + { + $process = $this->runWorker('retry'); + $this->assertSame(SIGKILL, $process->getTermSignal()); + $this->assertSame("attempt:1\ntimed-out\nstopping\nkill-callback\n", $process->getOutput()); + + $database = new PDO('sqlite:' . $this->directory . '/queue.sqlite'); + $this->assertSame(0, (int) $database->query('SELECT COUNT(*) FROM failed_jobs')->fetchColumn()); + $job = $database->query('SELECT attempts, reserved_at FROM jobs')->fetch(PDO::FETCH_ASSOC); + $this->assertSame(1, (int) $job['attempts']); + $this->assertNotNull($job['reserved_at']); + + // Advance only the reservation expiry; the next process uses the real queue pop path. + $database->exec('UPDATE jobs SET reserved_at = 0'); + $retry = $this->runWorker('finish'); + $this->assertTrue($retry->isSuccessful()); + $this->assertSame("attempt:2\nretried\nstopping\nfinished\n", $retry->getOutput()); + $this->assertSame(0, (int) $database->query('SELECT COUNT(*) FROM jobs')->fetchColumn()); + } + + #[TestWith(['reblock'])] + #[TestWith(['blocked-cleanup'])] + public function testKernelAlarmTerminatesRecoveryThatBlocksAgain(string $mode): void + { + $process = $this->runWorker($mode); + + $this->assertSame(SIGALRM, $process->getTermSignal()); + $this->assertSame("attempt:1\nfailed-start\nfailed-finished\nfailure-recorded\ntimed-out\n", $process->getOutput()); + } + + #[TestWith(['unwind-stall', "unwind-start\n"])] + #[TestWith(['late-park', ''])] + public function testUnfinishedAttemptIsLeftReservedInsteadOfFailedConcurrently(string $mode, string $cleanupOutput): void + { + $process = $this->runWorker($mode); + + $this->assertSame(SIGKILL, $process->getTermSignal()); + $this->assertSame("attempt:1\n" . $cleanupOutput . "timed-out\nstopping\nkill-callback\n", $process->getOutput()); + $database = new PDO('sqlite:' . $this->directory . '/queue.sqlite'); + $this->assertSame(0, (int) $database->query('SELECT COUNT(*) FROM failed_jobs')->fetchColumn()); + $jobs = $database->query('SELECT attempts, reserved_at FROM jobs')->fetchAll(PDO::FETCH_ASSOC); + $this->assertCount(1, $jobs); + $this->assertSame(1, (int) $jobs[0]['attempts']); + $this->assertNotNull($jobs[0]['reserved_at']); + } + + #[TestWith(['released-timeout'])] + #[TestWith(['catch-release'])] + public function testReleasedAttemptIsNotFailedByTimeoutHandling(string $mode): void + { + $process = $this->runWorker($mode); + + $this->assertSame(SIGKILL, $process->getTermSignal()); + $this->assertSame("attempt:1\nreleased\ntimed-out\nstopping\nkill-callback\n", $process->getOutput()); + $database = new PDO('sqlite:' . $this->directory . '/queue.sqlite'); + $this->assertSame(0, (int) $database->query('SELECT COUNT(*) FROM failed_jobs')->fetchColumn()); + $jobs = $database->query('SELECT attempts, reserved_at FROM jobs')->fetchAll(PDO::FETCH_ASSOC); + $this->assertCount(1, $jobs); + $this->assertSame(1, (int) $jobs[0]['attempts']); + $this->assertNull($jobs[0]['reserved_at']); + } + + /** + * Run real worker signals in a child with an independent outer time limit. + */ + protected function runWorker(string $mode): Process + { + $process = new Process([ + PHP_BINARY, + __DIR__ . '/Fixtures/timeout-worker.php', + $this->directory . '/queue.sqlite', + $mode, + ], timeout: 20); + + try { + $process->run(); + } catch (ProcessSignaledException) { + // Expected worker termination is asserted by the caller. + } finally { + $process->stop(0, SIGKILL); + } + + $this->assertSame('', $process->getErrorOutput()); + + return $process; + } +} diff --git a/tests/Saloon/Pagination/PaginatorTest.php b/tests/Saloon/Pagination/PaginatorTest.php index 7185100363..031be5bbd8 100644 --- a/tests/Saloon/Pagination/PaginatorTest.php +++ b/tests/Saloon/Pagination/PaginatorTest.php @@ -37,6 +37,7 @@ use LogicException; use Mockery as m; use PHPUnit\Framework\Attributes\DataProvider; +use PHPUnit\Framework\Attributes\TestWith; use Psr\Http\Message\UriInterface; use RuntimeException; use Swoole\Coroutine\CanceledException; @@ -505,7 +506,9 @@ public function testFirstPageMappingFailureStopsBeforeSchedulingThePool(): void $this->assertSame(0, $paginator->totalResults()); } - public function testFirstPageHandlerCancellationStopsBeforeSchedulingRemainingPages(): void + #[TestWith([false])] + #[TestWith([true])] + public function testFirstPageHandlerCancellationStopsBeforeSchedulingRemainingPages(bool $cleanupThrows): void { $manager = $this->manager(); $requested = []; @@ -519,10 +522,11 @@ public function testFirstPageHandlerCancellationStopsBeforeSchedulingRemainingPa $started = new Channel(1); $blocker = new Channel(1); $nativeCancellation = null; + $cleanupException = new RuntimeException('Response handler cleanup failed.'); $outcome = null; - $runner = EngineCoroutine::create(static function () use ($paginator, $started, $blocker, &$nativeCancellation, &$outcome): void { + $runner = EngineCoroutine::create(static function () use ($paginator, $started, $blocker, $cleanupThrows, $cleanupException, &$nativeCancellation, &$outcome): void { try { - $paginator->pool(responseHandler: static function () use ($started, $blocker, &$nativeCancellation): void { + $paginator->pool(responseHandler: static function () use ($started, $blocker, $cleanupThrows, $cleanupException, &$nativeCancellation): void { $started->push(true); try { @@ -530,6 +534,10 @@ public function testFirstPageHandlerCancellationStopsBeforeSchedulingRemainingPa } catch (CanceledException $exception) { $nativeCancellation = $exception; throw $exception; + } finally { + if ($cleanupThrows) { + throw $cleanupException; + } } }); } catch (Throwable $exception) { @@ -543,7 +551,7 @@ public function testFirstPageHandlerCancellationStopsBeforeSchedulingRemainingPa EngineCoroutine::join([$runner->getId()], 5); $this->assertFalse(EngineCoroutine::exists($runner->getId())); $this->assertInstanceOf(CanceledException::class, $nativeCancellation); - $this->assertSame($nativeCancellation, $outcome); + $this->assertSame($cleanupThrows ? $cleanupException : $nativeCancellation, $outcome); $this->assertSame([1], $requested); } finally { if (EngineCoroutine::exists($runner->getId())) { diff --git a/tests/Saloon/Unit/Oauth2/TokenAcquisitionTest.php b/tests/Saloon/Unit/Oauth2/TokenAcquisitionTest.php index 004e763c87..8284162a09 100644 --- a/tests/Saloon/Unit/Oauth2/TokenAcquisitionTest.php +++ b/tests/Saloon/Unit/Oauth2/TokenAcquisitionTest.php @@ -4,11 +4,15 @@ namespace Hypervel\Tests\Saloon\Unit\Oauth2; +use Hypervel\Coroutine\Coroutine; use Hypervel\Engine\Channel; use Hypervel\Engine\Coroutine as EngineCoroutine; use Hypervel\Saloon\Http\Auth\TokenAcquisition; use Hypervel\Tests\TestCase; +use PHPUnit\Framework\Attributes\TestWith; +use RuntimeException; use Swoole\Coroutine\CanceledException; +use Throwable; use function Hypervel\Coroutine\go; @@ -47,21 +51,32 @@ public function testCancelingAWaiterDoesNotReleaseTheOwnersAcquisition(): void } } - public function testCanceledOwnerLeavesOneSuccessorAndDoesNotCancelWaiters(): void + #[TestWith([false])] + #[TestWith([true])] + public function testCanceledOwnerLeavesOneSuccessorAndDoesNotCancelWaiters(bool $cleanupThrows): void { $acquisition = new TokenAcquisition; $release = new Channel(1); $results = new Channel(3); - $owner = go(function () use ($acquisition, $release, $results): void { + $cleanupException = new RuntimeException('Token acquisition cleanup failed.'); + $owner = go(function () use ($acquisition, $release, $results, $cleanupThrows, $cleanupException): void { try { - $acquisition->run('grant', function () use ($release): void { - $release->pop(1); + $acquisition->run('grant', function () use ($release, $cleanupThrows, $cleanupException): void { + try { + $release->pop(1); - if ($release->isCanceled()) { - throw new CanceledException('Owner canceled.'); + if ($release->isCanceled()) { + throw new CanceledException('Owner canceled.'); + } + } finally { + if ($cleanupThrows) { + usleep(1000); + + throw $cleanupException; + } } }); - } catch (CanceledException $exception) { + } catch (Throwable $exception) { $results->push($exception); } }); @@ -72,12 +87,26 @@ public function testCanceledOwnerLeavesOneSuccessorAndDoesNotCancelWaiters(): vo return 'replacement'; }; - $first = go(fn () => $results->push($acquisition->run('grant', $resolve))); - $second = go(fn () => $results->push($acquisition->run('grant', $resolve))); + $wait = static function () use ($acquisition, $results, $resolve): void { + try { + $results->push($acquisition->run('grant', $resolve)); + } catch (Throwable $exception) { + $results->push($exception); + } + }; + $first = go($wait); + $second = go($wait); try { $this->assertTrue(EngineCoroutine::cancelById($owner)); - $this->assertInstanceOf(CanceledException::class, $results->pop(1)); + $ownerException = $results->pop(1); + $this->assertTrue(Coroutine::causedByCancellation($ownerException)); + if ($cleanupThrows) { + $this->assertSame($cleanupException, $ownerException); + $this->assertInstanceOf(CanceledException::class, $ownerException->getPrevious()); + } else { + $this->assertInstanceOf(CanceledException::class, $ownerException); + } $this->assertSame('replacement', $results->pop(1)); $this->assertSame('replacement', $results->pop(1)); $this->assertSame(1, $calls); diff --git a/tests/Saloon/Unit/PoolTest.php b/tests/Saloon/Unit/PoolTest.php index 07c3ae526b..cc83c88753 100644 --- a/tests/Saloon/Unit/PoolTest.php +++ b/tests/Saloon/Unit/PoolTest.php @@ -48,6 +48,7 @@ use Hypervel\Tests\Saloon\Fixtures\Requests\UserRequest; use InvalidArgumentException; use Mockery as m; +use PHPUnit\Framework\Attributes\TestWith; use RuntimeException; use Swoole\Coroutine\CanceledException; use Throwable; @@ -507,7 +508,9 @@ public function testProducerFailuresTakePrecedenceAndWaitForStartedChildren(): v } } - public function testIndependentRequestCancellationIsAChildFailureWithoutInvokingTheExceptionHandler(): void + #[TestWith([false])] + #[TestWith([true])] + public function testIndependentRequestCancellationIsAChildFailureWithoutInvokingTheExceptionHandler(bool $cleanupThrows): void { $manager = $this->manager(); $requestStarted = new Channel(1); @@ -516,10 +519,13 @@ public function testIndependentRequestCancellationIsAChildFailureWithoutInvoking $nativeCancellation = null; $handlerInvoked = false; $outcome = null; + $cleanupException = new RuntimeException('Request cleanup failed.'); $manager->fake([ PoolRequestStub::class => function () use ( $requestStarted, $blocker, + $cleanupThrows, + $cleanupException, &$childCoroutineId, &$nativeCancellation, ): never { @@ -531,6 +537,11 @@ public function testIndependentRequestCancellationIsAChildFailureWithoutInvoking } catch (CanceledException $exception) { $nativeCancellation = $exception; throw $exception; + } finally { + if ($cleanupThrows) { + Coroutine::sleep(0.001); + throw $cleanupException; + } } throw new RuntimeException('The request was not canceled.'); @@ -550,72 +561,109 @@ public function testIndependentRequestCancellationIsAChildFailureWithoutInvoking } }); - $this->assertTrue($requestStarted->pop()); - $this->assertIsInt($childCoroutineId); - $this->assertTrue(EngineCoroutine::cancelById($childCoroutineId, throwException: true)); - Coroutine::join([$runner->getId()]); + try { + $this->assertTrue($requestStarted->pop(1)); + $this->assertIsInt($childCoroutineId); + $this->assertTrue(EngineCoroutine::cancelById($childCoroutineId, throwException: true)); + Coroutine::join([$runner->getId()], 1); + } finally { + $blocker->close(); + Coroutine::join([$runner->getId()], 1); + } $this->assertInstanceOf(PoolException::class, $outcome); $this->assertInstanceOf(ChildCancellationException::class, $outcome->failures()['request']); - $this->assertSame($nativeCancellation, $outcome->failures()['request']->getPrevious()); + $this->assertSame($cleanupThrows ? $cleanupException : $nativeCancellation, $outcome->failures()['request']->getPrevious()); + $this->assertFalse(Coroutine::causedByCancellation($outcome)); + if ($cleanupThrows) { + $this->assertSame($nativeCancellation, $cleanupException->getPrevious()); + } $this->assertFalse($handlerInvoked); } - public function testResponseHandlerCancellationRetainsTheResponseAsAChildCallbackFailure(): void + #[TestWith([false, false])] + #[TestWith([false, true])] + #[TestWith([true, false])] + #[TestWith([true, true])] + public function testHandlerCancellationRetainsTheRequestOutcomeAsAChildCallbackFailure(bool $requestFails, bool $cleanupThrows): void { $manager = $this->manager(); - $manager->fake([PoolRequestStub::class => MockResponse::make(['ok' => true])]); + $manager->fake([PoolRequestStub::class => MockResponse::make(['ok' => ! $requestFails], $requestFails ? 500 : 200)]); $callbackStarted = new Channel(1); $blocker = new Channel(1); $childCoroutineId = null; $nativeCancellation = null; $outcome = null; $connector = new PoolConnectorStub($manager); + $cleanupException = new RuntimeException('Callback cleanup failed.'); $runner = EngineCoroutine::create(function () use ( $connector, $callbackStarted, $blocker, + $requestFails, + $cleanupThrows, + $cleanupException, &$childCoroutineId, &$nativeCancellation, &$outcome, ): void { try { - $connector->pool(['request' => new PoolRequestStub(1)]) - ->withResponseHandler(static function () use ( - $callbackStarted, - $blocker, - &$childCoroutineId, - &$nativeCancellation, - ): void { - $childCoroutineId = Coroutine::id(); - $callbackStarted->push(true); - - try { - $blocker->pop(); - } catch (CanceledException $exception) { - $nativeCancellation = $exception; - throw $exception; + $pool = $connector->pool(['request' => new PoolRequestStub(1)]); + $handler = static function () use ( + $callbackStarted, + $blocker, + $cleanupThrows, + $cleanupException, + &$childCoroutineId, + &$nativeCancellation, + ): void { + $childCoroutineId = Coroutine::id(); + $callbackStarted->push(true); + + try { + $blocker->pop(); + } catch (CanceledException $exception) { + $nativeCancellation = $exception; + throw $exception; + } finally { + if ($cleanupThrows) { + Coroutine::sleep(0.001); + throw $cleanupException; } - }) - ->send(); + } + }; + $requestFails ? $pool->withExceptionHandler($handler) : $pool->withResponseHandler($handler); + $pool->send(); } catch (Throwable $exception) { $outcome = $exception; } }); - $this->assertTrue($callbackStarted->pop()); - $this->assertIsInt($childCoroutineId); - $this->assertTrue(EngineCoroutine::cancelById($childCoroutineId, throwException: true)); - Coroutine::join([$runner->getId()]); + try { + $this->assertTrue($callbackStarted->pop(1)); + $this->assertIsInt($childCoroutineId); + $this->assertTrue(EngineCoroutine::cancelById($childCoroutineId, throwException: true)); + Coroutine::join([$runner->getId()], 1); + } finally { + $blocker->close(); + Coroutine::join([$runner->getId()], 1); + } $this->assertInstanceOf(PoolException::class, $outcome); $this->assertInstanceOf(ChildCancellationException::class, $outcome->callbackFailures()['request']); - $this->assertSame($nativeCancellation, $outcome->callbackFailures()['request']->getPrevious()); - $this->assertSame(['request'], array_keys($outcome->responses())); + $this->assertSame($cleanupThrows ? $cleanupException : $nativeCancellation, $outcome->callbackFailures()['request']->getPrevious()); + $this->assertSame($requestFails ? [] : ['request'], array_keys($outcome->responses())); + $this->assertSame($requestFails ? ['request'] : [], array_keys($outcome->failures())); + $this->assertFalse(Coroutine::causedByCancellation($outcome)); + if ($cleanupThrows) { + $this->assertSame($nativeCancellation, $cleanupException->getPrevious()); + } } - public function testProducerCancellationCancelsStartedRequestsAndEscapesExactly(): void + #[TestWith([false])] + #[TestWith([true])] + public function testProducerCancellationCancelsStartedRequestsAndEscapesExactly(bool $cleanupThrows): void { $manager = $this->manager(); $requestStarted = new Channel(1); @@ -624,6 +672,8 @@ public function testProducerCancellationCancelsStartedRequestsAndEscapesExactly( $producerBlocker = new Channel(1); $childCancellation = null; $parentCancellation = null; + $cleanupException = new RuntimeException('Request producer cleanup failed.'); + $outcome = null; $manager->fake([ PoolRequestStub::class => function () use ($requestStarted, $requestBlocker, &$childCancellation): never { $requestStarted->push(true); @@ -645,24 +695,49 @@ public function testProducerCancellationCancelsStartedRequestsAndEscapesExactly( $producerSuspended, $producerBlocker, &$parentCancellation, + $cleanupThrows, + $cleanupException, + &$outcome, ): void { try { - $connector->pool((static function () use ($producerSuspended, $producerBlocker): iterable { + $connector->pool((static function () use ($producerSuspended, $producerBlocker, &$parentCancellation, $cleanupThrows, $cleanupException): iterable { yield 'request' => new PoolRequestStub(1); $producerSuspended->push(true); - $producerBlocker->pop(); + try { + $producerBlocker->pop(); + } catch (CanceledException $exception) { + $parentCancellation = $exception; + + throw $exception; + } finally { + if ($cleanupThrows) { + usleep(1000); + + throw $cleanupException; + } + } })())->send(); - } catch (CanceledException $exception) { - $parentCancellation = $exception; + } catch (Throwable $exception) { + $outcome = $exception; } }); - $this->assertTrue($requestStarted->pop()); - $this->assertTrue($producerSuspended->pop()); - $this->assertTrue(EngineCoroutine::cancelById($runner->getId(), throwException: true)); - - $this->assertInstanceOf(CanceledException::class, $parentCancellation); - $this->assertInstanceOf(CanceledException::class, $childCancellation); + try { + $this->assertTrue($requestStarted->pop(1)); + $this->assertTrue($producerSuspended->pop(1)); + $this->assertTrue(EngineCoroutine::cancelById($runner->getId(), throwException: true)); + Coroutine::join([$runner->getId()], 1); + + $this->assertFalse(Coroutine::exists($runner->getId())); + $this->assertInstanceOf(CanceledException::class, $parentCancellation); + $this->assertInstanceOf(CanceledException::class, $childCancellation); + $this->assertSame($cleanupThrows ? $cleanupException : $parentCancellation, $outcome); + $this->assertTrue(Coroutine::causedByCancellation($outcome)); + } finally { + $requestBlocker->close(); + $producerBlocker->close(); + Coroutine::join([$runner->getId()], 1); + } } public function testRepeatedKeysUseTheLaterInputAndProcessDoesNotCollectResponses(): void diff --git a/tests/Scout/Unit/Console/ConcurrentImportRunnerTest.php b/tests/Scout/Unit/Console/ConcurrentImportRunnerTest.php index e0c6a06465..f4a80e3c6e 100644 --- a/tests/Scout/Unit/Console/ConcurrentImportRunnerTest.php +++ b/tests/Scout/Unit/Console/ConcurrentImportRunnerTest.php @@ -6,16 +6,21 @@ use Hypervel\Config\Repository; use Hypervel\Container\Container; +use Hypervel\Context\CoroutineContext; use Hypervel\Coroutine\Coroutine; use Hypervel\Coroutine\Exceptions\ChildCancellationException; use Hypervel\Database\Connectors\ConnectionFactory; use Hypervel\Database\DatabaseManager; +use Hypervel\Database\Pool\ConnectionLease; use Hypervel\Engine\Channel; use Hypervel\Engine\Coroutine as EngineCoroutine; use Hypervel\Scout\Console\ConcurrentImportRunner; use Hypervel\Tests\TestCase; +use Mockery as m; +use PHPUnit\Framework\Attributes\TestWith; use RuntimeException; use Swoole\Coroutine\CanceledException; +use Throwable; class ConcurrentImportRunnerTest extends TestCase { @@ -114,6 +119,45 @@ public function testWaitDrainsActiveChildrenBeforeRethrowingTheFirstFailure(): v $this->assertTrue($completed); } + #[TestWith([false])] + #[TestWith([true])] + public function testCancellationWhileReleasingTheParentConnectionCancelsActiveImports(bool $cleanupThrows): void + { + $failure = $cleanupThrows ? new RuntimeException('Release cleanup failed.', previous: new CanceledException) : new CanceledException; + $lease = m::mock(ConnectionLease::class); + $lease->shouldReceive('releaseIfIdle')->once()->andThrow($failure); + $runner = new ConcurrentImportRunner(1); + $blocker = new Channel(1); + $childCanceled = false; + $childId = null; + $caught = null; + $runner->create(static function () use ($blocker, &$childCanceled, &$childId): void { + $childId = Coroutine::id(); + + try { + $blocker->pop(1); + } catch (CanceledException) { + $childCanceled = true; + } + }); + CoroutineContext::set('__scout.test.connection_lease', $lease); + + try { + $runner->wait(); + } catch (Throwable $exception) { + $caught = $exception; + } finally { + CoroutineContext::forget('__scout.test.connection_lease'); + $blocker->close(); + if ($childId !== null) { + Coroutine::join([$childId], 1); + } + } + + $this->assertSame($failure, $caught); + $this->assertTrue($childCanceled); + } + public function testWaitPreservesTheFirstFailureWhenMultipleChildrenFail(): void { $runner = new ConcurrentImportRunner(2); @@ -144,17 +188,22 @@ public function testWaitPreservesTheFirstFailureWhenMultipleChildrenFail(): void } } - public function testWaitSurfacesIndependentChildCancellationAsAChildFailure(): void + #[TestWith([false])] + #[TestWith([true])] + public function testWaitSurfacesIndependentChildCancellationAsAChildFailure(bool $cleanupThrows): void { $runner = new ConcurrentImportRunner(1); $operationStarted = new Channel(1); $blocker = new Channel(1); $childCoroutineId = null; $nativeCancellation = null; + $cleanupException = new RuntimeException('Import cleanup failed.'); $runner->create(static function () use ( $operationStarted, $blocker, + $cleanupThrows, + $cleanupException, &$childCoroutineId, &$nativeCancellation, ): void { @@ -166,19 +215,33 @@ public function testWaitSurfacesIndependentChildCancellationAsAChildFailure(): v } catch (CanceledException $exception) { $nativeCancellation = $exception; throw $exception; + } finally { + if ($cleanupThrows) { + Coroutine::sleep(0.001); + throw $cleanupException; + } } }); - $this->assertTrue($operationStarted->pop()); - $this->assertIsInt($childCoroutineId); - $this->assertTrue(EngineCoroutine::cancelById($childCoroutineId, throwException: true)); - try { + $this->assertTrue($operationStarted->pop(1)); + $this->assertIsInt($childCoroutineId); + $this->assertTrue(EngineCoroutine::cancelById($childCoroutineId, throwException: true)); $runner->wait(); $this->fail('The child cancellation was not rethrown.'); } catch (ChildCancellationException $exception) { $this->assertSame('A child coroutine running a Scout import was canceled while its owner remained active.', $exception->getMessage()); - $this->assertSame($nativeCancellation, $exception->getPrevious()); + $this->assertSame($cleanupThrows ? $cleanupException : $nativeCancellation, $exception->getPrevious()); + $this->assertFalse(Coroutine::causedByCancellation($exception)); + + if ($cleanupThrows) { + $this->assertSame($nativeCancellation, $cleanupException->getPrevious()); + } + } finally { + $blocker->close(); + if ($childCoroutineId !== null) { + Coroutine::join([$childCoroutineId], 1); + } } } } diff --git a/tests/Scout/Unit/EngineOperationRunnerTest.php b/tests/Scout/Unit/EngineOperationRunnerTest.php index d5ba7a7f7b..8f38b3e900 100644 --- a/tests/Scout/Unit/EngineOperationRunnerTest.php +++ b/tests/Scout/Unit/EngineOperationRunnerTest.php @@ -10,6 +10,7 @@ use Hypervel\Scout\EngineOperation; use Hypervel\Scout\EngineOperationRunner; use Hypervel\Tests\TestCase; +use PHPUnit\Framework\Attributes\TestWith; use RuntimeException; use Swoole\Coroutine\CanceledException; use Throwable; @@ -195,10 +196,12 @@ function () use (&$finished, $secondFailure): never { $this->assertSame(['first', 'second'], $finished); } - public function testStartingCancellationSkipsAllCompletionAndOperationWork(): void + #[TestWith([false])] + #[TestWith([true])] + public function testStartingCancellationSkipsAllCompletionAndOperationWork(bool $cleanupThrows): void { $runner = new EngineOperationRunner; - $cancellation = new CanceledException; + $cancellation = $cleanupThrows ? new RuntimeException('Cleanup failed.', previous: new CanceledException) : new CanceledException; $finished = false; $callbackCalled = false; @@ -230,10 +233,12 @@ function () use ($cancellation): never { $this->assertFalse($callbackCalled); } - public function testOperationCancellationSkipsAllCompletion(): void + #[TestWith([false])] + #[TestWith([true])] + public function testOperationCancellationSkipsAllCompletion(bool $cleanupThrows): void { $runner = new EngineOperationRunner; - $cancellation = new CanceledException; + $cancellation = $cleanupThrows ? new RuntimeException('Cleanup failed.', previous: new CanceledException) : new CanceledException; $finished = false; $runner->observe($this->observer( @@ -257,11 +262,13 @@ function () use (&$finished): void { $this->assertFalse($finished); } - public function testCompletionCancellationStopsLaterObserversAndSupersedesOperationFailure(): void + #[TestWith([false])] + #[TestWith([true])] + public function testCompletionCancellationStopsLaterObserversAndSupersedesOperationFailure(bool $cleanupThrows): void { $runner = new EngineOperationRunner; $operationFailure = new RuntimeException('operation failed'); - $cancellation = new CanceledException; + $cancellation = $cleanupThrows ? new RuntimeException('Cleanup failed.', previous: new CanceledException) : new CanceledException; $secondFinished = false; $runner->observe($this->observer( diff --git a/tests/ServerProcess/ListenMethodTest.php b/tests/ServerProcess/ListenMethodTest.php index b63a8b03c1..ac6501f888 100644 --- a/tests/ServerProcess/ListenMethodTest.php +++ b/tests/ServerProcess/ListenMethodTest.php @@ -218,7 +218,7 @@ public function testListenerStopsWhenTheProcessSocketCannotBeExported(): void public function testListenerClosesItsChannelWhenTheExceptionReporterThrows(): void { - Coroutine::enableReportException(false); + Coroutine::reportExceptions(false); $reporterFailure = new RuntimeException('reporter failed'); $handler = m::mock(ExceptionHandlerContract::class); $handler->shouldReceive('report') diff --git a/tests/Support/SupportHelpersTest.php b/tests/Support/SupportHelpersTest.php index 25c1e16c51..c25613444d 100644 --- a/tests/Support/SupportHelpersTest.php +++ b/tests/Support/SupportHelpersTest.php @@ -24,10 +24,12 @@ use IteratorAggregate; use LogicException; use PHPUnit\Framework\Attributes\DataProvider; +use PHPUnit\Framework\Attributes\TestWith; use ReflectionClass; use RuntimeException; use stdClass; use Swoole\Coroutine\CanceledException; +use Throwable; use Traversable; class SupportHelpersTest extends TestCase @@ -1239,9 +1241,12 @@ public function testRetryWithFractionalSleep(): void Sleep::assertSequence([Sleep::usleep(100_500)]); } - public function testCancellationIsNotRetriedOrPassedToRetryPolicy(): void + #[TestWith([false])] + #[TestWith([true])] + public function testCancellationIsNotRetriedOrPassedToRetryPolicy(bool $cleanupThrows): void { $cancellation = new CanceledException('operation cancelled'); + $failure = $cleanupThrows ? new RuntimeException('Cleanup failed.', previous: $cancellation) : $cancellation; $attempts = 0; $policyCalled = false; $sleepCalled = false; @@ -1249,10 +1254,10 @@ public function testCancellationIsNotRetriedOrPassedToRetryPolicy(): void try { retry( 3, - function () use (&$attempts, $cancellation): never { + function () use (&$attempts, $failure): never { ++$attempts; - throw $cancellation; + throw $failure; }, function () use (&$sleepCalled): int { $sleepCalled = true; @@ -1266,8 +1271,8 @@ function () use (&$policyCalled): bool { }, ); $this->fail('Expected the operation cancellation to be thrown.'); - } catch (CanceledException $exception) { - $this->assertSame($cancellation, $exception); + } catch (Throwable $exception) { + $this->assertSame($failure, $exception); } $this->assertSame(1, $attempts);