[danielgerlag]

Fixes #1451

What and why

If GetWorkflowInstance throws for a runnable instance (for example its stored Data no longer deserializes), the instance is polled, fails, and is polled again every poll interval forever. Each attempt takes a lock, does a database read and logs two errors.

This change counts consecutive load failures per instance in WorkflowConsumer. After MaxWorkflowLoadAttempts (default 5) the instance is quarantined on that node:

• one error is logged saying it is quarantined and that its stored state is unchanged
• later queue items for it return immediately without taking the lock or reading the database
• it stays greylisted, so RunnablePoller stops re-queuing it except once per greylist TTL (5 minutes)

A successful load clears the count. Only failures from GetWorkflowInstance are counted, so exceptions while running steps keep the existing retry behavior.

Design choices

• I picked an in-memory quarantine over marking the row Suspended. Changing status without loading the instance would need a new persistence method on IPersistenceProvider, which every provider (and third party providers) would have to implement. This keeps the fix inside core with no provider changes.
• New option WorkflowOptions.UseMaxWorkflowLoadAttempts(int), following the existing Use... style.
• Added InternalsVisibleTo for WorkflowCore.UnitTests so the consumer can be tested directly.
• Documented in docs/error-handling.md.

How it was tested

New WorkflowConsumerFixture (6 tests) drives ProcessItem with a fake persistence provider that throws on load. With the old WorkflowConsumer.cs restored, 3 of them fail: the load was attempted 20 times out of 20 instead of stopping at the limit. With the fix all 6 pass.

• dotnet build WorkflowCore.sln: succeeded, 0 errors
• WorkflowCore.UnitTests: 68 tests per framework. All pass on net6.0 and net8.0. On net10.0 should_run_pre_middlewares (SyncWorkflowRunnerTests) failed once; it also failed on master before any change (net8.0 run), so it looks flaky and unrelated.
• WorkflowCore.IntegrationTests: 45/45 on net6.0, net8.0 and net10.0

Limits

• Quarantine is per node and resets on restart, so each node and each restart allows another 5 attempts. In a cluster the instance is still tried up to 5 times per node.
• The instance's stored status stays Runnable. Nothing outside the logs shows it is quarantined; no lifecycle event is published.
• To bring it back after repairing the data, restart the host.