Keep armed faults across Durable Object eviction

Hosted instances kept faults only in memory. When Cloudflare replaced an
instance, its armed faults and fault counter were gone while the store and
ledger came back, so a fault armed for three requests read as spent after
one. Persist faults with the rest of the instance, and store a consumed
count before a delayed fault's response.

Author: RhysSullivan

README.md

@@ -73,7 +73,7 @@ The request ledger is a core feature, not a debug afterthought. Each entry recor
 
 ### Fault injection
 
-Tests and development harnesses can arm one-shot faults with `POST /_emulate/faults`. A fault matches all provided criteria: `operationId` when the emulator can resolve one from its service manifest, `method`, and `pathPattern`. `pathPattern` uses a simple glob where `*` matches any characters in the request path. Matching requests short-circuit with the configured response, decrement `remaining`, and are removed at zero. Faulted requests are still written to `/_emulate/ledger` with `faulted: true` and `faultId`.
+Tests and development harnesses can arm one-shot faults with `POST /_emulate/faults`. A fault matches all provided criteria: `operationId` when the emulator can resolve one from its service manifest, `method`, and `pathPattern`. `pathPattern` uses a simple glob where `*` matches any characters in the request path. Matching requests short-circuit with the configured response, decrement `remaining`, and are removed at zero. Faulted requests are still written to `/_emulate/ledger` with `faulted: true` and `faultId`. On the hosted Cloudflare surface armed faults, their remaining counts and the next fault id are persisted with the instance, so a fault keeps its count when Cloudflare replaces the instance between arming it and the requests it targets.
 
 ```bash
 curl -s -X POST "$EMULATOR_URL/_emulate/faults" \
@@ -213,7 +213,7 @@ Documentation is a separate site at `https://docs.emulators.dev`. Per-service do
 
 ### Deployment
 
-The emulator worker is named `emulate-hosts`. It serves `emulators.dev/*` (the apex catalog) and `*.emulators.dev/*`, that is every service and instance subdomain. The `EmulatorDurableObject` class is declared through a wrangler `migrations` entry (`new_classes`). Each stateful instance is backed by one `EmulatorDurableObject`; both the store snapshot and the request ledger are persisted to Durable Object storage so instances survive eviction.
+The emulator worker is named `emulate-hosts`. It serves `emulators.dev/*` (the apex catalog) and `*.emulators.dev/*`, that is every service and instance subdomain. The `EmulatorDurableObject` class is declared through a wrangler `migrations` entry (`new_classes`). Each stateful instance is backed by one `EmulatorDurableObject`; the store snapshot, the request ledger and armed faults are persisted to Durable Object storage so instances survive eviction.
 
 The docs site (`apps/web`, worker name `emulate-docs`) serves `docs.emulators.dev`. Because that host is more specific than `*.emulators.dev`, it wins for docs traffic.
 

apps/web/app/docs/architecture/page.mdx

@@ -85,6 +85,6 @@ The `@emulators/cloudflare` package runs the same plugins on Cloudflare Workers
 - A service host such as `github.emulators.dev` serves a service-level control plane and is useful before any instance exists. It has no shared instance of its own; provider traffic requires a created instance.
 - An instance host such as `github.my-run.emulators.dev` (or the compatibility path form `emulators.dev/github/my-run`) serves one stateful instance.
 
-Each instance is backed by one Cloudflare Durable Object. The `EmulatorDurableObject` class is declared through a wrangler `migrations` entry (`new_classes`), and both the instance state and the request ledger are snapshotted to Durable Object storage so an instance survives eviction. Instances are created lazily through `POST /_emulate/instances`.
+Each instance is backed by one Cloudflare Durable Object. The `EmulatorDurableObject` class is declared through a wrangler `migrations` entry (`new_classes`), and the instance state, the request ledger and armed faults are snapshotted to Durable Object storage so an instance survives eviction. Instances are created lazily through `POST /_emulate/instances`.
 
 Docs are a separate site on the `emulate-docs` worker at `docs.emulators.dev`, with per-service docs at `https://docs.emulators.dev/<service>`. Because that host is more specific than the apex, it wins for the docs subdomain. See [Deployment](/docs/deployment).

apps/web/app/docs/deployment/page.mdx

@@ -77,7 +77,7 @@ A bare service host such as `https://github.emulators.dev`, with no instance, se
 
 ## Instance model
 
-The `emulate-hosts` worker serves `emulators.dev/*` (the apex catalog) and `*.emulators.dev/*` (service and instance hosts). Each stateful instance is backed by one Cloudflare Durable Object. The `EmulatorDurableObject` class is declared through a wrangler `migrations` entry (`new_classes`), and both the instance state and the request ledger are snapshotted to Durable Object storage, so an instance survives eviction.
+The `emulate-hosts` worker serves `emulators.dev/*` (the apex catalog) and `*.emulators.dev/*` (service and instance hosts). Each stateful instance is backed by one Cloudflare Durable Object. The `EmulatorDurableObject` class is declared through a wrangler `migrations` entry (`new_classes`), and the instance state, the request ledger and armed faults are snapshotted to Durable Object storage, so an instance survives eviction.
 
 ```text
 POST https://github.emulators.dev/_emulate/instances

packages/@emulators/cloudflare/src/__tests__/worker.test.ts

@@ -598,6 +598,58 @@ describe("cloudflare durable object control plane", () => {
     expect(((await exchange.json()) as { error: string }).error).toBe("invalid_grant");
   });
 
+  // Executor's slow-Autumn e2e arms a delayed fault for three requests, sends one, gives up on
+  // it, and reads the remaining count. Cloudflare can replace the instance at any point of
+  // that, so the count and the ids must outlive it.
+  it("keeps armed faults, their counts and their ids across eviction", async () => {
+    const { storage, state } = makeState();
+    const first = new EmulatorDurableObject(state, {});
+    const arm = (durableObject: EmulatorDurableObject, delayMs: number) =>
+      durableObject
+        .fetch(
+          new Request("https://github.my-run.emulators.dev/_emulate/faults", {
+            method: "POST",
+            headers: idHeaders({ "content-type": "application/json" }),
+            body: JSON.stringify({
+              match: { method: "GET", pathPattern: "/repos/*" },
+              response: { status: 503, body: { message: "synthetic outage" } },
+              times: 3,
+              delayMs,
+            }),
+          }),
+        )
+        .then(async (res) => ((await res.json()) as { fault: { id: string } }).fault.id);
+    const remaining = async (durableObject: EmulatorDurableObject) => {
+      const res = await durableObject.fetch(
+        new Request("https://github.my-run.emulators.dev/_emulate/faults", { headers: idHeaders() }),
+      );
+      const body = (await res.json()) as { faults: Array<{ id: string; remaining: number }> };
+      return new Map(body.faults.map((fault) => [fault.id, fault.remaining]));
+    };
+
+    const delayed = await arm(first, 300);
+    // The caller abandons this request; its consumed count is stored before the delay ends.
+    const pending = first.fetch(
+      new Request("https://github.my-run.emulators.dev/repos/acme/widget", { headers: idHeaders() }),
+    );
+    for (let attempt = 0; attempt < 50; attempt++) {
+      const stored = storage.get("faults") as { faults: Array<{ remaining: number }> } | undefined;
+      if (stored?.faults[0]?.remaining === 2) break;
+      await new Promise((resolve) => setTimeout(resolve, 5));
+    }
+
+    const evicted = new EmulatorDurableObject(state, {});
+    expect((await remaining(evicted)).get(delayed)).toBe(2);
+    const next = await arm(evicted, 0);
+    expect(next).not.toBe(delayed);
+    const faulted = await evicted.fetch(
+      new Request("https://github.my-run.emulators.dev/repos/acme/widget", { headers: idHeaders() }),
+    );
+    expect(faulted.status).toBe(503);
+    expect((await remaining(evicted)).get(delayed)).toBe(1);
+    expect((await pending).status).toBe(503);
+  });
+
   it("reports the real instance id in the manifest", async () => {
     const { state } = makeState();
     const durableObject = new EmulatorDurableObject(state, {});

packages/@emulators/cloudflare/src/durable-object.ts

@@ -1,6 +1,8 @@
 import {
   createServer,
   type AppKeyResolver,
+  type FaultRegistry,
+  type FaultRegistrySnapshot,
   type LedgerEntry,
   type LedgerSnapshot,
   type RequestLedger,
@@ -51,6 +53,7 @@ interface Live {
   store: Store;
   tokenMap: TokenMap;
   ledger: RequestLedger;
+  faults: FaultRegistry;
   service: string;
   instance: string;
   baseUrl: string;
@@ -65,6 +68,9 @@ const SNAPSHOT_DATA_PREFIX = "snapshot:data:";
 const LEDGER_META_KEY = "ledger:meta";
 const LEDGER_ENTRY_PREFIX = "ledger:entry:";
 const MINTED_PREFIX = "minted:";
+// Armed faults and the next fault id. Cloudflare can replace an instance at any time, even
+// between a test arming a fault and the request it targets, so faults persist with the store.
+const FAULTS_KEY = "faults";
 
 interface SnapshotCollectionMeta {
   autoId: number;
@@ -308,6 +314,7 @@ export class EmulatorDurableObject {
     const strict = persisted.strict ?? true;
     const snapshot = await this.readStoreSnapshot(persisted.snapshot);
     const ledgerSnapshot = await this.readLedger(persisted.ledger);
+    const faultSnapshot = await this.state.storage.get<FaultRegistrySnapshot>(FAULTS_KEY);
     const minted = await this.readMinted(persisted.minted);
     const tokens = this.buildTokens(persisted.seed, strict, minted);
 
@@ -323,7 +330,7 @@ export class EmulatorDurableObject {
       : entry.defaultFallback(persisted.seed?.[service] as Record<string, unknown> | undefined);
 
     let resetService = async () => {};
-    const { app, store, tokenMap, webhooks, ledger } = createServer(entry.plugin, {
+    const { app, store, tokenMap, webhooks, ledger, faults } = createServer(entry.plugin, {
       baseUrl,
       tokens,
       appKeyResolver,
@@ -332,6 +339,7 @@ export class EmulatorDurableObject {
       instance,
       ledgerPersistent: true,
       reset: () => resetService(),
+      onFaultConsumed: () => this.persist(),
       seed: async (seed) => {
         if (seed && entry.seedFromConfig) {
           entry.seedFromConfig(store, baseUrl, seed as Record<string, unknown>);
@@ -360,6 +368,7 @@ export class EmulatorDurableObject {
     if (snapshot) store.restore(snapshot);
     else this.seedInto(store, service, baseUrl, persisted.seed);
     ledger.restore(ledgerSnapshot);
+    faults.restore(faultSnapshot);
     resetService = async () => {
       store.reset();
       webhooks.clear();
@@ -379,7 +388,17 @@ export class EmulatorDurableObject {
       }
     }
 
-    this.live = { app, store, tokenMap: tokenMap as TokenMap, ledger, service, instance, baseUrl, reset: resetService };
+    this.live = {
+      app,
+      store,
+      tokenMap: tokenMap as TokenMap,
+      ledger,
+      faults,
+      service,
+      instance,
+      baseUrl,
+      reset: resetService,
+    };
     if (entry.prepare) {
       // Persist prepared state now: a read-only first request would not persist it,
       // and a rebuild must not replace state a client has already observed.
@@ -408,6 +427,7 @@ export class EmulatorDurableObject {
       counter: ledger.counter,
       ids: ledger.entries.map((entry) => entry.id),
     } satisfies LedgerMeta);
+    entries.set(FAULTS_KEY, live.faults.serialize());
     return entries;
   }
 
@@ -416,7 +436,7 @@ export class EmulatorDurableObject {
     for (const prefix of [SNAPSHOT_ITEM_PREFIX, SNAPSHOT_DATA_PREFIX, LEDGER_ENTRY_PREFIX]) {
       for (const [key, value] of await this.state.storage.list({ prefix })) written.set(key, JSON.stringify(value));
     }
-    for (const key of [SNAPSHOT_META_KEY, LEDGER_META_KEY]) {
+    for (const key of [SNAPSHOT_META_KEY, LEDGER_META_KEY, FAULTS_KEY]) {
       const value = await this.state.storage.get(key);
       if (value !== undefined) written.set(key, JSON.stringify(value));
     }
@@ -469,6 +489,7 @@ export class EmulatorDurableObject {
       await this.clearStoreSnapshot();
       await this.clearLedger();
       await this.clearMinted();
+      await this.state.storage.delete(FAULTS_KEY);
       await this.writeStateMeta({ seed, strict: strict !== false });
       this.live = undefined;
       this.written = undefined;

packages/@emulators/core/src/faults.ts

@@ -37,10 +37,25 @@ export interface FaultLedgerMarker {
   faultId: string;
 }
 
+/** Every armed fault and the next id, so a host can keep them across a restart. */
+export interface FaultRegistrySnapshot {
+  counter: number;
+  faults: ArmedFault[];
+}
+
 export class FaultRegistry {
   private faults: ArmedFault[] = [];
   private counter = 1;
 
+  serialize(): FaultRegistrySnapshot {
+    return { counter: this.counter, faults: this.faults.map(cloneFault) };
+  }
+
+  restore(snapshot: FaultRegistrySnapshot | undefined): void {
+    this.faults = (snapshot?.faults ?? []).map(cloneFault);
+    this.counter = snapshot?.counter ?? 1;
+  }
+
   arm(input: FaultArmInput): ArmedFault {
     validateFaultInput(input);
     const fault: ArmedFault = {
@@ -87,7 +102,15 @@ export class FaultRegistry {
   }
 }
 
-export function createFaultMiddleware(faults: FaultRegistry, manifest: ServiceManifest): MiddlewareHandler<AppEnv> {
+export function createFaultMiddleware(
+  faults: FaultRegistry,
+  manifest: ServiceManifest,
+  /**
+   * Awaited after a request consumes a fault and before its delay, so a host that persists
+   * faults stores the new count before a delayed response, which its caller may abandon.
+   */
+  onConsumed?: () => void | Promise<void>,
+): MiddlewareHandler<AppEnv> {
   const operations = manifest.specs.flatMap((spec) => spec.operations ?? []);
 
   return async (c, next) => {
@@ -113,6 +136,7 @@ export function createFaultMiddleware(faults: FaultRegistry, manifest: ServiceMa
     }
 
     c.set("fault", { faulted: true, faultId: fault.id });
+    await onConsumed?.();
     if (fault.delayMs && fault.delayMs > 0) {
       await new Promise((resolve) => setTimeout(resolve, fault.delayMs));
     }

packages/@emulators/core/src/index.ts

@@ -79,6 +79,7 @@ export {
   type FaultArmInput,
   type FaultLedgerMarker,
   type FaultMatch,
+  type FaultRegistrySnapshot,
   type FaultResponse,
 } from "./faults.js";
 export {

packages/@emulators/core/src/server.ts

@@ -33,6 +33,8 @@ export interface ServerOptions {
   reset?: () => void | Promise<void>;
   seed?: (seed: unknown) => void | Promise<void>;
   issueCredential?: (request: CredentialRequest) => IssuedCredential | Promise<IssuedCredential>;
+  /** Awaited after a request consumes a fault, before its delay (see `createFaultMiddleware`). */
+  onFaultConsumed?: () => void | Promise<void>;
 }
 
 export function createServer(plugin: ServicePlugin, options: ServerOptions = {}) {
@@ -90,7 +92,10 @@ export function createServer(plugin: ServicePlugin, options: ServerOptions = {})
   }
 
   app.use("*", createLedgerMiddleware(ledger, { webhooks }));
-  app.use("*", createFaultMiddleware(faults, options.manifest ?? createDefaultManifest(plugin.name)));
+  app.use(
+    "*",
+    createFaultMiddleware(faults, options.manifest ?? createDefaultManifest(plugin.name), options.onFaultConsumed),
+  );
 
   const rateLimitCounters = new Map<string, { remaining: number; resetAt: number }>();
   let lastPruneAt = Math.floor(Date.now() / 1000);