Skip to content

Use fixed stores for CLI E2E tests - #8949

Draft
isaacroldan wants to merge 1 commit into
codex/e2e-parallel-store-cleanupfrom
codex/e2e-persistent-stores
Draft

isaacroldan wants to merge 1 commit into
codex/e2e-parallel-store-cleanupfrom
codex/e2e-persistent-stores

Conversation

@isaacroldan

@isaacroldan isaacroldan commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

WHY are these changes introduced?

Creating and deleting a development store for every app test adds provisioning delays and store-quota failures.

WHAT is this pull request doing?

Give each of the seven dev test cases a fixed store from the E2E_STORES map, with an eighth store kept as a spare. An optional E2E_STORE_FQDN overrides these assignments locally. Playwright handles sharding directly; store allocation, capacity checks, and store-sharing locks are removed. Four workers per shard retain the current CI load.

Every test creates a separate app and cleans only that app and preview. Concurrent runs may share stores. App names and cleanup patterns include the run, attempt, and shard. Remote cleanup failures produce warnings without blocking store reuse; failure to stop CLI processes remains an error.

Add a dedicated CLI store creation/deletion lifecycle case to retain provisioning coverage, and fix cleanup links to use numeric app IDs. This layer retains browser app deletion; #8947 replaces it with HTTP cleanup. Store commands are unchanged.

How to manually test your changes?

Set E2E_STORE_FQDN in packages/e2e/.env to use a development store accessible to your account, or leave it empty to use the fixed assignments in setup/constants.ts. Review the assignments and cleanup recovery instructions in packages/e2e/README.md.

With two separate app projects, run pnpm shopify app dev --path <app-path> --store <store.myshopify.com> for each in turn. Confirm both apps can use the store even when the first app remains installed.

Validation and remaining limits

33 local E2E tests pass on this layer, including cleanup-failure and store-protection checks. E2E lint and type checks pass. Playwright discovery selects all remote cases exactly once across the two shards without a configured store list. A targeted live store lifecycle check passed during development.

Concurrent use of one store by separate apps is supported by inspection of the backend's app-and-shop session keys; overlapping live E2E runs have not been stress-tested. Full live app-dev coverage remains unverified because the last attempt hit the generated template's unsupported 2025-10 API version. Remote E2E jobs remain non-blocking in CI.

Checklist

  • I've considered possible cross-platform impacts (Mac, Linux, Windows)
  • I've considered possible documentation changes
  • I've considered analytics changes to measure impact

@github-actions github-actions Bot added the no-changelog This PR doesn't include a changeset entry. Is an internal only change not relevant to end users. label Oct 19, 2026
@isaacroldan
isaacroldan added this pull request to stack #8950 October 19, 2026 13:50
@isaacroldan
isaacroldan removed this pull request from stack #8950 October 19, 2026 14:10
@isaacroldan
isaacroldan force-pushed the codex/e2e-persistent-stores branch from 5daa401 to 1dcbdb5 Compare October 19, 2026 14:11
@isaacroldan isaacroldan changed the title Reuse persistent stores in CLI E2E tests Use fixed stores for CLI E2E tests Oct 19, 2026
@isaacroldan
isaacroldan added this pull request to stack #8951 October 19, 2026 14:11
@github-actions

Copy link
Copy Markdown
Contributor

Differences in type declarations

We detected differences in the type declarations generated by Typescript for this branch compared to the baseline ('main' branch). Please, review them to ensure they are backward-compatible. Here are some important things to keep in mind:

  • Some seemingly private modules might be re-exported through public modules.
  • If the branch is behind main you might see odd diffs, rebase main into this branch.

New type declarations

We found no new type declarations in this PR

Existing type declarations

packages/cli-kit/dist/public/node/base-command.d.ts
@@ -21,7 +21,6 @@ declare abstract class BaseCommand extends Command {
     static descriptionWithoutMarkdown(): string | undefined;
     static analyticsNameOverride(): string | undefined;
     static analyticsStopCommand(): string | undefined;
-    private jsonOutputFromFlags;
     catch(error: Error & {
         skipOclifErrorHandling: boolean;
     }): Promise<void>;
@@ -37,7 +36,6 @@ declare abstract class BaseCommand extends Command {
     }, TGlobalFlags extends FlagOutput, TArgs extends ArgOutput>(options?: Input<TFlags, TGlobalFlags, TArgs>, argv?: string[]): Promise<ParserOutput<TFlags, TGlobalFlags, TArgs> & {
         argv: string[];
     }>;
-    protected runWithParsedOutput<T>(flags: FlagOutput, execute: () => T): T;
     protected environmentsFilename(): string | undefined;
     protected parseEnvironmentFlags<TFlags extends FlagOutput, TGlobalFlags extends FlagOutput, TArgs extends ArgOutput>(environment: JsonMap, options: Input<TFlags, TGlobalFlags, TArgs>, explicitFlags: FlagOutput): Promise<FlagOutput>;
     protected validateNonTTYFlags(flags: FlagOutput): void;
packages/cli-kit/dist/public/node/themes/theme-manager.d.ts
@@ -11,12 +11,5 @@ export declare abstract class ThemeManager {
     findOrCreate(name?: string, role?: Role): Promise<Theme>;
     fetch(name?: string, role?: Role): Promise<Theme | undefined>;
     generateThemeName(context: string): string;
-    create(themeRole?: Role, themeName?: string): Promise<{
-        createdAtRuntime: boolean;
-        id: number;
-        name: string;
-        processing: boolean;
-        role: string;
-        src?: string;
-    }>;
+    create(themeRole?: Role, themeName?: string): Promise<Theme>;
 }
\ No newline at end of file
packages/cli-kit/dist/cli/api/graphql/admin/generated/metafield_definitions_by_owner_type.d.ts
@@ -2,14 +2,9 @@ import * as Types from './types.js';
 import { TypedDocumentNode as DocumentNode } from '@graphql-typed-document-node/core';
 export type MetafieldDefinitionsByOwnerTypeQueryVariables = Types.Exact<{
     ownerType: Types.MetafieldOwnerType;
-    after?: Types.InputMaybe<Types.Scalars['String']['input']>;
 }>;
 export type MetafieldDefinitionsByOwnerTypeQuery = {
     metafieldDefinitions: {
-        pageInfo: {
-            hasNextPage: boolean;
-            endCursor?: string | null;
-        };
         nodes: {
             key: string;
             name: string;

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-changelog This PR doesn't include a changeset entry. Is an internal only change not relevant to end users.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant