diff --git a/endform-lambda-playwright-fast-start-plan.md b/endform-lambda-playwright-fast-start-plan.md new file mode 100644 index 0000000000000..4cfbd9fb37ff7 --- /dev/null +++ b/endform-lambda-playwright-fast-start-plan.md @@ -0,0 +1,240 @@ +# Endform Lambda Playwright Fast Start Plan + +## Context + +Endform runs Playwright tests remotely on AWS Lambda. Each invocation downloads only the exact test file and exact dependencies required for that run. There is no large test tree for Playwright to discover. + +Because Lambda execution is per-invocation, a long-lived daemon across invocations is not the primary design target. The goal is to start the required Node Playwright host and worker pair as early as possible within each invocation, overlapping their startup with Endform dependency download, proxy setup, and scratch preparation. + +## Key Implication + +Skipping broad Playwright discovery is likely not the first optimization to implement. + +Playwright discovery scans `project.testDir`, skips `node_modules`, and filters by extension, `testMatch`, and `testIgnore`. If the scratch filesystem contains only the selected test universe, discovery is bounded by a tiny directory tree. + +The larger initial costs are likely: + +- Node host startup and Playwright import graph. +- Worker process startup and worker import graph. +- Config load. +- Runner-side test file load. +- Worker-side test file load. +- Browser startup. +- Reporter finalization. + +## Target Per-Invocation Architecture + +1. Rust Lambda handler receives invocation. +2. Rust immediately starts `endform-playwright-host`. +3. Node host imports Playwright runner, worker host, config, reporter, and IPC internals. +4. Node host immediately pre-forks one Playwright worker. +5. Worker reaches `ready`, but does not receive `__init__` yet. +6. Rust downloads the exact test file and dependencies, prepares scratch, starts proxy, and writes/configures the Playwright config shim. +7. Rust sends run parameters to the Node host: + - scratch root + - config path + - selected project + - selected test file/test identity + - env overrides + - reporter transport/artifact settings +8. Node host loads config and runs mostly normal Playwright flow against the tiny scratch filesystem. +9. Dispatcher consumes the already-preforked worker instead of forking late. +10. Worker runs normal Playwright fixture/test/browser code. +11. Node host owns native reporters and streams completion data back to Rust. + +## Why Reporter Machinery Should Stay In The Host + +Native reporters should not run in the worker process. + +Playwright's current architecture assumes workers emit primitive IPC events and the runner reconstructs reporter-facing objects: + +- `Suite` +- `TestCase` +- `TestResult` +- `TestStep` +- attachments +- stdout/stderr attribution +- retries and final status + +Keeping reporters in the host preserves: + +- `onConfigure` +- `onBegin(rootSuite)` +- `onTestBegin` +- `onStepBegin` / `onStepEnd` +- `onTestEnd` +- `onError` +- `onEnd` +- `onExit` + +Running reporters in workers risks wrong object identity, duplicated output, artifact races, and incorrect stdout/stderr attribution. + +## Phase 1: Eager Boot With Minimal Playwright Changes + +Implement per-invocation startup overlap while preserving normal Playwright loading, dispatching, execution, and reporting. + +### Required Changes + +1. Add a custom `endform-playwright-host` entrypoint outside the normal Playwright CLI. +2. Split `ProcessHost.startRunner()` into two phases: + - fork child and wait for `ready` + - later send `__init__` +3. Add delayed initialization support to `WorkerHost`. +4. Allow `Dispatcher` to consume an already-preforked worker slot. + +### Expected Benefits + +- Worker Node startup overlaps with Endform dependency download. +- Worker module import graph overlaps with Endform setup. +- Host Node startup and Playwright imports overlap with Endform setup. +- Native Playwright reporting remains intact. +- Test execution remains close to upstream Playwright. + +### Expected Fork Surface + +Rough estimate: 300-700 LoC total, depending on how isolated the host entrypoint is. + +The most important Playwright-internal patch is the fork/init split in `ProcessHost` and delayed-init worker support. + +### Implemented Internal Shape + +The first implementation exposes this internal flow: + +```ts +const worker = new WorkerHost(0); +await worker.prefork(); + +// Later, after scratch/config/test files are ready: +await testRunner.runTests(reporter, { + locations: [selectedTestFile], + projects: [selectedProjectName], + preforkedWorkers: [worker], +}); +``` + +The dispatcher initializes the preforked worker with the real test group, serialized config, output directory, pause flags, and per-run env before sending `runTestGroup`. + +## Phase 2: Measure Before Skipping Discovery + +After Phase 1, instrument timings for: + +- Node host startup/imports. +- Worker fork-to-ready. +- Config load. +- Tiny-tree file collection. +- Runner-side test file load. +- Worker init. +- Worker-side test file load. +- Browser launch/context/page setup. +- Reporter `onEnd`/artifact finalization. + +If tiny-tree collection is negligible, do not patch discovery. Spend effort on worker-side test resolution, browser prewarm, or reporter finalization instead. + +## Phase 3: Avoid Double Test File Load + +Normal Playwright loads the selected file twice: + +1. Loader/runner loads it to build suite, test ids, groups, and reporter-facing objects. +2. Worker loads it again to execute. + +Since Endform already downloads only the exact selected test universe, removing this duplicate load may matter more than skipping discovery. + +### Safer Option: Host-Loaded Known File + +Keep runner-side file loading and suite construction, but provide a fast host command that runs against the tiny scratch tree and selected project/test params. + +Benefits: + +- Minimal semantic risk. +- Native reporters remain unchanged. +- Existing `Dispatcher` and `WorkerMain.runTestGroup()` remain mostly unchanged. + +Cost: + +- Worker still loads the file again. + +### Faster Option: Worker-Resolved Known Test + +Add a worker IPC method such as `prepareKnownTest`. + +Flow: + +1. Host initializes preforked worker with config/project/env. +2. Host sends `{ file, project, titlePath or test selector, repeatEachIndex }`. +3. Worker loads the selected file. +4. Worker binds the file suite to the project. +5. Worker builds fixture pools. +6. Worker finds the selected test. +7. Worker returns serialized suite/test metadata to the host. +8. Host reconstructs reporter-facing `Suite`/`TestCase` objects. +9. Host sends normal `runTestGroup` to the same worker. +10. Worker should reuse its already-loaded file suite via `testLoader` cache. + +Benefits: + +- Moves test loading to the worker. +- Can avoid the runner-side test file import. +- Keeps native reporters in host. +- Keeps actual test execution close to native worker code. + +Costs and risks: + +- Larger Playwright fork. +- Host must synthesize a correct reporter-facing suite from worker metadata. +- Duplicate titles need strict handling. +- Worker reuse must remain guarded by project/repeat/worker fixture compatibility. + +## Phase 4: Browser Prewarm + +If measurement shows browser startup dominates, add optional browser prewarm. + +Potential levels: + +- Start browser server while dependencies download, then connect from Playwright. +- Keep browser process alive for the duration of the invocation. +- Create a fresh browser context per test to preserve isolation. +- Only consider page/context reuse for explicitly safe tests. + +Fresh context reuse is the likely acceptable semantic boundary. Page reuse is a much larger compatibility tradeoff. + +## Phase 5: Reporter Finalization Optimization + +If blob reporter or artifact finalization is material on the critical path: + +- Use a lightweight Endform reporter for pass/fail and completion data. +- Enable blob only when requested. +- Defer blob/archive/upload work out of the critical completion path. +- Use host-side reporter events as the source of completion data. + +Do not move blob/native reporter finalization into the worker. + +## Recommended Implementation Order + +1. Build `endform-playwright-host` with normal Playwright internals. +2. Add per-invocation host startup from Rust before dependency download completes. +3. Split `ProcessHost` fork and init. +4. Add preforked delayed-init worker support. +5. Route normal Playwright run through the preforked worker against the tiny scratch tree. +6. Add timing instrumentation. +7. Decide whether discovery is worth patching. It likely is not if the scratch tree is tiny. +8. Prototype `prepareKnownTest` to avoid double test-file load. +9. Add browser prewarm if measurements justify it. +10. Optimize reporter finalization only if it appears in the critical path. + +## Maintained Fork Strategy + +Keep the fork surface focused on stable seams: + +- process host fork/init split +- worker host delayed init +- dispatcher preforked-worker consumption +- optional worker `prepareKnownTest` IPC + +Avoid rewriting: + +- fixture execution +- reporter lifecycle +- test result construction unless strictly necessary +- browser fixtures unless browser prewarm measurements justify it + +This should make the fork easier to carry across Playwright versions while still attacking the real Lambda cold-start bottleneck. diff --git a/packages/playwright/src/common/ipc.ts b/packages/playwright/src/common/ipc.ts index 3fdaeccfcb6d5..6f6ac2c38d8dd 100644 --- a/packages/playwright/src/common/ipc.ts +++ b/packages/playwright/src/common/ipc.ts @@ -66,6 +66,7 @@ export type WorkerInitParams = { projectId: string; config: SerializedConfig; artifactsDir: string; + extraEnv?: Record; pauseOnError: boolean; pauseAtEnd: boolean; }; diff --git a/packages/playwright/src/runner/dispatcher.ts b/packages/playwright/src/runner/dispatcher.ts index 29f4f4e84eb1a..10e98fbfc9e20 100644 --- a/packages/playwright/src/runner/dispatcher.ts +++ b/packages/playwright/src/runner/dispatcher.ts @@ -46,8 +46,9 @@ export class Dispatcher { private _extraEnvByProjectId: EnvByProjectId = new Map(); private _producedEnvByProjectId: EnvByProjectId = new Map(); - constructor(testRun: TestRun) { + constructor(testRun: TestRun, preforkedWorkers: WorkerHost[] = []) { this._testRun = testRun; + this._workerSlots = preforkedWorkers.map(worker => ({ worker })); for (const project of testRun.config.projects) { if (project.workers) this._workerLimitPerProjectId.set(project.id, project.workers); @@ -117,7 +118,7 @@ export class Dispatcher { let worker = this._workerSlots[index].worker; // 1. Restart the worker if it has the wrong hash or is being stopped already. - if (worker && (worker.hash() !== job.workerHash || worker.didSendStop())) { + if (worker && (worker.didSendStop() || (worker.isInitialized() && worker.hash() !== job.workerHash))) { await worker.stop(); worker = undefined; if (this._isStopped) // Check stopped signal after async hop. @@ -133,6 +134,12 @@ export class Dispatcher { startError = await worker.start(); if (this._isStopped) // Check stopped signal after async hop. return; + } else if (!worker.isInitialized()) { + this._initializeWorker(worker, job, index, ipc.serializeConfig(this._testRun.config, true)); + worker.on('exit', () => this._workerSlots[index].worker = undefined); + startError = await worker.start(); + if (this._isStopped) // Check stopped signal after async hop. + return; } // 3. Finally, run some tests in the worker! Or fail all of them because of startup error... @@ -173,12 +180,15 @@ export class Dispatcher { } private _isWorkerRedundant(worker: WorkerHost) { + const hash = worker.hash(); + if (!hash) + return false; let workersWithSameHash = 0; for (const slot of this._workerSlots) { - if (slot.worker && !slot.worker.didSendStop() && slot.worker.hash() === worker.hash()) + if (slot.worker && !slot.worker.didSendStop() && slot.worker.hash() === hash) workersWithSameHash++; } - return workersWithSameHash > this._queuedOrRunningHashCount.get(worker.hash())!; + return workersWithSameHash > this._queuedOrRunningHashCount.get(hash)!; } private _updateCounterForWorkerHash(hash: string, delta: number) { @@ -191,12 +201,15 @@ export class Dispatcher { for (const group of testGroups) this._updateCounterForWorkerHash(group.workerHash, +1); this._isStopped = false; - this._workerSlots = []; + const preforkedSlots = this._workerSlots + .filter(slot => slot.worker && !slot.worker.isInitialized() && !slot.worker.didSendStop()) + .slice(0, this._testRun.config.config.workers); + this._workerSlots = preforkedSlots; // 0. Stop right away if we have reached max failures. if (this._testRun.hasReachedMaxFailures()) void this.stop(); // 1. Allocate workers. - for (let i = 0; i < this._testRun.config.config.workers; i++) + for (let i = this._workerSlots.length; i < this._testRun.config.config.workers; i++) this._workerSlots.push({}); // 2. Schedule enough jobs. for (let i = 0; i < this._workerSlots.length; i++) @@ -207,13 +220,14 @@ export class Dispatcher { await this._finished; } - _createWorker(testGroup: TestGroup, parallelIndex: number, loaderData: ipc.SerializedConfig) { + _initializeWorker(worker: WorkerHost, testGroup: TestGroup, parallelIndex: number, loaderData: ipc.SerializedConfig) { const project = this._testRun.config.projects.find(p => p.id === testGroup.projectId)!; const pauseAtEnd = this._testRun.topLevelProjects.includes(project) && !!this._testRun.options.pauseAtEnd; - const worker = new WorkerHost(testGroup, { + worker.initialize({ + testGroup, parallelIndex, config: loaderData, - extraEnv: this._extraEnvByProjectId.get(testGroup.projectId) || {}, + extraEnv: { ...this._testRun.options.workerEnv, ...this._extraEnvByProjectId.get(testGroup.projectId) }, outputDir: project.project.outputDir, pauseOnError: !!this._testRun.options.pauseOnError, pauseAtEnd, @@ -260,6 +274,11 @@ export class Dispatcher { const producedEnv = this._producedEnvByProjectId.get(testGroup.projectId) || {}; this._producedEnvByProjectId.set(testGroup.projectId, { ...producedEnv, ...worker.producedEnv() }); }); + } + + _createWorker(testGroup: TestGroup, parallelIndex: number, loaderData: ipc.SerializedConfig) { + const worker = new WorkerHost(parallelIndex); + this._initializeWorker(worker, testGroup, parallelIndex, loaderData); return worker; } diff --git a/packages/playwright/src/runner/index.ts b/packages/playwright/src/runner/index.ts index deca5ea33718e..9881886fba753 100644 --- a/packages/playwright/src/runner/index.ts +++ b/packages/playwright/src/runner/index.ts @@ -20,6 +20,7 @@ // imports. export * as testRunner from './testRunner'; +export * as workerHost from './workerHost'; export * as testServer from './testServer'; export * as watchMode from './watchMode'; export * as projectUtils from './projectUtils'; diff --git a/packages/playwright/src/runner/processHost.ts b/packages/playwright/src/runner/processHost.ts index 5c0257933f8a8..e3d00ea537e57 100644 --- a/packages/playwright/src/runner/processHost.ts +++ b/packages/playwright/src/runner/processHost.ts @@ -32,6 +32,7 @@ export type ProcessExitData = { export class ProcessHost extends EventEmitter { private process: child_process.ChildProcess | undefined; + private _didSendInit = false; private _didSendStop = false; private _processDidExit = false; private _didExitAndRanOnExit = false; @@ -50,7 +51,7 @@ export class ProcessHost extends EventEmitter { this._extraEnv = env; } - async startRunner(runnerParams: any, options: { onStdOut?: (chunk: Buffer | string) => void, onStdErr?: (chunk: Buffer | string) => void } = {}): Promise { + async preforkRunner(options: { onStdOut?: (chunk: Buffer | string) => void, onStdErr?: (chunk: Buffer | string) => void } = {}): Promise { assert(!this.process, 'Internal error: starting the same process twice'); this.process = child_process.fork(this._entryScript, { // Note: we pass detached:false, so that workers are in the same process group. @@ -125,6 +126,12 @@ export class ProcessHost extends EventEmitter { if (error) return error; + } + + initRunner(runnerParams: any) { + assert(this.process, 'Internal error: initializing a process before it starts'); + assert(!this._didSendInit, 'Internal error: initializing the same process twice'); + this._didSendInit = true; const processParams: ipc.ProcessInitParams = { processName: this._processName, @@ -139,6 +146,17 @@ export class ProcessHost extends EventEmitter { }); } + async startRunner(runnerParams: any, options: { onStdOut?: (chunk: Buffer | string) => void, onStdErr?: (chunk: Buffer | string) => void } = {}): Promise { + const error = await this.preforkRunner(options); + if (error) + return error; + this.initRunner(runnerParams); + } + + hasProcess() { + return !!this.process; + } + sendMessage(message: { method: string, params?: any }) { const id = ++this._lastMessageId; this.send({ diff --git a/packages/playwright/src/runner/tasks.ts b/packages/playwright/src/runner/tasks.ts index 50b6951177ffd..0135c3dd4f99b 100644 --- a/packages/playwright/src/runner/tasks.ts +++ b/packages/playwright/src/runner/tasks.ts @@ -35,6 +35,7 @@ import { createTitleMatcher, forceRegExp, removeDirAndLogToConsole } from '../ut import type { TestGroup } from '../runner/testGroups'; import type { EnvByProjectId } from './dispatcher'; +import type { WorkerHost } from './workerHost'; import type { TestRunnerPluginRegistration } from '../plugins'; import type { Task } from './taskRunner'; import type { ReporterDescription } from '../../types/test'; @@ -74,6 +75,8 @@ export type TestRunOptions = { preserveOutputDir?: boolean; additionalReporters?: ReporterDescription[]; shardWeights?: number[]; + preforkedWorkers?: WorkerHost[]; + workerEnv?: Record; }; export type TestPausedParams = { @@ -423,7 +426,7 @@ function createPhasesTask(): Task { processed.add(project); if (phaseProjects.length) { let testGroupsInPhase = 0; - const phase: Phase = { dispatcher: new Dispatcher(testRun), projects: [] }; + const phase: Phase = { dispatcher: new Dispatcher(testRun, testRun.options.preforkedWorkers), projects: [] }; testRun.phases.push(phase); for (const project of phaseProjects) { const projectSuite = projectToSuite.get(project)!; diff --git a/packages/playwright/src/runner/testRunner.ts b/packages/playwright/src/runner/testRunner.ts index 3f1f250dee0e6..51aa6ab6c0b7b 100644 --- a/packages/playwright/src/runner/testRunner.ts +++ b/packages/playwright/src/runner/testRunner.ts @@ -42,6 +42,7 @@ import type * as reporterTypes from '../../types/testReporter'; import type { ConfigLocation } from '../common'; import type { TestRunnerPluginRegistration } from '../plugins'; import type { AnyReporter } from '../reporters/reporterV2'; +import type { WorkerHost } from './workerHost'; export const TestRunnerEvent = { TestFilesChanged: 'testFilesChanged', @@ -84,6 +85,8 @@ export type RunTestsParams = { doNotRunDepsOutsideProjectFilter?: boolean; disableConfigReporters?: boolean; failOnLoadErrors?: boolean; + preforkedWorkers?: WorkerHost[]; + workerEnv?: Record; }; export type FullResultStatus = reporterTypes.FullResult['status']; @@ -327,6 +330,8 @@ export class TestRunner extends EventEmitter { pauseAtEnd: params.pauseAtEnd, preserveOutputDir: true, onTestPaused: params => this.emit(TestRunnerEvent.TestPaused, params), + preforkedWorkers: params.preforkedWorkers, + workerEnv: params.workerEnv, }; const configReporters = params.disableConfigReporters ? [] : await createReporters(config, 'test', undefined, options); diff --git a/packages/playwright/src/runner/workerHost.ts b/packages/playwright/src/runner/workerHost.ts index 2e1781c2d0762..3b9d6236a7f2a 100644 --- a/packages/playwright/src/runner/workerHost.ts +++ b/packages/playwright/src/runner/workerHost.ts @@ -17,6 +17,7 @@ import fs from 'fs'; import path from 'path'; +import { assert } from '@isomorphic/assert'; import { removeFolders } from '@utils/fileUtils'; import { ProcessHost } from './processHost'; @@ -37,46 +38,81 @@ type WorkerHostOptions = { pauseAtEnd: boolean; }; +type WorkerHostInitOptions = WorkerHostOptions & { + testGroup: TestGroup; +}; + export class WorkerHost extends ProcessHost { - readonly parallelIndex: number; readonly workerIndex: number; - private _hash: string; - private _params: ipc.WorkerInitParams; + private _parallelIndex: number; + private _hash: string | undefined; + private _params: ipc.WorkerInitParams | undefined; private _didFail = false; - constructor(testGroup: TestGroup, options: WorkerHostOptions) { + constructor(testGroup: TestGroup, options: WorkerHostOptions); + constructor(parallelIndex: number); + constructor(testGroupOrParallelIndex: TestGroup | number, options?: WorkerHostOptions) { const workerIndex = lastWorkerIndex++; + const extraEnv = options?.extraEnv || {}; super(require.resolve('../worker/workerProcessEntry.js'), `worker-${workerIndex}`, { - ...options.extraEnv, + ...extraEnv, FORCE_COLOR: '1', DEBUG_COLORS: process.env.DEBUG_COLORS === undefined ? '1' : process.env.DEBUG_COLORS, }); this.workerIndex = workerIndex; - this.parallelIndex = options.parallelIndex; - this._hash = testGroup.workerHash; + this._parallelIndex = typeof testGroupOrParallelIndex === 'number' ? testGroupOrParallelIndex : options!.parallelIndex; + + if (typeof testGroupOrParallelIndex !== 'number') + this.initialize({ ...options!, testGroup: testGroupOrParallelIndex }); + } + + get parallelIndex() { + return this._parallelIndex; + } + + initialize(options: WorkerHostInitOptions) { + if (this._params) + return; + + this._parallelIndex = options.parallelIndex; + this._hash = options.testGroup.workerHash; this._params = { workerIndex: this.workerIndex, parallelIndex: options.parallelIndex, - repeatEachIndex: testGroup.repeatEachIndex, - projectId: testGroup.projectId, + repeatEachIndex: options.testGroup.repeatEachIndex, + projectId: options.testGroup.projectId, config: options.config, - artifactsDir: path.join(options.outputDir, artifactsFolderName(workerIndex)), + artifactsDir: path.join(options.outputDir, artifactsFolderName(this.workerIndex)), + extraEnv: options.extraEnv, pauseOnError: options.pauseOnError, pauseAtEnd: options.pauseAtEnd, }; } async start() { + assert(this._params, 'Internal error: starting a worker before it is initialized'); await fs.promises.mkdir(this._params.artifactsDir, { recursive: true }); + if (this.hasProcess()) { + this.initRunner(this._params); + return; + } return await this.startRunner(this._params, { onStdOut: chunk => this.emit('stdOut', ipc.stdioChunkToParams(chunk)), onStdErr: chunk => this.emit('stdErr', ipc.stdioChunkToParams(chunk)), }); } + async prefork() { + return await this.preforkRunner({ + onStdOut: chunk => this.emit('stdOut', ipc.stdioChunkToParams(chunk)), + onStdErr: chunk => this.emit('stdErr', ipc.stdioChunkToParams(chunk)), + }); + } + override async onExit() { - await removeFolders([this._params.artifactsDir]); + if (this._params) + await removeFolders([this._params.artifactsDir]); } override async stop(didFail?: boolean) { @@ -101,8 +137,12 @@ export class WorkerHost extends ProcessHost { return this._hash; } + isInitialized() { + return !!this._params; + } + projectId() { - return this._params.projectId; + return this._params?.projectId; } didFail() { diff --git a/packages/playwright/src/worker/workerMain.ts b/packages/playwright/src/worker/workerMain.ts index 22e8f46451639..409f94c2264c8 100644 --- a/packages/playwright/src/worker/workerMain.ts +++ b/packages/playwright/src/worker/workerMain.ts @@ -64,6 +64,12 @@ export class WorkerMain extends ProcessRunner { constructor(params: ipc.WorkerInitParams) { super(); + for (const [key, value] of Object.entries(params.extraEnv || {})) { + if (value === undefined) + delete process.env[key]; + else + process.env[key] = value; + } process.env.TEST_WORKER_INDEX = String(params.workerIndex); process.env.TEST_PARALLEL_INDEX = String(params.parallelIndex); globals.setIsWorkerProcess(); diff --git a/tests/playwright-test/prefork-worker.spec.ts b/tests/playwright-test/prefork-worker.spec.ts new file mode 100644 index 0000000000000..39ccf58dababa --- /dev/null +++ b/tests/playwright-test/prefork-worker.spec.ts @@ -0,0 +1,31 @@ +/** + * Copyright Microsoft Corporation. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import path from 'path'; + +import { test, expect } from './playwright-test-fixtures'; + +test('preforked worker can be initialized later and run a test', async ({ childProcess }, testInfo) => { + const repoRoot = path.resolve(__dirname, '..', '..'); + const script = path.join(repoRoot, 'utils', 'endform', 'prefork-worker-smoke.js'); + const proc = childProcess({ + command: ['node', script, '--playwright-root', repoRoot], + cwd: testInfo.outputPath(), + }); + const { exitCode } = await proc.exited; + expect(proc.output).toContain('PREFORK_WORKER_SMOKE_OK'); + expect(exitCode).toBe(0); +}); diff --git a/utils/endform/prefork-worker-smoke.js b/utils/endform/prefork-worker-smoke.js new file mode 100644 index 0000000000000..87b731e2c5a29 --- /dev/null +++ b/utils/endform/prefork-worker-smoke.js @@ -0,0 +1,235 @@ +#!/usr/bin/env node +/** + * Copyright Microsoft Corporation. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +const fs = require('fs'); +const os = require('os'); +const path = require('path'); + +function parseArgs() { + const result = { + playwrightRoot: process.cwd(), + }; + for (let i = 2; i < process.argv.length; i++) { + const arg = process.argv[i]; + if (arg === '--playwright-root') { + result.playwrightRoot = path.resolve(process.argv[++i]); + } else if (arg.startsWith('--playwright-root=')) { + result.playwrightRoot = path.resolve(arg.substring('--playwright-root='.length)); + } else { + throw new Error(`Unknown argument: ${arg}`); + } + } + return result; +} + +function requireFirst(candidates, label) { + const errors = []; + for (const candidate of candidates) { + try { + if (fs.existsSync(candidate)) + return require(candidate); + } catch (e) { + errors.push(`${candidate}: ${e.message}`); + } + } + throw new Error([ + `Could not load ${label}.`, + `Tried:`, + ...candidates.map(candidate => ` ${candidate}`), + `Build Playwright first, or pass --playwright-root to a built patched checkout/package.`, + ...errors.map(error => ` ${error}`), + ].join('\n')); +} + +function internals(root) { + const candidates = relative => [ + path.join(root, 'packages', 'playwright', 'lib', relative), + path.join(root, 'lib', relative), + path.join(root, 'node_modules', 'playwright', 'lib', relative), + ]; + const runnerIndex = requireFirst(candidates(path.join('runner', 'index.js')), 'runner internals'); + if (runnerIndex.testRunner && runnerIndex.workerHost) { + return { + testRunner: runnerIndex.testRunner, + workerHost: runnerIndex.workerHost, + }; + } + return { + testRunner: requireFirst(candidates(path.join('runner', 'testRunner.js')), 'TestRunner internals'), + workerHost: requireFirst(candidates(path.join('runner', 'workerHost.js')), 'WorkerHost internals'), + }; +} + +async function mkdirp(filePath) { + await fs.promises.mkdir(path.dirname(filePath), { recursive: true }); +} + +async function writeFile(filePath, text) { + await mkdirp(filePath); + await fs.promises.writeFile(filePath, text); +} + +function assert(condition, message) { + if (!condition) + throw new Error(message); +} + +class SmokeReporter { + constructor() { + this.events = []; + this.stdoutWithTest = false; + this.stderrWithTest = false; + this.attachments = []; + this.finalStatus = undefined; + } + + version() { + return 'v2'; + } + + onConfigure() { + this.events.push('onConfigure'); + } + + onBegin(suite) { + this.events.push('onBegin'); + this.testCount = suite.allTests().length; + } + + onTestBegin(test, result) { + this.events.push('onTestBegin'); + this.testTitle = test.title; + this.workerIndex = result.workerIndex; + } + + onStepBegin(test, result, step) { + this.events.push('onStepBegin:' + step.title); + } + + onStepEnd(test, result, step) { + this.events.push('onStepEnd:' + step.title); + } + + onStdOut(chunk, test, result) { + const text = String(chunk); + if (text.includes('stdout-from-prefork-test')) { + this.events.push('onStdOut'); + this.stdoutWithTest = !!test && !!result; + } + } + + onStdErr(chunk, test, result) { + const text = String(chunk); + if (text.includes('stderr-from-prefork-test')) { + this.events.push('onStdErr'); + this.stderrWithTest = !!test && !!result; + } + } + + onTestEnd(test, result) { + this.events.push('onTestEnd:' + result.status); + this.attachments = result.attachments.map(a => ({ name: a.name, contentType: a.contentType, body: a.body && a.body.toString() })); + } + + onEnd(result) { + this.events.push('onEnd:' + result.status); + this.finalStatus = result.status; + } + + onExit() { + this.events.push('onExit'); + } +} + +async function main() { + const { playwrightRoot } = parseArgs(); + const { testRunner, workerHost } = internals(playwrightRoot); + const { TestRunner } = testRunner; + const { WorkerHost } = workerHost; + + const tmpDir = await fs.promises.mkdtemp(path.join(os.tmpdir(), 'pw-prefork-smoke-')); + const configFile = path.join(tmpDir, 'playwright.config.js'); + const testFile = path.join(tmpDir, 'prefork-smoke.spec.js'); + const nodeModulesTest = path.join(tmpDir, 'node_modules', '@playwright', 'test', 'index.js'); + const playwrightTestEntry = path.join(playwrightRoot, 'packages', 'playwright', 'test.js'); + const fallbackPlaywrightTestEntry = path.join(playwrightRoot, 'test.js'); + const testEntry = fs.existsSync(playwrightTestEntry) ? playwrightTestEntry : fallbackPlaywrightTestEntry; + + const worker = new WorkerHost(0); + const preforkError = await worker.prefork(); + assert(!preforkError, `Prefork failed: ${JSON.stringify(preforkError)}`); + + const expectedWorkerIndex = String(worker.workerIndex); + + await writeFile(nodeModulesTest, `module.exports = require(${JSON.stringify(testEntry)});\n`); + await writeFile(configFile, ` +module.exports = { + testDir: ${JSON.stringify(tmpDir)}, + workers: 1, + reporter: 'null', +}; +`); + await writeFile(testFile, ` +const { test, expect } = require('@playwright/test'); + +test('prefork smoke', async ({}, testInfo) => { + expect(process.env.ENDFORM_LATE_ENV).toBe('from-init'); + expect(process.env.TEST_WORKER_INDEX).toBe(process.env.EXPECTED_WORKER_INDEX); + console.log('stdout-from-prefork-test'); + console.error('stderr-from-prefork-test'); + await test.step('prefork step', async () => { + expect(1 + 1).toBe(2); + }); + await testInfo.attach('prefork-attachment', { + body: Buffer.from('attachment-body'), + contentType: 'text/plain', + }); +}); +`); + + const reporter = new SmokeReporter(); + const runner = new TestRunner({ configDir: tmpDir, resolvedConfigFile: configFile }, { workers: 1 }); + await runner.initialize({}); + const { status } = await runner.runTests(reporter, { + locations: [testFile], + projects: [], + preforkedWorkers: [worker], + workerEnv: { ENDFORM_LATE_ENV: 'from-init', EXPECTED_WORKER_INDEX: expectedWorkerIndex }, + }); + await runner.stop(); + + assert(status === 'passed', `Expected run status passed, got ${status}. Events: ${reporter.events.join(', ')}`); + assert(reporter.finalStatus === 'passed', `Expected reporter final status passed, got ${reporter.finalStatus}`); + assert(reporter.testCount === 1, `Expected one test in onBegin, got ${reporter.testCount}`); + assert(reporter.testTitle === 'prefork smoke', `Expected prefork smoke test, got ${reporter.testTitle}`); + assert(reporter.workerIndex === worker.workerIndex, `Expected reporter workerIndex ${worker.workerIndex}, got ${reporter.workerIndex}`); + assert(reporter.stdoutWithTest, 'Expected stdout to be attributed to test/result'); + assert(reporter.stderrWithTest, 'Expected stderr to be attributed to test/result'); + assert(reporter.events.includes('onStepBegin:prefork step'), `Missing step begin. Events: ${reporter.events.join(', ')}`); + assert(reporter.events.includes('onStepEnd:prefork step'), `Missing step end. Events: ${reporter.events.join(', ')}`); + assert(reporter.attachments.some(a => a.name === 'prefork-attachment' && a.body === 'attachment-body'), `Missing attachment. Attachments: ${JSON.stringify(reporter.attachments)}`); + for (const event of ['onConfigure', 'onBegin', 'onTestBegin', 'onStdOut', 'onStdErr', 'onTestEnd:passed', 'onEnd:passed', 'onExit']) + assert(reporter.events.includes(event), `Missing reporter event ${event}. Events: ${reporter.events.join(', ')}`); + + console.log('PREFORK_WORKER_SMOKE_OK'); + console.log(JSON.stringify({ workerIndex: worker.workerIndex, events: reporter.events })); +} + +main().catch(e => { + console.error(e.stack || e.message || String(e)); + process.exit(1); +});