Skip to content

Commit 29d666e

Browse files
authored
Add parallel task execution cookbook guide (#693)
1 parent f1f836a commit 29d666e

3 files changed

Lines changed: 121 additions & 0 deletions

File tree

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
---
2+
title: "Parallel Task Execution"
3+
sidebarTitle: "Parallel Task Execution"
4+
description: "Execute multiple tasks in parallel on a single profiled browser session."
5+
---
6+
7+
Use one profiled browser when several tasks need the same browser identity at the same time. Give each task a named [Playwright executor](/browsers/playwright-execution#executors), so each task gets its own tab while tasks on different executors run concurrently.
8+
9+
This guide is a concise overview of the full, runnable [parallel profile Playwright executors cookbook](https://github.com/kernel/cookbooks/pull/10), which this pattern is based on.
10+
11+
This pattern is particularly useful when the browser must save changes to a profile. All task tabs share the browser's cookies and persistent origin storage, and the browser writes the complete profile once when the run ends. If tasks only read profile state, you can use a separate read-only browser for each task instead; see [Sharing and concurrency](/browsers/profiles/concurrency).
12+
13+
## How it works
14+
15+
1. Start one browser with the shared profile and `save_changes: true`. Use `start_url` when you want to replace a tab restored from an earlier run.
16+
2. Bound the number of active tasks. A browser supports up to eight named executors, and memory, CPU, site behavior, or model limits can become constraints.
17+
3. Assign each task an executor name such as `lane-1` or `lane-2`. Send every Playwright call for each task to the same executor so its `page` remains bound to one tab.
18+
4. Keep each task's work in its executor-owned tab. Browser state persists between calls, but local JavaScript variables do not.
19+
5. Delete an executor and close its tab before reusing its name for another task. Delete the browser after all tasks finish so the profile is saved.
20+
21+
## Run tasks in parallel
22+
23+
Use `Promise.all` or `asyncio.gather` for concurrent tasks. A production worker pool can reuse a fixed set of executor names as tasks finish.
24+
25+
The examples assume that the `parallel-task-demo` profile already exists.
26+
27+
<CodeGroup>
28+
```typescript TypeScript
29+
import Kernel from '@onkernel/sdk';
30+
31+
const kernel = new Kernel();
32+
const browser = await kernel.browsers.create({
33+
stealth: true,
34+
start_url: 'https://www.kernel.sh/',
35+
profile: {
36+
name: 'parallel-task-demo',
37+
save_changes: true,
38+
},
39+
});
40+
41+
const runTask = (executor: string, url: string) =>
42+
kernel.browsers.playwright.execute(browser.session_id, {
43+
executor,
44+
code: `
45+
await page.goto(${JSON.stringify(url)});
46+
return { title: await page.title(), url: page.url() };
47+
`,
48+
timeout_sec: 10,
49+
});
50+
51+
const results = await Promise.all([
52+
runTask('lane-1', 'https://www.kernel.sh/docs/'),
53+
runTask('lane-2', 'https://www.kernel.sh/changelog'),
54+
]);
55+
56+
console.log(results.map((result) => result.result));
57+
58+
for (const executor of ['lane-1', 'lane-2']) {
59+
await kernel.browsers.playwright.executors.delete(executor, {
60+
id_or_name: browser.session_id,
61+
close_tab: true,
62+
});
63+
}
64+
await kernel.browsers.deleteByID(browser.session_id);
65+
```
66+
67+
```python Python
68+
import asyncio
69+
70+
from kernel import AsyncKernel
71+
72+
kernel = AsyncKernel()
73+
74+
75+
async def main():
76+
browser = await kernel.browsers.create(
77+
stealth=True,
78+
start_url="https://www.kernel.sh/",
79+
profile={
80+
"name": "parallel-task-demo",
81+
"save_changes": True,
82+
},
83+
)
84+
85+
async def run_task(executor: str, url: str):
86+
return await kernel.browsers.playwright.execute(
87+
browser.session_id,
88+
executor=executor,
89+
code=f"""
90+
await page.goto({url!r});
91+
return {{ title: await page.title(), url: page.url() }};
92+
""",
93+
timeout_sec=10,
94+
)
95+
96+
results = await asyncio.gather(
97+
run_task("lane-1", "https://www.kernel.sh/docs/"),
98+
run_task("lane-2", "https://www.kernel.sh/changelog"),
99+
)
100+
print([result.result for result in results])
101+
102+
for executor in ("lane-1", "lane-2"):
103+
await kernel.browsers.playwright.executors.delete(
104+
executor,
105+
id_or_name=browser.session_id,
106+
close_tab=True,
107+
)
108+
await kernel.browsers.delete_by_id(browser.session_id)
109+
110+
111+
asyncio.run(main())
112+
```
113+
</CodeGroup>
114+
115+
One task failing doesn't need to stop the other tasks, but the run must still clean up every executor and the browser. Publish task results as they finish when callers need progress before the full run completes.
116+
117+
For the full cookbook—including bounded concurrency, lane reuse, result publishing, and fault-tolerant cleanup—see the [parallel profile Playwright executors cookbook](https://github.com/kernel/cookbooks/pull/10).

‎cookbooks.mdx‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,10 @@ import { CookbookSearch } from '/snippets/cookbook-search.jsx';
3434
<div className="cookbook-tag">common patterns</div>
3535
Hold an agent while Kernel's captcha solver works, and tell it what actually happened.
3636
</Card>
37+
<Card title="Parallel Task Execution" img="/images/cookbooks/parallel-task-execution.png" href="/browsers/parallel-task-execution">
38+
<div className="cookbook-tag">common patterns</div>
39+
Execute multiple tasks in parallel on a single profiled browser session.
40+
</Card>
3741
</Columns>
3842

3943
## Harnesses & models
13.6 KB
Loading

0 commit comments

Comments
 (0)