Skip to content

Commit fec0ea8

Browse files
l2yshoclaude
andauthored
feat: add GitLab and Bitbucket to --source (#1356)
> [!NOTE] > **TL;DR** — adds `--source gitlab` and `--source bitbucket` alongside `github`. Both tested with personal and repo token. Follows #1348. Part of #1153. ## No longer blocked | PR | What it gives us | State | |---|---|---| | [apify-core#30208](apify/apify-core#30208) | GitLab/Bitbucket branches for `/integrations/git`, `create-repo` and `register-deployment-key` | merged, live | | [apify-core#30279](apify/apify-core#30279) | the Console page that starts the OAuth connect for the CLI | merged, live | ## The shape change GitHub is one connection addressed by the fixed id `github-app`. GitLab and Bitbucket are a `UserIntegration` **per connected account**, each addressed by its own id and reporting its own workspaces — one GitLab account can carry several namespaces, including subgroups. So: - integrations are matched by `provider`, not by a hardcoded id - every workspace from every matching integration is merged into one list, each keeping the `providerId` that `create-repo` and `register-deployment-key` address it by Without that, a second connected account would have been silently ignored. ## `--git-repo` matches labels GitLab identifies a namespace by a number, so `--git-repo 4711/my-scraper` is what the API wants and nobody would type it. `--git-repo my-group/sub` resolves through the label. Since a GitLab label is a `full_path` and contains slashes of its own, the flag splits at the **last** slash — `my-group/sub/my-scraper` is workspace `my-group/sub`, repository `my-scraper`. The `workspaces` array in `--json` reports labels for the same reason. ## Connecting GitHub's authorize URL is two public query params, so the CLI builds it. GitLab and Bitbucket mint a CSRF `state` server-side and need a `UserIntegration` row to exist first — neither is something the CLI can produce. So `connectViaConsole` opens the Console integrations page with the `localCli*` deep-link params and waits on a loopback server, exactly as `apify login` does. The callback is not treated as the only signal. The CLI polls `GET /integrations/git` alongside it and takes whichever lands first, because a user who connects on the Console page by hand never triggers the callback — and that is a real path, not a hypothetical: it is what happens when the hand-off does not fire. An `AbortSignal` releases whichever wait loses, so the loopback server never holds the command open. ## Which Apify account the connection lands on #1371 fixed this for GitHub. These two providers reach Console a different way — the CLI navigates to the integrations page rather than sending a `routePrefix` through an OAuth callback — so the bug survived. Console creates the `UserIntegration` for the account its own route names: `useEffectiveUserSession` resolves the session token from `viewedUserId`, which comes only from the path. Where the path says nothing, Console redirects to whichever account the browser was last on — `/settings/*` is `ROUTE_KEYS.ACCOUNT`, which is in `REMEMBER_LAST_ACCOUNT_ROUTE_NAMES`. `GET /integrations/git` is scoped to the token owner, so an integration created elsewhere is invisible to the CLI, which then waits out its full timeout and reports the wrong cause. It fails in **both** directions, unlike GitHub: | CLI token | Browser last used | Integration created on | Result | |---|---|---|---| | organization | personal | personal | CLI never sees it | | personal | organization | that organization | CLI never sees it | The account now rides in the path. An explicit `/organization/<id>` route is never redirected away (`if (!isUserRoute) return` in `useRedirectToLastAccount`), so it pins both cases. `getConsoleRoutePrefix` moved to `console-url.ts` beside `getConsoleIntegrationsUrl`, which now takes the account — GitHub's query param and these providers' path prefix share one source. The account reaches the page the CLI opens, the connect URL printed for agents and CI, and the `noWorkspace` next steps. ## Waiting for the connection Two defects in the race, both found in review: - **A failed listing read ended the wait.** `pollForWorkspaces` had no guard around `load()`, so one 5xx or a dropped connection rejected out of the race and skipped `controller.abort()` — the only thing that closes the loopback server and clears its timer. `apify create` printed its next steps and then sat there for the remaining five minutes. The poll now survives a failed read, and the abort runs in a `finally`. - **A successful callback got one read, not a poll.** Console reports the connection as soon as the integration row exists, which can beat its workspaces into the listing. That single empty read reported a connected account as a missing one. It now waits the way the authorize branch already does. ## A failed clone no longer costs the Actor The clone is the only step needing the user's own provider credentials, and the only purely local one. It used to abort the run, leaving a repository with no Actor and a name that the next attempt could not reuse. Now the error is held, the Actor and the deploy key are still created, and the stop reports the missing local copy with a `git clone` to run — over HTTPS or SSH, whichever the user has credentials for. Bitbucket also returns its clone URL with the account name embedded, which makes git ask only for a password and defeats every token; the CLI strips the userinfo before cloning or printing it. ## Reviewer notes **This branch depends on Console's routing, and none of it is a contract.** Three assumptions, all in apify-core, none covered by a test that would fail if someone changed them: 1. every USER route gets an `/organization/:viewedUserId` twin (`route_definitions.ts`) 2. an explicit organization route is never redirected to the last-used account (`if (!isUserRoute) return`) 3. the path is what selects the effective account (`useEffectiveUserSession`) A regression in any of them is silent — no error, no 404, the connection just lands on the wrong account and the user is told Apify has no GitLab account. Cheapest mitigation is a test on the apify-core side; the real one is the first follow-up below. **GitHub's `provider` value is confirmed, not assumed.** `src/api/src/routes/integrations/git_providers.ts` builds the GitHub entry as `{ id: GITHUB_APP_PROVIDER_ID, provider: 'github' }` with `GITHUB_APP_PROVIDER_ID = 'github-app'`, so matching by `provider` is correct for all three. ## Verification Verified live against production, on a personal login: - **GitLab** — full chain: connect, create-repo, clone, Actor, deploy key. - **Bitbucket** — connect through the Console hand-off, create-repo, Actor, deploy key. The clone was confirmed separately with a working credential; the 403 first seen there was a workspace the account had no create rights in, not a CLI or scope fault. The account routing and the two waiting fixes landed after those runs and are covered by unit tests only — an organization login has not been exercised end to end. 64 tests across `gitSource.test.ts` and `connectViaConsole.test.ts`. Full suite: 527 passed, 4 skipped. `lint`, `format`, `build` clean, `docs/reference.md` regenerated. ## Follow-ups, not in this PR - **Drop the loopback server.** Print the account-scoped Console URL, let the user connect in Console's normal UI, and rely on the poll the CLI already runs. Nothing changes in Console — the deep link just stops being sent, and the hook it triggers is never entered. Removes ~110 lines plus its test, and both defects above were in code this deletes. The cost is the `/exit` signal, which tells the CLI the user closed the tab; GitHub has never had it, so this is parity rather than a regression. Losing it means a user who gives up waits out the timeout. - **Have the CLI create the integration.** A public `POST /v2/integrations/git` returning `{ integrationId, loginUrl }` would let the CLI start the flow under its own token. The account question then never gets asked, and all three Console-routing assumptions above go away. - **Clone over SSH when a key is available.** The recovery steps offer it, but the run itself always tries HTTPS, which fails for anyone who only uses SSH. - **Core-side, worth reporting:** a 403 from `create-repo` maps to "Reconnect your account", which cannot help when the cause is missing permission; and `shouldValidateScopes: false` lets a Bitbucket token without `repository:admin` connect cleanly and fail three steps later. - The Console hand-off scaffolding is duplicated between `connectViaConsole.ts` and `auth/login.ts`. Worth extracting next to `local-api-server.ts`, which also touches `edit-input-schema.ts`. Note the double `encodeURIComponent` on `localCliComputerName` in both — `searchParams.set` already encodes, so whoever extracts this has to keep the quirk or fix both sides at once. - Cloning a private repository still needs the user's own provider credentials. The deploy key is Apify's and only lets the build worker read it. --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1 parent 87dd140 commit fec0ea8

8 files changed

Lines changed: 844 additions & 146 deletions

File tree

‎docs/reference.md‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -280,7 +280,8 @@ USAGE
280280
$ apify create [actorName] [--git-repo <value>] [--json]
281281
[-l javascript|js|typescript|ts|python|py]
282282
[--omit-optional-deps] [--skip-dependency-install]
283-
[--skip-git-init] [--source apify|github] [-t <value>]
283+
[--skip-git-init] [--source apify|github|gitlab|bitbucket]
284+
[-t <value>]
284285
[-u web-scraper|ai-agent|data-pipeline|browser-automation]
285286
286287
ARGUMENTS
@@ -308,11 +309,11 @@ FLAGS
308309
--skip-git-init Skip initializing a git
309310
repository in the Actor directory.
310311
--source=<option> Where the Actor source
311-
code will live. With "github", Apify creates the
312-
repository on your connected GitHub account from
313-
the template, clones it here, and creates an Actor
314-
that builds from it.
315-
<options: apify|github>
312+
code will live. With a Git provider, Apify creates
313+
the repository on your connected account from the
314+
template, clones it here, and creates an Actor that
315+
builds from it.
316+
<options: apify|github|gitlab|bitbucket>
316317
-t, --template=<value> Template for the
317318
Actor. If not provided, the command will prompt for
318319
it. Visit

‎src/commands/auth/login.ts‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ import { cryptoRandomObjectId } from '@apify/utilities';
99

1010
import { ApifyCommand } from '../../lib/command-framework/apify-command.js';
1111
import { Flags } from '../../lib/command-framework/flags.js';
12-
import { getConsoleUrl } from '../../lib/console-url.js';
12+
import { getConsoleIntegrationsUrl, getConsoleUrl } from '../../lib/console-url.js';
1313
import { AUTH_FILE_PATH } from '../../lib/consts.js';
1414
import { getBackend } from '../../lib/credentials.js';
1515
import { updateUserId } from '../../lib/hooks/telemetry/useTelemetryState.js';
@@ -19,8 +19,6 @@ import { createLocalApiServer } from '../../lib/local-api-server.js';
1919
import { error, info, success } from '../../lib/outputs.js';
2020
import { getLocalUserInfo, getLoggedClient, tildify } from '../../lib/utils.js';
2121

22-
const CONSOLE_INTEGRATIONS_PATH = '/settings/integrations';
23-
2422
// When logging in against a local Console instance (local platform development), validate the token
2523
// against the local API rather than production.
2624
const LOCAL_API_BASE_URL = 'http://localhost:3333';
@@ -108,7 +106,7 @@ export class AuthLoginCommand extends ApifyCommand<typeof AuthLoginCommand> {
108106

109107
const consoleUrl = getConsoleUrl();
110108
const consoleOrigin = new URL(consoleUrl).origin;
111-
const consoleIntegrationsUrl = `${consoleUrl}${CONSOLE_INTEGRATIONS_PATH}`;
109+
const consoleIntegrationsUrl = getConsoleIntegrationsUrl();
112110

113111
let selectedMethod = method;
114112

‎src/commands/create.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -140,7 +140,7 @@ export class CreateCommand extends ApifyCommand<typeof CreateCommand> {
140140
}),
141141
source: Flags.string({
142142
description:
143-
'Where the Actor source code will live. With "github", Apify creates the repository on your connected GitHub account from the template, clones it here, and creates an Actor that builds from it.',
143+
'Where the Actor source code will live. With a Git provider, Apify creates the repository on your connected account from the template, clones it here, and creates an Actor that builds from it.',
144144
choices: [...GIT_SOURCE_CHOICES],
145145
// No default: an omitted flag triggers the wizard prompt, or "apify" when it cannot be asked.
146146
required: false,
@@ -292,7 +292,7 @@ export class CreateCommand extends ApifyCommand<typeof CreateCommand> {
292292
// Catches a non-interactive run that named a repository but omitted --source, which would
293293
// otherwise silently take the Apify path.
294294
if (!gitProvider && gitRepo) {
295-
throw new Error('--git-repo only applies to a Git source, so add --source github or drop it.');
295+
throw new Error('--git-repo only applies to a Git source, so add --source github|gitlab|bitbucket or drop it.');
296296
}
297297

298298
// The platform only accepts archive URLs listed in the official manifest.

‎src/lib/console-url.ts‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,29 @@ export function getConsoleUrl(): string {
2020
return DEFAULT_CONSOLE_URL;
2121
}
2222

23+
/** The fields of a stored login that decide which Apify account a Console URL opens. */
24+
export interface ConsoleAccount {
25+
id?: string;
26+
organizationOwnerUserId?: string;
27+
}
28+
29+
/**
30+
* The Console route an organization login has to run under. Console takes the account from the path, and
31+
* where the path does not say it falls back to the account the browser was last on — so a bare URL can act
32+
* on an account the CLI never asked for.
33+
*/
34+
export function getConsoleRoutePrefix(account?: ConsoleAccount): string | null {
35+
return account?.organizationOwnerUserId && account.id ? `/organization/${account.id}` : null;
36+
}
37+
38+
/**
39+
* Where a user manages the accounts Apify is connected to, including the git providers. Pass the account
40+
* whenever the page is opened to act on it, rather than only to be read.
41+
*/
42+
export function getConsoleIntegrationsUrl(account?: ConsoleAccount): string {
43+
return `${getConsoleUrl()}${getConsoleRoutePrefix(account) ?? ''}/settings/integrations`;
44+
}
45+
2346
function stripTrailingSlash(value: string): string {
2447
return value.replace(/\/+$/, '');
2548
}
Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
import type { AddressInfo } from 'node:net';
2+
3+
import computerName from 'computer-name';
4+
import open from 'open';
5+
6+
import { cryptoRandomObjectId } from '@apify/utilities';
7+
8+
import { type ConsoleAccount, getConsoleIntegrationsUrl, getConsoleUrl } from '../console-url.js';
9+
import { createLocalApiServer } from '../local-api-server.js';
10+
import { info } from '../outputs.js';
11+
import { cliDebugPrint } from '../utils/cliDebugPrint.js';
12+
import type { GitProvider } from './gitSource.js';
13+
14+
// Matches the version `apify login` serves its loopback routes under; Console reads it from the
15+
// `localCliApiVersion` query param and builds the callback URL from it.
16+
const API_VERSION = 'v1';
17+
18+
export const CONNECT_TIMEOUT_MS = 5 * 60_000;
19+
20+
export type ConnectViaConsoleResult =
21+
| { connected: true }
22+
/** The caller stopped waiting — it learned of the connection another way. Nothing to report. */
23+
| { abandoned: true }
24+
| { stopReason: 'connectCancelled' | 'connectTimedOut'; message: string };
25+
26+
/**
27+
* Connects a git provider that the CLI cannot authorize on its own.
28+
*
29+
* GitHub is a public OAuth app, so the CLI builds its authorize URL directly. GitLab and Bitbucket are
30+
* `UserIntegration` documents whose CSRF `state` is minted and stored server-side, so the browser has to
31+
* start the flow. Console does that and reports back to a loopback server here, the same hand-off
32+
* `apify login` uses.
33+
*
34+
* The caller re-reads the integration list afterwards, so the reported id is only logged.
35+
*/
36+
export const connectViaConsole = async (
37+
provider: GitProvider,
38+
label: string,
39+
{ abortSignal, account }: { abortSignal?: AbortSignal; account?: ConsoleAccount } = {},
40+
): Promise<ConnectViaConsoleResult> => {
41+
const consoleUrl = getConsoleUrl();
42+
const authToken = cryptoRandomObjectId();
43+
44+
let resolve!: (result: ConnectViaConsoleResult) => void;
45+
const finished = new Promise<ConnectViaConsoleResult>((resolveFinished) => {
46+
resolve = resolveFinished;
47+
});
48+
49+
const server = createLocalApiServer({
50+
// Only the origin, matching `apify login`: it is compared against the browser's Origin header.
51+
corsOrigin: new URL(consoleUrl).origin,
52+
authToken,
53+
routes: {
54+
[`POST /api/${API_VERSION}/git-provider-connected`]: (body, res) => {
55+
res.end();
56+
cliDebugPrint('git-source', 'connected', body);
57+
resolve({ connected: true });
58+
},
59+
[`POST /api/${API_VERSION}/exit`]: (body, res) => {
60+
res.end();
61+
resolve({
62+
stopReason: 'connectCancelled',
63+
message: body.isWindowClosed
64+
? `Connecting your ${label} account stopped: the Apify Console window was closed.`
65+
: `Connecting your ${label} account was canceled in Apify Console.`,
66+
});
67+
},
68+
},
69+
});
70+
71+
const timer = setTimeout(
72+
() =>
73+
resolve({
74+
stopReason: 'connectTimedOut',
75+
message: `Connecting your ${label} account did not finish within ${CONNECT_TIMEOUT_MS / 60_000} minutes.`,
76+
}),
77+
CONNECT_TIMEOUT_MS,
78+
);
79+
80+
abortSignal?.addEventListener('abort', () => resolve({ abandoned: true }), { once: true });
81+
82+
// Port 0 assigns a random free port.
83+
server.listen(0);
84+
const { port } = server.address() as AddressInfo;
85+
86+
// Console creates the integration for the account its own route names, so the CLI has to name the one
87+
// its token belongs to. Without it the connection can land on another account this user can reach,
88+
// where the poll below — and every later build — cannot see it.
89+
const url = new URL(getConsoleIntegrationsUrl(account));
90+
url.searchParams.set('localCliCommand', 'connect-git-provider');
91+
url.searchParams.set('gitProvider', provider);
92+
url.searchParams.set('localCliPort', `${port}`);
93+
url.searchParams.set('localCliToken', authToken);
94+
url.searchParams.set('localCliApiVersion', API_VERSION);
95+
try {
96+
url.searchParams.set('localCliComputerName', encodeURIComponent(computerName()));
97+
} catch {
98+
// Not critical, and it only labels the request in Console.
99+
}
100+
101+
info({ message: `Connect your ${label} account to Apify: ${url.href}` });
102+
// Printed above as well — a headless session, or a machine with no usable default browser, still needs it.
103+
open(url.href).catch(() => undefined);
104+
info({ message: 'Waiting for the connection to complete in your browser...' });
105+
106+
try {
107+
return await finished;
108+
} finally {
109+
clearTimeout(timer);
110+
// `close` alone waits for open sockets, and the browser can leave one behind. This runs at the
111+
// start of `apify create`, so a stray handle would hang the command long after its work is done.
112+
server.closeAllConnections();
113+
server.close();
114+
}
115+
};

0 commit comments

Comments
 (0)