Repository navigation
Commit fec0ea8
feat: add GitLab and Bitbucket to
> [!NOTE]
> **TL;DR** — adds `--source gitlab` and `--source bitbucket` alongside
`github`. Both tested with personal and repo token.
Follows #1348. Part of #1153.
## No longer blocked
| PR | What it gives us | State |
|---|---|---|
| [apify-core#30208](apify/apify-core#30208) |
GitLab/Bitbucket branches for `/integrations/git`, `create-repo` and
`register-deployment-key` | merged, live |
| [apify-core#30279](apify/apify-core#30279) |
the Console page that starts the OAuth connect for the CLI | merged,
live |
## The shape change
GitHub is one connection addressed by the fixed id `github-app`. GitLab
and Bitbucket are a `UserIntegration` **per connected account**, each
addressed by its own id and reporting its own workspaces — one GitLab
account can carry several namespaces, including subgroups. So:
- integrations are matched by `provider`, not by a hardcoded id
- every workspace from every matching integration is merged into one
list, each keeping the `providerId` that `create-repo` and
`register-deployment-key` address it by
Without that, a second connected account would have been silently
ignored.
## `--git-repo` matches labels
GitLab identifies a namespace by a number, so `--git-repo
4711/my-scraper` is what the API wants and nobody would type it.
`--git-repo my-group/sub` resolves through the label. Since a GitLab
label is a `full_path` and contains slashes of its own, the flag splits
at the **last** slash — `my-group/sub/my-scraper` is workspace
`my-group/sub`, repository `my-scraper`. The `workspaces` array in
`--json` reports labels for the same reason.
## Connecting
GitHub's authorize URL is two public query params, so the CLI builds it.
GitLab and Bitbucket mint a CSRF `state` server-side and need a
`UserIntegration` row to exist first — neither is something the CLI can
produce. So `connectViaConsole` opens the Console integrations page with
the `localCli*` deep-link params and waits on a loopback server, exactly
as `apify login` does.
The callback is not treated as the only signal. The CLI polls `GET
/integrations/git` alongside it and takes whichever lands first, because
a user who connects on the Console page by hand never triggers the
callback — and that is a real path, not a hypothetical: it is what
happens when the hand-off does not fire. An `AbortSignal` releases
whichever wait loses, so the loopback server never holds the command
open.
## Which Apify account the connection lands on
#1371 fixed this for GitHub. These two providers reach Console a
different way — the CLI navigates to the integrations page rather than
sending a `routePrefix` through an OAuth callback — so the bug survived.
Console creates the `UserIntegration` for the account its own route
names: `useEffectiveUserSession` resolves the session token from
`viewedUserId`, which comes only from the path. Where the path says
nothing, Console redirects to whichever account the browser was last on
— `/settings/*` is `ROUTE_KEYS.ACCOUNT`, which is in
`REMEMBER_LAST_ACCOUNT_ROUTE_NAMES`. `GET /integrations/git` is scoped
to the token owner, so an integration created elsewhere is invisible to
the CLI, which then waits out its full timeout and reports the wrong
cause.
It fails in **both** directions, unlike GitHub:
| CLI token | Browser last used | Integration created on | Result |
|---|---|---|---|
| organization | personal | personal | CLI never sees it |
| personal | organization | that organization | CLI never sees it |
The account now rides in the path. An explicit `/organization/<id>`
route is never redirected away (`if (!isUserRoute) return` in
`useRedirectToLastAccount`), so it pins both cases.
`getConsoleRoutePrefix` moved to `console-url.ts` beside
`getConsoleIntegrationsUrl`, which now takes the account — GitHub's
query param and these providers' path prefix share one source. The
account reaches the page the CLI opens, the connect URL printed for
agents and CI, and the `noWorkspace` next steps.
## Waiting for the connection
Two defects in the race, both found in review:
- **A failed listing read ended the wait.** `pollForWorkspaces` had no
guard around `load()`, so one 5xx or a dropped connection rejected out
of the race and skipped `controller.abort()` — the only thing that
closes the loopback server and clears its timer. `apify create` printed
its next steps and then sat there for the remaining five minutes. The
poll now survives a failed read, and the abort runs in a `finally`.
- **A successful callback got one read, not a poll.** Console reports
the connection as soon as the integration row exists, which can beat its
workspaces into the listing. That single empty read reported a connected
account as a missing one. It now waits the way the authorize branch
already does.
## A failed clone no longer costs the Actor
The clone is the only step needing the user's own provider credentials,
and the only purely local one. It used to abort the run, leaving a
repository with no Actor and a name that the next attempt could not
reuse.
Now the error is held, the Actor and the deploy key are still created,
and the stop reports the missing local copy with a `git clone` to run —
over HTTPS or SSH, whichever the user has credentials for. Bitbucket
also returns its clone URL with the account name embedded, which makes
git ask only for a password and defeats every token; the CLI strips the
userinfo before cloning or printing it.
## Reviewer notes
**This branch depends on Console's routing, and none of it is a
contract.** Three assumptions, all in apify-core, none covered by a test
that would fail if someone changed them:
1. every USER route gets an `/organization/:viewedUserId` twin
(`route_definitions.ts`)
2. an explicit organization route is never redirected to the last-used
account (`if (!isUserRoute) return`)
3. the path is what selects the effective account
(`useEffectiveUserSession`)
A regression in any of them is silent — no error, no 404, the connection
just lands on the wrong account and the user is told Apify has no GitLab
account. Cheapest mitigation is a test on the apify-core side; the real
one is the first follow-up below.
**GitHub's `provider` value is confirmed, not assumed.**
`src/api/src/routes/integrations/git_providers.ts` builds the GitHub
entry as `{ id: GITHUB_APP_PROVIDER_ID, provider: 'github' }` with
`GITHUB_APP_PROVIDER_ID = 'github-app'`, so matching by `provider` is
correct for all three.
## Verification
Verified live against production, on a personal login:
- **GitLab** — full chain: connect, create-repo, clone, Actor, deploy
key.
- **Bitbucket** — connect through the Console hand-off, create-repo,
Actor, deploy key. The clone was confirmed separately with a working
credential; the 403 first seen there was a workspace the account had no
create rights in, not a CLI or scope fault.
The account routing and the two waiting fixes landed after those runs
and are covered by unit tests only — an organization login has not been
exercised end to end.
64 tests across `gitSource.test.ts` and `connectViaConsole.test.ts`.
Full suite: 527 passed, 4 skipped. `lint`, `format`, `build` clean,
`docs/reference.md` regenerated.
## Follow-ups, not in this PR
- **Drop the loopback server.** Print the account-scoped Console URL,
let the user connect in Console's normal UI, and rely on the poll the
CLI already runs. Nothing changes in Console — the deep link just stops
being sent, and the hook it triggers is never entered. Removes ~110
lines plus its test, and both defects above were in code this deletes.
The cost is the `/exit` signal, which tells the CLI the user closed the
tab; GitHub has never had it, so this is parity rather than a
regression. Losing it means a user who gives up waits out the timeout.
- **Have the CLI create the integration.** A public `POST
/v2/integrations/git` returning `{ integrationId, loginUrl }` would let
the CLI start the flow under its own token. The account question then
never gets asked, and all three Console-routing assumptions above go
away.
- **Clone over SSH when a key is available.** The recovery steps offer
it, but the run itself always tries HTTPS, which fails for anyone who
only uses SSH.
- **Core-side, worth reporting:** a 403 from `create-repo` maps to
"Reconnect your account", which cannot help when the cause is missing
permission; and `shouldValidateScopes: false` lets a Bitbucket token
without `repository:admin` connect cleanly and fail three steps later.
- The Console hand-off scaffolding is duplicated between
`connectViaConsole.ts` and `auth/login.ts`. Worth extracting next to
`local-api-server.ts`, which also touches `edit-input-schema.ts`. Note
the double `encodeURIComponent` on `localCliComputerName` in both —
`searchParams.set` already encodes, so whoever extracts this has to keep
the quirk or fix both sides at once.
- Cloning a private repository still needs the user's own provider
credentials. The deploy key is Apify's and only lets the build worker
read it.
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>--source (#1356)1 parent 87dd140 commit fec0ea8
8 files changed
Lines changed: 844 additions & 146 deletions
File tree
- docs
- src
- commands
- auth
- lib
- git-source
- test/local/lib
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
280 | 280 | | |
281 | 281 | | |
282 | 282 | | |
283 | | - | |
| 283 | + | |
| 284 | + | |
284 | 285 | | |
285 | 286 | | |
286 | 287 | | |
| |||
308 | 309 | | |
309 | 310 | | |
310 | 311 | | |
311 | | - | |
312 | | - | |
313 | | - | |
314 | | - | |
315 | | - | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
316 | 317 | | |
317 | 318 | | |
318 | 319 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
| 12 | + | |
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
23 | | - | |
24 | 22 | | |
25 | 23 | | |
26 | 24 | | |
| |||
108 | 106 | | |
109 | 107 | | |
110 | 108 | | |
111 | | - | |
| 109 | + | |
112 | 110 | | |
113 | 111 | | |
114 | 112 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
140 | 140 | | |
141 | 141 | | |
142 | 142 | | |
143 | | - | |
| 143 | + | |
144 | 144 | | |
145 | 145 | | |
146 | 146 | | |
| |||
292 | 292 | | |
293 | 293 | | |
294 | 294 | | |
295 | | - | |
| 295 | + | |
296 | 296 | | |
297 | 297 | | |
298 | 298 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
23 | 46 | | |
24 | 47 | | |
25 | 48 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
0 commit comments