diff --git a/.claude/rules/api-tests.md b/.claude/rules/api-tests.md new file mode 100644 index 00000000000..2341283c060 --- /dev/null +++ b/.claude/rules/api-tests.md @@ -0,0 +1,58 @@ +--- +paths: + - "test/lightning_web/api/**/*.exs" + - "test/lightning_web/controllers/api/**/*.exs" + - "test/lightning_web/controllers/webhooks_controller*_test.exs" + - "test/lightning_web/controllers/project_mfa_enforcement_test.exs" + - "test/lightning_web/controllers/user_auth_test.exs" + - "test/lightning_web/controllers/adaptor_controller_test.exs" + - "test/lightning_web/collections_controller_test.exs" + - "test/lightning_web/collections/**/*.exs" + - "test/lightning_web/controllers/webhooks/**/*.exs" + - "test/lightning_web/plugs/channel_proxy/**/*.exs" + - "test/lightning_web/auth_test.exs" + - "test/lightning_web/plugs/access_token_auth_test.exs" + - "test/lightning_web/plugs/person_or_service_account_auth_test.exs" + - "test/lightning_web/plugs/webhook_auth_test.exs" + - "test/lightning_web/plugs/channel_proxy*_test.exs" + - "test/support/api_*.ex" + - "test/lightning/collections_test.exs" + - "test/lightning/collections_item*_test.exs" + - "test/lightning/invocation/query_test.exs" + - "test/lightning/projects/provisioner_test.exs" + - "test/lightning/workflows/graph_rules_test.exs" +--- + +# API error tests + +These files test what clients of `/api`, `/collections`, `/i/*` webhooks and the `/channels/*` proxy get back, and the modules that build those answers. Each promise a client relies on is tested in exactly one home. Before writing a test, find that home. + +## Where each promise is tested + +- **No token, a garbage token, a token of the wrong kind:** `api/sentinel_sweep_test.exs`, which walks every route from the router, and `auth_test.exs`, which holds the token-kind table and the lockout example. Controller test files keep no "without a token" or "with an invalid token" test. +- **A rule in shared code** (`Filters`, `Request`, `Problem`'s Accept handling, pointer escaping, `Errors`): the module's unit test holds the cases. Each route that uses the module keeps one test that fails if the route stops calling it, for example one paging test per route that calls `Filters.page/1`. +- **Who may reach a resource, what role it needs:** one table test per route family. +- **Each code's status and `detail` wording:** the table in `api/codes_test.exs`, written out literally. Rewording a code means editing its row there. +- **Everything else a route does:** the route test, through the router (`put(conn, ~p"/api/...", body)`), asserting status, headers and the whole body. + +A unit test of an API module exists for a promise no route can show (a 16-byte string as an id, the five-way Accept table) or a guard a developer hits. When a route test and a unit test both hold a promise, the unit test keeps the cases and the route keeps its one wiring test. + +## Writing a test + +- When a status or body changes, change the existing test, renaming it if its name stops being true. Add a test only for a promise that did not exist before. A new test with the same route and setup as an existing one means the existing one should have been edited. +- One operation per test, every assertion in it. Compare the whole body with `==`. Several inputs to one rule go in one table test (`for {input, expected} <- [...]`); different setups, or success against refusal, stay separate tests. +- Assert errors with `assert_problem(conn, status, code, entries)` by code, listing the entries exactly (`[]` when there are none). The response `detail` comes from `codes_test.exs`; each entry's `detail` is written out in the route test that produces it. Write challenge strings out in full with `assert_unauthenticated/3`. +- Every `/api` route that writes has its own stale-write test: another request deletes or changes the record mid-write (`Lightning.RaceHelpers.after_first_query/4`), and the route answers `assert_problem(409, :concurrent_edit, [])` and leaves the record as the other request left it. A new write route gets one too. +- `Lightning.Extensions.UsageLimiting.error_reason` lists four reasons, and the production limiter returns others as well. A test that stubs a limiter (`MockUsageLimiter`, `MockRateLimiter`) stubs at least one reason outside that type alongside the ones it checks. A stub that always returns `:exceeds_limit` passes even when every other reason crashes with a 500. +- When the same setup or assertion appears a third time, move it to `test/support`. +- Deleting or merging a test: name the test that still holds its promise, then break that behaviour in `lib` and watch the surviving test fail. + +## Names and layout + +- Describes and test names say what the client is doing: "reading", "creating", "refusals", "stale writes", "who may reach it". Put a new test beside the behaviour it refines. +- Names, describes and comments cite no spec section numbers, commit refs, ticket numbers or files outside this repo. Say the rule itself. +- Keep a file near 400 lines and always under 1,000. Split by behaviour (`workflows/creating_test.exs`, `workflows/stale_writes_test.exs`), not by position in the file. + +## Core context tests + +The `test/lightning/` files here alias and import only `Lightning` modules. Assert on `changeset.errors`, with each error's `code:` and key list. Pointer strings and problem bodies are a web promise, so they belong in a route test. diff --git a/.claude/rules/comments.md b/.claude/rules/comments.md new file mode 100644 index 00000000000..2587023e912 --- /dev/null +++ b/.claude/rules/comments.md @@ -0,0 +1,109 @@ +--- +paths: + - "lib/**/*.ex" + - "test/**/*.exs" + - "assets/js/**/*.ts" + - "assets/js/**/*.tsx" +--- + +# Comments and docs + +A comment promises the reader something the code can't tell them. Every one +that breaks the promise teaches people to skip the next, and agents read every +comment and act on it, so a wrong one misleads twice. Fewer, true, plain. + +## When a comment earns its place + +Keep a comment only if it carries one of these: + +- a constraint the names don't show, +- a decision a competent reader would otherwise reverse ("we deliberately + don't retry here"), +- a workaround for something outside the codebase, +- a pointer to where a rule comes from. + +Anything else goes. If a comment exists because the code is unclear, rename or +extract instead. + +```elixir +# Ecto's :utc_datetime type rejects non-zero microseconds, so truncate +# before casting or the changeset fails. +scheduled_at = DateTime.truncate(DateTime.utc_now(), :second) +``` + +## Which kind of comment + +- `@moduledoc`: what the module is for and when to reach for it. One or two + sentences is usually enough. Use `@moduledoc false` rather than a filler + sentence. +- `@doc`: how to call the function and what happens. Read in IEx, editor + hovers and by agents, usually without the source. The first line is one + sentence that stands alone. +- `#`: why this line is like this, for someone already in the body. + +In TypeScript, JSDoc on an export plays the part of `@doc`. + +## A `@doc` describes what the caller gets, not the steps inside + +Say what the inputs must be, every return shape including error tuples, and +side effects the caller can't see (writes, broadcasts, audit rows). Don't walk +through the body in order: that goes wrong the moment someone reorders it. + +```elixir +@doc """ +Replaces a credential's body on behalf of `actor`. + +Returns `{:error, :unauthorized}` if the actor can't write credentials in the +project, or `{:error, changeset}` if the body is invalid. The audit event is +written in the same transaction. +""" +``` + +## Explain a rule once, where it's enforced + +When several modules rely on the same rule, the explanation lives in the +module that enforces it. The others say nothing, or one line pointing there if +the caller would otherwise look wrong. Don't describe another module's +behaviour in a comment; it goes wrong when that module changes and nothing +flags it. + +## Write what will still be true next month + +| Goes stale | Lasts | +|---|---| +| Step-by-step account of the body | What the function promises | +| Callers, routes ("used by the API controller") | Constraints on inputs | +| Counts, line numbers, file paths | Why a rule exists | +| "Currently", "for now", "new", "now also" | Error shapes and side effects | +| Ticket, PR and refactor history | Link to the external source of a rule | + +Never narrate the change ("added for CON-331", "now also handles service +accounts"). That is the commit message's job. Code reads as if it was always +written that way. + +When you change behaviour, re-read the comments in that function and its +`@doc`, and fix anything the change made false in the same commit. + +A TODO says what is left and why it isn't done now. "TODO: fix" is not a TODO. + +## Plain words + +- Full sentences a colleague would say out loud. If a reader would need a + follow-up question to decode it, rewrite it. +- No coined shorthand: "the state rule", "project-less scope", "raw-actor + clause". Say the thing it stands for. +- No filler: "simply", "note that", "it's worth noting", "this ensures that". +- Don't weld clauses together with colons and semicolons. Two sentences, or + one simpler one. +- No spaced em dashes; prefer a full stop. +- `mix format` doesn't wrap comments. Reflow a paragraph after editing it so + every line but the last runs to 80 columns. In a `@doc` or `@moduledoc`, a + blank line separates ideas, and three or more parallel items go in a list. + +## Tests + +- Put the intent in the test name, not a comment above it. +- Comment setup only when the reason isn't visible ("inserted out of order so + the query has to sort"). +- Don't restate an assertion. Do say why when it pins deliberate behaviour a + reader might "fix" (a refusal that looks like a bug). diff --git a/.credo.exs b/.credo.exs index 8f9a5eafdfa..f2a5c40bd9a 100644 --- a/.credo.exs +++ b/.credo.exs @@ -43,7 +43,7 @@ # If you create your own checks, you must specify the source files for # them here, so they can be loaded by Credo before running the analysis. # - requires: [], + requires: ["credo/checks/*.ex"], # # If you want to enforce a style guide and need a more traditional linting # experience, you can change `strict` to `true` below: @@ -171,7 +171,9 @@ {Credo.Check.Consistency.MultiAliasImportRequireUse, []}, {Credo.Check.Readability.MultiAlias, []}, {Credo.Check.Readability.SeparateAliasRequire, []}, - {Credo.Check.Readability.StrictModuleLayout, []} + {Credo.Check.Readability.StrictModuleLayout, []}, + # Lightning's own, from credo/checks/ + {Lightning.Credo.Check.ChangesetErrorCode, []} ], disabled: [ # diff --git a/.formatter.exs b/.formatter.exs index 581ee697e61..14f2d638956 100644 --- a/.formatter.exs +++ b/.formatter.exs @@ -4,7 +4,7 @@ inputs: [ "*.{heex,ex,exs}", "priv/*/seeds.exs", - "{config,lib,test}/**/*.{heex,ex,exs}", + "{config,credo,lib,test}/**/*.{heex,ex,exs}", "tooling/adaptor_cache/lib/**/*.ex", "bin/adaptor_cache" ], diff --git a/CHANGELOG.md b/CHANGELOG.md index 141122dd7d6..48304f1d1cf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,11 +15,472 @@ and this project adheres to ## [Unreleased] -### Added - -### Changed - -### Fixed +### Breaking + +- Every `/api` route except `/api/oauth/token`, every `/collections` route and + `/adaptors/catalogue` answer these errors with a problem document, whatever + the request's `Accept` header, where they used to answer an HTML page or + `{"error": ""}`: + - A body that isn't valid JSON answers 400 `malformed`, and one over the size + limit answers 413 `content_too_large`, both before authentication. + - A query string or form body that isn't valid UTF-8 answers 400 `bad_request` + before authentication, as does any other request Lightning can't read. + - A `POST`, `PUT`, `PATCH` or `DELETE` whose `Content-Type` is one Lightning + doesn't parse, such as `text/plain`, answers 415 `unsupported_media_type` + before authentication, where the request used to reach the route with its + body ignored. A request with no body is let through whatever its type. + - A path that matches no route answers 404 `not_found` before authentication. + - An `Accept` header that doesn't allow JSON answers 406 `not_acceptable` with + a problem document as `application/json`. A body that doesn't arrive in time + answers 408 `request_timeout`, and a query string over 1,000,000 bytes 414 + `uri_too_long`. Every other request Lightning refuses answers by its own + status, so only a failure inside Lightning answers 500 `internal`, with a + problem document whose content type follows `Accept` like the rest. That + includes a failure in a plug before the router, such as the session or CORS, + which used to answer an HTML page to `Accept: */*`. + - An id that isn't a UUID answers 404 `not_found` instead of 400 on + `GET /api/jobs/:id`, `/api/runs/:id`, `/api/work_orders/:id` and + `/api/provision/:id`, their `/api/projects/:project_id/...` forms, the + `/api/projects/:project_id/jobs`, `/runs` and `/work_orders` lists, and the + `run_id` of `GET /api/log_lines`, the `project_id` of `GET /api/runs` and + the `id` of `GET /api/provision/yaml`. +- A failed login on a JSON route answers a 401 problem document instead of + `{"error": "Unauthorized"}`: + - Every `/api` route that takes a bearer token, and every `/collections` + route, answers a request with no bearer token with + `"code": "unauthenticated"` and `WWW-Authenticate: Bearer`, and one whose + token is refused (unknown, expired, revoked, or a locked-out user's) with + `"code": "invalid_token"` and + `WWW-Authenticate: Bearer error="invalid_token"`. These 401s sent no + `WWW-Authenticate` header before. + - A `Bearer` header with no token after it, with whitespace inside the token, + or sent twice answers 400 `invalid_request` instead, with + `WWW-Authenticate: Bearer error="invalid_request"` (RFC 6750 §3.1), where it + used to answer 401 `{"error": "Unauthorized"}`. + - These routes read the `Authorization` scheme without regard to case, so + `bearer ` works as `Bearer ` does, and a header with another + scheme, such as `Basic`, counts as no bearer token. + - `/api/users` answers a request with no token with + `"code": "unauthenticated"` and `WWW-Authenticate: Bearer`, and a refused + token with `"code": "invalid_token"`, both instead of + `{"error": "invalid_token"}`. + - The routes that take only the session cookie (`/api/ai_assistant/sessions`, + the workflow health routes under `/api/projects` and `/adaptors/catalogue`) + answer a missing session, a pending MFA code and a locked-out user with + `"code": "unauthenticated"` and + `WWW-Authenticate: Session realm="Lightning"`, a challenge no browser + prompts for. +- A token without the scope a route needs gets a 403 problem document with + `"code": "insufficient_scope"`, and its `WWW-Authenticate` header names the + one scope the request needed, as in + `Bearer error="insufficient_scope", scope="credentials:write"`: + - Where a route already checked scopes, this replaces + `{"error": "insufficient_scope"}`. + - On the projects, workflows, jobs, runs, work orders and log lines routes, + the 403 comes whether or not what the token names exists, instead of 404 + or 500. +- `/api/users` answers every error with a problem document, whose `errors` list + has one entry per field, each with a `pointer`, a `code` and a `detail`: + - `POST /api/users` with an email already taken answers 409 with + `"code": "taken"`, an entry at `#/email` and the existing user's id in + `existing.id`, instead of the existing user as `{"data": {...}}`. When + another field is also wrong it answers 422 `invalid`, with the email as one + of the entries, instead of 409. + - `POST /api/users` and `PATCH /api/users/:id` answer a field error with 422 + `invalid` instead of `{"errors": {"": [""]}}`. + - `POST /api/users` and `PATCH /api/users/:id` with a body that isn't a JSON + object, such as an array, answer 400 `malformed`, where a create answered + 422 and an update changed nothing and answered 200. + - `POST /api/users` and `PATCH /api/users/:id` with a key they don't take, + `email` on a `PATCH` included, answer 422 `invalid` with an `unknown_key` + entry at that key, as in `#/email`, where the key used to be ignored and the + request answered 201 or 200. + - `POST /api/users` and `PATCH /api/users/:id` take the user's fields from the + body alone, where a field in the query string, `role` included, was written + as if the body held it. + - `GET /api/users` with an `email` that isn't a string answers 400 + `invalid_parameter` with an entry naming the `email` parameter, instead + of 422. + - `GET /api/users/:id` and `PATCH /api/users/:id` for a user that doesn't + exist, or an id that isn't a UUID, answer 404 `not_found` instead of + `{"error": "Not Found"}`. +- `POST /api/users/register` on an instance with no registration extension + answers 501 `not_implemented` with a problem document instead of + `{"error": "Not Implemented"}`. +- `GET /api/projects/:id` answers every error with a problem document: + - A project that doesn't exist answers 404 `not_found` instead of + `{"error": "Not Found"}`, and so does an id that isn't a UUID, instead of + 400 `{"error": "Bad Request"}`. + - A caller who isn't a member of the project gets 404 `not_found`, the same + answer as for a project that doesn't exist, and a member who hasn't met its + MFA requirement gets 403 `forbidden`, both instead of 401 + `{"error": "Unauthorized"}`. + - A project scheduled for deletion answers 409 `scheduled_for_deletion` to a + caller who may see the project, 404 `not_found` to a non-member and 403 + `forbidden` to a member who hasn't met its MFA requirement, instead of 401 + `{"error": "Unauthorized"}`. +- `/api/credentials` answers every error with a problem document, whose `errors` + list has one entry per refused field, each with a `pointer`, a `code` and a + `detail`: + - `GET /api/credentials` and `GET /api/projects/:project_id/credentials`, with + a repo connection's token, answer 403 `insufficient_scope` instead of + `{"error": "Forbidden"}`. + - `GET /api/projects/:project_id/credentials` (and `?project_id=`) for a + project that doesn't exist answers 404 `not_found` instead of + `{"error": "Not Found"}`, and so does an id that isn't a UUID, instead of + 400 `{"error": "Bad Request"}`. A caller who isn't a member gets 404 + `not_found`, the same answer as for a project that doesn't exist, instead of + 403 `{"error": "Forbidden"}`. A project scheduled for deletion answers 409 + `scheduled_for_deletion` to a member, instead of 403. + - `POST /api/credentials` with a link to a project the caller can't see, or + that doesn't exist, answers 422 `invalid` with a `not_found` entry at that + link's `project_id`, and a link to a project where the caller may not add + credentials answers a `no_access` entry there, instead of 403 + `{"error": "Forbidden"}` for the whole request. Each entry points at the + link's position in the request, as in `#/project_credentials/1/project_id`. + - `POST /api/credentials` answers a field error with 422 `invalid` instead of + `{"errors": {"": [""]}}`. A credential body's errors point + into the body at its position: a typed body's at each field its schema + refuses (`#/credential_bodies/0/body/host`), where every refusal used to be + one `"Environment 1: body Invalid body types"`, and an OAuth token's at the + field it refuses, where the request used to fail with a 500. + - `POST /api/credentials` with a body that isn't a JSON object, such as an + array, answers 400 `malformed` instead of 422. + - `POST /api/credentials` with a name the user already holds, or an external + ID already in use, answers 409 `taken` with an entry at `#/name` or + `#/external_id`, instead of 422. + - `POST /api/credentials` with a typed body while the adaptor catalogue is + still loading answers 503 `not_ready` with `Retry-After: 5`, instead of 422. + - `DELETE /api/credentials/:id` for a credential that doesn't exist, or an id + that isn't a UUID, answers 404 `not_found` instead of + `{"error": "Not Found"}`. Someone else's credential answers 404 `not_found`, + the same answer as for one that doesn't exist, unless the caller sees it + linked to a project whose credentials they may list, when it answers 403 + `forbidden`; both were `{"error": "Forbidden"}`. + - `DELETE /api/credentials/:id` for a credential that runs have used answers + 409 `in_use` and keeps the credential, instead of failing with a 500. +- `POST /api/provision` answers every error with a problem document, whose + `errors` list has one entry per problem, each with a `pointer`, a `code` and a + `detail`: + - A document that isn't valid answers 422 `invalid` with entries that point at + the request's own keys and positions, as in `#/workflows/0/jobs/1/name`, + instead of `{"errors": {"workflows": {"": ...}}}` keyed by workflow + and job names. Each `detail` is the validation's own message, such as + `can't be blank`, instead of `This field can't be blank.`. + - A key the document doesn't accept gets its own `unknown_key` entry at that + key, as in `#/workflows/0/colour`, instead of one + `extraneous parameters: ` error listing them all. + - A `project_credentials` entry whose name matches none of the caller's + credentials answers a `not_found` entry at `#/project_credentials//name`, + instead of an error quoting the name. A channel's credential errors point at + `#/channels//destination_credential_id`, the key the document sends. + - Two jobs with one name in a workflow, or any other value that must be + unique, answers 409 `taken` with an entry at the clashing field, instead + of 422. + - A project credential, collection or channel `id` that another project + already uses answers 409 `taken` with an entry at that item's `id`, as in + `#/collections/1/id`, instead of failing with a 500. + - A project `id` naming a project the caller has no standing on (not a member, + or for a repo connection any project but its own) answers 404 `not_found`, + the same answer as for one that doesn't exist, instead of 403 + `{"error": "Forbidden"}`. So does an `id` that isn't a UUID, instead of 400. + - A viewer, an editor, a member who hasn't met the project's MFA requirement + and a caller who may not create a project answer 403 `forbidden` instead of + `{"error": "Forbidden"}`. + - A project scheduled for deletion answers 409 `scheduled_for_deletion` to a + caller who may provision it, instead of 403. + - A repo connection refused by the GitHub sync plan limit answers 402 + `exceeds_limit`, instead of 403 with the limiter's message, and only once + the document is valid, so an invalid document gets its 422 first. + - A job whose adaptor needs checking while the adaptor catalogue is still + loading answers 503 `not_ready` with `Retry-After: 5` at once, instead of + waiting up to 90 seconds for the catalogue. +- A write that loses a race with another request answers 409 `concurrent_edit`, + with nothing written, instead of `{"error": "Conflict"}`: + - `POST /api/provision`, when another request changed a workflow or channel + the document writes, or deleted a row it writes. + - `PUT /api/projects/:id`, `PUT /api/credentials/:id`, + `PUT /api/projects/:project_id/collections/:id` and `PATCH /api/users/:id`, + when another request deleted the row they change, or a project member, + credential link or credential body they write. + - `PUT` and `PATCH /api/projects/:project_id/workflows/:id`, when another + request saved the workflow first, or deleted a job, edge or trigger it + writes. +- `GET /api/provision/:id` and `GET /api/provision/yaml` answer every error with + a problem document: + - A project the caller has no standing on (not a member, or for a repo + connection any project but its own) answers 404 `not_found`, the same answer + as for one that doesn't exist, instead of 403 `{"error": "Forbidden"}`. A + project that doesn't exist answers 404 `not_found` instead of + `{"error": "Not Found"}`. + - A member who hasn't met the project's MFA requirement answers 403 + `forbidden` instead of `{"error": "Forbidden"}`. + - A project scheduled for deletion answers 409 `scheduled_for_deletion` to a + caller who may see it, instead of 403. + - A `snapshots` that isn't a list of UUIDs answers 400 `invalid_parameter` + with an entry naming the `snapshots` parameter, on both routes. One that + isn't a list, such as `?snapshots=true`, used to fail with a 500. + - `GET /api/provision/yaml` without `id` answers 400 `invalid_parameter` with + an entry naming the `id` parameter, instead of a 400 that named nothing. +- The workflows routes (`GET /api/workflows`, `GET /api/workflows/:id`, and + `GET` and `POST /api/projects/:project_id/workflows`, `GET`, `PATCH` and + `PUT /api/projects/:project_id/workflows/:id`) answer every error with a + problem document, and their success bodies no longer carry `"errors": {}`: + - A project or workflow that doesn't exist, an id that isn't a UUID, or a + workflow outside the path's project answers 404 `not_found`, instead of + `{"id": ..., "errors": ["Not Found"]}`, 400 `{"error": "Bad Request"}` or a + 422 quoting the id. + - A caller who isn't a member of the project gets 404 `not_found`, the same + answer as for a project that doesn't exist, whether or not the workflow is + open in the editor. A viewer creating or updating a workflow, and a member + who hasn't met the project's MFA requirement, get 403 `forbidden`. Both used + to get 401 `{"error": "Unauthorized"}`. + - A project scheduled for deletion answers 409 `scheduled_for_deletion` to a + caller who may see it. + - `PATCH` and `PUT` on a workflow open in the editor answer 409 + `being_edited`, before the body is read, instead of + `{"id": ..., "errors": {"workflow": ["Cannot save a workflow () ..."]}}`. + - `POST`, `PATCH` and `PUT` with a body that isn't a JSON object, such as an + array, answer 400 `malformed`. + - `POST`, `PATCH` and `PUT` with a body that isn't valid answer one 422 + `invalid` with every problem as an entry pointing at the request's keys and + positions, as in `#/jobs/1/name`, instead of the first problem found as + `{"id": ..., "errors": {"": [""]}}`. The workflow rules + have their own codes: `cycle` at the edge that closes a cycle, as in + `#/edges/3/target_job_id`, instead of `Cycle detected on job `; + `one_enabled` and `trigger_removed` at `#/triggers`; `missing_trigger_edge`, + `one_enabled_edge` and `single_target` at `#/edges`; and `required` at the + `id` of a job or trigger sent without one, which used to fail with a 500 on + `POST`. A `project_id`, or on `PATCH` and `PUT` an `id`, that differs from + the path's is a `mismatch` entry in the same answer. + - On `POST`, an edge naming a job or trigger id that no job or trigger in the + request carries gets an entry at that edge's key, where the reference used + to be dropped. + - A job, trigger or edge id already in use, or any other value that must be + unique, answers 409 `taken` with an entry at the clashing field, instead + of 422. + - The plan limit on active workflows answers 402 `exceeds_limit`, instead of + 422 with the limiter's message at `project_id`, and only once the body is + valid, so an invalid body gets its 422 first. Only a request that enables a + trigger counts against the limit: a `PATCH` or `PUT` that leaves an enabled + trigger enabled no longer does. + - A deleted workflow answers 404 `not_found`. `GET` used to return it, and + `PATCH` and `PUT` failed with a 500. +- The jobs routes (`GET /api/jobs`, `GET /api/jobs/:id` and + `GET /api/projects/:project_id/jobs`) answer every error with a problem + document: + - A job or project the caller isn't a member of, including one named by + `GET /api/jobs?project_id=`, answers 404 `not_found`, the same answer as for + one that doesn't exist, instead of 401 `{"error": "Unauthorized"}`. A member + who hasn't met the project's MFA requirement gets 403 `forbidden` instead + of 401. + - The `self`, `first`, `last`, `next` and `prev` links carry `project_id` when + the request filtered by it, the page and the page size, and no other query + parameter from the request. +- The runs and work orders routes (`GET /api/runs`, `GET /api/runs/:id`, + `GET /api/work_orders`, `GET /api/work_orders/:id`, and + `GET /api/projects/:project_id/runs` and `/work_orders`) answer every error + with a problem document: + - A run, work order or project the caller isn't a member of answers 404 + `not_found`, the same answer as for one that doesn't exist, instead of 401 + `{"error": "Unauthorized"}`. A member who hasn't met the project's MFA + requirement gets 403 `forbidden` instead of 401. + - The caller is authorised before the query is read, so a caller who may not + see the project gets its 404 or 403 whatever the query holds, where a bad + query used to be answered first. + - Bad query parameters answer one 400 `invalid_parameter` with an entry per + bad parameter, as in + `{"parameter": "inserted_after", "code": "type", "detail": "is invalid"}` or + `{"parameter": "state", "code": "not_allowed", ...}`, instead of the first + one found as `{"error": "Invalid datetime format for '': "}` or + `{"error": "Invalid state filter: ..."}`. No entry quotes the value. + - A `workflow_id` or `work_order_id` filter on the runs lists, and an `id` or + `workflow_id` filter on the work orders lists, that isn't a UUID is one of + those entries. + - The `self`, `first`, `last`, `next` and `prev` links of a list carry the + filters the list applied, the page and the page size, and no other query + parameter from the request. +- The log lines route (`GET /api/log_lines`) answers every error with a problem + document: + - Bad query parameters answer one 400 `invalid_parameter` with an entry per + bad parameter, as in + `{"parameter": "timestamp_after", "code": "type", "detail": "is invalid"}`, + instead of the first bad timestamp as + `{"error": "Invalid datetime format for '': "}`. No entry quotes + the value. + - A `level` that isn't one of `success`, `always`, `info`, `warn`, `error` or + `debug` is one of those entries, with code `not_allowed`, where it used to + fail with a 500. + - A `project_id`, `workflow_id`, `job_id`, `work_order_id` or `run_id` filter + that isn't a UUID is one of those entries too, where it used to answer 404. + - The `self` and pagination links carry the filters the list applied, the page + and the page size, and no other query parameter from the request. +- `GET /api/ai_assistant/sessions` answers every error with a problem document: + - A missing or unknown `session_type`, or a missing `job_id` or `project_id`, + answers 400 `invalid_parameter` with one entry per bad parameter, instead of + `{"error": "Bad Request"}`. An `offset` or `limit` that isn't a whole + number, a negative `offset`, a `limit` below 1 and a `workflow_id` that + isn't a UUID answer the same, where they used to fail with a 500 or a 400 + with no entries. The job or project the query names is checked first, so a + caller who can't see it gets 404 whatever the other parameters say. + - A caller who isn't a member of the project, or of the job's project, and + someone else's session for an unsaved job with no workflow, answer 404 + `not_found`, the same answer as for a project that doesn't exist, instead of + 403 `{"error": "Forbidden"}`. A project that doesn't exist answers 404 + `not_found` instead of `{"error": "Not Found"}`, and so does a `job_id` or + `project_id` that isn't a UUID, instead of 400. + - A member who hasn't met the project's MFA requirement answers 403 + `forbidden`, and a project scheduled for deletion answers 409 + `scheduled_for_deletion` to a member, both instead of 403 + `{"error": "Forbidden"}`. +- The workflow health routes + (`GET /api/projects/:project_id/workflows/:workflow_id/health/outcomes`, + `/failures` and `/runs`) answer every error with a problem document: + - A `days` other than 1, 7 or 30 answers 400 `invalid_parameter` with an entry + naming `days`, instead of `{"error": "days must be one of 1, 7, 30"}`. + - On `/runs`, an `x-timezone` header that isn't an IANA timezone name, or more + than one of them, answers 400 `invalid_header` with an entry at + `"header": "x-timezone"`, instead of + `{"error": "x-timezone must be an IANA timezone name"}`. A bad `days` is + reported first. + - A workflow or project that doesn't exist, an id that isn't a UUID, and a + caller who isn't a member answer 404 `not_found` instead of + `{"error": "Not Found"}`. + - A member who hasn't met the project's MFA requirement answers 403 + `forbidden`, and a project scheduled for deletion answers 409 + `scheduled_for_deletion` to a member, both instead of 404. +- `GET /adaptors/catalogue` answers a catalogue that hasn't loaded yet with 503 + `not_ready`, still with `Retry-After: 5`, instead of + `{"error": "adaptor catalogue unavailable"}`. +- The collections API (`/collections/*`) answers every error with a problem + document. A missing item is still not an error: `GET /collections/:name/:key` + answers 204, and `DELETE /collections/:name/:key` answers 200 with + `"deleted": 0`. + - Every `DELETE` answer carries only `key` and `deleted`. A missing item no + longer gets `"error": "Item Not Found"`, and a deleted item and + `DELETE /collections/:name` no longer get `"error": null`. + - `PUT /collections/:name/:key` with a value Lightning can't store, such as + one that is too long or isn't a string, answers 422 `invalid` with an entry + at `#/value`, instead of 200 `{"upserted": 0, "error": "Format error"}`. The + collections adaptor treats that 200 as a success, so jobs that silently lost + format-error writes will now fail. A key too long to store answers 422 + `invalid` with an entry whose `path` is `key`, since the key is in the path, + not the body; beside a bad value, both entries come back. + - `PUT /collections/:name/:key` with no `value`, and `POST /collections/:name` + with no `items`, answer 422 `invalid` with a `required` entry, instead of + `{"error": "Missing required field: "}`. + - `POST /collections/:name` with a key repeated in `items` answers 422 + `invalid` with a `duplicate` entry at the later item's key, instead of + `{"upserted": 0, "error": "Duplicate key found"}`. An item that isn't an + object or fails validation answers 422 with an entry at that item, instead + of 500. + - `PUT /collections/:name/:key` and `POST /collections/:name` over a plan + limit answer 402 `exceeds_limit`, without the limiter's message, instead of + 422 `{"upserted": 0, "error": ""}`. A value that is invalid and + over the limit answers 422, not 402. + - A collection that doesn't exist answers 404 `not_found` instead of + `{"error": "Not Found"}`. A caller who isn't a member of its project, a run + of another project, and a `project_id` that isn't a UUID answer 404 + `not_found` too, instead of 401 `{"error": "Unauthorized"}` and 400 + `{"error": "Bad Request"}`. + - A member whose role doesn't allow the action, or who hasn't met the + project's MFA requirement, answers 403 `forbidden` instead of 401 + `{"error": "Unauthorized"}`. A project scheduled for deletion answers 409 + `scheduled_for_deletion` to a member who may act on it. + - A name held by collections in more than one of the caller's projects answers + 409 `ambiguous_name` instead of `{"error": "... Add ?project_id=..."}`. Only + projects the caller is a member of count, and a run counts only its own + project, so a name another project also uses no longer answers 409. + - `GET /collections/:name` with bad query parameters answers 400 + `invalid_parameter` with one entry per bad parameter, instead of + `{"error": "Bad Request"}`. An unknown parameter's entry has the code + `unknown_key`. + - `GET /collections/:name` with a negative `limit` answers 400 + `invalid_parameter` with an `out_of_range` entry, instead of failing with + a 500. +- Webhooks (`/i/*`) answer their errors with a problem document. Every status + stays as it was, and so does every success body, including the 200 + `{"work_order_id": ..., "error": "too_many_runs"}` for an asynchronous trigger + whose run was refused after its work order was saved: + - An unknown trigger, and wrong credentials, answer 404 `not_found` instead of + `{"error": "Webhook not found"}`. + - Missing credentials answer 401 `unauthenticated` instead of + `{"error": "Unauthorized"}`, with + `WWW-Authenticate: ApiKey header="x-api-key"` when the trigger takes an API + key. A trigger never challenges Basic, so a browser doesn't prompt for one + trigger's login and then send it to other triggers. + - A disabled trigger answers 403 `trigger_disabled`, without its `message`. + - A body whose content type Lightning doesn't read answers 415 + `unsupported_media_type` instead of `{"error": "Unsupported Media Type"}`. + - A JSON body that doesn't parse answers 400 `malformed`, and a form body that + isn't valid UTF-8 answers 400 `bad_request`, instead of + `{"error": "Bad Request"}`. A body over the size limit answers 413 + `content_too_large` instead of `{"error": "Request Entity Too Large"}`. + Without an `Accept` header naming JSON, both were the reason phrase as + `text/html`. + - A database outage answers 503 `service_unavailable` instead of + `{"error": "service_unavailable", "message": ..., "retry_after": ...}`. The + `Retry-After` header stays. + - A limiter refusal answers 429 whatever the reason, with `code` + `too_many_requests`, `too_many_runs` or `exceeds_limit` in place of `error`, + and without the limiter's `message`. + - A work order that can't be saved answers 422 `invalid` with entries, instead + of `{"error": "invalid_request", "details": ...}` or + `{"error": ""}`. + - A synchronous trigger whose run was refused answers 429 `too_many_runs`, and + one left without a run for any other reason answers 500 `internal`, instead + of `{"work_order_id": ..., "error": ...}`. A synchronous trigger whose + workflow doesn't finish in time answers 504 `timeout` instead of + `{"error": "timeout", "message": ...}`. All three keep `work_order_id`. +- The channel proxy (`/channels/*`) answers the errors it builds itself with a + problem document. Every status stays as it was, and whatever the destination + answers, error or not, still passes through untouched: + - An unknown or disabled channel, and an id that isn't a UUID, answer 404 + `not_found` instead of `{"error": "Not Found"}`. + - Refused client credentials answer 401 `unauthenticated` instead of + `{"error": "Unauthorized"}`, with + `WWW-Authenticate: ApiKey header="x-api-key"` when the channel takes an API + key. A channel never challenges Basic, so a browser doesn't prompt for one + channel's login and then send it to other channels. + - A failure before proxying answers 500 `internal` instead of + `{"error": "Internal Server Error"}`. + - A destination credential that can't be used answers 502 `bad_gateway` + instead of `{"error": "Bad Gateway"}`. + +### Added + +- `GET /api/credentials/:id` reads one credential. Its owner gets the credential + with its body. A caller who sees it linked to a project whose credentials they + may list gets it with the body hidden. Anyone else gets 404 `not_found`, the + same answer as for one that doesn't exist. + [#5239](https://github.com/OpenFn/lightning/pull/5239) + +### Changed + +- Service-account token requests must now name the scopes they want. A request + to `POST /api/oauth/token` without `scope`, or with an empty one, is refused + with `invalid_scope` instead of getting every scope. + [#5239](https://github.com/OpenFn/lightning/pull/5239) +- API page links (`self`, `first`, `last`, `next`, `prev`) carry the `page_size` + Lightning applied, so a request for more than 100 a page links to pages of + 100, and no query parameter the list doesn't apply: on `GET /api/projects` + only the page and page size, on `GET /api/users` those and `email`. A single + run's, job's, work order's or project's `self` link is its own URL, with no + query parameter from the request. +- Saving a job while Lightning is still loading its adaptor catalogue, which + happens just after it starts, is refused at once with "adaptor catalogue is + not ready yet, try again shortly" instead of waiting up to 90 seconds for the + catalogue. This applies wherever a job is saved: the workflow editor, the + other workflow pages and `POST /api/provision`. + +### Fixed + +- Toggling a project's MFA requirement or support access in its settings shows + an error when the change can't be saved, instead of crashing the page. +- The workflow editor and the settings forms for projects, sandboxes, users, + credentials and collections no longer crash when another save wins the race or + the record was deleted. The workflow editor and project settings ask you to + try again; the other forms close and say the record no longer exists. ## [2.20.0-pre] - 2026-10-23 diff --git a/SERVICE_ACCOUNTS.md b/SERVICE_ACCOUNTS.md index 4b8237135e8..15b525fecc7 100644 --- a/SERVICE_ACCOUNTS.md +++ b/SERVICE_ACCOUNTS.md @@ -5,8 +5,8 @@ signing in with a key pair instead of a password. Use one when infrastructure automation, rather than a person, creates the instance's users, including its first superuser. -Each instance has at most one service account. For now it can only create, -change and find users. +Each instance has at most one service account. It can create, change and find +users, and create, change and read projects, collections and credentials. > #### Still under development {: .warning} > @@ -92,6 +92,7 @@ const { access_token } = await fetch(token_endpoint, { client_assertion_type: 'urn:ietf:params:oauth:client-assertion-type:jwt-bearer', client_assertion: `${unsigned}.${signature}`, + scope: 'users:read users:write', }), }).then(response => response.json()); ``` @@ -116,11 +117,28 @@ Lightning refuses a JSON body. If the token endpoint answers `invalid_client`, Lightning's logs say why. See [Logs](#logs). +## What the token can do + +Send the access token as `Authorization: Bearer `. What it may do depends +on the scopes it carries: + +- `users:read` finds users, and `users:write` creates and changes them. +- `projects:write` reads, creates and changes projects. +- `collections:write` reads, creates and renames collections. +- `credentials:write` reads, creates and replaces credentials. + +Ask for the scopes you need with the `scope` parameter when you request the +token. Projects, collections and credentials have no read scopes: their `GET` +routes need the same `:write` scope as their `PUT` routes. + +The service account acts on every project, as if it held every project role, +owner included, but only as far as its token's scopes reach. It is never a +member of a project and never shows in a project's members. + ## Manage users -Send the access token as `Authorization: Bearer `. Creating and changing -users needs the `users:write` scope, and finding them needs `users:read`. A -token gets both unless you ask for fewer with a `scope` parameter. +Creating and changing users needs the `users:write` scope, and finding them +needs `users:read`. Create a user, here a superuser whose email is already verified. `role` is `user` or `superuser`, and defaults to `user`: @@ -163,8 +181,31 @@ user later: Lightning doesn't email the users it creates, so whoever runs the automation has to tell them their account exists. -If the email is already taken, Lightning answers 409 with the existing user and -leaves it alone, so automation can run the same request again safely. +If a user already has the email, in any case, Lightning answers 409 `taken` and +leaves that user alone, so automation can run the same request again safely. The +answer carries the existing user's id in `existing`, and nothing else about +them: + +```json +{ + "type": "about:blank", + "title": "Conflict", + "status": 409, + "code": "taken", + "detail": "A value that must be unique is already in use. See errors.", + "errors": [ + { + "pointer": "#/email", + "code": "taken", + "detail": "has already been taken" + } + ], + "existing": { "id": "604f0cd2-37fb-4833-a5e1-821555b5b314" } +} +``` + +If another field is wrong as well, Lightning answers 422 `invalid` instead, with +the email's `taken` entry among the others and no `existing`. Find a user by email: @@ -174,7 +215,8 @@ curl "https://lightning.example.org/api/users?email=ada@example.com" \ ``` The list pages 10 users at a time by default, and at most 100 with `page_size`. -An `email` that isn't a single string answers 422. +An `email` that isn't a single string, such as `email[]=...`, answers 400 +`invalid_parameter` with an entry whose `parameter` is `email`. Change a user's role or password: @@ -186,7 +228,142 @@ curl -X PATCH https://lightning.example.org/api/users/$USER_ID \ ``` A new role or password signs the user out of every browser session. The email -can't be changed, and the API can't delete or disable a user. +can't be changed: a `PATCH` ignores it, as it ignores any key it doesn't take. +The API can't delete or disable a user. + +## Manage projects + +`GET /api/projects/:id` shows a project. `PUT /api/projects/:id` creates the +project under that id, or, if a project already has it, replaces its name and +description. Both need `projects:write`. + +To create a project, send its `name` and `description`, its `members` as +`{email, role}` pairs, and `notify`. Each member must already be a user. +`notify` is `true` by default, and decides whether Lightning emails the members +it adds. `members` and `notify` only count when the project is created: + +```bash +curl -X PUT https://lightning.example.org/api/projects/$PROJECT_ID \ + -H "Authorization: Bearer $ACCESS_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "malaria-programme", + "description": "Malaria case reporting", + "members": [ + {"email": "ada@example.com", "role": "owner"}, + {"email": "grace@example.com", "role": "editor"} + ], + "notify": false + }' +``` + +Lightning answers 201 with the project and its members when it creates one, and +200 when it replaces one. + +## Manage collections + +`GET` and `PUT /api/projects/:project_id/collections/:id` read and write a +project's collection. They need `collections:write`. `PUT` creates the +collection under that id, or renames the one that has it and keeps its items: + +```bash +curl -X PUT https://lightning.example.org/api/projects/$PROJECT_ID/collections/$COLLECTION_ID \ + -H "Authorization: Bearer $ACCESS_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"name": "patients"}' +``` + +Lightning answers `{"data": {"id", "project_id", "name"}}`, with 201 for a new +collection and 200 for a rename. The items inside a collection are not part of +this API. + +## Manage credentials + +`GET` and `PUT /api/credentials/:id` read and write a credential. They need +`credentials:write`. `PUT` creates the credential under that id, or replaces the +name, schema and `main` body of the one that has it. + +A `PUT` must name the credential's `owner` by email, and the owner must be an +existing user: + +```bash +curl -X PUT https://lightning.example.org/api/credentials/$CREDENTIAL_ID \ + -H "Authorization: Bearer $ACCESS_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "Acme DHIS2", + "schema": "raw", + "owner": "ada@example.com", + "credential_bodies": [ + {"name": "main", "body": {"username": "admin", "password": "district"}} + ], + "project_credentials": [{"project_id": "'"$PROJECT_ID"'"}] + }' +``` + +- `credential_bodies` must be exactly one body, named `main`, holding an object. + Replacing a credential leaves any other body it has alone. +- `project_credentials` is optional. A `PUT` only adds links, and never removes + one the credential already has. Linking a project needs `credentials:write` + only. Naming a project that doesn't exist answers 422 `invalid`, with a + `not_found` entry at that link, as in `#/project_credentials/0/project_id`. + Naming a project the owner isn't a member of answers the same way, with a + `no_access` entry. +- The response never includes the credential's body. It lists every project the + credential is linked to, but names a linked project only if the token also + holds `projects:write`. +- A service account can write credentials but never delete one. Listing, + creating with `POST` and deleting are for people: a service account's token + gets a 401 `invalid_token` from those routes. + +## When Lightning refuses + +Every refusal is an [RFC 9457](https://www.rfc-editor.org/rfc/rfc9457) problem +document with `type`, `title`, `status`, `code` and `detail`, as in the 409 +example in [Manage users](#manage-users). It is sent as `application/json`, or +as `application/problem+json` when the request's `Accept` names that. Branch on +`status` and `code`, not on `detail`. + +When the problem lies in particular parts of the request, the document also has +`errors`, a list of entries. Each entry has a `code` and a `detail`, and says +where the problem is: a `pointer` into the body, as in `#/members/0/email`, or +the name of a query `parameter`. Entries never quote a value from the request. + +- 400 `malformed`: the body isn't a JSON object. +- 400 `invalid_parameter`: a query parameter isn't valid. `errors` names it. +- 400 `invalid_request`, with + `WWW-Authenticate: Bearer error="invalid_request"`: the `Authorization` header + has no token after `Bearer`, has whitespace inside the token, or was sent + twice. +- 401 `invalid_token`, with `WWW-Authenticate: Bearer error="invalid_token"`: + the access token is expired or bad, or isn't one this route accepts. +- 401 `unauthenticated`, with `WWW-Authenticate: Bearer`: the request sent no + `Authorization` header. +- 402 `exceeds_limit`: creating a collection would go over the instance's limit, + or the plan has no seat for a member a project `PUT` adds. For a member, + `errors` has an entry at `#/members`. +- 403 `insufficient_scope`, with + `WWW-Authenticate: Bearer error="insufficient_scope"` naming the one scope the + request needed, as in `scope="credentials:write"`: the token lacks that scope. + Lightning checks this first, so it comes before any 404 or 422, whether the + project is missing or the body is wrong. +- 404 `not_found`: the project, collection, credential or user doesn't exist, or + the id in the path isn't a UUID. +- 409 `scheduled_for_deletion`: the project (on the projects and collections + routes) or the credential (on a credential `PUT`) is scheduled for deletion. + Lightning refuses changes until the deletion is cancelled or completes. +- 409 `id_taken`: the id already belongs to a collection in another project, or + to a credential with a different owner. Lightning never moves either. +- 409 `taken`: the value at the entry's pointer must be unique and is already in + use. That is a user's email, a collection's name within its project, or a + credential's name among its owner's credentials. +- 415 `unsupported_media_type`: the body isn't sent as JSON. +- 422 `invalid`: the body doesn't fit. `errors` has an entry for each field that + is wrong. On the projects, collections and credentials routes it also has one + for each key the route doesn't accept. The users routes ignore keys they don't + take. +- 503 `not_ready`, with `Retry-After`: Lightning can't check a credential's + schema yet. Try again after the number of seconds it gives. ## Change or remove the key @@ -214,6 +391,12 @@ there as "Service account", named by its key's thumbprint. It records: - `updated` for each user it changes, with the before and after of the email, names, role and confirmation. A new password shows only as `password_changed`. A change that changes nothing isn't recorded. +- `created` for each project it creates, with `collaborator_added` for each + member it adds, and `name_updated` or `description_updated` when it changes + one. +- `created` and `updated` for each collection it creates or renames. +- `created` and `updated` for each credential it writes, and `added_to_project` + for each project it links. ### Logs diff --git a/config/config.exs b/config/config.exs index 19206ea48ad..1bfa8765950 100644 --- a/config/config.exs +++ b/config/config.exs @@ -31,8 +31,7 @@ config :hammer, config :lightning, LightningWeb.Endpoint, url: [host: "localhost"], render_errors: [ - view: LightningWeb.ErrorView, - accepts: ~w(html json), + formats: [html: LightningWeb.ErrorView, json: LightningWeb.ErrorView], layout: false ], pubsub_server: Lightning.PubSub, @@ -226,9 +225,9 @@ config :lightning, :dataclip_search_indexing, batch_size: 250, max_batches: 10 config :lightning, Lightning.Runtime.RuntimeManager, start: false -config :lightning, LightningWeb.CollectionsController, - default_stream_limit: 1_000, - max_database_limit: 500 +config :lightning, :collections_api, default_stream_limit: 1_000 + +config :lightning, Lightning.Collections, stream_page_size: 500 # Configuration for injecting a custom feedback mechanism or component collection # for the AI Assistant. This can include interactive elements such as a button diff --git a/config/test.exs b/config/test.exs index 6125eb5baaf..0cf6e2cbdfc 100644 --- a/config/test.exs +++ b/config/test.exs @@ -188,10 +188,6 @@ config :lightning, :github_app, -----END RSA PRIVATE KEY----- """ -config :lightning, LightningWeb.CollectionsController, - default_stream_limit: 25, - max_database_limit: 15 - # The OIDC test suite serves discovery/JWKS/userinfo/token over http on # localhost via Bypass; allow those loopback endpoints to skip TLS verification. config :lightning, :auth_providers_allow_insecure_loopback, true diff --git a/credo/checks/changeset_error_code.ex b/credo/checks/changeset_error_code.ex new file mode 100644 index 00000000000..fb9adeddf31 --- /dev/null +++ b/credo/checks/changeset_error_code.ex @@ -0,0 +1,206 @@ +defmodule Lightning.Credo.Check.ChangesetErrorCode do + use Credo.Check, + id: "LX0001", + base_priority: :high, + category: :warning, + param_defaults: [ + files: %{included: ["lib/lightning/**/*.ex", "lib/lightning.ex"]} + ], + explanations: [ + check: """ + Every changeset error in `lib/lightning` names the rule it broke with a + `code:` in its options, as Ecto's own validators leave `validation:` or + `constraint:`. Code that reads a changeset can then tell its errors apart + without matching on message text. + + add_error(changeset, :owner, "must have one owner", code: :one_owner) + + validate_change(changeset, :url, fn :url, url -> + if ok?(url), do: [], else: [url: {"is not a URL", code: :format}] + end) + + A validator fn may return its errors from a function in the same module; + the check follows it. + + The API is one such reader: it sends each error with its code, and an + error with neither `code:` nor an Ecto option raises there. Its codes + are listed in `LightningWeb.API.Codes`; pick one from there or add one. + + `lib/lightning_web` is out of scope. + """ + ] + + @block_keys [:do, :else, :after, :rescue, :catch] + @branching [:if, :unless, :case, :cond, :with] + + @doc false + @impl true + def run(%SourceFile{} = source_file, params) do + issue_meta = IssueMeta.for(source_file, params) + ast = SourceFile.ast(source_file) + defs = local_defs(ast) + + {_ast, issues} = + Macro.prewalk(ast, [], fn node, issues -> + {node, check(node, defs, issue_meta) ++ issues} + end) + + Enum.reverse(issues) + end + + defp check({call, meta, args} = _node, defs, issue_meta) + when is_list(args) do + case {call_name(call), args} do + {:add_error, [_ | _]} -> + if coded_opts?(List.last(args)), + do: [], + else: [add_error_issue(issue_meta, meta)] + + {:validate_change, [_, _ | _]} -> + args |> List.last() |> validator_issues(defs, issue_meta, meta) + + _ -> + [] + end + end + + defp check(_node, _defs, _issue_meta), do: [] + + defp call_name({:., _, [_module, name]}) when is_atom(name), do: name + defp call_name(name) when is_atom(name), do: name + defp call_name(_), do: nil + + defp validator_issues({:fn, _, clauses}, defs, issue_meta, meta) do + clauses + |> Enum.flat_map(fn {:->, _, [_head, body]} -> tails(body, defs, []) end) + |> Enum.flat_map(&returned_issues(&1, issue_meta, meta)) + end + + defp validator_issues(_not_fn, _defs, issue_meta, meta), + do: [unreadable_issue(issue_meta, meta)] + + # The expressions a body can return, following branches and calls to + # functions defined in the same file. `seen` stops recursion. + defp tails({:__block__, _, [_ | _] = exprs}, defs, seen), + do: tails(List.last(exprs), defs, seen) + + defp tails({:|>, _, [_lhs, {name, _, _} = rhs]}, defs, seen) + when name in @branching, + do: tails(rhs, defs, seen) + + defp tails({name, _, [_ | _] = args}, defs, seen) when name in @branching do + branches = args |> List.last() |> List.wrap() + + Enum.flat_map(@block_keys, fn key -> + case Keyword.fetch(branches, key) do + :error -> [] + {:ok, [{:->, _, _} | _] = clauses} -> clause_tails(clauses, defs, seen) + {:ok, body} -> tails(body, defs, seen) + end + end) + end + + defp tails({name, _, args} = call, defs, seen) + when is_atom(name) and is_list(args) do + key = {name, length(args)} + + case Map.fetch(defs, key) do + {:ok, bodies} -> + if key in seen, + do: [], + else: Enum.flat_map(bodies, &tails(&1, defs, [key | seen])) + + :error -> + [call] + end + end + + defp tails(expr, _defs, _seen), do: [expr] + + defp clause_tails(clauses, defs, seen) do + Enum.flat_map(clauses, fn {:->, _, [_head, body]} -> + tails(body, defs, seen) + end) + end + + defp returned_issues(nil, _issue_meta, _meta), do: [] + + defp returned_issues(list, issue_meta, meta) when is_list(list) do + list + |> Enum.reject(&coded_error?/1) + |> Enum.map(fn _ -> validate_change_issue(issue_meta, meta) end) + end + + defp returned_issues(_unreadable, issue_meta, meta), + do: [unreadable_issue(issue_meta, meta)] + + defp coded_error?({_field, {_message, opts}}), do: coded_opts?(opts) + defp coded_error?(_), do: false + + defp coded_opts?(opts) when is_list(opts), + do: Enum.any?(opts, &match?({:code, _}, &1)) + + defp coded_opts?(_), do: false + + defp local_defs(ast) do + {_ast, defs} = + Macro.prewalk(ast, %{}, fn + {kind, _, [head, body]} = node, acc when kind in [:def, :defp] -> + case {def_key(head), Keyword.get(List.wrap(body), :do)} do + {nil, _} -> + {node, acc} + + {_, nil} -> + {node, acc} + + {key, do_body} -> + {node, Map.update(acc, key, [do_body], &[do_body | &1])} + end + + node, acc -> + {node, acc} + end) + + defs + end + + defp def_key({:when, _, [head | _]}), do: def_key(head) + + defp def_key({name, _, args}) when is_atom(name), + do: {name, length(List.wrap(args))} + + defp def_key(_), do: nil + + defp add_error_issue(issue_meta, meta) do + format_issue(issue_meta, + message: + "add_error has no `code:`. Pass one in its options, e.g. " <> + "add_error(cs, :field, \"message\", code: :not_found); " <> + "codes are listed in LightningWeb.API.Codes.", + trigger: "add_error", + line_no: meta[:line] + ) + end + + defp validate_change_issue(issue_meta, meta) do + format_issue(issue_meta, + message: + "validate_change returns an error with no `code:`. Return " <> + "{field, {\"message\", code: :format}} instead of {field, \"message\"}; " <> + "codes are listed in LightningWeb.API.Codes.", + trigger: "validate_change", + line_no: meta[:line] + ) + end + + defp unreadable_issue(issue_meta, meta) do + format_issue(issue_meta, + message: + "Can't see the errors this validate_change returns. Pass an fn that " <> + "returns a literal list of {field, {\"message\", code: ...}}, " <> + "directly or from a function in this module.", + trigger: "validate_change", + line_no: meta[:line] + ) + end +end diff --git a/credo/checks/changeset_error_codes.md b/credo/checks/changeset_error_codes.md new file mode 100644 index 00000000000..361881acbac --- /dev/null +++ b/credo/checks/changeset_error_codes.md @@ -0,0 +1,173 @@ +# Changeset error codes: what was tagged with what + +The record of the sweep that gave every changeset error in `lib/lightning` a +`code:` (error-format spec §6.2 and §6.3, plan D3 and D4). Line numbers are at +`7e5c724f1c`, before the sweep. `Lightning.Credo.Check.ChangesetErrorCode` keeps +new errors tagged; this file is not kept up to date after the sweep. + +`add_error` rows name the line the call starts on. `validate_change` rows name +the line of the error the validator returns, which is where the code goes. + +| site | call | error | code | +| ------------------------------------------------------------ | --------------- | --------------------------------------------------------------------------------- | ---------------------- | +| lib/lightning/utils/validators.ex:85 | validate_change | message | format | +| lib/lightning/utils/validators.ex:88 | validate_change | "can't be blank" | required | +| lib/lightning/utils/validators.ex:117 | validate_change | message | format | +| lib/lightning/utils/validators.ex:131 | validate_change | message | format | +| lib/lightning/utils/validators.ex:213 | add_error | message | exclusive | +| lib/lightning/utils/validators.ex:231 | add_error | message | required | +| lib/lightning/utils/validators.ex:254 | add_error | message | required | +| lib/lightning/utils/validators.ex:305 | validate_change | "is not a valid UUID" | format | +| lib/lightning/utils/validators.ex:326 | validate_change | "must be either a http or https URL" | format | +| lib/lightning/utils/validators.ex:329 | validate_change | "host can't be blank" | format | +| lib/lightning/utils/validators.ex:332 | validate_change | "host must be less than 255 characters" | format | +| lib/lightning/utils/validators.ex:335 | validate_change | "host has invalid characters" | format | +| lib/lightning/utils/validators.ex:341 | validate_change | "must be a valid URL" | format | +| lib/lightning/accounts.ex:842 | validate_change | "has not changed" | unchanged | +| lib/lightning/accounts.ex:855 | validate_change | "does not match password" | mismatch | +| lib/lightning/accounts/user.ex:203 | validate_change | "Please accept the terms and conditions to register." | must_accept | +| lib/lightning/accounts/user.ex:245 | validate_change | "has already been taken" | taken | +| lib/lightning/accounts/user.ex:331 | add_error | "did not change" | unchanged | +| lib/lightning/accounts/user.ex:414 | add_error | "is not valid" | mismatch | +| lib/lightning/accounts/user.ex:420 | add_error | "You can't delete a superuser account." | not_allowed | +| lib/lightning/accounts/user.ex:434 | add_error | "This email doesn't match your current email" | mismatch | +| lib/lightning/accounts/user_totp.ex:41 | add_error | "invalid code" | mismatch | +| lib/lightning/adaptors/catalogue_adaptor.ex:127 | add_error | "must be nil when #{ext_field} is nil" | not_allowed | +| lib/lightning/adaptors/catalogue_adaptor.ex:134 | add_error | "must not be nil when #{ext_field} is set" | required | +| lib/lightning/ai_assistant/chat_session.ex:97 | add_error | "must be either 'job_code' or 'workflow_template'" | not_allowed | +| lib/lightning/auth_providers/auth_config_form.ex:75 | add_error | "could not connect to discovery endpoint" | discovery_failed | +| lib/lightning/auth_providers/auth_config_form.ex:78 | add_error | message | discovery_failed | +| lib/lightning/auth_providers/auth_config_form.ex:81 | add_error | "error parsing .well-known" | discovery_failed | +| lib/lightning/auth_providers/auth_config_form.ex:84 | add_error | message | discovery_failed | +| lib/lightning/auth_providers/auth_config_form.ex:88 | add_error | "discovery endpoint must use https" | format | +| lib/lightning/auth_providers/auth_config_form.ex:92 | add_error | "could not fetch the discovery document" | discovery_failed | +| lib/lightning/channels.ex:322 | add_error | Scoping.violation_message(:project_credential_id) | credential_not_allowed | +| lib/lightning/channels/channel_auth_method.ex:78 | add_error | "client auth must use a webhook auth method, not a project credential" | not_allowed | +| lib/lightning/channels/channel_auth_method.ex:89 | add_error | "destination auth must use a project credential, not a webhook auth method" | not_allowed | +| lib/lightning/credentials.ex:532 | add_error | "the owner has no access to these projects: " <> ... | no_access | +| lib/lightning/credentials.ex:580 | add_error | message (external id already used in the project) | taken | +| lib/lightning/credentials.ex:708 | add_error | "adaptor catalogue is not ready yet, try again shortly" | not_ready | +| lib/lightning/credentials.ex:715 | add_error | "Invalid body types" | type | +| lib/lightning/credentials.ex:1897 | add_error | "User doesn't have access to these projects: #{project_names}" | no_access | +| lib/lightning/credentials.ex:1965 | add_error | "You cannot transfer a credential to yourself" | not_allowed | +| lib/lightning/credentials.ex:1972 | add_error | "User does not exist" | not_found | +| lib/lightning/credentials/credential_body.ex:73 | validate_change | message | too_many_secrets | +| lib/lightning/credentials/keychain_credential.ex:66 | validate_change | message | format | +| lib/lightning/credentials/keychain_credential.ex:107 | add_error | "cannot be checked without knowing the project" | required | +| lib/lightning/credentials/keychain_credential.ex:117 | add_error | "must belong to the same project" | not_found | +| lib/lightning/credentials/schema.ex:189 | add_error | error_msg ("expected to be a URI" or "an email") | format | +| lib/lightning/credentials/schema.ex:200 | add_error | "expected to be #{formatted_types}" | type | +| lib/lightning/credentials/schema.ex:207 | add_error | "can't be blank" | required | +| lib/lightning/credentials/schema.ex:212 | add_error | "can't be blank" | required | +| lib/lightning/credentials/schema.ex:217 | add_error | "can't be blank" | required | +| lib/lightning/credentials/schema.ex:225 | add_error | "invalid JSON" | type | +| lib/lightning/credentials/schema.ex:230 | add_error | "must be an object" | type | +| lib/lightning/credentials/scoping.ex:143 | add_error | violation_message(field) | credential_not_allowed | +| lib/lightning/credentials/scoping.ex:166 | add_error | "#{subject}: #{violation_message(field)} (#{field})" | credential_not_allowed | +| lib/lightning/extensions/project_hook.ex:62 | add_error | "#{length(remaining)} stored file(s) could not be deleted from storage" | storage_failed | +| lib/lightning/invocation/dataclip.ex:86 | add_error | "must be a map" | type | +| lib/lightning/invocation/dataclip.ex:139 | add_error | "cannot be set for this type" | not_allowed | +| lib/lightning/invocation/log_line.ex:69 | add_error | "can't be blank" | required | +| lib/lightning/projects/project.ex:130 | add_error | "cannot be self" | not_allowed | +| lib/lightning/projects/project.ex:262 | add_error | "dataclip retention period must be less or equal to the history retention period" | out_of_range | +| lib/lightning/projects/project.ex:337 | add_error | "Every project must have exactly one owner. Please specify one below." | one_owner | +| lib/lightning/projects/project.ex:344 | add_error | "A project can have only one owner." | one_owner | +| lib/lightning/projects/provisioner.ex:312 | add_error | message (Scoping.violation_message) | credential_not_allowed | +| lib/lightning/projects/provisioner.ex:318 | add_error | message (Scoping.violation_message) | credential_not_allowed | +| lib/lightning/projects/provisioner.ex:569 | add_error | message (workflow activation limiter) | exceeds_limit | +| lib/lightning/projects/provisioner.ex:578 | add_error | message (collection creation limiter) | exceeds_limit | +| lib/lightning/projects/provisioner.ex:763 | add_error | "channel deletion is not supported via the provisioning API; ..." | not_supported | +| lib/lightning/projects/provisioner.ex:891 | add_error | "extraneous parameters: %{params}" | unknown_key | +| lib/lightning/projects/provisioner.ex:919 | add_error | "cannot change or add a record while deleting" | deleting | +| lib/lightning/projects/provisioner.ex:935 | add_error | "cannot change or add a record while deleting" | deleting | +| lib/lightning/projects/provisioner.ex:982 | add_error | "No credential found with name #{cred_params["name"]}" | not_found | +| lib/lightning/projects/provisioner.ex:1022 | add_error | "extraneous parameters: %{params}" | unknown_key | +| lib/lightning/runs.ex:303 | validate_change | "must be associated with the run" | not_found | +| lib/lightning/runs.ex:367 | validate_change | "must be associated with the run" | not_found | +| lib/lightning/runs/handlers.ex:311 | add_error | "does not exist" | not_found | +| lib/lightning/runs/handlers.ex:373 | add_error | "does not exist" | not_found | +| lib/lightning/runs/handlers.ex:394 | add_error | "does not exist" | not_found | +| lib/lightning/runs/handlers.ex:489 | add_error | "not found" | not_found | +| lib/lightning/runs/run.ex:243 | add_error | "cannot mark run #{to} that has not been claimed by a worker" | not_allowed | +| lib/lightning/runs/run.ex:252 | add_error | "already in completed state" | not_allowed | +| lib/lightning/version_control/project_repo_connection.ex:94 | validate_change | "please tick the box" | must_accept | +| lib/lightning/version_control/project_repo_connection.ex:144 | add_error | @tree_branch_error (was `reason:`) | tree_branch_conflict | +| lib/lightning/version_control/project_repo_connection.ex:208 | validate_change | "you must specify a path to an existing config file" | required | +| lib/lightning/webhook_auth_methods.ex:680 | add_error | "already scheduled for deletion" | already_scheduled | +| lib/lightning/work_orders.ex:349 | add_error | "cannot retry run using a wiped dataclip" | dataclip_wiped | +| lib/lightning/workflows.ex:549 | add_error | base_message (a rescued Ecto exception) | invalid | +| lib/lightning/workflows/edge.ex:127 | add_error | "source_job_id or source_trigger_id must be present" | required | +| lib/lightning/workflows/edge.ex:197 | add_error | "target_job_id must be different from source_job_id" | cycle | +| lib/lightning/workflows/job.ex:145 | validate_change | "adaptor has invalid format" | format | +| lib/lightning/workflows/job.ex:167 | validate_change | "is not a recognised adaptor" | not_found | +| lib/lightning/workflows/job.ex:170 | validate_change | "adaptor catalogue is not ready yet, try again shortly" | not_ready | +| lib/lightning/workflows/job.ex:193 | add_error | Scoping.violation_message(:keychain_credential_id) | credential_not_allowed | +| lib/lightning/workflows/trigger.ex:276 | add_error | "needs a project, which comes from the trigger's workflow" | required | +| lib/lightning/workflows/trigger.ex:312 | validate_change | "cannot be a UUID" | format | +| lib/lightning/workflows/trigger.ex:330 | validate_change | error_message | format | +| lib/lightning/workflows/trigger.ex:372 | add_error | "must only contain lowercase letters, numbers, hyphens and underscores" | format | +| lib/lightning/workorders/manual.ex:42 | validate_change | "Workflow must be saved first." | not_allowed | +| lib/lightning/workorders/manual.ex:66 | add_error | "Must be an object" | type | +| lib/lightning/workorders/manual.ex:67 | add_error | "Invalid JSON" | type | +| lib/lightning/workorders/manual.ex:82 | add_error | "does not belong to this project" | not_found | + +Two sites carry no code of their own, and the check is disabled on each. In +`lib/lightning/credentials.ex`, `handle_transaction_result/2` re-adds a +credential body's errors onto the credential, and `cast_credential_body_change` +re-adds a typed body's schema errors onto the body. Both keep the options +unchanged, so each error keeps the code its producer gave it, and add `key:` to +place it in the request. + +## Added with the credentials API's problem answers + +OAuth token refusals are not changeset errors where they are made. Each +`Lightning.Credentials.OauthValidation.Error` carries `code:` and the token +`field:` it is about, and `validate_credential_bodies` in +`lib/lightning/credentials.ex` adds it to the credential at +`#/credential_bodies//body/`. The message is the entry's detail, so +none quotes a value from the token. + +| type | field | code | message | +| ------------------------ | ------------- | ----------------- | -------------------------------------------------------------------------------------------------------------- | +| `invalid_token_format` | (the body) | type | "OAuth token must be a valid map" | +| `invalid_token_format` | token_type | type | "token_type must be a string" | +| `missing_access_token` | access_token | required | "Missing required OAuth field: access_token" | +| `invalid_access_token` | access_token | type | "access_token must be a non-empty string" | +| `invalid_access_token` | access_token | required | "access_token cannot be empty" | +| `missing_refresh_token` | refresh_token | required | "Missing required OAuth field: refresh_token" | +| `invalid_refresh_token` | refresh_token | type | "refresh_token must be a non-empty string" | +| `invalid_refresh_token` | refresh_token | required | "refresh_token cannot be empty" | +| `missing_token_type` | token_type | required | "Missing required OAuth field: token_type" | +| `unsupported_token_type` | token_type | not_allowed | "Unsupported token type. Expected 'Bearer'" | +| `missing_scope` | scope | required | "Missing required OAuth field: scope or scopes"; "scope field cannot be empty" | +| `missing_scope` | scopes | required | "scopes array cannot be empty" | +| `invalid_oauth_response` | scope | type | "scope must be a string" | +| `invalid_oauth_response` | scope | required | "OAuth token missing scope information" | +| `invalid_oauth_response` | scopes | type | "scopes must be a list"; "scopes array must contain only strings" | +| `missing_expiration` | expires_in | required | "Missing expiration field: either expires_in or expires_at is required" | +| `invalid_expiration` | expires_in | out_of_range | "expires_in must be greater than 0 seconds"; "expires_in seems unreasonably long" | +| `invalid_expiration` | expires_in | type | "expires_in string must represent a valid positive integer"; "expires_in must be a positive integer (seconds)" | +| `invalid_expiration` | expires_at | out_of_range | "expires_at timestamp appears to be too far in the past (possible clock skew)"; "... too far in the future" | +| `invalid_expiration` | expires_at | format | "expires_at must be a Unix timestamp or ISO 8601 date string" | +| `invalid_expiration` | expires_at | type | "expires_at must be an integer timestamp or date string" | +| `missing_scopes` | scope | scope_not_granted | "Missing required scopes. Please reauthorize and grant all selected permissions." | + +`one_body` is set by hand in +`lib/lightning_web/controllers/api/credential_controller.ex`, outside the check, +on a `PUT` body's `credential_bodies`. + +## Added with the workflows API's graph rules + +`Lightning.Workflows.GraphRules` in `lib/lightning/workflows/graph_rules.ex` +checks a workflow's jobs, triggers and edges for the workflows API. A missing +job or trigger id is Ecto's own `validate_required`, so it needs no code. + +| field | code | message | +| ----------------------- | -------------------- | ----------------------------------------------------------------- | +| triggers | trigger_removed | "a trigger cannot be removed, only edited or added" | +| triggers | one_enabled | "only one trigger may be enabled" | +| edges | missing_trigger_edge | "needs an enabled edge from a trigger" | +| edges | one_enabled_edge | "only one edge from a trigger may be enabled" | +| edges | single_target | "a trigger must have a single target" | +| edges//target_job_id | cycle | "is part of a cycle" | +| edges | cycle | "is part of a cycle", for a stored cycle the request leaves alone | diff --git a/lib/lightning/accounts.ex b/lib/lightning/accounts.ex index 4fdbc5e7481..9e0ca7770c3 100644 --- a/lib/lightning/accounts.ex +++ b/lib/lightning/accounts.ex @@ -171,9 +171,12 @@ defmodule Lightning.Accounts do Scheduling a deletion revokes every token the user has. Disabling them, or changing their password, email, role or support access, signs them out of every session but keeps their API tokens. + + Returns `{:error, :concurrent_edit}` when another request deleted the user + while this one ran. """ @spec update_user(User.t(), map(), User.t() | ServiceAccount.t() | nil) :: - {:ok, User.t()} | {:error, Changeset.t()} + {:ok, User.t()} | {:error, Changeset.t() | :concurrent_edit} def update_user(%User{} = user, attrs, actor \\ nil) do user |> User.details_changeset(attrs) |> write_user(actor) end @@ -182,7 +185,7 @@ defmodule Lightning.Accounts do revoke_contexts = revoke_contexts(changeset) Multi.new() - |> Multi.update(:user, changeset) + |> Multi.update(:user, changeset, stale_error_field: :id) |> maybe_audit_update(changeset, actor) |> maybe_revoke_tokens(user, revoke_contexts) |> Repo.transaction() @@ -196,6 +199,9 @@ defmodule Lightning.Accounts do {:ok, updated} + {:error, :user, %Changeset{errors: [id: {_, stale: true}]}, _} -> + {:error, :concurrent_edit} + {:error, _step, changeset, _changes} -> {:error, changeset} end @@ -839,7 +845,7 @@ defmodule Lightning.Accounts do defp validate_email_changed(changeset, user) do Changeset.validate_change(changeset, :email, fn :email, email -> if user.email == email do - [email: "has not changed"] + [email: {"has not changed", code: :unchanged}] else [] end @@ -852,7 +858,7 @@ defmodule Lightning.Accounts do if User.valid_password?(user, password) do [] else - [current_password: "does not match password"] + [current_password: {"does not match password", code: :mismatch}] end end) end diff --git a/lib/lightning/accounts/user.ex b/lib/lightning/accounts/user.ex index ddaa0ce091b..4bbd9d42831 100644 --- a/lib/lightning/accounts/user.ex +++ b/lib/lightning/accounts/user.ex @@ -200,7 +200,11 @@ defmodule Lightning.Accounts.User do if terms_accepted do [] else - [terms_accepted: "Please accept the terms and conditions to register."] + [ + terms_accepted: + {"Please accept the terms and conditions to register.", + code: :must_accept} + ] end end) end @@ -242,7 +246,10 @@ defmodule Lightning.Accounts.User do changeset |> validate_change(:email, fn :email, email -> if Lightning.Repo.exists?(User |> where(email: ^email)) do - [email: {"has already been taken", validation: :unsafe_unique}] + [ + email: + {"has already been taken", validation: :unsafe_unique, code: :taken} + ] else [] end @@ -327,8 +334,11 @@ defmodule Lightning.Accounts.User do |> cast(attrs, [:email]) |> validate_email() |> case do - %{changes: %{email: _}} = changeset -> changeset - %{} = changeset -> add_error(changeset, :email, "did not change") + %{changes: %{email: _}} = changeset -> + changeset + + %{} = changeset -> + add_error(changeset, :email, "did not change", code: :unchanged) end end @@ -411,7 +421,7 @@ defmodule Lightning.Accounts.User do if valid_password?(changeset.data, password) do changeset else - add_error(changeset, :current_password, "is not valid") + add_error(changeset, :current_password, "is not valid", code: :mismatch) end end @@ -420,7 +430,8 @@ defmodule Lightning.Accounts.User do add_error( changeset, :scheduled_deletion_email, - "You can't delete a superuser account." + "You can't delete a superuser account.", + code: :not_allowed ) else changeset @@ -434,7 +445,8 @@ defmodule Lightning.Accounts.User do add_error( changeset, :scheduled_deletion_email, - "This email doesn't match your current email" + "This email doesn't match your current email", + code: :mismatch ) end end diff --git a/lib/lightning/accounts/user_totp.ex b/lib/lightning/accounts/user_totp.ex index 1ccf6986334..305122aad90 100644 --- a/lib/lightning/accounts/user_totp.ex +++ b/lib/lightning/accounts/user_totp.ex @@ -38,7 +38,7 @@ defmodule Lightning.Accounts.UserTOTP do if changeset.valid? and valid_totp?(totp, code) do changeset else - add_error(changeset, :code, "invalid code") + add_error(changeset, :code, "invalid code", code: :mismatch) end end diff --git a/lib/lightning/actor.ex b/lib/lightning/actor.ex new file mode 100644 index 00000000000..3e4bfc28007 --- /dev/null +++ b/lib/lightning/actor.ex @@ -0,0 +1,8 @@ +defmodule Lightning.Actor do + @moduledoc """ + Whoever a context function records as having done something: a person or a + service account. + """ + + @type t :: Lightning.Accounts.User.t() | Lightning.ServiceAccount.t() +end diff --git a/lib/lightning/adaptors/catalogue_adaptor.ex b/lib/lightning/adaptors/catalogue_adaptor.ex index dd07c454cc2..ab91900e620 100644 --- a/lib/lightning/adaptors/catalogue_adaptor.ex +++ b/lib/lightning/adaptors/catalogue_adaptor.ex @@ -127,14 +127,16 @@ defmodule Lightning.Adaptors.Catalogue.Adaptor do add_error( changeset, sha_field, - "must be nil when #{ext_field} is nil" + "must be nil when #{ext_field} is nil", + code: :not_allowed ) {_ext, nil} -> add_error( changeset, sha_field, - "must not be nil when #{ext_field} is set" + "must not be nil when #{ext_field} is set", + code: :required ) {_ext, _sha} -> diff --git a/lib/lightning/ai_assistant/ai_assistant.ex b/lib/lightning/ai_assistant/ai_assistant.ex index beff2f37824..05d9d447a03 100644 --- a/lib/lightning/ai_assistant/ai_assistant.ex +++ b/lib/lightning/ai_assistant/ai_assistant.ex @@ -326,6 +326,22 @@ defmodule Lightning.AiAssistant do end end + @doc """ + Returns a job-code session for the job, matching either its saved `job_id` or + the id of the unsaved job in its `meta`, or `nil` when there is none. + """ + @spec unsaved_job_session(Ecto.UUID.t()) :: ChatSession.t() | nil + def unsaved_job_session(job_id) do + from(s in ChatSession, + where: s.session_type == "job_code", + where: + s.job_id == ^job_id or + fragment("? -> 'unsaved_job' ->> 'id' = ?", s.meta, ^job_id), + limit: 1 + ) + |> Repo.one() + end + defp update_session_if_job_exists(session, job_ids, count) do unsaved_job = session.meta["unsaved_job"] unsaved_job_id = unsaved_job["id"] diff --git a/lib/lightning/ai_assistant/chat_session.ex b/lib/lightning/ai_assistant/chat_session.ex index 521891f7cda..26fe318d2cb 100644 --- a/lib/lightning/ai_assistant/chat_session.ex +++ b/lib/lightning/ai_assistant/chat_session.ex @@ -100,7 +100,8 @@ defmodule Lightning.AiAssistant.ChatSession do add_error( changeset, :session_type, - "must be either 'job_code' or 'workflow_template'" + "must be either 'job_code' or 'workflow_template'", + code: :not_allowed ) end end diff --git a/lib/lightning/auditing/audit.ex b/lib/lightning/auditing/audit.ex index 6f3899999df..844fbe37d70 100644 --- a/lib/lightning/auditing/audit.ex +++ b/lib/lightning/auditing/audit.ex @@ -7,6 +7,8 @@ defmodule Lightning.Auditing.Audit do require Ecto.Query require Logger + @type t :: %__MODULE__{} + @doc """ Callback to update changes before saving to the database. """ diff --git a/lib/lightning/auth_providers/auth_config_form.ex b/lib/lightning/auth_providers/auth_config_form.ex index ab944475534..af9ce20c3eb 100644 --- a/lib/lightning/auth_providers/auth_config_form.ex +++ b/lib/lightning/auth_providers/auth_config_form.ex @@ -72,24 +72,33 @@ defmodule Lightning.AuthProviders.AuthConfigForm do |> case do {:error, %{reason: :econnrefused}} -> changeset - |> add_error(:discovery_url, "could not connect to discovery endpoint") + |> add_error(:discovery_url, "could not connect to discovery endpoint", + code: :discovery_failed + ) {:error, %HTTPoison.Error{reason: message}} when is_binary(message) -> - changeset |> add_error(:discovery_url, message) + changeset |> add_error(:discovery_url, message, code: :discovery_failed) {:error, %Jason.DecodeError{}} -> - changeset |> add_error(:discovery_url, "error parsing .well-known") + changeset + |> add_error(:discovery_url, "error parsing .well-known", + code: :discovery_failed + ) {:error, message} when is_binary(message) -> - changeset |> add_error(:discovery_url, message) + changeset |> add_error(:discovery_url, message, code: :discovery_failed) {:error, :insecure_discovery_url} -> changeset - |> add_error(:discovery_url, "discovery endpoint must use https") + |> add_error(:discovery_url, "discovery endpoint must use https", + code: :format + ) {:error, _reason} -> changeset - |> add_error(:discovery_url, "could not fetch the discovery document") + |> add_error(:discovery_url, "could not fetch the discovery document", + code: :discovery_failed + ) {:ok, _provider} -> changeset diff --git a/lib/lightning/channels.ex b/lib/lightning/channels.ex index e2f019f17dd..107c131c6a5 100644 --- a/lib/lightning/channels.ex +++ b/lib/lightning/channels.ex @@ -220,7 +220,7 @@ defmodule Lightning.Channels do Updates a channel's config fields, bumping lock_version. """ @spec update_channel(Channel.t(), map(), actor: User.t()) :: - {:ok, Channel.t()} | {:error, Ecto.Changeset.t()} + {:ok, Channel.t()} | {:error, Ecto.Changeset.t() | :concurrent_edit} def update_channel(%Channel{} = channel, attrs, actor: %User{} = actor) do changeset = Channel.changeset(channel, attrs) @@ -240,9 +240,18 @@ defmodule Lightning.Channels do end) |> Repo.transaction() |> case do - {:ok, %{channel: channel}} -> {:ok, channel} - {:error, :channel, changeset, _} -> {:error, changeset} - {:error, :credential_scope_check, changeset, _} -> {:error, changeset} + {:ok, %{channel: channel}} -> + {:ok, channel} + + {:error, :channel, + %Ecto.Changeset{errors: [lock_version: {_, stale: true}]}, _} -> + {:error, :concurrent_edit} + + {:error, :channel, changeset, _} -> + {:error, changeset} + + {:error, :credential_scope_check, changeset, _} -> + {:error, changeset} end end @@ -322,7 +331,8 @@ defmodule Lightning.Channels do Ecto.Changeset.add_error( dest, :project_credential_id, - Scoping.violation_message(:project_credential_id) + Scoping.violation_message(:project_credential_id), + code: :credential_not_allowed ) changeset diff --git a/lib/lightning/channels/channel_auth_method.ex b/lib/lightning/channels/channel_auth_method.ex index c30c452ae36..e4769953aa3 100644 --- a/lib/lightning/channels/channel_auth_method.ex +++ b/lib/lightning/channels/channel_auth_method.ex @@ -78,7 +78,8 @@ defmodule Lightning.Channels.ChannelAuthMethod do add_error( changeset, :project_credential_id, - "client auth must use a webhook auth method, not a project credential" + "client auth must use a webhook auth method, not a project credential", + code: :not_allowed ) else changeset @@ -89,7 +90,8 @@ defmodule Lightning.Channels.ChannelAuthMethod do add_error( changeset, :webhook_auth_method_id, - "destination auth must use a project credential, not a webhook auth method" + "destination auth must use a project credential, not a webhook auth method", + code: :not_allowed ) else changeset diff --git a/lib/lightning/collaboration/session.ex b/lib/lightning/collaboration/session.ex index 8602db001f6..5632d47d835 100644 --- a/lib/lightning/collaboration/session.ex +++ b/lib/lightning/collaboration/session.ex @@ -222,6 +222,9 @@ defmodule Lightning.Collaboration.Session do save's transaction, so the whole save rolled back and nothing persisted - `{:error, :adaptor_catalogue_unavailable}` - The adaptor catalogue's first load did not complete, so no validation could run + - `{:error, :concurrent_edit}` - Another save changed the workflow while + this one ran + - `{:error, :read_only_view}` - The session is a read-only view - `{:error, changeset}` - Validation or persistence error ## Examples @@ -240,6 +243,8 @@ defmodule Lightning.Collaboration.Session do | :deserialization_failed | :internal_error | :adaptor_catalogue_unavailable + | :concurrent_edit + | :read_only_view | Ecto.Changeset.t()} def save_workflow(session_pid, user) do GenServer.call( @@ -569,6 +574,13 @@ defmodule Lightning.Collaboration.Session do {:reply, {:error, :snapshot_failed}, state} + {:error, :concurrent_edit} -> + Logger.info( + "Cannot save workflow #{state.workflow.id}: another save changed it first" + ) + + {:reply, {:error, :concurrent_edit}, state} + {:error, %Ecto.Changeset{} = changeset} -> all_errors = Ecto.Changeset.traverse_errors(changeset, fn {msg, opts} -> diff --git a/lib/lightning/collections.ex b/lib/lightning/collections.ex index 0bccf3104d7..516c2229176 100644 --- a/lib/lightning/collections.ex +++ b/lib/lightning/collections.ex @@ -4,13 +4,19 @@ defmodule Lightning.Collections do """ import Ecto.Query + alias Ecto.Changeset alias Ecto.Multi + alias Lightning.Accounts.User + alias Lightning.Collections.Audit alias Lightning.Collections.Collection alias Lightning.Collections.Item + alias Lightning.Config alias Lightning.Extensions.Message alias Lightning.Projects.Project alias Lightning.Repo + alias Lightning.Run + alias Lightning.Scope alias Lightning.Services.CollectionHook @doc """ @@ -55,21 +61,9 @@ defmodule Lightning.Collections do Repo.all(query) end - @doc """ - Looks up a collection by name across all projects. - - Returns `{:error, :conflict}` when the name exists in more than one - project; the caller should then retry with `get_collection/2`. - """ - @spec get_collection(String.t()) :: - {:ok, Collection.t()} | {:error, :not_found} | {:error, :conflict} - def get_collection(name) do - case Repo.all(from c in Collection, where: c.name == ^name) do - [] -> {:error, :not_found} - [collection] -> {:ok, collection} - [_ | _] -> {:error, :conflict} - end - end + @doc "The collection with primary key `id`, or `nil`." + @spec get_collection_by_id(Ecto.UUID.t()) :: Collection.t() | nil + def get_collection_by_id(id), do: Repo.get(Collection, id) @doc """ Looks up a collection scoped to a specific project. @@ -87,30 +81,85 @@ defmodule Lightning.Collections do end @doc """ - Creates a new collection with the given attributes. + The collection `name` names for `subject`, among the collections it has + standing on: a user's standing is a project role or support access, a run's + is its own project. `project_id`, as the request sent it, narrows the search + to one project; one that isn't a UUID names nothing. + + A collection the subject has no standing on never counts, so neither + `{:error, :not_found}` nor `{:error, :ambiguous_name}` says anything about + projects it can't see. Whether the subject may act on the collection is still + the policy's call. + """ + @spec resolve_collection( + User.t() | Run.t(), + String.t(), + String.t() | nil + ) :: + {:ok, Collection.t()} | {:error, :not_found | :ambiguous_name} + def resolve_collection(subject, name, project_id \\ nil) do + case named(name, project_id) do + nil -> {:error, :not_found} + query -> query |> standing_collections(subject) |> only_one() + end + end - ## Parameters + defp only_one([]), do: {:error, :not_found} + defp only_one([collection]), do: {:ok, collection} + defp only_one([_ | _]), do: {:error, :ambiguous_name} + + defp named(name, nil), do: from(c in Collection, where: c.name == ^name) + + defp named(name, project_id) do + if Lightning.Validators.valid_uuid?(project_id), + do: from(c in named(name, nil), where: c.project_id == ^project_id) + end + + defp standing_collections(query, %Run{} = run) do + project_id = + run |> Ecto.assoc([:work_order, :workflow]) |> select([w], w.project_id) + + query |> where([c], c.project_id in subquery(project_id)) |> Repo.all() + end + + defp standing_collections(query, %User{} = user) do + collections = Repo.all(query) + scopes = Scope.fetch_all(user, Enum.map(collections, & &1.project_id)) + + standing = + for {project_id, scope} <- scopes, + Scope.has_standing?(scope), + into: MapSet.new(), + do: project_id + + Enum.filter(collections, &(&1.project_id in standing)) + end + + @doc """ + Creates a collection, recording `actor` as having created it. + + Returns `{:error, :exceeds_limit, message}`, and writes nothing, when the + collection hook refuses the create. + + ## Options - - `attrs`: A map of attributes to create the collection. + * `:id` - The id to create the collection under, rather than a new one ## Examples - iex> create_collection(%{name: "New Collection", project_id: "a7895d29-02a9-42cd-845b-7ad893557c24"}) + iex> create_collection(%{"name" => "patients", "project_id" => project_id}, actor) {:ok, %Collection{}} - iex> create_collection(%{name: nil, project_id: "a7895d29-02a9-42cd-845b-7ad893557c24"}}) + iex> create_collection(%{"name" => nil, "project_id" => project_id}, actor) {:error, %Ecto.Changeset{}} - - ## Returns - - - `{:ok, %Collection{}}` on success. - - `{:error, %Ecto.Changeset{}}` on failure due to validation errors. """ - @spec create_collection(map()) :: + @spec create_collection(map(), Lightning.Actor.t(), [{:id, Ecto.UUID.t()}]) :: {:ok, Collection.t()} | {:error, Ecto.Changeset.t()} | {:error, :exceeds_limit, Message.t()} - def create_collection(attrs) do + def create_collection(attrs, actor, opts \\ []) do + changeset = Collection.changeset(%Collection{id: opts[:id]}, attrs) + Multi.new() |> Multi.run(:limiter, fn _repo, _changes -> case CollectionHook.handle_create(attrs) do @@ -118,42 +167,68 @@ defmodule Lightning.Collections do {:error, :exceeds_limit, message} -> {:error, message} end end) - |> Multi.insert(:create, Collection.changeset(%Collection{}, attrs)) + |> Multi.insert(:create, changeset) + |> Multi.insert(:audit, fn %{create: collection} -> + Audit.event("created", collection.id, actor, changeset) + end) |> Repo.transaction() |> case do - {:ok, %{create: collection}} -> {:ok, collection} - {:error, :limiter, message, _changes} -> {:error, :exceeds_limit, message} - {:error, _op, changeset, _changes} -> {:error, changeset} + {:ok, %{create: collection}} -> + {:ok, collection} + + {:error, :limiter, message, _changes} -> + {:error, :exceeds_limit, message} + + {:error, :create, changeset, _changes} -> + {:error, changeset} + + {:error, :audit, changeset, _changes} -> + raise Ecto.InvalidChangesetError, action: :insert, changeset: changeset end end @doc """ - Updates an existing collection with the given attributes. - - ## Parameters + Renames a collection, recording `actor` as having renamed it. A write that + leaves the name as it was records nothing. - - `collection`: The existing `%Collection{}` struct to update. - - `attrs`: A map of attributes to update the collection. + Returns `{:error, :concurrent_edit}` when another request deleted the + collection while this one ran. ## Examples - iex> update_collection(collection, %{name: "Updated Name"}) + iex> update_collection(collection, %{"name" => "patients"}, actor) {:ok, %Collection{}} - iex> update_collection(collection, %{name: nil}) + iex> update_collection(collection, %{"name" => nil}, actor) {:error, %Ecto.Changeset{}} + """ + @spec update_collection(Collection.t(), map(), Lightning.Actor.t()) :: + {:ok, Collection.t()} | {:error, Ecto.Changeset.t() | :concurrent_edit} + def update_collection(collection, attrs, actor) do + changeset = Collection.rename_changeset(collection, attrs) - ## Returns + Multi.new() + |> Multi.update(:update, changeset, stale_error_field: :id) + |> then(fn multi -> + case Audit.event("updated", collection.id, actor, changeset) do + :no_changes -> multi + event -> Multi.insert(multi, :audit, event) + end + end) + |> Repo.transaction() + |> case do + {:ok, %{update: collection}} -> + {:ok, collection} - - `{:ok, %Collection{}}` on success. - - `{:error, %Ecto.Changeset{}}` on failure due to validation errors. - """ - @spec update_collection(Collection.t(), map()) :: - {:ok, Collection.t()} | {:error, Ecto.Changeset.t()} - def update_collection(collection, attrs) do - collection - |> Collection.changeset(attrs) - |> Repo.update() + {:error, :update, %Changeset{errors: [id: {_, stale: true}]}, _} -> + {:error, :concurrent_edit} + + {:error, :update, changeset, _changes} -> + {:error, changeset} + + {:error, :audit, changeset, _changes} -> + raise Ecto.InvalidChangesetError, action: :insert, changeset: changeset + end end @spec delete_collection(Ecto.UUID.t()) :: @@ -161,7 +236,7 @@ defmodule Lightning.Collections do | {:error, Ecto.Changeset.t()} | {:error, :not_found} def delete_collection(collection_id) do - case Repo.get(Collection, collection_id) do + case get_collection_by_id(collection_id) do nil -> {:error, :not_found} @@ -203,28 +278,155 @@ defmodule Lightning.Collections do |> Repo.all() end - @spec put(Collection.t(), String.t(), String.t()) :: - :ok | {:error, Ecto.Changeset.t()} + @doc """ + Every item `get_all/3` would find for `filters` and `key_pattern`, from + `filters`' cursor on and at most `filters`' `:limit` of them when it has one. + Items are read a page at a time, so that no transaction is held while the + caller consumes the stream. + """ + @spec stream_all(Collection.t(), map(), String.t() | nil) :: Enumerable.t() + def stream_all(collection, filters, key_pattern \\ nil) do + {limit, filters} = Map.pop(filters, :limit) + max_page_size = Config.collections_stream_page_size() + page_size = if limit, do: min(limit, max_page_size), else: max_page_size + filters = Map.put(filters, :limit, page_size) + + Map.get(filters, :cursor) + |> Stream.unfold(fn + :drained -> + nil + + cursor -> + case get_all(collection, Map.put(filters, :cursor, cursor), key_pattern) do + [] -> nil + page when length(page) < page_size -> {page, :drained} + page -> {page, List.last(page).id} + end + end) + |> Stream.flat_map(& &1) + |> then(&if(limit, do: Stream.take(&1, limit), else: &1)) + end + + @doc """ + Writes `value` under `key`. The item is validated before the collection hook + is asked about its size, so an invalid item is refused as invalid, never as + over a limit. + + A bad value or a key that can't be stored is refused with one changeset + holding every error. The key's errors are on `:key`. + """ + @spec put(Collection.t(), String.t(), term()) :: + :ok + | {:error, Changeset.t()} + | {:error, :exceeds_limit, Message.t()} def put(collection, key, value) do - case get(collection, key) do - nil -> - upsert_item(collection, %Item{collection_id: collection.id}, key, value) + item = get(collection, key) || %Item{collection_id: collection.id} - item -> - upsert_item(collection, item, key, value) - end + changeset = + item + |> Item.value_changeset(%{value: value}) + |> Item.key_changeset(%{key: key}) + + if changeset.valid?, + do: upsert_item(collection, changeset), + else: {:error, changeset} end - @spec put_all(Collection.t(), [{String.t(), String.t()}]) :: + @doc """ + Writes every item of `items`, the request's list of + `%{"key" => _, "value" => _}` objects, in one transaction. + + A list with any bad item is refused whole, writing nothing, with a changeset + whose errors carry each bad item's position in `items` as their `key:`. An + item is bad if it isn't an object or fails validation, or if an earlier item + already uses its key, and then the error names the earlier one. The + collection hook is asked about the size only once every item is valid. + """ + @spec put_all(Collection.t(), term()) :: {:ok, non_neg_integer()} - | {:error, :duplicate_key} + | {:error, Changeset.t()} | {:error, :exceeds_limit, Message.t()} - def put_all(collection, kv_list) do + def put_all(collection, items) do + with {:ok, items} <- validate_items(collection, items) do + insert_items(collection, items) + end + end + + defp validate_items(collection, items) when is_list(items) do + {checked, _firsts} = + items + |> Enum.with_index() + |> Enum.map_reduce(%{}, fn {item, index}, firsts -> + key = request_key(item) + first = if key, do: Map.get(firsts, key) + firsts = if key, do: Map.put_new(firsts, key, index), else: firsts + + {check_item(collection, item, index, first), firsts} + end) + + case Enum.flat_map(checked, fn {_item, errors} -> errors end) do + [] -> {:ok, Enum.map(checked, fn {item, []} -> item end)} + errors -> refuse_items(errors) + end + end + + defp validate_items(_collection, nil), + do: refuse_items(items: {"can't be blank", validation: :required}) + + defp validate_items(_collection, _items), + do: + refuse_items( + items: {"is invalid", validation: :cast, type: {:array, :map}} + ) + + defp request_key(%{"key" => key}) when is_binary(key), do: key + defp request_key(_item), do: nil + + # `{item, errors}`, each error placed at this item's position by `key:`. + # `first` is the position of the earlier item holding the same key. + defp check_item(collection, item, index, first) when is_map(item) do + changeset = + Item.changeset( + %Item{collection_id: collection.id}, + Map.take(item, ["key", "value"]) + ) + + own = + for {field, {message, opts}} <- Enum.reverse(changeset.errors), + do: {:items, {message, [{:key, ["items", index, field]} | opts]}} + + duplicate = + if first, + do: [ + items: + {"names the same key as #/items/#{first}", + code: :duplicate, key: ["items", index, "key"]} + ], + else: [] + + {Changeset.apply_changes(changeset), own ++ duplicate} + end + + defp check_item(_collection, _item, index, _first) do + {nil, + [ + items: + {"is invalid", key: ["items", index], validation: :cast, type: :map} + ]} + end + + # A changeset holds its errors newest first, as add_error leaves them. + defp refuse_items(errors) do + changeset = Changeset.change({%{}, %{items: {:array, :map}}}) + {:error, %{changeset | errors: Enum.reverse(errors), valid?: false}} + end + + defp insert_items(collection, items) do now = DateTime.utc_now() {item_list, {key_list, new_used_mem}} = - Enum.map_reduce(kv_list, {[], 0}, fn %{"key" => key, "value" => value}, - {key_list, mem_used} -> + Enum.map_reduce(items, {[], 0}, fn %Item{key: key, value: value}, + {key_list, mem_used} -> { %{ collection_id: collection.id, @@ -276,13 +478,6 @@ defmodule Lightning.Collections do {:error, :hook, message, _changes} -> {:error, :exceeds_limit, message} {:error, _op, changeset, _changes} -> {:error, changeset} end - rescue - e in Postgrex.Error -> - if e.postgres.code == :cardinality_violation do - {:error, :duplicate_key} - else - reraise e, __STACKTRACE__ - end end @spec delete(Collection.t(), String.t()) :: :ok | {:error, :not_found} @@ -347,39 +542,25 @@ defmodule Lightning.Collections do end end - defp upsert_item( - collection, - %Item{value: old_value} = item, - key, - value - ) do + defp upsert_item(collection, %Changeset{data: %Item{} = stored} = changeset) do + %Item{key: key, value: value} = Changeset.apply_changes(changeset) + + increment_size = + if is_nil(stored.value), + do: byte_size(key) + byte_size(value), + else: byte_size(value) - byte_size(stored.value) + Multi.new() - |> Multi.run(:increment_size, fn _repo, _changes -> - if is_nil(old_value) do - {:ok, byte_size(key) + byte_size(value)} - else - {:ok, byte_size(value) - byte_size(old_value)} - end - end) - |> Multi.run(:limiter, fn _repo, %{increment_size: increment_size} -> + |> Multi.run(:limiter, fn _repo, _changes -> handle_put_items(collection, increment_size) end) - |> Multi.insert( - :item, - fn _no_change -> - Item.changeset(item, %{ - key: key, - value: value - }) - end, + |> Multi.insert(:item, changeset, conflict_target: [:collection_id, :key], - on_conflict: [ - set: [value: value, updated_at: DateTime.utc_now()] - ] + on_conflict: [set: [value: value, updated_at: DateTime.utc_now()]] ) |> Multi.update_all( :collection, - fn %{item: %{collection_id: collection_id}, increment_size: increment_size} -> + fn %{item: %{collection_id: collection_id}} -> from(c in Collection, where: c.id == ^collection_id, update: [inc: ^[byte_size_sum: increment_size]] diff --git a/lib/lightning/collections/audit.ex b/lib/lightning/collections/audit.ex new file mode 100644 index 00000000000..6ca0af676fe --- /dev/null +++ b/lib/lightning/collections/audit.ex @@ -0,0 +1,9 @@ +defmodule Lightning.Collections.Audit do + @moduledoc """ + Audit events for collections: who created one, and who renamed it. + """ + use Lightning.Auditing.Audit, + repo: Lightning.Repo, + item: "collection", + events: ["created", "updated"] +end diff --git a/lib/lightning/collections/collection.ex b/lib/lightning/collections/collection.ex index fc6b947b2f2..11969ff2d18 100644 --- a/lib/lightning/collections/collection.ex +++ b/lib/lightning/collections/collection.ex @@ -27,26 +27,29 @@ defmodule Lightning.Collections.Collection do timestamps() end - @doc false + @doc """ + Builds a new collection from its project and name. The id is never cast, so + a caller that wants a particular id puts it on the struct. + """ def changeset(entry, attrs) do entry |> cast(attrs, [:project_id, :name]) |> validate_required([:project_id, :name]) |> validate() + |> unique_constraint(:id, name: :collections_pkey) end - def validate(changeset) do - changeset - |> validate_format(:name, ~r/^[a-z0-9]+([\-_.][a-z0-9]+)*$/, - message: "Collection name must be URL safe" - ) - |> unique_constraint(:name, - name: :collections_project_id_name_index, - message: "A collection with this name already exists" - ) + @doc """ + Changes a collection's name. Only the name is cast, so it never moves project. + """ + def rename_changeset(collection, attrs) do + collection + |> cast(attrs, [:name]) + |> validate_required([:name]) + |> validate() end - defp validate_changeset(changeset) do + def validate(changeset) do changeset |> validate_format(:name, ~r/^[a-z0-9]+([\-_.][a-z0-9]+)*$/, message: "Collection name must be URL safe" @@ -71,6 +74,6 @@ defmodule Lightning.Collections.Collection do |> put_change(:name, Lightning.Helpers.url_safe_name(raw_name)) end end) - |> validate_changeset() + |> validate() end end diff --git a/lib/lightning/collections/item.ex b/lib/lightning/collections/item.ex index a36e0661148..fe01a72c693 100644 --- a/lib/lightning/collections/item.ex +++ b/lib/lightning/collections/item.ex @@ -35,14 +35,43 @@ defmodule Lightning.Collections.Item do @doc false def changeset(entry, attrs) do entry - |> cast(attrs, [:collection_id, :key, :value]) - |> validate_required([:collection_id, :key, :value]) - |> validate_length(:value, max: 1_000_000) - |> Lightning.Validators.validate_name_fits_column( - :value, - "value is too long, please use a shorter one", - 1_000_000 - ) + |> value_changeset(attrs) + |> key_changeset(attrs) + end + + @doc false + def value_changeset(entry, attrs) do + entry + |> cast(attrs, [:value]) + |> validate_required([:value]) + |> validate_value_length() + end + + @max_value_length 1_000_000 + + # A string has no more graphemes or codepoints than bytes, so a value that + # fits in bytes needn't have either counted. + defp validate_value_length(changeset) do + case get_change(changeset, :value) do + value when is_binary(value) and byte_size(value) > @max_value_length -> + changeset + |> validate_length(:value, max: @max_value_length) + |> Lightning.Validators.validate_name_fits_column( + :value, + "value is too long, please use a shorter one", + @max_value_length + ) + + _fits -> + changeset + end + end + + @doc false + def key_changeset(entry, attrs) do + entry + |> cast(attrs, [:collection_id, :key]) + |> validate_required([:collection_id, :key]) |> Lightning.Validators.validate_name_fits_column( :key, "key is too long, please use a shorter one" diff --git a/lib/lightning/config.ex b/lib/lightning/config.ex index 92391159585..bc5989187ec 100644 --- a/lib/lightning/config.ex +++ b/lib/lightning/config.ex @@ -86,6 +86,18 @@ defmodule Lightning.Config do Application.get_env(:lightning, :activity_cleanup_chunk_size) end + @impl true + def collections_stream_page_size do + Application.get_env(:lightning, Lightning.Collections, []) + |> Keyword.fetch!(:stream_page_size) + end + + @impl true + def collections_default_stream_limit do + Application.get_env(:lightning, :collections_api, []) + |> Keyword.fetch!(:default_stream_limit) + end + @impl true def log_lines_search_indexing_batch_size do log_lines_search_indexing_config() |> Keyword.fetch!(:batch_size) @@ -434,6 +446,8 @@ defmodule Lightning.Config do @callback promex_enabled?() :: boolean() @callback purge_deleted_after_days() :: integer() @callback activity_cleanup_chunk_size() :: integer() + @callback collections_stream_page_size() :: pos_integer() + @callback collections_default_stream_limit() :: non_neg_integer() @callback log_lines_search_indexing_batch_size() :: pos_integer() @callback log_lines_search_indexing_max_batches() :: pos_integer() @callback dataclip_search_indexing_batch_size() :: pos_integer() @@ -539,6 +553,14 @@ defmodule Lightning.Config do impl().activity_cleanup_chunk_size() end + def collections_stream_page_size do + impl().collections_stream_page_size() + end + + def collections_default_stream_limit do + impl().collections_default_stream_limit() + end + def log_lines_search_indexing_batch_size do impl().log_lines_search_indexing_batch_size() end diff --git a/lib/lightning/credentials.ex b/lib/lightning/credentials.ex index e27d48c0178..52e058d30fd 100644 --- a/lib/lightning/credentials.ex +++ b/lib/lightning/credentials.ex @@ -153,6 +153,21 @@ defmodule Lightning.Credentials do def get_credential(id), do: Repo.get(Credential, id) + @doc """ + Loads the credential's project links. + """ + @spec preload_links(Credential.t()) :: Credential.t() + def preload_links(%Credential{} = credential), + do: Repo.preload(credential, :project_credentials) + + @doc """ + Loads the credential's project links and the projects they name, fresh + even when either was loaded before. + """ + @spec preload_links_and_projects(Credential.t()) :: Credential.t() + def preload_links_and_projects(%Credential{} = credential), + do: Repo.preload(credential, [:project_credentials, :projects], force: true) + @doc """ Gets a credential body for a specific environment. @@ -187,32 +202,43 @@ defmodule Lightning.Credentials do * `name` - Environment name (e.g., "production", "staging") * `body` - Credential configuration data * `expected_scopes` - List of expected scopes (for OAuth credentials) + * `actor` - Who is asking: the owner, or a service account writing for them + * `opts`: + * `:id` - The id to create the credential under, rather than a new one ## Returns * `{:ok, credential}` - Successfully created credential - * `{:error, error}` - Error with creation process + * `{:error, changeset}` - With an error on `project_credentials` if the + owner has no access to a project it links to. A refused body, OAuth token + included, puts its errors on `credential_bodies`, each with a `key:` + option naming its place in `attrs`, as + `[:credential_bodies, index, :body, key]` + * `{:error, :unauthorized}` - `actor` may not create it for its owner """ - @spec create_credential(map(), User.t()) :: - {:ok, Credential.t()} | {:error, any()} - def create_credential(attrs, %User{} = actor) do + @spec create_credential(map(), Lightning.Actor.t(), [{:id, Ecto.UUID.t()}]) :: + {:ok, Credential.t()} + | {:error, :unauthorized | Ecto.Changeset.t()} + def create_credential(attrs, actor, opts \\ []) do attrs = normalize_keys(attrs) with :ok <- - authorize_credential_owner(actor, %Credential{ + authorize_credential(:edit_credential, actor, %Credential{ user_id: attrs["user_id"] }) do - do_create_credential(attrs) + do_create_credential(attrs, actor, opts[:id]) end end - defp do_create_credential(attrs) do + defp do_create_credential(attrs, actor, id) do credential_bodies = get_credential_bodies(attrs) - with :ok <- validate_credential_bodies(credential_bodies, attrs), - changeset <- Credential.create_changeset(%Credential{}, attrs), - :ok <- validate_external_id(changeset) do + changeset = Credential.create_changeset(%Credential{id: id}, attrs) + + with :ok <- validate_credential_bodies(changeset, credential_bodies, attrs), + :ok <- validate_external_id(changeset), + :ok <- validate_owner_access(changeset) do build_create_multi(changeset, credential_bodies) - |> derive_events(changeset) + |> derive_events(changeset, actor) |> Repo.transaction() |> handle_transaction_result(changeset) end @@ -227,7 +253,7 @@ defmodule Lightning.Credentials do |> Enum.with_index() |> Enum.reduce(multi, fn {body_attrs, index}, acc -> acc - |> Multi.insert(:"credential_body_#{index}", fn %{credential: credential} -> + |> Multi.insert({:credential_body, index}, fn %{credential: credential} -> %CredentialBody{} |> CredentialBody.changeset(%{ credential_id: credential.id, @@ -249,25 +275,56 @@ defmodule Lightning.Credentials do ## Returns * `{:ok, credential}` - Successfully updated credential - * `{:error, error}` - Error with update process + * `{:error, :scheduled_for_deletion}` - The credential is scheduled for + deletion. Only cancelling that, or deleting it now, can change it + * `{:error, changeset}` - With an error on `project_credentials` if the + owner has no access to a project a new link points to. Links the + credential already holds are not checked. A refused body is placed as + `create_credential/3` places it + * `{:error, :unauthorized}` - `actor` may not edit it + * `{:error, :concurrent_edit}` - another request deleted the credential + while this one ran """ - @spec update_credential(Credential.t(), map(), User.t()) :: - {:ok, Credential.t()} | {:error, any()} - def update_credential(%Credential{} = credential, attrs, %User{} = actor) do - with :ok <- authorize_credential_owner(actor, credential) do - do_update_credential(credential, attrs) + @spec update_credential(Credential.t(), map(), Lightning.Actor.t()) :: + {:ok, Credential.t()} + | {:error, + :unauthorized + | :scheduled_for_deletion + | :concurrent_edit + | Ecto.Changeset.t()} + def update_credential(%Credential{} = credential, attrs, actor) do + with :ok <- + authorize_credential( + :edit_credential, + actor, + with_stored_schedule(credential) + ) do + do_update_credential(credential, attrs, actor) end end - defp do_update_credential(%Credential{} = credential, attrs) do - credential = Repo.preload(credential, :project_credentials) + # A screen may hand in a struct loaded before the deletion was scheduled, so + # the policy is asked about the schedule as stored. + defp with_stored_schedule(%Credential{id: id} = credential) do + scheduled_deletion = + Repo.one( + from c in Credential, where: c.id == ^id, select: c.scheduled_deletion + ) + + %{credential | scheduled_deletion: scheduled_deletion} + end + + defp do_update_credential(%Credential{} = credential, attrs, actor) do + credential = preload_links(credential) attrs = normalize_keys(attrs) + changeset = Credential.changeset(credential, attrs) + schema = Ecto.Changeset.get_field(changeset, :schema) credential_bodies = attrs |> get_credential_bodies() |> then(fn bodies -> - if credential.schema == "oauth" do + if schema == "oauth" do preserve_refresh_tokens(bodies, credential) else bodies @@ -276,14 +333,15 @@ defmodule Lightning.Credentials do with :ok <- validate_credential_bodies( + changeset, credential_bodies, attrs, - credential.schema + schema ), - changeset <- Credential.changeset(credential, attrs), - :ok <- validate_external_id(changeset) do + :ok <- validate_external_id(changeset), + :ok <- validate_owner_access(changeset) do build_update_multi(credential, changeset, credential_bodies) - |> derive_events(changeset) + |> derive_events(changeset, actor) |> Repo.transaction() |> handle_transaction_result(changeset) end @@ -294,7 +352,7 @@ defmodule Lightning.Credentials do delete_environments = Map.get(changeset.params, "delete_environments", []) Multi.new() - |> Multi.update(:credential, changeset) + |> Multi.update(:credential, changeset, stale_error_field: :id) |> add_environment_deletions(credential.id, delete_environments) |> add_credential_body_upserts(credential.id, credential_bodies, schema_name) end @@ -303,7 +361,7 @@ defmodule Lightning.Credentials do defp add_environment_deletions(multi, credential_id, delete_environments) do Enum.reduce(delete_environments, multi, fn env_name, acc -> - Multi.run(acc, :"delete_env_#{env_name}", fn _repo, _changes -> + Multi.run(acc, {:delete_environment, env_name}, fn _repo, _changes -> delete_credential_body_if_exists(credential_id, env_name) end) end) @@ -318,8 +376,7 @@ defmodule Lightning.Credentials do {:ok, :not_found} credential_body -> - Repo.delete(credential_body) - {:ok, :deleted} + with {:ok, _} <- Repo.delete(credential_body), do: {:ok, :deleted} end end @@ -335,7 +392,7 @@ defmodule Lightning.Credentials do credential_bodies |> Enum.with_index() |> Enum.reduce(multi, fn {body_attrs, index}, acc -> - Multi.run(acc, :"credential_body_#{index}", fn _repo, _changes -> + Multi.run(acc, {:credential_body, index}, fn _repo, _changes -> upsert_credential_body(credential_id, body_attrs, schema_name) end) end) @@ -363,10 +420,18 @@ defmodule Lightning.Credentials do credential_body |> CredentialBody.changeset(%{body: body_data}) |> cast_credential_body_change(schema_name) - |> Repo.update() + |> update_body_if_changed() end end + # Compared after casting to the schema's types, so a form resending "5432" + # for a stored 5432 is no change. + defp update_body_if_changed(%Ecto.Changeset{valid?: true, changes: changes}) + when map_size(changes) == 0, + do: {:ok, :unchanged} + + defp update_body_if_changed(changeset), do: Repo.update(changeset) + defp get_credential_bodies(attrs) do case Map.get(attrs, "credential_bodies") do nil -> @@ -413,15 +478,29 @@ defmodule Lightning.Credentials do defp parse_nested_value(_), do: %{} - defp validate_credential_bodies(credential_bodies, attrs, schema \\ nil) do - credential_schema = schema || Map.get(attrs, "schema") - - if credential_schema == "oauth" do + # An OAuth token is refused at the key it is about, inside the body at its + # position in `credential_bodies`. + defp validate_credential_bodies( + changeset, + credential_bodies, + attrs, + schema \\ nil + ) do + if (schema || Map.get(attrs, "schema")) == "oauth" do credential_bodies - |> Enum.reduce_while(:ok, fn body_attrs, _acc -> + |> Enum.with_index() + |> Enum.find_value(:ok, fn {body_attrs, index} -> case validate_oauth_body(body_attrs["body"], attrs) do - :ok -> {:cont, :ok} - error -> {:halt, error} + :ok -> + nil + + {:error, %OauthValidation.Error{} = error} -> + changeset + |> Ecto.Changeset.add_error(:credential_bodies, error.message, + code: error.code, + key: [:credential_bodies, index, :body | List.wrap(error.field)] + ) + |> refused() end end) else @@ -429,6 +508,13 @@ defmodule Lightning.Credentials do end end + defp refused(changeset) do + action = + if changeset.data.__meta__.state == :built, do: :insert, else: :update + + {:error, Map.put(changeset, :action, action)} + end + defp validate_oauth_body(body_data, attrs) do with {:ok, _} <- OauthValidation.validate_token_data(body_data) do validate_expected_scopes(body_data, attrs) @@ -488,6 +574,37 @@ defmodule Lightning.Credentials do |> Enum.reject(&is_nil/1) end + # Only links being added are checked. Some existing links point to projects + # the owner has since lost access to, and the credential must stay editable. + # The error names projects by id because an API caller may link to a project + # whose name it isn't allowed to see. + defp validate_owner_access(%Ecto.Changeset{valid?: false}), do: :ok + + defp validate_owner_access(changeset) do + project_ids = + changeset + |> Ecto.Changeset.get_change(:project_credentials, []) + |> Enum.map(&Ecto.Changeset.get_change(&1, :project_id)) + |> Enum.reject(&is_nil/1) + + owner_id = Ecto.Changeset.get_field(changeset, :user_id) + + case projects_without_access(owner_id, project_ids) do + [] -> + :ok + + projects -> + changeset + |> Ecto.Changeset.add_error( + :project_credentials, + "the owner has no access to these projects: " <> + Enum.map_join(projects, ", ", & &1.id), + code: :no_access + ) + |> refused() + end + end + # NOTE: This is an app-level check only (not DB-enforced). It has a TOCTOU # race window — two concurrent inserts can both pass Repo.exists? before # either commits. Acceptable for now given low collision likelihood; @@ -520,15 +637,9 @@ defmodule Lightning.Credentials do end if Repo.exists?(query) do - action = - if changeset.data.__meta__.state == :built, do: :insert, else: :update - - changeset = - changeset - |> Ecto.Changeset.add_error(:external_id, message) - |> Map.put(:action, action) - - {:error, changeset} + changeset + |> Ecto.Changeset.add_error(:external_id, message, code: :taken) + |> refused() else :ok end @@ -538,33 +649,20 @@ defmodule Lightning.Credentials do @doc false def handle_transaction_result(transaction_result, credential_changeset) do case transaction_result do + {:error, :credential, %Ecto.Changeset{errors: [id: {_, stale: true}]}, _} -> + {:error, :concurrent_edit} + {:error, :credential, %Ecto.Changeset{} = changeset, _changes} -> {:error, changeset} - {:error, op, %Ecto.Changeset{} = body_changeset, _changes} -> - # When a credential_body operation fails, we need to return - # a Credential changeset with the error, not a CredentialBody changeset. - # Extract which environment failed from the operation name. - env_index = - op - |> Atom.to_string() - |> String.replace("credential_body_", "") - |> String.to_integer() - - errors = body_changeset.errors - - changeset_with_errors = - Enum.reduce(errors, credential_changeset, fn {field, {msg, opts}}, - acc -> - Ecto.Changeset.add_error( - acc, - :credential_bodies, - "Environment #{env_index + 1}: #{field} #{msg}", - opts - ) - end) - - {:error, changeset_with_errors} + {:error, {:credential_body, index}, %Ecto.Changeset{} = failed, _changes} -> + {:error, + nest_errors( + credential_changeset, + :credential_bodies, + [:credential_bodies, index], + failed.errors + )} {:error, _op, error, _changes} -> {:error, error} @@ -582,6 +680,23 @@ defmodule Lightning.Credentials do end end + defp nest_errors(changeset, field, prefix, child_errors) do + Enum.reduce(child_errors, changeset, fn + {child_field, {message, opts}}, acc -> + key = prefix ++ List.wrap(opts[:key] || child_field) + + # The options are the child changeset's own, so they already carry a + # code or Ecto's validation options. + # credo:disable-for-next-line Lightning.Credo.Check.ChangesetErrorCode + Ecto.Changeset.add_error( + acc, + field, + message, + Keyword.put(opts, :key, key) + ) + end) + end + @doc """ Creates a credential schema from credential json schema. @@ -652,15 +767,28 @@ defmodule Lightning.Credentials do {:ok, updated_body} -> Ecto.Changeset.put_change(changeset, :body, updated_body) + {:error, %Ecto.Changeset{errors: typed_errors}} -> + nest_errors( + changeset, + :body, + [:body], + Enum.map(typed_errors, fn {field, error} -> + {Atom.to_string(field), error} + end) + ) + {:error, reason} when reason in [:not_ready, :timeout, :unavailable] -> Ecto.Changeset.add_error( changeset, :body, - "adaptor catalogue is not ready yet, try again shortly" + "adaptor catalogue is not ready yet, try again shortly", + code: :not_ready ) {:error, _reason} -> - Ecto.Changeset.add_error(changeset, :body, "Invalid body types") + Ecto.Changeset.add_error(changeset, :body, "Invalid body types", + code: :type + ) end end @@ -684,63 +812,63 @@ defmodule Lightning.Credentials do end defp extract_environment_bodies(changes) do - Enum.reduce(changes, [], fn - {key, credential_body}, acc when is_atom(key) -> - case Atom.to_string(key) do - "credential_body_" <> _ -> - [{credential_body.name, credential_body.body} | acc] - - _ -> - acc - end + for {{:credential_body, _index}, %CredentialBody{} = body} <- changes, + do: {body.name, body.body} + end - _, acc -> - acc - end) - |> Enum.reverse() + defp extract_deleted_environments(changes) do + Enum.sort(for {{:delete_environment, name}, :deleted} <- changes, do: name) end defp derive_events( multi, %Ecto.Changeset{data: %Credential{__meta__: %{state: state}}} = - changeset + changeset, + actor ) do - case changeset.changes do - map when map_size(map) == 0 -> - multi - - _ -> - project_credentials_multi = - Ecto.Changeset.get_change(changeset, :project_credentials, []) - |> Enum.reduce(Multi.new(), fn changeset, multi -> - derive_event(multi, changeset) - end) + project_credentials_multi = + Ecto.Changeset.get_change(changeset, :project_credentials, []) + |> Enum.reduce(Multi.new(), fn changeset, multi -> + derive_event(multi, changeset, actor) + end) - multi - |> Multi.run(:collect_env_bodies, fn _repo, changes -> - {:ok, extract_environment_bodies(changes)} - end) - |> Multi.insert( - :audit, - fn %{credential: credential, collect_env_bodies: env_bodies} -> - Audit.user_initiated_event( - if(state == :built, do: "created", else: "updated"), - credential, - changeset, - env_bodies - ) - end - ) - |> Multi.append(project_credentials_multi) - end + multi + |> Multi.run(:audit, fn repo, %{credential: credential} = changes -> + case {extract_environment_bodies(changes), + extract_deleted_environments(changes)} do + {[], []} when map_size(changeset.changes) == 0 -> + {:ok, nil} + + {env_bodies, deleted} -> + Audit.user_initiated_event( + if(state == :built, do: "created", else: "updated"), + credential, + actor, + field_changes(changeset), + bodies: env_bodies, + deleted: deleted + ) + |> repo.insert() + end + end) + |> Multi.append(project_credentials_multi) end + # An empty changeset makes Audit.event record nothing, so a write that changed + # only a body passes an empty map instead. + defp field_changes(%Ecto.Changeset{changes: changes}) + when map_size(changes) == 0, + do: %{} + + defp field_changes(changeset), do: changeset + defp derive_event( multi, %Ecto.Changeset{ action: :delete, data: %Lightning.Projects.ProjectCredential{} - } = changeset + } = changeset, + actor ) do Multi.insert( multi, @@ -749,6 +877,7 @@ defmodule Lightning.Credentials do Audit.user_initiated_event( "removed_from_project", credential, + actor, %{ before: %{ project_id: Ecto.Changeset.get_field(changeset, :project_id) @@ -765,7 +894,8 @@ defmodule Lightning.Credentials do %Ecto.Changeset{ action: :insert, data: %Lightning.Projects.ProjectCredential{} - } = changeset + } = changeset, + actor ) do project_id = Ecto.Changeset.get_field(changeset, :project_id) @@ -773,7 +903,7 @@ defmodule Lightning.Credentials do |> Multi.insert( {:audit, project_id}, fn %{credential: credential} -> - Audit.user_initiated_event("added_to_project", credential, %{ + Audit.user_initiated_event("added_to_project", credential, actor, %{ before: %{project_id: nil}, after: %{project_id: project_id} }) @@ -787,10 +917,14 @@ defmodule Lightning.Credentials do ) end - defp derive_event(multi, %Ecto.Changeset{ - action: :update, - data: %Lightning.Projects.ProjectCredential{} - }) do + defp derive_event( + multi, + %Ecto.Changeset{ + action: :update, + data: %Lightning.Projects.ProjectCredential{} + }, + _actor + ) do multi end @@ -822,30 +956,53 @@ defmodule Lightning.Credentials do end @doc """ - Deletes a credential. + Deletes a credential, recording an audit event in the same transaction. + + Returns `{:error, :unauthorized}` when `actor` doesn't own it, + `{:error, :in_use}` when runs have used it, and `{:error, :not_found}` when + it is already gone, deleted since it was loaded. A refused audit event is + `{:error, changeset}`. ## Examples iex> delete_credential(credential, actor) - {:ok, %{credential: %Credential{}, audit: %Audit{}}} + {:ok, %{credential: %Credential{}, audit: %Lightning.Auditing.Audit{}}} iex> delete_credential(credential, actor) {:error, :unauthorized} """ + @spec delete_credential(Credential.t(), User.t()) :: + {:ok, + %{credential: Credential.t(), audit: Lightning.Auditing.Audit.t()}} + | {:error, :unauthorized | :in_use | :not_found | Ecto.Changeset.t()} def delete_credential(%Credential{} = credential, %User{} = actor) do - with :ok <- authorize_credential_owner(actor, credential) do + with :ok <- authorize_credential(:delete_credential, actor, credential), + :ok <- refuse_in_use(credential) do do_delete_credential(credential) end end + defp refuse_in_use(credential) do + if has_activity_in_projects?(credential), + do: {:error, :in_use}, + else: :ok + end + defp do_delete_credential(%Credential{} = credential) do + %{user: owner} = Repo.preload(credential, :user) + Multi.new() - |> Multi.delete(:credential, credential) + |> Multi.delete(:credential, credential, stale_error_field: :id) |> Multi.insert(:audit, fn _ -> - Audit.user_initiated_event("deleted", credential) + Audit.user_initiated_event("deleted", credential, owner) end) |> Repo.transaction() + |> case do + {:ok, deleted} -> {:ok, deleted} + {:error, :credential, _stale, _changes} -> {:error, :not_found} + {:error, :audit, changeset, _changes} -> {:error, changeset} + end end @doc """ @@ -869,6 +1026,8 @@ defmodule Lightning.Credentials do - `{:ok, credential}`: Returns an `:ok` tuple with the updated credential struct if the update was successful. + - `{:error, :scheduled_for_deletion}`: The credential is already scheduled + for deletion. Scheduling it again would move the date. - `{:error, changeset}`: Returns an `:error` tuple with the changeset if the update failed. ## Examples @@ -881,7 +1040,8 @@ defmodule Lightning.Credentials do """ def schedule_credential_deletion(%Credential{} = credential, %User{} = actor) do - with :ok <- authorize_credential_owner(actor, credential) do + with :ok <- authorize_credential(:delete_credential, actor, credential), + :ok <- refuse_scheduled(credential) do do_schedule_credential_deletion(credential) end end @@ -929,33 +1089,37 @@ defmodule Lightning.Credentials do Takes an id rather than a struct because the screen that calls it has only the id to hand. It loads the credential and asks the same ownership question as - every other write here, so the id being caller-supplied does not matter. + deleting it, so the id being caller-supplied does not matter. """ @spec cancel_scheduled_deletion(Ecto.UUID.t(), User.t()) :: - {:ok, Credential.t()} | {:error, :unauthorized | Ecto.Changeset.t()} + {:ok, Credential.t()} + | {:error, :unauthorized | Ecto.Changeset.t() | :concurrent_edit} def cancel_scheduled_deletion(credential_id, %User{} = actor) do credential = get_credential!(credential_id) - with :ok <- authorize_credential_owner(actor, credential) do - do_update_credential(credential, %{scheduled_deletion: nil}) + with :ok <- authorize_credential(:delete_credential, actor, credential) do + do_update_credential(credential, %{scheduled_deletion: nil}, actor) end end - # A credential belongs to the person who made it, and that is the rule the UI - # has always drawn its buttons from. Asking here rather than at each screen - # means a screen cannot act on someone else's credential by forgetting to - # check - the signature will not let it call in without saying who is asking. - defp authorize_credential_owner(%User{} = actor, %Credential{} = credential) do - if Permissions.can?( - :users, - :delete_credential, - actor, - credential - ) do - :ok - else - {:error, :unauthorized} - end + # Authorising here rather than in each caller means no caller can act on + # someone else's credential by forgetting to check. Every caller has to say + # who is asking. + defp authorize_credential(action, actor, %Credential{} = credential), + do: Permissions.can(:credentials, action, actor, credential) + + # Refused for every caller, so it lives here rather than in the policy. It + # runs after the policy so a caller without rights learns nothing about the + # credential. It reads the database because a screen may hand in a struct + # loaded before the deletion was scheduled. + defp refuse_scheduled(%Credential{id: id}) do + scheduled? = + Repo.exists?( + from c in Credential, + where: c.id == ^id and not is_nil(c.scheduled_deletion) + ) + + if scheduled?, do: {:error, :scheduled_for_deletion}, else: :ok end defp maybe_revoke_oauth_tokens(%Credential{schema: "oauth"} = credential) do @@ -1681,7 +1845,7 @@ defmodule Lightning.Credentials do }) end) |> Multi.insert(:audit, fn %{credential: updated_credential} -> - Audit.user_initiated_event("transfered", credential, %{ + Audit.user_initiated_event("transfered", credential, owner, %{ before: %{user_id: credential.user_id}, after: %{user_id: updated_credential.user_id} }) @@ -1826,30 +1990,61 @@ defmodule Lightning.Credentials do Ecto.Changeset.add_error( changeset, :email, - "User doesn't have access to these projects: #{project_names}" + "User doesn't have access to these projects: #{project_names}", + code: :no_access ) else _ -> changeset end end - @spec projects_blocking_credential_transfer( - Lightning.Credentials.Credential.t(), - Lightning.Accounts.User.t() - ) :: [Lightning.Projects.Project.t()] - defp projects_blocking_credential_transfer( - %Credential{id: credential_id}, - %User{id: user_id} - ) do - from(p in Lightning.Projects.Project, - join: pc in Lightning.Projects.ProjectCredential, - on: pc.project_id == p.id and pc.credential_id == ^credential_id, - left_join: pu in Lightning.Projects.ProjectUser, - on: pu.project_id == p.id and pu.user_id == ^user_id, - where: is_nil(pu.id), - distinct: true, - select: p - ) + defp projects_blocking_credential_transfer(%Credential{id: id}, %User{} = user) do + project_ids = + from(pc in Lightning.Projects.ProjectCredential, + where: pc.credential_id == ^id, + select: pc.project_id + ) + |> Repo.all() + + projects_without_access(user.id, project_ids, support_access: false) + end + + @doc """ + Returns the projects among `project_ids` that the user has no access to. + + A user has access to a project they are a member of. A support user also has + access to a project that allows support access, unless `support_access: false` + is given. Returns `[]` for a user that doesn't exist. + """ + @spec projects_without_access(Ecto.UUID.t(), [Ecto.UUID.t()], [ + {:support_access, boolean()} + ]) :: [Project.t()] + def projects_without_access(user_id, project_ids, opts \\ []) + + def projects_without_access(_user_id, [], _opts), do: [] + + def projects_without_access(user_id, project_ids, opts) do + query = + from(p in Project, + join: u in User, + on: u.id == ^user_id, + left_join: pu in Lightning.Projects.ProjectUser, + on: pu.project_id == p.id and pu.user_id == u.id, + where: p.id in ^project_ids and is_nil(pu.id), + order_by: p.name, + select: p + ) + + if Keyword.get(opts, :support_access, true) do + where( + query, + [p, u], + not (coalesce(u.support_user, false) and + coalesce(p.allow_support_access, false)) + ) + else + query + end |> Repo.all() end @@ -1864,11 +2059,14 @@ defmodule Lightning.Credentials do Ecto.Changeset.add_error( changeset, :email, - "You cannot transfer a credential to yourself" + "You cannot transfer a credential to yourself", + code: :not_allowed ) !Lightning.Repo.exists?(User |> where(email: ^recipient_email)) -> - Ecto.Changeset.add_error(changeset, :email, "User does not exist") + Ecto.Changeset.add_error(changeset, :email, "User does not exist", + code: :not_found + ) true -> changeset diff --git a/lib/lightning/credentials/audit.ex b/lib/lightning/credentials/audit.ex index 92f2f18df7e..b893d10ea09 100644 --- a/lib/lightning/credentials/audit.ex +++ b/lib/lightning/credentials/audit.ex @@ -22,17 +22,37 @@ defmodule Lightning.Credentials.Audit do end @doc """ - Creates a user-initiated audit event for credential operations. + Builds the audit event changeset for a change `actor` made to a credential. - For multi-environment credentials, environment body data is stored - in metadata rather than in the changes map. - """ - def user_initiated_event(event, credential, changes \\ %{}, env_bodies \\ []) do - %{id: id, user: user} = Lightning.Repo.preload(credential, :user) - - metadata = build_metadata_with_bodies(env_bodies) + Environment changes go in metadata rather than in the changes map: - event(event, id, user, changes, metadata) + * `:bodies` - `{name, body}` for each environment written. The bodies are + stored encrypted and their names listed under `environments` + * `:deleted` - names of environments removed, listed under + `deleted_environments` + """ + @spec user_initiated_event( + String.t(), + Lightning.Credentials.Credential.t(), + Lightning.Actor.t(), + map() | Ecto.Changeset.t(), + bodies: [{String.t(), map()}], + deleted: [String.t()] + ) :: Ecto.Changeset.t() + def user_initiated_event( + event, + credential, + actor, + changes \\ %{}, + environments \\ [] + ) do + metadata = + environments + |> Keyword.get(:bodies, []) + |> build_metadata_with_bodies() + |> put_deleted_environments(Keyword.get(environments, :deleted, [])) + + event(event, credential.id, actor, changes, metadata) end defp build_metadata_with_bodies([]), do: %{} @@ -51,6 +71,11 @@ defmodule Lightning.Credentials.Audit do } end + defp put_deleted_environments(metadata, []), do: metadata + + defp put_deleted_environments(metadata, deleted), + do: Map.put(metadata, :deleted_environments, deleted) + defp encrypt_body_for_audit(body) when is_map(body) do {:ok, encrypted} = Lightning.Encrypted.Map.dump(body) Base.encode64(encrypted) diff --git a/lib/lightning/credentials/credential.ex b/lib/lightning/credentials/credential.ex index a3574ac4b07..252bb3cee2b 100644 --- a/lib/lightning/credentials/credential.ex +++ b/lib/lightning/credentials/credential.ex @@ -59,6 +59,7 @@ defmodule Lightning.Credentials.Credential do :user_id ]) |> shared_validations() + |> unique_constraint(:id, name: :credentials_pkey) end @doc "Changeset for the guarded credential-transfer flow. Wraps the generic changeset so it keeps all validations/constraints, and additionally allows :user_id and :transfer_status." @@ -73,7 +74,7 @@ defmodule Lightning.Credentials.Credential do |> resolve_schema_name() |> normalize_external_id() |> cast_assoc(:project_credentials) - |> validate_required([:name, :user_id]) + |> validate_required([:name, :user_id, :schema]) |> unique_constraint([:name, :user_id], message: "you have another credential with the same name" ) diff --git a/lib/lightning/credentials/credential_body.ex b/lib/lightning/credentials/credential_body.ex index 5d0a74858a4..341e133c2f4 100644 --- a/lib/lightning/credentials/credential_body.ex +++ b/lib/lightning/credentials/credential_body.ex @@ -70,7 +70,7 @@ defmodule Lightning.Credentials.CredentialBody do validate_change(changeset, :body, fn :body, body -> case validate_sensitive_values_count(body) do :ok -> [] - {:error, message} -> [body: message] + {:error, message} -> [body: {message, code: :too_many_secrets}] end end) end diff --git a/lib/lightning/credentials/keychain_credential.ex b/lib/lightning/credentials/keychain_credential.ex index 4c6c3808113..3b5e89d046d 100644 --- a/lib/lightning/credentials/keychain_credential.ex +++ b/lib/lightning/credentials/keychain_credential.ex @@ -63,7 +63,7 @@ defmodule Lightning.Credentials.KeychainCredential do validate_change(changeset, field, fn ^field, path -> case validate_jsonpath_syntax(path) do :ok -> [] - {:error, message} -> [{field, message}] + {:error, message} -> [{field, {message, code: :format}}] end end) end @@ -107,7 +107,8 @@ defmodule Lightning.Credentials.KeychainCredential do add_error( changeset, :default_credential_id, - "cannot be checked without knowing the project" + "cannot be checked without knowing the project", + code: :required ) shared_with_project?(project_id, default_credential_id) -> @@ -117,7 +118,8 @@ defmodule Lightning.Credentials.KeychainCredential do add_error( changeset, :default_credential_id, - "must belong to the same project" + "must belong to the same project", + code: :not_found ) end end diff --git a/lib/lightning/credentials/oauth_validation.ex b/lib/lightning/credentials/oauth_validation.ex index addff2f5365..080c81c2b17 100644 --- a/lib/lightning/credentials/oauth_validation.ex +++ b/lib/lightning/credentials/oauth_validation.ex @@ -18,14 +18,6 @@ defmodule Lightning.Credentials.OauthValidation do that can occur during authentication flows, token validation, and scope verification. Each error includes a type for programmatic handling, a human-readable message, and optional details for additional context. - - ## Examples - - iex> Error.new(:missing_scopes, "Required scopes not granted") - %Error{type: :missing_scopes, message: "Required scopes not granted", details: nil} - - iex> Error.new(:missing_refresh_token, "No refresh token", %{existing_token_available: true}) - %Error{type: :missing_refresh_token, message: "No refresh token", details: %{existing_token_available: true}} """ @type error_type :: @@ -43,18 +35,21 @@ defmodule Lightning.Credentials.OauthValidation do | :missing_token_type | :unsupported_token_type + @typedoc """ + `code` names the rule the token broke, and `field` the token key it is + about, or `nil` for the token as a whole. `message` never quotes a value + from the token, since it can reach a client as it is. Values a screen + needs go in `details`. + """ @type t :: %__MODULE__{ type: error_type(), + code: atom() | nil, + field: String.t() | nil, message: String.t(), details: map() | nil } - defstruct [:type, :message, :details] - - @spec new(error_type(), String.t(), map() | nil) :: t() - def new(type, message, details \\ nil) do - %__MODULE__{type: type, message: message, details: details} - end + defstruct [:type, :code, :field, :message, :details] end @doc """ @@ -141,11 +136,12 @@ defmodule Lightning.Credentials.OauthValidation do check_scope_match(granted_scopes, expected_scopes) {:error, :no_scope_data} -> - {:error, - Error.new( - :invalid_oauth_response, - "OAuth token missing scope information" - )} + refuse( + :invalid_oauth_response, + "scope", + "OAuth token missing scope information", + code: :required + ) end end @@ -279,8 +275,26 @@ defmodule Lightning.Credentials.OauthValidation do defp convert_to_string(_), do: nil + # Callers name the code under the `code:` keyword, as changeset errors do, + # so searching lib for that keyword finds every code Lightning sets. + defp refuse(type, field, message, code: code), + do: refuse(type, field, message, code: code, details: nil) + + defp refuse(type, field, message, code: code, details: details) do + {:error, + %Error{ + type: type, + code: code, + field: field, + message: message, + details: details + }} + end + defp validate_token_format(token_data) when not is_map(token_data) do - {:error, Error.new(:invalid_token_format, "OAuth token must be a valid map")} + refuse(:invalid_token_format, nil, "OAuth token must be a valid map", + code: :type + ) end defp validate_token_format(_), do: :ok @@ -292,27 +306,27 @@ defmodule Lightning.Credentials.OauthValidation do defp validate_access_token(%{"access_token" => token}) when not is_binary(token) do - {:error, - Error.new( - :invalid_access_token, - "access_token must be a non-empty string, got: #{inspect(token)}" - )} + refuse( + :invalid_access_token, + "access_token", + "access_token must be a non-empty string", + code: :type + ) end defp validate_access_token(%{"access_token" => ""}) do - {:error, - Error.new( - :invalid_access_token, - "access_token cannot be empty" - )} + refuse(:invalid_access_token, "access_token", "access_token cannot be empty", + code: :required + ) end defp validate_access_token(_) do - {:error, - Error.new( - :missing_access_token, - "Missing required OAuth field: access_token" - )} + refuse( + :missing_access_token, + "access_token", + "Missing required OAuth field: access_token", + code: :required + ) end defp validate_refresh_token(%{"refresh_token" => token}) @@ -322,54 +336,58 @@ defmodule Lightning.Credentials.OauthValidation do defp validate_refresh_token(%{"refresh_token" => token}) when not is_binary(token) do - {:error, - Error.new( - :invalid_refresh_token, - "refresh_token must be a non-empty string, got: #{inspect(token)}" - )} + refuse( + :invalid_refresh_token, + "refresh_token", + "refresh_token must be a non-empty string", + code: :type + ) end defp validate_refresh_token(%{"refresh_token" => ""}) do - {:error, - Error.new( - :invalid_refresh_token, - "refresh_token cannot be empty" - )} + refuse( + :invalid_refresh_token, + "refresh_token", + "refresh_token cannot be empty", + code: :required + ) end defp validate_refresh_token(_) do - {:error, - Error.new( - :missing_refresh_token, - "Missing required OAuth field: refresh_token" - )} + refuse( + :missing_refresh_token, + "refresh_token", + "Missing required OAuth field: refresh_token", + code: :required + ) end defp validate_token_type(%{"token_type" => "Bearer"}), do: :ok defp validate_token_type(%{"token_type" => "bearer"}), do: :ok defp validate_token_type(%{"token_type" => type}) when is_binary(type) do - {:error, - Error.new( - :unsupported_token_type, - "Unsupported token type: '#{type}'. Expected 'Bearer'" - )} + refuse( + :unsupported_token_type, + "token_type", + "Unsupported token type. Expected 'Bearer'", + code: :not_allowed, + details: %{token_type: type} + ) end - defp validate_token_type(%{"token_type" => type}) do - {:error, - Error.new( - :invalid_token_format, - "token_type must be a string, got: #{inspect(type)}" - )} + defp validate_token_type(%{"token_type" => _type}) do + refuse(:invalid_token_format, "token_type", "token_type must be a string", + code: :type + ) end defp validate_token_type(_) do - {:error, - Error.new( - :missing_token_type, - "Missing required OAuth field: token_type" - )} + refuse( + :missing_token_type, + "token_type", + "Missing required OAuth field: token_type", + code: :required + ) end defp validate_scope_field(%{"scope" => scope}) @@ -381,52 +399,46 @@ defmodule Lightning.Credentials.OauthValidation do if Enum.all?(scopes, &is_binary/1) do :ok else - {:error, - Error.new( - :invalid_oauth_response, - "scopes array must contain only strings" - )} + refuse( + :invalid_oauth_response, + "scopes", + "scopes array must contain only strings", + code: :type + ) end end defp validate_scope_field(%{"scope" => ""}) do - {:error, - Error.new( - :missing_scope, - "scope field cannot be empty" - )} + refuse(:missing_scope, "scope", "scope field cannot be empty", + code: :required + ) end defp validate_scope_field(%{"scopes" => []}) do - {:error, - Error.new( - :missing_scope, - "scopes array cannot be empty" - )} + refuse(:missing_scope, "scopes", "scopes array cannot be empty", + code: :required + ) end defp validate_scope_field(%{"scope" => scope}) when not is_binary(scope) do - {:error, - Error.new( - :invalid_oauth_response, - "scope must be a string, got: #{inspect(scope)}" - )} + refuse(:invalid_oauth_response, "scope", "scope must be a string", + code: :type + ) end defp validate_scope_field(%{"scopes" => scopes}) when not is_list(scopes) do - {:error, - Error.new( - :invalid_oauth_response, - "scopes must be a list, got: #{inspect(scopes)}" - )} + refuse(:invalid_oauth_response, "scopes", "scopes must be a list", + code: :type + ) end defp validate_scope_field(_) do - {:error, - Error.new( - :missing_scope, - "Missing required OAuth field: scope or scopes" - )} + refuse( + :missing_scope, + "scope", + "Missing required OAuth field: scope or scopes", + code: :required + ) end defp validate_expiration(%{"expires_in" => expires_in}) do @@ -438,28 +450,31 @@ defmodule Lightning.Credentials.OauthValidation do end defp validate_expiration(_) do - {:error, - Error.new( - :missing_expiration, - "Missing expiration field: either expires_in or expires_at is required" - )} + refuse( + :missing_expiration, + "expires_in", + "Missing expiration field: either expires_in or expires_at is required", + code: :required + ) end defp validate_expires_in(expires_in) when is_integer(expires_in) do cond do expires_in <= 0 -> - {:error, - Error.new( - :invalid_expiration, - "expires_in must be greater than 0 seconds, got: #{expires_in}" - )} + refuse( + :invalid_expiration, + "expires_in", + "expires_in must be greater than 0 seconds", + code: :out_of_range + ) expires_in > 365 * 24 * 3600 -> - {:error, - Error.new( - :invalid_expiration, - "expires_in seems unreasonably long (#{expires_in} seconds = #{div(expires_in, 3600)} hours)" - )} + refuse( + :invalid_expiration, + "expires_in", + "expires_in seems unreasonably long", + code: :out_of_range + ) true -> :ok @@ -472,20 +487,22 @@ defmodule Lightning.Credentials.OauthValidation do validate_expires_in(int_value) _ -> - {:error, - Error.new( - :invalid_expiration, - "expires_in string must represent a valid positive integer, got: #{expires_in}" - )} + refuse( + :invalid_expiration, + "expires_in", + "expires_in string must represent a valid positive integer", + code: :type + ) end end - defp validate_expires_in(expires_in) do - {:error, - Error.new( - :invalid_expiration, - "expires_in must be a positive integer (seconds), got: #{inspect(expires_in)}" - )} + defp validate_expires_in(_expires_in) do + refuse( + :invalid_expiration, + "expires_in", + "expires_in must be a positive integer (seconds)", + code: :type + ) end defp validate_expires_at(expires_at) when is_integer(expires_at) do @@ -495,18 +512,20 @@ defmodule Lightning.Credentials.OauthValidation do cond do expires_at < min_valid_time -> - {:error, - Error.new( - :invalid_expiration, - "expires_at timestamp appears to be too far in the past (possible clock skew)" - )} + refuse( + :invalid_expiration, + "expires_at", + "expires_at timestamp appears to be too far in the past (possible clock skew)", + code: :out_of_range + ) expires_at > max_valid_time -> - {:error, - Error.new( - :invalid_expiration, - "expires_at timestamp appears to be too far in the future" - )} + refuse( + :invalid_expiration, + "expires_at", + "expires_at timestamp appears to be too far in the future", + code: :out_of_range + ) true -> :ok @@ -515,39 +534,32 @@ defmodule Lightning.Credentials.OauthValidation do defp validate_expires_at(expires_at) when is_binary(expires_at) and byte_size(expires_at) > 0 do - cond do - Regex.match?(~r/^\d+$/, expires_at) -> - case Integer.parse(expires_at) do - {timestamp, ""} -> - validate_expires_at(timestamp) - - _ -> - {:error, - Error.new( - :invalid_expiration, - "expires_at string must be a valid integer timestamp" - )} + case Integer.parse(expires_at) do + {timestamp, ""} -> + validate_expires_at(timestamp) + + _not_a_timestamp -> + if String.contains?(expires_at, ["T", "Z"]) or + String.match?(expires_at, ~r/\d{4}-\d{2}-\d{2}/) do + :ok + else + refuse( + :invalid_expiration, + "expires_at", + "expires_at must be a Unix timestamp or ISO 8601 date string", + code: :format + ) end - - String.contains?(expires_at, ["T", "Z"]) or - String.match?(expires_at, ~r/\d{4}-\d{2}-\d{2}/) -> - :ok - - true -> - {:error, - Error.new( - :invalid_expiration, - "expires_at must be a Unix timestamp or ISO 8601 date string, got: #{expires_at}" - )} end end - defp validate_expires_at(expires_at) do - {:error, - Error.new( - :invalid_expiration, - "expires_at must be an integer timestamp or date string, got: #{inspect(expires_at)}" - )} + defp validate_expires_at(_expires_at) do + refuse( + :invalid_expiration, + "expires_at", + "expires_at must be an integer timestamp or date string", + code: :type + ) end defp extract_granted_scopes(token_data) do @@ -572,18 +584,19 @@ defmodule Lightning.Credentials.OauthValidation do String.downcase(scope) in missing_normalized end) - scope_list = Enum.join(missing_original, ", ") - - message = - "Missing required scopes: #{scope_list}. Please reauthorize and grant all selected permissions." - details = %{ expected_scopes: expected_scopes, granted_scopes: granted_scopes, missing_scopes: missing_original } - {:error, Error.new(:missing_scopes, message, details)} + refuse( + :missing_scopes, + "scope", + "Missing required scopes. Please reauthorize and grant all selected permissions.", + code: :scope_not_granted, + details: details + ) end end diff --git a/lib/lightning/credentials/schema.ex b/lib/lightning/credentials/schema.ex index 7ef0317d93a..497f4ab65d4 100644 --- a/lib/lightning/credentials/schema.ex +++ b/lib/lightning/credentials/schema.ex @@ -186,7 +186,7 @@ defmodule Lightning.Credentials.Schema do defp handle_error(%{expected: format}, changeset, field) when format in @expected_formats do error_msg = Map.fetch!(@error_messages, format) - Changeset.add_error(changeset, field, error_msg) + Changeset.add_error(changeset, field, error_msg, code: :format) end defp handle_error(%{any_of: formats}, changeset, field) do @@ -197,37 +197,45 @@ defmodule Lightning.Credentials.Schema do <<"ipv", char>> -> "an IPv#{char - ?0} address" end) - Changeset.add_error(changeset, field, "expected to be #{formatted_types}") + Changeset.add_error(changeset, field, "expected to be #{formatted_types}", + code: :type + ) end defp handle_error(%{missing: fields}, changeset, _field) do Enum.reduce(fields, changeset, fn field, changeset -> # Convert string field name to atom to match schema.fields field_atom = String.to_existing_atom(field) - Changeset.add_error(changeset, field_atom, "can't be blank") + + Changeset.add_error(changeset, field_atom, "can't be blank", + code: :required + ) end) end defp handle_error(%{actual: 0, expected: _}, changeset, field) do - Changeset.add_error(changeset, field, "can't be blank") + Changeset.add_error(changeset, field, "can't be blank", code: :required) end defp handle_error(%{actual: "null", expected: expected}, changeset, field) when is_list(expected) do - Changeset.add_error(changeset, field, "can't be blank") + Changeset.add_error(changeset, field, "can't be blank", code: :required) end defp handle_error(%{expected: ["object"], actual: "string"}, changeset, field) do value = Changeset.get_field(changeset, field) case validate_json_object(value) do - :ok -> changeset - :error -> Changeset.add_error(changeset, field, "invalid JSON") + :ok -> + changeset + + :error -> + Changeset.add_error(changeset, field, "invalid JSON", code: :type) end end defp handle_error(%{expected: ["object"], actual: _}, changeset, field) do - Changeset.add_error(changeset, field, "must be an object") + Changeset.add_error(changeset, field, "must be an object", code: :type) end defp get_types(root) do diff --git a/lib/lightning/credentials/scoping.ex b/lib/lightning/credentials/scoping.ex index c36b44dc28e..d13958ec906 100644 --- a/lib/lightning/credentials/scoping.ex +++ b/lib/lightning/credentials/scoping.ex @@ -140,7 +140,9 @@ defmodule Lightning.Credentials.Scoping do changeset = Enum.reduce(matched, changeset, fn %{field: field}, cs -> - Ecto.Changeset.add_error(cs, field, violation_message(field)) + Ecto.Changeset.add_error(cs, field, violation_message(field), + code: :credential_not_allowed + ) end) {changeset, rest} @@ -166,7 +168,8 @@ defmodule Lightning.Credentials.Scoping do Ecto.Changeset.add_error( cs, :base, - "#{subject}: #{violation_message(field)} (#{field})" + "#{subject}: #{violation_message(field)} (#{field})", + code: :credential_not_allowed ) end) |> Map.put(:valid?, false) diff --git a/lib/lightning/extensions/project_hook.ex b/lib/lightning/extensions/project_hook.ex index 594f9d2bcd3..6196e8b5715 100644 --- a/lib/lightning/extensions/project_hook.ex +++ b/lib/lightning/extensions/project_hook.ex @@ -61,7 +61,8 @@ defmodule Lightning.Extensions.ProjectHook do |> Changeset.change() |> Changeset.add_error( :project_files, - "#{length(remaining)} stored file(s) could not be deleted from storage" + "#{length(remaining)} stored file(s) could not be deleted from storage", + code: :storage_failed )} end end diff --git a/lib/lightning/invocation.ex b/lib/lightning/invocation.ex index 97a039930ae..c95ea2fe7b1 100644 --- a/lib/lightning/invocation.ex +++ b/lib/lightning/invocation.ex @@ -480,6 +480,18 @@ defmodule Lightning.Invocation do |> Repo.paginate(params) end + @doc """ + A page of the log lines of every project `user` may access, narrowed by + `filters` (see `Query.filter_log_lines/2`). + """ + @spec paginate_log_lines_for(User.t(), map(), keyword()) :: Scrivener.Page.t() + def paginate_log_lines_for(%User{} = user, filters, page) do + user + |> Query.log_lines_for() + |> Query.filter_log_lines(filters) + |> Repo.paginate(page) + end + @doc """ Gets a single step. diff --git a/lib/lightning/invocation/dataclip.ex b/lib/lightning/invocation/dataclip.ex index 1ffe894cef4..664ca710891 100644 --- a/lib/lightning/invocation/dataclip.ex +++ b/lib/lightning/invocation/dataclip.ex @@ -83,7 +83,7 @@ defmodule Lightning.Invocation.Dataclip do changeset _other -> - add_error(changeset, :body, "must be a map") + add_error(changeset, :body, "must be a map", code: :type) end end @@ -136,7 +136,9 @@ defmodule Lightning.Invocation.Dataclip do changeset {_, request} when not is_nil(request) -> - add_error(changeset, :request, "cannot be set for this type") + add_error(changeset, :request, "cannot be set for this type", + code: :not_allowed + ) {_, _} -> changeset diff --git a/lib/lightning/invocation/log_line.ex b/lib/lightning/invocation/log_line.ex index dd5626a2642..14fef3219be 100644 --- a/lib/lightning/invocation/log_line.ex +++ b/lib/lightning/invocation/log_line.ex @@ -66,7 +66,7 @@ defmodule Lightning.Invocation.LogLine do fetch_field(changeset, :message) |> case do {_type, message} when is_nil(message) -> - add_error(changeset, :message, "can't be blank") + add_error(changeset, :message, "can't be blank", code: :required) _ -> changeset diff --git a/lib/lightning/invocation/query.ex b/lib/lightning/invocation/query.ex index 2bb3749e987..2bc163770ce 100644 --- a/lib/lightning/invocation/query.ex +++ b/lib/lightning/invocation/query.ex @@ -6,6 +6,7 @@ defmodule Lightning.Invocation.Query do alias Lightning.Accounts.User alias Lightning.Invocation.Dataclip + alias Lightning.Invocation.LogLine alias Lightning.Invocation.Step alias Lightning.Projects.Project alias Lightning.Run @@ -93,141 +94,57 @@ defmodule Lightning.Invocation.Query do ) end - @doc """ - Validate datetime parameters for filtering - """ - @spec validate_datetime_params(map(), list(String.t())) :: - :ok | {:error, String.t()} - def validate_datetime_params(params, keys) do - keys - |> Enum.find_value(fn key -> - case params[key] do - nil -> - nil - - value -> - case parse_datetime(value) do - {:ok, _} -> - nil - - :error -> - {:error, "Invalid datetime format for '#{key}': #{inspect(value)}"} - end - end - end) - |> case do - nil -> :ok - error -> error - end - end + @run_filters %{ + inserted_after: :utc_datetime_usec, + inserted_before: :utc_datetime_usec, + updated_after: :utc_datetime_usec, + updated_before: :utc_datetime_usec, + project_id: Ecto.UUID, + workflow_id: Ecto.UUID, + work_order_id: Ecto.UUID, + state: {:array, :string} + } @doc """ - Filter runs by inserted_at date range + The filters `filter_runs/2` takes, each with its type. """ - @spec filter_runs_by_date(Ecto.Queryable.t(), map()) :: Ecto.Queryable.t() - def filter_runs_by_date(query, params) do - query - |> filter_by_inserted_after(params["inserted_after"]) - |> filter_by_inserted_before(params["inserted_before"]) - |> filter_by_updated_after(params["updated_after"]) - |> filter_by_updated_before(params["updated_before"]) - end + @spec run_filter_types() :: %{atom() => Ecto.Type.t()} + def run_filter_types, do: @run_filters @doc """ - Filter runs by various criteria + Narrows a query of runs by `filters`, keyed and typed as + `run_filter_types/0` says. """ @spec filter_runs(Ecto.Queryable.t(), map()) :: Ecto.Queryable.t() - def filter_runs(query, params) do - query - |> filter_runs_by_date(params) - |> filter_runs_by_project(params["project_id"]) - |> filter_runs_by_workflow(params["workflow_id"]) - |> filter_runs_by_work_order(params["work_order_id"]) - |> filter_runs_by_state(params["state"]) - end - - defp filter_runs_by_project(query, nil), do: query - - defp filter_runs_by_project(query, project_id) do - from([workflow: w] in query, where: w.project_id == ^project_id) - end - - defp filter_runs_by_workflow(query, nil), do: query - - defp filter_runs_by_workflow(query, workflow_id) do - from([workflow: w] in query, where: w.id == ^workflow_id) - end - - defp filter_runs_by_work_order(query, nil), do: query - - defp filter_runs_by_work_order(query, work_order_id) do - from([work_order: wo] in query, where: wo.id == ^work_order_id) - end - - @valid_run_states Lightning.Run.states() |> Enum.map(&to_string/1) - - defp filter_runs_by_state(query, nil), do: query - - defp filter_runs_by_state(query, state) when is_binary(state) do - states = - state - |> String.split(",", trim: true) - |> Enum.map(&String.to_existing_atom/1) - - from(r in query, where: r.state in ^states) - end - - @doc """ - Validates the `state` query parameter against known run states. + def filter_runs(query, filters), + do: Enum.reduce(filters, query, &where_run/2) - Returns `:ok` or `{:error, message}`. - """ - @spec validate_run_state_param(map()) :: :ok | {:error, String.t()} - def validate_run_state_param(%{"state" => state}) when is_binary(state) do - invalid = - state - |> String.split(",", trim: true) - |> Enum.reject(&(&1 in @valid_run_states)) - - case invalid do - [] -> - :ok - - bad -> - {:error, - "Invalid state filter: #{inspect(bad)}. Valid states are: #{Enum.join(@valid_run_states, ", ")}"} - end - end + defp where_run({:project_id, id}, query), + do: from([workflow: w] in query, where: w.project_id == ^id) - def validate_run_state_param(_params), do: :ok + defp where_run({:workflow_id, id}, query), + do: from([workflow: w] in query, where: w.id == ^id) - defp filter_by_inserted_after(query, nil), do: query + defp where_run({:work_order_id, id}, query), + do: from([work_order: wo] in query, where: wo.id == ^id) - defp filter_by_inserted_after(query, date_string) do - {:ok, datetime} = parse_datetime(date_string) - from(r in query, where: r.inserted_at >= ^datetime) - end + defp where_run({:state, states}, query), + do: from(r in query, where: r.state in ^states) - defp filter_by_inserted_before(query, nil), do: query + defp where_run(timestamp_filter, query), + do: where_timestamps(timestamp_filter, query) - defp filter_by_inserted_before(query, date_string) do - {:ok, datetime} = parse_datetime(date_string) - from(r in query, where: r.inserted_at <= ^datetime) - end + defp where_timestamps({:inserted_after, at}, query), + do: from(q in query, where: q.inserted_at >= ^at) - defp filter_by_updated_after(query, nil), do: query + defp where_timestamps({:inserted_before, at}, query), + do: from(q in query, where: q.inserted_at <= ^at) - defp filter_by_updated_after(query, date_string) do - {:ok, datetime} = parse_datetime(date_string) - from(r in query, where: r.updated_at >= ^datetime) - end + defp where_timestamps({:updated_after, at}, query), + do: from(q in query, where: q.updated_at >= ^at) - defp filter_by_updated_before(query, nil), do: query - - defp filter_by_updated_before(query, date_string) do - {:ok, datetime} = parse_datetime(date_string) - from(r in query, where: r.updated_at <= ^datetime) - end + defp where_timestamps({:updated_before, at}, query), + do: from(q in query, where: q.updated_at <= ^at) @doc """ Steps for a specific user @@ -373,135 +290,45 @@ defmodule Lightning.Invocation.Query do ) end - @doc """ - Filter work orders by date range - """ - @spec filter_work_orders_by_date(Ecto.Queryable.t(), map()) :: - Ecto.Queryable.t() - def filter_work_orders_by_date(query, params) do - query - |> filter_wo_by_inserted_after(params["inserted_after"]) - |> filter_wo_by_inserted_before(params["inserted_before"]) - |> filter_wo_by_updated_after(params["updated_after"]) - |> filter_wo_by_updated_before(params["updated_before"]) - end + @work_order_filters %{ + id: {:array, Ecto.UUID}, + inserted_after: :utc_datetime_usec, + inserted_before: :utc_datetime_usec, + updated_after: :utc_datetime_usec, + updated_before: :utc_datetime_usec, + project_id: Ecto.UUID, + workflow_id: Ecto.UUID, + state: {:array, :string} + } @doc """ - Filter work orders by various criteria + The filters `filter_work_orders/2` takes, each with its type. """ - @spec filter_work_orders(Ecto.Queryable.t(), map()) :: Ecto.Queryable.t() - def filter_work_orders(query, params) do - query - |> filter_work_orders_by_ids(params["id"]) - |> filter_work_orders_by_date(params) - |> filter_work_orders_by_project(params["project_id"]) - |> filter_work_orders_by_workflow(params["workflow_id"]) - |> filter_work_orders_by_state(params["state"]) - end - - defp filter_work_orders_by_ids(query, nil), do: query - - defp filter_work_orders_by_ids(query, ids) when is_list(ids) do - from(wo in query, where: wo.id in ^ids) - end - - defp filter_work_orders_by_ids(query, id) when is_binary(id) do - case String.split(id, ",", trim: true) do - [single_id] -> - from(wo in query, where: wo.id == ^single_id) - - ids -> - from(wo in query, where: wo.id in ^ids) - end - end - - defp filter_work_orders_by_project(query, nil), do: query - - defp filter_work_orders_by_project(query, project_id) do - from([workflow: w] in query, where: w.project_id == ^project_id) - end - - defp filter_work_orders_by_workflow(query, nil), do: query - - defp filter_work_orders_by_workflow(query, workflow_id) do - from([workflow: w] in query, where: w.id == ^workflow_id) - end - - @valid_states Lightning.WorkOrder.states() |> Enum.map(&to_string/1) - - defp filter_work_orders_by_state(query, nil), do: query - - defp filter_work_orders_by_state(query, state) when is_binary(state) do - states = - state - |> String.split(",", trim: true) - |> Enum.map(&String.to_existing_atom/1) - - from(wo in query, where: wo.state in ^states) - end + @spec work_order_filter_types() :: %{atom() => Ecto.Type.t()} + def work_order_filter_types, do: @work_order_filters @doc """ - Validates the `state` query parameter against known work order states. - - Returns `:ok` or `{:error, message}`. + Narrows a query of work orders by `filters`, keyed and typed as + `work_order_filter_types/0` says. """ - @spec validate_state_param(map()) :: :ok | {:error, String.t()} - def validate_state_param(%{"state" => state}) when is_binary(state) do - invalid = - state - |> String.split(",", trim: true) - |> Enum.reject(&(&1 in @valid_states)) - - case invalid do - [] -> - :ok - - bad -> - {:error, - "Invalid state filter: #{inspect(bad)}. Valid states are: #{Enum.join(@valid_states, ", ")}"} - end - end - - def validate_state_param(_params), do: :ok - - defp filter_wo_by_inserted_after(query, nil), do: query - - defp filter_wo_by_inserted_after(query, date_string) do - {:ok, datetime} = parse_datetime(date_string) - from(wo in query, where: wo.inserted_at >= ^datetime) - end - - defp filter_wo_by_inserted_before(query, nil), do: query - - defp filter_wo_by_inserted_before(query, date_string) do - {:ok, datetime} = parse_datetime(date_string) - from(wo in query, where: wo.inserted_at <= ^datetime) - end - - defp filter_wo_by_updated_after(query, nil), do: query + @spec filter_work_orders(Ecto.Queryable.t(), map()) :: Ecto.Queryable.t() + def filter_work_orders(query, filters), + do: Enum.reduce(filters, query, &where_work_order/2) - defp filter_wo_by_updated_after(query, date_string) do - {:ok, datetime} = parse_datetime(date_string) - from(wo in query, where: wo.updated_at >= ^datetime) - end + defp where_work_order({:id, ids}, query), + do: from(wo in query, where: wo.id in ^ids) - defp filter_wo_by_updated_before(query, nil), do: query + defp where_work_order({:project_id, id}, query), + do: from([workflow: w] in query, where: w.project_id == ^id) - defp filter_wo_by_updated_before(query, date_string) do - {:ok, datetime} = parse_datetime(date_string) - from(wo in query, where: wo.updated_at <= ^datetime) - end + defp where_work_order({:workflow_id, id}, query), + do: from([workflow: w] in query, where: w.id == ^id) - defp parse_datetime(nil), do: :error - - defp parse_datetime(datetime_string) when is_binary(datetime_string) do - case DateTime.from_iso8601(datetime_string) do - {:ok, datetime, _offset} -> {:ok, datetime} - {:error, _} -> :error - end - end + defp where_work_order({:state, states}, query), + do: from(wo in query, where: wo.state in ^states) - defp parse_datetime(_), do: :error + defp where_work_order(timestamp_filter, query), + do: where_timestamps(timestamp_filter, query) @doc """ Log lines for a specific user, filtered by their accessible projects @@ -510,7 +337,7 @@ defmodule Lightning.Invocation.Query do def log_lines_for(%User{} = user) do projects = Ecto.assoc(user, :projects) |> select([:id]) - from(log in Lightning.Invocation.LogLine, + from(log in LogLine, as: :log, join: r in assoc(log, :run), as: :run, @@ -524,82 +351,56 @@ defmodule Lightning.Invocation.Query do ) end + @log_line_filters %{ + timestamp_after: :utc_datetime_usec, + timestamp_before: :utc_datetime_usec, + project_id: Ecto.UUID, + workflow_id: Ecto.UUID, + job_id: Ecto.UUID, + work_order_id: Ecto.UUID, + run_id: Ecto.UUID, + level: :string + } + @doc """ - Filter log lines by various criteria + The filters `filter_log_lines/2` takes, each with its type. """ - @spec filter_log_lines(Ecto.Queryable.t(), map()) :: Ecto.Queryable.t() - def filter_log_lines(query, params) do - query - |> filter_log_by_timestamp_after(params["timestamp_after"]) - |> filter_log_by_timestamp_before(params["timestamp_before"]) - |> filter_log_by_project(params["project_id"]) - |> filter_log_by_workflow(params["workflow_id"]) - |> filter_log_by_job(params["job_id"]) - |> filter_log_by_work_order(params["work_order_id"]) - |> filter_log_by_run(params["run_id"]) - |> filter_log_by_level(params["level"]) - end - - defp filter_log_by_timestamp_after(query, nil), do: query - - defp filter_log_by_timestamp_after(query, date_string) do - {:ok, datetime} = parse_datetime(date_string) - from([log: log] in query, where: log.timestamp >= ^datetime) - end + @spec log_line_filter_types() :: %{atom() => Ecto.Type.t()} + def log_line_filter_types, do: @log_line_filters - defp filter_log_by_timestamp_before(query, nil), do: query - - defp filter_log_by_timestamp_before(query, date_string) do - {:ok, datetime} = parse_datetime(date_string) - from([log: log] in query, where: log.timestamp <= ^datetime) - end - - defp filter_log_by_project(query, nil), do: query + @doc """ + Narrows a query of log lines by `filters`, keyed and typed as + `log_line_filter_types/0` says. + """ + @spec filter_log_lines(Ecto.Queryable.t(), map()) :: Ecto.Queryable.t() + def filter_log_lines(query, filters), + do: Enum.reduce(filters, query, &where_log_line/2) - defp filter_log_by_project(query, project_id) do - from([workflow: w] in query, where: w.project_id == ^project_id) - end + defp where_log_line({:timestamp_after, at}, query), + do: from([log: log] in query, where: log.timestamp >= ^at) - defp filter_log_by_workflow(query, nil), do: query + defp where_log_line({:timestamp_before, at}, query), + do: from([log: log] in query, where: log.timestamp <= ^at) - defp filter_log_by_workflow(query, workflow_id) do - from([work_order: wo] in query, where: wo.workflow_id == ^workflow_id) - end + defp where_log_line({:project_id, id}, query), + do: from([workflow: w] in query, where: w.project_id == ^id) - defp filter_log_by_job(query, nil), do: query + defp where_log_line({:workflow_id, id}, query), + do: from([work_order: wo] in query, where: wo.workflow_id == ^id) - defp filter_log_by_job(query, job_id) do + defp where_log_line({:job_id, id}, query) do from([log: log] in query, join: s in assoc(log, :step), - where: s.job_id == ^job_id + where: s.job_id == ^id ) end - defp filter_log_by_work_order(query, nil), do: query - - defp filter_log_by_work_order(query, work_order_id) do - from([run: r] in query, where: r.work_order_id == ^work_order_id) - end - - defp filter_log_by_run(query, nil), do: query + defp where_log_line({:work_order_id, id}, query), + do: from([run: r] in query, where: r.work_order_id == ^id) - defp filter_log_by_run(query, run_id) do - from([log: log] in query, where: log.run_id == ^run_id) - end - - defp filter_log_by_level(query, nil), do: query - - defp filter_log_by_level(query, level) when is_binary(level) do - level_atom = String.to_existing_atom(level) - - if level_atom in [:success, :always, :info, :warn, :error, :debug] do - from([log: log] in query, where: log.level == ^level_atom) - else - query - end - rescue - ArgumentError -> query - end + defp where_log_line({:run_id, id}, query), + do: from([log: log] in query, where: log.run_id == ^id) - defp filter_log_by_level(query, _), do: query + defp where_log_line({:level, level}, query), + do: from([log: log] in query, where: log.level == ^level) end diff --git a/lib/lightning/jobs.ex b/lib/lightning/jobs.ex index 1f2baf00bac..4851547fe11 100644 --- a/lib/lightning/jobs.ex +++ b/lib/lightning/jobs.ex @@ -6,6 +6,7 @@ defmodule Lightning.Jobs do import Ecto.Query alias Ecto.Multi + alias Lightning.Accounts.User alias Lightning.Projects.Project alias Lightning.Repo alias Lightning.Workflows @@ -27,14 +28,20 @@ defmodule Lightning.Jobs do |> Enum.map(fn e -> e.target_job end) end - @spec jobs_for_project_query(Project.t()) :: Ecto.Queryable.t() - def jobs_for_project_query(%Project{} = project) do - Query.jobs_for(project) + @doc """ + A page of the jobs of `project`, or of every project `user` may access. + """ + @spec paginate_jobs_for(Project.t() | User.t(), keyword()) :: + Scrivener.Page.t() + def paginate_jobs_for(project_or_user, page) do + project_or_user + |> Query.jobs_for() + |> Repo.paginate(page) end @spec jobs_for_project(Project.t()) :: [Job.t()] def jobs_for_project(%Project{} = project) do - jobs_for_project_query(project) |> Repo.all() + Query.jobs_for(project) |> Repo.all() end @doc """ @@ -46,9 +53,7 @@ defmodule Lightning.Jobs do @spec job_in_project?(Ecto.UUID.t(), Project.t()) :: boolean() def job_in_project?(job_id, %Project{} = project) do match?({:ok, _}, Ecto.UUID.cast(job_id)) and - Repo.exists?( - from(j in jobs_for_project_query(project), where: j.id == ^job_id) - ) + Repo.exists?(from(j in Query.jobs_for(project), where: j.id == ^job_id)) end @doc """ diff --git a/lib/lightning/policies/collections.ex b/lib/lightning/policies/collections.ex index a6e2585d36f..59c9ccd4927 100644 --- a/lib/lightning/policies/collections.ex +++ b/lib/lightning/policies/collections.ex @@ -4,14 +4,11 @@ defmodule Lightning.Policies.Collections do Access to collections is controlled by the project the collection belongs to. - Read access (`:access_collection`) is allowed for any project member, or for a - run that belongs to the project (via its workflow). - Access is gated on the caller's project role, mirroring the read/write asymmetry used elsewhere in the app: * `:access_collection` (read: get, stream, download) is allowed for any - project member, or for a run that belongs to the project. + project member. * `:put_collection_item` / `:delete_collection_item` (put, put_all and single-key delete) require at least the `:editor` role. * `:delete_all_collection_items` (wiping/matching-delete of a collection's @@ -20,20 +17,18 @@ defmodule Lightning.Policies.Collections do itself) requires `:owner` or `:admin`. Creation is authorized against the `%Project{}`, since no collection exists yet. + Every action needs the `collections:write` grant. + The `*_collection_item(s)` actions target the key-value entries exposed by the Collections API; `:manage_collection` targets the collection record itself. - - Runs retain full access to collections within their own project, since jobs - need to read and mutate collection data while executing. """ @behaviour Bodyguard.Policy - alias Lightning.Accounts.User alias Lightning.Collections.Collection + alias Lightning.Policies.Decision alias Lightning.Policies.Permissions alias Lightning.Projects.Project - alias Lightning.Projects.Scope - alias Lightning.Run + alias Lightning.Scope @type actions :: :access_collection @@ -47,39 +42,26 @@ defmodule Lightning.Policies.Collections do @spec authorize( actions(), - User.t() | Run.t(), - Collection.t() | Project.t() - ) :: :ok | {:error, :unauthorized} | boolean() - def authorize(:access_collection, %User{} = user, %Collection{} = collection) do - Permissions.can(:project_users, :access_project, user, collection) - end - - # Reads honour support access, via `:access_project` above. The writes below - # do not — `Scope.role_in?/3` reads `role` only, so a support user with no - # membership row is refused. Inherited behaviour, not a ruling. - def authorize(action, %User{} = user, %Collection{} = collection) - when action in [:put_collection_item, :delete_collection_item] do - Scope.role_in?(user, collection.project_id, @editor_roles) - end + Scope.t() | Scope.actor(), + Collection.t() | Project.t() | :before_target + ) :: Decision.answer() + def authorize(action, %Scope{} = scope, subject), + do: + Decision.project(scope, subject, "collections:write", judge(action, scope)) - def authorize( - :delete_all_collection_items, - %User{} = user, - %Collection{} = collection - ) do - Scope.role_in?(user, collection.project_id, @admin_roles) - end + def authorize(action, actor, subject), + do: Permissions.for_actor(actor, subject, &authorize(action, &1, subject)) - def authorize(:manage_collection, %User{} = user, %Project{} = project) do - Scope.role_in?(user, project.id, @admin_roles) - end + # Reads admit a support user with no membership row, as `:access_project` + # does. The writes don't, because `Scope.holds_role?/2` ignores support + # access. + defp judge(:access_collection, scope), do: Scope.has_standing?(scope) - def authorize(:manage_collection, %User{} = user, %Collection{} = collection) do - Scope.role_in?(user, collection.project_id, @admin_roles) - end + defp judge(action, scope) + when action in [:put_collection_item, :delete_collection_item], + do: Scope.holds_role?(scope, @editor_roles) - # Runs may perform any collection action within their own project. - def authorize(_action, %Run{} = run, %Collection{} = collection) do - Lightning.Runs.get_project_id_for_run(run) == collection.project_id - end + defp judge(action, scope) + when action in [:delete_all_collection_items, :manage_collection], + do: Scope.holds_role?(scope, @admin_roles) end diff --git a/lib/lightning/policies/credentials.ex b/lib/lightning/policies/credentials.ex index 5a409c04e2a..ff244e835d9 100644 --- a/lib/lightning/policies/credentials.ex +++ b/lib/lightning/policies/credentials.ex @@ -1,65 +1,130 @@ defmodule Lightning.Policies.Credentials do @moduledoc """ The Bodyguard Policy module for authorizing credential actions. + + Every action needs the `credentials:write` grant. + + A credential belongs to its owner, not to a project. Only the owner or an + operator may edit a credential, and only the owner may delete one. Only an + operator may write a credential for someone else (`:name_credential_owner`). + Listing your credentials, or creating one for yourself, needs a caller that + can own credentials, which only a person can. + + `:see_credential` lets in whoever may edit the credential, and a caller who + may list the credentials of a project it is linked to. It needs the + credential's `project_credentials` loaded. Anyone else hears + `{:error, :no_standing}`, since they may not learn that the credential + exists. + + A caller who may edit a credential scheduled for deletion hears + `{:error, :scheduled_for_deletion}` for `:edit_credential`. Seeing it and + deleting it are still allowed, so the deletion can be cancelled or brought + forward. + + Keychain actions take a project and decide through + `Lightning.Policies.Decision.project/4`. They need an owner or admin, or a + support user with no role on a project that allows support access. """ @behaviour Bodyguard.Policy - alias Lightning.Accounts.User + alias Lightning.Credentials.Credential alias Lightning.Credentials.KeychainCredential + alias Lightning.Policies.Decision + alias Lightning.Policies.Permissions + alias Lightning.Policies.ProjectUsers alias Lightning.Projects.Project - alias Lightning.Projects.Scope + alias Lightning.Scope require Logger @type actions :: - :create_keychain_credential + :see_credential + | :edit_credential + | :delete_credential + | :name_credential_owner + | :list_credentials + | :create_keychain_credential | :edit_keychain_credential | :delete_keychain_credential | :view_keychain_credential - @doc """ - Authorize credential actions based on the user's project role. + @ownership_actions [ + :edit_credential, + :delete_credential, + :name_credential_owner, + :list_credentials + ] - For KeychainCredential resources, users must have owner or admin role - in the associated project. - """ - def authorize(action, user, resource) + @keychain_actions [ + :create_keychain_credential, + :edit_keychain_credential, + :delete_keychain_credential, + :view_keychain_credential + ] @spec authorize( - action :: actions(), - user :: User.t(), - resource :: Project.t() - ) :: boolean + actions(), + Scope.t() | Scope.actor(), + Credential.t() + | KeychainCredential.t() + | Project.t() + | nil + | :before_target + ) :: + Decision.answer() def authorize( - :create_keychain_credential, - %User{} = user, - %Project{} = project - ) do - admin_or_support?(user, project) + :see_credential, + %Scope{} = scope, + %Credential{project_credentials: links} = credential + ) + when is_list(links) do + case ownership(:edit_credential, scope, credential) do + false -> + linked_to_listable_project?(scope, links) || {:error, :no_standing} + + answer -> + answer + end end - # KeychainCredential actions - require owner or admin role - # - # :create_keychain_credential is answerable from either shape. The context - # function holds an unsaved keychain rather than a project, and without this - # it would fall through to the deny-by-default clause and refuse every - # creation, including legitimate ones. - @spec authorize( - action :: actions(), - user :: User.t(), - resource :: KeychainCredential.t() - ) :: boolean def authorize( - action, - %User{} = user, - %KeychainCredential{} = keychain_credential - ) - when action in [ - :create_keychain_credential, - :edit_keychain_credential, - :delete_keychain_credential, - :view_keychain_credential - ] do - admin_or_support?(user, keychain_credential.project_id) + :edit_credential, + %Scope{} = scope, + %Credential{scheduled_deletion: %DateTime{}} = credential + ) do + case ownership(:edit_credential, scope, credential) do + true -> {:error, :scheduled_for_deletion} + answer -> answer + end + end + + def authorize(action, %Scope{} = scope, subject) + when action in @ownership_actions, + do: ownership(action, scope, subject) + + def authorize(action, %Scope{} = scope, subject) + when action in @keychain_actions, + do: + Decision.project( + scope, + subject, + "credentials:write", + admin_or_support?(scope) + ) + + def authorize(action, actor, subject) + when action in @ownership_actions and + (is_struct(subject, Credential) or is_nil(subject)) do + authorize(action, Scope.new(actor), subject) + end + + # :create_keychain_credential can be asked with a project or a keychain + # credential. The context function holds an unsaved keychain rather than a + # project, so without the keychain clause it would refuse every creation. + def authorize(action, actor, subject) + when action in @keychain_actions and + (is_struct(subject, Project) or + is_struct(subject, KeychainCredential)) do + Permissions.for_actor(actor, subject, &authorize(action, &1, subject)) end # Deny by default: a call site passing the wrong shape fails closed rather @@ -75,25 +140,51 @@ defmodule Lightning.Policies.Credentials do false end + defp ownership(action, scope, subject), + do: + Decision.instance( + scope, + subject, + "credentials:write", + owns?(action, scope, subject) + ) + + # Creating a credential for yourself makes you its owner, which only a caller + # that can own credentials may be. + defp owns?( + :edit_credential, + %Scope{actor: %{id: id}} = scope, + %Credential{id: nil, user_id: id} + ), + do: scope.can_own_credentials? + + defp owns?(:edit_credential, scope, %Credential{user_id: owner_id}), + do: scope.actor.id == owner_id or Scope.operator?(scope) + + # An operator may write a credential but never delete one. + defp owns?(:delete_credential, scope, %Credential{user_id: owner_id}), + do: scope.actor.id == owner_id + + defp owns?(:name_credential_owner, scope, nil), do: Scope.operator?(scope) + + defp owns?(:list_credentials, scope, nil), do: scope.can_own_credentials? + + defp owns?(_action, _scope, _subject), do: false + + defp linked_to_listable_project?(scope, links) do + scope + |> Scope.fetch_all(Enum.map(links, & &1.project_id)) + |> Enum.any?(fn {_id, project_scope} -> + ProjectUsers.permitted?(:list_project_credentials, project_scope) + end) + end + # Deliberately looser than ProjectUsers, where a support user with no # membership row gets nothing at admin level. Keychain access on a consenting # project is the pre-existing rule from allow_as_support_user?/2, kept rather # than tightened — narrowing it is a support-workflow decision, not a fix. - # - # The project's MFA requirement binds both branches, support included: it means - # "no un-MFA'd human touches this project's data", and a support user is one. - # Read here because this function pattern-matches the Scope directly rather - # than going through ProjectUsers.permitted?/2 or Scope.role_in?/3. - defp admin_or_support?(%User{} = user, subject) do - case Scope.fetch(user, subject) do - {:ok, %Scope{role: nil, support?: support?, mfa_satisfied?: mfa}} -> - support? and mfa - - {:ok, %Scope{role: role, mfa_satisfied?: mfa}} -> - role in [:owner, :admin] and mfa - - {:error, _reason} -> - false - end - end + defp admin_or_support?(scope), + do: + Scope.holds_role?(scope, [:owner, :admin]) or + (is_nil(scope.role) and scope.support?) end diff --git a/lib/lightning/policies/decision.ex b/lib/lightning/policies/decision.ex new file mode 100644 index 00000000000..d4906733080 --- /dev/null +++ b/lib/lightning/policies/decision.ex @@ -0,0 +1,76 @@ +defmodule Lightning.Policies.Decision do + @moduledoc """ + The order a policy decides in, so each policy supplies only its grant and + its rule. + + The order is what keeps refusals from leaking: a caller without the grant + hears `{:error, {:insufficient_scope, grant}}` before anything about the + target, and only a caller the rule allows learns + `{:error, :scheduled_for_deletion}`. A caller the rule refuses who has no + standing on the project hears `{:error, :no_standing}`, before the MFA check, + so it learns nothing about a project it has no standing on, not even that + it exists. + + `allowed?` is computed before the call, so it reads only facts already on the + scope and subject. A rule that needs to know about a project its subject links + to loads that project's scope through `Lightning.Scope`, outside + `allowed?` and only after the answer it builds on. `:see_credential` does this + for a credential's linked projects, and the sandbox rules for a sandbox's root + project. + + A `:before_target` subject asks only about the grant, so `true` for it is not + permission; only `Lightning.Policies.Permissions.permit/3` passes it. + """ + alias Lightning.Scope + + @typedoc """ + The refusal of a caller whose token lacks `grant`, the one grant the + refusing check needed. + """ + @type insufficient_scope :: {:insufficient_scope, grant :: String.t()} + + @type answer :: + boolean() + | {:error, + insufficient_scope() | :no_standing | :scheduled_for_deletion} + + @doc """ + Decides an action on a project. Without the grant the answer is + `{:error, {:insufficient_scope, grant}}`. A refused caller with no standing + on the project hears `{:error, :no_standing}`, and + `{:error, :scheduled_for_deletion}` comes only to a caller every other check + allows. + """ + @spec project(Scope.t(), term(), String.t(), boolean()) :: answer() + def project(%Scope{} = scope, subject, grant, allowed?) do + cond do + not Scope.allows?(scope, grant) -> {:error, {:insufficient_scope, grant}} + subject == :before_target -> true + not Scope.describes?(scope, subject) -> false + true -> on_project(scope, allowed?) + end + end + + defp on_project(scope, allowed?) do + cond do + not allowed? and not Scope.has_standing?(scope) -> {:error, :no_standing} + not scope.mfa_satisfied? -> false + not allowed? -> false + scope.scheduled_for_deletion? -> {:error, :scheduled_for_deletion} + true -> true + end + end + + @doc """ + Decides an action with no project, answering `allowed?` once the grant is + held. + """ + @spec instance(Scope.t(), term(), String.t(), boolean()) :: answer() + def instance(%Scope{} = scope, subject, grant, allowed?) do + cond do + not Scope.allows?(scope, grant) -> {:error, {:insufficient_scope, grant}} + subject == :before_target -> true + true -> allowed? + end + end +end diff --git a/lib/lightning/policies/permissions.ex b/lib/lightning/policies/permissions.ex index 454867a453a..845a7419c3f 100644 --- a/lib/lightning/policies/permissions.ex +++ b/lib/lightning/policies/permissions.ex @@ -87,6 +87,66 @@ defmodule Lightning.Policies.Permissions do |> Bodyguard.permit?(action, user, params) end + @doc """ + Whether the caller described by `scope` may apply `action` to `subject`. + + The one entry point for API callers. Unlike `can/4`, a refusal is + `{:error, :forbidden}`, so it renders as 403 rather than 401, and a policy's + more specific answer (`{:insufficient_scope, grant}`, `:no_standing`, + `:scheduled_for_deletion`) reaches the caller unchanged. + """ + @spec permit(atom(), atom(), Lightning.Scope.t(), term()) :: + :ok + | {:error, + :forbidden + | Lightning.Policies.Decision.insufficient_scope() + | :no_standing + | :scheduled_for_deletion} + def permit(policy, action, %Lightning.Scope{} = scope, subject) do + case can(policy, action, scope, subject) do + {:error, :unauthorized} -> {:error, :forbidden} + result -> result + end + end + + @doc """ + Whether the caller's grant already refuses `action`, before any target is + loaded. A pass here is not permission. The action still has to ask + `permit/4` about its target. + """ + @spec permit(atom(), atom(), Lightning.Scope.t()) :: + :ok + | {:error, + :forbidden + | Lightning.Policies.Decision.insufficient_scope() + | :no_standing + | :scheduled_for_deletion} + def permit(policy, action, %Lightning.Scope{} = scope), + do: permit(policy, action, scope, :before_target) + + @doc """ + Resolves `actor`'s scope on the subject's project and passes it to `decide`, + for a policy handed an actor rather than a `%Scope{}`. Bodyguard's contract + is boolean, and channel and LiveView callers match `{:error, :unauthorized}`, + so a missing project, no standing and a scheduled project are all `false`. + Only callers going through `permit/4` with a `%Scope{}` are told apart. + """ + @spec for_actor( + Lightning.Scope.actor(), + term(), + (Lightning.Scope.t() -> term()) + ) :: term() + def for_actor(actor, subject, decide) do + case Lightning.Scope.fetch(actor, subject) do + {:ok, scope} -> scope |> decide.() |> as_refusal() + {:error, _reason} -> false + end + end + + defp as_refusal({:error, :no_standing}), do: false + defp as_refusal({:error, :scheduled_for_deletion}), do: false + defp as_refusal(answer), do: answer + defp resolve_policy_module(policy) when is_atom(policy) do if Code.ensure_loaded?(policy) do policy diff --git a/lib/lightning/policies/project_users.ex b/lib/lightning/policies/project_users.ex index fe6e2f38eac..ccb449617f4 100644 --- a/lib/lightning/policies/project_users.ex +++ b/lib/lightning/policies/project_users.ex @@ -4,27 +4,26 @@ defmodule Lightning.Policies.ProjectUsers do Two steps, deliberately separate. - `Lightning.Projects.Scope` establishes the **facts**: which project, and what - standing this actor has in it. It refuses outright for a project that does not - exist or is scheduled for deletion, so nothing downstream can be handed a - standing in a wound-down project. + `Lightning.Scope` establishes the **facts**: which project, and what + standing this actor has in it, including whether the project is scheduled + for deletion. - `permitted?/2` makes the **judgement**: given that standing, is this action - allowed. It decides on `role` and `mfa_satisfied?`, and never mentions - `Project.scheduled_deletion` or `Project.requires_mfa` — there is no - shut-down project left for it to see, and the MFA rule reaches it as a fact - on the scope. + `authorize/3` on a `%Scope{}` makes the **judgement**, in the order + `Lightning.Policies.Decision` sets. It never mentions + `Project.scheduled_deletion` or `Project.requires_mfa`. Both reach it as + facts on the scope. Adding an action means adding an atom to `@admin_actions` or - `@editor_actions`. It inherits the guard; there is nothing to remember. + `@editor_actions`. It inherits the guards; there is nothing to remember. We deny by default: an action in none of the lists is refused. """ @behaviour Bodyguard.Policy - alias Lightning.Accounts.User + alias Lightning.Policies.Decision + alias Lightning.Policies.Permissions alias Lightning.Projects.ProjectUser - alias Lightning.Projects.Scope + alias Lightning.Scope @admin_actions [ :write_webhook_auth_method, @@ -50,10 +49,20 @@ defmodule Lightning.Policies.ProjectUsers do @self_actions [:edit_digest_alerts, :edit_failure_alerts] - @other_actions [:access_project, :delete_project, :publish_template] + @other_actions [ + :access_project, + :list_project_credentials, + :delete_project, + :publish_template + ] + + @owner_roles [:owner] + @admin_roles [:owner, :admin] + @editor_roles [:owner, :admin, :editor] @type actions :: :access_project + | :list_project_credentials | :delete_project | :publish_template | :run_workflow @@ -84,112 +93,103 @@ defmodule Lightning.Policies.ProjectUsers do do: @admin_actions ++ @editor_actions ++ @self_actions ++ @other_actions @doc """ - Whether `user` may perform `action`, where the subject is anything that - identifies a project — see `t:Lightning.Projects.Scope.subject/0`. - """ - @spec authorize(actions(), User.t(), Scope.subject()) :: boolean() + Whether the caller may perform `action`, where the subject is anything that + identifies a project — see `t:Lightning.Scope.subject/0`. - # These act on a specific membership row, not on project standing. The alert - # handlers in `ProjectLive.Settings` pass a client-supplied "project_user_id", - # so dropping the `id == user_id` comparison is a privilege escalation — - # pinned by two tests in project_live_test.exs. - def authorize( - action, - %User{id: id} = user, - %ProjectUser{user_id: user_id} = project_user - ) - when action in @self_actions do - # This clause never reaches permitted?/2, so the guards there do not cover - # it and it checks the scope itself. Without that, a member the project - # blocks could still edit their own preferences on it. - id == user_id and - match?( - {:ok, %Scope{mfa_satisfied?: true}}, - Scope.fetch(user, project_user) + Given a `%Scope{}`, the subject must be the scope's own project or one of + its membership rows. + """ + @spec authorize(actions(), Scope.t() | Scope.actor(), Scope.subject()) :: + Decision.answer() + def authorize(action, %Scope{} = scope, subject), + do: + Decision.project( + scope, + subject, + grant(action), + judge(action, scope, subject) ) - end - - def authorize(action, %User{} = user, subject) do - case Scope.fetch(user, subject) do - {:ok, scope} -> - permitted?(action, scope) - # Bodyguard's contract is boolean, so the reason stops here. A caller - # wanting the distinction resolves Scope itself. - {:error, _reason} -> - false - end - end + def authorize(action, actor, subject), + do: Permissions.for_actor(actor, subject, &authorize(action, &1, subject)) @doc """ - Decide an action against an already-resolved `%Scope{}`. + Decide an action against an already-resolved `%Scope{}`, as a boolean. Public so a caller needing several answers about one project resolves once - rather than once per question. + rather than once per question. It is `false` when the scope lacks the + action's grant, and on a project scheduled for deletion. """ @spec permitted?(actions(), Scope.t()) :: boolean() + def permitted?(action, %Scope{} = scope), + do: authorize(action, scope, scope.project) == true - # First, so it denies every action this module decides — including ones added - # later — without anyone having to remember @admin_actions/@editor_actions. - def permitted?(_action, %Scope{mfa_satisfied?: false}), do: false + @doc """ + Whether this actor would have standing on the project if not for the + project's MFA requirement. - def permitted?(:access_project, %Scope{} = scope), do: has_standing?(scope) + Lets a caller distinguish "no access at all" from "access except MFA" — + `LightningWeb.Hooks` sends the first to not-found and the second to + `/mfa_required` — without duplicating what counts as standing. It cannot ask + `permitted?(:access_project, ...)` for that, since that answers `false` for + both cases. A project scheduled for deletion is not-found for everyone, so + it is never "blocked by MFA". + """ + @spec blocked_by_mfa?(Scope.t()) :: boolean() + def blocked_by_mfa?(%Scope{mfa_satisfied?: mfa_satisfied?} = scope), + do: + Scope.has_standing?(scope) and not mfa_satisfied? and + not scope.scheduled_for_deletion? - def permitted?(:delete_project, %Scope{role: role}), do: role == :owner + defp grant(action) + when action in [:create_project_credential, :list_project_credentials], + do: "credentials:write" + + defp grant(_action), do: "projects:write" + + # These act on a specific membership row, not on project standing. The alert + # handlers in `ProjectLive.Settings` pass a client-supplied "project_user_id", + # so dropping the comparison with the caller's own row is a privilege + # escalation. Two tests in project_live_test.exs pin it. + defp judge(action, %Scope{project_user: own_row}, %ProjectUser{id: id}) + when action in @self_actions, + do: not is_nil(own_row) and own_row.id == id + + defp judge(action, scope, _subject), do: judge(action, scope) + + defp judge(action, scope) + when action in [:access_project, :list_project_credentials], + do: Scope.has_standing?(scope) + + defp judge(:delete_project, scope), do: Scope.holds_role?(scope, @owner_roles) # Support-staff-only: a project owner cannot publish, and staff can publish # without holding a membership row. Needs both support fields separately, so # do not fold it into @editor_actions — that drops the support_user # requirement silently. - def permitted?(:publish_template, %Scope{ - support_user?: true, - role: role, - support?: support? - }), - do: not is_nil(role) or support? + defp judge(:publish_template, %Scope{support_user?: true} = scope), + do: not is_nil(scope.role) or scope.support? - def permitted?(:publish_template, %Scope{}), do: false + defp judge(:publish_template, _scope), do: false # Reached only when the subject wasn't a %ProjectUser{} — Scope resolves the - # caller's own standing, so there is no "which row" ambiguity here. - def permitted?(action, %Scope{role: role}) when action in @self_actions, - do: not is_nil(role) + # caller's own standing, so there is no "which row" ambiguity here. An + # operator has no row, so gets nothing. + defp judge(action, scope) when action in @self_actions, + do: not is_nil(scope.role) - # Admin actions have never been available to a support user without a - # membership row. Unchanged here. - def permitted?(action, %Scope{role: role}) when action in @admin_actions, - do: role in [:owner, :admin] + # A support user with no membership row holds no admin action. + defp judge(action, scope) when action in @admin_actions, + do: Scope.holds_role?(scope, @admin_roles) # An explicit row wins; support access only fills in for someone with no row # at all. So adding a support user as a viewer gives them viewer permissions # rather than their account flag overriding it. Under the opposite rule, # "this person is read-only here" could not be expressed at all. - def permitted?(action, %Scope{role: nil, support?: support?}) - when action in @editor_actions, - do: support? - - def permitted?(action, %Scope{role: role}) when action in @editor_actions, - do: role in [:owner, :admin, :editor] - - def permitted?(_action, %Scope{}), do: false - - @doc """ - Whether this actor would have standing on the project if not for the - project's MFA requirement. - - Lets a caller distinguish "no access at all" from "access except MFA" — - `LightningWeb.Hooks` sends the first to not-found and the second to - `/mfa_required` — without duplicating what counts as standing. It cannot ask - `permitted?(:access_project, ...)` for that, since the guard above answers - `false` for both cases. - """ - @spec blocked_by_mfa?(Scope.t()) :: boolean() - def blocked_by_mfa?(%Scope{mfa_satisfied?: mfa_satisfied?} = scope), - do: has_standing?(scope) and not mfa_satisfied? + defp judge(action, scope) when action in @editor_actions, + do: + Scope.holds_role?(scope, @editor_roles) or + (is_nil(scope.role) and scope.support?) - # The one definition of "has any standing at all". `:access_project` is - # exactly this plus the MFA guard; `blocked_by_mfa?/1` is exactly this minus - # it. Written once so the two cannot answer differently. - defp has_standing?(%Scope{role: role, support?: support?}), - do: not is_nil(role) or support? + defp judge(_action, _scope), do: false end diff --git a/lib/lightning/policies/provisioning.ex b/lib/lightning/policies/provisioning.ex index f0b6d1cee67..9228d0e8003 100644 --- a/lib/lightning/policies/provisioning.ex +++ b/lib/lightning/policies/provisioning.ex @@ -2,71 +2,92 @@ defmodule Lightning.Policies.Provisioning do @moduledoc """ The Bodyguard Policy module for the provisioning API. - Two kinds of caller reach this: a person with an API token, and a - `%ProjectRepoConnection{}` — a GitHub connection pushing a project definition - back to us. Both resolve through `Lightning.Projects.Scope`, so both are - refused on a project scheduled for deletion. A person is additionally held to - the project's MFA requirement; a repo connection has no MFA concept, and - `Scope` says so for it. + Three kinds of caller reach this: a person with an API token, a service + account creating a project, and a `%ProjectRepoConnection{}` (a GitHub + connection pushing a project definition back to us). All resolve through + `Lightning.Scope`. A person is additionally held to the project's MFA + requirement; a repo connection has no MFA concept, and `Scope` says so for + it. A caller who would otherwise be allowed on a project scheduled for + deletion hears `{:error, :scheduled_for_deletion}`, and one with no standing + on the project `{:error, :no_standing}`, which reveals nothing about the + project, not even that it exists. - The repo-connection clauses previously compared - `repo_connection.project_id == project.id`, which is trivially true for the - project the connection belongs to and asks nothing about the project's state — - so a push could re-enable the triggers that shutting the project down had - disabled. - - Only a superuser can provision a project that does not exist yet. Owners and - admins can update an existing one. + Only a superuser or an operator holding `projects:write` can create a + project. Owners and admins can update an existing one. + `:provision_project` and `:describe_project` on an existing project need + `projects:provision`, the one grant a repo connection holds. """ @behaviour Bodyguard.Policy - alias Lightning.Accounts.User - alias Lightning.Policies.Permissions + alias Lightning.Policies.Decision + alias Lightning.Policies.ProjectUsers alias Lightning.Projects.Project - alias Lightning.Projects.Scope + alias Lightning.Scope alias Lightning.VersionControl.ProjectRepoConnection - @type actor :: User.t() | ProjectRepoConnection.t() - @type actions :: :provision_project | :describe_project + @type actions :: :create_project | :provision_project | :describe_project - @spec authorize(actions(), actor(), Project.t()) :: - boolean() | {:error, :forbidden} + @spec authorize( + actions(), + Scope.t(), + Project.t() | nil | :before_target + ) :: + boolean() + | {:error, + :forbidden + | Decision.insufficient_scope() + | :no_standing + | :scheduled_for_deletion} - # STAYS FIRST. A project that does not exist yet cannot be scheduled for - # deletion, and has no members to consult. Reordering this below the clauses - # that resolve a Scope breaks project creation through the API. - def authorize(:provision_project, %User{role: role}, %Project{id: nil}) do - role in [:superuser] or {:error, :forbidden} + # `false` becomes `{:error, :forbidden}`, so `/api/provision` answers 403, not + # 401. + def authorize(:create_project, %Scope{} = scope, subject) do + Decision.instance( + scope, + subject, + "projects:write", + scope.instance_role in [:superuser, :operator] + ) || {:error, :forbidden} end - def authorize(:provision_project, %User{} = user, %Project{} = project) do - case Scope.fetch(user, project) do - {:ok, %Scope{role: role, mfa_satisfied?: true}} - when role in [:owner, :admin] -> - true + def authorize(:describe_project, %Scope{} = scope, :before_target), + do: Decision.instance(scope, :before_target, "projects:provision", true) - _ -> - {:error, :forbidden} - end - end - - # Delegated rather than copied, so it cannot drift from `:access_project`. - def authorize(:describe_project, %User{} = user, %Project{} = project) do - Permissions.can?(:project_users, :access_project, user, project) or - {:error, :forbidden} + # Which grant applies waits for the body's project: creating one needs + # projects:write, provisioning one that exists projects:provision. + def authorize(:provision_project, %Scope{} = scope, :before_target) do + Scope.allows?(scope, "projects:write") or + Decision.instance(scope, :before_target, "projects:provision", true) end - # Resolving a Scope at all is the whole check: Scope refuses a project this - # connection does not belong to, and one scheduled for deletion. - def authorize( - action, - %ProjectRepoConnection{} = repo_connection, - %Project{} = project - ) + def authorize(action, %Scope{} = scope, %Project{} = project) when action in [:provision_project, :describe_project] do - match?({:ok, _}, Scope.fetch(repo_connection, project)) or - {:error, :forbidden} + case Decision.project( + scope, + project, + "projects:provision", + judge(action, scope) + ) do + false -> {:error, :forbidden} + answer -> answer + end end def authorize(_, _, _), do: false + + # A repo connection's standing is the project it belongs to, which + # `describes?/2` has already checked. + defp judge(_action, %Scope{actor: %ProjectRepoConnection{}}), do: true + + defp judge(:provision_project, scope), + do: Scope.holds_role?(scope, [:owner, :admin]) + + # Delegated rather than copied, so it cannot drift from `:access_project`. + # It answers `:scheduled_for_deletion` only to a caller it would allow. + defp judge(:describe_project, scope), + do: + ProjectUsers.authorize(:access_project, scope, scope.project) in [ + true, + {:error, :scheduled_for_deletion} + ] end diff --git a/lib/lightning/policies/sandboxes.ex b/lib/lightning/policies/sandboxes.ex index 9345459a213..8ac246611f4 100644 --- a/lib/lightning/policies/sandboxes.ex +++ b/lib/lightning/policies/sandboxes.ex @@ -29,7 +29,7 @@ defmodule Lightning.Policies.Sandboxes do alias Lightning.Accounts.User alias Lightning.Projects alias Lightning.Projects.Project - alias Lightning.Projects.Scope + alias Lightning.Scope @type actions :: :delete_sandbox @@ -100,8 +100,8 @@ defmodule Lightning.Policies.Sandboxes do end # The root cascade is a role rule, not a bypass: a root owner/admin counts as - # admin *on this sandbox*, so the sandbox still has to be operable. Resolving - # it first keeps the cascade unreachable for a sandbox scheduled for deletion. + # admin *on this sandbox*, so the sandbox still has to be operable. Refusing a + # scheduled sandbox first keeps the cascade unreachable for it. # # Only this branch reads `role` off the Scope directly, so only it needs the # MFA fact spelled out; has_root_project_permission?/2 gets the same guard for @@ -109,19 +109,20 @@ defmodule Lightning.Policies.Sandboxes do def authorize(action, %User{} = user, %Project{} = sandbox) when action in [:delete_sandbox, :update_sandbox] do case Scope.fetch(user, sandbox) do - {:ok, %Scope{role: role, mfa_satisfied?: mfa}} -> + {:ok, + %Scope{role: role, mfa_satisfied?: mfa, scheduled_for_deletion?: false}} -> (role in [:owner, :admin] and mfa) or has_root_project_permission?(sandbox, user) - {:error, _reason} -> + _scheduled_or_missing -> false end end - # Deliberately does not go through Scope: the subject is scheduled for - # deletion by definition, and Scope refuses those, so the liveness gate would - # make a sandbox scheduled for deletion impossible to restore. Same role rule as - # :delete_sandbox, read directly. + # Deliberately reads roles directly rather than through `Scope.role_in?/3`, + # which refuses a project scheduled for deletion. This subject is scheduled by + # definition, so that would make it impossible to restore. Same role rule as + # :delete_sandbox. def authorize(:cancel_scheduled_deletion, %User{} = user, %Project{} = sandbox) do root_project = Projects.root_of(sandbox) diff --git a/lib/lightning/policies/users.ex b/lib/lightning/policies/users.ex index 1f8c8c23a73..f4eb9c88576 100644 --- a/lib/lightning/policies/users.ex +++ b/lib/lightning/policies/users.ex @@ -6,16 +6,27 @@ defmodule Lightning.Policies.Users do alias Lightning.Accounts alias Lightning.Accounts.User - alias Lightning.Credentials.Credential alias Lightning.Credentials.OauthClient + alias Lightning.Policies.Decision + alias Lightning.Scope @type actions :: :access_admin_space | :edit_credential | :delete_credential | :delete_account + | :read_users + | :write_users + + @spec authorize(actions(), User.t() | Scope.t(), any()) :: + boolean() | {:error, Decision.insufficient_scope()} + + # `operator?` is a fact about the caller alone, so the check made before any + # target decides it. + def authorize(action, %Scope{} = scope, _subject) + when action in [:read_users, :write_users], + do: Decision.instance(scope, nil, grant(action), Scope.operator?(scope)) - @spec authorize(actions(), Lightning.Accounts.User.t(), any()) :: boolean def authorize(:access_admin_space, %User{role: role}, _params) do role in [:superuser] end @@ -37,9 +48,11 @@ defmodule Lightning.Policies.Users do end end - def authorize(action, %User{} = authenticated_user, %module{} = credential) - when module in [Credential, OauthClient] and - action in [:edit_credential, :delete_credential] do - authenticated_user.id == credential.user_id + def authorize(action, %User{} = authenticated_user, %OauthClient{} = client) + when action in [:edit_credential, :delete_credential] do + authenticated_user.id == client.user_id end + + defp grant(:read_users), do: "users:read" + defp grant(:write_users), do: "users:write" end diff --git a/lib/lightning/projects.ex b/lib/lightning/projects.ex index 4ac23885296..d36b45f34fa 100644 --- a/lib/lightning/projects.ex +++ b/lib/lightning/projects.ex @@ -503,16 +503,16 @@ defmodule Lightning.Projects do end @doc """ - Gets a single project with it's members via `project_users`. + Gets a single project with its members via `project_users`. Raises `Ecto.NoResultsError` if the Project does not exist. ## Examples - iex> get_project!(123) - %Project{} + iex> get_project_with_users!(123) + %Project{project_users: [%ProjectUser{user: %User{}}]} - iex> get_project!(456) + iex> get_project_with_users!(456) ** (Ecto.NoResultsError) """ @@ -548,21 +548,30 @@ defmodule Lightning.Projects do end @doc """ - Creates a project. + Creates a project, recording `actor` as having created it and added each of + its initial members. + + ## Options + + * `:notify` - email each initial member that they were added (default `true`) ## Examples - iex> create_project(%{field: value}) + iex> create_project(%{field: value}, actor) {:ok, %Project{}} - iex> create_project(%{field: bad_value}) + iex> create_project(%{field: bad_value}, actor) {:error, %Ecto.Changeset{}} """ - def create_project(attrs \\ %{}, schedule_email? \\ true) do + @spec create_project(map(), Lightning.Actor.t(), keyword()) :: + {:ok, Project.t()} | {:error, Ecto.Changeset.t()} + def create_project(attrs, actor, opts \\ []) do Repo.transact(fn -> - with {:ok, project} <- ProjectHook.handle_create_project(attrs) do - if schedule_email? do + with {:ok, project} <- ProjectHook.handle_create_project(attrs), + project = Repo.preload(project, :project_users), + {:ok, _events} <- record_creation(project, actor) do + if Keyword.get(opts, :notify, true) do schedule_project_addition_emails(%Project{project_users: []}, project) end @@ -574,27 +583,47 @@ defmodule Lightning.Projects do end) end + defp record_creation(project, actor) do + Multi.new() + |> Audit.derive_creation_events(project, actor) + |> Repo.transaction() + |> case do + {:ok, events} -> {:ok, events} + {:error, _operation, changeset, _changes_so_far} -> {:error, changeset} + end + end + @doc """ - Updates a project. + Updates a project, recording an audit event for each audited field it + changes in the same transaction. + + A refused audit event is `{:error, :audit_failed}`, not a changeset, so a + form never renders it as the caller's mistake. + + Returns `{:error, :concurrent_edit}` when another request deleted the + project while this one ran. ## Examples - iex> update_project(project, %{field: new_value}) + iex> update_project(project, %{field: new_value}, actor) {:ok, %Project{}} - iex> update_project(project, %{field: bad_value}) + iex> update_project(project, %{field: bad_value}, actor) {:error, %Ecto.Changeset{}} """ - def update_project(%Project{} = project, attrs, user \\ nil) do + @spec update_project(Project.t(), map(), Lightning.Actor.t()) :: + {:ok, Project.t()} + | {:error, Ecto.Changeset.t() | :audit_failed | :concurrent_edit} + def update_project(%Project{} = project, attrs, actor) do changeset = project |> Project.changeset(attrs) |> ProjectHook.handle_project_validation() Multi.new() - |> Multi.update(:project, changeset) - |> maybe_audit_changes(changeset, user) + |> Multi.update(:project, changeset, stale_error_field: :id) + |> Audit.derive_events(changeset, actor) |> Repo.transaction() |> case do {:ok, %{project: updated_project}} -> @@ -606,24 +635,17 @@ defmodule Lightning.Projects do {:ok, updated_project} + {:error, :project, %Ecto.Changeset{errors: [id: {_, stale: true}]}, _} -> + {:error, :concurrent_edit} + {:error, :project, changeset, _changes_so_far} -> {:error, changeset} - # 2024-10-29 Not tested at module-level - # due to the difficulty of simulating a failure - # without mocking {:error, _operation, _changeset, _changes_so_far} -> - {:error, :not_related_to_project} + {:error, :audit_failed} end end - defp maybe_audit_changes(multi, _changeset, nil), do: multi - - defp maybe_audit_changes(multi, changeset, user) do - multi - |> Audit.derive_events(changeset, user) - end - @spec update_project_with_users(Project.t(), map(), User.t(), boolean()) :: {:ok, Project.t()} | {:error, Ecto.Changeset.t()} def update_project_with_users( @@ -638,6 +660,7 @@ defmodule Lightning.Projects do project |> membership_multi(changeset, actor) + |> Audit.derive_events(changeset, actor) |> Repo.transaction() |> case do {:ok, %{project: updated_project, membership_changes: changes}} -> @@ -1124,18 +1147,7 @@ defmodule Lightning.Projects do |> Repo.all() end - @doc """ - Builds a query to retrieve projects associated with a user. - - ## Parameters - - user: The user struct for which projects are being queried. - - opts: Keyword list of options including :include for associations to preload and :order_by for sorting. - - ## Returns - - An Ecto queryable struct to fetch projects. - """ - @spec projects_for_user_query(user :: User.t()) :: Ecto.Queryable.t() - def projects_for_user_query(%User{id: user_id}) do + defp projects_for_user_query(%User{id: user_id}) do from(p in Project, join: pu in assoc(p, :project_users), where: @@ -1146,6 +1158,16 @@ defmodule Lightning.Projects do ) end + @doc """ + A page of the projects `user` is a member of, by name. + """ + @spec paginate_projects_for_user(User.t(), keyword()) :: Scrivener.Page.t() + def paginate_projects_for_user(%User{} = user, page) do + user + |> projects_for_user_query() + |> Repo.paginate(page) + end + @doc """ Fetches projects for a given user from the database. @@ -1175,17 +1197,6 @@ defmodule Lightning.Projects do |> Repo.all() end - @doc """ - Returns the ids of every project the user holds a membership on, at any - depth. Unlike `get_projects_for_user/1` (top-level roots only), this includes - sandboxes, so it matches the membership check used by `:access_project`. - """ - @spec member_project_ids(User.t()) :: [Ecto.UUID.t()] - def member_project_ids(%User{id: user_id}) do - from(pu in ProjectUser, where: pu.user_id == ^user_id, select: pu.project_id) - |> Repo.all() - end - @doc """ Returns the user-visible project tree as a list of `ProjectTreeItem`s ready for a hierarchical render (walk by grouping on `parent_id`). @@ -2241,10 +2252,10 @@ defmodule Lightning.Projects do end @doc """ - Creates a sandbox under the given `parent` by delegating to `create_project/2`. + Creates a sandbox under `parent` through `create_project/3`, recording + `actor` as having created it. - This is a convenience wrapper that sets `:parent_id` and preserves the - existing behavior around collaborator emails (off by default unless `schedule_email?` is `true`). + Emails no collaborators. ## Notes @@ -2258,10 +2269,12 @@ defmodule Lightning.Projects do * `{:ok, %Project{}}` on success * `{:error, %Ecto.Changeset{}}` on validation/unique errors """ - @spec create_sandbox(Project.t(), map(), boolean()) :: + @spec create_sandbox(Project.t(), map(), Lightning.Actor.t()) :: {:ok, Project.t()} | {:error, Ecto.Changeset.t()} - def create_sandbox(%Project{id: parent_id}, attrs, schedule_email? \\ false) do - attrs |> Map.put(:parent_id, parent_id) |> create_project(schedule_email?) + def create_sandbox(%Project{id: parent_id}, attrs, actor) do + attrs + |> Map.put(:parent_id, parent_id) + |> create_project(actor, notify: false) end @doc """ @@ -2494,11 +2507,19 @@ defmodule Lightning.Projects do * `{:ok, updated_sandbox}` - Successfully updated * `{:error, :unauthorized}` - Actor lacks permission * `{:error, :not_found}` - Sandbox not found + * `{:error, :audit_failed}` - The audit event was refused + * `{:error, :concurrent_edit}` - Another request deleted the sandbox while + this one ran * `{:error, changeset}` - Validation error """ @spec update_sandbox(Project.t() | Ecto.UUID.t(), User.t(), map()) :: {:ok, Project.t()} - | {:error, :unauthorized | :not_found | Ecto.Changeset.t()} + | {:error, + :unauthorized + | :not_found + | :audit_failed + | :concurrent_edit + | Ecto.Changeset.t()} defdelegate update_sandbox(sandbox, actor, attrs), to: Sandboxes, as: :update_sandbox diff --git a/lib/lightning/projects/audit.ex b/lib/lightning/projects/audit.ex index 5ddeac1d857..8fc28f16b97 100644 --- a/lib/lightning/projects/audit.ex +++ b/lib/lightning/projects/audit.ex @@ -10,18 +10,24 @@ defmodule Lightning.Projects.Audit do "collaborator_added", "collaborator_removed", "collaborator_role_changed", + "created", "dataclip_retention_period_updated", + "description_updated", "history_retention_period_updated", + "name_updated", "requires_mfa_updated" ] alias Ecto.Multi + alias Lightning.Projects.Project def derive_events(multi, changeset, user) do [ :allow_support_access, :dataclip_retention_period, + :description, :history_retention_period, + :name, :requires_mfa ] |> Enum.reduce(multi, fn field, multi -> @@ -38,6 +44,26 @@ defmodule Lightning.Projects.Audit do end) end + @doc """ + Appends the events for a newly created project: `created`, then one + `collaborator_added` per member in its `project_users`. + """ + def derive_creation_events(multi, %Project{} = project, actor) do + multi + |> Multi.insert( + :audit_created, + event("created", project.id, actor, %{ + before: nil, + after: %{"name" => project.name, "description" => project.description} + }) + ) + |> derive_membership_events( + project.id, + Enum.map(project.project_users, &{:added, &1}), + actor + ) + end + @doc """ Appends an audit event for every collaborator change in `changes`. diff --git a/lib/lightning/projects/environment.ex b/lib/lightning/projects/environment.ex index 3e059ff2439..ab580b61aec 100644 --- a/lib/lightning/projects/environment.ex +++ b/lib/lightning/projects/environment.ex @@ -46,7 +46,7 @@ defmodule Lightning.Projects.Environment do end # Repo.get raises Ecto.Query.CastError on a malformed binary_id, which turns a - # refusal into a 500. Cast first, the same way `Projects.Scope` does. + # refusal into a 500. Cast first, the same way `Lightning.Scope` does. def fetch(project_id) when is_binary(project_id) do case Ecto.UUID.cast(project_id) do {:ok, uuid} -> Project |> Repo.get(uuid) |> from_project() diff --git a/lib/lightning/projects/project.ex b/lib/lightning/projects/project.ex index b8046faf01b..9defc5e6c40 100644 --- a/lib/lightning/projects/project.ex +++ b/lib/lightning/projects/project.ex @@ -127,7 +127,7 @@ defmodule Lightning.Projects.Project do defp disallow_self_parent(changeset) do if get_field(changeset, :parent_id) == changeset.data.id, - do: add_error(changeset, :parent_id, "cannot be self"), + do: add_error(changeset, :parent_id, "cannot be self", code: :not_allowed), else: changeset end @@ -262,7 +262,8 @@ defmodule Lightning.Projects.Project do add_error( changeset, :dataclip_retention_period, - "dataclip retention period must be less or equal to the history retention period" + "dataclip retention period must be less or equal to the history retention period", + code: :out_of_range ) else changeset @@ -299,6 +300,31 @@ defmodule Lightning.Projects.Project do |> cast_assoc(:project_users, required: true, sort_param: :users_sort) |> validate() |> validate_project_owner() + |> write_owner_last() + |> unique_constraint(:id, name: :projects_pkey) + end + + # Children are written in list order and the one-owner index can't be + # deferred, so an ownership handover only succeeds if the old owner's + # demotion reaches the database before the new owner's row. A new project + # has no owner to demote, so its members keep the order they were given. + defp write_owner_last(%{data: %{__meta__: %{state: :built}}} = changeset), + do: changeset + + defp write_owner_last(changeset) do + case get_change(changeset, :project_users) do + nil -> + changeset + + project_users -> + {owners, others} = + Enum.split_with(project_users, fn project_user -> + project_user.action in [:insert, :update] and + get_field(project_user, :role) == :owner + end) + + put_change(changeset, :project_users, others ++ owners) + end end defp validate_project_owner(changeset) do @@ -316,11 +342,14 @@ defmodule Lightning.Projects.Project do add_error( changeset, :owner, - "Every project must have exactly one owner. Please specify one below." + "Every project must have exactly one owner. Please specify one below.", + code: :one_owner ) _more_than_1 -> - add_error(changeset, :owner, "A project can have only one owner.") + add_error(changeset, :owner, "A project can have only one owner.", + code: :one_owner + ) end end end diff --git a/lib/lightning/projects/provisioner.ex b/lib/lightning/projects/provisioner.ex index dc3ab049022..b7ea274ae6b 100644 --- a/lib/lightning/projects/provisioner.ex +++ b/lib/lightning/projects/provisioner.ex @@ -28,7 +28,6 @@ defmodule Lightning.Projects.Provisioner do alias Lightning.Services.CollectionHook alias Lightning.Services.UsageLimiter alias Lightning.VersionControl.ProjectRepoConnection - alias Lightning.VersionControl.VersionControlUsageLimiter alias Lightning.Validators alias Lightning.Workflows @@ -43,6 +42,8 @@ defmodule Lightning.Projects.Provisioner do alias Lightning.Workflows.WorkflowUsageLimiter alias Lightning.WorkflowVersions + require Logger + @doc """ Import a project document into the database. @@ -50,6 +51,10 @@ defmodule Lightning.Projects.Provisioner do (workflows, project credentials, collections) inside a single transaction, then fires audit, version-bump, and snapshot side effects. + Returns `{:error, :concurrent_edit}`, with nothing written, when another + request changed a workflow or channel the document writes, or deleted any + row it writes, while the import ran, unless `:allow_stale` is set. + Generic pipeline shared by YAML provisioning, CLI deploys, GitHub syncs, and sandbox merges. It only acts on what the document contains — sandbox-specific behaviours (credential cloning, dataclip copying, @@ -75,10 +80,10 @@ defmodule Lightning.Projects.Provisioner do map(), keyword() ) :: - {:error, - Ecto.Changeset.t(Project.t()) - | Lightning.Extensions.UsageLimiting.message()} - | {:ok, Project.t()} + {:ok, Project.t()} + | {:error, Ecto.Changeset.t(Project.t())} + | {:error, :concurrent_edit} + | Lightning.Extensions.UsageLimiting.error() def import_document(project, user_or_repo_connection, data, opts \\ []) def import_document(nil, %User{} = user, data, opts) do @@ -93,53 +98,55 @@ defmodule Lightning.Projects.Provisioner do release = Keyword.get(opts, :release) + project_changeset = + build_import_changeset(project, user_or_repo_connection, data) + result = - Repo.transact(fn -> - with :ok <- - maybe_limit_provisioning(project.id, user_or_repo_connection), - project_changeset <- - build_import_changeset(project, user_or_repo_connection, data), - edges_to_cleanup <- - edges_referencing_deleted_jobs(project_changeset), - :ok <- release_paths_of_soft_deleted_workflows(project_changeset), - {:ok, %{workflows: workflows} = project} <- - Repo.insert_or_update(project_changeset, allow_stale: allow_stale), - :ok <- check_credential_scoping(project, project_changeset), - :ok <- cleanup_orphaned_edges(edges_to_cleanup), - :ok <- - disable_triggers_for_soft_deleted_workflows(project_changeset), - :ok <- release_paths_of_hidden_workflows(project), - :ok <- handle_collection_deletion(project_changeset), - updated_project <- preload_dependencies(project), - {:ok, _changes} <- - audit_workflows(project_changeset, user_or_repo_connection), - {:ok, _changes} <- - audit_channels(project_changeset, user_or_repo_connection), - {:ok, _changes} <- - update_workflows_version( - project_changeset, - updated_project.workflows - ), - {:ok, _changes} <- - create_snapshots( - project_changeset, - updated_project.workflows, - user_or_repo_connection, - release - ) do - Enum.each(workflows, &Workflows.Events.workflow_updated/1) - - workflows - |> Enum.filter(& &1.deleted_at) - |> Enum.each(&Lightning.Projects.Events.workflow_deleted/1) - - Lightning.Projects.SandboxPromExPlugin.fire_provisioner_import_event( - Lightning.Projects.Project.sandbox?(updated_project) - ) + with :ok <- + maybe_limit_provisioning(project_changeset, user_or_repo_connection) do + Repo.transact(fn -> + with edges_to_cleanup <- + edges_referencing_deleted_jobs(project_changeset), + :ok <- release_paths_of_soft_deleted_workflows(project_changeset), + {:ok, %{workflows: workflows} = project} <- + write_document(project_changeset, allow_stale), + :ok <- check_credential_scoping(project, project_changeset), + :ok <- cleanup_orphaned_edges(edges_to_cleanup), + :ok <- + disable_triggers_for_soft_deleted_workflows(project_changeset), + :ok <- release_paths_of_hidden_workflows(project), + :ok <- handle_collection_deletion(project_changeset), + updated_project <- preload_dependencies(project), + {:ok, _changes} <- + audit_workflows(project_changeset, user_or_repo_connection), + {:ok, _changes} <- + audit_channels(project_changeset, user_or_repo_connection), + {:ok, _changes} <- + update_workflows_version( + project_changeset, + updated_project.workflows + ), + {:ok, _changes} <- + create_snapshots( + project_changeset, + updated_project.workflows, + user_or_repo_connection, + release + ) do + Enum.each(workflows, &Workflows.Events.workflow_updated/1) + + workflows + |> Enum.filter(& &1.deleted_at) + |> Enum.each(&Lightning.Projects.Events.workflow_deleted/1) + + Lightning.Projects.SandboxPromExPlugin.fire_provisioner_import_event( + Lightning.Projects.Project.sandbox?(updated_project) + ) - {:ok, updated_project} - end - end) + {:ok, updated_project} + end + end) + end case result do {:ok, updated_project} -> @@ -151,11 +158,28 @@ defmodule Lightning.Projects.Provisioner do {:ok, updated_project} + {:error, %Ecto.Changeset{} = changeset} -> + {:error, + changeset + |> report_destination_credentials() + |> report_project_credential_links()} + error -> error end end + defp write_document(changeset, allow_stale) do + Repo.insert_or_update(changeset, allow_stale: allow_stale) + rescue + # Ecto hands `allow_stale` down to the workflows, jobs and channels this + # changeset writes through cast_assoc, but not `stale_error_field`, so a + # nested row another request changed can only be reported by its raise. + e in Ecto.StaleEntryError -> + Logger.info("Import lost a race: #{Exception.message(e)}") + {:error, :concurrent_edit} + end + @doc """ Extracts the workflow ids carried by an import/merge document that a live collaborative document should reconcile against, excluding workflows the @@ -305,21 +329,84 @@ defmodule Lightning.Projects.Provisioner do end defp add_channel_credential_error(channel_cs) do - message = Scoping.violation_message(:project_credential_id) + add_error( + channel_cs, + :destination_credential_id, + Scoping.violation_message(:project_credential_id), + code: :credential_not_allowed + ) + end - case get_change(channel_cs, :destination_auth_method) do + # The request names a channel's credential as `destination_credential_id`, + # which `channel_changeset/2` turns into a `destination_auth_method`, so that + # child's errors are reported on the key the request sent. + defp report_destination_credentials(changeset) do + case get_change(changeset, :channels) do nil -> - add_error(channel_cs, :project_credential_id, message) + changeset - auth_method_cs -> + channels -> put_change( - channel_cs, - :destination_auth_method, - add_error(auth_method_cs, :project_credential_id, message) + changeset, + :channels, + Enum.map(channels, &lift_destination_errors/1) ) end end + defp lift_destination_errors( + %Ecto.Changeset{ + changes: %{ + destination_auth_method: %Ecto.Changeset{errors: [_ | _]} = method + } + } = channel_cs + ) do + lifted = + Enum.map(method.errors, fn {_field, error} -> + {:destination_credential_id, error} + end) + + %{channel_cs | errors: lifted ++ channel_cs.errors, valid?: false} + |> put_change(:destination_auth_method, %{method | errors: [], valid?: true}) + end + + defp lift_destination_errors(channel_cs), do: channel_cs + + # A link `maybe_add_project_credentials/2` builds carries no params, and the + # links aren't in the request's order, so its one possible error, an id + # another link already holds, is reported on the project at the position the + # request gave that id. + defp report_project_credential_links(changeset) do + case get_change(changeset, :project_credentials) do + nil -> + changeset + + links -> + request_ids = + Enum.map(changeset.params["project_credentials"], & &1["id"]) + + {links, changeset} = + Enum.map_reduce(links, changeset, fn + %{valid?: true} = link, project_cs -> + {link, project_cs} + + link, project_cs -> + index = Enum.find_index(request_ids, &(&1 == link.data.id)) + + {%{link | errors: [], valid?: true}, + add_error( + project_cs, + :project_credentials, + "has already been taken", + key: ["project_credentials", index, "id"], + code: :taken + )} + end) + + put_change(changeset, :project_credentials, links) + end + end + defp build_import_changeset(project, user_or_repo_connection, data) do project |> preload_dependencies() @@ -566,7 +653,7 @@ defmodule Lightning.Projects.Provisioner do changeset {:error, _reason, %{text: message}} -> - add_error(changeset, :id, message) + add_error(changeset, :id, message, code: :exceeds_limit) end end) |> then(fn changeset -> @@ -575,7 +662,7 @@ defmodule Lightning.Projects.Provisioner do changeset {:error, _reason, %{text: message}} -> - add_error(changeset, :id, message) + add_error(changeset, :id, message, code: :exceeds_limit) end end) end @@ -731,6 +818,7 @@ defmodule Lightning.Projects.Provisioner do collection |> cast(attrs, [:id, :name, :delete]) |> validate_required([:id]) + |> unique_constraint(:id, name: :collections_pkey) |> maybe_mark_for_deletion() |> validate_extraneous_params() |> Collection.validate() @@ -748,6 +836,7 @@ defmodule Lightning.Projects.Provisioner do :delete ]) |> validate_required([:id, :name, :destination_url]) + |> unique_constraint(:id, name: :channels_pkey) |> block_channel_deletion() |> cast_assoc(:destination_auth_method, with: &ChannelAuthMethod.changeset/2) |> validate_extraneous_params() @@ -764,7 +853,8 @@ defmodule Lightning.Projects.Provisioner do changeset, :delete, "channel deletion is not supported via the provisioning API; " <> - "delete from the dashboard instead" + "delete from the dashboard instead", + code: :not_supported ) else changeset @@ -888,7 +978,7 @@ defmodule Lightning.Projects.Provisioner do if is_map(attrs) and (Map.has_key?(attrs, key) or Map.has_key?(attrs, field)) do - add_error(acc, :base, "extraneous parameters: %{params}", params: key) + add_unknown_key(acc, key) else acc end @@ -916,7 +1006,9 @@ defmodule Lightning.Projects.Provisioner do {true, others} when map_size(others) > 0 -> changeset - |> add_error(:delete, "cannot change or add a record while deleting") + |> add_error(:delete, "cannot change or add a record while deleting", + code: :deleting + ) _ -> changeset @@ -932,13 +1024,19 @@ defmodule Lightning.Projects.Provisioner do {true, others} when map_size(others) > 0 -> changeset - |> add_error(:delete, "cannot change or add a record while deleting") + |> add_error(:delete, "cannot change or add a record while deleting", + code: :deleting + ) _ -> changeset end end + # New links are filtered, then put ahead of the stored ones, so their + # positions aren't the request's. A name that matches none of the caller's + # credentials is reported on the project, at the link's position in the + # request. defp maybe_add_project_credentials(changeset, user_or_repo_connection) do credentials_params = changeset.params["project_credentials"] @@ -951,38 +1049,34 @@ defmodule Lightning.Projects.Provisioner do existing_project_credential_ids = Enum.map(changeset.data.project_credentials, fn pc -> pc.id end) - new_credential_params = - Enum.filter(credentials_params, fn cred_params -> + {new_links, changeset} = + credentials_params + |> Enum.with_index() + # Keeps only unlinked items the caller owns. An item already linked, or + # an unlinked one naming another owner, is skipped with no error and the + # request still answers 201, as kit's sync sends links it doesn't own. + |> Enum.filter(fn {cred_params, _index} -> cred_params["id"] not in existing_project_credential_ids and cred_params["owner"] == user_or_repo_connection.email end) - - new_project_creds_to_add = - Enum.map(new_credential_params, fn cred_params -> - # The stored name went through `validate_name/3` and the spec body - # never did. Without normalising both sides, a spec carrying a - # decomposed accent is refused with an error naming a credential the - # user can see on screen. - wanted = Validators.normalize_name_for_match(cred_params["name"]) - - credential = - Enum.find(user_credentials, fn cred -> - Validators.normalize_name_for_match(cred.name) == wanted - end) - - if credential do - change(%ProjectCredential{ - id: cred_params["id"], - credential_id: credential.id - }) - else - change(%ProjectCredential{ - id: cred_params["id"] - }) - |> add_error( - :credential, - "No credential found with name #{cred_params["name"]}" - ) + |> Enum.flat_map_reduce(changeset, fn {cred_params, index}, acc -> + case find_credential_by_name(user_credentials, cred_params["name"]) do + nil -> + {[], + add_error(acc, :project_credentials, "names nothing you can see", + key: ["project_credentials", index, "name"], + code: :not_found + )} + + credential -> + {[ + %ProjectCredential{ + id: cred_params["id"], + credential_id: credential.id + } + |> change() + |> unique_constraint(:id, name: :project_credentials_pkey) + ], acc} end end) @@ -992,13 +1086,24 @@ defmodule Lightning.Projects.Provisioner do put_assoc( changeset, :project_credentials, - new_project_creds_to_add ++ project_credentials + new_links ++ project_credentials ) else changeset end end + # The stored name went through `validate_name/3` and the spec body never + # did. Without normalising both sides, a spec carrying a decomposed accent is + # refused, naming a credential the user can see on screen. + defp find_credential_by_name(credentials, name) do + wanted = Validators.normalize_name_for_match(name) + + Enum.find(credentials, fn credential -> + Validators.normalize_name_for_match(credential.name) == wanted + end) + end + @doc """ Validate that there are no extraneous parameters in the changeset. @@ -1016,16 +1121,20 @@ defmodule Lightning.Projects.Provisioner do field_keys = changeset.types |> Map.keys() |> MapSet.new(&to_string/1) - extraneous_params = MapSet.difference(param_keys, field_keys) + param_keys + |> MapSet.difference(field_keys) + |> Enum.sort() + |> Enum.reduce(changeset, &add_unknown_key(&2, &1)) + end - if MapSet.size(extraneous_params) > 0 do - add_error(changeset, :base, "extraneous parameters: %{params}", - params: MapSet.to_list(extraneous_params) |> Enum.join(", ") + # A key from the request can't safely become an atom, so the error names it + # in `key:` instead of as its field. + defp add_unknown_key(changeset, key), + do: + add_error(changeset, :base, "is not accepted", + key: key, + code: :unknown_key ) - else - changeset - end - end # A hidden row must not keep a name reserved. Released whatever the trigger's # enabled state, and before the insert so one document can drop a workflow and @@ -1088,9 +1197,17 @@ defmodule Lightning.Projects.Provisioner do end end - defp maybe_limit_provisioning(project_id, %ProjectRepoConnection{}) do - VersionControlUsageLimiter.limit_github_sync(project_id) + # A limit is checked once the document is valid, so an invalid one is refused + # for what is wrong with it. The limiter's reason is kept, since each answers + # differently. + defp maybe_limit_provisioning( + %Ecto.Changeset{valid?: true, data: %Project{id: project_id}}, + %ProjectRepoConnection{} + ) do + UsageLimiter.limit_action(%Action{type: :github_sync}, %Context{ + project_id: project_id + }) end - defp maybe_limit_provisioning(_project_id, %User{}), do: :ok + defp maybe_limit_provisioning(_changeset, _actor), do: :ok end diff --git a/lib/lightning/projects/sandboxes.ex b/lib/lightning/projects/sandboxes.ex index 2bfeb556f47..801c636f394 100644 --- a/lib/lightning/projects/sandboxes.ex +++ b/lib/lightning/projects/sandboxes.ex @@ -726,6 +726,9 @@ defmodule Lightning.Projects.Sandboxes do * `{:ok, updated_sandbox}` - Successfully updated sandbox * `{:error, :unauthorized}` - Actor lacks permission on sandbox * `{:error, :not_found}` - Sandbox ID not found (when using string ID) + * `{:error, :audit_failed}` - The audit event was refused + * `{:error, :concurrent_edit}` - Another request deleted the sandbox while + this one ran * `{:error, changeset}` - Validation error ## Example @@ -739,7 +742,12 @@ defmodule Lightning.Projects.Sandboxes do """ @spec update_sandbox(Project.t() | Ecto.UUID.t(), User.t(), map()) :: {:ok, Project.t()} - | {:error, :unauthorized | :not_found | Ecto.Changeset.t()} + | {:error, + :unauthorized + | :not_found + | :audit_failed + | :concurrent_edit + | Ecto.Changeset.t()} def update_sandbox(%Project{} = sandbox, %User{} = actor, attrs) when is_map(attrs) do Permissions.can?( @@ -1027,7 +1035,11 @@ defmodule Lightning.Projects.Sandboxes do sandbox_env ) - case create_empty_sandbox(parent_with_data, sandbox_attrs) do + case Lightning.Projects.create_sandbox( + parent_with_data, + sandbox_attrs, + actor + ) do {:ok, sandbox} -> sandbox |> Repo.preload(:project_users) @@ -1083,10 +1095,6 @@ defmodule Lightning.Projects.Sandboxes do }) end - defp create_empty_sandbox(parent, attrs) do - Lightning.Projects.create_sandbox(parent, attrs, false) - end - defp clone_credentials_from_parent(sandbox, parent) do current_time = DateTime.utc_now() |> DateTime.truncate(:second) diff --git a/lib/lightning/projects/scope.ex b/lib/lightning/projects/scope.ex deleted file mode 100644 index bcfeacd5cd3..00000000000 --- a/lib/lightning/projects/scope.ex +++ /dev/null @@ -1,225 +0,0 @@ -defmodule Lightning.Projects.Scope do - @moduledoc """ - What standing does an actor have in a project, right now. - - `fetch/2` establishes facts, not permission: it reports the actor's role, and - refuses outright for a project that does not exist or is scheduled for - deletion. Whether that standing is enough for an action is the policy's - judgement — see `Lightning.Policies.ProjectUsers.permitted?/2`. - - {:ok, %Scope{role: :editor}} - {:ok, %Scope{role: nil}} no membership row - {:error, :no_such_project} - {:error, :project_scheduled_for_deletion} - {:error, :connection_not_for_this_project} repo-connection actor only - - So `{:ok, _}` is not itself an authorisation result — a non-member of a live - project gets `role: nil`, which support access can still make meaningful. - - Two support fields, deliberately. `support_user?` is the raw account flag. - `support?` additionally requires the project's `allow_support_access`. - Anything granting access to a customer's project wants `support?`. - - A `%ProjectRepoConnection{}` has no membership row, so its standing *is* - which project it belongs to; `fetch/2` checks that, and its `role` is always - `nil`. Ownership is checked before liveness, so a connection learns nothing - about a project that is not its own. - - Scheduling deletion removes no membership rows and revokes no token, so this - refusal is the whole of the offboarding gate during the purge window. - - ## When not to reach for it - - * `ProjectUsers` self-actions — they act on a *specific* membership row and - ask whether it is the caller's own, which a scope cannot answer. - * `Sandboxes` `:cancel_scheduled_deletion` — its subject is scheduled by - definition, so the liveness gate would make restore impossible. - * `Provisioning`'s `%Project{id: nil}` clause — no row to read, no members - to consult. - - Every other project-scoped decision resolves here; no policy module reads - `Project.scheduled_deletion` directly. - - Likewise, no policy module reads `requires_mfa`/`mfa_enabled` directly. That - rule is written once, privately, below; `mfa_satisfied?` is the only way to - reach it. - """ - import Ecto.Query - - alias Lightning.Accounts.User - alias Lightning.Projects.Project - alias Lightning.Projects.ProjectUser - alias Lightning.Repo - alias Lightning.VersionControl.ProjectRepoConnection - - # `mfa_satisfied?` defaults to false so a `%Scope{}` that never went through - # `build/2` — a fixture, or a clause for a future actor type that forgets to - # set it — fails closed rather than silently bypassing the requirement. - defstruct [ - :project, - :actor, - :project_user, - :role, - support_user?: false, - support?: false, - mfa_satisfied?: false - ] - - @type t :: %__MODULE__{ - project: Project.t(), - actor: actor(), - project_user: ProjectUser.t() | nil, - role: :owner | :admin | :editor | :viewer | nil, - support_user?: boolean(), - support?: boolean(), - mfa_satisfied?: boolean() - } - - @typedoc """ - Whoever is asking. A `%User{}` may hold a role; a `%ProjectRepoConnection{}` - never does. - """ - @type actor :: User.t() | ProjectRepoConnection.t() - - @typedoc """ - Anything that identifies a project: a loaded `%Project{}`, a project id, or - any struct or map carrying a `:project_id` — a `%Dataclip{}`, a - `%Collection{}`, a `%Projects.File{}`, a `%ProjectUser{}`. - """ - # The map member is deliberately open — a closed `%{project_id: Ecto.UUID.t()}` - # would put every real caller's struct out of contract. - @type subject :: - Project.t() - | Ecto.UUID.t() - | %{:project_id => Ecto.UUID.t(), optional(any()) => any()} - | nil - - @type error :: - :no_such_project - | :project_scheduled_for_deletion - | :connection_not_for_this_project - - @doc """ - The actor's standing in the project. - - Returns `{:error, :project_scheduled_for_deletion}` for a project that is - winding down, `{:error, :no_such_project}` when the subject does not identify - a project that exists, and — for a `%ProjectRepoConnection{}` actor — - `{:error, :connection_not_for_this_project}` when the resolved project is not - the one the connection belongs to. - """ - @spec fetch(actor(), subject()) :: {:ok, t()} | {:error, error()} - def fetch(%User{} = user, subject) do - with {:ok, project} <- resolve_project(subject), - :ok <- still_operable(project) do - {:ok, build(user, project)} - end - end - - # Ownership before liveness, deliberately, and pinned by a test. Reversed, a - # connection asking about someone else's project would be told whether that - # project is being wound down. - def fetch(%ProjectRepoConnection{} = repo_connection, subject) do - with {:ok, project} <- resolve_project(subject), - :ok <- connection_owns?(repo_connection, project), - :ok <- still_operable(project) do - {:ok, build(repo_connection, project)} - end - end - - @doc """ - Whether the user holds one of `roles` in the project. - - `false` for a project that does not exist or is scheduled for deletion, - `false` for a user with no membership row, and `false` for a user who has - not met the project's MFA requirement. It never consults `support?`, so a - rule that admits a support user has to decide on the `%Scope{}` itself. - """ - @spec role_in?(User.t(), subject(), [atom()]) :: boolean() - def role_in?(%User{} = user, subject, roles) do - case fetch(user, subject) do - {:ok, %__MODULE__{role: role, mfa_satisfied?: mfa_satisfied?}} -> - role in roles and mfa_satisfied? - - {:error, _reason} -> - false - end - end - - defp still_operable(%Project{scheduled_deletion: nil}), do: :ok - - defp still_operable(%Project{}), - do: {:error, :project_scheduled_for_deletion} - - # Answered here rather than at the call site so no policy can forget the - # comparison. The repeated `id` binds both patterns to the same value. - defp connection_owns?(%ProjectRepoConnection{project_id: id}, %Project{id: id}), - do: :ok - - defp connection_owns?(%ProjectRepoConnection{}, %Project{}), - do: {:error, :connection_not_for_this_project} - - # A subject names a project; it does not describe one. Even a `%Project{}` is - # read for its id and reloaded, so no caller can assert its own standing from - # a struct it happens to hold — whether assembled in code or loaded before the - # facts changed. - defp resolve_project(%Project{id: id}), do: load(id) - defp resolve_project(%{project_id: id}), do: load(id) - defp resolve_project(id) when is_binary(id), do: load(id) - defp resolve_project(_), do: {:error, :no_such_project} - - # Repo.get raises Ecto.Query.CastError on a malformed binary_id, which would - # turn a denial into a 500. Cast first. - defp load(id) when is_binary(id) do - with {:ok, uuid} <- Ecto.UUID.cast(id), - %Project{} = project <- Repo.get(Project, uuid) do - {:ok, project} - else - _ -> {:error, :no_such_project} - end - end - - defp load(_), do: {:error, :no_such_project} - - defp build(%User{} = user, %Project{} = project) do - project_user = - Repo.one( - from pu in ProjectUser, - where: pu.project_id == ^project.id and pu.user_id == ^user.id - ) - - %__MODULE__{ - project: project, - actor: user, - project_user: project_user, - role: project_user && project_user.role, - support_user?: user.support_user, - support?: user.support_user and project.allow_support_access, - mfa_satisfied?: mfa_satisfied?(project, user) - } - end - - # No membership row to look up, and no support-user concept. Its authority - # comes from the token, which the caller has already verified. - # - # No MFA concept either — a machine credential cannot enrol — so the - # exemption is set here rather than defaulted, to keep it a decision a - # reviewer can see and a future actor type cannot inherit by accident. - defp build(%ProjectRepoConnection{} = repo_connection, %Project{} = project) do - %__MODULE__{ - project: project, - actor: repo_connection, - role: nil, - mfa_satisfied?: true - } - end - - # Only an enrolled `true` counts; an unset flag is not enrolled. - # - # A support user is a human reading project data, so the rule binds them as - # much as a member and nothing here consults `support_user`. - defp mfa_satisfied?(%Project{requires_mfa: true}, %User{mfa_enabled: enrolled}), - do: enrolled == true - - defp mfa_satisfied?(%Project{}, %User{}), do: true -end diff --git a/lib/lightning/runs.ex b/lib/lightning/runs.ex index d9a2ec1674d..fa523780dcf 100644 --- a/lib/lightning/runs.ex +++ b/lib/lightning/runs.ex @@ -7,8 +7,10 @@ defmodule Lightning.Runs do import Ecto.Query alias Ecto.Multi + alias Lightning.Accounts.User alias Lightning.Extensions.UsageLimiting.Context alias Lightning.Invocation.LogLine + alias Lightning.Projects.Project alias Lightning.Repo alias Lightning.Run alias Lightning.Runs.Events @@ -300,7 +302,7 @@ defmodule Lightning.Runs do |> if do [] else - [{:step_id, "must be associated with the run"}] + [{:step_id, {"must be associated with the run", code: :not_found}}] end end end) @@ -364,7 +366,7 @@ defmodule Lightning.Runs do if step_id in valid_step_ids do [] else - [{:step_id, "must be associated with the run"}] + [{:step_id, {"must be associated with the run", code: :not_found}}] end end @@ -542,21 +544,16 @@ defmodule Lightning.Runs do defdelegate subscribe(run), to: Events @doc """ - Returns a query for runs belonging to a specific project + A page of the runs of `project`, or of every project `user` may access, + narrowed by `filters` (see `Lightning.Invocation.Query.filter_runs/2`). """ - @spec runs_for_project_query(Lightning.Projects.Project.t()) :: - Ecto.Queryable.t() - def runs_for_project_query(%Lightning.Projects.Project{} = project) do - Lightning.Invocation.Query.runs_for(project) - end - - @doc """ - Returns a query for runs accessible to a user - """ - @spec runs_for_user_query(Lightning.Accounts.User.t()) :: - Ecto.Queryable.t() - def runs_for_user_query(%Lightning.Accounts.User{} = user) do - Lightning.Invocation.Query.runs_for(user) + @spec paginate_runs_for(Project.t() | User.t(), map(), keyword()) :: + Scrivener.Page.t() + def paginate_runs_for(project_or_user, filters, page) do + project_or_user + |> Lightning.Invocation.Query.runs_for() + |> Lightning.Invocation.Query.filter_runs(filters) + |> Repo.paginate(page) end @spec get_project_id_for_run(Run.t()) :: Ecto.UUID.t() | nil diff --git a/lib/lightning/runs/handlers.ex b/lib/lightning/runs/handlers.ex index e3d7095f2d5..eed1e847120 100644 --- a/lib/lightning/runs/handlers.ex +++ b/lib/lightning/runs/handlers.ex @@ -308,7 +308,7 @@ defmodule Lightning.Runs.Handlers do |> fetch_existing_job(job_id) |> Enum.reduce(changeset, fn {k, v}, changeset -> if is_nil(v) do - add_error(changeset, k, "does not exist") + add_error(changeset, k, "does not exist", code: :not_found) else changeset end @@ -370,7 +370,9 @@ defmodule Lightning.Runs.Handlers do changeset true -> - add_error(changeset, :input_dataclip_id, "does not exist") + add_error(changeset, :input_dataclip_id, "does not exist", + code: :not_found + ) end end @@ -391,7 +393,9 @@ defmodule Lightning.Runs.Handlers do changeset true -> - add_error(changeset, :credential_id, "does not exist") + add_error(changeset, :credential_id, "does not exist", + code: :not_found + ) end end end @@ -486,7 +490,7 @@ defmodule Lightning.Runs.Handlers do {:error, complete_step |> change() - |> add_error(:step_id, "not found")} + |> add_error(:step_id, "not found", code: :not_found)} error -> error diff --git a/lib/lightning/runs/run.ex b/lib/lightning/runs/run.ex index 1cc4a632c30..9cc851236f9 100644 --- a/lib/lightning/runs/run.ex +++ b/lib/lightning/runs/run.ex @@ -242,14 +242,17 @@ defmodule Lightning.Run do changeset |> add_error( :state, - "cannot mark run #{to} that has not been claimed by a worker" + "cannot mark run #{to} that has not been claimed by a worker", + code: :not_allowed ) {:claimed, :started} -> changeset |> validate_required([:started_at]) {from, to} when from in @final_states and to in @final_states -> - add_error(changeset, :state, "already in completed state") + add_error(changeset, :state, "already in completed state", + code: :not_allowed + ) {from, to} when from in [:claimed, :started] and to in @final_states -> changeset |> validate_required([:finished_at]) diff --git a/lib/lightning/scope.ex b/lib/lightning/scope.ex new file mode 100644 index 00000000000..6b452fd210d --- /dev/null +++ b/lib/lightning/scope.ex @@ -0,0 +1,343 @@ +defmodule Lightning.Scope do + @moduledoc """ + The facts about whoever is calling, and their standing in a project right now. + + `new/1` gives the facts about the caller alone: the `grants` its token + carries (`:all` for a person), its `instance_role`, MFA status, and whether + it can own a credential (only a person can). `fetch/2` adds the facts about + one project, including whether it is scheduled for deletion, and refuses a + project that does not exist. `fetch_all/2` does the same for several projects + in one query. Whether that standing is enough for an action is the policy's + judgement — see `Lightning.Policies.ProjectUsers.permitted?/2`. + + {:ok, %Scope{role: :editor}} + {:ok, %Scope{role: nil}} no membership row + {:ok, %Scope{scheduled_for_deletion?: true}} winding down + {:error, :no_such_project} no such project + {:error, :no_standing} a repo connection naming + another project + + So `{:ok, _}` is not itself an authorisation result — a non-member of a live + project gets `role: nil`, which support access can still make meaningful. + + Two support fields, deliberately. `support_user?` is the raw account flag. + `support?` additionally requires the project's `allow_support_access`. + Anything granting access to a customer's project wants `support?`. + + A `%ProjectRepoConnection{}` has no membership row, so its standing *is* + which project it belongs to; `fetch/2` checks that, and its `role` is always + `nil`. Ownership is checked before anything else is read, so a connection + learns nothing about a project that is not its own, and hears the + `:no_standing` a refused user without a membership row hears from a policy. + + A `%ServiceAccount{}` has no membership row either and is not tied to one + project; its `role` is always `nil`. Its `instance_role` is `:operator`, which + `holds_role?/2` treats as holding every project role. + + Scheduling deletion removes no membership rows and revokes no token, so + during the purge window only the policies' check of `scheduled_for_deletion?` + keeps members out. + + ## When not to reach for it + + `Sandboxes` `:cancel_scheduled_deletion` reads its subject's role directly, + since its subject is scheduled by definition. + + Every other project-scoped decision resolves here; no policy module reads + `Project.scheduled_deletion` directly. + + Likewise, no policy module reads `requires_mfa`/`mfa_enabled` directly. That + rule is written once, privately, below; `mfa_satisfied?` is the only way to + reach it. + """ + import Ecto.Query + + alias Lightning.Accounts.User + alias Lightning.Projects.Project + alias Lightning.Projects.ProjectUser + alias Lightning.Repo + alias Lightning.ServiceAccount + alias Lightning.VersionControl.ProjectRepoConnection + + # `grants`, `mfa_satisfied?` and `can_own_credentials?` default to refusing, + # so a `%Scope{}` that never went through `new/1` — a fixture, or a clause for + # a future actor type that forgets to set them — fails closed rather than + # silently passing. + @enforce_keys [:actor] + defstruct [ + :actor, + grants: MapSet.new(), + instance_role: nil, + support_user?: false, + mfa_satisfied?: false, + can_own_credentials?: false, + project: nil, + project_user: nil, + role: nil, + support?: false, + scheduled_for_deletion?: false + ] + + @type t :: %__MODULE__{ + actor: actor(), + grants: :all | MapSet.t(String.t()), + instance_role: :superuser | :operator | nil, + support_user?: boolean(), + mfa_satisfied?: boolean(), + can_own_credentials?: boolean(), + project: Project.t() | nil, + project_user: ProjectUser.t() | nil, + role: :owner | :admin | :editor | :viewer | nil, + support?: boolean(), + scheduled_for_deletion?: boolean() + } + + @typedoc """ + Whoever is asking. A `%User{}` may hold a role; a `%ProjectRepoConnection{}` + and a `%ServiceAccount{}` never do. + """ + @type actor :: Lightning.Actor.t() | ProjectRepoConnection.t() + + @typedoc """ + Anything that identifies a project: a loaded `%Project{}`, a project id, or + any struct or map carrying a `:project_id` — a `%Dataclip{}`, a + `%Collection{}`, a `%Projects.File{}`, a `%ProjectUser{}`. + """ + # The map member is deliberately open — a closed `%{project_id: Ecto.UUID.t()}` + # would put every real caller's struct out of contract. + @type subject :: + Project.t() + | Ecto.UUID.t() + | %{:project_id => Ecto.UUID.t(), optional(any()) => any()} + | nil + + @type error :: :no_such_project | :no_standing + + @doc """ + The facts about the caller alone, with no project. + + No project means no MFA requirement to meet, so `mfa_satisfied?` is `true`. + """ + @spec new(actor()) :: t() + def new(%User{} = user) do + %__MODULE__{ + actor: user, + grants: :all, + instance_role: if(user.role == :superuser, do: :superuser), + support_user?: user.support_user, + mfa_satisfied?: true, + can_own_credentials?: true + } + end + + def new(%ServiceAccount{} = account) do + %__MODULE__{ + actor: account, + grants: account.grants, + instance_role: :operator, + mfa_satisfied?: true + } + end + + # A repo connection holds only `projects:provision`, so every other + # grant-checked action refuses it. + def new(%ProjectRepoConnection{} = repo_connection) do + %__MODULE__{ + actor: repo_connection, + grants: MapSet.new(["projects:provision"]), + mfa_satisfied?: true + } + end + + @doc """ + The actor's standing in the project. Given a `%Scope{}` from `new/1`, adds + the project's facts to it rather than building it again. + + Returns `{:error, :no_such_project}` when the subject does not identify a + project that exists, and `{:error, :no_standing}` when a + `%ProjectRepoConnection{}` actor names a project other than the connection's + own. A project scheduled for deletion resolves, with `scheduled_for_deletion?` + set; refusing it is the policy's call. + + A `%ServiceAccount{}` resolves any project, with no role. + """ + @spec fetch(t() | actor(), subject()) :: {:ok, t()} | {:error, error()} + def fetch(%__MODULE__{} = scope, subject) do + case resolve(scope, List.wrap(subject_id(subject))) do + [result] -> result + [] -> {:error, :no_such_project} + end + end + + def fetch(actor, subject), do: fetch(new(actor), subject) + + @doc """ + The actor's standing in each of several projects, read in one query, as a + map from project id to the scope `fetch/2` would give. + + An id that is malformed, names no project, or names a project a + `%ProjectRepoConnection{}` actor does not belong to is absent from the map. + """ + @spec fetch_all(t() | actor(), [Ecto.UUID.t()]) :: %{Ecto.UUID.t() => t()} + def fetch_all(%__MODULE__{} = scope, project_ids) do + for {:ok, %__MODULE__{project: project} = project_scope} <- + resolve(scope, project_ids), + into: %{}, + do: {project.id, project_scope} + end + + def fetch_all(actor, project_ids), do: fetch_all(new(actor), project_ids) + + @doc "Whether the caller's token carries `grant`. A person holds every grant." + @spec allows?(t(), String.t()) :: boolean() + def allows?(%__MODULE__{grants: :all}, _grant), do: true + + def allows?(%__MODULE__{grants: grants}, grant), + do: MapSet.member?(grants, grant) + + @doc """ + Whether the caller stands in one of `roles` in the scope's project. An + operator stands in every role. Standing only: it does not consult MFA or + whether the project is being deleted. + """ + @spec holds_role?(t(), [atom()]) :: boolean() + def holds_role?(%__MODULE__{} = scope, roles), + do: operator?(scope) or scope.role in roles + + @doc """ + Whether the caller has any standing in the scope's project at all: a role, + or support access. This is the one definition of standing, so reading a + project and being blocked by MFA can't disagree about it. + """ + @spec has_standing?(t()) :: boolean() + def has_standing?(%__MODULE__{} = scope), + do: holds_role?(scope, [:owner, :admin, :editor, :viewer]) or scope.support? + + @doc "Whether the caller acts for the instance on every project." + @spec operator?(t()) :: boolean() + def operator?(%__MODULE__{instance_role: instance_role}), + do: instance_role == :operator + + @doc """ + Whether `subject` names this scope's project: the project itself, its id, or + anything carrying its `project_id`. + """ + @spec describes?(t(), term()) :: boolean() + def describes?(%__MODULE__{project: %Project{id: id}}, %Project{id: id}), + do: true + + def describes?(%__MODULE__{project: %Project{id: id}}, %{project_id: id}), + do: true + + def describes?(%__MODULE__{project: %Project{id: id}}, subject) + when is_binary(subject), + do: Ecto.UUID.cast(subject) == {:ok, id} + + def describes?(%__MODULE__{}, _subject), do: false + + @doc """ + Whether the actor holds one of `roles` in the project. + + `false` for a project that does not exist or is scheduled for deletion, + `false` for a user with no membership row, and `false` for a user who has + not met the project's MFA requirement. It never consults `support?`, so a + rule that admits a support user has to decide on the `%Scope{}` itself. + """ + @spec role_in?(actor(), subject(), [atom()]) :: boolean() + def role_in?(actor, subject, roles) do + case fetch(actor, subject) do + {:ok, %__MODULE__{} = scope} -> + scope.mfa_satisfied? and not scope.scheduled_for_deletion? and + holds_role?(scope, roles) + + {:error, _reason} -> + false + end + end + + # Answered here rather than at the call site so no policy can forget the + # comparison. The repeated `id` binds both patterns to the same value. Another + # project is refused like a missing one: the connection has no standing on + # it, and a distinct refusal would confirm it exists. + defp connection_owns?(%ProjectRepoConnection{project_id: id}, %Project{id: id}), + do: :ok + + defp connection_owns?(%ProjectRepoConnection{}, %Project{}), + do: {:error, :no_standing} + + defp connection_owns?(_actor, %Project{}), do: :ok + + # Ownership before any other fact, deliberately, and pinned by a test. + # Reversed, a connection asking about someone else's project would be told + # whether that project is being wound down. + defp resolve(%__MODULE__{actor: actor} = scope, project_ids) do + for {project, project_user} <- load(actor, project_ids) do + with :ok <- connection_owns?(actor, project), + do: {:ok, put_project(scope, project, project_user)} + end + end + + # A subject names a project; it does not describe one. Even a `%Project{}` is + # read for its id and reloaded, so no caller can assert its own standing from + # a struct it happens to hold — whether assembled in code or loaded before the + # facts changed. + defp subject_id(%Project{id: id}), do: id + defp subject_id(%{project_id: id}), do: id + defp subject_id(id) when is_binary(id), do: id + defp subject_id(_), do: nil + + # Each project comes with the user's own membership row, or nil. Querying a + # malformed binary_id raises Ecto.Query.CastError, which would turn a denial + # into a 500, so ids are cast first. + defp load(actor, project_ids) do + case for(id <- project_ids, {:ok, uuid} <- [Ecto.UUID.cast(id)], do: uuid) do + [] -> [] + uuids -> Repo.all(load_query(actor, uuids)) + end + end + + defp load_query(%User{id: user_id}, uuids) do + from p in Project, + where: p.id in ^uuids, + left_join: pu in ProjectUser, + on: pu.project_id == p.id and pu.user_id == ^user_id, + select: {p, pu} + end + + defp load_query(_actor, uuids), + do: from(p in Project, where: p.id in ^uuids, select: {p, nil}) + + defp put_project( + %__MODULE__{actor: %User{} = user} = scope, + project, + project_user + ) do + %{ + scope + | project: project, + project_user: project_user, + role: project_user && project_user.role, + support?: scope.support_user? and project.allow_support_access, + mfa_satisfied?: mfa_satisfied?(project, user), + scheduled_for_deletion?: scheduled?(project) + } + end + + # Any other actor has no membership row, and no support-user or MFA concept. + # A machine credential cannot enrol, so `new/1` sets the exemption where a + # reviewer can see it. + defp put_project(%__MODULE__{} = scope, project, _project_user), + do: %{scope | project: project, scheduled_for_deletion?: scheduled?(project)} + + defp scheduled?(%Project{scheduled_deletion: scheduled_deletion}), + do: not is_nil(scheduled_deletion) + + # Only an enrolled `true` counts; an unset flag is not enrolled. + # + # A support user is a human reading project data, so the rule binds them as + # much as a member and nothing here consults `support_user`. + defp mfa_satisfied?(%Project{requires_mfa: true}, %User{mfa_enabled: enrolled}), + do: enrolled == true + + defp mfa_satisfied?(%Project{}, %User{}), do: true +end diff --git a/lib/lightning/service_account.ex b/lib/lightning/service_account.ex index 11a7d06dddd..4c92e2fd5e2 100644 --- a/lib/lightning/service_account.ex +++ b/lib/lightning/service_account.ex @@ -6,15 +6,20 @@ defmodule Lightning.ServiceAccount do service account. Its `uuid`, for places that need one such as an audit event's actor, is a UUIDv5 of the thumbprint's RFC 9278 URI in the RFC 4122 URL namespace, which any stock `uuid5` implementation reproduces. + + `grants` are the scopes of the access token the request came in with. The + account registered at boot holds none, so it is refused every action that + needs a grant. """ @enforce_keys [:id, :uuid, :public_key] - defstruct @enforce_keys + defstruct @enforce_keys ++ [grants: MapSet.new()] @type t :: %__MODULE__{ id: String.t(), uuid: Ecto.UUID.t(), - public_key: JOSE.JWK.t() + public_key: JOSE.JWK.t(), + grants: MapSet.t(String.t()) } @url_namespace <<0x6BA7B8119DAD11D180B400C04FD430C8::128>> diff --git a/lib/lightning/service_account/access_token.ex b/lib/lightning/service_account/access_token.ex index 459e73c5fcc..d96f620dec7 100644 --- a/lib/lightning/service_account/access_token.ex +++ b/lib/lightning/service_account/access_token.ex @@ -9,7 +9,13 @@ defmodule Lightning.ServiceAccount.AccessToken do alias Lightning.ServiceAccount - @scopes ["users:read", "users:write"] + @scopes [ + "users:read", + "users:write", + "projects:write", + "credentials:write", + "collections:write" + ] @lifetime 300 @typ "at+jwt" @@ -47,6 +53,17 @@ defmodule Lightning.ServiceAccount.AccessToken do token end + @doc """ + Whether `token` declares itself an access token by its `typ`, without + checking anything else about it. + """ + @spec access_token?(String.t()) :: boolean() + def access_token?(token) do + match?(%{"typ" => @typ}, Jason.decode!(JOSE.JWS.peek_protected(token))) + rescue + _ -> false + end + @doc """ Returns the claims of `token` when its `typ`, signature, `iss`, `aud` and `exp` all check out. diff --git a/lib/lightning/setup_utils.ex b/lib/lightning/setup_utils.ex index fab20c43446..d087a74d166 100644 --- a/lib/lightning/setup_utils.ex +++ b/lib/lightning/setup_utils.ex @@ -197,7 +197,8 @@ defmodule Lightning.SetupUtils do Application.get_env(:lightning, :default_retention_period), project_users: project_users }, - false + owner!(project_users), + notify: false ) if with_workflow do @@ -341,7 +342,8 @@ defmodule Lightning.SetupUtils do id: "4adf2644-ed4e-4f97-a24c-ab35b3cb1efa", project_users: project_users }, - false + owner!(project_users), + notify: false ) user = get_most_privileged_user!(openhie_project) @@ -592,7 +594,8 @@ defmodule Lightning.SetupUtils do name: "dhis2-project", project_users: project_users }, - false + owner!(project_users), + notify: false ) user = get_most_privileged_user!(project) @@ -944,6 +947,11 @@ defmodule Lightning.SetupUtils do dataclip end + defp owner!(project_users) do + %{user_id: user_id} = Enum.find(project_users, &(&1.role == :owner)) + Repo.get!(User, user_id) + end + defp get_most_privileged_user!(project) do role_ordering_query = from( diff --git a/lib/lightning/utils/validators.ex b/lib/lightning/utils/validators.ex index b63b708a189..84bc82024b1 100644 --- a/lib/lightning/utils/validators.ex +++ b/lib/lightning/utils/validators.ex @@ -1,6 +1,11 @@ defmodule Lightning.Validators do @moduledoc """ Extra validators for Ecto.Changeset. + + `code:` on a changeset error is the rule + `Lightning.Credo.Check.ChangesetErrorCode` enforces. An error about one + element of a list field also carries `key:`, the path to that element, such + as `["items", 2, "key"]`. """ import Ecto.Changeset @@ -82,10 +87,12 @@ defmodule Lightning.Validators do changeset |> update_change(field, &normalize_name/1) |> validate_change(field, fn ^field, value -> - if valid_name?(value), do: [], else: [{field, message}] + if valid_name?(value), do: [], else: [{field, {message, code: :format}}] end) |> validate_change(field, fn ^field, value -> - if invisible_only?(value), do: [{field, "can't be blank"}], else: [] + if invisible_only?(value), + do: [{field, {"can't be blank", code: :required}}], + else: [] end) end @@ -114,7 +121,9 @@ defmodule Lightning.Validators do Ecto.Changeset.t() def validate_no_null_bytes(changeset, field, message) do validate_change(changeset, field, fn ^field, value -> - if storable_in_jsonb?(value), do: [], else: [{field, message}] + if storable_in_jsonb?(value), + do: [], + else: [{field, {message, code: :format}}] end) end @@ -128,7 +137,7 @@ defmodule Lightning.Validators do Ecto.Changeset.t() def validate_no_null_bytes_deep(changeset, field, message) do validate_change(changeset, field, fn ^field, value -> - if jsonb_safe?(value), do: [], else: [{field, message}] + if jsonb_safe?(value), do: [], else: [{field, {message, code: :format}}] end) end @@ -210,7 +219,7 @@ defmodule Lightning.Validators do |> Enum.find(fn [_, {kind, _}] -> kind == :changes end) |> List.first() - add_error(changeset, error_field, message) + add_error(changeset, error_field, message, code: :exclusive) else changeset end @@ -228,7 +237,7 @@ defmodule Lightning.Validators do |> Enum.reject(&is_nil/1) |> case do [] -> - add_error(changeset, fields |> List.first(), message) + add_error(changeset, fields |> List.first(), message, code: :required) _any -> changeset @@ -251,7 +260,7 @@ defmodule Lightning.Validators do |> get_field(assoc) |> case do nil -> - add_error(changeset, assoc, message) + add_error(changeset, assoc, message, code: :required) _any -> changeset @@ -302,7 +311,9 @@ defmodule Lightning.Validators do def validate_uuid(changeset, field) when is_atom(field) do validate_change(changeset, field, fn _, value -> - if valid_uuid?(value), do: [], else: [{field, "is not a valid UUID"}] + if valid_uuid?(value), + do: [], + else: [{field, {"is not a valid UUID", code: :format}}] end) end @@ -323,22 +334,22 @@ defmodule Lightning.Validators do {:ok, uri} <- URI.new(url) do cond do uri.scheme not in ["http", "https"] -> - [{field, "must be either a http or https URL"}] + [{field, {"must be either a http or https URL", code: :format}}] is_nil(uri.host) or byte_size(uri.host) == 0 -> - [{field, "host can't be blank"}] + [{field, {"host can't be blank", code: :format}}] byte_size(uri.host) > 255 -> - [{field, "host must be less than 255 characters"}] + [{field, {"host must be less than 255 characters", code: :format}}] not valid_host?(uri.host) -> - [{field, "host has invalid characters"}] + [{field, {"host has invalid characters", code: :format}}] true -> [] end else - _ -> [{field, "must be a valid URL"}] + _ -> [{field, {"must be a valid URL", code: :format}}] end end) end diff --git a/lib/lightning/version_control/project_repo_connection.ex b/lib/lightning/version_control/project_repo_connection.ex index af552ec13ee..5256f838543 100644 --- a/lib/lightning/version_control/project_repo_connection.ex +++ b/lib/lightning/version_control/project_repo_connection.ex @@ -98,7 +98,7 @@ defmodule Lightning.VersionControl.ProjectRepoConnection do ) |> validate_change(field, fn ^field, path -> if Path.type(path) == :absolute or ".." in Path.split(path) do - [{field, "must stay inside the repository"}] + [{field, {"must stay inside the repository", code: :not_allowed}}] else [] end @@ -115,7 +115,7 @@ defmodule Lightning.VersionControl.ProjectRepoConnection do if accept do [] else - [accept: "please tick the box"] + [accept: {"please tick the box", code: :must_accept}] end end) |> validate_sync_direction() @@ -176,7 +176,7 @@ defmodule Lightning.VersionControl.ProjectRepoConnection do if is_binary(root_project_id) and is_binary(repo) and is_binary(branch) do if tree_branch_conflict?(root_project_id, repo, branch, self_id) do add_error(changeset, :branch, @tree_branch_error, - reason: :tree_branch_conflict + code: :tree_branch_conflict ) else changeset @@ -241,7 +241,11 @@ defmodule Lightning.VersionControl.ProjectRepoConnection do # v2 syncs always use openfn.yaml, so only legacy ones need a path if value == :deploy and is_nil(path) and not get_field(changeset, :sync_version) do - [config_path: "you must specify a path to an existing config file"] + [ + config_path: + {"you must specify a path to an existing config file", + code: :required} + ] else [] end diff --git a/lib/lightning/webhook_auth_methods.ex b/lib/lightning/webhook_auth_methods.ex index 76ce220947a..3ce5dc10790 100644 --- a/lib/lightning/webhook_auth_methods.ex +++ b/lib/lightning/webhook_auth_methods.ex @@ -679,7 +679,8 @@ defmodule Lightning.WebhookAuthMethods do WebhookAuthMethod.changeset(webhook_auth_method, %{}) |> Ecto.Changeset.add_error( :scheduled_deletion, - "already scheduled for deletion" + "already scheduled for deletion", + code: :already_scheduled ) {:error, changeset} diff --git a/lib/lightning/work_orders.ex b/lib/lightning/work_orders.ex index 762a2488e63..a464a4ef5a2 100644 --- a/lib/lightning/work_orders.ex +++ b/lib/lightning/work_orders.ex @@ -41,6 +41,7 @@ defmodule Lightning.WorkOrders do alias Lightning.Graph alias Lightning.Invocation.Dataclip alias Lightning.Invocation.Step + alias Lightning.Projects.Project alias Lightning.Repo alias Lightning.Run alias Lightning.Runs @@ -348,7 +349,8 @@ defmodule Lightning.WorkOrders do |> Ecto.Changeset.change() |> Ecto.Changeset.add_error( :input_dataclip_id, - "cannot retry run using a wiped dataclip" + "cannot retry run using a wiped dataclip", + code: :dataclip_wiped )} end) |> Multi.run(:steps, &get_workflow_steps_from/2) @@ -859,12 +861,17 @@ defmodule Lightning.WorkOrders do end @doc """ - Returns a query for work orders belonging to a specific project + A page of the work orders of `project`, or of every project `user` may + access, narrowed by `filters` (see + `Lightning.Invocation.Query.filter_work_orders/2`). """ - @spec work_orders_for_project_query(Lightning.Projects.Project.t()) :: - Ecto.Queryable.t() - def work_orders_for_project_query(%Lightning.Projects.Project{} = project) do - Lightning.Invocation.Query.work_orders_for(project) + @spec paginate_work_orders_for(Project.t() | User.t(), map(), keyword()) :: + Scrivener.Page.t() + def paginate_work_orders_for(project_or_user, filters, page) do + project_or_user + |> Lightning.Invocation.Query.work_orders_for() + |> Lightning.Invocation.Query.filter_work_orders(filters) + |> Repo.paginate(page) end defp new_retry_run( diff --git a/lib/lightning/workflows.ex b/lib/lightning/workflows.ex index 7cd21348843..5f61b51b66e 100644 --- a/lib/lightning/workflows.ex +++ b/lib/lightning/workflows.ex @@ -10,6 +10,7 @@ defmodule Lightning.Workflows do import Ecto.Query alias Ecto.Multi + alias Lightning.Accounts.User alias Lightning.Config alias Lightning.Credentials.Scoping @@ -194,7 +195,8 @@ defmodule Lightning.Workflows do | {:error, Ecto.Changeset.t(Workflow.t()) | :workflow_deleted - | :snapshot_failed} + | :snapshot_failed + | :concurrent_edit} def save_workflow(changeset_or_attrs, actor, opts \\ []) def save_workflow( @@ -212,11 +214,6 @@ defmodule Lightning.Workflows do changeset |> build_save_multi(actor, opts) |> Repo.transaction() - - # NOTE: Ecto.StaleEntryError is deliberately NOT caught — optimistic - # lock conflicts have their own reload UX and workflows_test.exs asserts - # it raises. Anything off this allow-list re-raises automatically with - # the original stacktrace. rescue e in Ecto.ChangeError -> # Malformed values that pass cast but fail at dump (e.g. a 16-byte @@ -333,7 +330,9 @@ defmodule Lightning.Workflows do |> Multi.run(:orphan_deleted_jobs, fn repo, _changes -> orphan_jobs_being_deleted(repo, changeset) end) - |> Multi.insert_or_update(:workflow, changeset) + |> Multi.insert_or_update(:workflow, changeset, + stale_error_field: :lock_version + ) |> Multi.run(:credential_scope_check, fn _repo, %{workflow: workflow} -> credential_scope_check(workflow, changeset) end) @@ -475,6 +474,14 @@ defmodule Lightning.Workflows do defp handle_save_result({:error, :rescued, changeset}, _changeset, _skip), do: {:error, changeset} + defp handle_save_result( + {:error, :workflow, + %Ecto.Changeset{errors: [lock_version: {_, stale: true}]}, _changes}, + _changeset, + _skip + ), + do: {:error, :concurrent_edit} + defp handle_save_result( {:error, :workflow, changeset, _changes}, _changeset, @@ -546,7 +553,7 @@ defmodule Lightning.Workflows do Logger.log(level, log_message) changeset - |> Ecto.Changeset.add_error(:base, base_message) + |> Ecto.Changeset.add_error(:base, base_message, code: :invalid) |> Map.put(:action, derive_action(changeset)) end @@ -861,12 +868,17 @@ defmodule Lightning.Workflows do end @doc """ - Returns a query for workflows accessible to a user + Lists the workflows in the user's projects, newest first, preloading the + associations named in `include:`. """ - @spec workflows_for_user_query(Lightning.Accounts.User.t()) :: - Ecto.Queryable.t() - def workflows_for_user_query(%Lightning.Accounts.User{} = user) do - Query.workflows_for(user) + @spec list_workflows_for_user(User.t(), keyword()) :: [Workflow.t()] + def list_workflows_for_user(%User{} = user, opts \\ []) do + include = Keyword.get(opts, :include, []) + + user + |> Query.workflows_for() + |> preload(^include) + |> Repo.all() end @spec to_project_space([Workflow.t()]) :: %{} @@ -1433,7 +1445,8 @@ defmodule Lightning.Workflows do and saves in one transaction (capturing a snapshot and recording a version). """ @spec go_live(Workflow.t(), struct()) :: - {:ok, Workflow.t()} | {:error, Ecto.Changeset.t(Workflow.t())} + {:ok, Workflow.t()} + | {:error, Ecto.Changeset.t(Workflow.t()) | :concurrent_edit} def go_live(%Workflow{} = workflow, actor) do workflow |> Repo.preload(:triggers) @@ -1456,7 +1469,8 @@ defmodule Lightning.Workflows do replacement is more than the incremental reconciler can express. """ @spec restore_version(Workflow.t(), WorkflowRelease.t(), struct()) :: - {:ok, Workflow.t()} | {:error, Ecto.Changeset.t(Workflow.t())} + {:ok, Workflow.t()} + | {:error, Ecto.Changeset.t(Workflow.t()) | :concurrent_edit} def restore_version( %Workflow{} = workflow, %WorkflowRelease{snapshot: %Snapshot{} = snapshot} = release, @@ -1476,7 +1490,8 @@ defmodule Lightning.Workflows do disables its triggers, saved in one transaction. """ @spec switch_to_draft(Workflow.t(), struct()) :: - {:ok, Workflow.t()} | {:error, Ecto.Changeset.t(Workflow.t())} + {:ok, Workflow.t()} + | {:error, Ecto.Changeset.t(Workflow.t()) | :concurrent_edit} def switch_to_draft(%Workflow{} = workflow, actor) do workflow |> Repo.preload(:triggers) diff --git a/lib/lightning/workflows/edge.ex b/lib/lightning/workflows/edge.ex index 0e08e2a6b64..2c5045f29f7 100644 --- a/lib/lightning/workflows/edge.ex +++ b/lib/lightning/workflows/edge.ex @@ -127,7 +127,8 @@ defmodule Lightning.Workflows.Edge do add_error( changeset, :source_job_id, - "source_job_id or source_trigger_id must be present" + "source_job_id or source_trigger_id must be present", + code: :required ) end end @@ -197,7 +198,8 @@ defmodule Lightning.Workflows.Edge do add_error( changeset, :target_job_id, - "target_job_id must be different from source_job_id" + "target_job_id must be different from source_job_id", + code: :cycle ) _else -> diff --git a/lib/lightning/workflows/graph_rules.ex b/lib/lightning/workflows/graph_rules.ex new file mode 100644 index 00000000000..9c0fb808575 --- /dev/null +++ b/lib/lightning/workflows/graph_rules.ex @@ -0,0 +1,238 @@ +defmodule Lightning.Workflows.GraphRules do + @moduledoc """ + Rules for a workflow's jobs, triggers and edges, on top of + `Lightning.Workflows.Workflow.changeset/2`. + + - Every job and trigger the request sends carries its id, since edges name + them by it. + - A request that sends triggers removes none of the stored ones, and leaves + at most one enabled. + - Exactly one enabled edge leaves a trigger, and no trigger has two targets. + - The edges form no cycle. + + Each broken rule is a coded error on the changeset, on the item at fault for a + missing id or a cycle and on the list otherwise. + """ + import Ecto.Changeset, only: [add_error: 4, get_field: 2, validate_required: 2] + + alias Ecto.Changeset + alias Lightning.Workflows.Workflow + + @spec validate(Changeset.t(Workflow.t())) :: Changeset.t(Workflow.t()) + def validate(%Changeset{data: %Workflow{}} = changeset) do + changeset + |> update_sent(:jobs, &require_id/2) + |> update_sent(:triggers, &require_id/2) + |> validate_triggers() + |> validate_trigger_edges() + |> validate_no_cycle() + end + + defp validate_triggers(%Changeset{changes: %{triggers: triggers}} = changeset) do + enabled = changeset |> field_list(:triggers) |> Enum.count(& &1.enabled) + + changeset = + if Enum.any?(triggers, &(&1.action == :replace)), + do: + add_error( + changeset, + :triggers, + "a trigger cannot be removed, only edited or added", + code: :trigger_removed + ), + else: changeset + + if enabled > 1, + do: + add_error(changeset, :triggers, "only one trigger may be enabled", + code: :one_enabled + ), + else: changeset + end + + defp validate_triggers(changeset), do: changeset + + defp validate_trigger_edges(changeset) do + trigger_ids = + changeset + |> field_list(:triggers) + |> Enum.map(& &1.id) + |> Enum.reject(&is_nil/1) + |> MapSet.new() + + from_triggers = + changeset + |> edges() + |> Enum.filter(&MapSet.member?(trigger_ids, &1.from)) + + many_targets? = + from_triggers + |> Enum.frequencies_by(& &1.from) + |> Enum.any?(fn {_trigger_id, count} -> count > 1 end) + + # An edge that names a trigger as its source job counts toward the trigger's + # targets but not toward its enabled edges. + enabled_flags = + for edge <- from_triggers, edge.source_trigger_id, do: edge.enabled + + changeset = refuse_trigger_edges(changeset, enabled_flags) + + if many_targets?, + do: + add_error(changeset, :edges, "a trigger must have a single target", + code: :single_target + ), + else: changeset + end + + # A lone edge from a trigger counts whatever its enabled flag says: saving + # enables it (Edge.changeset/2). + defp refuse_trigger_edges(changeset, [_one]), do: changeset + + defp refuse_trigger_edges(changeset, enabled_flags) do + case Enum.count(enabled_flags, & &1) do + 0 -> + add_error(changeset, :edges, "needs an enabled edge from a trigger", + code: :missing_trigger_edge + ) + + 1 -> + changeset + + _more -> + add_error( + changeset, + :edges, + "only one edge from a trigger may be enabled", + code: :one_enabled_edge + ) + end + end + + defp validate_no_cycle(changeset) do + {sent, stored} = + changeset + |> edges() + |> closing_edges() + |> Enum.split_with(&is_integer(&1.index)) + + sent_indexes = MapSet.new(sent, & &1.index) + + changeset = + update_sent(changeset, :edges, fn edge, index -> + if MapSet.member?(sent_indexes, index), + do: + add_error(edge, :target_job_id, "is part of a cycle", code: :cycle), + else: edge + end) + + # Only true when the request leaves the edges alone and the stored ones + # already loop, so no edge of the request is at fault. + if stored != [], + do: add_error(changeset, :edges, "is part of a cycle", code: :cycle), + else: changeset + end + + # Depth-first from every node, in edge order: an edge that reaches a node + # still on the walk's path closes a cycle. An edge from a job to itself is + # Edge.changeset/2's to report. + defp closing_edges(edges) do + out = + edges + |> Enum.reject(&(&1.from == &1.to or is_nil(&1.from) or is_nil(&1.to))) + |> Enum.group_by(& &1.from) + + {_states, closing} = + edges + |> Enum.map(& &1.from) + |> Enum.uniq() + |> Enum.reduce({%{}, []}, fn node, acc -> visit(node, out, acc) end) + + Enum.reverse(closing) + end + + defp visit(node, out, {states, closing}) do + if Map.has_key?(states, node) do + {states, closing} + else + {states, closing} = + out + |> Map.get(node, []) + |> Enum.reduce( + {Map.put(states, node, :on_path), closing}, + &follow(&1, out, &2) + ) + + {Map.put(states, node, :done), closing} + end + end + + defp follow(edge, out, {states, closing}) do + case Map.get(states, edge.to) do + :on_path -> {states, [edge | closing]} + :done -> {states, closing} + nil -> visit(edge.to, out, {states, closing}) + end + end + + # The workflow's edges once this change applies. `index` is an edge's position + # among the changeset's edges, or nil for a stored edge the change didn't + # touch. + defp edges(%Changeset{changes: %{edges: edges}}) do + edges + |> Enum.with_index() + |> Enum.reject(fn {edge, _index} -> edge.action == :replace end) + |> Enum.map(fn {edge, index} -> + edge(index, &get_field(edge, &1)) + end) + end + + defp edges(%Changeset{data: %Workflow{edges: edges}}) when is_list(edges), + do: Enum.map(edges, &edge(nil, fn field -> Map.fetch!(&1, field) end)) + + defp edges(_changeset), do: [] + + defp edge(index, field) do + %{ + index: index, + from: field.(:source_trigger_id) || field.(:source_job_id), + to: field.(:target_job_id), + source_trigger_id: field.(:source_trigger_id), + enabled: field.(:enabled) + } + end + + defp require_id(child, _index), do: validate_required(child, :id) + + defp field_list(changeset, field) do + case get_field(changeset, field) do + list when is_list(list) -> list + %Ecto.Association.NotLoaded{} -> [] + end + end + + # Applies `fun` to each child the request sent, with its position in the + # changeset's list, and marks the parent invalid when a child becomes so. + defp update_sent(%Changeset{changes: changes} = changeset, field, fun) do + case Map.fetch(changes, field) do + {:ok, children} -> + children = + children + |> Enum.with_index() + |> Enum.map(fn + {%Changeset{params: nil} = child, _index} -> child + {%Changeset{action: :replace} = child, _index} -> child + {child, index} -> fun.(child, index) + end) + + %{ + changeset + | changes: Map.put(changes, field, children), + valid?: changeset.valid? and Enum.all?(children, & &1.valid?) + } + + :error -> + changeset + end + end +end diff --git a/lib/lightning/workflows/job.ex b/lib/lightning/workflows/job.ex index 2749190a959..4a670c1732e 100644 --- a/lib/lightning/workflows/job.ex +++ b/lib/lightning/workflows/job.ex @@ -142,7 +142,7 @@ defmodule Lightning.Workflows.Job do validate_change(changeset, :adaptor, fn :adaptor, adaptor -> if Adaptors.valid_format?(adaptor), do: [], - else: [adaptor: "adaptor has invalid format"] + else: [adaptor: {"adaptor has invalid format", code: :format}] end) if changeset.valid? do @@ -156,18 +156,27 @@ defmodule Lightning.Workflows.Job do # package cannot be persisted on a job. `fetch_adaptor` returns # `:not_found` once the catalogue has loaded and genuinely lacks the # package, but any other error means the catalogue itself isn't ready - # yet, so that case gets its own, retry-able message. + # yet, so that case gets its own, retry-able message. It doesn't wait for a + # first load, so a save while the catalogue loads is refused at once and the + # client retries. defp validate_known_adaptor(changeset) do validate_change(changeset, :adaptor, fn :adaptor, adaptor -> with {:ok, {name, _version}} <- Adaptors.parse_spec(adaptor), - {:ok, _package} <- Adaptors.fetch_adaptor(name) do + {:ok, _package} <- + Adaptors.fetch_adaptor(Adaptors.Config.default_instance(), name, + timeout: 0 + ) do [] else {:error, reason} when reason in [:not_found, :invalid_format] -> - [adaptor: "is not a recognised adaptor"] + [adaptor: {"is not a recognised adaptor", code: :not_found}] {:error, _reason} -> - [adaptor: "adaptor catalogue is not ready yet, try again shortly"] + [ + adaptor: + {"adaptor catalogue is not ready yet, try again shortly", + code: :not_ready} + ] end end) end @@ -193,7 +202,8 @@ defmodule Lightning.Workflows.Job do add_error( changeset, :keychain_credential_id, - Scoping.violation_message(:keychain_credential_id) + Scoping.violation_message(:keychain_credential_id), + code: :credential_not_allowed ) _ -> diff --git a/lib/lightning/workflows/trigger.ex b/lib/lightning/workflows/trigger.ex index 9cecd4033ef..fc881bc0185 100644 --- a/lib/lightning/workflows/trigger.ex +++ b/lib/lightning/workflows/trigger.ex @@ -276,7 +276,8 @@ defmodule Lightning.Workflows.Trigger do add_error( changeset, :custom_path, - "needs a project, which comes from the trigger's workflow" + "needs a project, which comes from the trigger's workflow", + code: :required ) else changeset @@ -309,7 +310,10 @@ defmodule Lightning.Workflows.Trigger do <<_::288>> -> case Ecto.UUID.cast(path) do {:ok, _} -> - [custom_path: {"cannot be a UUID", custom_path_shape: true}] + [ + custom_path: + {"cannot be a UUID", custom_path_shape: true, code: :format} + ] :error -> [] @@ -327,7 +331,7 @@ defmodule Lightning.Workflows.Trigger do Crontab.CronExpression.Parser.parse(cron_expression) |> case do {:error, error_message} -> - [{:cron_expression, error_message}] + [{:cron_expression, {error_message, code: :format}}] {:ok, _exp} -> [] @@ -372,7 +376,8 @@ defmodule Lightning.Workflows.Trigger do add_error( changeset, :custom_path, - "must only contain lowercase letters, numbers, hyphens and underscores" + "must only contain lowercase letters, numbers, hyphens and underscores", + code: :format ) else _usable_or_not_a_webhook -> changeset diff --git a/lib/lightning/workorders/graph.ex b/lib/lightning/workorders/graph.ex index 4ac13c09898..dca2dcbfda0 100644 --- a/lib/lightning/workorders/graph.ex +++ b/lib/lightning/workorders/graph.ex @@ -1,6 +1,6 @@ defmodule Lightning.Graph do @moduledoc """ - A graph model for workflows transversal. + A workflow's edges as a graph, for finding the steps downstream of a job. """ defstruct nodes: MapSet.new(), edges: MapSet.new() @type t :: %__MODULE__{nodes: MapSet.t(), edges: MapSet.t()} @@ -56,45 +56,4 @@ defmodule Lightning.Graph do def nodes(graph, opts \\ [as: []]) do graph.nodes |> Enum.into(opts[:as]) end - - def traverse(%{edges: edges}, trigger_id) do - edges - |> Enum.reduce(Map.new(), fn {from, to}, dag_edges -> - Map.update(dag_edges, from, [to], &[to | &1]) - end) - |> then(fn dag_edges -> - case Map.get(dag_edges, trigger_id) do - [] -> - :ok - - nil -> - {:error, :no_target_for_trigger} - - [initial_job_id] -> - traverse(initial_job_id, dag_edges, MapSet.new()) - - _multiple -> - {:error, :multiple_targets_for_trigger} - end - end) - end - - defp traverse(node_id, dag_edges, visited_nodes) do - cond do - not Map.has_key?(dag_edges, node_id) -> - :ok - - MapSet.member?(visited_nodes, node_id) -> - {:error, :graph_has_a_cycle, node_id} - - :else -> - visited_nodes = MapSet.put(visited_nodes, node_id) - - dag_edges - |> Map.get(node_id) - |> Enum.find_value(:ok, fn dest_node_id -> - with :ok <- traverse(dest_node_id, dag_edges, visited_nodes), do: nil - end) - end - end end diff --git a/lib/lightning/workorders/manual.ex b/lib/lightning/workorders/manual.ex index 41d79cb5148..4eeefc9e10f 100644 --- a/lib/lightning/workorders/manual.ex +++ b/lib/lightning/workorders/manual.ex @@ -36,15 +36,6 @@ defmodule Lightning.WorkOrders.Manual do |> validate_body_or_dataclip() |> validate_dataclip_in_project() |> validate_json(:body) - |> validate_change(:workflow, fn _field, workflow -> - case workflow do - %{__meta__: %{state: :built}} -> - [:workflow, "Workflow must be saved first."] - - _other -> - [] - end - end) end defp remove_body_if_dataclip_present(changeset) do @@ -62,9 +53,14 @@ defmodule Lightning.WorkOrders.Manual do body -> case Jason.decode(body) do - {:ok, body} when is_map(body) -> changeset - {:ok, _} -> add_error(changeset, field, "Must be an object") - {:error, _} -> add_error(changeset, field, "Invalid JSON") + {:ok, body} when is_map(body) -> + changeset + + {:ok, _} -> + add_error(changeset, field, "Must be an object", code: :type) + + {:error, _} -> + add_error(changeset, field, "Invalid JSON", code: :type) end end end @@ -79,7 +75,9 @@ defmodule Lightning.WorkOrders.Manual do Lightning.Invocation.dataclip_in_project?(dataclip_id, project.id) do changeset else - add_error(changeset, :dataclip_id, "does not belong to this project") + add_error(changeset, :dataclip_id, "does not belong to this project", + code: :not_found + ) end end diff --git a/lib/lightning_web/api/codes.ex b/lib/lightning_web/api/codes.ex new file mode 100644 index 00000000000..238867c9076 --- /dev/null +++ b/lib/lightning_web/api/codes.ex @@ -0,0 +1,176 @@ +defmodule LightningWeb.API.Codes do + @moduledoc """ + The API's codes: each response code with its status and fixed detail, and + every code an error entry may carry. Both lists are the API's contract, so + renaming or removing a code is a breaking change (error-format spec §6.4). + """ + + @response %{ + bad_request: {400, "The request could not be read."}, + invalid_request: {400, "The credentials could not be read."}, + malformed: {400, "The body is not a JSON object."}, + invalid_parameter: {400, "A query parameter is not valid. See errors."}, + invalid_header: {400, "A request header is not valid. See errors."}, + unauthenticated: {401, "Authentication is required."}, + invalid_token: {401, "The token is not valid."}, + exceeds_limit: {402, "This would exceed a limit of your plan."}, + forbidden: {403, "You may not do this."}, + insufficient_scope: {403, "The token lacks the scope this needs."}, + trigger_disabled: {403, "This trigger is disabled."}, + not_found: {404, "Not found, or you don't have access to it."}, + not_acceptable: + {406, "This resource cannot be returned in the requested format."}, + request_timeout: {408, "The request body did not arrive in time."}, + scheduled_for_deletion: + {409, "This is scheduled for deletion. Cancel the deletion first."}, + id_taken: {409, "The id is already in use."}, + taken: {409, "A value that must be unique is already in use. See errors."}, + being_edited: {409, "This workflow is open in the editor."}, + concurrent_edit: + {409, + "Another request changed or deleted this record. Fetch it again and retry."}, + in_use: {409, "This has been used by runs and cannot be deleted."}, + ambiguous_name: + {409, "More than one collection has this name. Add project_id."}, + export_conflict: + {409, "Two items in this project export under the same key."}, + content_too_large: {413, "The body is too large."}, + uri_too_long: {414, "The query string is too long."}, + unsupported_media_type: {415, "The body must be JSON."}, + invalid: {422, "The request is not valid. See errors."}, + too_many_runs: {429, "Too many runs. Try again later."}, + too_many_requests: {429, "Too many requests. Try again later."}, + internal: {500, "Something went wrong."}, + not_implemented: {501, "This is not available on this instance."}, + bad_gateway: {502, "The channel could not reach its destination."}, + service_unavailable: {503, "Try again shortly."}, + not_ready: {503, "This is not ready yet. Try again shortly."}, + timeout: {504, "The workflow did not finish in time."} + } + + @typedoc "A response code, one of the keys of the response list." + @type code :: + unquote(Enum.reduce(Map.keys(@response), &{:|, [], [&1, &2]})) + + # `/i/*` keeps 429 for every limiter refusal (spec §12). + @other_statuses %{exceeds_limit: [429]} + + @by_status %{ + 400 => :bad_request, + 403 => :forbidden, + 404 => :not_found, + 406 => :not_acceptable, + 408 => :request_timeout, + 413 => :content_too_large, + 414 => :uri_too_long, + 415 => :unsupported_media_type + } + + @ecto_entry_codes [ + :duplicate, + :format, + :invalid, + :must_accept, + :mismatch, + :not_allowed, + :not_found, + :out_of_range, + :required, + :reserved, + :taken, + :too_long, + :too_short, + :type, + :wrong_length + ] + + # `not_ready` is set with `code:` too, but it is not listed here. It never + # reaches the client as an entry, only as the 503 response code (spec §6.2). + @lightning_entry_codes [ + :already_scheduled, + :credential_not_allowed, + :cycle, + :dataclip_wiped, + :deleting, + :discovery_failed, + :exceeds_limit, + :exclusive, + :invalid, + :mismatch, + :missing_trigger_edge, + :no_access, + :not_found, + :not_supported, + :one_body, + :one_enabled, + :one_enabled_edge, + :one_owner, + :scope_not_granted, + :single_target, + :storage_failed, + :too_many_secrets, + :tree_branch_conflict, + :trigger_removed, + :unchanged, + :unknown_key + ] + + @spec status(code()) :: pos_integer() + def status(code), do: @response |> Map.fetch!(code) |> elem(0) + + @doc "Every status `code` may answer with, its own first." + @spec statuses(code()) :: [pos_integer()] + def statuses(code), do: [status(code) | Map.get(@other_statuses, code, [])] + + @spec detail(code()) :: String.t() + def detail(code), do: @response |> Map.fetch!(code) |> elem(1) + + @doc """ + The code a raise is answered with. Every exception a client can cause has a + 4xx status (`Plug.Exception.status/1`) and is answered by it. Anything else + is `:internal`: a 500 means Lightning raised where it should not have. + """ + @spec for_exception(:error | :exit | :throw, term()) :: code() + def for_exception(:error, %Plug.Parsers.ParseError{}), do: :malformed + + # A path id that isn't a UUID is not found, like any other id (spec §4). + def for_exception(:error, %Ecto.Query.CastError{}), do: :not_found + + # phoenix_ecto gives a stale write 409, but it is a race inside Lightning, + # not the client's doing: the producer has to return an error tuple instead. + def for_exception(:error, %Ecto.StaleEntryError{}), do: :internal + + def for_exception(:error, exception), + do: Map.get(@by_status, Plug.Exception.status(exception), :internal) + + def for_exception(_kind, _reason), do: :internal + + @doc """ + The code a usage or rate limiter's refusal is answered with (spec §4, §12). + The limiter's text is copy for the app's pages, so it never reaches the + answer. + """ + @spec for_limiter(atom()) :: code() + def for_limiter(:too_many_runs), do: :too_many_runs + + def for_limiter(reason) when reason in [:too_many_requests, :unknown], + do: :too_many_requests + + # The usage limiter is an extension that picks its own reasons, so any other + # is the plan's limit. + def for_limiter(_reason), do: :exceeds_limit + + @spec response_codes() :: [code()] + def response_codes, do: Map.keys(@response) + + @doc """ + The codes the walker maps from Ecto's own options, and those Lightning sets + with `code:`. + """ + @spec entry_codes() :: [atom()] + def entry_codes, do: Enum.uniq(@ecto_entry_codes ++ @lightning_entry_codes) + + @doc false + @spec lightning_entry_codes() :: [atom()] + def lightning_entry_codes, do: @lightning_entry_codes +end diff --git a/lib/lightning_web/api/errors.ex b/lib/lightning_web/api/errors.ex new file mode 100644 index 00000000000..7001d5889e7 --- /dev/null +++ b/lib/lightning_web/api/errors.ex @@ -0,0 +1,371 @@ +defmodule LightningWeb.API.Errors do + @moduledoc """ + Turns a refused changeset into error entries and a problem + (error-format spec §3, §4 and §7). + + Every error in the tree becomes one entry, placed by the request's keys and + positions. An error whose code can't be worked out raises: there is no + fallback code (spec §6.3). + """ + alias Ecto.Changeset + alias LightningWeb.API.Problem + + require Logger + + @type segment :: atom() | String.t() | non_neg_integer() + @type location :: + [segment()] + | {:parameter, String.t()} + | {:header, String.t()} + | {:path, String.t()} + @type entry :: %{ + required(:code) => atom(), + required(:detail) => String.t(), + optional(:pointer | :parameter | :header | :path) => String.t() + } + + @validation_codes %{ + required: :required, + format: :format, + inclusion: :not_allowed, + subset: :not_allowed, + exclusion: :reserved, + number: :out_of_range, + confirmation: :mismatch, + acceptance: :must_accept, + unsafe_unique: :taken + } + + @length_codes %{min: :too_short, max: :too_long, is: :wrong_length} + + @constraint_codes %{ + unique: :taken, + exclusion: :taken, + foreign: :not_found, + assoc: :not_found, + check: :invalid + } + + @doc """ + Every error in `changeset` and its children, as entries. + + `renames` maps an Ecto field to the request's key, at any depth: + `%{project_users: :members}`. An error whose options carry `key:` is placed + there instead of at its field: one request key, or a list of the request's + keys and positions below the error's level. + + `path:` lists the top-level request keys the request sent as path segments + rather than in its body, so their errors name the segment: `path: [:key]`. + + Raises `ArgumentError` for an error with no code, and for an error on a + child the request didn't send, since neither can be placed in the request. + """ + @spec from_changeset(Changeset.t(), %{atom() => atom()}, path: [atom()]) :: + [entry()] + def from_changeset(%Changeset{} = changeset, renames \\ %{}, opts \\ []) do + path_keys = Keyword.get(opts, :path, []) + + changeset + |> walk([], renames, nil) + |> Enum.map(fn {location, error} -> + entry(in_path(location, path_keys), error) + end) + end + + @doc """ + The problem for a refused changeset. The first rule that matches wins + (spec §4, "A changeset's status"): + + 1. any error coded `storage_failed`: 500 `internal`, since the server + failed, not the request (spec §6.2); + 2. any error coded `not_ready`: 503 `not_ready`, with no entries; + 3. every error `exceeds_limit`: 402 `exceeds_limit`; + 4. every error `taken`: 409 `taken`; + 5. otherwise 422 `invalid`, without any `exceeds_limit` entries. + + A unique constraint and an application-level check both answer 409, since + they report the same clash and which one caught it is a race. + """ + @spec problem(Changeset.t(), %{atom() => atom()}, path: [atom()]) :: + Problem.t() + def problem(changeset, renames \\ %{}, opts \\ []) + + def problem(%Changeset{valid?: true}, _renames, _opts) do + raise ArgumentError, "a changeset with no errors is not a problem" + end + + def problem(%Changeset{} = changeset, renames, opts) do + entries = from_changeset(changeset, renames, opts) + codes = Enum.map(entries, & &1.code) + + cond do + :storage_failed in codes -> + Logger.error("An API action could not delete stored files") + Problem.new(:internal) + + :not_ready in codes -> + Problem.new(:not_ready, retry_after: 5) + + Enum.all?(codes, &(&1 == :exceeds_limit)) -> + Problem.new(:exceeds_limit, errors: entries) + + Enum.all?(codes, &(&1 == :taken)) -> + Problem.new(:taken, errors: entries) + + true -> + Problem.new(:invalid, + errors: Enum.reject(entries, &(&1.code == :exceeds_limit)) + ) + end + end + + @doc """ + `:ok` when there are no entries, otherwise a 422 `invalid` problem holding + them all, for a producer that checks the request itself (spec §9.6). + """ + @spec refuse([entry()]) :: :ok | {:error, Problem.t()} + def refuse([]), do: :ok + def refuse(entries), do: {:error, Problem.new(:invalid, errors: entries)} + + @doc """ + The 400 `invalid_parameter` problem for a refused changeset over a route's + query parameters, with one entry per error, each naming its parameter + (spec §9.4). + """ + @spec parameters(Changeset.t()) :: Problem.t() + def parameters(%Changeset{valid?: true}) do + raise ArgumentError, "a changeset with no errors is not a problem" + end + + def parameters(%Changeset{errors: errors}) do + entries = + errors + |> Enum.reverse() + |> Enum.map(fn {field, error} -> + entry({:parameter, Atom.to_string(field)}, error) + end) + + Problem.new(:invalid_parameter, errors: entries) + end + + @doc """ + `{:ok, value}` passed through, or `{:error, changeset}` over a route's query + parameters turned into its 400 `invalid_parameter` problem (spec §9.4). + """ + @spec refuse_parameters({:ok, value} | {:error, Changeset.t()}) :: + {:ok, value} | {:error, Problem.t()} + when value: term() + def refuse_parameters({:ok, _value} = ok), do: ok + + def refuse_parameters({:error, %Changeset{} = changeset}), + do: {:error, parameters(changeset)} + + @doc """ + One entry, for a producer that validates the request itself (spec §7 items + 3 and 4). + + `error` is shaped as a changeset error, `{message, opts}`, and is coded and + interpolated the same way. `location` is a list of the request's keys and + positions, or `{:parameter, name}`, or `{:header, name}`, or `{:path, name}` + for a path segment, named by its route parameter. A `path` names a segment, + never carries its text. + """ + @spec entry(location(), {String.t(), keyword()}) :: entry() + def entry(location, {message, opts}) do + location + |> place() + |> Map.merge(%{ + code: code(location, message, opts), + detail: interpolate(message, opts) + }) + end + + @doc """ + The error for a reference in the body that matches nothing the caller can + see. Something missing and something hidden get the same error (spec §4). + """ + @spec not_visible() :: {String.t(), keyword()} + def not_visible, do: {"names nothing you can see", code: :not_found} + + @doc """ + An RFC 6901 JSON Pointer in URI-fragment form: each key has `~` written + `~0` and `/` written `~1`, then the whole is percent-encoded wherever a + character isn't allowed in a fragment (spec §3). + """ + @spec pointer([segment()]) :: String.t() + def pointer(segments) do + path = Enum.map_join(segments, &("/" <> escape(to_string(&1)))) + "#" <> URI.encode(path, &fragment_char?/1) + end + + defp escape(key), + do: key |> String.replace("~", "~0") |> String.replace("/", "~1") + + defp fragment_char?(char), + do: URI.char_unreserved?(char) or char in ~c"!$&'()*+,;=:@/?" + + defp place({:parameter, name}), do: %{parameter: name} + defp place({:header, name}), do: %{header: String.downcase(name)} + defp place({:path, name}), do: %{path: name} + defp place(segments) when is_list(segments), do: %{pointer: pointer(segments)} + + defp code(location, message, opts) do + opts[:code] || ecto_code(opts) || + raise ArgumentError, + "the error #{inspect(message)} at #{inspect(location)} has no code: " <> + "give it code: in its options (error-format spec §6.3)" + end + + defp ecto_code(opts) do + case Map.new(opts) do + %{validation: :cast, type: type} -> cast_code(type) + %{validation: :length, kind: kind} -> Map.get(@length_codes, kind) + %{validation: validation} -> Map.get(@validation_codes, validation) + %{constraint: constraint} -> Map.get(@constraint_codes, constraint) + %{type: type} when type in [:map, {:array, :map}] -> :type + _untagged -> nil + end + end + + # A value outside an enum's set is refused like validate_inclusion, not as + # a value of the wrong type (spec §6.2). `__enum_map__/0` is what ecto_enum's + # defenum modules export. + defp cast_code({:array, type}), do: cast_code(type) + defp cast_code({:parameterized, {Ecto.Enum, _params}}), do: :not_allowed + + defp cast_code(type) when is_atom(type) do + if Code.ensure_loaded?(type) and function_exported?(type, :__enum_map__, 0), + do: :not_allowed, + else: :type + end + + defp cast_code(_type), do: :type + + defp interpolate(message, opts) do + Regex.replace(~r"%{(\w+)}", message, fn whole, key -> + Enum.find_value(opts, whole, fn {name, value} -> + if Atom.to_string(name) == key, do: to_string(value) + end) + end) + end + + # A placed error is `{location, {message, opts}}`. `duplicate_of` is the + # location of the earlier request item holding the same primary key, when + # this changeset is a later one. + defp walk(%Changeset{} = changeset, path, renames, duplicate_of) do + own = + changeset.errors + |> Enum.reverse() + |> Enum.map(fn {field, error} -> + {error_location(changeset, path, field, error, renames), + duplicate(changeset, field, error, duplicate_of)} + end) + + children = + for {field, {tag, _relation}} <- changeset.types, + tag in [:assoc, :embed], + Map.has_key?(changeset.changes, field), + placed <- + walk_relation( + Map.fetch!(changeset.changes, field), + path ++ [Map.get(renames, field, field)], + renames + ), + do: placed + + own ++ children + end + + # Only children with params came from the request, and positions count + # only those (spec §7 item 3, plan D1). + defp walk_relation(children, path, renames) when is_list(children) do + {from_request, not_sent} = Enum.split_with(children, &from_request?/1) + Enum.each(not_sent, &refuse_errors(&1, path)) + + {placed, _firsts} = + from_request + |> Enum.with_index() + |> Enum.flat_map_reduce(%{}, fn {child, index}, firsts -> + key = primary_key(child) + first = Map.get(firsts, key, index) + duplicate_of = if first != index and nil not in key, do: path ++ [first] + + {walk(child, path ++ [index], renames, duplicate_of), + Map.put_new(firsts, key, index)} + end) + + placed + end + + defp walk_relation(%Changeset{} = child, path, renames) do + if from_request?(child) do + walk(child, path, renames, nil) + else + refuse_errors(child, path) + end + end + + defp walk_relation(nil, _path, _renames), do: [] + + defp from_request?(%Changeset{params: params}), do: params != nil + + defp refuse_errors(%Changeset{} = child, path) do + if child.valid? do + [] + else + raise ArgumentError, + "a child under #{pointer(path)} that the request didn't send has " <> + "errors, so they can't be placed in the request. Its producer " <> + "builds its own entries with Errors.entry/2 (error-format spec §7)" + end + end + + defp primary_key(%Changeset{data: %schema{}} = child) do + Enum.map(schema.__schema__(:primary_key), &Changeset.get_field(child, &1)) + end + + defp error_location(_changeset, path, :base, {_message, opts}, _renames), + do: path ++ List.wrap(opts[:key]) + + defp error_location(changeset, path, field, {_message, opts}, renames) do + path ++ + List.wrap( + opts[:key] || Map.get(renames, field) || + request_key(changeset, field, opts[:constraint]) + ) + end + + defp in_path([key] = location, path_keys) do + if key in path_keys, do: {:path, to_string(key)}, else: location + end + + defp in_path(location, _path_keys), do: location + + # assoc_constraint puts its error on the association, but the request sent + # the foreign key, and keys are the request's (spec §3 rule 2). + defp request_key(%Changeset{data: %schema{}}, field, :assoc) do + case schema.__schema__(:association, field) do + %Ecto.Association.BelongsTo{owner_key: owner_key} -> owner_key + _other -> field + end + end + + defp request_key(_changeset, field, _constraint), do: field + + # Ecto's duplicate-key check in cast_assoc and cast_embed is its one error + # with no options: "has already been taken", on the later child's primary + # key (spec §6.2). + defp duplicate( + %Changeset{data: %schema{}}, + field, + {_message, []} = error, + earlier + ) + when earlier != nil do + if field in schema.__schema__(:primary_key), + do: {"names the same id as " <> pointer(earlier), [code: :duplicate]}, + else: error + end + + defp duplicate(_changeset, _field, error, _earlier), do: error +end diff --git a/lib/lightning_web/api/filters.ex b/lib/lightning_web/api/filters.ex new file mode 100644 index 00000000000..54ff55090ce --- /dev/null +++ b/lib/lightning_web/api/filters.ex @@ -0,0 +1,169 @@ +defmodule LightningWeb.API.Filters do + @moduledoc """ + Reads a list route's query string into the typed values its context takes. + + A filter function answers `{:ok, filters}`, or `{:error, changeset}` with an + error on every bad parameter, for + `LightningWeb.API.Errors.refuse_parameters/1`. + `page/1` never refuses. + """ + alias Ecto.Changeset + alias Lightning.Config + alias Lightning.Invocation.LogLine + alias Lightning.Invocation.Query + + defmodule Cursor do + @moduledoc false + # The id of the last item a page carried, base64 encoded. + use Ecto.Type + + def type, do: :integer + + def cast(cursor) when is_binary(cursor) do + with {:ok, decoded} <- Base.decode64(cursor), + {id, ""} <- Integer.parse(decoded) do + {:ok, id} + else + _invalid -> :error + end + end + + def cast(_cursor), do: :error + + def load(id), do: {:ok, id} + def dump(id), do: {:ok, id} + + def encode(id), do: Base.encode64(to_string(id)) + end + + @collection_item_params %{ + created_after: :utc_datetime_usec, + created_before: :utc_datetime_usec, + cursor: Cursor, + key: :string, + limit: :integer, + updated_after: :utc_datetime_usec, + updated_before: :utc_datetime_usec + } + + @collection_item_parameters Enum.map( + Map.keys(@collection_item_params), + &Atom.to_string/1 + ) + + @run_states Lightning.Run.states() |> Enum.map(&to_string/1) + @work_order_states Lightning.WorkOrder.states() |> Enum.map(&to_string/1) + @log_levels Ecto.Enum.dump_values(LogLine, :level) + + @doc """ + The run filters in `params`, or a changeset with an error on every bad + parameter, none quoting its value. + + `state` is a comma-separated list of run states. + """ + @spec runs(map()) :: {:ok, map()} | {:error, Changeset.t()} + def runs(params) do + params + |> split_list("state") + |> cast( + Query.run_filter_types(), + &Changeset.validate_subset(&1, :state, @run_states) + ) + end + + @doc """ + The work order filters in `params`, or a changeset with an error on every bad + parameter, none quoting its value. + + `id` and `state` are comma-separated lists, `id` also a repeated parameter. + """ + @spec work_orders(map()) :: {:ok, map()} | {:error, Changeset.t()} + def work_orders(params) do + params + |> split_list("id") + |> split_list("state") + |> cast( + Query.work_order_filter_types(), + &Changeset.validate_subset(&1, :state, @work_order_states) + ) + end + + @doc """ + The log line filters in `params`, or a changeset with an error on every bad + parameter, none quoting its value. + """ + @spec log_lines(map()) :: {:ok, map()} | {:error, Changeset.t()} + def log_lines(params) do + cast( + params, + Query.log_line_filter_types(), + &Changeset.validate_inclusion(&1, :level, @log_levels) + ) + end + + @doc """ + The collection item filters in `params`, or a changeset with an error on + every bad parameter, none quoting its value. + + `limit` defaults to `Lightning.Config.collections_default_stream_limit/0`. + An empty value is a value, not a missing one. `cursor` is the one the + previous response carried. Parameters outside + `collection_item_parameters/0` are left out. + """ + @spec collection_items(map()) :: {:ok, map()} | {:error, Changeset.t()} + def collection_items(params) do + {%{limit: Config.collections_default_stream_limit()}, + @collection_item_params} + |> Changeset.cast(params, Map.keys(@collection_item_params), + empty_values: [] + ) + |> Changeset.validate_number(:limit, greater_than_or_equal_to: 0) + |> Changeset.apply_action(:validate) + end + + @doc "The names of the parameters `collection_items/1` reads." + @spec collection_item_parameters() :: [String.t()] + def collection_item_parameters, do: @collection_item_parameters + + @doc """ + The page `params` asks for, as `Lightning.Repo.paginate/2` takes it. + + Each value is read as Scrivener reads one: the number a string starts with, + so `5abc` is 5 and `2.0` is 2. A value below 1, or one with no number to + read, is left out and the default applies; it is never refused. Above 100, + `page_size` is cut to 100 by the repo. + """ + @spec page(map()) :: [page: pos_integer(), page_size: pos_integer()] + def page(params) do + for key <- [:page, :page_size], + {:ok, number} <- [leading_number(params[Atom.to_string(key)])], + do: {key, number} + end + + defp leading_number(value) when is_binary(value), + do: value |> Integer.parse() |> leading_number() + + defp leading_number({number, _rest}), do: leading_number(number) + + defp leading_number(number) when is_integer(number) and number > 0, + do: {:ok, number} + + defp leading_number(_value), do: :error + + defp cast(params, types, validate) do + {%{}, types} + |> Changeset.cast(params, Map.keys(types)) + |> validate.() + |> Changeset.apply_action(:validate) + end + + defp split_list(params, key) do + case params do + %{^key => value} when is_binary(value) -> + Map.put(params, key, String.split(value, ",", trim: true)) + + _other -> + params + end + end +end diff --git a/lib/lightning_web/api/no_route.ex b/lib/lightning_web/api/no_route.ex new file mode 100644 index 00000000000..1b843971aec --- /dev/null +++ b/lib/lightning_web/api/no_route.ex @@ -0,0 +1,25 @@ +defmodule LightningWeb.API.NoRoute do + @moduledoc """ + The catch-all route at the end of each JSON scope: 404 `not_found` as a + problem document that follows `Accept` (error-format spec §5). A path the + error format doesn't cover gets the router's own 404. + """ + @behaviour Plug + + alias LightningWeb.API.Problem + alias LightningWeb.API.ProblemPaths + + @impl true + def init(opts), do: opts + + @impl true + def call(conn, _opts) do + if ProblemPaths.member?(conn), + do: Problem.respond(conn, Problem.new(:not_found)), + else: + raise(Phoenix.Router.NoRouteError, + conn: conn, + router: LightningWeb.Router + ) + end +end diff --git a/lib/lightning_web/api/problem.ex b/lib/lightning_web/api/problem.ex new file mode 100644 index 00000000000..e248902f161 --- /dev/null +++ b/lib/lightning_web/api/problem.ex @@ -0,0 +1,129 @@ +defmodule LightningWeb.API.Problem do + @moduledoc """ + An RFC 9457 problem document (error-format spec §2), and the one function + that sends one. + """ + alias LightningWeb.API.Codes + alias LightningWeb.API.Errors + alias Plug.Conn.Utils + + @enforce_keys [:status, :code] + defstruct [:status, :code, errors: [], extensions: %{}, headers: []] + + @type t :: %__MODULE__{ + status: pos_integer(), + code: Codes.code(), + errors: [Errors.entry()], + extensions: map(), + headers: [{String.t(), String.t()}] + } + + @doc """ + Builds the problem for a response code from `LightningWeb.API.Codes`. + + Options: + + * `:errors` - the entries, from `LightningWeb.API.Errors`. + * `:extensions` - members the spec names for this answer, such as + `existing` or `work_order_id`. + * `:headers` - response headers that belong to the answer. + * `:retry_after` - seconds, sent as `retry-after`; `nil` sends none. + * `:status` - one of `LightningWeb.API.Codes.statuses/1` for `code` + other than its own, where the spec keeps a route's status (§12). + """ + @spec new(Codes.code(), keyword()) :: t() + def new(code, opts \\ []) do + status = Keyword.get(opts, :status, Codes.status(code)) + + unless status in Codes.statuses(code) do + raise ArgumentError, "#{inspect(code)} is never sent at #{status}" + end + + %__MODULE__{ + code: code, + status: status, + errors: Keyword.get(opts, :errors, []), + extensions: Keyword.get(opts, :extensions, %{}), + headers: + Keyword.get(opts, :headers, []) ++ + retry_after_header(Keyword.get(opts, :retry_after)) + } + end + + defp retry_after_header(nil), do: [] + + defp retry_after_header(seconds), + do: [{"retry-after", Integer.to_string(seconds)}] + + @doc """ + Sends `problem` as the response. + + The body is the same whatever the request's `Accept`. The content type is + `application/problem+json` only when `Accept` names it, and + `application/json` otherwise (spec §5). + """ + @spec respond(Plug.Conn.t(), t()) :: Plug.Conn.t() + def respond(conn, %__MODULE__{} = problem) do + conn + |> Plug.Conn.put_resp_content_type(content_type(conn)) + |> Plug.Conn.merge_resp_headers(problem.headers) + |> Plug.Conn.send_resp( + problem.status, + Jason.encode_to_iodata!(body(problem)) + ) + end + + @doc """ + Sends `problem` as the response, as `respond/2` does, and halts the plug + pipeline. + """ + @spec halt(Plug.Conn.t(), t()) :: Plug.Conn.t() + def halt(conn, %__MODULE__{} = problem) do + conn + |> respond(problem) + |> Plug.Conn.halt() + end + + @doc """ + The problem document `respond/2` sends. + """ + @spec body(t()) :: map() + def body(problem) do + %{ + type: "about:blank", + status: problem.status, + title: Plug.Conn.Status.reason_phrase(problem.status), + code: problem.code, + detail: Codes.detail(problem.code) + } + |> then(fn body -> + if problem.errors == [], + do: body, + else: Map.put(body, :errors, problem.errors) + end) + |> Map.merge(problem.extensions) + end + + @doc """ + The content type a problem sent on `conn` takes: `application/problem+json` + when the request's `Accept` names it, in any position and whatever its `q`, + and `application/json` otherwise (spec §5). + """ + @spec content_type(Plug.Conn.t()) :: String.t() + def content_type(conn) do + asks_for_problem? = + conn + |> Plug.Conn.get_req_header("accept") + |> Enum.flat_map(&Utils.list/1) + |> Enum.any?( + &match?( + {:ok, "application", "problem+json", _params}, + Utils.media_type(&1) + ) + ) + + if asks_for_problem?, + do: "application/problem+json", + else: "application/json" + end +end diff --git a/lib/lightning_web/api/problem_paths.ex b/lib/lightning_web/api/problem_paths.ex new file mode 100644 index 00000000000..e117209d387 --- /dev/null +++ b/lib/lightning_web/api/problem_paths.ex @@ -0,0 +1,36 @@ +defmodule LightningWeb.API.ProblemPaths do + @moduledoc """ + The paths whose errors before a controller runs are problem documents: every + `/api` route but the OAuth token endpoint, `/collections` and + `/adaptors/catalogue` (error-format spec §1). Webhooks under `/i` are not + among them, though a body they can't read is answered with a problem document + (§12). + """ + + @body_errors [ + Plug.Parsers.BadEncodingError, + Plug.Parsers.ParseError, + Plug.Parsers.RequestTooLargeError + ] + + @spec member?(Plug.Conn.t()) :: boolean() + def member?(%Plug.Conn{path_info: path_info}), do: problem_path?(path_info) + + @doc """ + Whether `exception`, raised while answering `conn`, is answered with a + problem document: on every path `member?/1` covers, and on `/i/*` when it is + a request body Lightning can't read. + """ + @spec answers?(Plug.Conn.t(), term()) :: boolean() + def answers?(%Plug.Conn{path_info: ["i" | _]}, %struct{}) + when struct in @body_errors, + do: true + + def answers?(conn, _exception), do: member?(conn) + + defp problem_path?(["api", "oauth", "token" | _]), do: false + defp problem_path?(["api" | _]), do: true + defp problem_path?(["collections" | _]), do: true + defp problem_path?(["adaptors", "catalogue" | _]), do: true + defp problem_path?(_path_info), do: false +end diff --git a/lib/lightning_web/api/request.ex b/lib/lightning_web/api/request.ex new file mode 100644 index 00000000000..5db7dec704c --- /dev/null +++ b/lib/lightning_web/api/request.ex @@ -0,0 +1,110 @@ +defmodule LightningWeb.API.Request do + @moduledoc """ + The checks on a request's path id and JSON body before the body reaches a + changeset (error-format spec §7 item 8, §9.5, §9.6), and telling when a + create-or-replace `PUT /:id` lost the race for its id. + """ + alias Ecto.Changeset + alias LightningWeb.API.Errors + alias LightningWeb.API.Problem + + @mismatch {"does not match the id in the path", code: :mismatch} + + @doc """ + Checks `body` against the keys the route accepts and the id in its path. + + `path_id` is the path's id, already cast as a UUID. + + Returns `{:error, problem}` with a 400 `malformed` problem when the body is + not a JSON object. Otherwise returns `{:ok, entries}`, empty when the body + passes: one `unknown_key` entry for each key outside `accepted`, and a + `mismatch` entry when the body's `id` is not `path_id`. + """ + @spec check(map(), Ecto.UUID.t(), [atom()]) :: + {:ok, [Errors.entry()]} | {:error, Problem.t()} + def check(body, path_id, accepted) do + with :ok <- check_object(body) do + {:ok, unknown_keys(body, [], accepted) ++ id_mismatch(body, path_id)} + end + end + + @doc """ + One `unknown_key` entry for each key of `object` outside `accepted`, in key + order, each under `path`. + """ + @spec unknown_keys(map(), [Errors.segment()], [atom()]) :: [Errors.entry()] + def unknown_keys(object, path, accepted) do + (Map.keys(object) -- Enum.map(accepted, &Atom.to_string/1)) + |> Enum.sort() + |> Enum.map( + &Errors.entry(path ++ [&1], {"is not accepted", code: :unknown_key}) + ) + end + + @doc """ + `:ok` when `body`, a request's body params, was a JSON object. Otherwise a + 400 `malformed` problem, for any route that takes an object (spec §9.5). + Plug.Parsers puts any other JSON value under `"_json"`. + """ + @spec check_object(map()) :: :ok | {:error, Problem.t()} + def check_object(%{"_json" => _not_an_object}), + do: {:error, Problem.new(:malformed)} + + def check_object(_body), do: :ok + + @doc """ + The path's id, written out as a UUID. + + Returns `{:error, :not_found}` for anything else, since an id that isn't a + UUID names nothing (spec §4). `Ecto.UUID.cast/1` would also take any 16-byte + string as a raw UUID, landing the request under an id it doesn't name. + """ + @spec path_id(term()) :: {:ok, Ecto.UUID.t()} | {:error, :not_found} + def path_id(id) do + with {:ok, raw} <- Ecto.UUID.dump(id), + {:ok, uuid} <- Ecto.UUID.load(raw) do + {:ok, uuid} + else + :error -> {:error, :not_found} + end + end + + @doc """ + Whether an insert at the path's id failed because the id's row now exists, + that is, another request created it after the route looked the id up. + """ + @spec id_taken?(Changeset.t()) :: boolean() + def id_taken?(changeset) do + changeset.errors + |> Keyword.get_values(:id) + |> Enum.any?(fn {_message, opts} -> opts[:constraint] == :unique end) + end + + @doc """ + `changeset` with a `mismatch` error on `field` when `body` names an id there + that is not `path_id`. + """ + @spec validate_matches_path(Changeset.t(), map(), atom(), Ecto.UUID.t()) :: + Changeset.t() + def validate_matches_path(changeset, body, field, path_id) do + if matches_path?(body, field, path_id) do + changeset + else + {message, opts} = @mismatch + Changeset.add_error(changeset, field, message, opts) + end + end + + defp id_mismatch(body, path_id) do + if matches_path?(body, :id, path_id), + do: [], + else: [Errors.entry([:id], @mismatch)] + end + + defp matches_path?(body, field, path_id) do + case Map.fetch(body, Atom.to_string(field)) do + {:ok, value} -> path_id(value) == {:ok, path_id} + :error -> true + end + end +end diff --git a/lib/lightning_web/api/target.ex b/lib/lightning_web/api/target.ex new file mode 100644 index 00000000000..a80f7e2ab2c --- /dev/null +++ b/lib/lightning_web/api/target.ex @@ -0,0 +1,80 @@ +defmodule LightningWeb.API.Target do + @moduledoc """ + Resolves the target an API action works on in a project, and asks the policy + whether the caller may act on it, in the order of error-format spec §4. + + The policy is asked only after the target is looked up, so a missing project + or target is 404 (row 3) even to a member the policy would refuse 403 (row 4). + A caller with no standing hears the policy's `:no_standing`, also + 404. The token's grant (row 2) has to be refused before any lookup, so an + action using this module checks it first with + `LightningWeb.Plugs.AuthorizeAction`. Every refusal is one + `LightningWeb.API.FallbackController` answers. + """ + alias Lightning.Policies.Permissions + alias Lightning.Projects.Project + alias Lightning.Scope + + @type permission :: {policy :: atom(), action :: atom()} + + @doc """ + The project `subject` identifies, once the caller may apply the permission to + it. + + Target.project(conn.assigns.scope, project_id, {:project_users, :access_project}) + """ + @spec project(Scope.t() | Scope.actor(), Scope.subject(), permission()) :: + {:ok, Project.t()} | {:error, term()} + def project(scope, subject, permission), + do: in_project(scope, subject, permission, &{:ok, &1}) + + @doc """ + The project `id` names, as `project/3` answers it, or a new, unsaved + `%Project{}` when no project has that id, or `id` is nil, and the caller may + create projects. + + An id taken by a project the caller has no standing on is still 404, not a new + project (plan D14). + """ + @spec project_or_new( + Scope.t() | Scope.actor(), + Ecto.UUID.t() | nil, + permission() + ) :: {:ok, Project.t()} | {:error, term()} + def project_or_new(scope, nil, _permission), do: new_project(scope) + + def project_or_new(scope, id, permission) do + case project(scope, id, permission) do + {:error, :no_such_project} -> new_project(scope) + found_or_refused -> found_or_refused + end + end + + defp new_project(scope) do + with :ok <- Permissions.permit(:provisioning, :create_project, scope, nil), + do: {:ok, %Project{}} + end + + @doc """ + What `lookup` finds in the project `subject` identifies, once the caller may + apply the permission to that project. `lookup` is given the project and + answers `{:ok, target}`, or a refusal such as `{:error, :not_found}`. + + Target.in_project(scope, project_id, {:project_users, :edit_workflow}, + &fetch_workflow(id, &1) + ) + """ + @spec in_project( + Scope.t() | Scope.actor(), + Scope.subject(), + permission(), + (Project.t() -> {:ok, target} | {:error, term()}) + ) :: {:ok, target} | {:error, term()} + when target: term() + def in_project(scope, subject, {policy, action}, lookup) do + with {:ok, scope} <- Scope.fetch(scope, subject), + {:ok, target} <- lookup.(scope.project), + :ok <- Permissions.permit(policy, action, scope, scope.project), + do: {:ok, target} + end +end diff --git a/lib/lightning_web/auth.ex b/lib/lightning_web/auth.ex index 9e53a272b67..563003065d0 100644 --- a/lib/lightning_web/auth.ex +++ b/lib/lightning_web/auth.ex @@ -1,18 +1,53 @@ defmodule LightningWeb.Auth do @moduledoc """ - Shared HTTP request authentication functions. + Request authentication for every HTTP route: what credentials the request + sent, whether they match a trigger's or channel's webhook auth methods, and + the refusal a failed login gets. - Validates inbound requests against `WebhookAuthMethod` records using - API key (`x-api-key` header) or Basic Auth (`authorization` header). - Used by both `WebhookAuth` plug (triggers) and `ChannelProxyPlug` - (channels). + Checking a bearer token is the core's job (`Lightning.Tokens`, + `Lightning.ServiceAccount.AccessToken`, `Lightning.Accounts`); the plugs call + this module to read the token and, when the check fails, to refuse. - All comparisons use `Plug.Crypto.secure_compare/2` to prevent - timing attacks. + A refusal is a problem document with the `WWW-Authenticate` challenge the + route's scheme calls for (RFC 6750 §3, error-format spec §9.1): + a 401, or a 400 when the `Bearer` header can't be read. A refused permission + is never a 401 (plan D9), so only auth plugs refuse here; the one 403 + challenge, `insufficient_scope`, is built for `API.FallbackController`. + + All secret comparisons use `Plug.Crypto.secure_compare/2`. """ import Plug.Conn, only: [get_req_header: 2] alias Lightning.Workflows.WebhookAuthMethod + alias LightningWeb.API.Problem + + @type scheme :: :bearer | :session + @type auth_methods :: [%{auth_type: :basic | :api}] + @type header :: {String.t(), String.t()} + + @doc """ + The bearer token the request sent in its `Authorization` header. + + The scheme is matched without regard to case (RFC 9110 §11.1). A header with + any other scheme, `Basic` included, is no token sent (error-format spec §9.1). + A `Bearer` header with no token after it, or with whitespace inside the token, + or more than one `Bearer` header, is a token sent but malformed. + """ + @spec bearer_token(Plug.Conn.t()) :: + {:ok, String.t()} | {:error, :missing | :malformed} + def bearer_token(conn) do + bearer_headers = + for value <- get_req_header(conn, "authorization"), + [scheme | token] <- [String.split(value)], + String.downcase(scheme) == "bearer", + do: token + + case bearer_headers do + [] -> {:error, :missing} + [[token]] -> {:ok, token} + _malformed -> {:error, :malformed} + end + end @doc """ Returns true if the request's `x-api-key` header matches any @@ -39,6 +74,62 @@ defmodule LightningWeb.Auth do get_req_header(conn, "authorization") != [] end + @doc """ + Refuses a failed login, and halts: on a route that takes a bearer token or + the session cookie, or on a trigger or channel, given its webhook auth + methods. + + A trigger or channel answers credentials sent but refused itself, so its + refusal is always `unauthenticated`, never `invalid_token`. + """ + @spec refuse(Plug.Conn.t(), scheme() | auth_methods()) :: Plug.Conn.t() + def refuse(conn, :bearer) do + case bearer_token(conn) do + {:error, :missing} -> + respond(conn, :unauthenticated, [{"www-authenticate", "Bearer"}]) + + {:error, :malformed} -> + respond(conn, :invalid_request, [bearer(~s(error="invalid_request"))]) + + {:ok, _token} -> + respond(conn, :invalid_token, [bearer(~s(error="invalid_token"))]) + end + end + + # No `Session` scheme is registered; browsers don't prompt for one they don't + # know, which is the point (RFC 9110 §15.5.2 requires a challenge). + def refuse(conn, :session) do + respond(conn, :unauthenticated, [ + {"www-authenticate", ~s(Session realm="Lightning")} + ]) + end + + # A trigger or channel that its webhook auth methods didn't let in. It is + # challenged for an API key when it takes one, and never for Basic: every + # trigger lives under `/i/` and every channel under `/channels/`, so a browser + # prompted for one Basic login may send it unasked to the others, where an + # open trigger stores it in the dataclip and an open channel forwards it + # upstream. A route that takes only Basic sends no challenge. + def refuse(conn, auth_methods) when is_list(auth_methods) do + challenges = + if Enum.any?(auth_methods, &(&1.auth_type == :api)), + do: [{"www-authenticate", ~s(ApiKey header="x-api-key")}], + else: [] + + respond(conn, :unauthenticated, challenges) + end + + @doc "The challenge on a 403 for a token that lacks `grant`." + @spec insufficient_scope_challenge(String.t()) :: header() + def insufficient_scope_challenge(grant), + do: bearer(~s(error="insufficient_scope", scope="#{grant}")) + + defp respond(conn, code, headers) do + Problem.halt(conn, Problem.new(code, headers: headers)) + end + + defp bearer(params), do: {"www-authenticate", "Bearer " <> params} + defp key_matches?( conn, %WebhookAuthMethod{ diff --git a/lib/lightning_web/channels/channel_helpers.ex b/lib/lightning_web/channels/channel_helpers.ex index dac113f72f6..14b5cbabf8c 100644 --- a/lib/lightning_web/channels/channel_helpers.ex +++ b/lib/lightning_web/channels/channel_helpers.ex @@ -3,6 +3,8 @@ defmodule LightningWeb.ChannelHelpers do Helper functions for channels """ + import LightningWeb.CoreComponents, only: [translate_error: 1] + @doc """ Replies, and reports an error reply to Sentry. @@ -20,9 +22,15 @@ defmodule LightningWeb.ChannelHelpers do {:reply, reply, socket} end - defp error_to_map(%Ecto.Changeset{} = error) do - LightningWeb.ChangesetJSON.errors(error) - end + @doc """ + A changeset's errors as the channels send them, `%{field => [message]}`. + ws-worker reads this shape. + """ + @spec changeset_errors(Ecto.Changeset.t()) :: %{atom() => [String.t()]} + def changeset_errors(%Ecto.Changeset{} = changeset), + do: Ecto.Changeset.traverse_errors(changeset, &translate_error/1) + + defp error_to_map(%Ecto.Changeset{} = error), do: changeset_errors(error) defp error_to_map(error), do: error diff --git a/lib/lightning_web/channels/run_channel.ex b/lib/lightning_web/channels/run_channel.ex index a2c2e30897e..7149e62a078 100644 --- a/lib/lightning_web/channels/run_channel.ex +++ b/lib/lightning_web/channels/run_channel.ex @@ -331,7 +331,7 @@ defmodule LightningWeb.RunChannel do end # This run's project changed who may see it. Re-derive the permission instead - # of matching on the event: `Projects.Scope` re-reads the project, so one + # of matching on the event: `Lightning.Scope` re-reads the project, so one # clause covers a membership being revoked and a support user losing the # support access that was their only standing. A role change cannot take # `:access_project` away, so it resolves to a no-op here. diff --git a/lib/lightning_web/channels/worker_channel.ex b/lib/lightning_web/channels/worker_channel.ex index af1e00b4918..1076687a900 100644 --- a/lib/lightning_web/channels/worker_channel.ex +++ b/lib/lightning_web/channels/worker_channel.ex @@ -9,6 +9,7 @@ defmodule LightningWeb.WorkerChannel do alias Lightning.Services.UsageLimiter alias Lightning.Workers + alias LightningWeb.ChannelHelpers alias LightningWeb.Observability alias LightningWeb.WorkerPresence @@ -63,7 +64,7 @@ defmodule LightningWeb.WorkerChannel do {:reply, {:ok, %{runs: runs}}, socket} {:error, changeset} -> - {:reply, {:error, LightningWeb.ChangesetJSON.errors(changeset)}, socket} + {:reply, {:error, ChannelHelpers.changeset_errors(changeset)}, socket} end end diff --git a/lib/lightning_web/channels/workflow_channel.ex b/lib/lightning_web/channels/workflow_channel.ex index e9d4fc0198c..02113365d81 100644 --- a/lib/lightning_web/channels/workflow_channel.ex +++ b/lib/lightning_web/channels/workflow_channel.ex @@ -24,8 +24,8 @@ defmodule LightningWeb.WorkflowChannel do alias Lightning.Projects.MergeProjects alias Lightning.Projects.ProjectLimiter alias Lightning.Projects.Sandboxes - alias Lightning.Projects.Scope alias Lightning.Repo + alias Lightning.Scope alias Lightning.VersionControl alias Lightning.VersionControl.VersionControlUsageLimiter alias Lightning.Workflows @@ -460,7 +460,7 @@ defmodule LightningWeb.WorkflowChannel do with :ok <- authorize_edit_workflow(socket), %WorkflowRelease{snapshot: %_{}} = release <- WorkflowReleases.get_by_version_number(workflow.id, number), - {:ok, restored} <- restore_or_conflict(workflow, release, user) do + {:ok, restored} <- Workflows.restore_version(workflow, release, user) do WorkflowReconciler.request_reconciliation(workflow.id) broadcast_workflow_saved(socket, restored) @@ -1426,12 +1426,6 @@ defmodule LightningWeb.WorkflowChannel do } end - defp restore_or_conflict(workflow, release, user) do - Workflows.restore_version(workflow, release, user) - rescue - Ecto.StaleEntryError -> {:error, :workflow_moved_on} - end - defp render_losing_triggers(workflow, snapshot) do kept = MapSet.new(snapshot.triggers, & &1.id) @@ -1529,7 +1523,8 @@ defmodule LightningWeb.WorkflowChannel do Permissions.can?(:sandboxes, :delete_sandbox, user, project) } - # No such project, or one scheduled for deletion. + # No such project. A project scheduled for deletion loads instead, and + # every check above refuses it. {:error, _reason} -> %{ can_edit_workflow: false, @@ -1668,11 +1663,11 @@ defmodule LightningWeb.WorkflowChannel do }} end - defp workflow_error_reply({:error, :workflow_moved_on}) do + defp workflow_error_reply({:error, :concurrent_edit}) do {:error, %{ errors: %{ - base: ["Someone else saved this workflow. Try the restore again."] + base: ["Someone else saved this workflow. Try again."] }, type: "workflow_moved_on" }} diff --git a/lib/lightning_web/controllers/adaptor_controller.ex b/lib/lightning_web/controllers/adaptor_controller.ex index 27c2f52627d..d522f2ddd12 100644 --- a/lib/lightning_web/controllers/adaptor_controller.ex +++ b/lib/lightning_web/controllers/adaptor_controller.ex @@ -7,14 +7,15 @@ defmodule LightningWeb.AdaptorController do matching `If-None-Match` answers 304 without touching Postgres, and a miss on the ETag still serves an already-rendered payload. - A catalogue that has never loaded answers 503 with a `retry-after` rather - than an empty list, so the picker shows its retry state instead of no + A catalogue that has never loaded answers 503 `not_ready` with a `retry-after` + rather than an empty list, so the picker shows its retry state instead of no adaptors. """ use LightningWeb, :controller alias Lightning.Adaptors + alias LightningWeb.API.Problem require Logger @@ -41,10 +42,7 @@ defmodule LightningWeb.AdaptorController do inspect(reason) ) - conn - |> put_resp_header("retry-after", "5") - |> put_status(:service_unavailable) - |> json(%{"error" => "adaptor catalogue unavailable"}) + Problem.respond(conn, Problem.new(:not_ready, retry_after: 5)) end end diff --git a/lib/lightning_web/controllers/api/ai_assistant_controller.ex b/lib/lightning_web/controllers/api/ai_assistant_controller.ex index 16d8d3db4f4..994231e067d 100644 --- a/lib/lightning_web/controllers/api/ai_assistant_controller.ex +++ b/lib/lightning_web/controllers/api/ai_assistant_controller.ex @@ -4,14 +4,31 @@ defmodule LightningWeb.API.AiAssistantController do """ use LightningWeb, :controller + import Ecto.Changeset + alias Lightning.AiAssistant alias Lightning.Jobs - alias Lightning.Policies.Permissions - alias Lightning.Projects alias Lightning.Workflows + alias LightningWeb.API.Errors + alias LightningWeb.API.FallbackController + alias LightningWeb.API.Request + alias LightningWeb.API.Target alias LightningWeb.Channels.AiAssistantJSON - action_fallback LightningWeb.FallbackController + action_fallback FallbackController + + @session_types ["job_code", "workflow_template"] + + @access_project {:project_users, :access_project} + + @params %{ + session_type: :string, + job_id: :string, + project_id: :string, + workflow_id: Ecto.UUID, + offset: :integer, + limit: :integer + } @doc """ List AI Assistant sessions for a given context (job or project). @@ -20,39 +37,30 @@ defmodule LightningWeb.API.AiAssistantController do - `session_type`: "job_code" or "workflow_template" (required) - `job_id`: Job ID (required if session_type is "job_code") - `project_id`: Project ID (required if session_type is "workflow_template") + - `workflow_id`: narrows "workflow_template" sessions to one workflow - `offset`: Pagination offset (optional, default: 0) - `limit`: Pagination limit (optional, default: 20) + The job or project the query names is the target: it is authorised before the + other parameters are read, and a caller with no standing on it gets 404 + (error-format spec §4). Every bad parameter is reported in one 400 + `invalid_parameter` (§9.4). + ## Examples - GET /api/ai_assistant/sessions?session_type=job_code&job_id=123 - GET /api/ai_assistant/sessions?session_type=workflow_template&project_id=456 """ def list_sessions(conn, params) do - user = conn.assigns[:current_user] - - with {:ok, session_type} <- validate_session_type(params), - {:ok, resource} <- get_resource(session_type, params), - :ok <- authorize_access(session_type, resource, user) do - offset = Map.get(params, "offset", "0") |> String.to_integer() - limit = Map.get(params, "limit", "20") |> String.to_integer() - - opts = [offset: offset, limit: limit] - - opts = - if session_type == "workflow_template" do - workflow = - case params["workflow_id"] do - nil -> nil - workflow_id -> Workflows.get_workflow(workflow_id) - end - - Keyword.put(opts, :workflow, workflow) - else - opts - end - + changeset = cast_params(params) + + with {:ok, target} <- target(changeset), + {:ok, resource} <- authorize(target, conn.assigns.current_user), + {:ok, query} <- + changeset + |> apply_action(:validate) + |> Errors.refuse_parameters() do %{sessions: sessions, pagination: pagination} = - AiAssistant.list_sessions(resource, :desc, opts) + AiAssistant.list_sessions(resource, :desc, list_opts(query)) json(conn, %{ sessions: AiAssistantJSON.format_sessions(sessions), @@ -65,101 +73,85 @@ defmodule LightningWeb.API.AiAssistantController do end end - defp validate_session_type(%{"session_type" => session_type}) - when session_type in ["job_code", "workflow_template"] do - {:ok, session_type} + defp cast_params(params) do + {%{offset: 0, limit: 20, workflow_id: nil}, @params} + |> cast(params, Map.keys(@params)) + |> validate_required([:session_type]) + |> validate_inclusion(:session_type, @session_types) + |> validate_target_id() + |> validate_number(:offset, greater_than_or_equal_to: 0) + |> validate_number(:limit, greater_than: 0) end - defp validate_session_type(_params) do - {:error, :bad_request} + defp validate_target_id(changeset) do + case get_change(changeset, :session_type) do + "job_code" -> validate_required(changeset, [:job_id]) + "workflow_template" -> validate_required(changeset, [:project_id]) + _other -> changeset + end end - defp get_resource("job_code", %{"job_id" => job_id}) do - {:ok, job_id} - end + # With no target named there is nothing to authorise, so every bad parameter + # is reported at once. + defp target(%{changes: %{session_type: "job_code", job_id: job_id}}), + do: {:ok, {:job_code, job_id}} - defp get_resource("job_code", _params) do - {:error, :bad_request} - end + defp target(%{changes: %{session_type: "workflow_template", project_id: id}}), + do: {:ok, {:workflow_template, id}} - defp get_resource("workflow_template", %{"project_id" => project_id}) do - case Projects.get_project(project_id) do - nil -> {:error, :not_found} - project -> {:ok, project} - end - end + defp target(changeset), do: {:error, Errors.parameters(changeset)} - defp get_resource("workflow_template", _params) do - {:error, :bad_request} + defp list_opts(%{session_type: "workflow_template"} = query) do + workflow = query.workflow_id && Workflows.get_workflow(query.workflow_id) + [offset: query.offset, limit: query.limit, workflow: workflow] end - defp authorize_access("job_code", job_id, user) when is_binary(job_id) do - alias Lightning.AiAssistant.ChatSession - alias Lightning.Repo - import Ecto.Query - - case Jobs.get_job(job_id, - include: [workflow: [project: :project_users]] - ) do - {:ok, job} -> - check_job_access(job, user) - - {:error, :not_found} -> - session = - from(s in ChatSession, - where: s.session_type == "job_code", - where: - s.job_id == ^job_id or - fragment("? -> 'unsaved_job' ->> 'id' = ?", s.meta, ^job_id), - limit: 1 - ) - |> Repo.one() - - check_unsaved_job_access(session, user) - end - end + defp list_opts(query), do: [offset: query.offset, limit: query.limit] - defp authorize_access("workflow_template", project, user) do - case Permissions.can(:workflows, :access_read, user, project) do - :ok -> :ok - {:error, _reason} -> {:error, :forbidden} - end - end + defp authorize({:workflow_template, project_id}, user), + do: Target.project(user, project_id, @access_project) - defp check_job_access(job, user) do - check_workflow_access(job.workflow, user) + defp authorize({:job_code, job_id}, user) do + with {:ok, job_id} <- Request.path_id(job_id), + :ok <- authorize_job(job_id, user), + do: {:ok, job_id} end - defp check_unsaved_job_access(nil, _user) do - # No matching session: listing returns nothing, so allowing is safe. - :ok + defp authorize_job(job_id, user) do + case Jobs.get_job(job_id, include: [:workflow]) do + {:ok, job} -> authorize_workflow(job.workflow, user) + {:error, :not_found} -> authorize_unsaved_job(job_id, user) + end end - defp check_unsaved_job_access( - %{meta: %{"unsaved_job" => %{"workflow_id" => workflow_id}}} = session, - user - ) do - case Workflows.get_workflow(workflow_id) do + defp authorize_unsaved_job(job_id, user) do + session = AiAssistant.unsaved_job_session(job_id) + + case session do + # No matching session: listing returns nothing, so allowing is safe. nil -> - owner_or_forbidden(session, user) + :ok + + %{meta: %{"unsaved_job" => %{"workflow_id" => workflow_id}}} -> + case Workflows.get_workflow(workflow_id) do + nil -> owner_only(session, user) + workflow -> authorize_workflow(workflow, user) + end - workflow -> - workflow = Lightning.Repo.preload(workflow, project: [:project_users]) - check_workflow_access(workflow, user) + # No workflow to authorise against: owner-only, never allow-all (H-8a). + session -> + owner_only(session, user) end end - # No workflow to authorise against: owner-only, never allow-all (H-8a). - defp check_unsaved_job_access(session, user), - do: owner_or_forbidden(session, user) + # Someone else's session with no workflow is one the caller has no standing + # on, so it answers as if it didn't exist (spec §4 row 3). + defp owner_only(%{user_id: user_id}, %{id: user_id}), do: :ok + defp owner_only(_session, _user), do: {:error, :not_found} - defp owner_or_forbidden(%{user_id: user_id}, %{id: user_id}), do: :ok - defp owner_or_forbidden(_session, _user), do: {:error, :forbidden} - - defp check_workflow_access(workflow, user) do - case Permissions.can(:workflows, :access_read, user, workflow.project) do - :ok -> :ok - {:error, _reason} -> {:error, :forbidden} - end + defp authorize_workflow(workflow, user) do + with {:ok, _project} <- + Target.project(user, workflow.project_id, @access_project), + do: :ok end end diff --git a/lib/lightning_web/controllers/api/collection_controller.ex b/lib/lightning_web/controllers/api/collection_controller.ex new file mode 100644 index 00000000000..86026531378 --- /dev/null +++ b/lib/lightning_web/controllers/api/collection_controller.ex @@ -0,0 +1,132 @@ +defmodule LightningWeb.API.CollectionController do + @moduledoc """ + Writes and reads a project's collection by id. + `LightningWeb.CollectionsController` serves the items inside it. + + GET /api/projects/:project_id/collections/:id + PUT /api/projects/:project_id/collections/:id {"name": "patients"} + + Both answer `{"data": {"id", "project_id", "name"}}`. + """ + use LightningWeb, :controller + + alias Lightning.Collections + alias Lightning.Collections.Collection + alias Lightning.Projects.Project + alias LightningWeb.API.Errors + alias LightningWeb.API.FallbackController + alias LightningWeb.API.Request + alias LightningWeb.API.Target + alias LightningWeb.Plugs.AuthorizeAction + + action_fallback FallbackController + + plug AuthorizeAction, + {:collections, :manage_collection} + when action in [:show, :update] + + @accepted [:id, :name] + @manage_collection {:collections, :manage_collection} + + @spec show(Plug.Conn.t(), map()) :: Plug.Conn.t() | {:error, term()} + def show(conn, %{"project_id" => project_id, "id" => id}) do + with {:ok, collection} <- + Target.in_project( + conn.assigns.scope, + project_id, + @manage_collection, + &fetch_collection(id, &1) + ) do + render_collection(conn, collection) + end + end + + @doc """ + Creates the collection under the path's id in the project, or renames the + project's collection that has it, keeping its items. + + An id held by another project's collection answers 409 `id_taken`, since + Lightning never moves a collection between projects. A refused body answers + one 422 with its keys and the collection's fields together, before the plan + limit is asked (error-format spec §4, D13). + """ + @spec update(Plug.Conn.t(), map()) :: Plug.Conn.t() | {:error, term()} + def update(conn, %{"project_id" => project_id, "id" => id}) do + body = conn.body_params + + with {:ok, project} <- + Target.project(conn.assigns.scope, project_id, @manage_collection), + {:ok, id} <- Request.path_id(id), + {:ok, target} <- update_target(id, project), + {:ok, entries} <- Request.check(body, id, @accepted) do + attrs = %{"name" => body["name"]} + + case target do + :new -> create(conn, project, id, attrs, entries) + collection -> rename(conn, collection, attrs, entries) + end + end + end + + defp fetch_collection(id, %Project{id: project_id}) do + with {:ok, id} <- Request.path_id(id) do + case Collections.get_collection_by_id(id) do + %Collection{project_id: ^project_id} = collection -> {:ok, collection} + _ -> {:error, :not_found} + end + end + end + + defp update_target(id, %Project{id: project_id}) do + case Collections.get_collection_by_id(id) do + nil -> {:ok, :new} + %Collection{project_id: ^project_id} = collection -> {:ok, collection} + %Collection{} -> {:error, :id_taken} + end + end + + defp create(conn, project, id, attrs, entries) do + attrs = Map.put(attrs, "project_id", project.id) + + changeset = Collection.changeset(%Collection{id: id}, attrs) + + with :ok <- Errors.refuse(entries ++ Errors.from_changeset(changeset)) do + case Collections.create_collection( + attrs, + conn.assigns.current_resource, + id: id + ) do + {:ok, collection} -> + conn |> put_status(:created) |> render_collection(collection) + + # Someone else created it since the lookup. Rerun the request so it + # meets that collection. + {:error, %Ecto.Changeset{} = changeset} -> + if Request.id_taken?(changeset), + do: update(conn, %{"project_id" => project.id, "id" => id}), + else: {:error, changeset} + + error -> + error + end + end + end + + defp rename(conn, collection, attrs, entries) do + changeset = Collection.rename_changeset(collection, attrs) + + with :ok <- Errors.refuse(entries ++ Errors.from_changeset(changeset)), + {:ok, collection} <- + Collections.update_collection( + collection, + attrs, + conn.assigns.current_resource + ) do + render_collection(conn, collection) + end + end + + defp render_collection(conn, collection) do + json(conn, %{data: Map.take(collection, [:id, :project_id, :name])}) + end +end diff --git a/lib/lightning_web/controllers/api/credential_controller.ex b/lib/lightning_web/controllers/api/credential_controller.ex index 9c5d372df7b..6c3200eb98d 100644 --- a/lib/lightning_web/controllers/api/credential_controller.ex +++ b/lib/lightning_web/controllers/api/credential_controller.ex @@ -17,55 +17,55 @@ defmodule LightningWeb.API.CredentialController do GET /api/credentials GET /api/credentials?project_id=a1b2c3d4-... POST /api/credentials + GET /api/credentials/a1b2c3d4-... + PUT /api/credentials/a1b2c3d4-... DELETE /api/credentials/a1b2c3d4-... """ use LightningWeb, :controller - alias Lightning.Accounts.User + alias Lightning.Accounts alias Lightning.Credentials + alias Lightning.Credentials.Credential alias Lightning.Policies.Permissions alias Lightning.Policies.ProjectUsers - alias Lightning.Projects - - action_fallback LightningWeb.FallbackController - - # `require_authenticated_api_resource` lets repo-connection tokens through - # this pipeline, and everything here is about credentials, which belong to a - # person. Every action would hand a machine token to a function that only - # takes a user, which raises rather than refusing. Answer it once here. - plug :require_user - - defp require_user(%{assigns: %{current_resource: %User{}}} = conn, _opts), - do: conn - - defp require_user(conn, _opts) do - conn - |> put_status(:forbidden) - |> put_view(LightningWeb.ErrorView) - |> render(:"403") - |> halt() - end + alias Lightning.Scope + alias LightningWeb.API.Errors + alias LightningWeb.API.FallbackController + alias LightningWeb.API.Request + alias LightningWeb.API.Target + alias LightningWeb.Plugs.AuthorizeAction + + action_fallback FallbackController + + # Both index routes need the `credentials:write` grant, and that is all this + # plug checks. + plug AuthorizeAction, + {:credentials, :list_credentials} + when action == :index + + plug AuthorizeAction, + {:credentials, :edit_credential} + when action in [:create, :show, :update] + + plug AuthorizeAction, + {:credentials, :delete_credential} + when action == :delete + + @accepted [ + :id, + :name, + :owner, + :schema, + :credential_bodies, + :project_credentials + ] @doc """ - Lists credentials with optional project filtering. - - This function has two variants: - - With `project_id`: Returns all credentials for a specific project (regardless of owner) - - Without `project_id`: Returns only credentials owned by the authenticated user - - Credential bodies are excluded from responses for security. - - ## Parameters - - - `conn` - The Plug connection struct with the current resource assigned - - `params` - Map containing: - - `project_id` - Project UUID (optional, filters to specific project) + Lists credentials, without their bodies. - ## Returns - - - `200 OK` with list of credentials (bodies excluded) - - `404 Not Found` if project doesn't exist (when project_id provided) - - `403 Forbidden` if user lacks project access (when project_id provided) + With `project_id`, lists every credential linked to that project, whoever + owns it. A project the caller isn't a member of answers 404 `not_found`, as + if it didn't exist. Without `project_id`, lists the caller's own credentials. ## Examples @@ -77,42 +77,38 @@ defmodule LightningWeb.API.CredentialController do """ @spec index(Plug.Conn.t(), map()) :: Plug.Conn.t() def index(conn, %{"project_id" => project_id}) do - current_user = conn.assigns.current_resource - - with project when not is_nil(project) <- Projects.get_project(project_id), - :ok <- - ProjectUsers - |> Permissions.can( - :access_project, - current_user, - project + with {:ok, project} <- + Target.project( + conn.assigns.scope, + project_id, + {:project_users, :list_project_credentials} ) do - # The requested project is already access-checked above, so always show - # it (covers membership at any depth and support-user access). credentials = project |> Credentials.list_credentials() - |> scope_to_visible_projects(current_user, [project.id]) + |> hide_unseen_links(conn.assigns.scope) render(conn, "index.json", credentials: credentials) - else - nil -> - {:error, :not_found} - - {:error, :unauthorized} -> - {:error, :forbidden} end end def index(conn, _params) do current_user = conn.assigns.current_resource - credentials = - current_user - |> Credentials.list_credentials() - |> scope_to_visible_projects(current_user) + with :ok <- + Permissions.permit( + :credentials, + :list_credentials, + conn.assigns.scope, + nil + ) do + credentials = + current_user + |> Credentials.list_credentials() + |> hide_unseen_links(conn.assigns.scope) - render(conn, "index.json", credentials: credentials) + render(conn, "index.json", credentials: credentials) + end end @doc """ @@ -133,8 +129,9 @@ defmodule LightningWeb.API.CredentialController do ## Returns - `201 Created` with credential JSON including body - - `422 Unprocessable Entity` on validation errors - - `403 Forbidden` if user lacks access to specified projects + - `400 Bad Request` (`malformed`) when the body isn't a JSON object + - `422 Unprocessable Entity` on validation errors, with one entry for each + project link the user may not make ## Examples @@ -155,14 +152,24 @@ defmodule LightningWeb.API.CredentialController do ] } """ - @spec create(Plug.Conn.t(), map()) :: Plug.Conn.t() + @spec create(Plug.Conn.t(), map()) :: Plug.Conn.t() | {:error, term()} def create(conn, params) do - current_user = conn.assigns.current_resource + scope = conn.assigns.scope + owner = scope.actor - with {:ok, validated_params} <- - validate_and_authorize_projects(params, current_user), + with :ok <- Request.check_object(conn.body_params), + {link_entries, project_ids} = + check_links(params, [], {:ok, owner}, scope), + :ok <- Errors.refuse(link_entries), {:ok, credential} <- - Credentials.create_credential(validated_params, current_user) do + Credentials.create_credential( + Map.merge(params, %{ + "user_id" => owner.id, + "project_credentials" => + Enum.map(project_ids, &%{"project_id" => &1}) + }), + owner + ) do conn |> put_status(:created) |> render("create.json", credential: credential) @@ -170,162 +177,424 @@ defmodule LightningWeb.API.CredentialController do end @doc """ - Deletes a credential owned by the authenticated user. + Shows a credential, without its body, to a caller who may edit it or who + sees it linked to a project whose credentials it may list. Anyone else hears + 404 `not_found`, as if it didn't exist. - Permanently removes a credential. Only the credential owner can delete it. - Credentials in use by workflows cannot be deleted and will return an error. - - ## Parameters - - - `conn` - The Plug connection struct with the current resource assigned - - `params` - Map containing: - - `id` - Credential UUID (required) + Links to projects whose credentials the caller may not list are left out. + """ + @spec show(Plug.Conn.t(), map()) :: Plug.Conn.t() | {:error, term()} + def show(conn, %{"id" => id}) do + with {:ok, id} <- Request.path_id(id), + {:ok, credential} <- fetch_credential(id), + :ok <- + Permissions.permit( + :credentials, + :see_credential, + conn.assigns.scope, + credential + ) do + render_credential(conn, credential) + end + end - ## Returns + @doc """ + Creates the credential under the path's id, or replaces the name, schema and + `main` body of the one that has it. + + Links are only ever added. A project the body lists gains a link, and a link + the credential already holds stays even if the body leaves it out. The + response leaves out links the caller may not see, as `show/2` does. + + The credential under the path's id is checked before the body: one the caller + may not edit is `id_taken`, and one being deleted is `scheduled_for_deletion`. + A refused body answers one 422 with every error in its keys, its owner and + its links (error-format spec §9.6). The credential's own fields and its body + are checked by `Lightning.Credentials` when it writes. + """ + @spec update(Plug.Conn.t(), map()) :: Plug.Conn.t() | {:error, term()} + def update(conn, %{"id" => id}) do + scope = conn.assigns.scope + body = conn.body_params + + with {:ok, id} <- Request.path_id(id), + {:ok, target} <- update_target(id, scope), + {:ok, naming} <- owner_naming(body["owner"], scope), + {:ok, key_entries} <- Request.check(body, id, @accepted), + {:ok, {owner, project_ids}} <- + check_put_body(body, key_entries, naming, target, scope) do + case target do + :new -> create_under_id(conn, id, body, owner, project_ids) + credential -> replace(conn, credential, body, owner, project_ids) + end + end + end - - `204 No Content` on successful deletion - - `404 Not Found` if credential doesn't exist or invalid UUID - - `403 Forbidden` if user is not the credential owner + defp update_target(id, scope) do + case Credentials.get_credential(id) do + nil -> {:ok, :new} + credential -> editable(credential, scope) + end + end - ## Examples + defp editable(credential, scope) do + case Permissions.permit(:credentials, :edit_credential, scope, credential) do + :ok -> + {:ok, Credentials.preload_links(credential)} - DELETE /api/credentials/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d - """ - @spec delete(Plug.Conn.t(), map()) :: Plug.Conn.t() - def delete(conn, %{"id" => id}) do - current_user = conn.assigns.current_resource + # The id belongs to a credential the caller can't write, which no body + # can change (spec §4). + {:error, :forbidden} -> + {:error, :id_taken} - with :ok <- validate_uuid(id), - credential when not is_nil(credential) <- - Credentials.get_credential(id), - :ok <- validate_credential_ownership(credential, current_user), - {:ok, _} <- Credentials.delete_credential(credential, current_user) do - send_resp(conn, :no_content, "") - else - {:error, :invalid_uuid} -> - {:error, :not_found} + error -> + error + end + end - nil -> - {:error, :not_found} + # A caller who may name an owner has to name one. Anyone else owns the + # credential and may name only themselves. Naming someone else is refused + # before any lookup, so the reply never reveals whether an email has an + # account. + defp owner_naming(owner, scope) do + case Permissions.permit(:credentials, :name_credential_owner, scope, nil) do + :ok -> + {:ok, :anyone} {:error, :forbidden} -> - {:error, :forbidden} + if names_someone_else?(owner, scope.actor), + do: {:error, :forbidden}, + else: {:ok, :self} error -> error end end - defp validate_uuid(id) do - case Ecto.UUID.dump(to_string(id)) do - {:ok, _bin} -> :ok - :error -> {:error, :invalid_uuid} + defp names_someone_else?(email, caller) when is_binary(email), + do: String.downcase(email) != String.downcase(caller.email) + + defp names_someone_else?(_owner, _caller), do: false + + defp check_put_body(body, key_entries, naming, target, scope) do + owner = resolve_owner(naming, body, scope.actor) + held = held_project_ids(target) + + {link_entries, project_ids} = check_links(body, held, owner, scope) + + entries = + key_entries ++ + body_entries(body) ++ + owner_entries(owner) ++ link_entries ++ link_key_entries(body) + + # Every refused owner has an entry, so it is refused with the rest. + with :ok <- Errors.refuse(entries), + {:ok, owner} <- owner, + do: {:ok, {owner, project_ids}} + end + + defp resolve_owner(_naming, %{"owner" => owner}, _caller) + when not is_binary(owner) and not is_nil(owner), + do: {:error, [Errors.entry([:owner], {"is invalid", code: :type})]} + + defp resolve_owner(:self, _body, caller), do: {:ok, caller} + + defp resolve_owner(:anyone, %{"owner" => email}, _caller) + when is_binary(email) do + case Accounts.list_users_by_emails([email]) do + [owner] -> + {:ok, owner} + + _none -> + {:error, [Errors.entry([:owner], Errors.not_visible())]} end end - # A shared credential may be linked to projects in other tenants; rendering - # those would leak their id/name/description across the tenant boundary. Prune - # each credential's project associations to the caller-visible set here so the - # JSON view can render whatever it is handed. - # - # The visible set is the caller's memberships (any depth), matching the - # :access_project check so a caller's own sandbox associations are not - # dropped, plus any `extra_project_ids` already access-checked by the caller - # (e.g. the requested project on the project-scoped endpoint). - defp scope_to_visible_projects( - credentials, - current_user, - extra_project_ids \\ [] - ) do - visible = - current_user - |> Projects.member_project_ids() - |> Enum.concat(extra_project_ids) - |> MapSet.new() + defp resolve_owner(:anyone, _body, _caller), + do: {:error, [Errors.entry([:owner], {"can't be blank", code: :required})]} - Enum.map(credentials, fn credential -> - projects = - filter_accessible_projects( - credential.projects, - visible, - fn project -> project.id end - ) - - project_credentials = - filter_accessible_projects( - credential.project_credentials, - visible, - fn project_credential -> project_credential.project_id end - ) + defp owner_entries({:ok, _owner}), do: [] + defp owner_entries({:error, entries}), do: entries - %{ - credential - | projects: projects, - project_credentials: project_credentials - } - end) + @bodies_message "must be one body named main, holding an object" + + defp body_entries(%{ + "credential_bodies" => [%{"name" => "main", "body" => %{}} = main] + }) + when map_size(main) == 2, + do: [] + + defp body_entries(_body), + do: [Errors.entry([:credential_bodies], {@bodies_message, code: :one_body})] + + # A PUT link holds a project_id and nothing else. + defp link_key_entries(%{"project_credentials" => links}) when is_list(links) do + for {%{} = link, index} <- Enum.with_index(links), + entry <- + Request.unknown_keys(link, [:project_credentials, index], [:project_id]), + do: entry end - defp filter_accessible_projects(assoc, visible, key_fun) when is_list(assoc) do - Enum.filter(assoc, fn record -> MapSet.member?(visible, key_fun.(record)) end) + defp link_key_entries(_body), do: [] + + defp held_project_ids(:new), do: [] + + defp held_project_ids(%Credential{project_credentials: links}), + do: Enum.map(links, & &1.project_id) + + defp check_links(%{"project_credentials" => links}, held, owner, scope) + when is_list(links) do + shaped = + links + |> Enum.with_index() + |> Enum.map(fn {link, index} -> {index, link_project_id(link, index)} end) + + ids = for {_index, {:ok, id}} <- shaped, id not in held, do: id + scopes = Scope.fetch_all(scope, ids) + unreachable = owner_unreachable(owner, ids) + + results = + Enum.map(shaped, fn + {index, {:ok, id}} -> + cond do + id in held -> + :held + + refusal = link_refusal(scopes[id], id in unreachable) -> + {:error, link_entry(index, refusal)} + + true -> + {:ok, id} + end + + {_index, error} -> + error + end) + + {for({:error, entry} <- results, do: entry), + Enum.uniq(for {:ok, id} <- results, do: id)} end - # Leave unloaded associations untouched; the view renders them as []. - defp filter_accessible_projects(assoc, _visible, _key_fun), do: assoc + defp check_links(%{"project_credentials" => _links}, _held, _owner, _scope), + do: {[Errors.entry([:project_credentials], {"is invalid", code: :type})], []} - defp validate_credential_ownership(credential, current_user) do - if credential.user_id == current_user.id do - :ok - else - {:error, :forbidden} + defp check_links(_body, _held, _owner, _scope), do: {[], []} + + defp link_project_id(%{"project_id" => id}, index) when is_binary(id) do + case Request.path_id(id) do + {:ok, id} -> {:ok, id} + {:error, :not_found} -> {:error, link_entry(index, Errors.not_visible())} end end - defp validate_and_authorize_projects(params, current_user) do - # Ensure user_id is set to the current authenticated user - params_with_user = Map.put(params, "user_id", current_user.id) + defp link_project_id(%{"project_id" => id}, index) when not is_nil(id), + do: {:error, link_entry(index, {"is invalid", code: :type})} - project_credentials = Map.get(params, "project_credentials", []) + defp link_project_id(%{}, index), + do: {:error, link_entry(index, {"can't be blank", code: :required})} - if Enum.empty?(project_credentials) do - {:ok, params_with_user} - else - case validate_project_access(project_credentials, current_user) do - :ok -> {:ok, params_with_user} - {:error, _} = error -> error - end + defp link_project_id(_link, index), + do: + {:error, + Errors.entry([:project_credentials, index], {"is invalid", code: :type})} + + defp link_entry(index, error), + do: Errors.entry([:project_credentials, index, :project_id], error) + + # A link's entry codes follow spec §7 item 3 and plan D13. + defp link_refusal(nil, _owner_unreachable?), do: Errors.not_visible() + + defp link_refusal(project_scope, owner_unreachable?) do + case ProjectUsers.authorize( + :create_project_credential, + project_scope, + project_scope.project + ) do + {:error, :no_standing} -> + Errors.not_visible() + + true when owner_unreachable? -> + {"the credential's owner is not a member of this project", + code: :no_access} + + true -> + nil + + _refused -> + {"you may not link a credential to this project", code: :no_access} end end - defp validate_project_access(project_credentials, current_user) do - project_ids = - project_credentials - |> Enum.map(&Map.get(&1, "project_id")) - |> Enum.filter(& &1) - - unauthorized_projects = - Enum.filter(project_ids, fn project_id -> - case Projects.get_project(project_id) do - nil -> - true - - project -> - case ProjectUsers - |> Permissions.can( - :create_project_credential, - current_user, - project - ) do - :ok -> false - _ -> true - end - end - end) + # An owner the body failed to name already has its own entry, so its + # membership isn't checked. + defp owner_unreachable({:error, _entries}, _project_ids), do: [] + + defp owner_unreachable({:ok, owner}, project_ids) do + owner.id + |> Credentials.projects_without_access(Enum.uniq(project_ids)) + |> Enum.map(& &1.id) + end + + defp create_under_id(conn, id, body, owner, project_ids) do + body + |> credential_params() + |> Map.merge(%{ + "user_id" => owner.id, + "project_credentials" => Enum.map(project_ids, &%{"project_id" => &1}) + }) + |> Credentials.create_credential(conn.assigns.current_resource, id: id) + |> case do + {:ok, credential} -> + conn |> put_status(:created) |> render_credential(credential) + + # Someone else created it since the lookup. Rerun the request so the + # caller is checked against that credential. + {:error, %Ecto.Changeset{} = changeset} -> + if Request.id_taken?(changeset), + do: update(conn, %{"id" => id}), + else: {:error, changeset} + + error -> + error + end + end - if Enum.empty?(unauthorized_projects) do - :ok + # Lightning never moves a credential to another owner. The links the + # credential holds go to `update_credential` with their ids, so its + # `cast_assoc` keeps them rather than replacing them. + defp replace(conn, credential, body, owner, project_ids) do + if credential.user_id == owner.id do + links = + Enum.map( + credential.project_credentials, + &%{"id" => &1.id, "project_id" => &1.project_id} + ) ++ Enum.map(project_ids, &%{"project_id" => &1}) + + params = + body |> credential_params() |> Map.put("project_credentials", links) + + with {:ok, credential} <- + Credentials.update_credential( + credential, + params, + conn.assigns.current_resource + ), + do: render_credential(conn, credential) else - {:error, :forbidden} + {:error, :id_taken} end end + + defp credential_params(body) do + body + |> Map.take(["schema"]) + |> Map.merge(%{ + "name" => body["name"], + "credential_bodies" => body["credential_bodies"] + }) + end + + defp fetch_credential(id) do + case Credentials.get_credential(id) do + nil -> {:error, :not_found} + credential -> {:ok, Credentials.preload_links(credential)} + end + end + + defp permitted_ids(scopes, action) do + for {id, project_scope} <- scopes, + ProjectUsers.permitted?(action, project_scope), + into: MapSet.new(), + do: id + end + + defp render_credential(conn, credential) do + [credential] = + [Credentials.preload_links_and_projects(credential)] + |> hide_unseen_links(conn.assigns.scope) + + render(conn, "create.json", credential: credential) + end + + # A credential can be linked to projects the caller has no standing in, + # because its owner lost access or because a list holds someone else's + # credentials. Those links are left out. Naming a project also needs + # `:access_project`, so a service account without `projects:write` sees every + # link but no project names. + defp hide_unseen_links(credentials, scope) do + scopes = + Scope.fetch_all( + scope, + credentials + |> Enum.flat_map(& &1.project_credentials) + |> Enum.map(& &1.project_id) + |> Enum.uniq() + ) + + shown = permitted_ids(scopes, :list_project_credentials) + named = permitted_ids(scopes, :access_project) + + Enum.map(credentials, fn credential -> + %{ + credential + | projects: Enum.filter(credential.projects, &(&1.id in named)), + project_credentials: + Enum.filter( + credential.project_credentials, + &(&1.project_id in shown) + ) + } + end) + end + + @doc """ + Deletes a credential owned by the authenticated user. + + Permanently removes a credential. Only the credential owner can delete it. + + ## Parameters + + - `conn` - The Plug connection struct with the current resource assigned + - `params` - Map containing: + - `id` - Credential UUID (required) + + ## Returns + + - `204 No Content` on successful deletion + - `404 Not Found` if the credential doesn't exist, the id isn't a UUID, or + the caller can't see it + - `403 Forbidden` if the caller can see it but doesn't own it + + ## Examples + + DELETE /api/credentials/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d + """ + @spec delete(Plug.Conn.t(), map()) :: Plug.Conn.t() | {:error, term()} + def delete(conn, %{"id" => id}) do + with {:ok, id} <- Request.path_id(id), + {:ok, credential} <- fetch_credential(id), + :ok <- deletable(credential, conn.assigns.scope), + {:ok, _deleted} <- + Credentials.delete_credential( + credential, + conn.assigns.current_resource + ) do + send_resp(conn, :no_content, "") + end + end + + # Seeing a credential through a link is standing to read it and nothing more + # (spec §15 row 49), so a caller who sees it but may not delete it hears 403. + defp deletable(credential, scope) do + with {:error, :forbidden} <- + Permissions.permit( + :credentials, + :delete_credential, + scope, + credential + ), + :ok <- + Permissions.permit(:credentials, :see_credential, scope, credential), + do: {:error, :forbidden} + end end diff --git a/lib/lightning_web/controllers/api/credential_json.ex b/lib/lightning_web/controllers/api/credential_json.ex index fcb31c298e3..57288a2aac0 100644 --- a/lib/lightning_web/controllers/api/credential_json.ex +++ b/lib/lightning_web/controllers/api/credential_json.ex @@ -20,6 +20,9 @@ defmodule LightningWeb.API.CredentialJSON do } end + # Renders every link and project the credential carries. The controller + # prunes both to what the caller may see, so another tenant's project ids, + # names and descriptions never reach this module. defp credential_data(%Credential{} = credential) do %{ id: credential.id, @@ -54,9 +57,6 @@ defmodule LightningWeb.API.CredentialJSON do } end - # Renders whatever projects the credential carries. The controller is - # responsible for pruning these to the caller-visible set before rendering, so - # that other tenants' project id/name/description are never disclosed here. defp render_projects(projects) when is_list(projects) do Enum.map(projects, &render_project/1) end diff --git a/lib/lightning_web/controllers/api/fallback_controller.ex b/lib/lightning_web/controllers/api/fallback_controller.ex new file mode 100644 index 00000000000..316342a00a7 --- /dev/null +++ b/lib/lightning_web/controllers/api/fallback_controller.ex @@ -0,0 +1,82 @@ +defmodule LightningWeb.API.FallbackController do + @moduledoc """ + Answers an API action's error result with a problem document + (error-format spec §8). + + Each result has its own clause, and there is deliberately no catch-all: a + result with no clause raises, so it is logged as a 500 and gets a clause. + `{:error, :unauthorized}` from `Permissions.can/4` has no clause either, + because a refused permission is a 403 or a 404, never a 401 (plan D9). + """ + alias Lightning.Extensions.Message + alias LightningWeb.API.Codes + alias LightningWeb.API.Errors + alias LightningWeb.API.Problem + alias LightningWeb.Auth + + require Logger + + # action_fallback hands the action's result to init/1, then to call/2. + @doc false + @spec init(term()) :: term() + def init(result), do: result + + @spec call(Plug.Conn.t(), term()) :: Plug.Conn.t() + def call(conn, {:error, %Ecto.Changeset{} = changeset}), + do: Problem.respond(conn, Errors.problem(changeset)) + + def call(conn, {:error, %Problem{} = problem}), + do: Problem.respond(conn, problem) + + def call(conn, {:error, reason}) + when reason in [ + :not_found, + :no_such_project, + :no_standing, + :workflow_deleted + ], + do: Problem.respond(conn, Problem.new(:not_found)) + + def call(conn, {:error, :forbidden}), + do: Problem.respond(conn, Problem.new(:forbidden)) + + def call(conn, {:error, {:insufficient_scope, grant}}) do + problem = + Problem.new(:insufficient_scope, + headers: [Auth.insufficient_scope_challenge(grant)] + ) + + Problem.respond(conn, problem) + end + + def call(conn, {:error, reason}) + when reason in [ + :scheduled_for_deletion, + :being_edited, + :concurrent_edit, + :id_taken, + :in_use, + :ambiguous_name, + :exceeds_limit + ], + do: Problem.respond(conn, Problem.new(reason)) + + def call(conn, {:error, :unknown_project, %Message{}}), + do: call(conn, {:error, :not_found}) + + def call(conn, {:error, reason, %Message{}}), + do: Problem.respond(conn, Problem.new(Codes.for_limiter(reason))) + + def call(conn, {:error, :audit_failed}) do + Logger.error( + "An API action could not write the audit events of a project change" + ) + + Problem.respond(conn, Problem.new(:internal)) + end + + def call(conn, {:error, :snapshot_failed}) do + Logger.error("An API action could not save a workflow snapshot") + Problem.respond(conn, Problem.new(:internal)) + end +end diff --git a/lib/lightning_web/controllers/api/helpers.ex b/lib/lightning_web/controllers/api/helpers.ex index 0c21dae60e6..b49340a8d6d 100644 --- a/lib/lightning_web/controllers/api/helpers.ex +++ b/lib/lightning_web/controllers/api/helpers.ex @@ -4,50 +4,54 @@ defmodule LightningWeb.API.Helpers do """ alias LightningWeb.Router.Helpers, as: Routes - def pagination_links(conn, page) do + @doc """ + The page's links, `self` included, carrying the filters the controller + applied, as cast, and the page and page size Scrivener applied, and nothing + else from the request (error-format spec §15 row 52). A filter the path + already names, such as a nested route's `project_id`, is left out. + """ + def pagination_links(conn, page, filters) do + query = + filters + |> Map.new(fn {key, value} -> + {Atom.to_string(key), query_value(value)} + end) + |> Map.drop(Map.keys(conn.path_params)) + %{ - first: pagination_link(conn, page, :first), - last: pagination_link(conn, page, :last), - next: pagination_link(conn, page, :next), - prev: pagination_link(conn, page, :prev) + self: page_url(conn, query, page, page.page_number), + first: page_url(conn, query, page, 1), + last: page_url(conn, query, page, page.total_pages), + next: + if(page.page_number < page.total_pages, + do: page_url(conn, query, page, page.page_number + 1) + ), + prev: + if(page.page_number > 1, + do: page_url(conn, query, page, page.page_number - 1) + ) } end - def pagination_link(conn, _page, :first) do - url_for(conn, conn.query_params |> Map.put("page", 1)) + defp page_url(conn, query, page, number) do + query + |> Map.merge(%{"page" => number, "page_size" => page.page_size}) + |> url_with(conn) end - def pagination_link(conn, page, :last) do - url_for(conn, conn.query_params |> Map.put("page", page.total_pages)) - end + defp query_value(%DateTime{} = at), do: DateTime.to_iso8601(at) + defp query_value(values) when is_list(values), do: Enum.join(values, ",") + defp query_value(value), do: value - def pagination_link(conn, page, :next) do - if page.page_number < page.total_pages do - url_for( - conn, - conn.query_params - |> Map.put("page", page.page_number + 1) - ) - end - end - - def pagination_link(conn, page, :prev) do - if page.page_number > 1 do - url_for( - conn, - conn.query_params - |> Map.put( - "page", - page.page_number - 1 - ) - ) - end - end + @doc """ + The request's URL without its query string, a single resource's `self`. + """ + def self_url(conn), do: Routes.url(conn) <> conn.request_path - def url_for(conn, params \\ %{}) do + defp url_with(query, conn) do %URI{ - URI.new!(Routes.url(conn) <> conn.request_path) - | query: conn.query_params |> Map.merge(params) |> Plug.Conn.Query.encode() + URI.new!(self_url(conn)) + | query: Plug.Conn.Query.encode(query) } |> URI.to_string() end diff --git a/lib/lightning_web/controllers/api/job_controller.ex b/lib/lightning_web/controllers/api/job_controller.ex index f4a8109e81e..fe9875d29d8 100644 --- a/lib/lightning_web/controllers/api/job_controller.ex +++ b/lib/lightning_web/controllers/api/job_controller.ex @@ -51,109 +51,67 @@ defmodule LightningWeb.API.JobController do use LightningWeb, :controller alias Lightning.Jobs - alias Lightning.Policies.Permissions - alias Lightning.Policies.ProjectUsers - alias Lightning.Workflows + alias LightningWeb.API.FallbackController + alias LightningWeb.API.Filters + alias LightningWeb.API.Request + alias LightningWeb.API.Target - action_fallback LightningWeb.FallbackController + action_fallback FallbackController - @doc """ - Lists jobs with optional project filtering. - - This function has two variants: - - With `project_id`: Returns jobs for a specific project - - Without `project_id`: Returns jobs across all accessible projects - - Returns a paginated list of jobs. - - ## Parameters - - - `conn` - The Plug connection struct with the current resource assigned - - `params` - Map containing: - - `project_id` - Project UUID (optional, filters to specific project) - - `page` - Page number (optional, default: 1) - - `page_size` - Items per page (optional, default: 10, at most 100) + plug LightningWeb.Plugs.AuthorizeAction, + {:project_users, :access_project} - ## Returns - - - Renders JSON with paginated list of jobs - - `404 Not Found` if project doesn't exist (when project_id provided) - - `403 Forbidden` if user lacks project access (when project_id provided) + @doc """ + Lists the jobs of the projects the caller may access, or of one project named + by the nested route or the `project_id` parameter. ## Examples - # All jobs accessible to user - GET /api/jobs GET /api/jobs?page=3&page_size=25 - - # Jobs for specific project GET /api/jobs?project_id=a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d - GET /api/jobs?project_id=a1b2c3d4-...&page=2&page_size=50 + GET /api/projects/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d/jobs """ - @spec index(Plug.Conn.t(), map()) :: Plug.Conn.t() + @spec index(Plug.Conn.t(), map()) :: Plug.Conn.t() | {:error, term()} def index(conn, %{"project_id" => project_id} = params) do - pagination_attrs = Map.take(params, ["page_size", "page"]) - - with project <- Lightning.Projects.get_project(project_id), - :ok <- - ProjectUsers - |> Permissions.can( - :access_project, - conn.assigns.current_resource, - project + with {:ok, project} <- + Target.project( + conn.assigns.scope, + project_id, + {:project_users, :access_project} ) do - page = - Jobs.jobs_for_project_query(project) - |> Lightning.Repo.paginate(pagination_attrs) - - render(conn, "index.json", page: page, conn: conn) + list(project, conn, params, %{project_id: project.id}) end end def index(conn, params) do - pagination_attrs = Map.take(params, ["page_size", "page"]) - - page = - Workflows.Query.jobs_for(conn.assigns.current_resource) - |> Lightning.Repo.paginate(pagination_attrs) - - render(conn, "index.json", page: page, conn: conn) + list(conn.assigns.current_resource, conn, params, %{}) end @doc """ - Retrieves a specific job by ID. - - Returns detailed information about a single job including its body, adaptor, - and workflow association. Access is granted if the user has access to the - job's parent project. - - ## Parameters - - - `conn` - The Plug connection struct with the current resource assigned - - `params` - Map containing: - - `id` - Job UUID (required) - - ## Returns - - - `200 OK` with job JSON on success - - `404 Not Found` if job doesn't exist - - `403 Forbidden` if user lacks project access + Shows a job to a caller who may access its project. A job the caller has no + standing on answers 404 `not_found`, as if it didn't exist. ## Examples GET /api/jobs/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d """ - @spec show(Plug.Conn.t(), map()) :: Plug.Conn.t() + @spec show(Plug.Conn.t(), map()) :: Plug.Conn.t() | {:error, term()} def show(conn, %{"id" => id}) do - with {:ok, job} <- Jobs.get_job(id, include: [workflow: :project]), - :ok <- - ProjectUsers - |> Permissions.can( - :access_project, - conn.assigns.current_resource, - job.workflow.project + with {:ok, id} <- Request.path_id(id), + {:ok, job} <- Jobs.get_job(id, include: [:workflow]), + {:ok, _project} <- + Target.project( + conn.assigns.scope, + job.workflow.project_id, + {:project_users, :access_project} ) do render(conn, "show.json", job: job, conn: conn) end end + + defp list(project_or_user, conn, params, filters) do + page = Jobs.paginate_jobs_for(project_or_user, Filters.page(params)) + + render(conn, "index.json", page: page, filters: filters, conn: conn) + end end diff --git a/lib/lightning_web/controllers/api/job_json.ex b/lib/lightning_web/controllers/api/job_json.ex index 0ed1895676e..f9bfcd36e5e 100644 --- a/lib/lightning_web/controllers/api/job_json.ex +++ b/lib/lightning_web/controllers/api/job_json.ex @@ -6,15 +6,11 @@ defmodule LightningWeb.API.JobJSON do @fields ~w(name)a - def render("index.json", %{page: page, conn: conn}) do + def render("index.json", %{page: page, filters: filters, conn: conn}) do %{ data: Enum.map(page.entries, &resource(conn, &1)), included: [], - links: - %{ - self: url_for(conn) - } - |> Map.merge(pagination_links(conn, page)) + links: pagination_links(conn, page, filters) } end @@ -23,7 +19,7 @@ defmodule LightningWeb.API.JobJSON do data: resource(conn, job), included: [], links: %{ - self: url_for(conn) + self: self_url(conn) } } end diff --git a/lib/lightning_web/controllers/api/log_lines_controller.ex b/lib/lightning_web/controllers/api/log_lines_controller.ex index 4b44a17359e..8606a73286a 100644 --- a/lib/lightning_web/controllers/api/log_lines_controller.ex +++ b/lib/lightning_web/controllers/api/log_lines_controller.ex @@ -26,41 +26,34 @@ defmodule LightningWeb.API.LogLinesController do use LightningWeb, :controller alias Lightning.Invocation + alias LightningWeb.API.Errors + alias LightningWeb.API.FallbackController + alias LightningWeb.API.Filters - action_fallback LightningWeb.FallbackController + action_fallback FallbackController - @doc """ - Lists log lines with pagination and filtering. - - Returns a paginated list of log lines filtered by the query parameters. - Validates datetime parameters before applying filters and enforces - authorization based on the current resource (user or API token). - - ## Parameters - - - `conn` - The Plug connection struct with the current resource assigned - - `params` - Map of query parameters for filtering and pagination + plug LightningWeb.Plugs.AuthorizeAction, + {:project_users, :access_project} - ## Returns - - - Renders JSON with paginated log lines on success - - Returns error via fallback controller if datetime validation fails + @doc """ + Lists the log lines of the projects the caller may access, narrowed by the + query's filters. Every bad filter is reported in one 400 `invalid_parameter` + (error-format spec §9.4). """ - @spec index(Plug.Conn.t(), map()) :: Plug.Conn.t() + @spec index(Plug.Conn.t(), map()) :: Plug.Conn.t() | {:error, term()} def index(conn, params) do - with :ok <- - Invocation.Query.validate_datetime_params(params, [ - "timestamp_after", - "timestamp_before" - ]) do - pagination_attrs = Map.take(params, ["page_size", "page"]) - + with {:ok, filters} <- + params + |> Filters.log_lines() + |> Errors.refuse_parameters() do page = - Invocation.Query.log_lines_for(conn.assigns.current_resource) - |> Invocation.Query.filter_log_lines(params) - |> Lightning.Repo.paginate(pagination_attrs) + Invocation.paginate_log_lines_for( + conn.assigns.current_resource, + filters, + Filters.page(params) + ) - render(conn, "index.json", page: page, conn: conn) + render(conn, "index.json", page: page, filters: filters, conn: conn) end end end diff --git a/lib/lightning_web/controllers/api/log_lines_json.ex b/lib/lightning_web/controllers/api/log_lines_json.ex index 6b4110f864c..cd6746a731b 100644 --- a/lib/lightning_web/controllers/api/log_lines_json.ex +++ b/lib/lightning_web/controllers/api/log_lines_json.ex @@ -5,7 +5,7 @@ defmodule LightningWeb.API.LogLinesJSON do @fields ~w(source level message timestamp step_id run_id)a - def render("index.json", %{page: page, conn: conn}) do + def render("index.json", %{page: page, filters: filters, conn: conn}) do %{ data: Enum.map(page.entries, &resource(conn, &1)), included: [], @@ -15,11 +15,7 @@ defmodule LightningWeb.API.LogLinesJSON do page_number: page.page_number, page_size: page.page_size }, - links: - %{ - self: url_for(conn) - } - |> Map.merge(pagination_links(conn, page)) + links: pagination_links(conn, page, filters) } end diff --git a/lib/lightning_web/controllers/api/project_controller.ex b/lib/lightning_web/controllers/api/project_controller.ex index f39db4da342..96e29e7ca90 100644 --- a/lib/lightning_web/controllers/api/project_controller.ex +++ b/lib/lightning_web/controllers/api/project_controller.ex @@ -2,8 +2,8 @@ defmodule LightningWeb.API.ProjectController do @moduledoc """ API controller for project management. - Provides read access to projects for authenticated users and API tokens. - Users can list projects they have access to and retrieve individual project details. + Lists, shows, creates and replaces projects for authenticated users and API + tokens. ## Query Parameters (index) @@ -33,11 +33,36 @@ defmodule LightningWeb.API.ProjectController do """ use LightningWeb, :controller - alias Lightning.Policies.Permissions - alias Lightning.Policies.ProjectUsers + import Ecto.Changeset + + alias Lightning.Accounts alias Lightning.Projects + alias Lightning.Projects.Project + alias Lightning.Projects.ProjectUser + alias LightningWeb.API.Errors + alias LightningWeb.API.FallbackController + alias LightningWeb.API.Filters + alias LightningWeb.API.Request + alias LightningWeb.API.Target + alias LightningWeb.Plugs.AuthorizeAction + + action_fallback FallbackController + + plug AuthorizeAction, + {:project_users, :access_project} + when action in [:index, :show] + + plug AuthorizeAction, + {:project_users, :edit_project} + when action == :update - action_fallback LightningWeb.FallbackController + @accepted [:id, :name, :description, :members, :notify] + @request_types %{members: {:array, :map}, notify: :boolean} + @member_types %{email: :string, role: ProjectUser.RolesEnum} + + # The project changeset reports member errors under `owner` and + # `project_users`. This API calls them `members`. + @renames %{owner: :members, project_users: :members} @doc """ Lists all projects accessible to the authenticated user. @@ -61,32 +86,22 @@ defmodule LightningWeb.API.ProjectController do """ @spec index(Plug.Conn.t(), map()) :: Plug.Conn.t() def index(conn, params) do - pagination_attrs = Map.take(params, ["page_size", "page"]) - page = - Projects.projects_for_user_query(conn.assigns.current_resource) - |> Lightning.Repo.paginate(pagination_attrs) + Projects.paginate_projects_for_user( + conn.assigns.current_resource, + Filters.page(params) + ) render(conn, "index.json", page: page, conn: conn) end @doc """ - Retrieves a specific project by ID. + Shows a project to a caller who may access it. - Returns detailed information about a single project if the authenticated - user has access to it. - - ## Parameters - - - `conn` - The Plug connection struct with the current resource assigned - - `params` - Map containing: - - `id` - Project UUID (required) - - ## Returns - - - `200 OK` with project JSON on success - - `404 Not Found` if project doesn't exist - - `403 Forbidden` if user lacks access to the project + A project the caller isn't a member of answers 404 `not_found`, as if it + didn't exist. A member who hasn't met its MFA requirement, or a token without + `projects:write`, gets 403. A project scheduled for deletion answers 409 + `scheduled_for_deletion` to a caller who may see it. ## Examples @@ -94,19 +109,185 @@ defmodule LightningWeb.API.ProjectController do """ @spec show(Plug.Conn.t(), map()) :: Plug.Conn.t() def show(conn, %{"id" => id}) do - with %Lightning.Projects.Project{} = project <- - Projects.get_project(id), - :ok <- - ProjectUsers - |> Permissions.can( - :access_project, - conn.assigns.current_resource, - project + with {:ok, project} <- + Target.project( + conn.assigns.scope, + id, + {:project_users, :access_project} + ) do + render_project(conn, project) + end + end + + @doc """ + Creates the project under the path's id, or replaces the name and description + of the one that has it. + + `members` and `notify` only apply when the project is created. A refused body + answers one 422 with every error in it, the body's keys, its members and the + project's own fields together (error-format spec §9.6). + """ + @spec update(Plug.Conn.t(), map()) :: Plug.Conn.t() | {:error, term()} + def update(conn, %{"id" => id}) do + body = conn.body_params + + with {:ok, id} <- Request.path_id(id), + {:ok, target} <- + Target.project_or_new( + conn.assigns.scope, + id, + {:project_users, :edit_project} + ), + {:ok, body_entries} <- Request.check(body, id, @accepted) do + request = + {%{notify: true}, @request_types} + |> cast(body, Map.keys(@request_types)) + |> validate_required(:notify) + + fields = %{name: body["name"], description: body["description"]} + entries = body_entries ++ Errors.from_changeset(request) + + case target do + %Project{__meta__: %{state: :built}} -> + create(conn, Map.put(fields, :id, id), request, entries) + + project -> + replace(conn, project, fields, entries) + end + end + end + + defp replace(conn, project, fields, entries) do + changeset = Project.changeset(project, fields) + + with :ok <- + Errors.refuse(entries ++ Errors.from_changeset(changeset, @renames)), + {:ok, project} <- + Projects.update_project( + project, + fields, + conn.assigns.current_resource ) do - render(conn, "show.json", project: project, conn: conn) + render_project(conn, project) else - nil -> {:error, :not_found} - error -> error + {:error, %Ecto.Changeset{} = changeset} -> + {:error, Errors.problem(changeset, @renames)} + + error -> + error end end + + defp create(conn, fields, request, entries) do + {member_entries, fields, changeset} = + case members(request) do + {:ok, project_users} -> + fields = Map.put(fields, :project_users, project_users) + {[], fields, Project.project_with_users_changeset(%Project{}, fields)} + + # Without its members a new project can't be checked for an owner, so + # only its own fields are. + {:error, member_entries} -> + {member_entries, fields, Project.changeset(%Project{}, fields)} + end + + with :ok <- + Errors.refuse( + entries ++ + member_entries ++ Errors.from_changeset(changeset, @renames) + ) do + insert(conn, fields, notify: get_field(request, :notify)) + end + end + + # A members list of the wrong type is already among the request's entries. + defp members(request) do + if Keyword.has_key?(request.errors, :members), + do: {:error, []}, + else: resolve_members(get_field(request, :members) || []) + end + + defp insert(conn, attrs, opts) do + case Projects.create_project(attrs, conn.assigns.current_resource, opts) do + {:ok, project} -> + conn |> put_status(:created) |> render_project(project) + + # Someone else created it since the lookup. Rerun the request so the + # caller is checked against that project. + {:error, changeset} -> + if Request.id_taken?(changeset), + do: update(conn, %{"id" => attrs.id}), + else: {:error, Errors.problem(changeset, @renames)} + end + end + + # Resolves each member's email to its user. Errors point at the member's + # position in the request, which the project changeset doesn't keep + # (error-format spec §7 items 3 and 4). + defp resolve_members(members) do + changesets = + Enum.map(members, fn member -> + {%{}, @member_types} + |> cast(member, Map.keys(@member_types)) + |> validate_required(Map.keys(@member_types)) + end) + + user_ids = + for( + %{valid?: true} = changeset <- changesets, + do: get_field(changeset, :email) + ) + |> Accounts.list_users_by_emails() + |> Map.new(&{String.downcase(&1.email), &1.id}) + + {project_users, entries, _firsts} = + changesets + |> Enum.with_index() + |> Enum.reduce({[], [], %{}}, fn {changeset, index}, acc -> + resolve_member(changeset, [:members, index], user_ids, acc) + end) + + if entries == [], + do: {:ok, Enum.reverse(project_users)}, + else: {:error, Enum.reverse(entries)} + end + + defp resolve_member(%{valid?: false} = changeset, path, _user_ids, acc) do + {project_users, entries, firsts} = acc + + member_entries = + for {field, error} <- changeset.errors, + do: Errors.entry(path ++ [field], error) + + {project_users, member_entries ++ entries, firsts} + end + + defp resolve_member(changeset, path, user_ids, acc) do + {project_users, entries, firsts} = acc + + refuse_email = + &{project_users, [Errors.entry(path ++ [:email], &1) | entries], firsts} + + case Map.fetch(user_ids, String.downcase(get_field(changeset, :email))) do + :error -> + refuse_email.(Errors.not_visible()) + + {:ok, user_id} when is_map_key(firsts, user_id) -> + refuse_email.( + {"names the same user as " <> Errors.pointer(firsts[user_id]), + code: :duplicate} + ) + + {:ok, user_id} -> + project_user = %{user_id: user_id, role: get_field(changeset, :role)} + {[project_user | project_users], entries, Map.put(firsts, user_id, path)} + end + end + + defp render_project(conn, project) do + render(conn, "show.json", + project: Projects.get_project_with_users!(project.id), + conn: conn + ) + end end diff --git a/lib/lightning_web/controllers/api/project_json.ex b/lib/lightning_web/controllers/api/project_json.ex index 9874b937dd9..82dbd68e38d 100644 --- a/lib/lightning_web/controllers/api/project_json.ex +++ b/lib/lightning_web/controllers/api/project_json.ex @@ -11,24 +11,29 @@ defmodule LightningWeb.API.ProjectJSON do %{ data: Enum.map(page.entries, &resource(conn, &1)), included: [], - links: - %{ - self: url_for(conn) - } - |> Map.merge(pagination_links(conn, page)) + links: pagination_links(conn, page, %{}) } end def render("show.json", %{project: project, conn: conn}) do %{ - data: resource(conn, project), + data: + put_in( + resource(conn, project), + [:attributes, :members], + members(project) + ), included: [], links: %{ - self: url_for(conn) + self: self_url(conn) } } end + defp members(project) do + Enum.map(project.project_users, &%{email: &1.user.email, role: &1.role}) + end + defp resource(conn, project) do %{ type: "projects", diff --git a/lib/lightning_web/controllers/api/provisioning_controller.ex b/lib/lightning_web/controllers/api/provisioning_controller.ex index 556e1622e5e..dbf1eb1c41f 100644 --- a/lib/lightning_web/controllers/api/provisioning_controller.ex +++ b/lib/lightning_web/controllers/api/provisioning_controller.ex @@ -13,15 +13,27 @@ defmodule LightningWeb.API.ProvisioningController do """ use LightningWeb, :controller - alias Lightning.Policies.Permissions - alias Lightning.Policies.Provisioning + alias Ecto.Changeset + alias Lightning.Extensions.Message alias Lightning.Projects - alias Lightning.Projects.Project alias Lightning.Projects.Provisioner alias Lightning.Workflows alias Lightning.WorkflowVersions + alias LightningWeb.API.Errors + alias LightningWeb.API.FallbackController + alias LightningWeb.API.Problem + alias LightningWeb.API.Request + alias LightningWeb.API.Target - action_fallback(LightningWeb.FallbackController) + action_fallback(FallbackController) + + plug LightningWeb.Plugs.AuthorizeAction, + {:provisioning, :provision_project} + when action == :create + + plug LightningWeb.Plugs.AuthorizeAction, + {:provisioning, :describe_project} + when action in [:show, :show_yaml] @doc """ Creates or updates a project based on a JSON payload. @@ -38,8 +50,16 @@ defmodule LightningWeb.API.ProvisioningController do ## Returns - `201 Created` with project JSON on success - - `422 Unprocessable Entity` with changeset errors on validation failure - - `403 Forbidden` if user lacks provisioning permissions + - `404 Not Found` if the body names a project the caller has no standing on + - `403 Forbidden` if the caller lacks the role or grant to provision it, or + to create a project + - `409 Conflict` to a caller who may provision the project, if it is + scheduled for deletion, or if a value that must be unique is taken + - `422 Unprocessable Content` when the document is not valid + - `402 Payment Required` when a plan limit is reached + - `503 Service Unavailable` while the adaptor catalogue is still loading + + Every error is a problem document. ## Examples @@ -58,47 +78,34 @@ defmodule LightningWeb.API.ProvisioningController do "workflows": [...] } """ - @spec create(Plug.Conn.t(), map()) :: Plug.Conn.t() + @spec create(Plug.Conn.t(), map()) :: + Plug.Conn.t() | {:error, term()} | {:error, term(), Message.t()} def create(conn, params) do - with project <- get_or_build_project(params), - :ok <- - Permissions.can( - Provisioning, - :provision_project, - conn.assigns.current_resource, - project - ) do - case Provisioner.import_document( + with {:ok, project} <- provision_target(conn.assigns.scope, params["id"]), + {:ok, project} <- + Provisioner.import_document( project, conn.assigns.current_resource, params ) do - {:ok, project} -> - conn - |> put_status(:created) - |> put_resp_header("location", ~p"/api/provision/#{project.id}") - |> render("create.json", project: project) - - {:error, %Ecto.Changeset{} = changeset} -> - conn - |> put_status(:unprocessable_entity) - |> render("error.json", changeset: changeset) - - {:error, error} -> - conn - |> put_status(:forbidden) - |> put_view(LightningWeb.ErrorView) - |> render(:"403", - error: - case error do - %Lightning.Extensions.Message{text: text} -> text - _ -> error - end - ) - end + conn + |> put_status(:created) + |> put_resp_header("location", ~p"/api/provision/#{project.id}") + |> render("create.json", project: project) end end + # The project the body names is the target, as if it were in the path (spec + # §4). + defp provision_target(scope, nil), + do: Target.project_or_new(scope, nil, {:provisioning, :provision_project}) + + defp provision_target(scope, id) do + with {:ok, id} <- Request.path_id(id), + do: + Target.project_or_new(scope, id, {:provisioning, :provision_project}) + end + @doc """ Returns a project state as JSON with UUIDs for idempotent deployments. @@ -111,13 +118,19 @@ defmodule LightningWeb.API.ProvisioningController do - `conn` - The Plug connection struct with the current resource assigned - `params` - Map containing: - `id` - Project UUID (required) - - `snapshots` - Whether to include workflow snapshots (optional) + - `snapshots` - A list of snapshot ids, each a UUID, to return in place + of the workflows' current state (optional) ## Returns - `200 OK` with project JSON on success - - `404 Not Found` if project doesn't exist - - `403 Forbidden` if user lacks describe permissions + - `400 Bad Request` if `snapshots` isn't a list of UUIDs + - `404 Not Found` if the project doesn't exist or the caller has no standing + on it + - `403 Forbidden` if the caller lacks the grant, or hasn't met the + project's MFA requirement + - `409 Conflict` to a caller who may see the project, if it is scheduled + for deletion ## Examples @@ -125,25 +138,19 @@ defmodule LightningWeb.API.ProvisioningController do GET /api/provision/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d # Get project state with snapshots - GET /api/provision/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d?snapshots=true + GET /api/provision/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d?snapshots[]=6f1e2d3c-4b5a-4968-8776-5a4b3c2d1e0f """ - @spec show(Plug.Conn.t(), map()) :: Plug.Conn.t() - def show(conn, params) do - with project = %Project{} <- - Projects.get_project(params["id"]) || {:error, :not_found}, - :ok <- - Permissions.can( - Provisioning, - :describe_project, - conn.assigns.current_resource, - project - ), - :ok <- ensure_workflows_have_versions(project), - project <- - Provisioner.preload_dependencies(project, params["snapshots"]) do + @spec show(Plug.Conn.t(), map()) :: Plug.Conn.t() | {:error, term()} + def show(conn, %{"id" => id} = params) do + with {:ok, project} <- describe_target(conn.assigns.scope, id), + {:ok, snapshot_ids} <- fetch_snapshot_ids(params) do + ensure_workflows_have_versions(project) + conn |> put_status(:ok) - |> render("create.json", project: project) + |> render("create.json", + project: Provisioner.preload_dependencies(project, snapshot_ids) + ) end end @@ -168,13 +175,20 @@ defmodule LightningWeb.API.ProvisioningController do - `conn` - The Plug connection struct with the current resource assigned - `params` - Map containing: - `id` - Project UUID (required) - - `snapshots` - Whether to include workflow snapshots (optional) + - `snapshots` - A list of snapshot ids, each a UUID, to return in place + of the workflows' current state (optional) ## Returns - `200 OK` with YAML content on success - - `404 Not Found` if project doesn't exist - - `403 Forbidden` if user lacks describe permissions + - `400 Bad Request` without an `id` + - `400 Bad Request` if `snapshots` isn't a list of UUIDs + - `404 Not Found` if the project doesn't exist or the caller has no standing + on it + - `403 Forbidden` if the caller lacks the grant, or hasn't met the + project's MFA requirement + - `409 Conflict` to a caller who may see the project, if it is scheduled + for deletion ## Examples @@ -182,21 +196,14 @@ defmodule LightningWeb.API.ProvisioningController do GET /api/provision/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d.yaml # Get project state as YAML with snapshots - GET /api/provision/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d.yaml?snapshots=true + GET /api/provision/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d.yaml?snapshots[]=6f1e2d3c-4b5a-4968-8776-5a4b3c2d1e0f """ - @spec show_yaml(Plug.Conn.t(), map()) :: Plug.Conn.t() + @spec show_yaml(Plug.Conn.t(), map()) :: Plug.Conn.t() | {:error, term()} def show_yaml(conn, %{"id" => id} = params) do - with %Projects.Project{} = project <- - Projects.get_project(id) || {:error, :not_found}, - :ok <- - Permissions.can( - Provisioning, - :describe_project, - conn.assigns.current_resource, - project - ), + with {:ok, project} <- describe_target(conn.assigns.scope, id), + {:ok, snapshot_ids} <- fetch_snapshot_ids(params), {:ok, yaml} <- - Projects.export_project(:yaml, id, params["snapshots"], :v1) do + Projects.export_project(:yaml, project.id, snapshot_ids, :v1) do conn |> put_resp_content_type("text/yaml") |> put_root_layout(false) @@ -204,11 +211,28 @@ defmodule LightningWeb.API.ProvisioningController do end end - defp get_or_build_project(params) do - params - |> case do - %{"id" => id} -> Projects.get_project(id) || %Project{} - _ -> %Project{} + def show_yaml(_conn, _params) do + {:error, + Problem.new(:invalid_parameter, + errors: [ + Errors.entry( + {:parameter, "id"}, + {"can't be blank", validation: :required} + ) + ] + )} + end + + defp fetch_snapshot_ids(params) do + with {:ok, filters} <- + {%{}, %{snapshots: {:array, Ecto.UUID}}} + |> Changeset.cast(params, [:snapshots]) + |> Changeset.apply_action(:validate) + |> Errors.refuse_parameters() do + {:ok, filters[:snapshots]} end end + + defp describe_target(scope, id), + do: Target.project(scope, id, {:provisioning, :describe_project}) end diff --git a/lib/lightning_web/controllers/api/provisioning_json.ex b/lib/lightning_web/controllers/api/provisioning_json.ex index 2e2c1f951f5..e809d54f167 100644 --- a/lib/lightning_web/controllers/api/provisioning_json.ex +++ b/lib/lightning_web/controllers/api/provisioning_json.ex @@ -1,9 +1,6 @@ defmodule LightningWeb.API.ProvisioningJSON do @moduledoc false - import LightningWeb.CoreComponents, only: [translate_error: 1] - import Ecto.Changeset - alias Lightning.Channels.Channel alias Lightning.Channels.ChannelAuthMethod alias Lightning.Collections.Collection @@ -20,10 +17,6 @@ defmodule LightningWeb.API.ProvisioningJSON do %{"data" => as_json(project)} end - def error(%{changeset: %Ecto.Changeset{} = changeset}) do - %{errors: traverse_provision_errors(changeset)} - end - def as_json(%Project{} = project) do Ecto.embedded_dump(project, :json) |> Map.put( @@ -161,122 +154,6 @@ defmodule LightningWeb.API.ProvisioningJSON do Map.reject(map, fn {_, v} -> is_nil(v) end) end - # inspired by: https://github.com/elixir-ecto/ecto/blob/v3.11.2/lib/ecto/changeset.ex#L4037 - defp traverse_provision_errors(%Ecto.Changeset{errors: errors} = changeset) do - errors - |> Enum.reverse() - |> merge_keyword_keys() - |> merge_related_keys(changeset) - end - - defp merge_keyword_keys(keyword_list) do - Enum.reduce(keyword_list, %{}, fn {key, val}, acc -> - val = translate_error(val) - Map.update(acc, key, [val], &[val | &1]) - end) - end - - defp merge_related_keys( - map, - %Ecto.Changeset{changes: changes, data: %schema_module{}} = changeset - ) do - fields = - schema_module.__schema__(:associations) ++ - schema_module.__schema__(:embeds) - - Enum.reduce(fields, map, fn - :workflows, acc -> - traverse_field_errors(acc, changes, :workflows, fn workflow_changeset -> - get_field(workflow_changeset, :name) - end) - - :triggers, acc -> - traverse_field_errors(acc, changes, :triggers, fn trigger_changeset -> - get_field(trigger_changeset, :type) - end) - - :jobs, acc -> - traverse_field_errors(acc, changes, :jobs, fn job_changeset -> - get_field(job_changeset, :name) - end) - - :edges, acc -> - traverse_field_errors(acc, changes, :edges, fn edge_changeset -> - source_trigger_id = get_field(edge_changeset, :source_trigger_id) - source_job_id = get_field(edge_changeset, :source_job_id) - target_job_id = get_field(edge_changeset, :target_job_id) - jobs = get_assoc(changeset, :jobs, :struct) - triggers = get_assoc(changeset, :triggers, :struct) - - source_name = - case {source_trigger_id, source_job_id} do - {nil, nil} -> - nil - - {nil, job_id} -> - job = find_item_by_id(jobs, job_id) - job && job.name - - {trigger_id, nil} -> - trigger = find_item_by_id(triggers, trigger_id) - trigger && trigger.type - end - - target_name = - if target_job_id do - job = find_item_by_id(jobs, target_job_id) - job && job.name - end - - "#{source_name}->#{target_name}" - end) - - field, acc -> - traverse_field_errors(acc, changes, field, fn _ -> field end) - end) - end - - defp traverse_field_errors(acc, changes, field, child_name_func) - when is_function(child_name_func, 1) do - changesets = - case Map.get(changes, field) do - %{} = change -> - [change] - - changes -> - changes - end - - if changesets do - child = traverse_nested_changesets(changesets, child_name_func) - - if child == %{} do - acc - else - Map.put(acc, field, child) - end - else - acc - end - end - - defp traverse_nested_changesets(changesets, child_name_func) do - Enum.reduce(changesets, %{}, fn changeset, acc -> - child = traverse_provision_errors(changeset) - - if child == %{} do - acc - else - child_name = changeset |> child_name_func.() |> to_string() - Map.put(acc, child_name, child) - end - end) - end - - defp find_item_by_id(items, id) do - Enum.find(items, fn item -> item.id == id end) - end - # A path written before the naming rules would fail validation when this # document is deployed into another project and take the whole run with it, # the same guard `ExportUtils` and `MergeProjects` apply. diff --git a/lib/lightning_web/controllers/api/registration_controller.ex b/lib/lightning_web/controllers/api/registration_controller.ex index 2c1ebf5fdf7..9ca8bca8e96 100644 --- a/lib/lightning_web/controllers/api/registration_controller.ex +++ b/lib/lightning_web/controllers/api/registration_controller.ex @@ -1,11 +1,13 @@ defmodule LightningWeb.API.RegistrationController do use LightningWeb, :controller + alias LightningWeb.API.Problem + def create(conn, _params) do Lightning.Config.get_extension_mod(:registration_controller) |> case do nil -> - conn |> put_status(501) |> json(%{error: "Not Implemented"}) + Problem.respond(conn, Problem.new(:not_implemented)) controller -> controller.call(conn, :create) diff --git a/lib/lightning_web/controllers/api/run_controller.ex b/lib/lightning_web/controllers/api/run_controller.ex index 80bb2789721..7fcd873f5e3 100644 --- a/lib/lightning_web/controllers/api/run_controller.ex +++ b/lib/lightning_web/controllers/api/run_controller.ex @@ -52,141 +52,89 @@ defmodule LightningWeb.API.RunController do """ use LightningWeb, :controller - alias Lightning.Invocation - alias Lightning.Policies.Permissions - alias Lightning.Policies.ProjectUsers alias Lightning.Runs + alias LightningWeb.API.Errors + alias LightningWeb.API.FallbackController + alias LightningWeb.API.Filters + alias LightningWeb.API.Request + alias LightningWeb.API.Target - action_fallback LightningWeb.FallbackController + action_fallback FallbackController - @doc """ - Lists runs with optional project filtering. - - This function has two variants: - - With `project_id`: Returns runs for a specific project - - Without `project_id`: Returns runs across all accessible projects - - Returns a paginated list of runs with optional datetime filtering. + plug LightningWeb.Plugs.AuthorizeAction, + {:project_users, :access_project} - ## Parameters - - - `conn` - The Plug connection struct with the current resource assigned - - `params` - Map containing: - - `project_id` - Project UUID (optional, filters to specific project) - - `page` - Page number (optional, default: 1) - - `page_size` - Items per page (optional, default: 10, at most 100) - - `state` - Comma-separated list of states to filter by (optional) - - `inserted_after` - Filter runs created after ISO8601 datetime (optional) - - `inserted_before` - Filter runs created before ISO8601 datetime (optional) - - `updated_after` - Filter runs updated after ISO8601 datetime (optional) - - `updated_before` - Filter runs updated before ISO8601 datetime (optional) - - ## Returns + @doc """ + Lists the runs of the projects the caller may access, or of one project on the + nested route, narrowed by the query's filters. - - Renders JSON with paginated list of runs - - `404 Not Found` if project doesn't exist (when project_id provided) - - `403 Forbidden` if user lacks project access (when project_id provided) - - `422 Unprocessable Entity` if datetime parameters are invalid + The caller is authorised before the query is read, so a caller who may not see + the project learns nothing from its filters. Every bad filter is reported in + one 400 `invalid_parameter` (error-format spec §9.4). ## Examples - # All runs accessible to user - GET /api/runs - GET /api/runs?page=2&page_size=50 - GET /api/runs?inserted_after=2024-01-01T00:00:00Z - - # Runs for specific project - GET /api/projects/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d/runs - GET /api/projects/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d/runs?page=2 + GET /api/runs?page=2&page_size=50&state=failed,crashed GET /api/projects/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d/runs?inserted_after=2024-01-01T00:00:00Z """ - @spec index(Plug.Conn.t(), map()) :: Plug.Conn.t() + @spec index(Plug.Conn.t(), map()) :: Plug.Conn.t() | {:error, term()} def index(conn, %{"project_id" => project_id} = params) do - pagination_attrs = Map.take(params, ["page_size", "page"]) - - with :ok <- - Invocation.Query.validate_datetime_params(params, [ - "inserted_after", - "inserted_before", - "updated_after", - "updated_before" - ]), - :ok <- Invocation.Query.validate_run_state_param(params), - project <- Lightning.Projects.get_project(project_id), - :ok <- - ProjectUsers - |> Permissions.can( - :access_project, - conn.assigns.current_resource, - project - ) do - page = - Runs.runs_for_project_query(project) - |> Invocation.Query.filter_runs(params) - |> Lightning.Repo.paginate(pagination_attrs) - - render(conn, "index.json", page: page, conn: conn) + with {:ok, project} <- + Target.project( + conn.assigns.scope, + project_id, + {:project_users, :access_project} + ), + {:ok, filters} <- + params + |> Filters.runs() + |> Errors.refuse_parameters() do + list(project, conn, params, filters) end end def index(conn, params) do - with :ok <- - Invocation.Query.validate_datetime_params(params, [ - "inserted_after", - "inserted_before", - "updated_after", - "updated_before" - ]), - :ok <- Invocation.Query.validate_run_state_param(params) do - pagination_attrs = Map.take(params, ["page_size", "page"]) - - page = - Runs.runs_for_user_query(conn.assigns.current_resource) - |> Invocation.Query.filter_runs(params) - |> Lightning.Repo.paginate(pagination_attrs) - - render(conn, "index.json", page: page, conn: conn) + with {:ok, filters} <- + params + |> Filters.runs() + |> Errors.refuse_parameters() do + list(conn.assigns.current_resource, conn, params, filters) end end @doc """ - Retrieves a specific run by ID. - - Returns detailed information about a single run including its associated - work order and workflow. Access is granted if the user has access to the - run's parent project. - - ## Parameters - - - `conn` - The Plug connection struct with the current resource assigned - - `params` - Map containing: - - `id` - Run UUID (required) - - ## Returns - - - `200 OK` with run JSON including work order and workflow - - `404 Not Found` if run doesn't exist - - `403 Forbidden` if user lacks project access + Shows a run to a caller who may access its project. A run the caller has no + standing on answers 404 `not_found`, as if it didn't exist. ## Examples GET /api/runs/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d """ - @spec show(Plug.Conn.t(), map()) :: Plug.Conn.t() + @spec show(Plug.Conn.t(), map()) :: Plug.Conn.t() | {:error, term()} def show(conn, %{"id" => id}) do - with %Lightning.Run{} = run <- - Runs.get(id, include: [work_order: [workflow: :project]]), - :ok <- - ProjectUsers - |> Permissions.can( - :access_project, - conn.assigns.current_resource, - run.work_order.workflow.project + with {:ok, id} <- Request.path_id(id), + {:ok, run} <- fetch_run(id), + {:ok, _project} <- + Target.project( + conn.assigns.scope, + run.work_order.workflow.project_id, + {:project_users, :access_project} ) do render(conn, "show.json", run: run, conn: conn) - else + end + end + + defp fetch_run(id) do + case Runs.get(id, include: [work_order: :workflow]) do nil -> {:error, :not_found} - error -> error + run -> {:ok, run} end end + + defp list(project_or_user, conn, params, filters) do + page = + Runs.paginate_runs_for(project_or_user, filters, Filters.page(params)) + + render(conn, "index.json", page: page, filters: filters, conn: conn) + end end diff --git a/lib/lightning_web/controllers/api/run_json.ex b/lib/lightning_web/controllers/api/run_json.ex index bbc975f0600..b29fade2a38 100644 --- a/lib/lightning_web/controllers/api/run_json.ex +++ b/lib/lightning_web/controllers/api/run_json.ex @@ -7,7 +7,7 @@ defmodule LightningWeb.API.RunJSON do @fields ~w(state started_at finished_at priority error_type)a - def render("index.json", %{page: page, conn: conn}) do + def render("index.json", %{page: page, filters: filters, conn: conn}) do %{ data: Enum.map(page.entries, &resource(conn, &1)), included: [], @@ -17,11 +17,7 @@ defmodule LightningWeb.API.RunJSON do page_number: page.page_number, page_size: page.page_size }, - links: - %{ - self: url_for(conn) - } - |> Map.merge(pagination_links(conn, page)) + links: pagination_links(conn, page, filters) } end @@ -30,7 +26,7 @@ defmodule LightningWeb.API.RunJSON do data: resource(conn, run), included: [], links: %{ - self: url_for(conn) + self: self_url(conn) } } end diff --git a/lib/lightning_web/controllers/api/user_controller.ex b/lib/lightning_web/controllers/api/user_controller.ex index a6049bb53a5..302c9cd9c29 100644 --- a/lib/lightning_web/controllers/api/user_controller.ex +++ b/lib/lightning_web/controllers/api/user_controller.ex @@ -3,36 +3,59 @@ defmodule LightningWeb.API.UserController do The users resource, for service accounts only. `POST /api/users` takes a flat object of `email`, `password`, `first_name`, - `last_name`, `role` (`user` or `superuser`) and `confirmed`, and ignores any - other key. It answers 201 with the new user, or 409 with the existing user, - rendered the same way, when the email is already taken in any case. + `last_name`, `role` (`user` or `superuser`) and `confirmed`, and refuses any + other key with a 422 `unknown_key` entry at that key, alongside every other + error in the body. It answers 201 with the new user. When the email is + already taken in any case and nothing else is wrong, it answers 409 `taken` + with the existing user's id in `existing`, and nothing else of that user + (error-format spec §9.9). `PATCH /api/users/:id` takes any of `first_name`, `last_name`, `role`, - `password` and `confirmed`, with the same rules as a create, and answers 200 - with the user rendered as a 201 renders one. The email can't be changed, and - `confirmed` only ever confirms. + `password` and `confirmed`, with the same rules as a create, and refuses any + other key the same way, `email` included, since the email can't be changed. + It answers 200 with the user rendered as a 201 renders one. `confirmed` only + ever confirms. + + Either answers 400 `malformed` to a body that isn't a JSON object. `GET /api/users` is paginated, at most 100 a page, and filters by `?email=` - without regard to case. An `email` that isn't a string answers 422. + without regard to case. An `email` that isn't a string answers 400 + `invalid_parameter`. """ use LightningWeb, :controller - import LightningWeb.Plugs.AccessTokenAuth, only: [require_scope: 2] - + alias Ecto.Changeset alias Lightning.Accounts alias Lightning.Accounts.User + alias LightningWeb.API.Errors + alias LightningWeb.API.FallbackController + alias LightningWeb.API.Problem + alias LightningWeb.API.Request + alias LightningWeb.Plugs.AuthorizeAction + + action_fallback FallbackController - action_fallback LightningWeb.FallbackController + @create_fields ~w(email password first_name last_name role confirmed)a + @update_fields ~w(first_name last_name role password confirmed)a + @create_keys Enum.map(@create_fields, &Atom.to_string/1) + @update_keys Enum.map(@update_fields, &Atom.to_string/1) - @create_fields ~w(email password first_name last_name role confirmed) - @update_fields ~w(first_name last_name role password confirmed) + plug AuthorizeAction, + {:users, :read_users} + when action in [:index, :show] - plug :require_scope, "users:read" when action in [:index, :show] - plug :require_scope, "users:write" when action in [:create, :update] + plug AuthorizeAction, + {:users, :write_users} + when action in [:create, :update] def index(conn, params) do - with {:ok, page} <- Accounts.paginate_users(params) do - render(conn, "index.json", page: page, conn: conn) + with {:ok, page} <- + params |> Accounts.paginate_users() |> Errors.refuse_parameters() do + render(conn, "index.json", + page: page, + filters: applied_filters(params), + conn: conn + ) end end @@ -42,59 +65,81 @@ defmodule LightningWeb.API.UserController do end end - def update(conn, %{"id" => id} = params) do + def update(conn, %{"id" => id}) do + body = conn.body_params + attrs = Map.take(body, @update_keys) + with {:ok, user} <- fetch_user(id), + :ok <- Request.check_object(body), + :ok <- + refuse_unknown_keys(body, @update_fields, fn -> + User.details_changeset(user, attrs) + end), {:ok, user} <- - Accounts.update_user( - user, - Map.take(params, @update_fields), - conn.assigns.service_account - ) do + Accounts.update_user(user, attrs, conn.assigns.current_resource) do render(conn, "show.json", user: user, conn: conn) end end - def create(conn, params) do - case Accounts.create_user( - Map.take(params, @create_fields), - conn.assigns.service_account - ) do - {:ok, user} -> - conn - |> put_status(:created) - |> render("show.json", user: user, conn: conn) - - {:error, changeset} -> - case holder_of_taken_email(changeset) do - %User{} = user -> - conn - |> put_status(:conflict) - |> render("show.json", user: user, conn: conn) - - nil -> - {:error, changeset} - end + def create(conn, _params) do + body = conn.body_params + attrs = Map.take(body, @create_keys) + + with :ok <- Request.check_object(body), + :ok <- + refuse_unknown_keys(body, @create_fields, fn -> + Accounts.change_user(%User{}, attrs) + end) do + case Accounts.create_user(attrs, conn.assigns.current_resource) do + {:ok, user} -> + conn + |> put_status(:created) + |> render("show.json", user: user, conn: conn) + + {:error, changeset} -> + {:error, creation_problem(changeset)} + end + end + end + + # An unknown key refuses the body together with every error its changeset + # holds, so a taken email beside one is a 422 and never the 409. + defp refuse_unknown_keys(body, accepted, build_changeset) do + case Request.unknown_keys(body, [], accepted) do + [] -> + :ok + + unknown -> + Errors.refuse(unknown ++ Errors.from_changeset(build_changeset.())) end end - defp holder_of_taken_email(changeset) do - taken? = - changeset.errors - |> Keyword.get_values(:email) - |> Enum.any?(fn {_message, opts} -> - opts[:validation] == :unsafe_unique or opts[:constraint] == :unique - end) + # A create's only unique field is the email, so a `taken` problem means a + # user holds it, unless that user was deleted since. + defp creation_problem(changeset) do + case Errors.problem(changeset) do + %Problem{code: :taken} = problem -> + case Accounts.get_user_by_email(Changeset.get_field(changeset, :email)) do + %User{id: id} -> %{problem | extensions: %{existing: %{id: id}}} + nil -> problem + end - if taken?, - do: Accounts.get_user_by_email(Ecto.Changeset.get_field(changeset, :email)) + problem -> + problem + end end + # Reached only once `paginate_users` has accepted the email as a string. + defp applied_filters(%{"email" => email}), do: %{email: email} + defp applied_filters(_params), do: %{} + defp fetch_user(id) do - with {:ok, id} <- Ecto.UUID.cast(id), + with {:ok, id} <- Request.path_id(id), %User{} = user <- Accounts.get_user(id) do {:ok, user} else - _ -> {:error, :not_found} + nil -> {:error, :not_found} + error -> error end end end diff --git a/lib/lightning_web/controllers/api/user_json.ex b/lib/lightning_web/controllers/api/user_json.ex index 237ab75c888..c2d3e1398c9 100644 --- a/lib/lightning_web/controllers/api/user_json.ex +++ b/lib/lightning_web/controllers/api/user_json.ex @@ -7,10 +7,10 @@ defmodule LightningWeb.API.UserJSON do @fields ~w(email first_name last_name role)a - def render("index.json", %{page: page, conn: conn}) do + def render("index.json", %{page: page, filters: filters, conn: conn}) do %{ data: Enum.map(page.entries, &resource(conn, &1)), - links: Map.merge(%{self: url_for(conn)}, pagination_links(conn, page)) + links: pagination_links(conn, page, filters) } end diff --git a/lib/lightning_web/controllers/api/work_orders_controller.ex b/lib/lightning_web/controllers/api/work_orders_controller.ex index 811da0af814..248f9c96612 100644 --- a/lib/lightning_web/controllers/api/work_orders_controller.ex +++ b/lib/lightning_web/controllers/api/work_orders_controller.ex @@ -59,141 +59,94 @@ defmodule LightningWeb.API.WorkOrdersController do """ use LightningWeb, :controller - alias Lightning.Invocation - alias Lightning.Policies.Permissions - alias Lightning.Policies.ProjectUsers alias Lightning.WorkOrders + alias LightningWeb.API.Errors + alias LightningWeb.API.FallbackController + alias LightningWeb.API.Filters + alias LightningWeb.API.Request + alias LightningWeb.API.Target - action_fallback LightningWeb.FallbackController + action_fallback FallbackController - @doc """ - Lists work orders with optional project filtering. - - This function has two variants: - - With `project_id`: Returns work orders for a specific project - - Without `project_id`: Returns work orders across all accessible projects - - Returns a paginated list of work orders with optional datetime filtering. + plug LightningWeb.Plugs.AuthorizeAction, + {:project_users, :access_project} - ## Parameters - - - `conn` - The Plug connection struct with the current resource assigned - - `params` - Map containing: - - `project_id` - Project UUID (optional, filters to specific project) - - `page` - Page number (optional, default: 1) - - `page_size` - Items per page (optional, default: 10, at most 100) - - `state` - Comma-separated list of states to filter by (optional) - - `inserted_after` - Filter work orders created after ISO8601 datetime (optional) - - `inserted_before` - Filter work orders created before ISO8601 datetime (optional) - - `updated_after` - Filter work orders updated after ISO8601 datetime (optional) - - `updated_before` - Filter work orders updated before ISO8601 datetime (optional) - - ## Returns + @doc """ + Lists the work orders of the projects the caller may access, or of one project + on the nested route, narrowed by the query's filters. - - Renders JSON with paginated list of work orders - - `404 Not Found` if project doesn't exist (when project_id provided) - - `403 Forbidden` if user lacks project access (when project_id provided) - - `422 Unprocessable Entity` if datetime parameters are invalid + The caller is authorised before the query is read, so a caller who may not see + the project learns nothing from its filters. Every bad filter is reported in + one 400 `invalid_parameter` (error-format spec §9.4). ## Examples - # All work orders accessible to user - GET /api/work_orders - GET /api/work_orders?page=2&page_size=50 - GET /api/work_orders?inserted_after=2024-01-01T00:00:00Z - - # Work orders for specific project - GET /api/projects/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d/work_orders - GET /api/projects/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d/work_orders?page=2 + GET /api/work_orders?page=2&page_size=50&state=failed,crashed GET /api/projects/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d/work_orders?inserted_after=2024-01-01T00:00:00Z """ - @spec index(Plug.Conn.t(), map()) :: Plug.Conn.t() + @spec index(Plug.Conn.t(), map()) :: Plug.Conn.t() | {:error, term()} def index(conn, %{"project_id" => project_id} = params) do - pagination_attrs = Map.take(params, ["page_size", "page"]) - - with :ok <- - Invocation.Query.validate_datetime_params(params, [ - "inserted_after", - "inserted_before", - "updated_after", - "updated_before" - ]), - :ok <- Invocation.Query.validate_state_param(params), - project <- Lightning.Projects.get_project(project_id), - :ok <- - ProjectUsers - |> Permissions.can( - :access_project, - conn.assigns.current_resource, - project - ) do - page = - WorkOrders.work_orders_for_project_query(project) - |> Invocation.Query.filter_work_orders(params) - |> Lightning.Repo.paginate(pagination_attrs) - - render(conn, "index.json", page: page, conn: conn) + with {:ok, project} <- + Target.project( + conn.assigns.scope, + project_id, + {:project_users, :access_project} + ), + {:ok, filters} <- + params + |> Filters.work_orders() + |> Errors.refuse_parameters() do + list(project, conn, params, filters) end end def index(conn, params) do - with :ok <- - Invocation.Query.validate_datetime_params(params, [ - "inserted_after", - "inserted_before", - "updated_after", - "updated_before" - ]), - :ok <- Invocation.Query.validate_state_param(params) do - pagination_attrs = Map.take(params, ["page_size", "page"]) - - page = - Invocation.Query.work_orders_for(conn.assigns.current_resource) - |> Invocation.Query.filter_work_orders(params) - |> Lightning.Repo.paginate(pagination_attrs) - - render(conn, "index.json", page: page, conn: conn) + with {:ok, filters} <- + params + |> Filters.work_orders() + |> Errors.refuse_parameters() do + list(conn.assigns.current_resource, conn, params, filters) end end @doc """ - Retrieves a specific work order by ID. - - Returns detailed information about a single work order including its - associated workflow and runs. Access is granted if the user has access - to the work order's parent project. - - ## Parameters - - - `conn` - The Plug connection struct with the current resource assigned - - `params` - Map containing: - - `id` - Work order UUID (required) - - ## Returns - - - `200 OK` with work order JSON including workflow and runs - - `404 Not Found` if work order doesn't exist - - `403 Forbidden` if user lacks project access + Shows a work order, with its runs, to a caller who may access its project. A + work order the caller has no standing on answers 404 `not_found`, as if it + didn't exist. ## Examples GET /api/work_orders/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d """ - @spec show(Plug.Conn.t(), map()) :: Plug.Conn.t() + @spec show(Plug.Conn.t(), map()) :: Plug.Conn.t() | {:error, term()} def show(conn, %{"id" => id}) do - with %Lightning.WorkOrder{} = work_order <- - WorkOrders.get(id, include: [workflow: :project, runs: []]), - :ok <- - ProjectUsers - |> Permissions.can( - :access_project, - conn.assigns.current_resource, - work_order.workflow.project + with {:ok, id} <- Request.path_id(id), + {:ok, work_order} <- fetch_work_order(id), + {:ok, _project} <- + Target.project( + conn.assigns.scope, + work_order.workflow.project_id, + {:project_users, :access_project} ) do render(conn, "show.json", work_order: work_order, conn: conn) - else + end + end + + defp fetch_work_order(id) do + case WorkOrders.get(id, include: [:workflow, runs: []]) do nil -> {:error, :not_found} - error -> error + work_order -> {:ok, work_order} end end + + defp list(project_or_user, conn, params, filters) do + page = + WorkOrders.paginate_work_orders_for( + project_or_user, + filters, + Filters.page(params) + ) + + render(conn, "index.json", page: page, filters: filters, conn: conn) + end end diff --git a/lib/lightning_web/controllers/api/work_orders_json.ex b/lib/lightning_web/controllers/api/work_orders_json.ex index 10c40cacdfb..97ebb047927 100644 --- a/lib/lightning_web/controllers/api/work_orders_json.ex +++ b/lib/lightning_web/controllers/api/work_orders_json.ex @@ -7,7 +7,7 @@ defmodule LightningWeb.API.WorkOrdersJSON do @fields ~w(state last_activity inserted_at updated_at)a - def render("index.json", %{page: page, conn: conn}) do + def render("index.json", %{page: page, filters: filters, conn: conn}) do %{ data: Enum.map(page.entries, &resource(conn, &1)), included: [], @@ -17,11 +17,7 @@ defmodule LightningWeb.API.WorkOrdersJSON do page_number: page.page_number, page_size: page.page_size }, - links: - %{ - self: url_for(conn) - } - |> Map.merge(pagination_links(conn, page)) + links: pagination_links(conn, page, filters) } end @@ -30,7 +26,7 @@ defmodule LightningWeb.API.WorkOrdersJSON do data: resource(conn, work_order), included: [], links: %{ - self: url_for(conn) + self: self_url(conn) } } end diff --git a/lib/lightning_web/controllers/api/workflow_health_controller.ex b/lib/lightning_web/controllers/api/workflow_health_controller.ex index 48be2fe73d6..8a9d19a2259 100644 --- a/lib/lightning_web/controllers/api/workflow_health_controller.ex +++ b/lib/lightning_web/controllers/api/workflow_health_controller.ex @@ -6,55 +6,50 @@ defmodule LightningWeb.API.WorkflowHealthController do health LiveView mounts. This is its own trust boundary — the LiveView's `on_mount` guards do not protect it, since a client can request any path. Access to the project is re-checked here, via the same policy the rest of the - app uses, so support users get in and soft-deleted projects don't. + app uses, so support users get in and a project scheduled for deletion is + refused. One action per chart, so a cheap chart isn't held up by an expensive one. """ use LightningWeb, :controller - alias Lightning.Policies.Permissions - alias Lightning.Projects.Project + alias Ecto.Changeset alias Lightning.Workflows + alias LightningWeb.API.Errors + alias LightningWeb.API.FallbackController + alias LightningWeb.API.Problem + alias LightningWeb.API.Target - plug :authorize_workflow - # After :authorize_workflow so a 404 wins over a 400. - plug :validate_days - plug :validate_timezone when action in [:runs] + action_fallback FallbackController - def outcomes(conn, _params) do - json( - conn, - Workflows.Stats.outcomes(conn.assigns.workflow, conn.assigns.days_back) - ) + def outcomes(conn, params) do + with {:ok, workflow} <- authorize(conn, params), + {:ok, days_back} <- fetch_days(params) do + json(conn, Workflows.Stats.outcomes(workflow, days_back)) + end end - def error_signatures(conn, _params) do - json( - conn, - Workflows.Stats.error_signatures( - conn.assigns.workflow, - conn.assigns.days_back - ) - ) + def error_signatures(conn, params) do + with {:ok, workflow} <- authorize(conn, params), + {:ok, days_back} <- fetch_days(params) do + json(conn, Workflows.Stats.error_signatures(workflow, days_back)) + end end # `vary` because this is the one action whose body depends on a request # header, and nothing between the browser and here would guess that. - def runs(conn, _params) do - conn - |> put_resp_header("vary", "x-timezone") - |> json( - Workflows.Stats.runs( - conn.assigns.workflow, - conn.assigns.days_back, - conn.assigns.timezone - ) - ) + def runs(conn, params) do + with {:ok, workflow} <- authorize(conn, params), + {:ok, days_back} <- fetch_days(params), + {:ok, timezone} <- fetch_timezone(conn) do + conn + |> put_resp_header("vary", "x-timezone") + |> json(Workflows.Stats.runs(workflow, days_back, timezone)) + end end - # Closed set, string-matched — no free integer, no parse to defend. + # A closed set matched as strings, so there is no integer to parse. @days %{"1" => 1, "7" => 7, "30" => 30} - @default_days "7" @default_timezone "Etc/UTC" # CLDR's sentinel for a host clock it could not map to an IANA zone. A @@ -62,17 +57,16 @@ defmodule LightningWeb.API.WorkflowHealthController do # as not telling us. @unknown_timezone "Etc/Unknown" - defp validate_days(conn, _opts) do - case Map.fetch(@days, conn.params["days"] || @default_days) do - {:ok, days_back} -> - assign(conn, :days_back, days_back) + @not_allowed {"is invalid", validation: :inclusion} - :error -> - conn - |> put_status(:bad_request) - |> json(%{error: "days must be one of 1, 7, 30"}) - |> halt() - end + defp fetch_days(params) do + with {:ok, %{days: days}} <- + {%{days: "7"}, %{days: :string}} + |> Changeset.cast(params, [:days], empty_values: []) + |> Changeset.validate_inclusion(:days, Map.keys(@days)) + |> Changeset.apply_action(:validate) + |> Errors.refuse_parameters(), + do: {:ok, Map.fetch!(@days, days)} end # The reader's timezone, because nothing in Lightning records one. The only @@ -82,53 +76,44 @@ defmodule LightningWeb.API.WorkflowHealthController do # # Validated before it reaches the cache key, so `:workflow_stats` is keyed on # the tz database rather than on anything a header can carry. - defp validate_timezone(conn, _opts) do + defp fetch_timezone(conn) do case get_req_header(conn, "x-timezone") do [] -> - assign(conn, :timezone, @default_timezone) + {:ok, @default_timezone} [@unknown_timezone] -> - assign(conn, :timezone, @default_timezone) + {:ok, @default_timezone} [timezone] -> if Tzdata.zone_exists?(timezone), - do: assign(conn, :timezone, timezone), - else: reject_timezone(conn) + do: {:ok, timezone}, + else: refuse_timezone() _ -> - reject_timezone(conn) + refuse_timezone() end end - defp reject_timezone(conn) do - conn - |> put_status(:bad_request) - |> json(%{error: "x-timezone must be an IANA timezone name"}) - |> halt() + defp refuse_timezone do + {:error, + Problem.new(:invalid_header, + errors: [Errors.entry({:header, "x-timezone"}, @not_allowed)] + )} end - defp authorize_workflow(conn, _opts) do - %{"project_id" => project_id, "workflow_id" => workflow_id} = conn.params - - with %_{} = user <- conn.assigns[:current_user], - {:ok, project_id} <- Ecto.UUID.cast(project_id), - %{project: project} = workflow <- - Workflows.get_workflow_for_project( - %Project{id: project_id}, - workflow_id, - include: [:project] - ), - :ok <- Permissions.can(:project_users, :access_project, user, project) do - assign(conn, :workflow, workflow) - else - # 404 for every failure, not 403: a 403 would confirm the workflow exists - # to someone who can't see the project. - _ -> - conn - |> put_status(:not_found) - |> put_view(LightningWeb.ErrorView) - |> render(:"404") - |> halt() + defp authorize(conn, %{"project_id" => project_id, "workflow_id" => id}) do + Target.in_project( + conn.assigns.current_user, + project_id, + {:project_users, :access_project}, + &fetch_workflow(&1, id) + ) + end + + defp fetch_workflow(project, id) do + case Workflows.get_workflow_for_project(project, id) do + nil -> {:error, :not_found} + workflow -> {:ok, workflow} end end end diff --git a/lib/lightning_web/controllers/api/workflows_controller.ex b/lib/lightning_web/controllers/api/workflows_controller.ex index 727fa5628cc..776fd880066 100644 --- a/lib/lightning_web/controllers/api/workflows_controller.ex +++ b/lib/lightning_web/controllers/api/workflows_controller.ex @@ -14,11 +14,16 @@ defmodule LightningWeb.API.WorkflowsController do ## Validation Rules - - Workflows must be valid DAGs (no cycles) - - Only one trigger can be enabled at a time - - All jobs and triggers must have valid UUIDs - - Edges must reference existing jobs/triggers - - Cannot modify workflows with active presence (being edited in UI) + The rules in `Lightning.Workflows.GraphRules`: every job and trigger carries + an id, the edges form no cycle, one enabled edge leaves the one enabled + trigger, and an update removes no trigger. On create, the ids of jobs and + triggers are the client's own and are replaced with new UUIDs. A workflow open + in the editor can't be updated. + + Errors are problem documents (`LightningWeb.API.FallbackController`), checked + in the order of error-format spec §4: the token's scope, the caller's standing + and role, the workflow being open in the editor, the body, then the plan + limit. ## Examples @@ -78,723 +83,266 @@ defmodule LightningWeb.API.WorkflowsController do """ use LightningWeb, :controller - import Lightning.Workflows.WorkflowUsageLimiter, - only: [limit_workflow_activation: 2] - alias Ecto.Changeset - - alias Lightning.Extensions.Message - alias Lightning.Graph - alias Lightning.Policies.Permissions - alias Lightning.Projects.Project - alias Lightning.Repo alias Lightning.Workflows - alias Lightning.Workflows.Edge + alias Lightning.Workflows.GraphRules alias Lightning.Workflows.Presence alias Lightning.Workflows.Workflow - alias LightningWeb.ChangesetJSON - - action_fallback LightningWeb.FallbackController - - require Logger - - @doc """ - Lists workflows with optional project filtering. - - This function has two variants: - - With `project_id`: Returns workflows for a specific project - - Without `project_id`: Returns workflows across all accessible projects + alias Lightning.Workflows.WorkflowUsageLimiter + alias LightningWeb.API.FallbackController + alias LightningWeb.API.Request + alias LightningWeb.API.Target - Returns all workflows including their jobs, triggers, and edges. + action_fallback FallbackController - ## Parameters + @access_project {:project_users, :access_project} - - `conn` - The Plug connection struct with the current resource assigned - - `params` - Map containing: - - `project_id` - Project UUID (optional, filters to specific project) + plug LightningWeb.Plugs.AuthorizeAction, + {:project_users, :access_project} - ## Returns + @doc """ + Lists workflows, with their jobs, triggers and edges. - - `200 OK` with workflows list and empty errors map - - `404 Not Found` if project doesn't exist (when project_id provided) - - `403 Forbidden` if user lacks project access (when project_id provided) + With `project_id`, in the path or the query, lists that project's workflows; a + project the caller can't see answers 404 `not_found`. Without it, lists the + workflows of every project the caller can access. ## Examples - # All workflows accessible to user GET /api/workflows - - # Workflows for specific project GET /api/workflows?project_id=a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d + GET /api/projects/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d/workflows """ - @spec index(Plug.Conn.t(), map()) :: Plug.Conn.t() + @spec index(Plug.Conn.t(), map()) :: Plug.Conn.t() | {:error, term()} def index(conn, %{"project_id" => project_id}) do - with :ok <- validate_project_id(%{"project_id" => project_id}, project_id), - :ok <- authorize_read(conn, project_id) do + with {:ok, project} <- + Target.project(conn.assigns.scope, project_id, @access_project) do list = - Workflows.list_project_workflows(project_id, + Workflows.list_project_workflows(project.id, include: [:edges, :jobs, :triggers] ) - json(conn, %{workflows: Enum.map(list, &public_workflow/1), errors: %{}}) + json(conn, %{workflows: Enum.map(list, &public_workflow/1)}) end - |> then(&maybe_handle_error(conn, &1)) end def index(conn, _params) do list = - Workflows.workflows_for_user_query(conn.assigns.current_resource) - |> Repo.all() - |> Repo.preload([:edges, :jobs, :triggers]) + Workflows.list_workflows_for_user(conn.assigns.current_resource, + include: [:edges, :jobs, :triggers] + ) - json(conn, %{workflows: Enum.map(list, &public_workflow/1), errors: %{}}) + json(conn, %{workflows: Enum.map(list, &public_workflow/1)}) end @doc """ - Creates a new workflow in a project. - - Creates a workflow with jobs, triggers, and edges. Validates the workflow - structure and ensures it forms a valid DAG with no cycles. + Creates a workflow in the path's project, with its jobs, triggers and edges. - ## Parameters - - - `conn` - The Plug connection struct with the current resource assigned - - `params` - Map containing: - - `project_id` - Project UUID (required) - - `name` - Workflow name (required) - - `jobs` - List of job definitions with UUIDs (required) - - `triggers` - List of trigger definitions with UUIDs (required) - - `edges` - List of edge definitions connecting jobs/triggers (required) - - ## Returns - - - `201 Created` with workflow JSON on success - - `422 Unprocessable Entity` with validation errors - - `403 Forbidden` if user lacks write access + Edges name jobs and triggers by the ids the request gives them, which are + replaced with new UUIDs. A refused body answers one 422 with every error in + it, and the plan limit on enabled triggers is checked only once the body is + valid (402 `exceeds_limit`). ## Examples - POST /api/workflows + POST /api/projects/a1b2c3d4-.../workflows { - "project_id": "a1b2c3d4-...", "name": "Data Processing Pipeline", "jobs": [ - { - "id": "job-uuid-1", - "name": "Extract Data", - "body": "fn(state => state)" - } - ], - "triggers": [ - { - "id": "trigger-uuid-1", - "type": "webhook", - "enabled": true - } + {"id": "extract", "name": "Extract Data", "body": "fn(state => state)"} ], + "triggers": [{"id": "webhook", "type": "webhook", "enabled": true}], "edges": [ { - "source_trigger_id": "trigger-uuid-1", - "target_job_id": "job-uuid-1", - "condition": "always" + "source_trigger_id": "webhook", + "target_job_id": "extract", + "condition_type": "always" } ] } """ - @spec create(Plug.Conn.t(), map()) :: Plug.Conn.t() - def create(conn, %{"project_id" => project_id} = params) do - with :ok <- validate_project_id(conn.body_params, project_id), - :ok <- authorize_write(conn, project_id), - {:ok, workflow} <- save_workflow(params, conn.assigns.current_resource) do + @spec create(Plug.Conn.t(), map()) :: Plug.Conn.t() | {:error, term()} + def create(conn, %{"project_id" => project_id}) do + body = conn.body_params + + with {:ok, project} <- + Target.project( + conn.assigns.scope, + project_id, + {:project_users, :create_workflow} + ), + :ok <- Request.check_object(body), + {:ok, workflow} <- + %Workflow{} + |> Workflow.changeset( + body + |> remap_client_ids() + |> Map.put("project_id", project.id) + ) + |> Request.validate_matches_path(body, :project_id, project.id) + |> save(conn.assigns.current_resource) do conn |> put_status(:created) - |> json(%{workflow: public_workflow(workflow), errors: %{}}) + |> json(%{workflow: public_workflow(workflow)}) end - |> then(&maybe_handle_error(conn, &1)) end @doc """ - Retrieves a workflow by ID. - - This function has two variants: - - With `project_id`: Validates workflow belongs to specified project - - Without `project_id`: Determines project access from workflow association + Shows a workflow, with its jobs, triggers and edges. - Returns a workflow with all its jobs, triggers, and edges. - - ## Parameters - - - `conn` - The Plug connection struct with the current resource assigned - - `params` - Map containing: - - `id` - Workflow UUID (required) - - `project_id` - Project UUID (optional, for validation) - - ## Returns - - - `200 OK` with workflow JSON on success - - `404 Not Found` if workflow doesn't exist - - `400 Bad Request` if workflow exists but project_id mismatch - - `403 Forbidden` if user lacks project access + A workflow the caller can't see, a deleted one, or one outside the path's + project answers 404 `not_found`. ## Examples - # Get workflow by ID only GET /api/workflows/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d - - # Get workflow with project validation - GET /api/workflows/workflow-uuid-1?project_id=a1b2c3d4-... + GET /api/projects/a1b2c3d4-.../workflows/a1b2c3d4-... """ - @spec show(Plug.Conn.t(), map()) :: Plug.Conn.t() - def show(conn, %{"project_id" => project_id, "id" => workflow_id}) do - with :ok <- validate_uuid(project_id), - :ok <- validate_uuid(workflow_id), - :ok <- authorize_read(conn, project_id), - {:ok, workflow} <- get_workflow(workflow_id, project_id) do - json(conn, %{workflow: public_workflow(workflow), errors: %{}}) + @spec show(Plug.Conn.t(), map()) :: Plug.Conn.t() | {:error, term()} + def show(conn, %{"project_id" => project_id, "id" => id}) do + with {:ok, workflow} <- + Target.in_project( + conn.assigns.scope, + project_id, + @access_project, + &fetch_workflow(id, &1) + ) do + json(conn, %{workflow: public_workflow(workflow)}) end - |> then(&maybe_handle_error(conn, &1)) end - def show(conn, %{"id" => workflow_id}) do - with :ok <- validate_uuid(workflow_id), - workflow when not is_nil(workflow) <- - Workflows.get_workflow(workflow_id, - include: [:edges, :jobs, :triggers] - ), - :ok <- authorize_read_workflow(conn, workflow) do - json(conn, %{workflow: public_workflow(workflow), errors: %{}}) - else - nil -> {:error, :not_found} - error -> error + def show(conn, %{"id" => id} = params) do + with {:ok, project_id} <- workflow_project_id(id) do + show(conn, Map.put(params, "project_id", project_id)) end - |> then(&maybe_handle_error(conn, &1, workflow_id)) end @doc """ - Updates an existing workflow. - - Modifies a workflow's structure, including its jobs, triggers, and edges. - Validates the updated workflow forms a valid DAG. Prevents updates while - workflow is being edited in the UI (active presence). - - ## Parameters - - - `conn` - The Plug connection struct with the current resource assigned - - `params` - Map containing: - - `project_id` - Project UUID (required, must match workflow's project) - - `id` - Workflow UUID (required) - - `name` - Updated workflow name (optional) - - `jobs` - Updated jobs list (optional) - - `triggers` - Updated triggers list (optional, cannot replace existing) - - `edges` - Updated edges list (optional) + Updates a workflow's fields, jobs, triggers and edges. - ## Returns - - - `200 OK` with updated workflow JSON on success - - `422 Unprocessable Entity` with validation errors - - `409 Conflict` if workflow has active presence - - `403 Forbidden` if user lacks write access - - `404 Not Found` if workflow doesn't exist + A list the body sends replaces the stored one; triggers may be edited or added + but not removed. A workflow open in the editor answers 409 `being_edited`, + before the body is read. ## Examples - PATCH /api/workflows/workflow-uuid-1?project_id=a1b2c3d4-... - { - "name": "Updated Pipeline Name", - "jobs": [...] - } + PATCH /api/projects/a1b2c3d4-.../workflows/a1b2c3d4-... + {"name": "Updated Pipeline Name"} """ - @spec update(Plug.Conn.t(), map()) :: Plug.Conn.t() - def update(conn, %{"project_id" => project_id, "id" => workflow_id} = params) do - with :ok <- validate_project_id(conn.body_params, project_id), - :ok <- validate_workflow_id(conn.body_params, workflow_id), - :ok <- authorize_write(conn, project_id), - {:ok, workflow} <- get_workflow(workflow_id, project_id), - :ok <- authorize_write(conn, workflow), + @spec update(Plug.Conn.t(), map()) :: Plug.Conn.t() | {:error, term()} + def update(conn, %{"project_id" => project_id, "id" => id}) do + body = conn.body_params + + with {:ok, workflow} <- + Target.in_project( + conn.assigns.scope, + project_id, + {:project_users, :edit_workflow}, + &fetch_workflow(id, &1) + ), + :ok <- refuse_being_edited(workflow), + :ok <- Request.check_object(body), {:ok, workflow} <- - save_workflow(workflow, params, conn.assigns.current_resource) do - json(conn, %{workflow: public_workflow(workflow), errors: %{}}) + workflow + |> Workflows.change_workflow(body) + |> Request.validate_matches_path( + body, + :project_id, + workflow.project_id + ) + |> Request.validate_matches_path(body, :id, workflow.id) + |> save(conn.assigns.current_resource) do + json(conn, %{workflow: public_workflow(workflow)}) end - |> then(&maybe_handle_error(conn, &1, workflow_id)) end - defp public_workflow(workflow) do - Map.take(workflow, Workflow.json_fields() -- [:state]) - end - - defp count_enabled_triggers(params), - do: params |> Map.get("triggers", []) |> Enum.count(& &1["enabled"]) - - defp save_workflow(%{"project_id" => project_id} = params, user) do - active_triggers_count = count_enabled_triggers(params) - - if active_triggers_count > 1 do - {:error, :too_many_active_triggers} + defp workflow_project_id(id) do + with {:ok, id} <- Request.path_id(id), + %Workflow{project_id: project_id} <- Workflows.get_workflow(id) do + {:ok, project_id} else - save_workflow(params, active_triggers_count > 0, project_id, user) - end - end - - defp save_workflow( - %{triggers: triggers} = workflow, - params, - user - ) do - changes_triggers? = Map.has_key?(params, "triggers") - - triggers_ids = - params - |> Map.get("triggers", []) - |> Enum.map(& &1["id"]) - - active_triggers_count = count_enabled_triggers(params) - - cond do - changes_triggers? and Enum.any?(triggers, &(&1.id not in triggers_ids)) -> - {:error, :cannot_replace_trigger} - - active_triggers_count > 1 -> - {:error, :too_many_active_triggers} - - :else -> - workflow - |> Workflows.change_workflow(params) - |> save_workflow(active_triggers_count > 0, workflow.project_id, user) + nil -> {:error, :not_found} + refusal -> refusal end end - defp save_workflow(params_or_changeset, activate?, project_id, user) do - with :ok <- limit_workflow_activation(activate?, project_id), - {params_or_changeset, ids_map} <- - remap_arbitrary_ids(params_or_changeset), - :ok <- validate_workflow(params_or_changeset, ids_map), - {:error, %{changes: changes} = _changeset} <- - Workflows.save_workflow(params_or_changeset, user) do - triggers_with_errors = map_errors_to_ids(changes[:triggers]) - jobs_with_errors = map_errors_to_ids(changes[:jobs]) - edges_with_errors = map_errors_to_ids(changes[:edges]) - - cond do - Enum.any?(triggers_with_errors) -> - {:error, {:invalid_triggers, triggers_with_errors, ids_map}} - - Enum.any?(jobs_with_errors) -> - {:error, {:invalid_jobs, jobs_with_errors, ids_map}} - - Enum.any?(edges_with_errors) -> - {:error, {:invalid_edges, edges_with_errors, ids_map}} - - :else -> - {:error, :invalid_workflow} - end + defp fetch_workflow(id, project) do + case Workflows.get_workflow_for_project(project, id, + include: [:edges, :jobs, :triggers] + ) do + nil -> {:error, :not_found} + workflow -> {:ok, workflow} end end - # list of edges, jobs or triggers - defp map_errors_to_ids(nil), do: Map.new() - - defp map_errors_to_ids(list) do - list - |> Enum.filter(&Enum.any?(&1.errors)) - |> Map.new(&{Ecto.Changeset.fetch_field!(&1, :id), ChangesetJSON.errors(&1)}) + defp refuse_being_edited(workflow) do + if Presence.has_any_presence?(workflow), + do: {:error, :being_edited}, + else: :ok end - defp validate_workflow(%Changeset{} = changeset, ids_map) do - edges = Changeset.get_field(changeset, :edges) - jobs = Changeset.get_field(changeset, :jobs) - triggers = Changeset.get_field(changeset, :triggers) - - validate_workflow(edges, jobs, triggers, ids_map) - end + # Spec §4: the body is checked whole before the plan limit, so a 422 never + # carries a limit and a 402 never carries a field error. + defp save(changeset, user) do + changeset = GraphRules.validate(changeset) - defp validate_workflow( - %{ - "edges" => edges, - "jobs" => jobs, - "triggers" => triggers - }, - ids_map - ), - do: validate_workflow(edges, jobs, triggers, ids_map) - - defp validate_workflow(%{} = params, ids_map) do - edges = Map.get(params, "edges", []) - jobs = Map.get(params, "jobs", []) - triggers = Map.get(params, "triggers", []) - - validate_workflow(edges, jobs, triggers, ids_map) - end - - defp validate_workflow(edges, jobs, triggers, ids_map) do - # {:ok, _ids} <- validate_ids(edges), - with {:ok, triggers_ids} <- validate_ids(triggers), - {:ok, _ids} <- validate_ids(jobs), - {:ok, source_trigger_id} <- get_initial_node(edges, triggers_ids) do - edges - |> make_graph() - |> Graph.traverse(source_trigger_id) - |> case do - {:error, :graph_has_a_cycle, node_id} -> - client_id = - Enum.find_value(ids_map, fn {client_id, id} -> - if id == node_id, do: client_id - end) - - {:error, :graph_has_a_cycle, client_id} - - result -> - result - end + with :ok <- refuse_invalid(changeset), + :ok <- WorkflowUsageLimiter.limit_workflow_activation(changeset) do + Workflows.save_workflow(changeset, user) end end - defp validate_ids(list) do - Enum.reduce_while(list, [], fn item, acc -> - case Map.get(item, :id) || Map.get(item, "id") do - nil -> - {:halt, {:error, :missing_id}} - - id -> - case Ecto.UUID.dump(id) do - {:ok, _bin} -> {:cont, [id | acc]} - :error -> {:halt, {:error, :invalid_id, id}} - end - end - end) - |> then(fn result -> - with ids_list when is_list(ids_list) <- result, do: {:ok, ids_list} - end) - end + defp refuse_invalid(%Changeset{valid?: true}), do: :ok + defp refuse_invalid(changeset), do: {:error, changeset} - defp get_initial_node(edges, triggers_ids, all_enabled \\ false) do - edges - |> Enum.map( - &(Map.get(&1, :source_trigger_id) || Map.get(&1, "source_trigger_id")) - ) - |> Enum.filter(&(&1 in triggers_ids)) - |> case do - [] -> - {:error, :edges_misses_a_trigger} - - [source_trigger_id] -> - {:ok, source_trigger_id} - - _list -> - if all_enabled do - {:error, :edges_has_many_triggers} - else - edges - |> Enum.filter(&(Map.get(&1, :enabled) || Map.get(&1, "enabled"))) - |> get_initial_node(triggers_ids, true) - end - end + defp public_workflow(workflow) do + Map.take(workflow, Workflow.json_fields() -- [:state]) end - defp make_graph(edges) do - Enum.reduce(edges, Graph.new(), fn - %Edge{} = edge, graph -> - Graph.add_edge(graph, edge) - - edge, graph -> - edge - |> Map.take(["source_trigger_id", "source_job_id", "target_job_id"]) - |> Map.new(fn {key, value} -> {String.to_existing_atom(key), value} end) - |> then(&Graph.add_edge(graph, struct(Edge, &1))) - end) - end + # On create, jobs and triggers carry ids of the client's choosing, which the + # edges refer to. Each becomes a new UUID; an edge naming an id no job or + # trigger carries keeps it, so the changeset reports it where it was sent. + defp remap_client_ids(body) do + {body, ids} = remap_ids(body, "jobs", %{}) + {body, ids} = remap_ids(body, "triggers", ids) - defp get_workflow(workflow_id, project_id) do - case Workflows.get_workflow(workflow_id, include: [:edges, :jobs, :triggers]) do - nil -> {:error, :not_found} - %{project_id: ^project_id} = workflow -> {:ok, workflow} - _project_mismatch -> {:error, :bad_request} - end - end + case body do + %{"edges" => edges} when is_list(edges) -> + Map.put(body, "edges", Enum.map(edges, &remap_edge(&1, ids))) - defp validate_uuid(project_id) do - case Ecto.UUID.dump(to_string(project_id)) do - {:ok, _bin} -> :ok - :error -> {:error, :invalid_id, project_id} + _no_edges -> + body end end - defp validate_workflow_id(%{"id" => workflow_id}, workflow_id), - do: validate_uuid(workflow_id) - - defp validate_workflow_id(%{"id" => _workflow_id1}, _workflow_id2), - do: {:error, :invalid_path_workflow_id} - - defp validate_workflow_id(_no_body_id, _workflow_id2), do: :ok - - defp validate_project_id(%{"project_id" => project_id}, project_id), - do: validate_uuid(project_id) - - defp validate_project_id(%{"project_id" => _project_id1}, _project_id2), - do: {:error, :invalid_project_id} - - defp validate_project_id(_patch, project_id), - do: validate_uuid(project_id) + defp remap_ids(body, key, ids) do + case body do + %{^key => items} when is_list(items) -> + {items, ids} = Enum.map_reduce(items, ids, &remap_id/2) + {Map.put(body, key, items), ids} - defp authorize_write(_conn, %Workflow{name: name} = workflow) do - if Presence.has_any_presence?(workflow) do - {:error, :conflict, name} - else - :ok + _no_list -> + {body, ids} end end - defp authorize_write(conn, project_id) do - authorize_for_project(conn, project_id, :edit_workflow) - end - - defp authorize_read(conn, project_id) do - authorize_for_project(conn, project_id, :access_project) - end - - defp authorize_read_workflow(conn, %Workflow{project_id: project_id}) do - authorize_read(conn, project_id) - end - - defp authorize_for_project(conn, project_id, access) do - project = Repo.get(Project, project_id) - - Permissions.can( - Lightning.Policies.ProjectUsers, - access, - conn.assigns.current_resource, - project - ) + defp remap_id(%{"id" => client_id} = item, ids) when not is_nil(client_id) do + id = Ecto.UUID.generate() + {Map.put(item, "id", id), Map.put(ids, client_id, id)} end - defp maybe_handle_error(conn, result, workflow_id \\ nil) + defp remap_id(item, ids), do: {item, ids} - defp maybe_handle_error(conn, {:error, :not_found}, workflow_id) do - conn - |> put_status(:not_found) - |> json(%{id: workflow_id, errors: ["Not Found"]}) - end - - defp maybe_handle_error(_conn, result, _workflow_id) when is_map(result), - do: result - - defp maybe_handle_error(conn, {:error, :conflict, name}, workflow_id), - do: - conn - |> put_status(:conflict) - |> json(%{ - id: workflow_id, - errors: %{ - workflow: [ - "Cannot save a workflow (#{name}) while it is being edited on the App UI" - ] - } - }) - - defp maybe_handle_error(conn, {:error, :invalid_id, id}, workflow_id), - do: - reply_422( - conn, - workflow_id, - :workflow, - "Id #{id} should be a UUID." - ) - - defp maybe_handle_error( - conn, - {:error, :invalid_path_workflow_id}, - workflow_id - ), - do: - reply_422( - conn, - workflow_id, - :id, - "Workflow ID doesn't match with the one on the path." - ) - - defp maybe_handle_error(conn, {:error, :missing_id}, workflow_id), - do: - reply_422( - conn, - workflow_id, - :workflow, - "All jobs and triggers should have an id (UUID)." - ) + defp remap_edge(%{} = edge, ids) do + Map.new(edge, fn + {key, value} + when key in ["source_job_id", "source_trigger_id", "target_job_id"] -> + {key, Map.get(ids, value, value)} - @reason_entity_field %{ - invalid_triggers: {"Trigger", :triggers}, - invalid_jobs: {"Job", :jobs}, - invalid_edges: {"Edge", :edges} - } - defp maybe_handle_error( - conn, - {:error, {reason, id_to_errors_map, ids_map}}, - workflow_id - ) - when reason in [:invalid_triggers, :invalid_jobs, :invalid_edges] do - {entity, workflow_field} = Map.get(@reason_entity_field, reason) - - client_ids = Map.new(ids_map, fn {k, v} -> {v, k} end) - - error_msgs = - id_to_errors_map - |> Enum.map(fn {id, errors} -> - errors = - Enum.map(errors, fn - {field, [error]} -> "#{field}: #{error}" - {field, errors} -> "#{field}: #{inspect(errors)}" - end) - - "#{entity} #{Map.get(client_ids, id, id)} has the errors: [#{errors}]" - end) - - reply_422( - conn, - workflow_id, - workflow_field, - error_msgs - ) - end - - defp maybe_handle_error( - conn, - {:error, :too_many_workflows, %Message{text: error_msg}}, - workflow_id - ) do - reply_422( - conn, - workflow_id, - :project_id, - error_msg - ) - end - - defp maybe_handle_error(conn, {:error, reason}, workflow_id) - when reason in [:invalid_project_id, :invalid_project_id_format] do - case reason do - :invalid_project_id -> - "The project_id of the body does not match the one the path." - - :invalid_project_id_format -> - "The project_id is not a UUID." - end - |> then( - &reply_422( - conn, - workflow_id, - :project_id, - &1 - ) - ) - end - - defp maybe_handle_error(conn, result, workflow_id) - when is_tuple(result) do - case result do - {:error, :invalid_jobs_ids, job_ids} -> - reply_422( - conn, - workflow_id, - :jobs, - "The jobs #{inspect(job_ids)} should be present both in the jobs and on an edge." - ) - - {:error, :edges_misses_a_trigger} -> - reply_422( - conn, - workflow_id, - :edges, - "Missing edge with source_trigger_id." - ) - - {:error, :multiple_targets_for_trigger} -> - reply_422( - conn, - workflow_id, - :edges, - "source_trigger_id must have a single target." - ) - - {:error, :edges_has_many_triggers} -> - reply_422( - conn, - workflow_id, - :edges, - "There should be only one enabled edge with source_trigger_id." - ) - - {:error, :graph_has_a_cycle, node_id} -> - reply_422( - conn, - workflow_id, - :edges, - "Cycle detected on job #{node_id}." - ) - - {:error, :cannot_replace_trigger} -> - reply_422( - conn, - workflow_id, - :triggers, - "A trigger cannot be replaced, only edited or added." - ) - - {:error, :too_many_active_triggers} -> - reply_422( - conn, - workflow_id, - :triggers, - "A workflow can have only one trigger enabled at a time." - ) - - result -> - result - end - end - - defp reply_422(conn, workflow_id, field, msgs) do - conn - |> put_status(:unprocessable_entity) - |> json(%{id: workflow_id, errors: %{field => List.wrap(msgs)}}) - end - - defp remap_arbitrary_ids( - %{"jobs" => jobs, "edges" => edges, "triggers" => triggers} = workflow - ) do - {jobs, ids_map} = - Enum.map_reduce(jobs, Map.new(), fn %{"id" => client_id} = job, ids -> - insert_uuid = Ecto.UUID.generate() - {Map.put(job, "id", insert_uuid), Map.put(ids, client_id, insert_uuid)} - end) - - {triggers, ids_map} = - Enum.map_reduce(triggers, ids_map, fn %{"id" => client_id} = trigger, - ids -> - insert_uuid = Ecto.UUID.generate() - - {Map.put(trigger, "id", insert_uuid), - Map.put(ids, client_id, insert_uuid)} - end) - - edges = - Enum.map(edges, fn edge -> - edge - |> Map.update("source_job_id", nil, &Map.get(ids_map, &1)) - |> Map.update("source_trigger_id", nil, &Map.get(ids_map, &1)) - |> Map.update("target_job_id", nil, &Map.get(ids_map, &1)) - end) - - { - Map.merge(workflow, %{ - "jobs" => jobs, - "edges" => edges, - "triggers" => triggers - }), - ids_map - } + pair -> + pair + end) end - defp remap_arbitrary_ids(changeset) do - {changeset, Map.new()} - end + defp remap_edge(edge, _ids), do: edge end diff --git a/lib/lightning_web/controllers/fallback_controller.ex b/lib/lightning_web/controllers/browser_fallback_controller.ex similarity index 67% rename from lib/lightning_web/controllers/fallback_controller.ex rename to lib/lightning_web/controllers/browser_fallback_controller.ex index 5098c4aea94..66d964ff183 100644 --- a/lib/lightning_web/controllers/fallback_controller.ex +++ b/lib/lightning_web/controllers/browser_fallback_controller.ex @@ -1,12 +1,14 @@ -defmodule LightningWeb.FallbackController do +defmodule LightningWeb.BrowserFallbackController do @moduledoc """ - Translates controller action results into valid `Plug.Conn` responses. + Turns the errors the browser-pipeline controllers return into responses. API + controllers answer through `LightningWeb.API.FallbackController`. See `Phoenix.Controller.action_fallback/1` for more details. """ use LightningWeb, :controller - # This clause is an example of how to handle resources that cannot be found. + import LightningWeb.CoreComponents, only: [translate_errors: 1] + def call(conn, {:error, :not_found}) do conn |> put_status(:not_found) @@ -28,16 +30,6 @@ defmodule LightningWeb.FallbackController do |> render(:"401") end - def call(conn, {:error, :conflict}) do - conn - |> put_status(:conflict) - |> json(%{ - error: - "Multiple collections found with this name. " <> - "Add ?project_id= to disambiguate." - }) - end - def call(conn, {:error, :forbidden}) do conn |> put_status(:forbidden) @@ -48,12 +40,7 @@ defmodule LightningWeb.FallbackController do def call(conn, {:error, %Ecto.Changeset{} = changeset}) do conn |> put_status(:unprocessable_entity) - |> put_view(LightningWeb.ChangesetView) - |> render("error.json", changeset: changeset) - end - - def call(conn, {:error, _reason, %Lightning.Extensions.Message{} = message}) do - call(conn, {:error, message}) + |> json(%{errors: translate_errors(changeset)}) end def call(conn, {:error, %Lightning.Extensions.Message{} = message}) do @@ -62,6 +49,7 @@ defmodule LightningWeb.FallbackController do |> json(%{error: message.text}) end + # The OIDC token endpoint's error body, a string or a map. def call(conn, {:error, error}) when is_binary(error) do conn |> put_status(:bad_request) diff --git a/lib/lightning_web/controllers/changeset_json.ex b/lib/lightning_web/controllers/changeset_json.ex deleted file mode 100644 index 79145ae1b3a..00000000000 --- a/lib/lightning_web/controllers/changeset_json.ex +++ /dev/null @@ -1,18 +0,0 @@ -defmodule LightningWeb.ChangesetJSON do - @moduledoc """ - Renders changesets as JSON. - """ - - import LightningWeb.CoreComponents, only: [translate_error: 1] - - def errors(%Ecto.Changeset{} = changeset) do - Ecto.Changeset.traverse_errors( - changeset, - &translate_error/1 - ) - end - - def errors(%{changeset: %Ecto.Changeset{} = changeset}) do - %{errors: errors(changeset)} - end -end diff --git a/lib/lightning_web/controllers/collection_download_controller.ex b/lib/lightning_web/controllers/collection_download_controller.ex new file mode 100644 index 00000000000..edeb4121762 --- /dev/null +++ b/lib/lightning_web/controllers/collection_download_controller.ex @@ -0,0 +1,60 @@ +defmodule LightningWeb.CollectionDownloadController do + @moduledoc """ + Downloads a collection's items from the browser, as one JSON array. It + answers errors as the other browser controllers do, not as the + `/collections` API does. + """ + use LightningWeb, :controller + + alias Lightning.Collections + alias Lightning.Policies.Permissions + + action_fallback LightningWeb.BrowserFallbackController + + def download(conn, %{"project_id" => project_id, "name" => name}) do + with {:ok, uuid} <- cast_uuid(project_id), + {:ok, collection} <- Collections.get_collection(uuid, name), + :ok <- + Permissions.can( + :collections, + :access_collection, + conn.assigns.current_user, + collection + ) do + items_stream = Collections.stream_all(collection, %{}) + + conn + |> put_resp_content_type("application/json") + |> put_resp_header( + "content-disposition", + ~s(attachment; filename="#{name}.json") + ) + |> stream_as_json_array(items_stream) + end + end + + defp cast_uuid(project_id) do + case Ecto.UUID.cast(project_id) do + {:ok, uuid} -> {:ok, uuid} + :error -> {:error, :bad_request} + end + end + + defp stream_as_json_array(conn, items_stream) do + conn = send_chunked(conn, 200) + {:ok, conn} = Plug.Conn.chunk(conn, "[") + + {conn, _first?} = + Enum.reduce_while(items_stream, {conn, true}, fn item, {conn, first?} -> + prefix = if first?, do: "", else: "," + + case Plug.Conn.chunk(conn, prefix <> Jason.encode!(item)) do + {:ok, conn} -> {:cont, {conn, false}} + {:error, :closed} -> {:halt, {conn, false}} + end + end) + + {:ok, conn} = Plug.Conn.chunk(conn, "]") + conn + end +end diff --git a/lib/lightning_web/controllers/collections_controller.ex b/lib/lightning_web/controllers/collections_controller.ex index 76eb88f41cd..31024cb8c53 100644 --- a/lib/lightning_web/controllers/collections_controller.ex +++ b/lib/lightning_web/controllers/collections_controller.ex @@ -1,67 +1,48 @@ defmodule LightningWeb.CollectionsController do use LightningWeb, :controller + alias Ecto.Changeset alias Lightning.Collections - alias Lightning.Extensions.Message alias Lightning.Policies.Permissions + alias Lightning.Run + alias Lightning.Scope + alias LightningWeb.API.Errors + alias LightningWeb.API.FallbackController + alias LightningWeb.API.Filters + alias LightningWeb.API.Filters.Cursor + alias LightningWeb.API.Problem - action_fallback LightningWeb.FallbackController + action_fallback FallbackController require Logger @max_chunk_size 100 - @limits Application.compile_env!(:lightning, __MODULE__) - - @default_stream_limit @limits[:default_stream_limit] - @max_database_limit @limits[:max_database_limit] - - @timestamp_params [ - "created_after", - "created_before", - "updated_after", - "updated_before" - ] - - @valid_params ["key", "cursor", "limit" | @timestamp_params] + def put(conn, %{"name" => name, "key" => key}) do + with {:ok, collection} <- fetch_collection(conn, name, :put_collection_item) do + case Collections.put(collection, key, conn.body_params["value"]) do + :ok -> + json(conn, %{upserted: 1, error: nil}) - def put(conn, %{"name" => name, "key" => key, "value" => value}) do - with {:ok, collection} <- resolve(conn, name), - :ok <- authorize(conn, :put_collection_item, collection), - :ok <- Collections.put(collection, key, value) do - json(conn, %{upserted: 1, error: nil}) - else - {:error, %Ecto.Changeset{}} -> - json(conn, %{upserted: 0, error: "Format error"}) + {:error, %Changeset{} = changeset} -> + {:error, Errors.problem(changeset, %{}, path: [:key])} - error -> - maybe_handle_limit_error(conn, error) + error -> + error + end end end - def put(conn, %{"name" => _, "key" => _}), do: missing_body(conn, "value") - - def put_all(conn, %{"name" => name, "items" => items}) do - with {:ok, collection} <- resolve(conn, name), - :ok <- authorize(conn, :put_collection_item, collection), - {:ok, count} <- Collections.put_all(collection, items) do + def put_all(conn, %{"name" => name}) do + with {:ok, collection} <- fetch_collection(conn, name, :put_collection_item), + {:ok, count} <- + Collections.put_all(collection, conn.body_params["items"]) do json(conn, %{upserted: count, error: nil}) - else - {:error, :duplicate_key} -> - conn - |> put_status(:unprocessable_entity) - |> json(%{upserted: 0, error: "Duplicate key found"}) - - error -> - maybe_handle_limit_error(conn, error) end end - def put_all(conn, %{"name" => _}), do: missing_body(conn, "items") - def get(conn, %{"name" => name, "key" => key}) do - with {:ok, collection} <- resolve(conn, name), - :ok <- authorize(conn, :access_collection, collection) do + with {:ok, collection} <- fetch_collection(conn, name, :access_collection) do case Collections.get(collection, key) do nil -> resp(conn, :no_content, "") item -> json(conn, item) @@ -69,144 +50,68 @@ defmodule LightningWeb.CollectionsController do end end + # A missing item is a success with nothing in it (error-format spec §10). def delete(conn, %{"name" => name, "key" => key}) do - with {:ok, collection} <- resolve(conn, name), - :ok <- authorize(conn, :delete_collection_item, collection) do + with {:ok, collection} <- + fetch_collection(conn, name, :delete_collection_item) do case Collections.delete(collection, key) do - :ok -> - json(conn, %{key: key, deleted: 1, error: nil}) - - {:error, :not_found} -> - json(conn, %{key: key, deleted: 0, error: "Item Not Found"}) + :ok -> json(conn, %{key: key, deleted: 1}) + {:error, :not_found} -> json(conn, %{key: key, deleted: 0}) end end end def delete_all(conn, %{"name" => name} = params) do - with {:ok, collection} <- resolve(conn, name), - :ok <- authorize(conn, :delete_all_collection_items, collection) do + with {:ok, collection} <- + fetch_collection(conn, name, :delete_all_collection_items) do key_param = params["key"] {:ok, n} = Collections.delete_all(collection, key_param) - json(conn, %{key: key_param, deleted: n, error: nil}) + json(conn, %{key: key_param, deleted: n}) end end def stream(conn, %{"name" => name}) do - with {:ok, collection} <- resolve(conn, name), - :ok <- authorize(conn, :access_collection, collection), + with {:ok, collection} <- fetch_collection(conn, name, :access_collection), {:ok, filters} <- - parse_query_params(Map.drop(conn.query_params, ["project_id"])) do - key_pattern = conn.query_params["key"] - items_stream = stream_all_in_chunks(collection, filters, key_pattern) - response_limit = Map.fetch!(filters, :limit) + stream_filters(Map.delete(conn.query_params, "project_id")) do + {key_pattern, filters} = Map.pop(filters, :key) + response_limit = filters.limit - case stream_chunked(conn, items_stream, response_limit) do - {:error, conn} -> conn - {:ok, conn} -> conn - end - end - end - - def download(conn, %{"project_id" => project_id, "name" => name}) do - with {:ok, uuid} <- cast_uuid(project_id), - {:ok, collection} <- Collections.get_collection(uuid, name), - :ok <- authorize(conn, :access_collection, collection) do + # One item past the limit tells the response whether to carry a cursor. items_stream = - stream_all_in_chunks( + Collections.stream_all( collection, - %{cursor: nil, limit: @max_database_limit + 1}, - nil + %{filters | limit: response_limit + 1}, + key_pattern ) - conn - |> put_resp_content_type("application/json") - |> put_resp_header( - "content-disposition", - ~s(attachment; filename="#{name}.json") - ) - |> stream_as_json_array(items_stream) - end - end - - defp resolve(conn, name) do - case Map.get(conn.query_params, "project_id") do - nil -> - Collections.get_collection(name) - - project_id -> - with {:ok, uuid} <- cast_uuid(project_id) do - Collections.get_collection(uuid, name) - end - end - end - - defp cast_uuid(project_id) do - case Ecto.UUID.cast(project_id) do - {:ok, uuid} -> {:ok, uuid} - :error -> {:error, :bad_request} + case stream_chunked(conn, items_stream, response_limit) do + {:error, conn} -> conn + {:ok, conn} -> conn + end end end - defp authorize(conn, action, collection) do - subject = conn.assigns[:subject] || conn.assigns[:current_user] - - Permissions.can( - Lightning.Policies.Collections, - action, - subject, - collection - ) - end - - defp missing_body(conn, field) do - conn - |> put_status(:unprocessable_entity) - |> json(%{error: "Missing required field: #{field}"}) - end - - defp stream_as_json_array(conn, items_stream) do - conn = send_chunked(conn, 200) - {:ok, conn} = Plug.Conn.chunk(conn, "[") - - {conn, _first?} = - Enum.reduce_while(items_stream, {conn, true}, fn item, {conn, first?} -> - prefix = if first?, do: "", else: "," - - case Plug.Conn.chunk(conn, prefix <> Jason.encode!(item)) do - {:ok, conn} -> {:cont, {conn, false}} - {:error, :closed} -> {:halt, {conn, false}} - end - end) - - {:ok, conn} = Plug.Conn.chunk(conn, "]") - conn + defp fetch_collection(conn, name, action) do + subject = conn.assigns.subject + + with {:ok, collection} <- + Collections.resolve_collection( + subject, + name, + conn.query_params["project_id"] + ), + :ok <- authorize(subject, action, collection), + do: {:ok, collection} end - # Streams records from database without depending on holding a transaction from database pool. - # It streams one more than the limit to allow determining if there are more items for the response cursor. - defp stream_all_in_chunks(collection, %{limit: limit} = filters, key_pattern) - when limit <= @max_database_limit do - filters = Map.put(filters, :limit, limit + 1) + # A run is never asked Lightning.Policies.Collections. It may do anything to + # its own project's collections, and resolve_collection/3 finds it no others. + defp authorize(%Run{}, _action, _collection), do: :ok - Collections.get_all(collection, filters, key_pattern) - end - - defp stream_all_in_chunks( - collection, - %{cursor: initial_cursor} = filters, - key_pattern - ) do - filters = Map.put(filters, :limit, @max_database_limit + 1) - - Stream.unfold(initial_cursor, fn cursor -> - filters = Map.put(filters, :cursor, cursor) - - case Collections.get_all(collection, filters, key_pattern) do - [] -> nil - list -> {list, List.last(list).id} - end - end) - |> Stream.flat_map(& &1) + defp authorize(actor, action, collection) do + with {:ok, scope} <- Scope.fetch(actor, collection), + do: Permissions.permit(:collections, action, scope, collection) end defmodule ChunkAcc do @@ -230,55 +135,39 @@ defmodule LightningWeb.CollectionsController do end end - defp parse_query_params(query_params) do - query_params - |> Enum.into(%{ - "cursor" => nil, - "limit" => "#{@default_stream_limit}" - }) - |> validate_query_params() - end + # Every bad parameter is reported, in name order, and none is quoted + # (error-format spec §9.4). + defp stream_filters(query_params) do + {params, unknown} = + Map.split(query_params, Filters.collection_item_parameters()) - defp validate_query_params( - %{"cursor" => cursor, "limit" => limit} = query_params - ) do - with invalid_params when map_size(invalid_params) == 0 <- - Map.drop(query_params, @valid_params), - {:ok, cursor} <- validate_cursor(cursor), - {limit, ""} <- Integer.parse(limit), - :ok <- validate_timestamps(query_params) do - filters = - query_params - |> Map.new(fn {key, value} -> {String.to_existing_atom(key), value} end) - |> Map.put(:limit, limit) - |> Map.put(:cursor, cursor) - - {:ok, filters} - else - _invalid -> - {:error, :bad_request} - end + params + |> Filters.collection_items() + |> Errors.refuse_parameters() + |> refuse_unknown(Map.keys(unknown)) end - defp validate_cursor(nil), do: {:ok, nil} + defp refuse_unknown({:ok, _filters} = ok, []), do: ok - defp validate_cursor(cursor) do - with {:ok, decoded} <- Base.decode64(cursor), - {id, ""} <- Integer.parse(decoded) do - {:ok, id} - end - end + defp refuse_unknown(result, unknown) do + refused = + case result do + {:ok, _filters} -> [] + {:error, %Problem{errors: errors}} -> errors + end - defp validate_timestamps(params) do - params - |> Map.take(@timestamp_params) - |> Enum.all?(fn {_key, value} -> - match?({:ok, _dt, _time}, DateTime.from_iso8601(value)) - end) - |> case do - true -> :ok - false -> :error - end + entries = + for name <- unknown, + do: + Errors.entry( + {:parameter, name}, + {"is not accepted", code: :unknown_key} + ) + + {:error, + Problem.new(:invalid_parameter, + errors: Enum.sort_by(refused ++ entries, & &1.parameter) + )} end defp begin_chunking(conn) do @@ -297,10 +186,7 @@ defmodule LightningWeb.CollectionsController do end defp finish_chunking(%ChunkAcc{conn: conn, cursor_data: cursor_data}) do - cursor = - if cursor_data do - Base.encode64(to_string(cursor_data)) - end + cursor = if cursor_data, do: Cursor.encode(cursor_data) Plug.Conn.chunk(conn, ~S(], "cursor":) <> Jason.encode!(cursor) <> "}") end @@ -373,15 +259,4 @@ defmodule LightningWeb.CollectionsController do {:halt, {:error, conn}} end end - - defp maybe_handle_limit_error( - conn, - {:error, :exceeds_limit, %Message{text: error_msg}} - ) do - conn - |> put_status(:unprocessable_entity) - |> json(%{upserted: 0, error: error_msg}) - end - - defp maybe_handle_limit_error(_conn, error), do: error end diff --git a/lib/lightning_web/controllers/dataclip_controller.ex b/lib/lightning_web/controllers/dataclip_controller.ex index ff4149c2471..62a1ebe9496 100644 --- a/lib/lightning_web/controllers/dataclip_controller.ex +++ b/lib/lightning_web/controllers/dataclip_controller.ex @@ -9,7 +9,7 @@ defmodule LightningWeb.DataclipController do alias Lightning.Runs alias LightningWeb.WorkflowLive.NewManualRun - action_fallback LightningWeb.FallbackController + action_fallback LightningWeb.BrowserFallbackController @max_age 86_400 diff --git a/lib/lightning_web/controllers/downloads_controller.ex b/lib/lightning_web/controllers/downloads_controller.ex index ee98f044b6b..0b1f95bba37 100644 --- a/lib/lightning_web/controllers/downloads_controller.ex +++ b/lib/lightning_web/controllers/downloads_controller.ex @@ -5,7 +5,7 @@ defmodule LightningWeb.DownloadsController do alias Lightning.Policies.ProjectUsers alias Lightning.Projects - action_fallback(LightningWeb.FallbackController) + action_fallback(LightningWeb.BrowserFallbackController) def download_project_yaml(conn, %{"id" => id}) do with %Projects.Project{} = project <- diff --git a/lib/lightning_web/controllers/oidc_controller.ex b/lib/lightning_web/controllers/oidc_controller.ex index ce7cd4fb845..aa566666e07 100644 --- a/lib/lightning_web/controllers/oidc_controller.ex +++ b/lib/lightning_web/controllers/oidc_controller.ex @@ -7,7 +7,7 @@ defmodule LightningWeb.OidcController do alias LightningWeb.OauthCredentialHelper alias LightningWeb.UserAuth - action_fallback LightningWeb.FallbackController + action_fallback LightningWeb.BrowserFallbackController plug :fetch_current_user diff --git a/lib/lightning_web/controllers/project_file_controller.ex b/lib/lightning_web/controllers/project_file_controller.ex index e6712ef3bcd..633f88d5a03 100644 --- a/lib/lightning_web/controllers/project_file_controller.ex +++ b/lib/lightning_web/controllers/project_file_controller.ex @@ -9,7 +9,7 @@ defmodule LightningWeb.ProjectFileController do # Without this the `with` below returns a bare {:error, :forbidden} and never # sends a response. Pre-existing, but the denial branch was near-unreachable # until project liveness started being enforced here. - action_fallback LightningWeb.FallbackController + action_fallback LightningWeb.BrowserFallbackController # sobelow_skip ["Traversal.SendDownload"] # Path is safe: generated by storage_path_for_exports/2 using system UUIDs only diff --git a/lib/lightning_web/controllers/token_exchange_controller.ex b/lib/lightning_web/controllers/token_exchange_controller.ex index e14f9b58a1d..f017f8d0e4d 100644 --- a/lib/lightning_web/controllers/token_exchange_controller.ex +++ b/lib/lightning_web/controllers/token_exchange_controller.ex @@ -81,19 +81,15 @@ defmodule LightningWeb.TokenExchangeController do defp client_assertion(_params), do: {:error, :invalid_request} defp requested_scopes(%{"scope" => scope}) when is_binary(scope) do - case String.split(scope) do - [] -> - {:ok, AccessToken.scopes()} - - scopes -> - if Enum.all?(scopes, &(&1 in AccessToken.scopes())), - do: {:ok, Enum.uniq(scopes)}, - else: {:error, :invalid_scope} - end + scopes = String.split(scope) + + if scopes != [] and Enum.all?(scopes, &(&1 in AccessToken.scopes())), + do: {:ok, Enum.uniq(scopes)}, + else: {:error, :invalid_scope} end defp requested_scopes(%{"scope" => _}), do: {:error, :invalid_request} - defp requested_scopes(_params), do: {:ok, AccessToken.scopes()} + defp requested_scopes(_params), do: {:error, :invalid_scope} defp authenticate(assertion, params) do case Assertion.verify(assertion, url(~p"/api/oauth/token")) do diff --git a/lib/lightning_web/controllers/user_auth.ex b/lib/lightning_web/controllers/user_auth.ex index 40c70399fc6..5b7f0c63eed 100644 --- a/lib/lightning_web/controllers/user_auth.ex +++ b/lib/lightning_web/controllers/user_auth.ex @@ -11,6 +11,7 @@ defmodule LightningWeb.UserAuth do alias Lightning.Accounts.User alias Lightning.Accounts.UserToken alias Lightning.VersionControl.ProjectRepoConnection + alias LightningWeb.Auth alias LightningWeb.Router.Helpers, as: Routes # Make the remember me cookie valid for 60 days. @@ -239,7 +240,7 @@ defmodule LightningWeb.UserAuth do end def authenticate_bearer(conn, _opts) do - with {:ok, bearer_token} <- get_bearer(conn), + with {:ok, bearer_token} <- Auth.bearer_token(conn), %{__struct__: type} = resource <- get_current_resource(bearer_token) do if type == User do update_last_used(bearer_token) @@ -251,15 +252,6 @@ defmodule LightningWeb.UserAuth do end end - def get_bearer(conn) do - conn - |> get_req_header("authorization") - |> case do - ["Bearer " <> bearer_token] -> {:ok, bearer_token} - _ -> {:error, "Bearer Token not found"} - end - end - defp get_current_resource(token) do Accounts.get_user_by_api_token(token) || Lightning.VersionControl.get_repo_connection_for_token(token) @@ -290,7 +282,7 @@ defmodule LightningWeb.UserAuth do is_nil(conn.assigns[:current_user]) -> case get_format(conn) do "json" -> - render_unauthorized(conn) + Auth.refuse(conn, :session) _ -> conn @@ -313,7 +305,7 @@ defmodule LightningWeb.UserAuth do defp refuse_totp_pending(conn) do case get_format(conn) do "json" -> - render_unauthorized(conn) + Auth.refuse(conn, :session) _ -> conn @@ -332,7 +324,7 @@ defmodule LightningWeb.UserAuth do defp refuse_locked_out(conn) do case get_format(conn) do "json" -> - render_unauthorized(conn) + Auth.refuse(conn, :session) _ -> conn @@ -357,24 +349,18 @@ defmodule LightningWeb.UserAuth do def require_authenticated_api_resource(conn, _opts) do case conn.assigns[:current_resource] do nil -> - render_unauthorized(conn) + Auth.refuse(conn, :bearer) %User{} = user -> - if Accounts.locked_out?(user), do: render_unauthorized(conn), else: conn + if Accounts.locked_out?(user), + do: Auth.refuse(conn, :bearer), + else: conn %ProjectRepoConnection{} -> conn end end - defp render_unauthorized(conn) do - conn - |> put_status(:unauthorized) - |> put_view(LightningWeb.ErrorView) - |> render(:"401") - |> halt() - end - defp put_user_token(conn, _ \\ nil) do session_token = get_session(conn, :user_token) @@ -419,21 +405,9 @@ defmodule LightningWeb.UserAuth do _ -> conn - |> get_format() - |> case do - "json" -> - conn - |> put_status(:forbidden) - |> put_view(LightningWeb.ErrorView) - |> render(:"403") - |> halt() - - _ -> - conn - |> put_flash(:nav, :no_access_no_back) - |> redirect(to: signed_in_path(conn)) - |> halt() - end + |> put_flash(:nav, :no_access_no_back) + |> redirect(to: signed_in_path(conn)) + |> halt() end end diff --git a/lib/lightning_web/controllers/webhooks_controller.ex b/lib/lightning_web/controllers/webhooks_controller.ex index fa9c5050bc4..363b40de5af 100644 --- a/lib/lightning_web/controllers/webhooks_controller.ex +++ b/lib/lightning_web/controllers/webhooks_controller.ex @@ -1,12 +1,16 @@ defmodule LightningWeb.WebhooksController do use LightningWeb, :controller + alias Lightning.Extensions.Message alias Lightning.Extensions.RateLimiting alias Lightning.Invocation.RequestHeaders alias Lightning.Retry alias Lightning.Services.RateLimiter alias Lightning.Workflows alias Lightning.WorkOrders + alias LightningWeb.API.Codes + alias LightningWeb.API.Errors + alias LightningWeb.API.Problem require Logger @@ -15,11 +19,7 @@ defmodule LightningWeb.WebhooksController do defp reject_unfetched(conn, _) do case conn.body_params do %Plug.Conn.Unfetched{} -> - conn - |> put_status(415) - |> put_view(LightningWeb.ErrorView) - |> render(:"415") - |> halt() + Problem.halt(conn, Problem.new(:unsupported_media_type)) _ -> conn @@ -78,44 +78,39 @@ defmodule LightningWeb.WebhooksController do workflow_id: trigger.workflow.id, project_id: project_id }, - message: - "Unable to process request due to temporary database issues. Please try again in %{s}s.", halt?: false ) - {:error, %Ecto.Changeset{} = changeset} -> - errors = Ecto.Changeset.traverse_errors(changeset, fn {m, _} -> m end) - - conn - |> put_status(:unprocessable_entity) - |> json(%{error: :invalid_request, details: errors}) - - {:error, reason} when is_atom(reason) -> - conn - |> put_status(:unprocessable_entity) - |> json(%{error: reason}) + # The work order is built from the trigger and the workflow, not shaped + # by the request, so its errors have no place in the body to point at + # and the walker would raise on them. The body as a whole is what could + # not be saved, at 422 as /i/* always answered (spec §12). + {:error, reason} + when is_atom(reason) or is_struct(reason, Ecto.Changeset) -> + entry = Errors.entry([], {"is invalid", code: :invalid}) + Problem.respond(conn, Problem.new(:invalid, errors: [entry])) end else - {:error, reason, %{text: message}} -> - conn - |> put_status(:too_many_requests) - |> json(%{error: reason, message: message}) + {:error, reason, %Message{}} -> + Problem.respond(conn, limited(reason)) nil -> - conn - |> put_status(:not_found) - |> json(%{"error" => "Webhook not found"}) + Problem.respond(conn, Problem.new(:not_found)) _disabled -> - put_status(conn, :forbidden) - |> json(%{ - error: :trigger_disabled, - message: - "Unable to process request, trigger is disabled. Enable it on OpenFn to allow requests to this endpoint." - }) + Problem.respond(conn, Problem.new(:trigger_disabled)) end end + # Every limiter refusal on /i/* is a 429, whatever the reason, and an unknown + # project's is `exceeds_limit` where the API's is 404 (spec §12). + defp limited(reason, opts \\ []) + + defp limited(:unknown_project, opts), do: limited(:exceeds_limit, opts) + + defp limited(reason, opts), + do: Problem.new(Codes.for_limiter(reason), [status: 429] ++ opts) + defp put_work_order_headers(conn, work_order) do conn |> put_resp_header("x-meta-work-order-id", work_order.id) @@ -149,12 +144,10 @@ defmodule LightningWeb.WebhooksController do end end - # The limiter's own message is deliberately not echoed here: it is copy written - # for the project UI, and `/i/*` is anonymous whenever the trigger carries no - # auth methods. The reason code says why no run was created without disclosing - # a project's plan or quota wording to whoever holds the webhook URL. + # The one success body that carries `error`: the payload was saved, so a 4xx + # would make senders resend it (spec §10). defp rejection_details(nil), do: %{} - defp rejection_details(reason) when is_atom(reason), do: %{error: reason} + defp rejection_details({reason, %Message{}}), do: %{error: reason} defp has_run?(%{runs: [_ | _]}), do: true defp has_run?(_work_order), do: false @@ -163,26 +156,23 @@ defmodule LightningWeb.WebhooksController do # by a completion broadcast, so reply now with the reason the run was not # created rather than holding the connection open for the full timeout. defp respond_without_run(conn, work_order, run_rejection) do - status = - if is_nil(run_rejection) do - Logger.warning( - "Synchronous webhook work order created without a run: " <> - inspect(work_order.id) - ) + extensions = %{work_order_id: work_order.id} + + problem = + case run_rejection do + {reason, _message} -> + limited(reason, extensions: extensions) + + nil -> + Logger.warning( + "Synchronous webhook work order created without a run: " <> + inspect(work_order.id) + ) - :internal_server_error - else - :too_many_requests + Problem.new(:internal, extensions: extensions) end - conn - |> put_status(status) - |> json( - Map.merge( - %{work_order_id: work_order.id}, - rejection_details(run_rejection || :no_run_created) - ) - ) + Problem.respond(conn, problem) end defp handle_delayed_response(conn, work_order) do @@ -203,27 +193,24 @@ defmodule LightningWeb.WebhooksController do "Webhook response timeout for work_order: #{inspect(work_order.id)}" ) - conn - |> put_status(:gateway_timeout) - |> json(%{ - error: :timeout, - message: "Workflow did not complete within timeout period", - work_order_id: work_order.id - }) + Problem.respond( + conn, + Problem.new(:timeout, extensions: %{work_order_id: work_order.id}) + ) end end # `{:error, :too_many_runs, _}` is not fatal: the payload is still recorded as - # a rejected work order, just without a run. The reason travels with that + # a rejected work order, just without a run. The refusal travels with that # decision instead of being collapsed into a bare boolean, so the response can - # say why no run was created. + # say why no run was created, and when to try again. defp check_skip_run_creation(project_id) do case WorkOrders.limit_run_creation(project_id) do :ok -> {:ok, nil} - {:error, :too_many_runs, _message} -> - {:ok, :too_many_runs} + {:error, :too_many_runs, message} -> + {:ok, {:too_many_runs, message}} error -> error diff --git a/lib/lightning_web/controllers/workflow_controller.ex b/lib/lightning_web/controllers/workflow_controller.ex index 01c40d4328b..275bedcb04c 100644 --- a/lib/lightning_web/controllers/workflow_controller.ex +++ b/lib/lightning_web/controllers/workflow_controller.ex @@ -9,7 +9,7 @@ defmodule LightningWeb.WorkflowController do alias Lightning.WorkOrders alias LightningWeb.WorkflowLive.Helpers - action_fallback LightningWeb.FallbackController + action_fallback LightningWeb.BrowserFallbackController @doc """ Create a manual workflow run. @@ -266,7 +266,7 @@ defmodule LightningWeb.WorkflowController do # reason atom; hand it to the fallback controller instead, so this route # answers a refused caller the way the rest of the app does. {:error, :unauthorized} = error -> - LightningWeb.FallbackController.call(conn, error) + LightningWeb.BrowserFallbackController.call(conn, error) {:error, reason} -> conn diff --git a/lib/lightning_web/endpoint.ex b/lib/lightning_web/endpoint.ex index cfdb8cc8f1e..268fa290136 100644 --- a/lib/lightning_web/endpoint.ex +++ b/lib/lightning_web/endpoint.ex @@ -1,7 +1,21 @@ defmodule LightningWeb.Endpoint do + @moduledoc """ + The HTTP entry point. It declares the sockets and the plugs every request + passes through before `LightningWeb.Router`. + + A raise on a problem path is answered in `LightningWeb.Endpoint.call/2` and + nowhere else. `ProblemPaths.answers?/2` is the only place that says which + raises qualify (error-format spec §5), and `Codes.for_exception/2` the only + place that says which code a raise gets: its own status if a client caused + it, and a 500 for anything a client did not cause. + """ + use Sentry.PlugCapture use Phoenix.Endpoint, otp_app: :lightning + alias LightningWeb.API.Codes + alias LightningWeb.API.Problem + alias LightningWeb.API.ProblemPaths alias LightningWeb.Plugs require LightningWeb.Utils @@ -66,9 +80,11 @@ defmodule LightningWeb.Endpoint do param_key: "request_logger", cookie_key: "request_logger" + @cors_options [origin: &Lightning.Config.cors_origin/0] + plug Plug.RequestId - plug CORSPlug, origin: &Lightning.Config.cors_origin/0 + plug CORSPlug, @cors_options plug Plug.Telemetry, event_prefix: [:phoenix, :endpoint] @@ -107,4 +123,74 @@ defmodule LightningWeb.Endpoint do LightningWeb.Utils.add_dynamic_plugs(@post_session_plugs) plug LightningWeb.Router + + # Wraps the plug pipeline from inside Sentry's and Phoenix's own call/2 + # wrappers, so it sees the request's path and Accept whether the raise came + # before the router or inside it. + def call(conn, opts) do + super(conn, opts) + rescue + e in Plug.Conn.WrapperError -> + %{conn: conn, kind: kind, reason: reason, stack: stack} = e + + case answer_raise(conn, kind, reason, stack, & &1) do + # Keep the wrapper so Phoenix renders with the router's conn and format. + :raise -> reraise e, __STACKTRACE__ + conn -> conn + end + catch + kind, reason -> + case answer_raise(conn, kind, reason, __STACKTRACE__, &put_entry_headers/1) do + :raise -> :erlang.raise(kind, reason, __STACKTRACE__) + conn -> conn + end + end + + # A raise, exit or throw that Plug did not wrap leaves only the conn that + # entered call/2, so an answer to it lacks the headers the endpoint's plugs + # set. The preflight response is off so that an OPTIONS request still gets + # the error, not a 204. + defp put_entry_headers(conn) do + conn + |> put_request_id() + |> CORSPlug.call( + CORSPlug.init([send_preflight_response?: false] ++ @cors_options) + ) + end + + # Plug.RequestId would mint a fresh id when the request sent none, and the + # answer would not match the id the request's log lines already carry. + defp put_request_id(conn) do + case Logger.metadata()[:request_id] do + nil -> Plug.RequestId.call(conn, Plug.RequestId.init([])) + request_id -> Plug.Conn.put_resp_header(conn, "x-request-id", request_id) + end + end + + defp answer_raise(conn, kind, reason, stack, prepare) do + exception = Exception.normalize(kind, reason, stack) + + if ProblemPaths.answers?(conn, exception) do + code = Codes.for_exception(kind, exception) + conn = respond_unless_sent(prepare.(conn), Problem.new(code)) + + # Raise the 500 after answering it, so Sentry and Phoenix still log the + # crash. + if code == :internal, do: :raise, else: Plug.Conn.halt(conn) + else + :raise + end + end + + # The same check Phoenix.Endpoint.RenderErrors makes: send_resp/3 tells the + # owner process, so a response a plug sent before raising is left alone. + defp respond_unless_sent(conn, problem) do + receive do + {:plug_conn, :sent} -> + send(self(), {:plug_conn, :sent}) + %Plug.Conn{conn | state: :sent} + after + 0 -> Problem.respond(conn, problem) + end + end end diff --git a/lib/lightning_web/hooks.ex b/lib/lightning_web/hooks.ex index 2358270237a..a4ec91fc549 100644 --- a/lib/lightning_web/hooks.ex +++ b/lib/lightning_web/hooks.ex @@ -20,7 +20,7 @@ defmodule LightningWeb.Hooks do alias Lightning.Projects.Events.SupportAccessUpdated alias Lightning.Projects.Events.WorkflowDeleted alias Lightning.Projects.ProjectLimiter - alias Lightning.Projects.Scope + alias Lightning.Scope alias Lightning.Services.UsageLimiter alias Lightning.VersionControl.VersionControlUsageLimiter alias LightningWeb.LiveHelpers @@ -124,8 +124,8 @@ defmodule LightningWeb.Hooks do redirect(socket, to: "/projects") |> put_flash(:nav, :not_found)} end - # No such project, or one scheduled for deletion. Both were already the - # not-found redirect before Scope answered them here. + # No such project. A project scheduled for deletion loads instead, and + # ends in the same not-found redirect above. {:error, _reason} -> {:halt, redirect(socket, to: "/projects") |> put_flash(:nav, :not_found)} end @@ -287,10 +287,9 @@ defmodule LightningWeb.Hooks do end # The project is wound down. Every socket on it is in scope, whatever standing - # it holds, and no later change can bring the project back — `Scope` refuses - # it for good — so there is nothing to re-mount into. Leave the project - # rather than bouncing through a mount that would only redirect again with a - # less useful message. + # it holds. The policies refuse it while it stays scheduled, so there is + # nothing to re-mount into. Leave the project rather than bouncing through a + # mount that would only redirect again with a less useful message. defp handle_project_user_event(%ProjectDeletionScheduled{} = event, socket) do destination = if Lightning.Accounts.experimental_features_enabled?( diff --git a/lib/lightning_web/live/channel_live/form_component.ex b/lib/lightning_web/live/channel_live/form_component.ex index 7042271f8c5..e881ebd0cd9 100644 --- a/lib/lightning_web/live/channel_live/form_component.ex +++ b/lib/lightning_web/live/channel_live/form_component.ex @@ -269,6 +269,12 @@ defmodule LightningWeb.ChannelLive.FormComponent do |> put_flash(:info, "Channel updated successfully.") |> push_patch(to: ~p"/projects/#{socket.assigns.project}/channels")} + {:error, :concurrent_edit} -> + {:noreply, + socket + |> put_flash(:error, save_error_message(:concurrent_edit)) + |> push_patch(to: ~p"/projects/#{socket.assigns.project}/channels")} + {:error, changeset} -> {:noreply, assign(socket, changeset: changeset)} end diff --git a/lib/lightning_web/live/channel_live/index.ex b/lib/lightning_web/live/channel_live/index.ex index 169fb7f7176..e95fc99da14 100644 --- a/lib/lightning_web/live/channel_live/index.ex +++ b/lib/lightning_web/live/channel_live/index.ex @@ -198,6 +198,11 @@ defmodule LightningWeb.ChannelLive.Index do |> push_patch(to: ~p"/projects/#{socket.assigns.project.id}/channels") |> noreply() + {:error, :concurrent_edit} -> + socket + |> put_flash(:error, save_error_message(:concurrent_edit)) + |> noreply() + {:error, _changeset} -> socket |> put_flash(:error, "Failed to update channel. Please try again.") diff --git a/lib/lightning_web/live/collection_live/collection_creation_modal.ex b/lib/lightning_web/live/collection_live/collection_creation_modal.ex index 6648c80244a..1503014b8d4 100644 --- a/lib/lightning_web/live/collection_live/collection_creation_modal.ex +++ b/lib/lightning_web/live/collection_live/collection_creation_modal.ex @@ -54,18 +54,21 @@ defmodule LightningWeb.CollectionLive.CollectionCreationModal do end def handle_event("save", %{"collection" => collection_params}, socket) do - %{mode: mode, return_to: return_to} = socket.assigns + %{mode: mode, return_to: return_to, current_user: current_user} = + socket.assigns + collection_params = Helpers.derive_name_param(collection_params) result = case mode do :create -> - Collections.create_collection(collection_params) + Collections.create_collection(collection_params, current_user) :update -> Collections.update_collection( socket.assigns.collection, - collection_params + collection_params, + current_user ) end @@ -82,6 +85,12 @@ defmodule LightningWeb.CollectionLive.CollectionCreationModal do |> put_flash(:error, error_msg) |> push_navigate(to: return_to)} + {:error, :concurrent_edit} -> + {:noreply, + socket + |> put_flash(:error, save_error_message(:concurrent_edit)) + |> push_navigate(to: return_to)} + {:error, changeset} -> {:noreply, assign( @@ -148,6 +157,7 @@ defmodule LightningWeb.CollectionLive.CollectionCreationModal do field={f[:project_id]} label="Project" options={@projects_options} + disabled={@mode == :update} required="true" /> diff --git a/lib/lightning_web/live/collection_live/components.ex b/lib/lightning_web/live/collection_live/components.ex index 31659176641..0a0fc92d0a3 100644 --- a/lib/lightning_web/live/collection_live/components.ex +++ b/lib/lightning_web/live/collection_live/components.ex @@ -139,6 +139,7 @@ defmodule LightningWeb.CollectionLive.Components do id={"update-collection-#{collection.id}-modal"} module={LightningWeb.CollectionLive.CollectionCreationModal} collection={collection} + current_user={@user} mode={:update} return_to={~p"/settings/collections"} /> diff --git a/lib/lightning_web/live/collection_live/index.ex b/lib/lightning_web/live/collection_live/index.ex index 06889aca4c1..5794194eb40 100644 --- a/lib/lightning_web/live/collection_live/index.ex +++ b/lib/lightning_web/live/collection_live/index.ex @@ -109,6 +109,7 @@ defmodule LightningWeb.CollectionLive.Index do id="create-collection-modal" module={LightningWeb.CollectionLive.CollectionCreationModal} collection={%Collection{}} + current_user={@current_user} return_to={~p"/settings/collections"} /> diff --git a/lib/lightning_web/live/components/credential_deletion_modal.ex b/lib/lightning_web/live/components/credential_deletion_modal.ex index 53aa13098eb..d49b1fe38a4 100644 --- a/lib/lightning_web/live/components/credential_deletion_modal.ex +++ b/lib/lightning_web/live/components/credential_deletion_modal.ex @@ -38,36 +38,36 @@ defmodule LightningWeb.Components.CredentialDeletionModal do end defp delete_or_schedule(credential, socket) do - cond do - not socket.assigns.delete_now? -> - case Credentials.schedule_credential_deletion( - credential, - socket.assigns.current_user - ) do - {:ok, %Credential{}} -> - {:noreply, - socket - |> put_flash(:info, "Credential scheduled for deletion") - |> push_navigate(to: socket.assigns.return_to)} - - {:error, :unauthorized} -> - {:noreply, unauthorized(socket)} - - {:error, %Ecto.Changeset{} = _changeset} -> - {:noreply, socket} - end - - socket.assigns.has_activity_in_projects? -> + if socket.assigns.delete_now?, + do: delete_now(credential, socket), + else: schedule(credential, socket) + end + + defp schedule(credential, socket) do + case Credentials.schedule_credential_deletion( + credential, + socket.assigns.current_user + ) do + {:ok, %Credential{}} -> + {:noreply, + socket + |> put_flash(:info, "Credential scheduled for deletion") + |> push_navigate(to: socket.assigns.return_to)} + + {:error, :unauthorized} -> + {:noreply, unauthorized(socket)} + + {:error, :scheduled_for_deletion} -> {:noreply, socket |> put_flash( :error, - "Cannot delete a credential that has activities in projects" + "This credential is already scheduled for deletion" ) - |> push_patch(to: socket.assigns.return_to)} + |> push_navigate(to: socket.assigns.return_to)} - true -> - delete_now(credential, socket) + {:error, %Ecto.Changeset{} = _changeset} -> + {:noreply, socket} end end @@ -89,6 +89,15 @@ defmodule LightningWeb.Components.CredentialDeletionModal do {:error, :unauthorized} -> {:noreply, unauthorized(socket)} + {:error, :in_use} -> + {:noreply, + socket + |> put_flash( + :error, + "Cannot delete a credential that has activities in projects" + ) + |> push_patch(to: socket.assigns.return_to)} + _error -> {:noreply, socket diff --git a/lib/lightning_web/live/credential_live/credential_form_component.ex b/lib/lightning_web/live/credential_live/credential_form_component.ex index 4e0fe2702cb..e04c56c7a72 100644 --- a/lib/lightning_web/live/credential_live/credential_form_component.ex +++ b/lib/lightning_web/live/credential_live/credential_form_component.ex @@ -1017,6 +1017,11 @@ defmodule LightningWeb.CredentialLive.CredentialFormComponent do workflows_using_credentials={@workflows_using_credentials} phx_target={@myself} /> + <.error :for={ + msg <- translate_errors(@changeset.errors, :project_credentials) + }> + {msg} +
put_flash(:error, "You are not authorized to do that.") |> close_or_stay() + {:error, :scheduled_for_deletion} -> + socket + |> put_flash( + :error, + "This credential is scheduled for deletion. Cancel the deletion to edit it." + ) + |> close_or_stay() + + {:error, :concurrent_edit} -> + socket + |> put_flash(:error, save_error_message(:concurrent_edit)) + |> close_or_stay() + {:error, %Ecto.Changeset{} = changeset} -> {:noreply, assign(socket, :changeset, changeset)} end diff --git a/lib/lightning_web/live/credential_live/credential_index_component.ex b/lib/lightning_web/live/credential_live/credential_index_component.ex index 888ba504870..5be72b5378d 100644 --- a/lib/lightning_web/live/credential_live/credential_index_component.ex +++ b/lib/lightning_web/live/credential_live/credential_index_component.ex @@ -3,7 +3,9 @@ defmodule LightningWeb.CredentialLive.CredentialIndexComponent do use LightningWeb, :live_component alias Lightning.Credentials + alias Lightning.Credentials.Credential alias Lightning.Credentials.KeychainCredential + alias Lightning.Credentials.OauthClient alias Lightning.OauthClients alias Lightning.Policies @@ -389,9 +391,9 @@ defmodule LightningWeb.CredentialLive.CredentialIndexComponent do ) end - defp can_delete_credential(current_user, credential) do + defp can_delete_credential(current_user, %Credential{} = credential) do Policies.Permissions.can?( - :users, + :credentials, :delete_credential, current_user, credential @@ -407,15 +409,24 @@ defmodule LightningWeb.CredentialLive.CredentialIndexComponent do ) end - defp can_edit_credential(current_user, credential) do + defp can_edit_credential(current_user, %Credential{} = credential) do Policies.Permissions.can?( - :users, + :credentials, :edit_credential, current_user, credential ) end + defp can_edit_credential(current_user, %OauthClient{} = client) do + Policies.Permissions.can?( + :users, + :edit_credential, + current_user, + client + ) + end + defp not_authorized(socket) do {:noreply, socket diff --git a/lib/lightning_web/live/credential_live/credential_index_component.html.heex b/lib/lightning_web/live/credential_live/credential_index_component.html.heex index cf4ea55a0b2..42d08494af5 100644 --- a/lib/lightning_web/live/credential_live/credential_index_component.html.heex +++ b/lib/lightning_web/live/credential_live/credential_index_component.html.heex @@ -69,7 +69,10 @@ /> <:actions :let={credential}> -
+
<:options> <.link + :if={can_edit_credential(@current_user, credential)} id={"credential-actions-#{credential.id}-transfer"} phx-target={@myself} phx-click="request_credential_transfer" @@ -94,6 +98,7 @@ <% end %> <.link + :if={can_edit_credential(@current_user, credential)} id={"credential-actions-#{credential.id}-edit"} phx-target={@myself} phx-click="edit_credential" @@ -102,6 +107,7 @@ Edit <.delete_action + :if={can_delete_credential(@current_user, credential)} myself={@myself} socket={@socket} credential={credential} diff --git a/lib/lightning_web/live/dashboard_live/project_creation_modal.ex b/lib/lightning_web/live/dashboard_live/project_creation_modal.ex index fbb2b1f31c4..680c940860c 100644 --- a/lib/lightning_web/live/dashboard_live/project_creation_modal.ex +++ b/lib/lightning_web/live/dashboard_live/project_creation_modal.ex @@ -52,7 +52,7 @@ defmodule LightningWeb.DashboardLive.ProjectCreationModal do "role" => "owner" } }) - |> Projects.create_project(false) + |> Projects.create_project(current_user, notify: false) |> case do {:ok, project} -> {:noreply, diff --git a/lib/lightning_web/live/live_helpers.ex b/lib/lightning_web/live/live_helpers.ex index 27c4d700604..b9ed578fdaa 100644 --- a/lib/lightning_web/live/live_helpers.ex +++ b/lib/lightning_web/live/live_helpers.ex @@ -210,6 +210,17 @@ defmodule LightningWeb.LiveHelpers do end end + @doc """ + Why a form's save failed when the reason isn't the user's input. + """ + @spec save_error_message(:concurrent_edit | :audit_failed) :: String.t() + def save_error_message(:concurrent_edit), + do: + "Someone else changed or deleted this record while you were editing it. Reload and try again." + + def save_error_message(:audit_failed), + do: "Changes couldn't be saved, please try again" + @doc """ Refuses a collaborator change the current user is not allowed to make. diff --git a/lib/lightning_web/live/project_live/collections_component.ex b/lib/lightning_web/live/project_live/collections_component.ex index 6f689404d52..bbc68537a09 100644 --- a/lib/lightning_web/live/project_live/collections_component.ex +++ b/lib/lightning_web/live/project_live/collections_component.ex @@ -197,7 +197,7 @@ defmodule LightningWeb.ProjectLive.CollectionsComponent do defp save_collection(socket, :new, params) do params |> Map.put("project_id", socket.assigns.project.id) - |> Collections.create_collection() + |> Collections.create_collection(socket.assigns.current_user) |> case do {:ok, _collection} -> socket @@ -219,12 +219,21 @@ defmodule LightningWeb.ProjectLive.CollectionsComponent do end defp save_collection(socket, :edit, params) do - case Collections.update_collection(socket.assigns.collection, params) do + case Collections.update_collection( + socket.assigns.collection, + params, + socket.assigns.current_user + ) do {:ok, _collection} -> socket |> put_flash(:info, "Collection updated") |> push_navigate(to: socket.assigns.return_to) + {:error, :concurrent_edit} -> + socket + |> put_flash(:error, save_error_message(:concurrent_edit)) + |> push_navigate(to: socket.assigns.return_to) + {:error, changeset} -> assign( socket, diff --git a/lib/lightning_web/live/project_live/form_component.ex b/lib/lightning_web/live/project_live/form_component.ex index 21e58442c3d..62616aa7317 100644 --- a/lib/lightning_web/live/project_live/form_component.ex +++ b/lib/lightning_web/live/project_live/form_component.ex @@ -181,7 +181,7 @@ defmodule LightningWeb.ProjectLive.FormComponent do end defp save_project(socket, :new, project_params) do - case Projects.create_project(project_params) do + case Projects.create_project(project_params, socket.assigns.current_user) do {:ok, _project} -> {:noreply, socket diff --git a/lib/lightning_web/live/project_live/github_sync_component.ex b/lib/lightning_web/live/project_live/github_sync_component.ex index 89a8929aa0b..9c19c68c566 100644 --- a/lib/lightning_web/live/project_live/github_sync_component.ex +++ b/lib/lightning_web/live/project_live/github_sync_component.ex @@ -322,7 +322,7 @@ defmodule LightningWeb.ProjectLive.GithubSyncComponent do branch_errors = Keyword.get_values(changeset.errors, :branch) if Enum.any?(branch_errors, fn {_msg, opts} -> - Keyword.get(opts, :reason) == :tree_branch_conflict + Keyword.get(opts, :code) == :tree_branch_conflict end) do Map.put(changeset, :action, :validate) else diff --git a/lib/lightning_web/live/project_live/settings.ex b/lib/lightning_web/live/project_live/settings.ex index 6d2c3cc5ffb..d6e512ef948 100644 --- a/lib/lightning_web/live/project_live/settings.ex +++ b/lib/lightning_web/live/project_live/settings.ex @@ -380,28 +380,16 @@ defmodule LightningWeb.ProjectLive.Settings do def handle_event("toggle_support_access", _params, socket) do case authorize(socket, :edit_project) do :ok -> - project = socket.assigns.project - - {:ok, project} = - Projects.update_project( - project, - %{ - allow_support_access: !project.allow_support_access - }, - socket.assigns.current_user + granting? = !socket.assigns.project.allow_support_access + + toggle_project( + socket, + %{allow_support_access: granting?}, + if(granting?, + do: "Granted access to support users successfully", + else: "Revoked access to support users successfully" ) - - flash_msg = - if project.allow_support_access do - "Granted access to support users successfully" - else - "Revoked access to support users successfully" - end - - {:noreply, - socket - |> assign(:project, project) - |> put_flash(:info, flash_msg)} + ) {:error, :unauthorized} -> deny_action(socket, "toggle_support_access") @@ -412,19 +400,11 @@ defmodule LightningWeb.ProjectLive.Settings do # `can_require_mfa` comes from the `:limit_mfa` hook - a usage limit rather # than a role-derived boolean if :ok == authorize(socket, :edit_project) and socket.assigns.can_require_mfa do - %{project: project, current_user: current_user} = socket.assigns - - {:ok, project} = - Projects.update_project( - project, - %{requires_mfa: !project.requires_mfa}, - current_user - ) - - {:noreply, - socket - |> assign(:project, project) - |> put_flash(:info, "Project MFA requirement updated successfully")} + toggle_project( + socket, + %{requires_mfa: !socket.assigns.project.requires_mfa}, + "Project MFA requirement updated successfully" + ) else deny_action(socket, "toggle-mfa") end @@ -782,10 +762,24 @@ defmodule LightningWeb.ProjectLive.Settings do changeset = Helpers.copy_error(changeset, :name, :raw_name) {:noreply, assign(socket, :project_changeset, changeset)} - {:error, :not_related_to_project} -> + {:error, reason} when reason in [:audit_failed, :concurrent_edit] -> + {:noreply, put_flash(socket, :error, save_error_message(reason))} + end + end + + defp toggle_project(socket, attrs, success) do + case Projects.update_project( + socket.assigns.project, + attrs, + socket.assigns.current_user + ) do + {:ok, project} -> + {:noreply, + socket |> assign(:project, project) |> put_flash(:info, success)} + + {:error, _reason} -> {:noreply, - socket - |> put_flash(:error, "Changes couldn't be saved, please try again")} + put_flash(socket, :error, "Changes couldn't be saved, please try again")} end end diff --git a/lib/lightning_web/live/sandbox_live/form_component.ex b/lib/lightning_web/live/sandbox_live/form_component.ex index 417649b7bc1..b4063e0fd43 100644 --- a/lib/lightning_web/live/sandbox_live/form_component.ex +++ b/lib/lightning_web/live/sandbox_live/form_component.ex @@ -135,6 +135,18 @@ defmodule LightningWeb.SandboxLive.FormComponent do |> put_flash(:info, "Sandbox updated") |> push_navigate(to: return_to || ~p"/projects/#{sandbox.id}/w")} + {:error, reason} when reason in [:concurrent_edit, :audit_failed] -> + {:noreply, + socket + |> put_flash(:error, save_error_message(reason)) + |> push_navigate(to: return_path(socket))} + + {:error, :unauthorized} -> + {:noreply, + socket + |> put_flash(:error, "You are not authorized to perform this action.") + |> push_navigate(to: return_path(socket))} + {:error, %Ecto.Changeset{}} -> changeset = socket.assigns diff --git a/lib/lightning_web/live/user_live/form_component.ex b/lib/lightning_web/live/user_live/form_component.ex index 1f644bf9a50..5dea1d6a41a 100644 --- a/lib/lightning_web/live/user_live/form_component.ex +++ b/lib/lightning_web/live/user_live/form_component.ex @@ -69,6 +69,12 @@ defmodule LightningWeb.UserLive.FormComponent do {:error, %Ecto.Changeset{} = changeset} -> {:noreply, assign(socket, :changeset, changeset)} + + {:error, :concurrent_edit} -> + {:noreply, + socket + |> put_flash(:error, save_error_message(:concurrent_edit)) + |> push_navigate(to: socket.assigns.return_to)} end end diff --git a/lib/lightning_web/live/workflow_live/collaborate.ex b/lib/lightning_web/live/workflow_live/collaborate.ex index 7453aef0d02..6465a27f226 100644 --- a/lib/lightning_web/live/workflow_live/collaborate.ex +++ b/lib/lightning_web/live/workflow_live/collaborate.ex @@ -13,7 +13,6 @@ defmodule LightningWeb.WorkflowLive.Collaborate do alias Lightning.AiAssistant alias Lightning.Credentials.Credential alias Lightning.Policies.Permissions - alias Lightning.Policies.Users alias Lightning.Projects alias Lightning.Projects.Project alias Lightning.Workflows @@ -155,7 +154,7 @@ defmodule LightningWeb.WorkflowLive.Collaborate do Lightning.Credentials.get_credential(credential_id), true <- Permissions.can?( - Users, + :credentials, :edit_credential, socket.assigns.current_user, credential diff --git a/lib/lightning_web/live/workflow_live/helpers.ex b/lib/lightning_web/live/workflow_live/helpers.ex index 5821290ab8c..db57322c44c 100644 --- a/lib/lightning_web/live/workflow_live/helpers.ex +++ b/lib/lightning_web/live/workflow_live/helpers.ex @@ -8,24 +8,9 @@ defmodule LightningWeb.WorkflowLive.Helpers do alias Lightning.Workflows alias Lightning.Workflows.Workflow - alias Lightning.Workflows.WorkflowUsageLimiter alias Lightning.WorkOrder alias Lightning.WorkOrders - @spec save_workflow(Ecto.Changeset.t(), struct()) :: - {:ok, Workflows.Workflow.t()} - | {:error, - Ecto.Changeset.t() | UsageLimiting.message() | :workflow_deleted} - def save_workflow(changeset, actor) do - case WorkflowUsageLimiter.limit_workflow_activation(changeset) do - :ok -> - Workflows.save_workflow(changeset, actor) - - {:error, _reason, message} -> - {:error, message} - end - end - @spec run_workflow( Ecto.Changeset.t(Workflows.Workflow.t()) | Workflows.Workflow.t(), map(), diff --git a/lib/lightning_web/plugs/access_token_auth.ex b/lib/lightning_web/plugs/access_token_auth.ex index a62aa06686c..d4070bf8916 100644 --- a/lib/lightning_web/plugs/access_token_auth.ex +++ b/lib/lightning_web/plugs/access_token_auth.ex @@ -5,41 +5,31 @@ defmodule LightningWeb.Plugs.AccessTokenAuth do `call/2` goes in the pipeline. It accepts only tokens issued to the service account registered now, so unsetting `SERVICE_ACCOUNT_PUBLIC_KEY` cuts off - tokens already issued, and assigns the token's claims as `:access_token` and - the service account as `:service_account`. Each controller then names the scope its actions need with - `plug :require_scope, "users:read"`. Refusals follow RFC 6750 §3. + tokens already issued. It assigns the token's claims as `:access_token`, and + the service account as `:current_resource` with the token's scopes as its + `grants`. It only authenticates. The policies decide whether those grants + allow an action. A refusal is `LightningWeb.Auth`'s. """ import Plug.Conn alias Lightning.ServiceAccount alias Lightning.ServiceAccount.AccessToken + alias LightningWeb.Auth def init(opts), do: opts def call(conn, _opts) do - with ["Bearer " <> token] <- get_req_header(conn, "authorization"), + with {:ok, token} <- Auth.bearer_token(conn), {:ok, %{"sub" => sub} = claims} <- AccessToken.verify(token), %ServiceAccount{id: ^sub} = service_account <- Lightning.Config.service_account() do + grants = claims |> Map.get("scope", "") |> String.split() |> MapSet.new() + conn |> assign(:access_token, claims) - |> assign(:service_account, service_account) + |> assign(:current_resource, %{service_account | grants: grants}) else - _other -> refuse(conn, 401, "invalid_token") + _other -> Auth.refuse(conn, :bearer) end end - - def require_scope(conn, scope) do - if scope in String.split(conn.assigns.access_token["scope"]), - do: conn, - else: refuse(conn, 403, "insufficient_scope") - end - - defp refuse(conn, status, error) do - conn - |> put_resp_header("www-authenticate", ~s(Bearer error="#{error}")) - |> put_status(status) - |> Phoenix.Controller.json(%{error: error}) - |> halt() - end end diff --git a/lib/lightning_web/plugs/api_auth.ex b/lib/lightning_web/plugs/api_auth.ex index 6a63c4966f2..9ece62108e6 100644 --- a/lib/lightning_web/plugs/api_auth.ex +++ b/lib/lightning_web/plugs/api_auth.ex @@ -2,46 +2,30 @@ defmodule LightningWeb.Plugs.ApiAuth do @moduledoc """ Authenticates api calls based on JWT bearer token. """ - use Phoenix.Controller import Plug.Conn + alias LightningWeb.Auth + def init(opts) do opts end def call(conn, _opts) do - with {:ok, bearer_token} <- get_bearer_token(conn), + with {:ok, bearer_token} <- Auth.bearer_token(conn), {:ok, claims} <- Lightning.Tokens.verify(bearer_token) do conn |> assign(:claims, claims) |> put_subject() else {:error, _reason} -> - deny_access(conn) - end - end - - defp get_bearer_token(conn) do - conn - |> get_req_header("authorization") - |> case do - ["Bearer " <> bearer] -> {:ok, bearer} - _none_or_many -> {:error, :token_not_found} + Auth.refuse(conn, :bearer) end end defp put_subject(conn) do case Lightning.Tokens.get_subject(conn.assigns.claims) do - nil -> deny_access(conn) + nil -> Auth.refuse(conn, :bearer) subject -> assign(conn, :subject, subject) end end - - defp deny_access(conn) do - conn - |> put_status(:unauthorized) - |> put_view(LightningWeb.ErrorView) - |> render(:"401") - |> halt() - end end diff --git a/lib/lightning_web/plugs/authorize_action.ex b/lib/lightning_web/plugs/authorize_action.ex new file mode 100644 index 00000000000..3e8d159f342 --- /dev/null +++ b/lib/lightning_web/plugs/authorize_action.ex @@ -0,0 +1,33 @@ +defmodule LightningWeb.Plugs.AuthorizeAction do + @moduledoc """ + Refuses an API action early, on facts about the caller alone, before the + action parses a body or loads anything. + + plug AuthorizeAction, {:project_users, :access_project} when action == :show + + It asks `Lightning.Policies.Permissions.permit/3` before any project is + loaded. A refusal is answered by `LightningWeb.API.FallbackController`. A pass + assigns the caller's scope as `:scope` for the action to reuse, and the action + still asks `permit/4` about its target. An action with no target asks + nothing more. + """ + alias Lightning.Policies.Permissions + alias Lightning.Scope + alias LightningWeb.API.FallbackController + + def init({_policy, _action} = permission), do: permission + + def call(conn, {policy, action}) do + scope = Scope.new(conn.assigns.current_resource) + + case Permissions.permit(policy, action, scope) do + :ok -> + Plug.Conn.assign(conn, :scope, scope) + + error -> + conn + |> FallbackController.call(error) + |> Plug.Conn.halt() + end + end +end diff --git a/lib/lightning_web/plugs/channel_proxy_plug.ex b/lib/lightning_web/plugs/channel_proxy_plug.ex index b88ba73adf9..db956e3f8ad 100644 --- a/lib/lightning_web/plugs/channel_proxy_plug.ex +++ b/lib/lightning_web/plugs/channel_proxy_plug.ex @@ -27,11 +27,11 @@ defmodule LightningWeb.ChannelProxyPlug do @behaviour Plug import Plug.Conn - import Phoenix.Controller, only: [json: 2] alias Lightning.Channels alias Lightning.Channels.PersistencePolicy alias Lightning.Projects.Environment + alias LightningWeb.API.Problem alias LightningWeb.Auth require Logger @@ -95,11 +95,11 @@ defmodule LightningWeb.ChannelProxyPlug do }} else :error -> - conn = error_response(conn, :not_found, "Not Found") + conn = Problem.halt(conn, Problem.new(:not_found)) {conn, %{outcome: :invalid_uuid, status: conn.status}} :not_found -> - conn = error_response(conn, :not_found, "Not Found") + conn = Problem.halt(conn, Problem.new(:not_found)) {conn, %{outcome: :unknown_channel, status: conn.status}} end end @@ -117,7 +117,7 @@ defmodule LightningWeb.ChannelProxyPlug do forward_request(conn, channel, rest, matched_auth) {:unauthorized, conn} -> - error_response(conn, :unauthorized, "Unauthorized") + Auth.refuse(conn, channel.client_webhook_auth_methods) end {result, @@ -143,7 +143,7 @@ defmodule LightningWeb.ChannelProxyPlug do handle_credential_error(conn, channel, reason) {:error, _} -> - error_response(conn, :internal_server_error, "Internal Server Error") + Problem.halt(conn, Problem.new(:internal)) end end @@ -395,21 +395,17 @@ defmodule LightningWeb.ChannelProxyPlug do event_attrs ) - error_response(conn, :bad_gateway, "Bad Gateway") + Problem.halt(conn, Problem.new(:bad_gateway)) {:error, _} -> Logger.error( "Failed to create snapshot for credential error on channel #{channel.id}" ) - error_response(conn, :bad_gateway, "Bad Gateway") + Problem.halt(conn, Problem.new(:bad_gateway)) end end - defp error_response(conn, status, message) do - conn |> put_status(status) |> json(%{"error" => message}) |> halt() - end - defp classify_credential_error(:environment_not_found), do: "credential_environment_not_found" diff --git a/lib/lightning_web/plugs/person_or_service_account_auth.ex b/lib/lightning_web/plugs/person_or_service_account_auth.ex new file mode 100644 index 00000000000..c5c7d56dd54 --- /dev/null +++ b/lib/lightning_web/plugs/person_or_service_account_auth.ex @@ -0,0 +1,33 @@ +defmodule LightningWeb.Plugs.PersonOrServiceAccountAuth do + @moduledoc """ + Authenticates a request by either a service account's access token or the + bearer tokens `LightningWeb.UserAuth` accepts (a personal access token or a + repo connection token). + + A token that declares itself an access token goes to + `LightningWeb.Plugs.AccessTokenAuth`. Any other request gets exactly the + answer `UserAuth.authenticate_bearer/2` and + `UserAuth.require_authenticated_api_resource/2` give it elsewhere. + + It only authenticates. The policies decide what the caller may do, including + whether its token holds the grant an action needs. + """ + alias Lightning.ServiceAccount.AccessToken + alias LightningWeb.Auth + alias LightningWeb.Plugs.AccessTokenAuth + alias LightningWeb.UserAuth + + def init(opts), do: opts + + def call(conn, _opts) do + with {:ok, token} <- Auth.bearer_token(conn), + true <- AccessToken.access_token?(token) do + AccessTokenAuth.call(conn, []) + else + _person -> + conn + |> UserAuth.authenticate_bearer([]) + |> UserAuth.require_authenticated_api_resource([]) + end + end +end diff --git a/lib/lightning_web/plugs/webhook_auth.ex b/lib/lightning_web/plugs/webhook_auth.ex index a93feeb4f5e..115a04155bc 100644 --- a/lib/lightning_web/plugs/webhook_auth.ex +++ b/lib/lightning_web/plugs/webhook_auth.ex @@ -7,6 +7,7 @@ defmodule LightningWeb.Plugs.WebhookAuth do alias Lightning.Retry alias Lightning.Workflows + alias LightningWeb.API.Problem alias LightningWeb.Auth require Logger @@ -29,11 +30,12 @@ defmodule LightningWeb.Plugs.WebhookAuth do `workflow` and `edges`) and its `webhook_auth_methods`, wrapped in `Lightning.Retry.with_webhook_retry/2` so transient DB errors are retried. - 2. If the trigger is missing → responds **404** `{"error":"webhook_not_found"}`. + 2. If the trigger is missing → responds **404** `not_found`. 3. If auth methods are configured: - If credentials match → assigns `:trigger` and continues. - If credentials are present but wrong → responds **404** (hide existence). - - If credentials are missing → responds **401**. + - If credentials are missing → responds **401** `unauthenticated`, + challenging for an API key when the trigger takes one. 4. If retries exhaust due to DB issues → responds **503** with `Retry-After` based on `WEBHOOK_RETRY_TIMEOUT_MS`. @@ -67,9 +69,7 @@ defmodule LightningWeb.Plugs.WebhookAuth do LightningWeb.Utils.respond_service_unavailable( conn, error, - %{op: :webhook_auth_lookup, webhook: log_key(segments)}, - message: - "Temporary database issue during webhook lookup. Please retry in %{s}s." + %{op: :webhook_auth_lookup, webhook: log_key(segments)} ) end @@ -83,7 +83,8 @@ defmodule LightningWeb.Plugs.WebhookAuth do # `/i//Patient`, and that is where record identifiers turn up. defp log_key(segments), do: segments |> Enum.take(1) |> Enum.join("/") - defp validate_auth(nil, _methods, conn), do: not_found_response(conn) + defp validate_auth(nil, _methods, conn), + do: Problem.halt(conn, Problem.new(:not_found)) defp validate_auth(trigger, methods, conn) do case methods do @@ -103,24 +104,10 @@ defmodule LightningWeb.Plugs.WebhookAuth do successful_response(conn, trigger) Auth.has_credentials?(conn) -> - not_found_response(conn) + Problem.halt(conn, Problem.new(:not_found)) true -> - unauthorized_response(conn) + Auth.refuse(conn, auth_methods) end end - - defp unauthorized_response(conn) do - conn - |> put_status(:unauthorized) - |> json(%{"error" => "Unauthorized"}) - |> halt() - end - - defp not_found_response(conn) do - conn - |> put_status(:not_found) - |> json(%{"error" => "Webhook not found"}) - |> halt() - end end diff --git a/lib/lightning_web/router.ex b/lib/lightning_web/router.ex index 68175472317..8fb9b1199e4 100644 --- a/lib/lightning_web/router.ex +++ b/lib/lightning_web/router.ex @@ -9,6 +9,7 @@ defmodule LightningWeb.Router do import Phoenix.LiveDashboard.Router alias CredentialLive + alias LightningWeb.API.Problem alias ProjectLive alias UserLive @@ -44,25 +45,57 @@ defmodule LightningWeb.Router do pipeline :api do plug :accepts, ["json"] + plug :refuse_unparsed end pipeline :access_token_api do plug :accepts, ["json"] + plug :refuse_unparsed plug LightningWeb.Plugs.AccessTokenAuth end + pipeline :person_or_service_account_api do + plug :accepts, ["json"] + plug :refuse_unparsed + plug LightningWeb.Plugs.PersonOrServiceAccountAuth + end + pipeline :authenticated_api do plug :accepts, ["json"] + plug :refuse_unparsed plug LightningWeb.Plugs.ApiAuth end + # Webhooks answer a body they can't read themselves (error-format spec §12). + pipeline :webhook do + plug :accepts, ["json"] + end + pipeline :authenticated_json do plug :accepts, ["json"] + plug :refuse_unparsed plug :fetch_session plug :protect_from_forgery plug :fetch_current_user end + # Plug.Parsers passes every content type (`pass: ["*/*"]`), so a body no + # parser read arrives unfetched. The JSON pipelines answer it with 415. A + # request with no body has nothing to misread, so it passes whatever its + # content type says. Reading is the only reliable way to tell, because a + # chunked or HTTP/2 request needn't send a content-length. + defp refuse_unparsed(%Plug.Conn{body_params: %Plug.Conn.Unfetched{}} = conn, _) do + case Plug.Conn.read_body(conn, length: 1, read_length: 1) do + {:ok, "", conn} -> + %{conn | body_params: %{}} + + {_ok_or_more, _data, conn} -> + Problem.halt(conn, Problem.new(:unsupported_media_type)) + end + end + + defp refuse_unparsed(conn, _opts), do: conn + scope "/", LightningWeb do pipe_through [:browser] @@ -101,6 +134,23 @@ defmodule LightningWeb.Router do patch "/users/:id", API.UserController, :update end + ## A service account's access token or a person's bearer token + scope "/api", LightningWeb, as: :api do + pipe_through [:person_or_service_account_api] + + get "/projects/:id", API.ProjectController, :show + put "/projects/:id", API.ProjectController, :update + + get "/credentials/:id", API.CredentialController, :show + put "/credentials/:id", API.CredentialController, :update + + get "/projects/:project_id/collections/:id", API.CollectionController, :show + + put "/projects/:project_id/collections/:id", + API.CollectionController, + :update + end + scope "/api", LightningWeb, as: :api do pipe_through [:api] @@ -114,9 +164,12 @@ defmodule LightningWeb.Router do get "/provision/yaml", API.ProvisioningController, :show_yaml resources "/provision", API.ProvisioningController, only: [:create, :show] - resources "/projects", API.ProjectController, only: [:index, :show] do + resources "/projects", API.ProjectController, only: [:index] do resources "/credentials", API.CredentialController, only: [:index] - resources "/workflows", API.WorkflowsController, except: [:delete] + + resources "/workflows", API.WorkflowsController, + only: [:index, :show, :create, :update] + resources "/jobs", API.JobController, only: [:index, :show] resources "/work_orders", API.WorkOrdersController, only: [:index, :show] resources "/runs", API.RunController, only: [:index, :show] @@ -152,6 +205,13 @@ defmodule LightningWeb.Router do :runs end + # An unmatched route is answered before authentication (error-format spec + # §4), so the catch-alls take no pipeline. The /api scopes share one prefix, + # so theirs follows the last of them. + scope "/api" do + match :*, "/*path", LightningWeb.API.NoRoute, :not_found + end + ## Adaptor catalogue (cookie-authenticated JSON) scope "/", LightningWeb do pipe_through [:authenticated_json, :require_authenticated_user] @@ -171,6 +231,10 @@ defmodule LightningWeb.Router do delete "/:name", CollectionsController, :delete_all end + scope "/collections" do + match :*, "/*path", LightningWeb.API.NoRoute, :not_found + end + ## Authentication routes scope "/", LightningWeb do pipe_through [:browser, :redirect_if_user_is_authenticated] @@ -195,7 +259,7 @@ defmodule LightningWeb.Router do get "/download/yaml", DownloadsController, :download_project_yaml get "/download/collections/:project_id/:name", - CollectionsController, + CollectionDownloadController, :download get "/dataclip/body/:id", DataclipController, :show @@ -331,7 +395,7 @@ defmodule LightningWeb.Router do end scope "/i", LightningWeb do - pipe_through :api + pipe_through :webhook post "/*path", WebhooksController, :create get "/*path", WebhooksController, :check diff --git a/lib/lightning_web/utils.ex b/lib/lightning_web/utils.ex index 6b523cf0985..365ae3ffc01 100644 --- a/lib/lightning_web/utils.ex +++ b/lib/lightning_web/utils.ex @@ -8,6 +8,7 @@ defmodule LightningWeb.Utils do * A macro to set up multiple plugs dynamically (`add_dynamic_plugs/1`) """ + alias LightningWeb.API.Problem alias Phoenix.HTML.Form alias Plug.Conn.Query @@ -92,17 +93,13 @@ defmodule LightningWeb.Utils do def ok(socket), do: {:ok, socket} @doc """ - Standard **503 Service Unavailable** response for transient DB outages - during webhook handling. - - This function is typically called inside a `with_webhook_retry/2` error branch - when retries against the database have been exhausted. + Answers 503 `service_unavailable` with `Retry-After` for a transient DB outage + during webhook handling (error-format spec §9.10, §12), once + `with_webhook_retry/2` has exhausted its retries against the database. ## Options - * `:message` — custom response message, may include `"%{s}"` placeholder - which will be replaced with the retry-after value. - * `:halt?` — whether to halt the connection after sending the response. + * `:halt?` - whether to halt the connection after sending the response. Defaults to `true`. """ @spec respond_service_unavailable( @@ -120,13 +117,6 @@ defmodule LightningWeb.Utils do ) do halt? = Keyword.get(opts, :halt?, true) - message = - Keyword.get( - opts, - :message, - "Temporary database issue. Please retry in %{s}s." - ) - retry_after = Lightning.Config.webhook_retry(:timeout_ms) |> div(1000) |> max(1) @@ -151,17 +141,11 @@ defmodule LightningWeb.Utils do " error=#{Exception.message(error)}" ) - body = %{ - error: :service_unavailable, - message: String.replace(message, "%{s}", Integer.to_string(retry_after)), - retry_after: retry_after - } - conn = - conn - |> Plug.Conn.put_resp_header("retry-after", Integer.to_string(retry_after)) - |> Plug.Conn.put_status(:service_unavailable) - |> Phoenix.Controller.json(body) + Problem.respond( + conn, + Problem.new(:service_unavailable, retry_after: retry_after) + ) if halt?, do: Plug.Conn.halt(conn), else: conn end diff --git a/lib/lightning_web/views/changeset_view.ex b/lib/lightning_web/views/changeset_view.ex deleted file mode 100644 index 35fa60ca8a9..00000000000 --- a/lib/lightning_web/views/changeset_view.ex +++ /dev/null @@ -1,11 +0,0 @@ -defmodule LightningWeb.ChangesetView do - use LightningWeb, :view - - import LightningWeb.CoreComponents, only: [translate_errors: 1] - - def render("error.json", %{changeset: changeset}) do - # When encoded, the changeset returns its errors - # as a JSON object. So we just pass it forward. - %{errors: translate_errors(changeset)} - end -end diff --git a/priv/bench/collections.exs b/priv/bench/collections.exs index 8cdd1b321dd..4bcfc89808f 100644 --- a/priv/bench/collections.exs +++ b/priv/bench/collections.exs @@ -6,16 +6,25 @@ keys_count = 5_000 Repo.delete_all(Collections.Collection) +user = Repo.get_by(Lightning.Accounts.User, email: "demo@openfn.org") + project = with nil <- Repo.get_by(Projects.Project, name: "benchee") do - user = Repo.get_by(Lightning.Accounts.User, email: "demo@openfn.org") - {:ok, project} = Projects.create_project(%{name: "benchee", project_users: [%{user_id: user.id, role: :owner}]}) + {:ok, project} = + Projects.create_project( + %{name: "benchee", project_users: [%{user_id: user.id, role: :owner}]}, + user + ) + project end {:ok, collection} = with {:error, :not_found} <- Collections.get_collection("benchee") do - Collections.create_collection(%{"project_id" => project.id, "name" => "benchee"}) + Collections.create_collection( + %{"project_id" => project.id, "name" => "benchee"}, + user + ) end IO.puts("\n### Setup:") @@ -23,14 +32,17 @@ IO.puts("Generating items for benchee collection...") record = fn prefix, i -> i_str = String.pad_leading(to_string(i), 5, "0") - {"#{prefix}:foo#{i_str}:bar#{i_str}", Jason.encode!(%{fieldA: "value#{1_000_000 + i}"})} + + {"#{prefix}:foo#{i_str}:bar#{i_str}", + Jason.encode!(%{fieldA: "value#{1_000_000 + i}"})} end sampleA = Enum.map(1..keys_count, fn i -> record.("keyA", i) end) sampleB = Enum.map(1..keys_count, fn i -> record.("keyB", i) end) sampleC1 = Enum.map(1..keys_count, fn i -> record.("keyC", i) end) + sampleC2 = - keys_count..keys_count * 2 + keys_count..(keys_count * 2) |> Enum.map(fn i -> record.("keyC", i) end) |> Enum.map(fn {k, v} -> %{"key" => k, "value" => v} end) @@ -42,6 +54,7 @@ sampleC2 = if rem(idx, 1000) == 0, do: IO.puts("Inserting " <> key) :ok = Collections.put(collection, key, value) end) + :ok end) end) @@ -53,6 +66,7 @@ end) end) IO.puts("Inserting #{length(sampleC2)} items with put_all...") + :timer.tc(fn -> {:ok, _n} = Collections.put_all(collection, sampleC2) end) @@ -63,6 +77,7 @@ end) sampleD = Enum.map(1..keys_count, fn i -> record.("keyD", i) end) IO.puts("Inserting sampleD (w/ sorted keys)...") + :timer.tc(fn -> sampleD |> Enum.chunk_every(1000) @@ -78,6 +93,7 @@ end) end) IO.puts("Upserting sampleD...") + :timer.tc(fn -> sampleD |> Enum.chunk_every(1000) @@ -119,7 +135,11 @@ stream_match_all = stream_match_prefix = fn -> Stream.unfold(nil, fn cursor -> - case Collections.get_all(collection, %{cursor: cursor, limit: 500}, "keyA*") do + case Collections.get_all( + collection, + %{cursor: cursor, limit: 500}, + "keyA*" + ) do [] -> nil list -> {list, List.last(list).updated_at} end @@ -131,7 +151,11 @@ stream_match_prefix = stream_match_trigram = fn -> Stream.unfold(nil, fn cursor -> - case Collections.get_all(collection, %{cursor: cursor, limit: 500}, "keyB*bar*") do + case Collections.get_all( + collection, + %{cursor: cursor, limit: 500}, + "keyB*bar*" + ) do [] -> nil list -> {list, List.last(list).updated_at} end @@ -143,10 +167,19 @@ stream_match_trigram = # Process.exit(self(), :normal) IO.puts("\n### Round record count ({microsecs, count}):") -:timer.tc(fn -> stream_all.() |> Enum.count() end) |> IO.inspect(label: "stream_all") -:timer.tc(fn -> stream_match_all.() |> Enum.count() end) |> IO.inspect(label: "stream_match_all") -:timer.tc(fn -> stream_match_prefix.() |> Enum.count() end) |> IO.inspect(label: "stream_match_prefix") -:timer.tc(fn -> stream_match_trigram.() |> Enum.count() end) |> IO.inspect(label: "stream_match_trigram") + +:timer.tc(fn -> stream_all.() |> Enum.count() end) +|> IO.inspect(label: "stream_all") + +:timer.tc(fn -> stream_match_all.() |> Enum.count() end) +|> IO.inspect(label: "stream_match_all") + +:timer.tc(fn -> stream_match_prefix.() |> Enum.count() end) +|> IO.inspect(label: "stream_match_prefix") + +:timer.tc(fn -> stream_match_trigram.() |> Enum.count() end) +|> IO.inspect(label: "stream_match_trigram") + IO.puts("\n") Benchee.run( diff --git a/priv/bench/collections_put_all.exs b/priv/bench/collections_put_all.exs index 2380475c25c..0a634c92c8e 100644 --- a/priv/bench/collections_put_all.exs +++ b/priv/bench/collections_put_all.exs @@ -2,8 +2,13 @@ alias Lightning.Repo alias Lightning.Collections alias Lightning.Projects -IO.puts "### Indexes on collection_items table" -%{rows: rows} = Repo.query!("select indexname, indexdef from pg_indexes where tablename = 'collection_items'") +IO.puts("### Indexes on collection_items table") + +%{rows: rows} = + Repo.query!( + "select indexname, indexdef from pg_indexes where tablename = 'collection_items'" + ) + Enum.each(rows, &IO.puts(Enum.join(&1, ":\n"))) keys_count = 5_000 @@ -13,52 +18,72 @@ rounds = 300 Repo.delete_all(Collections.Collection) +user = + Repo.get_by(Lightning.Accounts.User, email: "demo@openfn.org") || + raise "This benchmark requires demo/known user" + project = with nil <- Repo.get_by(Projects.Project, name: "bench") do - user = Repo.get_by(Lightning.Accounts.User, email: "demo@openfn.org") || raise "This benchmark requires demo/known user" - {:ok, project} = Projects.create_project(%{name: "bench", project_users: [%{user_id: user.id, role: :owner}]}) + {:ok, project} = + Projects.create_project( + %{name: "bench", project_users: [%{user_id: user.id, role: :owner}]}, + user + ) + project end {:ok, collection1} = with {:error, :not_found} <- Collections.get_collection("bench1") do - Collections.create_collection(%{"project_id" => project.id, "name" => "bench1"}) + Collections.create_collection( + %{"project_id" => project.id, "name" => "bench1"}, + user + ) end {:ok, collection2} = with {:error, :not_found} <- Collections.get_collection("bench2") do - Collections.create_collection(%{"project_id" => project.id, "name" => "bench2"}) + Collections.create_collection( + %{"project_id" => project.id, "name" => "bench2"}, + user + ) end record1 = fn prefix, i -> i_str = String.pad_leading(to_string(i), 5, "0") + %{ "key" => "#{prefix}:foo#{i_str}:bar#{i_str}", - "value" => Jason.encode!(%{ - someid1: "soo#{1_000_000 + i}", - somefield1: "zar#{1_000_000 + i}", - anotherfield1: "yaz#{1_000_000 + i}"}) + "value" => + Jason.encode!(%{ + someid1: "soo#{1_000_000 + i}", + somefield1: "zar#{1_000_000 + i}", + anotherfield1: "yaz#{1_000_000 + i}" + }) } end record2 = fn prefix, i -> i_str = String.pad_leading(to_string(i), 5, "0") + %{ "key" => "#{prefix}:foo#{i_str}:bar#{i_str}", - "value" => Jason.encode!(%{ - someid2: "soo#{1_000_000 + i}", - somefield2: "zar#{1_000_000 + i}", - anotherfield2: "yaz#{1_000_000 + i}"}) + "value" => + Jason.encode!(%{ + someid2: "soo#{1_000_000 + i}", + somefield2: "zar#{1_000_000 + i}", + anotherfield2: "yaz#{1_000_000 + i}" + }) } end samples1 = - 1..keys_count * rounds + 1..(keys_count * rounds) |> Enum.map(fn i -> record1.("keyA", i) end) |> Enum.chunk_every(keys_count) samples2 = - 1..keys_count * rounds + 1..(keys_count * rounds) |> Enum.map(fn i -> record2.("keyB", i) end) |> Enum.chunk_every(keys_count) @@ -66,18 +91,25 @@ IO.puts("\n### Inserting #{rounds} rounds of 2x#{keys_count} with put_all...") durations = Enum.zip(samples1, samples2) - |> Enum.flat_map(fn {sample1, sample2} -> [{sample1, collection1}, {sample2, collection2}] end) - |> Task.async_stream(fn {sample, collection} -> - {duration, _res} = :timer.tc(fn -> - {:ok, _n} = Collections.put_all(collection, sample) - end) - - div(duration, 1_000) - end, max_concurrency: 2, timeout: :infinity) + |> Enum.flat_map(fn {sample1, sample2} -> + [{sample1, collection1}, {sample2, collection2}] + end) + |> Task.async_stream( + fn {sample, collection} -> + {duration, _res} = + :timer.tc(fn -> + {:ok, _n} = Collections.put_all(collection, sample) + end) + + div(duration, 1_000) + end, + max_concurrency: 2, + timeout: :infinity + ) |> Enum.map(fn {:ok, duration_ms} -> IO.puts("elapsed time: #{duration_ms}ms") duration_ms end) -IO.puts "Average: #{Statistics.mean(durations)}ms" -IO.puts "Std Deviation: #{Statistics.stdev(durations)}ms" +IO.puts("Average: #{Statistics.mean(durations)}ms") +IO.puts("Std Deviation: #{Statistics.stdev(durations)}ms") diff --git a/test/integration/cli_deploy_test.exs b/test/integration/cli_deploy_test.exs index daa70358eb3..5bde3ebffc6 100644 --- a/test/integration/cli_deploy_test.exs +++ b/test/integration/cli_deploy_test.exs @@ -73,8 +73,8 @@ defmodule Lightning.CliDeployTest do env: @required_env ) - assert logs =~ "Failed to authorize request with endpoint" - assert logs =~ "403 Forbidden" + assert logs =~ "ERROR: Project not found." + assert logs =~ "Please check the UUID" # state and spec files are not saved refute File.exists?(config.statePath) diff --git a/test/lightning/accounts/user_test.exs b/test/lightning/accounts/user_test.exs index bd10a14a6be..6fea4689521 100644 --- a/test/lightning/accounts/user_test.exs +++ b/test/lightning/accounts/user_test.exs @@ -441,7 +441,8 @@ defmodule Lightning.Accounts.UserTest do changeset = apply(User, unquote(fun), [%User{}, attrs]) assert changeset.errors[:email] == - {"has already been taken", validation: :unsafe_unique} + {"has already been taken", + validation: :unsafe_unique, code: :taken} assert [%{field: :email, type: :unique}] = changeset.constraints end diff --git a/test/lightning/accounts/user_totp_test.exs b/test/lightning/accounts/user_totp_test.exs index 4d4c748afaa..c2f47e689d1 100644 --- a/test/lightning/accounts/user_totp_test.exs +++ b/test/lightning/accounts/user_totp_test.exs @@ -38,7 +38,7 @@ defmodule Lightning.Accounts.UserTOTPTest do UserTOTP.changeset(user_totp, %{}) assert errors == [ - {:code, {"invalid code", []}}, + {:code, {"invalid code", [code: :mismatch]}}, {:code, {"can't be blank", [validation: :required]}} ] end @@ -50,7 +50,7 @@ defmodule Lightning.Accounts.UserTOTPTest do UserTOTP.changeset(%UserTOTP{user_id: user_id}, %{code: "123456"}) assert errors == [ - {:code, {"invalid code", []}}, + {:code, {"invalid code", [code: :mismatch]}}, {:secret, {"can't be blank", [validation: :required]}} ] end @@ -65,7 +65,7 @@ defmodule Lightning.Accounts.UserTOTPTest do UserTOTP.changeset(user_totp, %{code: provided_code}) assert errors == [ - {:code, {"invalid code", []}}, + {:code, {"invalid code", [code: :mismatch]}}, {:code, {"should be a 6 digit number", [validation: :format]}} ] end @@ -82,7 +82,7 @@ defmodule Lightning.Accounts.UserTOTPTest do assert %Changeset{valid?: false, errors: errors} = UserTOTP.changeset(user_totp, %{code: invalid_code}) - assert errors == [code: {"invalid code", []}] + assert errors == [code: {"invalid code", [code: :mismatch]}] end end diff --git a/test/lightning/accounts_test.exs b/test/lightning/accounts_test.exs index 7327a569a04..b343c01eed1 100644 --- a/test/lightning/accounts_test.exs +++ b/test/lightning/accounts_test.exs @@ -846,7 +846,11 @@ defmodule Lightning.AccountsTest do test "purging a user sets all project credentials that use their credentials to nil" do user = insert(:user) - project = Lightning.ProjectsFixtures.project_fixture() + + project = + Lightning.ProjectsFixtures.project_fixture( + project_users: [%{user_id: user.id}] + ) project_credential_1 = CredentialsFixtures.project_credential_fixture( @@ -1421,6 +1425,14 @@ defmodule Lightning.AccountsTest do assert_transports_disconnected(user) end + test "a user another request deleted is a concurrent edit, not a raise" do + user = insert(:user) + Repo.delete!(user) + + assert Accounts.update_user(user, %{"first_name" => "Renamed"}) == + {:error, :concurrent_edit} + end + test "editing other fields leaves the user's sessions untouched" do user = insert(:user) session_token = Accounts.generate_user_session_token(user) diff --git a/test/lightning/ai_assistant/unsaved_job_test.exs b/test/lightning/ai_assistant/unsaved_job_test.exs index 1fd95acfc98..7ea1af38f33 100644 --- a/test/lightning/ai_assistant/unsaved_job_test.exs +++ b/test/lightning/ai_assistant/unsaved_job_test.exs @@ -265,6 +265,51 @@ defmodule Lightning.AiAssistant.UnsavedJobTest do end end + describe "unsaved_job_session/1" do + test "finds a job-code session by its unsaved job's id" do + job_id = Ecto.UUID.generate() + + session = + insert(:chat_session, + session_type: "job_code", + job_id: nil, + meta: %{"unsaved_job" => %{"id" => job_id}} + ) + + assert AiAssistant.unsaved_job_session(job_id).id == session.id + end + + test "finds a job-code session by its saved job" do + job = insert(:job) + session = insert(:chat_session, session_type: "job_code", job: job) + + assert AiAssistant.unsaved_job_session(job.id).id == session.id + end + + test "returns nil when no job-code session names the job" do + job_id = Ecto.UUID.generate() + + insert(:chat_session, + session_type: "workflow_template", + meta: %{"unsaved_job" => %{"id" => job_id}} + ) + + assert AiAssistant.unsaved_job_session(job_id) == nil + end + + test "returns nil when job-code sessions name only other jobs" do + insert(:chat_session, session_type: "job_code", job: insert(:job)) + + insert(:chat_session, + session_type: "job_code", + job_id: nil, + meta: %{"unsaved_job" => %{"id" => Ecto.UUID.generate()}} + ) + + assert AiAssistant.unsaved_job_session(Ecto.UUID.generate()) == nil + end + end + describe "cleanup_unsaved_job_sessions/1" do test "cleans up sessions when workflow is saved with matching jobs" do user = insert(:user) diff --git a/test/lightning/channels_test.exs b/test/lightning/channels_test.exs index 29e63d9f4f3..2457172720a 100644 --- a/test/lightning/channels_test.exs +++ b/test/lightning/channels_test.exs @@ -496,7 +496,8 @@ defmodule Lightning.ChannelsTest do assert reloaded.destination_auth_method == nil end - test "returns stale error on lock_version conflict", %{user: user} do + test "a channel another request changed is a concurrent edit, not a changeset", + %{user: user} do channel = insert(:channel) # Simulate concurrent update by updating lock_version in DB @@ -506,12 +507,12 @@ defmodule Lightning.ChannelsTest do set: [lock_version: channel.lock_version + 1] ) - assert {:error, changeset} = + assert {:error, :concurrent_edit} = Channels.update_channel(channel, %{name: "stale-update"}, actor: user ) - assert changeset.errors[:lock_version] + assert Lightning.Repo.reload(channel).name == channel.name end end diff --git a/test/lightning/collaboration/session_test.exs b/test/lightning/collaboration/session_test.exs index 918e3597157..f43838f6d82 100644 --- a/test/lightning/collaboration/session_test.exs +++ b/test/lightning/collaboration/session_test.exs @@ -5,6 +5,7 @@ defmodule Lightning.SessionTest do import Lightning.AdaptorTestHelpers import Lightning.Factories import Lightning.CollaborationHelpers + import Lightning.RaceHelpers import Mox alias Lightning.Collaboration.DocumentState @@ -1138,31 +1139,35 @@ defmodule Lightning.SessionTest do drain_document(instance, document_name) end - test "handles concurrent saves with optimistic locking", %{ - session: session, - user: user, - workflow: workflow, - instance: instance, - document_name: document_name - } do - # Another process updates the workflow (simulating concurrent edit) - {:ok, _updated} = - Lightning.Workflows.save_workflow( - Lightning.Workflows.change_workflow(workflow, %{ - name: "Concurrent Update" - }), - user - ) + test "a save another request beats replies :concurrent_edit and keeps the session", + %{ + session: session, + user: user, + workflow: workflow, + instance: instance, + document_name: document_name + } do + doc = Session.get_doc(session) + workflow_map = Yex.Doc.get_map(doc, "workflow") - # Our save should detect the lock_version conflict - # Note: This may succeed if Y.Doc has latest changes - # The test verifies the system handles it gracefully either way - result = Session.save_workflow(session, user) + Yex.Doc.transaction(doc, "test_update", fn -> + Yex.Map.set(workflow_map, "name", "Updated Name") + end) - case result do - {:ok, _} -> assert true - {:error, _} -> assert true - end + bump_workflow_lock_mid_save(workflow.id, session) + + assert {:error, :concurrent_edit} = Session.save_workflow(session, user) + assert Process.alive?(session) + + assert %{name: "Original Name", lock_version: lock_version} = + Repo.reload(workflow) + + assert lock_version == workflow.lock_version + 1 + + assert {:ok, %Workflow{name: "Updated Name", lock_version: retried}} = + Session.save_workflow(session, user) + + assert retried == workflow.lock_version + 2 drain_document(instance, document_name) end diff --git a/test/lightning/collections_item_writing_test.exs b/test/lightning/collections_item_writing_test.exs new file mode 100644 index 00000000000..b8837f1f812 --- /dev/null +++ b/test/lightning/collections_item_writing_test.exs @@ -0,0 +1,484 @@ +defmodule Lightning.CollectionsItemWritingTest do + use Lightning.DataCase, async: true + + alias Lightning.Collections + alias Lightning.Collections.Collection + alias Lightning.Collections.Item + + describe "Item.changeset/2 column width" do + # Key width only. Null bytes in item keys are not covered here. + test "a key short in graphemes but too wide for the column is rejected" do + collection = insert(:collection) + family = "\u{1F468}\u{200D}\u{1F469}\u{200D}\u{1F467}\u{200D}\u{1F466}" + + changeset = + Item.changeset(%Item{}, %{ + collection_id: collection.id, + key: String.duplicate(family, 200), + value: "v" + }) + + assert errors_on(changeset)[:key] == [ + "key is too long, please use a shorter one" + ] + end + + test "a value short in graphemes but too wide for the column is rejected" do + collection = insert(:collection) + family = "\u{1F468}\u{200D}\u{1F469}\u{200D}\u{1F467}\u{200D}\u{1F466}" + + # 200_000 graphemes, which clears the 1_000_000 grapheme cap, but + # 1_400_000 codepoints, which is what varchar(1000000) counts. + changeset = + Item.changeset(%Item{}, %{ + collection_id: collection.id, + key: "k", + value: String.duplicate(family, 200_000) + }) + + assert errors_on(changeset)[:value] == [ + "value is too long, please use a shorter one" + ] + end + + test "a value wider than the limit in bytes but not in characters is accepted" do + collection = insert(:collection) + + changeset = + Item.changeset(%Item{}, %{ + collection_id: collection.id, + key: "k", + value: String.duplicate("é", 1_000_000) + }) + + assert changeset.valid? + end + + test "a key at the column width is accepted" do + collection = insert(:collection) + + changeset = + Item.changeset(%Item{}, %{ + collection_id: collection.id, + key: String.duplicate("a", 255), + value: "v" + }) + + refute errors_on(changeset)[:key] + assert {:ok, _} = Repo.insert(changeset) + end + end + + describe "put/3" do + test "creates a new entry in the collection for the given collection" do + collection = insert(:collection) + + assert :ok = Collections.put(collection, "some-key1", "some-value1") + assert :ok = Collections.put(collection, "some-key12", "some-value12") + + assert %{key: "some-key1", value: "some-value1"} = + Repo.get_by!(Item, key: "some-key1") + + assert %{key: "some-key12", value: "some-value12"} = + Repo.get_by!(Item, key: "some-key12") + + byte_size_sum = + byte_size("some-key1") + byte_size("some-value1") + + byte_size("some-key12") + byte_size("some-value12") + + assert %{byte_size_sum: ^byte_size_sum} = + Repo.get!(Collection, collection.id) + end + + test "updates the value of an item when key exists" do + collection = insert(:collection) + + assert :ok = Collections.put(collection, "some-key", "some-value1") + + assert %{key: "some-key", value: "some-value1"} = + Repo.get_by!(Item, key: "some-key") + + assert :ok = Collections.put(collection, "some-key", "some-value12") + + assert %{key: "some-key", value: "some-value12"} = + Repo.get_by!(Item, key: "some-key") + + byte_size_sum = byte_size("some-key") + byte_size("some-value12") + + assert %{byte_size_sum: ^byte_size_sum} = + Repo.get!(Collection, collection.id) + end + + test "returns an :error if the value is bigger than max len" do + collection = insert(:collection) + + assert {:error, + %{ + errors: [ + value: + {"should be at most %{count} character(s)", + [ + count: 1_000_000, + validation: :length, + kind: :max, + type: :string + ]} + ] + }} = + Collections.put( + collection, + "key", + String.duplicate("a", 1_000_001) + ) + end + + test "refuses a key too long to store with a changeset erring on :key" do + collection = insert(:collection) + + assert {:error, %Ecto.Changeset{errors: errors}} = + Collections.put(collection, String.duplicate("k", 256), "v") + + assert [{:key, {"key is too long, please use a shorter one", _opts}}] = + errors + end + + test "returns an :error if the collection does not exist" do + assert {:error, + %{ + errors: [ + collection_id: + {"does not exist", + [ + constraint: :foreign, + constraint_name: "collection_items_collection_id_fkey" + ]} + ] + }} = + Collections.put( + %Collection{id: Ecto.UUID.generate()}, + "key", + "value" + ) + end + + test "refuses a value that isn't a string" do + collection = insert(:collection) + + assert {:error, changeset} = + Collections.put(collection, "key", %{"nested" => true}) + + assert changeset.errors == [ + value: {"is invalid", [type: :string, validation: :cast]} + ] + end + + test "answers the hook's size refusal as a limit, writing nothing" do + collection = insert(:collection) + message = %Lightning.Extensions.Message{text: "over the limit"} + + Mox.stub( + Lightning.Extensions.MockCollectionHook, + :handle_put_items, + fn _collection, _size -> {:error, :exceeds_limit, message} end + ) + + assert {:error, :exceeds_limit, ^message} = + Collections.put(collection, "key", "value") + + assert Repo.all(Item) == [] + end + + test "refuses an invalid value before asking the hook about its size" do + collection = insert(:collection) + + Mox.expect( + Lightning.Extensions.MockCollectionHook, + :handle_put_items, + 0, + fn _collection, _size -> :ok end + ) + + assert {:error, %Ecto.Changeset{}} = + Collections.put( + collection, + "key", + String.duplicate("a", 1_000_001) + ) + + Mox.verify!() + end + end + + describe "put_all/2" do + test "inserts multiple entries at once in a given collection" do + collection = insert(:collection) + + {items, storage_used} = + Enum.map_reduce(10..50, 0, fn i, mem_used -> + { + %{"key" => "key#{i}", "value" => "value#{i}"}, + mem_used + byte_size("key#{i}") + byte_size("value#{i}") + } + end) + + assert {:ok, 41} = Collections.put_all(collection, items) + + assert Item + |> Repo.all() + |> Enum.map(&Map.take(&1, [:key, :value])) == + Enum.map(items, fn item -> + %{ + key: item["key"], + value: item["value"] + } + end) + + assert %{byte_size_sum: ^storage_used} = + Repo.get!(Collection, collection.id) + end + + test "replaces conflicting values and timestamps" do + collection = insert(:collection) + + items = + Enum.map(1..5, fn i -> %{"key" => "key#{i}", "value" => "value#{i}"} end) + + assert {:ok, 5} = Collections.put_all(collection, items) + + assert %{updated_at: updated_at1} = Repo.get_by(Item, key: "key1") + assert %{updated_at: updated_at2} = Repo.get_by(Item, key: "key2") + assert %{updated_at: updated_at5} = Repo.get_by(Item, key: "key5") + + update_items = + Enum.map(1..2, fn i -> + %{"key" => "key#{i}", "value" => "value#{10 + i}"} + end) + + assert {:ok, 2} = Collections.put_all(collection, update_items) + + assert %{value: "value11", updated_at: updated_at} = + Repo.get_by(Item, key: "key1") + + assert DateTime.after?(updated_at, updated_at1) + + assert %{value: "value12", updated_at: updated_at} = + Repo.get_by(Item, key: "key2") + + assert DateTime.after?(updated_at, updated_at2) + + assert %{value: "value5", updated_at: ^updated_at5} = + Repo.get_by(Item, key: "key5") + + final_items = Enum.drop(items, 2) ++ update_items + + assert Item + |> Repo.all() + |> Enum.map(&Map.take(&1, [:key, :value])) == + Enum.map(final_items, fn item -> + %{ + key: item["key"], + value: item["value"] + } + end) + + storage_used = + Enum.map(final_items, &(byte_size(&1["key"]) + byte_size(&1["value"]))) + |> Enum.sum() + + assert %{byte_size_sum: ^storage_used} = + Repo.get!(Collection, collection.id) + end + + test "refuses every item whose key an earlier item uses, at its position" do + collection = insert(:collection) + + items = + Enum.map( + ["a", "b", "a", "a"], + &%{"key" => &1, "value" => "v"} + ) + + assert {:error, changeset} = Collections.put_all(collection, items) + + assert Enum.sort(changeset.errors) == + Enum.sort( + items: + {"names the same key as #/items/0", + [code: :duplicate, key: ["items", 2, "key"]]}, + items: + {"names the same key as #/items/0", + [code: :duplicate, key: ["items", 3, "key"]]} + ) + + assert Repo.all(Item) == [] + end + + test "refuses invalid items at their positions, writing none of the list" do + collection = insert(:collection) + + items = [ + %{"key" => "ok", "value" => "v"}, + %{"key" => "missing-value"}, + %{"key" => "too-long", "value" => String.duplicate("a", 1_000_001)}, + "not an object", + %{"key" => "not-a-string", "value" => %{"nested" => true}} + ] + + assert {:error, changeset} = Collections.put_all(collection, items) + + assert Enum.sort(changeset.errors) == + Enum.sort( + items: + {"can't be blank", + [key: ["items", 1, :value], validation: :required]}, + items: + {"should be at most %{count} character(s)", + [ + key: ["items", 2, :value], + count: 1_000_000, + validation: :length, + kind: :max, + type: :string + ]}, + items: + {"is invalid", + [key: ["items", 3], validation: :cast, type: :map]}, + items: + {"is invalid", + [ + key: ["items", 4, :value], + type: :string, + validation: :cast + ]} + ) + + assert Repo.all(Item) == [] + end + + test "refuses items that aren't a list" do + collection = insert(:collection) + + assert {:error, changeset} = Collections.put_all(collection, "items") + + assert changeset.errors == [ + items: {"is invalid", [validation: :cast, type: {:array, :map}]} + ] + end + + test "answers the hook's size refusal as a limit, writing nothing" do + collection = insert(:collection) + message = %Lightning.Extensions.Message{text: "over the limit"} + + Mox.stub( + Lightning.Extensions.MockCollectionHook, + :handle_put_items, + fn _collection, _size -> {:error, :exceeds_limit, message} end + ) + + assert {:error, :exceeds_limit, ^message} = + Collections.put_all(collection, [ + %{"key" => "k", "value" => "v"} + ]) + + assert Repo.all(Item) == [] + end + + test "refuses an invalid list before asking the hook about its size" do + collection = insert(:collection) + + Mox.expect( + Lightning.Extensions.MockCollectionHook, + :handle_put_items, + 0, + fn _collection, _size -> :ok end + ) + + assert {:error, %Ecto.Changeset{}} = + Collections.put_all(collection, [ + %{"key" => "k", "value" => "v"}, + %{"key" => "k", "value" => "v"} + ]) + + Mox.verify!() + end + end + + describe "delete/2" do + test "deletes an entry for the given collection" do + collection = insert(:collection) + + key1 = "áàâãäéèêëíìîï" + key2 = "óòôõöúùûüçñabc" + + :ok = Collections.put(collection, key1, "12345") + :ok = Collections.put(collection, key2, "12345") + + assert Collections.get(collection, key1) + assert Collections.get(collection, key2) + + assert :ok = Collections.delete(collection, key1) + + refute Collections.get(collection, key1) + assert Collections.get(collection, key2) + + byte_size2 = byte_size(key2) + byte_size("12345") + + assert %{byte_size_sum: ^byte_size2} = + Repo.get!(Collection, collection.id) + end + + test "answers not_found for an item or a collection that doesn't exist" do + collection = insert(:collection) + + assert {:error, :not_found} = + Collections.delete(collection, "nonexistent") + + assert {:error, :not_found} = + Collections.delete(%{id: Ecto.UUID.generate()}, "key") + end + end + + describe "delete_all/2" do + test "deletes all items of the given collection" do + collection = insert(:collection) + + items = insert_list(3, :collection_item, collection: collection) + + assert {:ok, 3} = Collections.delete_all(collection) + + refute Enum.any?(items, &Collections.get(collection, &1.key)) + + assert %{byte_size_sum: 0} = Repo.get!(Collection, collection.id) + end + + test "deletes the items matching a pattern and leaves the rest" do + collection = insert(:collection) + + for key <- ["foo:111:bär1", "foo:222:bär2", "foo:333:bär3", "foo:444:bár4"], + do: :ok = Collections.put(collection, key, "12345") + + kept_size = byte_size("foo:444:bár4") + byte_size("12345") + + assert {:ok, 3} = Collections.delete_all(collection, "foo:*:bär*") + + assert [%Item{key: "foo:444:bár4"}] = + Collections.get_all(collection, %{limit: 50}) + + assert %{byte_size_sum: ^kept_size} = + Repo.get!(Collection, collection.id) + end + + test "deletes every item for the match-all pattern, and nothing from an empty collection" do + collection = insert(:collection) + + :ok = Collections.put(collection, "foo:111:bär1", "12345") + :ok = Collections.put(collection, "foo:222:bär2", "12345") + + assert {:ok, 2} = Collections.delete_all(collection, "*") + + assert [] = Collections.get_all(collection, %{limit: 50}) + assert %{byte_size_sum: 0} = Repo.get!(Collection, collection.id) + assert {:ok, 0} = Collections.delete_all(collection, "*") + end + end +end diff --git a/test/lightning/collections_items_test.exs b/test/lightning/collections_items_test.exs new file mode 100644 index 00000000000..bacdca1c61b --- /dev/null +++ b/test/lightning/collections_items_test.exs @@ -0,0 +1,216 @@ +defmodule Lightning.CollectionsItemsTest do + use Lightning.DataCase, async: true + + import Lightning.QueryCountHelpers + + alias Lightning.Collections + alias Lightning.Collections.Item + + describe "get/2" do + test "returns the entry of the given collection, or nil" do + %{key: key, value: value, collection: collection} = + insert(:collection_item) |> Repo.preload(:collection) + + assert %Item{key: ^key, value: ^value} = Collections.get(collection, key) + refute Collections.get(collection, "nonexistent") + refute Collections.get(%{id: Ecto.UUID.generate()}, key) + end + end + + describe "get_all/3" do + test "returns all items for the given collection sorted by inserted_at" do + collection = insert(:collection) + + items = + 1..11 + |> Enum.map(fn _i -> + insert(:collection_item, + key: "rkey#{:rand.uniform()}", + collection: collection + ) + end) + + get_items = + Collections.get_all(collection, limit: 50) + |> Repo.preload(collection: :project) + + assert List.last(get_items) == + Enum.sort_by(items, & &1.inserted_at) |> List.last() + + assert MapSet.new(get_items) == MapSet.new(items) + end + + test "returns the items after a cursor up to a limited amount" do + collection = insert(:collection) + + items = + Enum.map(1..30, fn _i -> + insert(:collection_item, + key: "rkey#{:rand.uniform()}", + collection: collection + ) + end) + + %{id: cursor} = Enum.at(items, 4) + + assert Collections.get_all(collection, cursor: cursor, limit: 50) + |> Enum.count() == 30 - (4 + 1) + + assert Collections.get_all(collection, cursor: cursor, limit: 10) + |> Enum.count() == 10 + end + + test "returns an empty list for an empty or missing collection" do + assert [] = Collections.get_all(insert(:collection), limit: 50) + + insert(:collection_item, key: "existing_key") + + assert [] = Collections.get_all(%{id: Ecto.UUID.generate()}, limit: 50) + end + end + + describe "stream_all/3" do + setup do + collection = insert(:collection) + insert_list(20, :collection_item, collection: collection) + %{collection: collection} + end + + test "reads every item a page at a time, stopping at the short page", + %{collection: collection} do + Mox.stub(Lightning.MockConfig, :collections_stream_page_size, fn -> 15 end) + + {items, queries} = + count_queries( + fn -> collection |> Collections.stream_all(%{}) |> Enum.to_list() end, + "collection_items" + ) + + assert Enum.map(items, & &1.id) == + Enum.map(Collections.get_all(collection, limit: 50), & &1.id) + + assert queries == 2 + end + + test "reads no more than :limit asks for", %{collection: collection} do + {items, queries} = + count_queries( + fn -> + collection |> Collections.stream_all(%{limit: 5}) |> Enum.to_list() + end, + "collection_items" + ) + + assert Enum.map(items, & &1.id) == + Enum.map(Collections.get_all(collection, limit: 5), & &1.id) + + assert queries == 1 + end + end + + describe "get_all/3 with key pattern" do + test "returns item with exact match" do + collection = insert(:collection) + _itemA = insert(:collection_item, key: "keyA", collection: collection) + itemB = insert(:collection_item, key: "keyB", collection: collection) + + assert [itemB] == + Collections.get_all(collection, %{limit: 50}, "keyB*") + |> Repo.preload(collection: :project) + end + + test "returns matching items for the given collection sorted by id" do + collection = insert(:collection) + + insert(:collection_item, key: "rkeynomatch", collection: collection) + + items = + Enum.map(1..11, fn _i -> + insert(:collection_item, + key: "rkeymatch#{:rand.uniform()}", + collection: collection + ) + |> item_reload() + end) + + get_items = Collections.get_all(collection, %{limit: 50}, "rkeymatch*") + + assert List.last(get_items) == Enum.max_by(items, & &1.id) + + assert MapSet.new(get_items) == MapSet.new(items) + end + + test "returns matching items after a cursor up to a limited amount" do + collection = insert(:collection) + + items = + Enum.map(1..30, fn _i -> + insert(:collection_item, + key: "rkeyA#{:rand.uniform()}", + collection: collection + ) + end) + + %{id: cursor} = Enum.at(items, 9) + + insert(:collection_item, key: "rkeyB", collection: collection) + + assert Collections.get_all( + collection, + %{cursor: cursor, limit: 50}, + "rkeyA*" + ) + |> Enum.count() == 30 - (9 + 1) + + assert Collections.get_all( + collection, + %{cursor: cursor, limit: 16}, + "rkeyA*" + ) + |> Enum.count() == 16 + end + + test "returns an empty list for an empty or missing collection" do + assert [] = + Collections.get_all(insert(:collection), %{limit: 50}, "any-key") + + insert(:collection_item, key: "existing_key") + + assert [] = + Collections.get_all( + %{id: Ecto.UUID.generate()}, + %{limit: 50}, + "existing_key" + ) + end + + test "returns item escaping the %" do + collection = insert(:collection) + + item = + insert(:collection_item, key: "keyA%", collection: collection) + |> item_reload() + + assert [item] == Collections.get_all(collection, %{limit: 50}, "keyA%*") + + insert(:collection_item, key: "keyBC", collection: collection) + + assert [] = Collections.get_all(collection, %{limit: 50}, "keyB%") + end + + test "returns item escaping the \\" do + collection = insert(:collection) + + item = + insert(:collection_item, key: "keyA\\", collection: collection) + |> item_reload() + + assert [item] == Collections.get_all(collection, %{limit: 50}, "keyA\\*") + end + end + + defp item_reload(item) do + # composite primary + Repo.get_by(Item, id: item.id, collection_id: item.collection_id) + end +end diff --git a/test/lightning/collections_test.exs b/test/lightning/collections_test.exs index 15d19b24dcc..989cd47cb96 100644 --- a/test/lightning/collections_test.exs +++ b/test/lightning/collections_test.exs @@ -1,95 +1,17 @@ defmodule Lightning.CollectionsTest do use Lightning.DataCase, async: true + import Lightning.QueryCountHelpers + alias Lightning.Collections alias Lightning.Collections.Collection - alias Lightning.Collections.Item - - describe "Item.changeset/2 key width" do - # Key width only. Null bytes in item keys are not covered here. - test "an over-long key is a changeset error, not a 22001" do - collection = insert(:collection) - - changeset = - Item.changeset(%Item{}, %{ - collection_id: collection.id, - key: String.duplicate("a", 300), - value: "v" - }) - - assert errors_on(changeset)[:key] == [ - "key is too long, please use a shorter one" - ] - end - - test "a key short in graphemes but too wide for the column is rejected" do - collection = insert(:collection) - family = "\u{1F468}\u{200D}\u{1F469}\u{200D}\u{1F467}\u{200D}\u{1F466}" - - changeset = - Item.changeset(%Item{}, %{ - collection_id: collection.id, - key: String.duplicate(family, 200), - value: "v" - }) - - assert errors_on(changeset)[:key] == [ - "key is too long, please use a shorter one" - ] - end - - test "a value short in graphemes but too wide for the column is rejected" do - collection = insert(:collection) - family = "\u{1F468}\u{200D}\u{1F469}\u{200D}\u{1F467}\u{200D}\u{1F466}" - - # 200_000 graphemes, which clears the 1_000_000 grapheme cap, but - # 1_400_000 codepoints, which is what varchar(1000000) counts. - changeset = - Item.changeset(%Item{}, %{ - collection_id: collection.id, - key: "k", - value: String.duplicate(family, 200_000) - }) - - assert errors_on(changeset)[:value] == [ - "value is too long, please use a shorter one" - ] - end - - test "a key at the column width is accepted" do - collection = insert(:collection) - changeset = - Item.changeset(%Item{}, %{ - collection_id: collection.id, - key: String.duplicate("a", 255), - value: "v" - }) + describe "get_collection_by_id/1" do + test "returns the collection with the id, or nil" do + %{id: id} = insert(:collection) - refute errors_on(changeset)[:key] - assert {:ok, _} = Repo.insert(changeset) - end - end - - describe "get_collection/1" do - test "get a collection" do - %{id: collection_id, name: collection_name} = insert(:collection) - - assert {:ok, %Collection{id: ^collection_id}} = - Collections.get_collection(collection_name) - end - - test "returns an error when the collection does not exist" do - assert {:error, :not_found} = - Collections.get_collection("nonexistent") - end - - test "returns a conflict error when the same name exists in multiple projects" do - name = "shared-name" - insert(:collection, name: name) - insert(:collection, name: name) - - assert {:error, :conflict} = Collections.get_collection(name) + assert %Collection{id: ^id} = Collections.get_collection_by_id(id) + assert Collections.get_collection_by_id(Ecto.UUID.generate()) == nil end end @@ -116,587 +38,377 @@ defmodule Lightning.CollectionsTest do end end - describe "create_collection/2" do - test "creates a new collection" do - %{id: project_id} = insert(:project) - name = "col1_project1" + describe "resolve_collection/3" do + setup do + user = insert(:user) + mine = insert(:project, project_users: [%{user: user, role: :viewer}]) + theirs = insert(:project) - assert {:ok, %Collection{project_id: ^project_id, name: ^name}} = - Collections.create_collection(%{ - "project_id" => project_id, - "name" => name - }) + %{ + user: user, + mine: insert(:collection, name: "shared", project: mine), + theirs: insert(:collection, name: "shared", project: theirs) + } end - test "returns an error when collection name is taken" do - %{id: project_id1} = insert(:project) - name = "col1_project1" - - assert {:ok, %Collection{project_id: ^project_id1, name: ^name}} = - Collections.create_collection(%{ - "project_id" => project_id1, - "name" => name - }) - - assert {:error, - %{ - errors: [ - name: - {"A collection with this name already exists", - [ - constraint: :unique, - constraint_name: "collections_project_id_name_index" - ]} - ] - }} = - Collections.create_collection(%{ - "project_id" => project_id1, - "name" => name - }) + test "finds the one collection of that name the user has standing on", + %{user: user, mine: %{id: id}} do + assert {:ok, %Collection{id: ^id}} = + Collections.resolve_collection(user, "shared") end - test "returns an error when limit is exceeded" do - %{id: project_id1} = insert(:project) - %{id: project_id2} = insert(:project) - - assert {:ok, %Collection{project_id: ^project_id1, name: "col1"}} = - Collections.create_collection(%{ - "project_id" => project_id1, - "name" => "col1" - }) - - message = %Lightning.Extensions.Message{text: "some error"} + test "is ambiguous only among collections the user has standing on", + %{user: user, theirs: theirs} do + insert(:project_user, user: user, project_id: theirs.project_id) - Mox.stub( - Lightning.Extensions.MockCollectionHook, - :handle_create, - fn %{"project_id" => _project_id} -> - {:error, :exceeds_limit, message} - end - ) - - assert {:error, :exceeds_limit, ^message} = - Collections.create_collection(%{ - "project_id" => project_id2, - "name" => "col2" - }) + assert {:error, :ambiguous_name} = + Collections.resolve_collection(user, "shared") end - end - describe "delete_collection/1" do - test "deletes a collection" do - %{id: collection_id} = insert(:collection) + test "narrows to the project named by project_id", + %{user: user, theirs: %{id: id} = theirs} do + insert(:project_user, user: user, project_id: theirs.project_id) - assert {:ok, %Collection{id: ^collection_id}} = - Collections.delete_collection(collection_id) + assert {:ok, %Collection{id: ^id}} = + Collections.resolve_collection(user, "shared", theirs.project_id) end - test "returns an error when collection does not exist" do + test "finds nothing in a project the user has no standing on", + %{user: user, theirs: theirs} do assert {:error, :not_found} = - Collections.delete_collection(Ecto.UUID.generate()) - end - end + Collections.resolve_collection(user, "shared", theirs.project_id) - describe "get/2" do - test "returns an entry for the given collection" do - %{key: key, value: value, collection: collection} = - insert(:collection_item) |> Repo.preload(:collection) - - assert %Item{key: ^key, value: ^value} = Collections.get(collection, key) - end - - test "returns nil if the item key does not exist" do - collection = insert(:collection) - - refute Collections.get(collection, "nonexistent") + assert {:error, :not_found} = + Collections.resolve_collection(insert(:user), "shared") end - test "returns nil if the collection does not exist" do - insert(:collection_item, key: "existing_key") + test "finds nothing for a name no collection has, or a project_id that isn't a UUID", + %{user: user} do + assert {:error, :not_found} = + Collections.resolve_collection(user, "nonexistent") - refute Collections.get(%{id: Ecto.UUID.generate()}, "existing_key") + assert {:error, :not_found} = + Collections.resolve_collection(user, "shared", "not-a-uuid") end - end - - describe "get_all/3" do - test "returns all items for the given collection sorted by inserted_at" do - collection = insert(:collection) - items = - 1..11 - |> Enum.map(fn _i -> - insert(:collection_item, - key: "rkey#{:rand.uniform()}", - collection: collection - ) - end) + test "reads a 16-character project_id as text, not as a raw UUID", + %{user: user} do + raw = "abcdefghijklmnop" + {:ok, project_id} = Ecto.UUID.load(raw) - get_items = - Collections.get_all(collection, limit: 50) - |> Repo.preload(collection: :project) + project = + insert(:project, + id: project_id, + project_users: [%{user: user, role: :viewer}] + ) - assert List.last(get_items) == - Enum.sort_by(items, & &1.inserted_at) |> List.last() + insert(:collection, name: "shared", project: project) - assert MapSet.new(get_items) == MapSet.new(items) + assert {:error, :not_found} = + Collections.resolve_collection(user, "shared", raw) end - test "returns the items after a cursor up to a limited amount" do - collection = insert(:collection) + test "counts support access as standing, and only where it is allowed", + %{theirs: theirs} do + support_user = insert(:user, support_user: true) - items = - Enum.map(1..30, fn _i -> - insert(:collection_item, - key: "rkey#{:rand.uniform()}", - collection: collection - ) - end) + assert {:error, :not_found} = + Collections.resolve_collection(support_user, "shared") - %{id: cursor} = Enum.at(items, 4) + Repo.update_all( + from(p in Lightning.Projects.Project, + where: p.id == ^theirs.project_id + ), + set: [allow_support_access: true] + ) - assert Collections.get_all(collection, cursor: cursor, limit: 50) - |> Enum.count() == 30 - (4 + 1) + theirs_id = theirs.id - assert Collections.get_all(collection, cursor: cursor, limit: 10) - |> Enum.count() == 10 + assert {:ok, %Collection{id: ^theirs_id}} = + Collections.resolve_collection(support_user, "shared") end - test "returns empty list when collection is empty" do - collection = insert(:collection) + test "resolves for a member who hasn't met the project's MFA requirement" do + user = insert(:user, mfa_enabled: false) - assert [] = Collections.get_all(collection, limit: 50) - end + project = + insert(:project, requires_mfa: true, project_users: [%{user: user}]) - test "returns empty list when the collection doesn't exist" do - insert(:collection_item, key: "existing_key") + %{id: id} = insert(:collection, name: "guarded", project: project) - assert [] = Collections.get_all(%{id: Ecto.UUID.generate()}, limit: 50) + assert {:ok, %Collection{id: ^id}} = + Collections.resolve_collection(user, "guarded") end - end - describe "get_all/3 with key pattern" do - test "returns item with exact match" do - collection = insert(:collection) - _itemA = insert(:collection_item, key: "keyA", collection: collection) - itemB = insert(:collection_item, key: "keyB", collection: collection) + test "gives a run standing on its own project only", + %{mine: %{id: id} = mine, theirs: theirs} do + workflow = insert(:simple_workflow, project: mine.project) + work_order = insert(:workorder, workflow: workflow) - assert [itemB] == - Collections.get_all(collection, %{limit: 50}, "keyB*") - |> Repo.preload(collection: :project) - end + run = + insert(:run, + work_order: work_order, + dataclip: build(:dataclip), + starting_trigger: hd(workflow.triggers) + ) - test "returns matching items for the given collection sorted by id" do - collection = insert(:collection) + assert {:ok, %Collection{id: ^id}} = + Collections.resolve_collection(run, "shared") - insert(:collection_item, key: "rkeynomatch", collection: collection) - - items = - Enum.map(1..11, fn _i -> - insert(:collection_item, - key: "rkeymatch#{:rand.uniform()}", - collection: collection - ) - |> item_reload() - end) - - get_items = Collections.get_all(collection, %{limit: 50}, "rkeymatch*") - - assert List.last(get_items) == Enum.max_by(items, & &1.id) - - assert MapSet.new(get_items) == MapSet.new(items) + assert {:error, :not_found} = + Collections.resolve_collection(run, "shared", theirs.project_id) end - test "returns matching items after a cursor up to a limited amount" do - collection = insert(:collection) - - items = - Enum.map(1..30, fn _i -> - insert(:collection_item, - key: "rkeyA#{:rand.uniform()}", - collection: collection - ) - end) - - %{id: cursor} = Enum.at(items, 9) + test "reads standing in two queries however many projects share the name", + %{user: user, mine: %{id: id}} do + insert(:collection, name: "shared") - insert(:collection_item, key: "rkeyB", collection: collection) - - assert Collections.get_all( - collection, - %{cursor: cursor, limit: 50}, - "rkeyA*" - ) - |> Enum.count() == 30 - (9 + 1) - - assert Collections.get_all( - collection, - %{cursor: cursor, limit: 16}, - "rkeyA*" - ) - |> Enum.count() == 16 + assert {{:ok, %Collection{id: ^id}}, 2} = + count_queries(fn -> + Collections.resolve_collection(user, "shared") + end) end + end - test "returns empty list when collection is empty" do - collection = insert(:collection) - - assert [] = Collections.get_all(collection, %{limit: 50}, "any-key") + describe "create_collection/3" do + setup do + %{actor: insert(:user)} end - test "returns empty list when the collection doesn't exist" do - insert(:collection_item, key: "existing_key") + test "creates a collection and records its creator", %{actor: actor} do + %{id: project_id} = insert(:project) + name = "col1_project1" - assert [] = - Collections.get_all( - %{id: Ecto.UUID.generate()}, - %{limit: 50}, - "existing_key" + assert {:ok, %Collection{id: id, project_id: ^project_id, name: ^name}} = + Collections.create_collection( + %{"project_id" => project_id, "name" => name}, + actor ) - end - - test "returns item escaping the %" do - collection = insert(:collection) - item = - insert(:collection_item, key: "keyA%", collection: collection) - |> item_reload() + assert [audit] = collection_audits(id) + assert audit.event == "created" + assert audit.actor_id == actor.id + assert audit.actor_type == :user + assert audit.changes.before == nil - assert [item] == Collections.get_all(collection, %{limit: 50}, "keyA%*") - - insert(:collection_item, key: "keyBC", collection: collection) - - assert [] = Collections.get_all(collection, %{limit: 50}, "keyB%") + assert audit.changes.after == %{ + "name" => name, + "project_id" => project_id + } end - test "returns item escaping the \\" do - collection = insert(:collection) - - item = - insert(:collection_item, key: "keyA\\", collection: collection) - |> item_reload() + test "records a service account as the creator" do + {account, _key} = + Lightning.ServiceAccountHelpers.service_account_with_key() - assert [item] == Collections.get_all(collection, %{limit: 50}, "keyA\\*") - end - end - - describe "put/3" do - test "creates a new entry in the collection for the given collection" do - collection = insert(:collection) - - assert :ok = Collections.put(collection, "some-key1", "some-value1") - assert :ok = Collections.put(collection, "some-key12", "some-value12") - - assert %{key: "some-key1", value: "some-value1"} = - Repo.get_by!(Item, key: "some-key1") - - assert %{key: "some-key12", value: "some-value12"} = - Repo.get_by!(Item, key: "some-key12") + %{id: project_id} = insert(:project) - byte_size_sum = - byte_size("some-key1") + byte_size("some-value1") + - byte_size("some-key12") + byte_size("some-value12") + assert {:ok, %Collection{id: id}} = + Collections.create_collection( + %{"project_id" => project_id, "name" => "patients"}, + account + ) - assert %{byte_size_sum: ^byte_size_sum} = - Repo.get!(Collection, collection.id) + assert [audit] = collection_audits(id) + assert audit.actor_id == account.uuid + assert audit.actor_type == :service_account + assert audit.metadata["service_account_id"] == account.id end - test "updates the value of an item when key exists" do - collection = insert(:collection) + test "creates under the id it is given, never one cast from attrs", %{ + actor: actor + } do + %{id: project_id} = insert(:project) + id = Ecto.UUID.generate() - assert :ok = Collections.put(collection, "some-key", "some-value1") + assert {:ok, %Collection{id: ^id}} = + Collections.create_collection( + %{"project_id" => project_id, "name" => "given"}, + actor, + id: id + ) - assert %{key: "some-key", value: "some-value1"} = - Repo.get_by!(Item, key: "some-key") + assert {:ok, %Collection{id: other_id}} = + Collections.create_collection( + %{"project_id" => project_id, "name" => "cast", "id" => id}, + actor + ) - assert :ok = Collections.put(collection, "some-key", "some-value12") + refute other_id == id + end - assert %{key: "some-key", value: "some-value12"} = - Repo.get_by!(Item, key: "some-key") + test "refuses an id already in use as a changeset error", %{actor: actor} do + existing = insert(:collection) - byte_size_sum = byte_size("some-key") + byte_size("some-value12") + assert {:error, changeset} = + Collections.create_collection( + %{"project_id" => insert(:project).id, "name" => "other"}, + actor, + id: existing.id + ) - assert %{byte_size_sum: ^byte_size_sum} = - Repo.get!(Collection, collection.id) + assert {_message, opts} = changeset.errors[:id] + assert opts[:constraint] == :unique + assert collection_audits(existing.id) == [] end - test "returns an :error if the value is bigger than max len" do - collection = insert(:collection) + test "returns an error when collection name is taken", %{actor: actor} do + %{id: project_id1} = insert(:project) + name = "col1_project1" - assert {:error, - %{ - errors: [ - value: - {"should be at most %{count} character(s)", - [ - count: 1_000_000, - validation: :length, - kind: :max, - type: :string - ]} - ] - }} = - Collections.put( - collection, - "key", - String.duplicate("a", 1_000_001) + assert {:ok, %Collection{project_id: ^project_id1, name: ^name}} = + Collections.create_collection( + %{"project_id" => project_id1, "name" => name}, + actor ) - end - test "returns an :error if the collection does not exist" do assert {:error, %{ errors: [ - collection_id: - {"does not exist", + name: + {"A collection with this name already exists", [ - constraint: :foreign, - constraint_name: "collection_items_collection_id_fkey" + constraint: :unique, + constraint_name: "collections_project_id_name_index" ]} ] }} = - Collections.put( - %Collection{id: Ecto.UUID.generate()}, - "key", - "value" + Collections.create_collection( + %{"project_id" => project_id1, "name" => name}, + actor ) end - end - - describe "put_all/2" do - test "inserts multiple entries at once in a given collection" do - collection = insert(:collection) - - {items, storage_used} = - Enum.map_reduce(10..50, 0, fn i, mem_used -> - { - %{"key" => "key#{i}", "value" => "value#{i}"}, - mem_used + byte_size("key#{i}") + byte_size("value#{i}") - } - end) - - assert {:ok, 41} = Collections.put_all(collection, items) - - assert Item - |> Repo.all() - |> Enum.map(&Map.take(&1, [:key, :value])) == - Enum.map(items, fn item -> - %{ - key: item["key"], - value: item["value"] - } - end) - - assert %{byte_size_sum: ^storage_used} = - Repo.get!(Collection, collection.id) - end - - test "replaces conflicting values and timestamps" do - collection = insert(:collection) - items = - Enum.map(1..5, fn i -> %{"key" => "key#{i}", "value" => "value#{i}"} end) - - assert {:ok, 5} = Collections.put_all(collection, items) - - assert %{updated_at: updated_at1} = Repo.get_by(Item, key: "key1") - assert %{updated_at: updated_at2} = Repo.get_by(Item, key: "key2") - assert %{updated_at: updated_at5} = Repo.get_by(Item, key: "key5") - - update_items = - Enum.map(1..2, fn i -> - %{"key" => "key#{i}", "value" => "value#{10 + i}"} - end) - - assert {:ok, 2} = Collections.put_all(collection, update_items) - - assert %{value: "value11", updated_at: updated_at} = - Repo.get_by(Item, key: "key1") - - assert DateTime.after?(updated_at, updated_at1) - - assert %{value: "value12", updated_at: updated_at} = - Repo.get_by(Item, key: "key2") + test "writes nothing when the limit is exceeded", %{actor: actor} do + %{id: project_id1} = insert(:project) + %{id: project_id2} = insert(:project) - assert DateTime.after?(updated_at, updated_at2) + assert {:ok, %Collection{project_id: ^project_id1, name: "col1"}} = + Collections.create_collection( + %{"project_id" => project_id1, "name" => "col1"}, + actor + ) - assert %{value: "value5", updated_at: ^updated_at5} = - Repo.get_by(Item, key: "key5") + message = %Lightning.Extensions.Message{text: "some error"} - final_items = Enum.drop(items, 2) ++ update_items + Mox.stub( + Lightning.Extensions.MockCollectionHook, + :handle_create, + fn %{"project_id" => _project_id} -> + {:error, :exceeds_limit, message} + end + ) - assert Item - |> Repo.all() - |> Enum.map(&Map.take(&1, [:key, :value])) == - Enum.map(final_items, fn item -> - %{ - key: item["key"], - value: item["value"] - } - end) + id = Ecto.UUID.generate() - storage_used = - Enum.map(final_items, &(byte_size(&1["key"]) + byte_size(&1["value"]))) - |> Enum.sum() + assert {:error, :exceeds_limit, ^message} = + Collections.create_collection( + %{"project_id" => project_id2, "name" => "col2"}, + actor, + id: id + ) - assert %{byte_size_sum: ^storage_used} = - Repo.get!(Collection, collection.id) + refute Repo.get(Collection, id) + assert collection_audits(id) == [] end - test "raises Postgrex error when an item is bigger than allowed max len" do - collection = insert(:collection) + test "returns an error if invalid attributes are provided", %{actor: actor} do + attrs = %{"name" => nil, "project_id" => Ecto.UUID.generate()} - items = [ - %{"key" => "key1", "value" => "adsf"}, - %{"key" => "key1", "value" => String.duplicate("a", 1_000_001)} - ] + assert {:error, changeset} = Collections.create_collection(attrs, actor) - assert_raise Postgrex.Error, fn -> - Collections.put_all(collection, items) - end + assert %{name: ["can't be blank"]} == errors_on(changeset) end end - describe "delete/2" do - test "deletes an entry for the given collection" do - collection = insert(:collection) - - key1 = "áàâãäéèêëíìîï" - key2 = "óòôõöúùûüçñabc" - - :ok = Collections.put(collection, key1, "12345") - :ok = Collections.put(collection, key2, "12345") - - assert Collections.get(collection, key1) - assert Collections.get(collection, key2) - - assert :ok = Collections.delete(collection, key1) - - refute Collections.get(collection, key1) - assert Collections.get(collection, key2) - - byte_size2 = byte_size(key2) + byte_size("12345") - - assert %{byte_size_sum: ^byte_size2} = - Repo.get!(Collection, collection.id) + describe "update_collection/3" do + setup do + %{actor: insert(:user)} end - test "returns an :error if the collection does not exist" do - assert {:error, :not_found} = - Collections.delete(%{id: Ecto.UUID.generate()}, "key") - end - - test "returns an :error if item does not exist" do - collection = insert(:collection) - - assert {:error, :not_found} = - Collections.delete(collection, "nonexistent") - end - end + test "renames a collection, keeping its items, and records who did", %{ + actor: actor + } do + collection = insert(:collection, name: "old-name") + insert(:collection_item, collection: collection, key: "k", value: "v") - describe "delete_all/2" do - test "does nothing on an empty collection" do - collection = insert(:collection) + assert {:ok, %Collection{name: "updated-name"}} = + Collections.update_collection( + collection, + %{"name" => "updated-name"}, + actor + ) - assert %{byte_size_sum: 0} = Repo.get!(Collection, collection.id) + assert Collections.get(collection, "k").value == "v" - assert {:ok, 0} = Collections.delete_all(collection, "*") + assert [audit] = collection_audits(collection.id) + assert audit.event == "updated" + assert audit.actor_id == actor.id + assert audit.changes.before == %{"name" => "old-name"} + assert audit.changes.after == %{"name" => "updated-name"} end - test "deletes all items of the given collection" do - collection = insert(:collection) + test "a collection another request deleted is a concurrent edit, not a raise", + %{actor: actor} do + collection = insert(:collection, name: "old-name") + Repo.delete!(collection) - items = insert_list(3, :collection_item, collection: collection) - - assert {:ok, 3} = Collections.delete_all(collection) - - refute Enum.any?(items, &Collections.get(collection, &1.key)) - - assert %{byte_size_sum: 0} = Repo.get!(Collection, collection.id) + assert Collections.update_collection( + collection, + %{"name" => "updated-name"}, + actor + ) == {:error, :concurrent_edit} end - test "deletes matching items of the given collection" do - collection = insert(:collection) - - :ok = Collections.put(collection, "foo:111:bär1", "12345") - :ok = Collections.put(collection, "foo:222:bär2", "12345") - :ok = Collections.put(collection, "foo:333:bär3", "12345") - :ok = Collections.put(collection, "foo:444:bár4", "12345") - - item4_size = byte_size("foo:444:bár4") + byte_size("12345") - - collection_size = - item4_size + 3 * (byte_size("foo:333:bär3") + byte_size("12345")) + test "records nothing when the name is unchanged", %{actor: actor} do + collection = insert(:collection, name: "same-name") - assert %{byte_size_sum: ^collection_size} = - Repo.get!(Collection, collection.id) - - assert Collections.get(collection, "foo:111:bär1") - assert Collections.get(collection, "foo:222:bär2") - assert Collections.get(collection, "foo:333:bär3") - - assert {:ok, 3} = Collections.delete_all(collection, "foo:*:bär*") - - refute Collections.get(collection, "foo:111:bär1") - refute Collections.get(collection, "foo:222:bär2") - refute Collections.get(collection, "foo:333:bär3") - assert Collections.get(collection, "foo:444:bár4") + assert {:ok, %Collection{name: "same-name"}} = + Collections.update_collection( + collection, + %{"name" => "same-name"}, + actor + ) - assert %{byte_size_sum: ^item4_size} = Repo.get!(Collection, collection.id) + assert collection_audits(collection.id) == [] end - test "deletes all items by matching on the given collection" do + test "never moves a collection to another project", %{actor: actor} do collection = insert(:collection) + other_project = insert(:project) - :ok = Collections.put(collection, "foo:111:bär1", "12345") - :ok = Collections.put(collection, "foo:222:bär2", "12345") - - collection_size = 2 * (byte_size("foo:nnn:bärn") + byte_size("12345")) - - assert %{byte_size_sum: ^collection_size} = - Repo.get!(Collection, collection.id) - - assert Collections.get(collection, "foo:111:bär1") - assert Collections.get(collection, "foo:222:bär2") - - assert {:ok, 2} = Collections.delete_all(collection, "foo:*:bär*") - - refute Collections.get(collection, "foo:111:bär1") - refute Collections.get(collection, "foo:222:bär2") + assert {:ok, _collection} = + Collections.update_collection( + collection, + %{"name" => "moved", "project_id" => other_project.id}, + actor + ) - assert %{byte_size_sum: 0} = Repo.get!(Collection, collection.id) + assert Repo.reload!(collection).project_id == collection.project_id end - test "deletes matching all items on the given collection" do + test "returns an error if invalid attributes are provided", %{actor: actor} do collection = insert(:collection) - :ok = Collections.put(collection, "foo:111:bär1", "12345") - :ok = Collections.put(collection, "foo:222:bär2", "12345") - - collection_size = 2 * (byte_size("foo:nnn:bärn") + byte_size("12345")) - - assert %{byte_size_sum: ^collection_size} = - Repo.get!(Collection, collection.id) - - assert Collections.get(collection, "foo:111:bär1") - assert Collections.get(collection, "foo:222:bär2") - - assert {:ok, 2} = Collections.delete_all(collection, "*") - - refute Collections.get(collection, "foo:111:bär1") - refute Collections.get(collection, "foo:222:bär2") + assert {:error, changeset} = + Collections.update_collection(collection, %{name: nil}, actor) - assert %{byte_size_sum: 0} = Repo.get!(Collection, collection.id) + assert %{name: ["can't be blank"]} == errors_on(changeset) + assert collection_audits(collection.id) == [] end + end - test "returns an :error if the collection does not exist" do - assert {:error, :not_found} = - Collections.delete(%{id: Ecto.UUID.generate()}, "key") - end + describe "delete_collection/1" do + test "deletes a collection" do + %{id: collection_id} = insert(:collection) - test "returns an :error if item does not exist" do - collection = insert(:collection) + assert {:ok, %Collection{id: ^collection_id}} = + Collections.delete_collection(collection_id) + end + test "returns an error when collection does not exist" do assert {:error, :not_found} = - Collections.delete(collection, "nonexistent") + Collections.delete_collection(Ecto.UUID.generate()) end end @@ -729,46 +441,10 @@ defmodule Lightning.CollectionsTest do end end - describe "create_collection/1" do - test "creates a new collection with valid attributes" do - %{id: project_id} = insert(:project) - attrs = %{"name" => "new-collection", "project_id" => project_id} - - assert {:ok, %Collection{name: "new-collection"}} = - Collections.create_collection(attrs) - end - - test "returns an error if invalid attributes are provided" do - attrs = %{"name" => nil, "project_id" => Ecto.UUID.generate()} - - assert {:error, changeset} = Collections.create_collection(attrs) - - assert %{name: ["can't be blank"]} == errors_on(changeset) - end - end - - describe "update_collection/2" do - test "updates an existing collection with valid attributes" do - collection = insert(:collection, name: "Old Name") - attrs = %{name: "updated-name"} - - assert {:ok, %Collection{name: "updated-name"}} = - Collections.update_collection(collection, attrs) - end - - test "returns an error if invalid attributes are provided" do - collection = insert(:collection) - attrs = %{name: nil} - - assert {:error, changeset} = - Collections.update_collection(collection, attrs) - - assert %{name: ["can't be blank"]} == errors_on(changeset) - end - end - - defp item_reload(item) do - # composite primary - Repo.get_by(Item, id: item.id, collection_id: item.collection_id) + defp collection_audits(collection_id) do + Repo.all( + from a in Lightning.Collections.Audit.base_query(), + where: a.item_id == ^collection_id + ) end end diff --git a/test/lightning/credentials/audit_test.exs b/test/lightning/credentials/audit_test.exs index 42942e11be1..8fc660312bf 100644 --- a/test/lightning/credentials/audit_test.exs +++ b/test/lightning/credentials/audit_test.exs @@ -86,9 +86,14 @@ defmodule Lightning.Credentials.AuditTest do end describe ".user_initiated_event" do - test "generates changeset for event with environment bodies in metadata" do + test "names the actor passed in, not the owner, with environment bodies in metadata" do %{id: user_id} = user_fixture() + {account, _key} = + Lightning.ServiceAccountHelpers.service_account_with_key() + + account_uuid = account.uuid + %{id: credential_id} = credential = credential_fixture(user_id: user_id) @@ -103,15 +108,21 @@ defmodule Lightning.Credentials.AuditTest do ] audit_changeset = - Audit.user_initiated_event("updated", credential, changeset, env_bodies) + Audit.user_initiated_event( + "updated", + credential, + account, + changeset, + bodies: env_bodies + ) assert %{ changes: %{ event: "updated", item_id: ^credential_id, item_type: "credential", - actor_id: ^user_id, - actor_type: :user, + actor_id: ^account_uuid, + actor_type: :service_account, metadata: metadata } } = audit_changeset @@ -134,7 +145,12 @@ defmodule Lightning.Credentials.AuditTest do changeset = Credential.changeset(credential, %{name: "New Name"}) audit_changeset = - Audit.user_initiated_event("updated", credential, changeset) + Audit.user_initiated_event( + "updated", + credential, + credential.user, + changeset + ) assert %{ changes: %{ @@ -160,7 +176,8 @@ defmodule Lightning.Credentials.AuditTest do |> with_body(%{name: "main", body: %{}}) |> Repo.preload(:user) - audit_changeset = Audit.user_initiated_event("deleted", credential) + audit_changeset = + Audit.user_initiated_event("deleted", credential, credential.user) assert %{ changes: %{ @@ -194,7 +211,13 @@ defmodule Lightning.Credentials.AuditTest do changeset = Credential.changeset(credential, %{name: "New Credential"}) audit_changeset = - Audit.user_initiated_event("created", credential, changeset, env_bodies) + Audit.user_initiated_event( + "created", + credential, + credential.user, + changeset, + bodies: env_bodies + ) assert %{ changes: %{ diff --git a/test/lightning/credentials/oauth_client_test.exs b/test/lightning/credentials/oauth_client_test.exs index acff7df5e23..f279954ae2c 100644 --- a/test/lightning/credentials/oauth_client_test.exs +++ b/test/lightning/credentials/oauth_client_test.exs @@ -54,7 +54,7 @@ defmodule Lightning.Credentials.OauthClientTest do changeset = OauthClient.changeset(%OauthClient{}, @invalid_attrs) Enum.each(url_fields, fn field -> - assert {"must be either a http or https URL", []} == + assert {"must be either a http or https URL", [code: :format]} == changeset.errors[field] end) end diff --git a/test/lightning/credentials/oauth_validation_test.exs b/test/lightning/credentials/oauth_validation_test.exs index 48c21c69b82..fc741002faa 100644 --- a/test/lightning/credentials/oauth_validation_test.exs +++ b/test/lightning/credentials/oauth_validation_test.exs @@ -4,24 +4,6 @@ defmodule Lightning.Credentials.OauthValidationTest do alias Lightning.Credentials.OauthValidation alias Lightning.Credentials.OauthValidation.Error - describe "Error struct" do - test "creates error with type, message, and details" do - error = Error.new(:missing_scopes, "Test message", %{test: "data"}) - - assert error.type == :missing_scopes - assert error.message == "Test message" - assert error.details == %{test: "data"} - end - - test "creates error without details" do - error = Error.new(:invalid_token_format, "Test message") - - assert error.type == :invalid_token_format - assert error.message == "Test message" - assert error.details == nil - end - end - describe "validate_token_data/1" do test "validates complete valid token data with Bearer token type" do token_data = %{ @@ -229,10 +211,52 @@ defmodule Lightning.Credentials.OauthValidationTest do result = OauthValidation.validate_token_data(token_data) - assert {:error, %Error{type: :unsupported_token_type, message: message}} = - result + assert {:error, + %Error{ + type: :unsupported_token_type, + code: :not_allowed, + field: "token_type", + message: "Unsupported token type. Expected 'Bearer'", + details: %{token_type: "Basic"} + }} = result + end + + test "codes every refusal and quotes no value from the token" do + valid = %{ + "access_token" => "access_token_123", + "refresh_token" => "refresh_token_123", + "token_type" => "Bearer", + "expires_in" => 3600, + "scope" => "read write" + } - assert message == "Unsupported token type: 'Basic'. Expected 'Bearer'" + planted = "planted-1234" + + refused = [ + {%{"access_token" => [planted]}, :type, "access_token"}, + {%{"refresh_token" => %{"x" => planted}}, :type, "refresh_token"}, + {%{"token_type" => planted}, :not_allowed, "token_type"}, + {%{"token_type" => [planted]}, :type, "token_type"}, + {%{"scope" => [planted]}, :type, "scope"}, + {%{"scope" => nil, "scopes" => planted}, :type, "scopes"}, + {%{"expires_in" => planted}, :type, "expires_in"}, + {%{"expires_in" => 999_999_999}, :out_of_range, "expires_in"}, + {%{"expires_in" => nil, "expires_at" => planted}, :format, "expires_at"}, + {%{"expires_in" => nil, "expires_at" => "1700000000\n"}, :format, + "expires_at"} + ] + + for {overrides, code, field} <- refused do + assert {:error, %Error{code: ^code, field: ^field, message: message}} = + valid + |> Map.merge(overrides) + |> Map.reject(fn {_key, value} -> is_nil(value) end) + |> OauthValidation.validate_token_data(), + inspect(overrides) + + refute message =~ planted + refute message =~ "999999999" + end end test "rejects token without scope or scopes" do @@ -529,10 +553,10 @@ defmodule Lightning.Credentials.OauthValidationTest do assert details.expected_scopes == ["read", "write", "admin"] assert message == - "Missing required scopes: write, admin. Please reauthorize and grant all selected permissions." + "Missing required scopes. Please reauthorize and grant all selected permissions." end - test "preserves original case in error messages" do + test "preserves original case in the missing scopes" do token_data = %{"scope" => "read"} expected_scopes = ["read", "WRITE", "Admin"] @@ -543,9 +567,7 @@ defmodule Lightning.Credentials.OauthValidationTest do result assert details.missing_scopes == ["WRITE", "Admin"] - - assert message == - "Missing required scopes: WRITE, Admin. Please reauthorize and grant all selected permissions." + refute message =~ "WRITE" end test "fails when token has no scope data" do @@ -958,16 +980,19 @@ defmodule Lightning.Credentials.OauthValidationTest do assert ["a", "b"] = OauthValidation.normalize_scopes("A B ") end - test "validates error message formatting for missing scopes" do + test "tags missing scopes with an entry code, naming none of them in the message" do token_data = %{"scope" => "read"} expected_scopes = ["read", "write", "admin", "delete"] result = OauthValidation.validate_scope_grant(token_data, expected_scopes) - assert {:error, %Error{message: message}} = result - - assert message == - "Missing required scopes: write, admin, delete. Please reauthorize and grant all selected permissions." + assert {:error, + %Error{ + code: :scope_not_granted, + field: "scope", + message: + "Missing required scopes. Please reauthorize and grant all selected permissions." + }} = result end test "handles token with both expires_in and expires_at (expires_in takes precedence)" do diff --git a/test/lightning/credentials/schema_test.exs b/test/lightning/credentials/schema_test.exs index 8e50ec11b34..83a615ad443 100644 --- a/test/lightning/credentials/schema_test.exs +++ b/test/lightning/credentials/schema_test.exs @@ -110,7 +110,7 @@ defmodule Lightning.Credentials.SchemaTest do |> Ecto.Changeset.put_change(:baseUrl, "not a uri") |> Schema.validate(schema) - assert {:baseUrl, {"expected to be a URI", []}} in bad.errors + assert {:baseUrl, {"expected to be a URI", [code: :format]}} in bad.errors end test "creates a struct containing the schema, types and data", %{ @@ -383,7 +383,9 @@ defmodule Lightning.Credentials.SchemaTest do assert Enum.any?( errors, - &(&1 == {:host, {"expected to be a URI or an IPv4 address", []}}) + &(&1 == + {:host, + {"expected to be a URI or an IPv4 address", [code: :type]}}) ) end @@ -401,7 +403,7 @@ defmodule Lightning.Credentials.SchemaTest do assert Enum.any?( errors, - &(&1 == {:baseUrl, {"expected to be a URI", []}}) + &(&1 == {:baseUrl, {"expected to be a URI", [code: :format]}}) ) end @@ -431,7 +433,7 @@ defmodule Lightning.Credentials.SchemaTest do assert Enum.any?( errors, - &(&1 == {:email, {"expected to be an email", []}}) + &(&1 == {:email, {"expected to be an email", [code: :format]}}) ) end @@ -495,7 +497,7 @@ defmodule Lightning.Credentials.SchemaTest do validated = Schema.validate(changeset, schema) refute validated.valid? - assert {:settings, {"invalid JSON", []}} in validated.errors + assert {:settings, {"invalid JSON", [code: :type]}} in validated.errors end test "adds error when JSON array provided for object type field" do @@ -516,7 +518,7 @@ defmodule Lightning.Credentials.SchemaTest do validated = Schema.validate(changeset, schema) refute validated.valid? - assert {:data, {"invalid JSON", []}} in validated.errors + assert {:data, {"invalid JSON", [code: :type]}} in validated.errors end test "handles various JSON edge cases for object type validation" do @@ -532,28 +534,30 @@ defmodule Lightning.Credentials.SchemaTest do |> Ecto.Changeset.put_change(:config, "null") validated_null = Schema.validate(changeset_null, schema) - assert {:config, {"invalid JSON", []}} in validated_null.errors + assert {:config, {"invalid JSON", [code: :type]}} in validated_null.errors changeset_string = %Ecto.Changeset{data: %{}, types: %{config: :map}, valid?: true} |> Ecto.Changeset.put_change(:config, ~s("just a string")) validated_string = Schema.validate(changeset_string, schema) - assert {:config, {"invalid JSON", []}} in validated_string.errors + + assert {:config, {"invalid JSON", [code: :type]}} in validated_string.errors changeset_number = %Ecto.Changeset{data: %{}, types: %{config: :map}, valid?: true} |> Ecto.Changeset.put_change(:config, "123") validated_number = Schema.validate(changeset_number, schema) - assert {:config, {"invalid JSON", []}} in validated_number.errors + + assert {:config, {"invalid JSON", [code: :type]}} in validated_number.errors changeset_bool = %Ecto.Changeset{data: %{}, types: %{config: :map}, valid?: true} |> Ecto.Changeset.put_change(:config, "true") validated_bool = Schema.validate(changeset_bool, schema) - assert {:config, {"invalid JSON", []}} in validated_bool.errors + assert {:config, {"invalid JSON", [code: :type]}} in validated_bool.errors changeset_empty = %Ecto.Changeset{data: %{}, types: %{config: :map}, valid?: true} @@ -582,7 +586,7 @@ defmodule Lightning.Credentials.SchemaTest do validated = Schema.validate(changeset, schema) refute validated.valid? - assert {:config, {"must be an object", []}} in validated.errors + assert {:config, {"must be an object", [code: :type]}} in validated.errors end test "accepts valid JSON object string and continues without error" do @@ -628,7 +632,8 @@ defmodule Lightning.Credentials.SchemaTest do validated = Schema.validate(changeset_atom, schema) refute validated.valid? - assert {:settings, {"must be an object", []}} in validated.errors + + assert {:settings, {"must be an object", [code: :type]}} in validated.errors changeset_nil = %Ecto.Changeset{ data: %{settings: nil}, @@ -639,7 +644,8 @@ defmodule Lightning.Credentials.SchemaTest do validated_nil = Schema.validate(changeset_nil, schema) refute validated_nil.valid? - assert {:settings, {"can't be blank", []}} in validated_nil.errors + + assert {:settings, {"can't be blank", [code: :required]}} in validated_nil.errors changeset_map = %Ecto.Changeset{ diff --git a/test/lightning/credentials_test.exs b/test/lightning/credentials_test.exs index d995de4a664..93113884d97 100644 --- a/test/lightning/credentials_test.exs +++ b/test/lightning/credentials_test.exs @@ -147,6 +147,25 @@ defmodule Lightning.CredentialsTest do assert job.project_credential_id == nil end + test "delete_credential/2 answers not_found for a credential deleted since it was loaded" do + user = insert(:user) + credential = insert(:credential, user: user) + + assert {:ok, _deleted} = Credentials.delete_credential(credential, user) + + assert Credentials.delete_credential(credential, user) == + {:error, :not_found} + end + + test "delete_credential/2 refuses a credential runs have used, keeping it" do + user = insert(:user) + credential = insert(:credential, user: user) + insert(:step, credential: credential) + + assert Credentials.delete_credential(credential, user) == {:error, :in_use} + assert Credentials.get_credential(credential.id) + end + test "schedule_credential_deletion/1 schedules a deletion date according to the :purge_deleted_after_days env" do days = Lightning.Config.purge_deleted_after_days() @@ -343,6 +362,26 @@ defmodule Lightning.CredentialsTest do :ok end + test "creates under the id it is given, and refuses that id a second time" do + owner = insert(:user) + id = Ecto.UUID.generate() + + attrs = fn name -> + %{"name" => name, "schema" => "raw", "user_id" => owner.id} + end + + assert {:ok, %Credential{id: ^id}} = + Credentials.create_credential(attrs.("first"), owner, id: id) + + assert {:error, %Ecto.Changeset{errors: errors}} = + Credentials.create_credential(attrs.("second"), owner, id: id) + + assert {_, [constraint: :unique, constraint_name: "credentials_pkey"]} = + errors[:id] + + assert Repo.get!(Credential, id).name == "first" + end + test "fails if another cred exists with the same name for the same user" do user = insert(:user) @@ -379,7 +418,9 @@ defmodule Lightning.CredentialsTest do user_id: user.id, schema: "raw", project_credentials: [ - %{project_id: insert(:project).id} + %{ + project_id: insert(:project, project_users: [%{user_id: user.id}]).id + } ], credential_bodies: [ %{ @@ -440,7 +481,9 @@ defmodule Lightning.CredentialsTest do user_id: user.id, schema: "postgresql", project_credentials: [ - %{project_id: insert(:project).id} + %{ + project_id: insert(:project, project_users: [%{user_id: user.id}]).id + } ], # ✅ Use new credential_bodies structure credential_bodies: [ @@ -499,6 +542,153 @@ defmodule Lightning.CredentialsTest do end end + describe "writes by a service account" do + setup do + {account, _key} = + Lightning.ServiceAccountHelpers.service_account_with_key() + + owner = insert(:user) + + %{ + account: %{account | grants: MapSet.new(["credentials:write"])}, + owner: owner, + project: insert(:project, project_users: [%{user_id: owner.id}]) + } + end + + test "are refused without credentials:write, saying so", %{ + account: account, + owner: owner + } do + credential = insert(:credential, user: owner, name: "before") + account = %{account | grants: MapSet.new(["projects:write"])} + + assert Credentials.update_credential(credential, %{"name" => "x"}, account) == + {:error, {:insufficient_scope, "credentials:write"}} + end + + test "create a credential owned by the named user and record the account as the actor", + %{account: account, owner: owner, project: project} do + assert {:ok, credential} = + Credentials.create_credential( + %{ + "name" => "provisioned", + "schema" => "raw", + "user_id" => owner.id, + "credential_bodies" => [ + %{"name" => "main", "body" => %{"a" => "1"}} + ], + "project_credentials" => [%{"project_id" => project.id}] + }, + account + ) + + assert credential.user_id == owner.id + + assert [ + {"added_to_project", account.uuid, :service_account}, + {"created", account.uuid, :service_account} + ] == audit_actors(credential) + end + + test "update a credential and record the account as the actor", %{ + account: account, + owner: owner + } do + credential = + insert(:credential, user: owner, name: "before", schema: "raw") + + assert {:ok, _} = + Credentials.update_credential( + credential, + %{"name" => "after"}, + account + ) + + assert [{"updated", account.uuid, :service_account}] == + audit_actors(credential) + end + end + + describe "a credential scheduled for deletion" do + setup do + owner = insert(:user) + + credential = + insert(:credential, + user: owner, + name: "before", + schema: "raw", + scheduled_deletion: DateTime.utc_now() |> DateTime.truncate(:second) + ) + + %{owner: owner, credential: credential} + end + + test "cannot be updated by its owner", %{owner: owner, credential: c} do + assert Credentials.update_credential(c, %{"name" => "after"}, owner) == + {:error, :scheduled_for_deletion} + + assert Repo.reload!(c).name == "before" + end + + # A form opened before the credential was scheduled still holds this struct. + test "cannot be updated through a struct loaded before it was scheduled", + %{owner: owner, credential: c} do + stale = %{c | scheduled_deletion: nil} + + assert Credentials.update_credential(stale, %{"name" => "after"}, owner) == + {:error, :scheduled_for_deletion} + end + + test "tells a stranger only that they may not", %{credential: c} do + assert Credentials.update_credential(c, %{"name" => "x"}, insert(:user)) == + {:error, :unauthorized} + end + + test "cannot be scheduled again, which would move the date", %{ + owner: owner, + credential: c + } do + assert Credentials.schedule_credential_deletion(c, owner) == + {:error, :scheduled_for_deletion} + + assert Repo.reload!(c).scheduled_deletion == c.scheduled_deletion + end + + test "can still be deleted now, or have its deletion cancelled", %{ + owner: owner, + credential: c + } do + other = + insert(:credential, + user: owner, + scheduled_deletion: DateTime.utc_now() |> DateTime.truncate(:second) + ) + + assert {:ok, %Credential{scheduled_deletion: nil}} = + Credentials.cancel_scheduled_deletion(c.id, owner) + + assert {:ok, %{credential: _}} = + Credentials.delete_credential(other, owner) + + refute Repo.get(Credential, other.id) + end + end + + defp audit_query(credential) do + from(a in Audit.base_query(), where: a.item_id == ^credential.id) + end + + defp audit_actors(credential) do + from(a in Audit.base_query(), + where: a.item_id == ^credential.id, + order_by: a.event, + select: {a.event, a.actor_id, a.actor_type} + ) + |> Repo.all() + end + describe "update_credential/2" do setup :isolated_adaptors @@ -507,6 +697,91 @@ defmodule Lightning.CredentialsTest do :ok end + test "a credential another request deleted is a concurrent edit, not a raise" do + user = insert(:user) + credential = insert(:credential, schema: "raw", user: user) + Repo.delete!(credential) + + assert Credentials.update_credential(credential, %{name: "renamed"}, user) == + {:error, :concurrent_edit} + end + + test "holds a credential switched to oauth to an OAuth token body" do + user = insert(:user) + + credential = + insert(:credential, schema: "raw", user: user) + |> with_body(%{name: "main", body: %{"foo" => 1}}) + + assert {:error, + %Ecto.Changeset{ + errors: [ + credential_bodies: + {_message, + [ + code: :required, + key: [:credential_bodies, 0, :body, "access_token"] + ]} + ] + }} = + Credentials.update_credential( + credential, + %{ + "schema" => "oauth", + "credential_bodies" => [ + %{"name" => "main", "body" => %{"foo" => 2}} + ] + }, + user + ) + + assert Repo.reload!(credential).schema == "raw" + end + + test "refuses a typed body at each key its schema refuses" do + user = insert(:user) + + assert {:error, %Ecto.Changeset{errors: errors}} = + Credentials.create_credential( + %{ + "name" => "typed", + "schema" => "postgresql", + "user_id" => user.id, + "credential_bodies" => [ + %{ + "name" => "main", + "body" => %{ + "host" => "https://a.example", + "port" => 5432, + "database" => "acme" + } + }, + %{"name" => "staging", "body" => %{"host" => "b.example"}} + ] + }, + user + ) + + assert Enum.sort(errors) == + Enum.sort( + credential_bodies: + {"can't be blank", + [ + key: [:credential_bodies, 1, :body, "port"], + code: :required + ]}, + credential_bodies: + {"can't be blank", + [ + key: [:credential_bodies, 1, :body, "database"], + code: :required + ]}, + credential_bodies: + {"expected to be a URI or an IPv4 address", + [key: [:credential_bodies, 1, :body, "host"], code: :type]} + ) + end + test "updates an OAuth credential with new scopes" do user = insert(:user) oauth_client = insert(:oauth_client) @@ -588,7 +863,7 @@ defmodule Lightning.CredentialsTest do Enum.at(credential.project_credentials, 0) |> Map.from_struct() - new_project = insert(:project) + new_project = insert(:project, project_users: [%{user_id: user.id}]) update_attrs = %{ body: %{}, @@ -863,6 +1138,313 @@ defmodule Lightning.CredentialsTest do assert persisted.user_id == owner.id assert is_nil(persisted.transfer_status) end + + test "a write that changes only a body records one updated event holding only that body" do + owner = insert(:user) + + credential = + insert(:credential, user: owner, name: "unchanged", schema: "raw") + |> with_body(%{name: "main", body: %{"a" => "old-secret"}}) + |> with_body(%{name: "staging", body: %{"b" => "staging-secret"}}) + + assert {:ok, _} = + Credentials.update_credential( + credential, + %{ + "name" => "unchanged", + "credential_bodies" => [ + %{"name" => "main", "body" => %{"a" => "new-secret"}}, + %{"name" => "staging", "body" => %{"b" => "staging-secret"}} + ] + }, + owner + ) + + assert [audit] = Repo.all(audit_query(credential)) + assert audit.event == "updated" + assert audit.actor_id == owner.id + assert audit.metadata["environments"] == ["main"] + assert Map.keys(audit.metadata["credential_bodies"]) == ["body:main"] + + assert {:ok, %{"a" => "new-secret"}} = + audit.metadata["credential_bodies"]["body:main"] + |> Base.decode64!() + |> Lightning.Encrypted.Map.load() + + refute inspect(Map.from_struct(audit), limit: :infinity) =~ "secret" + end + + test "deleting only an environment records one updated event naming it" do + owner = insert(:user) + + credential = + insert(:credential, user: owner, name: "unchanged", schema: "raw") + |> with_body(%{name: "main", body: %{"a" => "main-secret"}}) + |> with_body(%{name: "staging", body: %{"b" => "staging-secret"}}) + + assert {:ok, _} = + Credentials.update_credential( + credential, + %{ + "name" => "unchanged", + "delete_environments" => ["staging", "never-existed"] + }, + owner + ) + + assert [audit] = Repo.all(audit_query(credential)) + assert audit.event == "updated" + assert audit.metadata == %{"deleted_environments" => ["staging"]} + end + + test "an edit that writes one environment and deletes another records both" do + owner = insert(:user) + + credential = + insert(:credential, user: owner, name: "unchanged", schema: "raw") + |> with_body(%{name: "main", body: %{"a" => "old-secret"}}) + |> with_body(%{name: "staging", body: %{"b" => "staging-secret"}}) + + assert {:ok, _} = + Credentials.update_credential( + credential, + %{ + "name" => "unchanged", + "credential_bodies" => [ + %{"name" => "main", "body" => %{"a" => "new-secret"}} + ], + "delete_environments" => ["staging"] + }, + owner + ) + + assert [audit] = Repo.all(audit_query(credential)) + + assert %{ + "environments" => ["main"], + "deleted_environments" => ["staging"], + "credential_bodies" => %{"body:main" => _} + } = audit.metadata + end + + test "a typed body resent as a form sends it records nothing" do + owner = insert(:user) + + stored = %{ + "user" => "user1", + "password" => "pass1", + "host" => "https://dbhost", + "database" => "test_db", + "port" => 5000, + "ssl" => true, + "allowSelfSignedCert" => false + } + + credential = + insert(:credential, user: owner, name: "pg", schema: "postgresql") + |> with_body(%{name: "main", body: stored}) + + assert {:ok, _} = + Credentials.update_credential( + credential, + %{ + "name" => "pg", + "credential_bodies" => [ + %{ + "name" => "main", + "body" => + Map.merge(stored, %{ + "port" => "5000", + "ssl" => "true", + "allowSelfSignedCert" => "false" + }) + } + ] + }, + owner + ) + + assert Repo.all(audit_query(credential)) == [] + + assert Credentials.get_credential_body(credential.id, "main").body == + stored + end + + test "re-authorising an OAuth credential records the new tokens, an unchanged save nothing" do + owner = insert(:user) + + credential = + insert(:credential, + user: owner, + name: "oauth", + schema: "oauth", + oauth_client: insert(:oauth_client) + ) + |> with_body(%{ + name: "main", + body: %{ + "access_token" => "old_token", + "refresh_token" => "existing_refresh", + "expires_in" => 3600, + "scope" => "read write", + "token_type" => "Bearer" + } + }) + + # The form never sends the refresh token back, so the stored one is kept. + save = fn access_token -> + Credentials.update_credential( + credential, + %{ + "name" => "oauth", + "credential_bodies" => [ + %{ + "name" => "main", + "body" => %{ + "access_token" => access_token, + "expires_in" => 3600, + "scope" => "read write", + "token_type" => "Bearer" + } + } + ] + }, + owner + ) + end + + assert {:ok, _} = save.("old_token") + assert Repo.all(audit_query(credential)) == [] + + assert {:ok, _} = save.("new_token") + + assert [%{event: "updated", metadata: %{"environments" => ["main"]}}] = + Repo.all(audit_query(credential)) + end + + test "existing links passed back with their ids are kept beside a new one" do + owner = insert(:user) + + [linked, added] = + insert_list(2, :project, project_users: [%{user_id: owner.id}]) + + credential = + insert(:credential, + user: owner, + schema: "raw", + project_credentials: [%{project_id: linked.id}] + ) + + [existing] = credential.project_credentials + + assert {:ok, updated} = + Credentials.update_credential( + credential, + %{ + "project_credentials" => [ + %{"id" => existing.id, "project_id" => linked.id}, + %{"project_id" => added.id} + ] + }, + owner + ) + + assert updated.project_credentials + |> Enum.map(& &1.project_id) + |> Enum.sort() == + Enum.sort([linked.id, added.id]) + + assert existing.id in Enum.map(updated.project_credentials, & &1.id) + + assert ["added_to_project", "updated"] == + credential + |> audit_query() + |> Repo.all() + |> Enum.map(& &1.event) + |> Enum.sort() + end + + test "refuses a new link to a project the owner has no access to" do + owner = insert(:user) + elsewhere = insert(:project) + credential = insert(:credential, user: owner, schema: "raw") + + assert {:error, changeset} = + Credentials.update_credential( + credential, + %{"project_credentials" => [%{"project_id" => elsewhere.id}]}, + owner + ) + + assert errors_on(changeset).project_credentials == [ + "the owner has no access to these projects: #{elsewhere.id}" + ] + + assert Repo.preload(credential, :project_credentials, force: true).project_credentials == + [] + end + + test "keeps editable a credential already linked to a project the owner has no access to" do + owner = insert(:user) + elsewhere = insert(:project) + + credential = + insert(:credential, + user: owner, + schema: "raw", + project_credentials: [%{project_id: elsewhere.id}] + ) + + [existing] = credential.project_credentials + + assert {:ok, %Credential{name: "Renamed"}} = + Credentials.update_credential( + credential, + %{ + "name" => "Renamed", + "project_credentials" => [ + %{"id" => existing.id, "project_id" => elsewhere.id} + ] + }, + owner + ) + end + end + + describe "preload_links/1" do + test "loads the credential's project links and not the projects" do + project = insert(:project) + + credential = + insert(:credential, project_credentials: [%{project: project}]) + + loaded = + credential.id + |> Credentials.get_credential() + |> Credentials.preload_links() + + assert [%{project_id: project_id}] = loaded.project_credentials + assert project_id == project.id + refute Ecto.assoc_loaded?(loaded.projects) + end + end + + describe "preload_links_and_projects/1" do + test "loads the credential's project links and the projects they name" do + project = insert(:project, name: "linked") + + credential = + insert(:credential, project_credentials: [%{project: project}]) + + loaded = + credential.id + |> Credentials.get_credential() + |> Credentials.preload_links_and_projects() + + assert [%{project_id: project_id}] = loaded.project_credentials + assert project_id == project.id + assert [%{name: "linked"}] = loaded.projects + end end describe "has_activity_in_projects?/1" do @@ -1517,7 +2099,9 @@ defmodule Lightning.CredentialsTest do result = Credentials.validate_credential_transfer(changeset, sender, credential) - assert {:email, {"You cannot transfer a credential to yourself", []}} in result.errors + assert {:email, + {"You cannot transfer a credential to yourself", + [code: :not_allowed]}} in result.errors end test "adds error when recipient does not exist" do @@ -1530,7 +2114,7 @@ defmodule Lightning.CredentialsTest do result = Credentials.validate_credential_transfer(changeset, sender, credential) - assert {:email, {"User does not exist", []}} in result.errors + assert {:email, {"User does not exist", [code: :not_found]}} in result.errors end test "adds error when recipient lacks access to required projects" do @@ -1549,7 +2133,8 @@ defmodule Lightning.CredentialsTest do Credentials.validate_credential_transfer(changeset, sender, credential) assert {:email, - {"User doesn't have access to these projects: Secret Project", []}} in result.errors + {"User doesn't have access to these projects: Secret Project", + [code: :no_access]}} in result.errors end end @@ -1588,9 +2173,15 @@ defmodule Lightning.CredentialsTest do } assert {:error, - %Lightning.Credentials.OauthValidation.Error{ - type: :missing_scope, - message: "Missing required OAuth field: scope or scopes" + %Ecto.Changeset{ + errors: [ + credential_bodies: + {"Missing required OAuth field: scope or scopes", + [ + code: :required, + key: [:credential_bodies, 0, :body, "scope"] + ]} + ] }} = Credentials.create_credential(attrs, user) end @@ -1634,9 +2225,15 @@ defmodule Lightning.CredentialsTest do } assert {:error, - %Lightning.Credentials.OauthValidation.Error{ - type: :missing_scope, - message: "Missing required OAuth field: scope or scopes" + %Ecto.Changeset{ + errors: [ + credential_bodies: + {"Missing required OAuth field: scope or scopes", + [ + code: :required, + key: [:credential_bodies, 0, :body, "scope"] + ]} + ] }} = Credentials.update_credential(credential, update_attrs, user) end end @@ -1670,10 +2267,14 @@ defmodule Lightning.CredentialsTest do assert {:error, changeset} = Credentials.create_credential(attrs, user) - # Error should be on :credential_bodies with environment info - assert [credential_bodies: {msg, _}] = changeset.errors - assert msg =~ "Environment 1" - assert msg =~ "must be a short slug" + assert [ + credential_bodies: + {"must be a short slug", + [ + key: [:credential_bodies, 0, :name], + validation: :format + ]} + ] = changeset.errors end test "returns credential changeset error when second credential body fails" do @@ -1691,10 +2292,14 @@ defmodule Lightning.CredentialsTest do assert {:error, changeset} = Credentials.create_credential(attrs, user) - # Error should reference Environment 2 (second body at index 1) - assert [credential_bodies: {msg, _}] = changeset.errors - assert msg =~ "Environment 2" - assert msg =~ "must be a short slug" + assert [ + credential_bodies: + {"must be a short slug", + [ + key: [:credential_bodies, 1, :name], + validation: :format + ]} + ] = changeset.errors end end @@ -1792,9 +2397,15 @@ defmodule Lightning.CredentialsTest do } assert {:error, - %Lightning.Credentials.OauthValidation.Error{ - type: :missing_access_token, - message: "Missing required OAuth field: access_token" + %Ecto.Changeset{ + errors: [ + credential_bodies: + {"Missing required OAuth field: access_token", + [ + code: :required, + key: [:credential_bodies, 0, :body, "access_token"] + ]} + ] }} = Credentials.create_credential(attrs, user) # ✅ Test missing refresh_token @@ -1816,9 +2427,15 @@ defmodule Lightning.CredentialsTest do } assert {:error, - %Lightning.Credentials.OauthValidation.Error{ - type: :missing_refresh_token, - message: "Missing required OAuth field: refresh_token" + %Ecto.Changeset{ + errors: [ + credential_bodies: + {"Missing required OAuth field: refresh_token", + [ + code: :required, + key: [:credential_bodies, 0, :body, "refresh_token"] + ]} + ] }} = Credentials.create_credential(attrs, user) # ✅ Test missing expiration fields @@ -1841,10 +2458,16 @@ defmodule Lightning.CredentialsTest do } assert {:error, - %Lightning.Credentials.OauthValidation.Error{ - type: :missing_expiration, - message: - "Missing expiration field: either expires_in or expires_at is required" + %Ecto.Changeset{ + errors: [ + credential_bodies: { + "Missing expiration field: either expires_in or expires_at is required", + [ + code: :required, + key: [:credential_bodies, 0, :body, "expires_in"] + ] + } + ] }} = Credentials.create_credential(attrs, user) # ✅ Test valid token data @@ -1895,13 +2518,16 @@ defmodule Lightning.CredentialsTest do } assert {:error, - %Lightning.Credentials.OauthValidation.Error{ - type: :missing_scopes, - message: message, - details: %{missing_scopes: ["write"]} + %Ecto.Changeset{ + errors: [ + credential_bodies: + {"Missing required scopes. Please reauthorize and grant all selected permissions.", + [ + code: :scope_not_granted, + key: [:credential_bodies, 0, :body, "scope"] + ]} + ] }} = Credentials.create_credential(attrs, user) - - assert message =~ "Missing required scopes: write" end end @@ -2092,9 +2718,15 @@ defmodule Lightning.CredentialsTest do } assert {:error, - %Lightning.Credentials.OauthValidation.Error{ - type: :unsupported_token_type, - message: "Unsupported token type: 'Basic'. Expected 'Bearer'" + %Ecto.Changeset{ + errors: [ + credential_bodies: + {"Unsupported token type. Expected 'Bearer'", + [ + code: :not_allowed, + key: [:credential_bodies, 0, :body, "token_type"] + ]} + ] }} = Credentials.create_credential(attrs, user) end @@ -2299,8 +2931,8 @@ defmodule Lightning.CredentialsTest do test "same external_id in different projects is allowed" do user1 = insert(:user) user2 = insert(:user) - project1 = insert(:project) - project2 = insert(:project) + project1 = insert(:project, project_users: [%{user_id: user1.id}]) + project2 = insert(:project, project_users: [%{user_id: user2.id}]) attrs1 = %{ name: "cred one", @@ -2327,7 +2959,11 @@ defmodule Lightning.CredentialsTest do test "cross-user duplicate external_id in the same project is rejected" do user1 = insert(:user) user2 = insert(:user) - project = insert(:project) + + project = + insert(:project, + project_users: [%{user_id: user1.id}, %{user_id: user2.id}] + ) attrs1 = %{ name: "cred one", @@ -2359,8 +2995,8 @@ defmodule Lightning.CredentialsTest do test "removing a project association resolves the conflict in that project" do user = insert(:user) - project1 = insert(:project) - project2 = insert(:project) + project1 = insert(:project, project_users: [%{user_id: user.id}]) + project2 = insert(:project, project_users: [%{user_id: user.id}]) # Create our credential in both projects (no conflict yet) {:ok, credential} = @@ -2415,6 +3051,7 @@ defmodule Lightning.CredentialsTest do # Now another user can use the same external_id in project1 other_user = insert(:user) + insert(:project_user, project: project1, user: other_user) {:ok, _other} = Credentials.create_credential( @@ -2433,8 +3070,13 @@ defmodule Lightning.CredentialsTest do test "removing credential from a conflicting project allows the update" do user_a = insert(:user) user_b = insert(:user) - shared_project = insert(:project) - other_project = insert(:project) + + shared_project = + insert(:project, + project_users: [%{user_id: user_a.id}, %{user_id: user_b.id}] + ) + + other_project = insert(:project, project_users: [%{user_id: user_b.id}]) # User A has a credential with external_id "conflict-id" in shared_project {:ok, _cred_a} = @@ -2530,6 +3172,8 @@ defmodule Lightning.CredentialsTest do env: "staging" }) + insert(:project_user, project: sandbox, user: user2) + # User A adds credential with external_id directly to sandbox {:ok, _sandbox_cred} = Credentials.create_credential( @@ -2566,7 +3210,7 @@ defmodule Lightning.CredentialsTest do test "updating a credential keeps its own external_id without error" do user = insert(:user) - project = insert(:project) + project = insert(:project, project_users: [%{user_id: user.id}]) {:ok, credential} = Credentials.create_credential( @@ -2927,8 +3571,8 @@ defmodule Lightning.CredentialsTest do assert [ credential_bodies: - {"Environment 1: body adaptor catalogue is not ready yet, try again shortly", - []} + {"adaptor catalogue is not ready yet, try again shortly", + [key: [:credential_bodies, 0, :body], code: :not_ready]} ] = changeset.errors end end diff --git a/test/lightning/invocation/dataclip_test.exs b/test/lightning/invocation/dataclip_test.exs index c7d26aed623..fa943e59446 100644 --- a/test/lightning/invocation/dataclip_test.exs +++ b/test/lightning/invocation/dataclip_test.exs @@ -57,7 +57,8 @@ defmodule Lightning.Invocation.DataclipTest do |> Dataclip.new() refute changeset.valid? - assert {:request, {"cannot be set for this type", []}} in changeset.errors + + assert {:request, {"cannot be set for this type", [code: :not_allowed]}} in changeset.errors end end end diff --git a/test/lightning/invocation/log_line_test.exs b/test/lightning/invocation/log_line_test.exs index 14a63b6b722..1123a5bc579 100644 --- a/test/lightning/invocation/log_line_test.exs +++ b/test/lightning/invocation/log_line_test.exs @@ -57,7 +57,7 @@ defmodule Lightning.Invocation.LogLineTest do nil ) - assert {:message, {"can't be blank", []}} in log_line.errors + assert {:message, {"can't be blank", [code: :required]}} in log_line.errors refute log_line.valid? end end diff --git a/test/lightning/invocation_test.exs b/test/lightning/invocation_test.exs index 856b6bc3f1c..e6eaa2ad1ea 100644 --- a/test/lightning/invocation_test.exs +++ b/test/lightning/invocation_test.exs @@ -2946,6 +2946,62 @@ defmodule Lightning.InvocationTest do end end + describe "paginate_log_lines_for/3" do + test "pages the log lines of the user's projects, narrowed by the filters" do + user = insert(:user) + project = insert(:project, project_users: [%{user_id: user.id}]) + + %{triggers: [trigger]} = + workflow = insert(:simple_workflow, project: project) + + %{runs: [run]} = + insert(:workorder, + workflow: workflow, + trigger: trigger, + dataclip: build(:dataclip), + runs: [ + %{ + state: :started, + dataclip: build(:dataclip), + starting_trigger: trigger + } + ] + ) + + info = insert(:log_line, run: run, level: :info) + error = insert(:log_line, run: run, level: :error) + + insert(:log_line, + run: + insert(:run, + work_order: insert(:workorder, dataclip: build(:dataclip)), + dataclip: build(:dataclip), + starting_trigger: build(:trigger) + ), + level: :error + ) + + assert %Scrivener.Page{entries: [%{id: id}], total_entries: 1} = + Invocation.paginate_log_lines_for(user, %{level: "error"}, []) + + assert id == error.id + + page = Invocation.paginate_log_lines_for(user, %{}, page_size: 1) + + assert %Scrivener.Page{ + page_number: 1, + page_size: 1, + total_entries: 2, + total_pages: 2 + } = page + + assert MapSet.new( + Invocation.paginate_log_lines_for(user, %{}, []).entries, + & &1.id + ) == MapSet.new([info.id, error.id]) + end + end + describe "scrubbed_io_for_run/2" do setup do project = insert(:project) diff --git a/test/lightning/jobs_test.exs b/test/lightning/jobs_test.exs index 526e6626caf..5798825ad94 100644 --- a/test/lightning/jobs_test.exs +++ b/test/lightning/jobs_test.exs @@ -192,6 +192,34 @@ defmodule Lightning.JobsTest do end end + describe "paginate_jobs_for/2" do + test "pages a project's jobs, or those of every project a user may access" do + user = insert(:user) + project = insert(:project, project_users: [%{user_id: user.id}]) + other_project = insert(:project, project_users: [%{user_id: user.id}]) + %{jobs: [job]} = insert(:simple_workflow, project: project) + %{jobs: [other_job]} = insert(:simple_workflow, project: other_project) + insert(:simple_workflow) + + assert %Scrivener.Page{entries: [%{id: job_id}], total_entries: 1} = + Jobs.paginate_jobs_for(project, []) + + assert job_id == job.id + + page = Jobs.paginate_jobs_for(user, page_size: 1) + + assert %Scrivener.Page{ + page_number: 1, + page_size: 1, + total_entries: 2, + total_pages: 2 + } = page + + assert MapSet.new(Jobs.paginate_jobs_for(user, []).entries, & &1.id) == + MapSet.new([job.id, other_job.id]) + end + end + describe "create_job/2" do setup :isolated_adaptors diff --git a/test/lightning/kickstart_test.exs b/test/lightning/kickstart_test.exs index cc9254b457b..ac7b426e959 100644 --- a/test/lightning/kickstart_test.exs +++ b/test/lightning/kickstart_test.exs @@ -235,9 +235,8 @@ defmodule Lightning.KickstartTest do scenario(), ["projects", Access.at(0), "members"], fn [owner_member, editor_member] -> - # The incoming owner is declared FIRST on purpose. Declared in the - # other order the handover succeeds even without reconcile_members/2 - # sorting owners last, so this order is what pins the sort. + # Project.project_with_users_changeset/2 writes the owner row last, + # so the handover works whichever order the members come in. [ Map.put(editor_member, "role", "owner"), Map.put(owner_member, "role", "editor") diff --git a/test/lightning/policies/collections_test.exs b/test/lightning/policies/collections_test.exs index 322858b1c67..40617e9312e 100644 --- a/test/lightning/policies/collections_test.exs +++ b/test/lightning/policies/collections_test.exs @@ -1,15 +1,7 @@ defmodule Lightning.Policies.CollectionsTest do @moduledoc """ - Collections access is gated on the caller's project role. - - Read (`:access_collection`) routes through `:project_users, :access_project`, - which loads the project and therefore sees `scheduled_deletion`. The write - actions do not: `Policies.Collections.has_project_role?/3` builds a - `%Project{id: project_id}` and asks `Projects.get_project_user_role/2` about - it, so whether the project is still operable is never consulted. - - The result is that writes are *more* permissive than reads on a project that - has been shut down. + Collections access is decided by the caller's project role, and every action + is refused on a project scheduled for deletion, reads and writes alike. """ use Lightning.DataCase, async: true @@ -65,9 +57,6 @@ defmodule Lightning.Policies.CollectionsTest do # below is about the project's lifecycle and not about the role. assert_can(:put_collection_item, editor, collection) - # Control: read access is ALREADY refused, because `:access_collection` - # delegates to `:access_project`, which loads the project. The writes - # below are therefore more permissive than the read. refute_can(:access_collection, editor, scheduled_collection) assert allowed_writes(editor, scheduled_collection) == [] @@ -95,6 +84,146 @@ defmodule Lightning.Policies.CollectionsTest do end end + describe ":manage_collection on a project" do + test "is held by its owner and admins, not its editors or viewers" do + [owner, admin, editor, viewer, outsider] = insert_list(5, :user) + + project = + insert(:project, + project_users: [ + %{user_id: owner.id, role: :owner}, + %{user_id: admin.id, role: :admin}, + %{user_id: editor.id, role: :editor}, + %{user_id: viewer.id, role: :viewer} + ] + ) + + assert_can(:manage_collection, owner, project) + assert_can(:manage_collection, admin, project) + refute_can(:manage_collection, editor, project) + refute_can(:manage_collection, viewer, project) + refute_can(:manage_collection, outsider, project) + end + + test "is held by a service account on any project" do + {account, _key} = + Lightning.ServiceAccountHelpers.service_account_with_key() + + account = %{account | grants: MapSet.new(["collections:write"])} + + assert Permissions.can?( + Collections, + :manage_collection, + account, + insert(:project) + ) + end + end + + describe "an operator holding collections:write" do + setup do + {account, _key} = + Lightning.ServiceAccountHelpers.service_account_with_key() + + project = insert(:project) + + %{ + account: %{account | grants: MapSet.new(["collections:write"])}, + project: project, + collection: insert(:collection, project: project) + } + end + + test "may manage a project or a collection, and put items", %{ + account: account, + project: project, + collection: collection + } do + assert Collections.authorize(:manage_collection, account, project) == true + + assert Collections.authorize(:manage_collection, account, collection) == + true + + assert Collections.authorize(:put_collection_item, account, collection) == + true + end + + test "may read items, needing no grant but its own", %{ + account: account, + collection: collection + } do + assert Collections.authorize(:access_collection, account, collection) == + true + end + + test "without the grant, is refused as insufficient_scope", %{ + account: account, + project: project + } do + account = %{account | grants: MapSet.new(["projects:write"])} + {:ok, scope} = Lightning.Scope.fetch(account, project) + + assert Collections.authorize(:manage_collection, scope, project) == + {:error, {:insufficient_scope, "collections:write"}} + end + end + + describe "asked with a scope, on a project scheduled for deletion" do + setup %{scheduled_project: scheduled_project} do + owner = insert(:user) + + insert(:project_user, + project: scheduled_project, + user: owner, + role: :owner + ) + + %{owner: owner} + end + + test "an owner is told, a raw owner or non-member is not", %{ + owner: owner, + scheduled_project: scheduled_project + } do + {:ok, scope} = Lightning.Scope.fetch(owner, scheduled_project) + + assert Collections.authorize(:manage_collection, scope, scheduled_project) == + {:error, :scheduled_for_deletion} + + assert Collections.authorize(:manage_collection, owner, scheduled_project) == + false + + scheduled_collection = insert(:collection, project: scheduled_project) + + assert Collections.authorize( + :access_collection, + scope, + scheduled_collection + ) == {:error, :scheduled_for_deletion} + + {:ok, outsider_scope} = + Lightning.Scope.fetch(insert(:user), scheduled_project) + + assert Collections.authorize( + :manage_collection, + outsider_scope, + scheduled_project + ) == {:error, :no_standing} + end + end + + test "a scope for one project is refused another project's subject" do + owner = insert(:user) + + project = + insert(:project, project_users: [%{user_id: owner.id, role: :owner}]) + + other = insert(:project) + {:ok, scope} = Lightning.Scope.fetch(owner, project) + + assert Collections.authorize(:manage_collection, scope, other) == false + end + defp allowed_writes(user, collection) do for action <- @write_actions, Permissions.can?(Collections, action, user, collection), diff --git a/test/lightning/policies/credentials_test.exs b/test/lightning/policies/credentials_test.exs index 8c7dc0b11e1..211aa1565ad 100644 --- a/test/lightning/policies/credentials_test.exs +++ b/test/lightning/policies/credentials_test.exs @@ -579,4 +579,231 @@ defmodule Lightning.Policies.CredentialsTest do end end end + + describe "Credential ownership" do + setup do + owner = insert(:user) + + %{ + owner: owner, + credential: insert(:credential, user: owner), + superuser: insert(:user, role: :superuser), + other_user: insert(:user) + } + end + + test "the owner can edit and delete their credential", %{ + owner: owner, + credential: credential + } do + assert Credentials + |> Bodyguard.permit?(:edit_credential, owner, credential) + + assert Credentials + |> Bodyguard.permit?(:delete_credential, owner, credential) + end + + test "anyone else, a superuser included, can do neither", %{ + credential: credential, + superuser: superuser, + other_user: other_user + } do + for user <- [superuser, other_user], + action <- [:edit_credential, :delete_credential] do + refute Credentials |> Bodyguard.permit?(action, user, credential), + "#{action} was granted to someone who does not own the credential" + end + end + + test "a role in a project the credential is linked to grants neither", %{ + owner: owner + } do + [admin, editor, viewer] = insert_list(3, :user) + + project = + insert(:project, + project_users: [ + %{user: owner, role: :owner}, + %{user: admin, role: :admin}, + %{user: editor, role: :editor}, + %{user: viewer, role: :viewer} + ] + ) + + credential = insert(:credential, user: owner) + insert(:project_credential, project: project, credential: credential) + + for user <- [admin, editor, viewer], + action <- [:edit_credential, :delete_credential] do + refute Credentials |> Bodyguard.permit?(action, user, credential), + "#{action} was granted through a project role" + end + end + + test "linking to a project takes an owner, admin or editor there, or a service account" do + [owner, admin, editor, viewer, outsider] = insert_list(5, :user) + + project = + insert(:project, + project_users: [ + %{user: owner, role: :owner}, + %{user: admin, role: :admin}, + %{user: editor, role: :editor}, + %{user: viewer, role: :viewer} + ] + ) + + {account, _key} = + Lightning.ServiceAccountHelpers.service_account_with_key() + + account = %{account | grants: MapSet.new(["credentials:write"])} + + may_link? = + &Bodyguard.permit?( + Lightning.Policies.ProjectUsers, + :create_project_credential, + &1, + project + ) + + assert Enum.map( + [owner, admin, editor, viewer, outsider, account], + may_link? + ) == + [true, true, true, false, false, true] + end + + test "a service account without credentials:write cannot link to a project" do + {account, _key} = + Lightning.ServiceAccountHelpers.service_account_with_key() + + account = %{account | grants: MapSet.new(["projects:write"])} + + assert Lightning.Policies.ProjectUsers.authorize( + :create_project_credential, + account, + insert(:project) + ) == {:error, {:insufficient_scope, "credentials:write"}} + end + + test "a service account can write any credential but never delete one", %{ + credential: credential + } do + {account, _key} = + Lightning.ServiceAccountHelpers.service_account_with_key() + + account = %{account | grants: MapSet.new(["credentials:write"])} + + assert Credentials + |> Bodyguard.permit?(:edit_credential, account, credential) + + refute Credentials + |> Bodyguard.permit?(:delete_credential, account, credential) + end + + test "only an operator may name another owner, and only with the grant" do + {account, _key} = + Lightning.ServiceAccountHelpers.service_account_with_key() + + operator = %{account | grants: MapSet.new(["credentials:write"])} + bare = %{account | grants: MapSet.new(["projects:write"])} + + assert Credentials.authorize(:name_credential_owner, operator, nil) == true + + assert Credentials.authorize(:name_credential_owner, insert(:user), nil) == + false + + assert Credentials.authorize(:name_credential_owner, bare, nil) == + {:error, {:insufficient_scope, "credentials:write"}} + + assert Credentials.authorize(:list_credentials, insert(:user), nil) == true + + assert Credentials.authorize(:edit_credential, bare, insert(:credential)) == + {:error, {:insufficient_scope, "credentials:write"}} + end + + test "only a person may list their credentials or create one for themselves" do + {account, _key} = + Lightning.ServiceAccountHelpers.service_account_with_key() + + operator = %{account | grants: MapSet.new(["credentials:write"])} + person = insert(:user) + + assert Credentials.authorize(:list_credentials, person, nil) == true + + assert Credentials.authorize( + :edit_credential, + person, + %Lightning.Credentials.Credential{user_id: person.id} + ) == true + + assert Credentials.authorize(:list_credentials, operator, nil) == false + + assert Credentials.authorize( + :edit_credential, + operator, + %Lightning.Credentials.Credential{user_id: operator.id} + ) == false + + # An operator can't own a credential, but can create one for a person. + assert Credentials.authorize( + :edit_credential, + operator, + %Lightning.Credentials.Credential{user_id: person.id} + ) == true + end + end + + describe "a credential scheduled for deletion" do + setup do + owner = insert(:user) + + {account, _key} = + Lightning.ServiceAccountHelpers.service_account_with_key() + + credential = + insert(:credential, + user: owner, + scheduled_deletion: DateTime.utc_now() |> DateTime.truncate(:second) + ) + + %{ + owner: owner, + operator: %{account | grants: MapSet.new(["credentials:write"])}, + bare: %{account | grants: MapSet.new(["projects:write"])}, + credential: Repo.preload(credential, :project_credentials) + } + end + + test "refuses editing with scheduled_for_deletion to a caller who may edit it", + %{owner: owner, operator: operator, credential: credential} do + for actor <- [owner, operator] do + assert Credentials.authorize(:edit_credential, actor, credential) == + {:error, :scheduled_for_deletion} + end + end + + test "tells anyone else only what it would tell them before the schedule", + %{bare: bare, credential: credential} do + assert Credentials.authorize(:edit_credential, insert(:user), credential) == + false + + assert Credentials.authorize(:edit_credential, bare, credential) == + {:error, {:insufficient_scope, "credentials:write"}} + end + + test "still lets the owner see it and delete it", %{ + owner: owner, + credential: credential + } do + assert Credentials.authorize( + :see_credential, + Lightning.Scope.new(owner), + credential + ) == true + + assert Credentials.authorize(:delete_credential, owner, credential) == + true + end + end end diff --git a/test/lightning/policies/decision_test.exs b/test/lightning/policies/decision_test.exs new file mode 100644 index 00000000000..44a54df3112 --- /dev/null +++ b/test/lightning/policies/decision_test.exs @@ -0,0 +1,168 @@ +defmodule Lightning.Policies.DecisionTest do + @moduledoc """ + Only a caller the policy would otherwise allow learns that a project is + scheduled for deletion, and a caller with no standing hears that apart from + a missing role or MFA, in every policy that decides through + `Lightning.Policies.Decision`. + """ + use Lightning.DataCase, async: true + + alias Lightning.Policies.Collections + alias Lightning.Policies.Credentials + alias Lightning.Policies.ProjectUsers + alias Lightning.Policies.Provisioning + alias Lightning.Policies.Users + alias Lightning.Scope + alias Lightning.ServiceAccount.AccessToken + + @project_actions [ + {ProjectUsers, ProjectUsers.actions()}, + {Collections, + [ + :access_collection, + :put_collection_item, + :delete_collection_item, + :delete_all_collection_items, + :manage_collection + ]}, + {Credentials, + [ + :create_keychain_credential, + :edit_keychain_credential, + :delete_keychain_credential, + :view_keychain_credential + ]} + ] + + @instance_actions [ + {Credentials, + [ + :edit_credential, + :delete_credential, + :name_credential_owner, + :list_credentials + ]}, + {Users, [:read_users, :write_users]} + ] + + setup do + project = + insert(:project, + scheduled_deletion: DateTime.add(DateTime.utc_now(), 7, :day) + ) + + {account, _key} = Lightning.ServiceAccountHelpers.service_account_with_key() + + {:ok, operator} = + Scope.fetch(%{account | grants: MapSet.new(AccessToken.scopes())}, project) + + {:ok, grantless} = Scope.fetch(%{account | grants: MapSet.new()}, project) + {:ok, stranger} = Scope.fetch(insert(:user), project) + + %{ + project: project, + operator: operator, + grantless: grantless, + stranger: stranger + } + end + + test "an allowed caller is told the project is scheduled for deletion", %{ + project: project, + operator: operator + } do + for {policy, action} <- [ + {ProjectUsers, :access_project}, + {Collections, :manage_collection}, + {Credentials, :view_keychain_credential} + ] do + assert policy.authorize(action, operator, project) == + {:error, :scheduled_for_deletion} + end + end + + test "a caller without the grant hears which grant, never the deletion", + %{project: project, grantless: grantless} do + for {policy, actions} <- @project_actions, action <- actions do + assert policy.authorize(action, grantless, project) == + {:error, {:insufficient_scope, grant(policy, action)}}, + "#{inspect(policy)} #{action}" + end + + instance = Scope.new(grantless.actor) + + for {policy, actions} <- @instance_actions, action <- actions do + assert policy.authorize(action, instance, nil) == + {:error, {:insufficient_scope, grant(policy, action)}}, + "#{inspect(policy)} #{action}" + end + end + + test "a stranger is told only that it has no standing, never about the deletion", + %{project: project, stranger: stranger} do + for {policy, actions} <- @project_actions, action <- actions do + assert policy.authorize(action, stranger, project) == + {:error, :no_standing}, + "#{inspect(policy)} #{action}" + end + end + + describe "on a project that requires MFA" do + setup do + %{project: insert(:project, requires_mfa: true)} + end + + test "a non-member has no standing, even without MFA", %{project: project} do + {:ok, scope} = Scope.fetch(insert(:user, mfa_enabled: false), project) + + assert ProjectUsers.authorize(:access_project, scope, project) == + {:error, :no_standing} + end + + test "a member without MFA is refused", %{project: project} do + member = insert(:user, mfa_enabled: false) + insert(:project_user, project: project, user: member, role: :owner) + {:ok, scope} = Scope.fetch(member, project) + + assert ProjectUsers.authorize(:access_project, scope, project) == false + end + + test "an enrolled member without the role is refused", %{project: project} do + member = insert(:user, mfa_enabled: true) + insert(:project_user, project: project, user: member, role: :viewer) + {:ok, scope} = Scope.fetch(member, project) + + assert ProjectUsers.authorize(:access_project, scope, project) == true + assert ProjectUsers.authorize(:edit_project, scope, project) == false + end + end + + test "a person asking without a scope is refused with false, standing or not" do + project = insert(:project) + + assert ProjectUsers.authorize(:access_project, insert(:user), project) == + false + end + + test "a repo connection is allowed on its own project and refused another" do + project = insert(:project) + repo_connection = insert(:project_repo_connection, project: project) + {:ok, scope} = Scope.fetch(repo_connection, project) + + assert Provisioning.authorize(:provision_project, scope, project) == true + assert Provisioning.authorize(:describe_project, scope, project) == true + + assert Scope.fetch(repo_connection, insert(:project)) == + {:error, :no_standing} + end + + defp grant(ProjectUsers, action) + when action in [:create_project_credential, :list_project_credentials], + do: "credentials:write" + + defp grant(ProjectUsers, _action), do: "projects:write" + defp grant(Collections, _action), do: "collections:write" + defp grant(Credentials, _action), do: "credentials:write" + defp grant(Users, :read_users), do: "users:read" + defp grant(Users, :write_users), do: "users:write" +end diff --git a/test/lightning/policies/project_user_permissions_test.exs b/test/lightning/policies/project_user_permissions_test.exs index 3df44ea5aa1..6f11dd98bd2 100644 --- a/test/lightning/policies/project_user_permissions_test.exs +++ b/test/lightning/policies/project_user_permissions_test.exs @@ -12,7 +12,7 @@ defmodule Lightning.Policies.ProjectUserPermissionsTest do alias Lightning.Accounts alias Lightning.Policies.{Permissions, ProjectUsers} - alias Lightning.Projects.Scope + alias Lightning.Scope @project_user_actions ~w( create_workflow @@ -112,9 +112,9 @@ defmodule Lightning.Policies.ProjectUserPermissionsTest do end # A %Project{} subject never carries a client-supplied project_user_id, so - # there is no "whose row is this" question to answer — Scope already - # resolved it to the caller's own standing. This goes through `permitted?` - # rather than the id == user_id clause above. + # there is no "whose row is this" question to answer. Scope already + # resolved it to the caller's own standing, so the check that a row + # belongs to the caller isn't consulted. test "can edit their own digest and failure alerts given only the project", %{project: project, editor: editor} do ~w( @@ -387,9 +387,8 @@ defmodule Lightning.Policies.ProjectUserPermissionsTest do "MFA: " <> Enum.join(allowed, ", ") end - # The sweep above passes a project, so it never reaches the clause that - # takes a `%ProjectUser{}`. That clause does not call permitted?/2 at all, - # so it needs — and has — its own MFA check. + # The sweep above passes a project, so it never reaches the rule for a + # `%ProjectUser{}` subject. This checks MFA refuses that subject too. test "refuses every action when the subject is the member's own row", context do actions = ProjectUsers.actions() @@ -457,6 +456,169 @@ defmodule Lightning.Policies.ProjectUserPermissionsTest do end end + describe "an operator" do + setup do + {account, _private_key} = + Lightning.ServiceAccountHelpers.service_account_with_key() + + %{ + account: %{ + account + | grants: MapSet.new(["projects:write", "credentials:write"]) + }, + bare_account: account + } + end + + test "is allowed what an owner is, but nothing about its own row", + %{account: account, project: project} do + # The expected set is written out rather than read from the module, so a + # new action fails here until someone decides whether an operator gets it. + expected = + ~w( + access_project + list_project_credentials + delete_project + add_project_user + remove_project_user + edit_project + edit_data_retention + edit_run_settings + write_webhook_auth_method + write_github_connection + create_workflow + edit_workflow + delete_workflow + run_workflow + create_project_credential + initiate_github_sync + create_channel + update_channel + delete_channel + )a + + refused = ~w(publish_template edit_digest_alerts edit_failure_alerts)a + + assert Enum.sort(ProjectUsers.actions()) == + Enum.sort(expected ++ refused), + "classify every new action as allowed or refused for an operator" + + allowed = + for action <- ProjectUsers.actions(), + Permissions.can?(ProjectUsers, action, account, project), + do: action + + assert Enum.sort(allowed) == Enum.sort(expected) + end + + test "is refused for want of the grant before anything else", + %{bare_account: account, project: project} do + projects_only = %{account | grants: MapSet.new(["projects:write"])} + credentials_only = %{account | grants: MapSet.new(["credentials:write"])} + + assert ProjectUsers.authorize(:access_project, credentials_only, project) == + {:error, {:insufficient_scope, "projects:write"}} + + assert ProjectUsers.authorize( + :list_project_credentials, + projects_only, + project + ) == {:error, {:insufficient_scope, "credentials:write"}} + + assert ProjectUsers.authorize( + :create_project_credential, + projects_only, + project + ) == {:error, {:insufficient_scope, "credentials:write"}} + + assert ProjectUsers.authorize( + :create_project_credential, + credentials_only, + project + ) == true + + # The grant is about the caller alone, so it is answered before any + # target is loaded. + assert ProjectUsers.authorize( + :edit_project, + Scope.new(credentials_only), + :before_target + ) == {:error, {:insufficient_scope, "projects:write"}} + + assert ProjectUsers.authorize( + :edit_project, + Scope.new(projects_only), + :before_target + ) == true + end + + test "is told a project is scheduled only when asking with a scope", + %{account: account} do + project = + insert(:project, + scheduled_deletion: DateTime.utc_now() |> DateTime.truncate(:second) + ) + + {:ok, scope} = Scope.fetch(account, project) + + assert ProjectUsers.authorize(:edit_project, scope, project) == + {:error, :scheduled_for_deletion} + + assert ProjectUsers.authorize(:edit_project, account, project) == false + refute ProjectUsers.permitted?(:edit_project, scope) + end + + # Only the row's member may change its alert settings, and an operator has + # no row. + test "is refused the self actions given someone's membership row", %{ + account: account, + project: project + } do + refute_can( + ProjectUsers, + [:edit_digest_alerts, :edit_failure_alerts], + account, + hd(project.project_users) + ) + end + end + + describe "a person asking with a scope" do + test "is told a project is scheduled only once otherwise allowed" do + owner = insert(:user) + outsider = insert(:user) + + project = + insert(:project, + project_users: [%{user: owner, role: :owner}], + scheduled_deletion: DateTime.utc_now() |> DateTime.truncate(:second) + ) + + {:ok, owner_scope} = Scope.fetch(owner, project) + {:ok, outsider_scope} = Scope.fetch(outsider, project) + + assert ProjectUsers.authorize(:edit_project, owner_scope, project) == + {:error, :scheduled_for_deletion} + + assert ProjectUsers.authorize(:edit_project, outsider_scope, project) == + {:error, :no_standing} + + assert ProjectUsers.authorize(:edit_project, owner, project) == false + end + + test "is refused when the subject is not the scope's project" do + owner = insert(:user) + project = insert(:project, project_users: [%{user: owner, role: :owner}]) + {:ok, scope} = Scope.fetch(owner, project) + + assert ProjectUsers.authorize(:edit_project, scope, project) == true + assert ProjectUsers.authorize(:edit_project, scope, project.id) == true + + assert ProjectUsers.authorize(:edit_project, scope, insert(:project)) == + false + end + end + describe "blocked_by_mfa?/1" do setup %{editor: editor} do enrolled = insert(:user, mfa_enabled: true) diff --git a/test/lightning/policies/provisioning_test.exs b/test/lightning/policies/provisioning_test.exs index fe038773cb2..7b5ee395116 100644 --- a/test/lightning/policies/provisioning_test.exs +++ b/test/lightning/policies/provisioning_test.exs @@ -2,17 +2,16 @@ defmodule Lightning.Policies.ProvisioningTest do @moduledoc """ Tests for the provisioning API's authorization policy. - The clauses that resolve a `Lightning.Projects.Scope` are held to the - project's MFA requirement; the ones that cannot be (a project that does not - exist yet, and a machine credential) are not. + A person is held to the project's MFA requirement; a superuser creating a + project and a repo connection are not. """ use Lightning.DataCase, async: true import Lightning.Factories alias Lightning.Policies.Permissions - alias Lightning.Policies.Provisioning - alias Lightning.Projects.Project + alias Lightning.Scope + alias Lightning.ServiceAccount.AccessToken describe ":provision_project on a project that requires MFA" do test "is refused for an unenrolled :owner or :admin" do @@ -25,12 +24,8 @@ defmodule Lightning.Policies.ProvisioningTest do # Control: the same unenrolled actor CAN provision a project that # does not require MFA, so the refusal below is about the # requirement and not about the role. - assert Permissions.can?( - Provisioning, - :provision_project, - actor, - unrestricted_project - ) + assert permit_target(actor, :provision_project, unrestricted_project) == + :ok mfa_project = insert(:project, @@ -38,14 +33,7 @@ defmodule Lightning.Policies.ProvisioningTest do project_users: [%{user_id: actor.id, role: role}] ) - refute Permissions.can?( - Provisioning, - :provision_project, - actor, - mfa_project - ) - - assert Provisioning.authorize(:provision_project, actor, mfa_project) == + assert permit_target(actor, :provision_project, mfa_project) == {:error, :forbidden} end end @@ -60,12 +48,7 @@ defmodule Lightning.Policies.ProvisioningTest do project_users: [%{user_id: actor.id, role: role}] ) - assert Permissions.can?( - Provisioning, - :provision_project, - actor, - mfa_project - ) + assert permit_target(actor, :provision_project, mfa_project) == :ok end end @@ -79,38 +62,19 @@ defmodule Lightning.Policies.ProvisioningTest do project_users: [%{user_id: actor.id, role: role}] ) - refute Permissions.can?( - Provisioning, - :provision_project, - actor, - mfa_project - ) + assert permit_target(actor, :provision_project, mfa_project) == + {:error, :forbidden} end end - end - describe "clauses that resolve no project membership" do - test "a superuser can still create a project that does not exist yet" do - # `%Project{id: nil}` has no `requires_mfa` to read and no members to - # consult; the MFA guard cannot apply and must not have been added here. - superuser = insert(:user, role: :superuser, mfa_enabled: false) - - assert Permissions.can?( - Provisioning, - :provision_project, - superuser, - %Project{id: nil} - ) - end - - test "a repo connection is unaffected by the project's MFA requirement" do + test "does not apply to a repo connection" do # No MFA concept applies to a machine credential, so the connection # keeps working on a project its human members must enrol to touch. project = insert(:project, requires_mfa: true) repo_connection = insert(:project_repo_connection, project: project) for action <- [:provision_project, :describe_project] do - assert Permissions.can?(Provisioning, action, repo_connection, project) + assert permit_target(repo_connection, action, project) == :ok end end end @@ -128,19 +92,200 @@ defmodule Lightning.Policies.ProvisioningTest do unrestricted_project = insert(:project, project_users: [%{user_id: actor.id, role: :viewer}]) - assert Permissions.can?( - Provisioning, - :describe_project, - actor, - unrestricted_project - ) + assert permit_target(actor, :describe_project, unrestricted_project) == + :ok + + assert permit_target(actor, :describe_project, mfa_project) == + {:error, :forbidden} + end + end + + describe "a scope for one project" do + test "is refused another project's subject" do + owner = insert(:user) + + project = + insert(:project, project_users: [%{user_id: owner.id, role: :owner}]) + + repo_connection = insert(:project_repo_connection, project: project) + other = insert(:project) + + {:ok, owner_scope} = Scope.fetch(owner, project) + {:ok, connection_scope} = Scope.fetch(repo_connection, project) + + for action <- [:provision_project, :describe_project], + scope <- [owner_scope, connection_scope, Scope.new(repo_connection)] do + assert Permissions.permit(:provisioning, action, scope, other) == + {:error, :forbidden} + end + + # Shows the refusals above are about the project, not the scopes. + for action <- [:provision_project, :describe_project], + scope <- [owner_scope, connection_scope] do + assert Permissions.permit(:provisioning, action, scope, project) == :ok + end + end + end + + describe "an operator on an existing project" do + setup do + {account, _private_key} = + Lightning.ServiceAccountHelpers.service_account_with_key() + + %{account: account, project: insert(:project)} + end + + test "is refused both actions without projects:provision, whatever else it holds", + %{account: account, project: project} do + account = %{account | grants: MapSet.new(AccessToken.scopes())} + + for action <- [:provision_project, :describe_project] do + assert permit_target(account, action, project) == + {:error, {:insufficient_scope, "projects:provision"}} + end + end + + test "may provision with projects:provision", %{ + account: account, + project: project + } do + account = %{account | grants: MapSet.new(["projects:provision"])} + + assert permit_target(account, :provision_project, project) == :ok + end + + test "needs projects:write as well to describe, as :access_project does", %{ + account: account, + project: project + } do + provision_only = %{account | grants: MapSet.new(["projects:provision"])} + + both = %{ + account + | grants: MapSet.new(["projects:provision", "projects:write"]) + } + + assert permit_target(provision_only, :describe_project, project) == + {:error, :forbidden} + + assert permit_target(both, :describe_project, project) == :ok + end + + test "hears the project is scheduled for deletion only once allowed", %{ + account: account + } do + project = + insert(:project, + scheduled_deletion: DateTime.utc_now() |> DateTime.add(1, :day) + ) + + assert permit_target(account, :provision_project, project) == + {:error, {:insufficient_scope, "projects:provision"}} + + account = %{account | grants: MapSet.new(["projects:provision"])} - refute Permissions.can?( - Provisioning, - :describe_project, - actor, - mfa_project - ) + assert permit_target(account, :provision_project, project) == + {:error, :scheduled_for_deletion} end end + + describe ":create_project" do + test "is allowed for a superuser, who is not held to MFA" do + superuser = insert(:user, role: :superuser, mfa_enabled: false) + + assert permit_create(superuser) == :ok + end + + test "is allowed for an operator holding projects:write, and only then" do + {account, _private_key} = + Lightning.ServiceAccountHelpers.service_account_with_key() + + assert permit_create(%{account | grants: MapSet.new(["projects:write"])}) == + :ok + + assert permit_create(%{account | grants: MapSet.new(["users:write"])}) == + {:error, {:insufficient_scope, "projects:write"}} + end + + test "is refused for a user who is not a superuser, whatever roles they hold elsewhere" do + for role <- [:owner, :admin, :editor, :viewer] do + user = insert(:user) + insert(:project, project_users: [%{user_id: user.id, role: role}]) + + assert permit_create(user) == {:error, :forbidden} + end + end + + # A repo connection holds no `projects:write`, so the grant refuses it first. + test "is refused for a repo connection" do + repo_connection = insert(:project_repo_connection) + + assert permit_create(repo_connection) == + {:error, {:insufficient_scope, "projects:write"}} + end + end + + # The routes reach only people (every grant) and repo connections + # (projects:provision), so these refusals are for tokens that can't reach + # /api/provision. + describe "before the target is loaded" do + setup do + {account, _private_key} = + Lightning.ServiceAccountHelpers.service_account_with_key() + + %{account: account} + end + + test "describing needs projects:provision", %{account: account} do + assert permit(account, [], :describe_project) == + {:error, {:insufficient_scope, "projects:provision"}} + + assert permit(account, ["projects:write"], :describe_project) == + {:error, {:insufficient_scope, "projects:provision"}} + + assert permit(account, ["projects:provision"], :describe_project) == :ok + end + + # Creating needs projects:write and provisioning an existing project + # projects:provision; which applies waits for the body's target. + test "provisioning needs either grant", %{account: account} do + assert permit(account, ["users:write"], :provision_project) == + {:error, {:insufficient_scope, "projects:provision"}} + + assert permit(account, ["projects:write"], :provision_project) == :ok + assert permit(account, ["projects:provision"], :provision_project) == :ok + end + + test "passes a repo connection" do + scope = Scope.new(insert(:project_repo_connection)) + + for action <- [:describe_project, :provision_project] do + assert Permissions.permit(:provisioning, action, scope) == :ok + end + end + + defp permit(account, grants, action) do + scope = Scope.new(%{account | grants: MapSet.new(grants)}) + Permissions.permit(:provisioning, action, scope) + end + end + + describe "a caller with no standing on the project" do + test "is told so, for the API to answer as a project that doesn't exist" do + user = insert(:user) + project = insert(:project) + + for action <- [:provision_project, :describe_project] do + assert permit_target(user, action, project) == {:error, :no_standing} + end + end + end + + defp permit_target(actor, action, project) do + {:ok, scope} = Scope.fetch(actor, project) + Permissions.permit(:provisioning, action, scope, project) + end + + defp permit_create(actor), + do: Permissions.permit(:provisioning, :create_project, Scope.new(actor), nil) end diff --git a/test/lightning/policies/user_permissions_test.exs b/test/lightning/policies/user_permissions_test.exs index acee0352be5..15dfa454ee5 100644 --- a/test/lightning/policies/user_permissions_test.exs +++ b/test/lightning/policies/user_permissions_test.exs @@ -7,8 +7,9 @@ defmodule Lightning.Policies.UserPermissionsTest do Typically, there is only one superuser per deployment of Lightning. - Regular users have full control over their own credentials, but cannot see or - modify other user's credentials. All other resources in Lightning are under + Regular users have full control over their own OAuth clients, but cannot + modify other users' clients. Credentials are decided by + `Lightning.Policies.Credentials`. All other resources in Lightning are under the control of Project User Permissions and are demonstrated in the ProjectUserPermissionsTest. """ @@ -28,7 +29,7 @@ defmodule Lightning.Policies.UserPermissionsTest do end describe "Users" do - test "can edit their own credentials and delete their own accounts, api tokens, and credentials.", + test "can edit and delete their own OAuth clients and delete their own accounts and api tokens", %{ user: user } do @@ -37,17 +38,15 @@ defmodule Lightning.Policies.UserPermissionsTest do |> with_personal_access_token() |> insert() - credential = insert(:credential, user: user) client = insert(:oauth_client, user: user) - assert Users |> Permissions.can?(:edit_credential, user, credential) + assert Users |> Permissions.can?(:edit_credential, user, client) assert Users |> Permissions.can?(:delete_account, user, user) assert Users |> Permissions.can?(:delete_api_token, user, api_token.token) - assert Users |> Permissions.can?(:delete_credential, user, credential) assert Users |> Permissions.can?(:delete_credential, user, client) end - test "cannot access admin space, edit other users credentials, and delete other users accounts, api tokens, and credentials", + test "cannot access admin space, edit or delete other users' OAuth clients, or delete other users' accounts and api tokens", %{ user: user, other_user: other_user @@ -57,18 +56,18 @@ defmodule Lightning.Policies.UserPermissionsTest do |> with_personal_access_token() |> insert() - credential = insert(:credential) + client = insert(:oauth_client, user: other_user) refute Users |> Permissions.can?(:access_admin_space, user) - refute Users |> Permissions.can?(:edit_credential, user, credential) + refute Users |> Permissions.can?(:edit_credential, user, client) refute Users |> Permissions.can?(:delete_account, user, other_user) refute Users |> Permissions.can?(:delete_api_token, user, api_token.token) - refute Users |> Permissions.can?(:delete_credential, user, credential) + refute Users |> Permissions.can?(:delete_credential, user, client) end end describe "Superusers" do - test "can access admin space, edit their own credentials, and delete their own accounts, api tokens, and credentials.", + test "can access admin space and delete their own accounts and api tokens", %{ superuser: superuser } do @@ -77,19 +76,14 @@ defmodule Lightning.Policies.UserPermissionsTest do |> with_personal_access_token() |> insert() - credential = insert(:credential, user: superuser) - assert Users |> Permissions.can?(:access_admin_space, superuser) - assert Users |> Permissions.can?(:edit_credential, superuser, credential) assert Users |> Permissions.can?(:delete_account, superuser, superuser) assert Users |> Permissions.can?(:delete_api_token, superuser, api_token.token) - - assert Users |> Permissions.can?(:delete_credential, superuser, credential) end - test "cannot edit other users credentials, and delete other users accounts, api tokens, and credentials", + test "cannot edit or delete other users' OAuth clients, or delete other users' accounts and api tokens", %{ superuser: superuser, other_user: other_user @@ -99,15 +93,62 @@ defmodule Lightning.Policies.UserPermissionsTest do |> with_personal_access_token() |> insert() - credential = insert(:credential) + client = insert(:oauth_client, user: other_user) - refute Users |> Permissions.can?(:edit_credential, superuser, credential) + refute Users |> Permissions.can?(:edit_credential, superuser, client) refute Users |> Permissions.can?(:delete_account, superuser, other_user) refute Users |> Permissions.can?(:delete_api_token, superuser, api_token.token) - refute Users |> Permissions.can?(:delete_credential, superuser, credential) + refute Users |> Permissions.can?(:delete_credential, superuser, client) + end + end + + describe "the users API" do + test "is for an operator holding the grant each action needs" do + {account, _key} = + Lightning.ServiceAccountHelpers.service_account_with_key() + + reader = + Lightning.Scope.new(%{account | grants: MapSet.new(["users:read"])}) + + assert Users.authorize(:read_users, reader, nil) == true + + assert Users.authorize(:write_users, reader, nil) == + {:error, {:insufficient_scope, "users:write"}} + end + + test "is refused to a person, even a superuser" do + for user <- [insert(:user), insert(:user, role: :superuser)] do + scope = Lightning.Scope.new(user) + + assert Users.authorize(:read_users, scope, nil) == false + assert Users.authorize(:write_users, scope, nil) == false + end + end + + test "is decided before any target, so a person never passes the first check" do + scope = Lightning.Scope.new(insert(:user, role: :superuser)) + + assert Permissions.permit(:users, :read_users, scope) == + {:error, :forbidden} + + assert Permissions.permit(:users, :write_users, scope) == + {:error, :forbidden} + end + + test "before any target, a service account passes with the grant and hears which one it lacks" do + {account, _key} = + Lightning.ServiceAccountHelpers.service_account_with_key() + + reader = + Lightning.Scope.new(%{account | grants: MapSet.new(["users:read"])}) + + assert Permissions.permit(:users, :read_users, reader) == :ok + + assert Permissions.permit(:users, :write_users, reader) == + {:error, {:insufficient_scope, "users:write"}} end end end diff --git a/test/lightning/projects/membership_events_test.exs b/test/lightning/projects/membership_events_test.exs index dfa026613d9..896846e9105 100644 --- a/test/lightning/projects/membership_events_test.exs +++ b/test/lightning/projects/membership_events_test.exs @@ -123,6 +123,39 @@ defmodule Lightning.Projects.MembershipEventsTest do assert audit_count(project_id) == 3 end + test "audits a rename alongside the membership it changes", %{ + project: %{id: project_id, name: old_name} = project, + actor: %{id: actor_id} = actor, + owner_project_user: owner_project_user, + member_project_user: member_project_user + } do + assert {:ok, _project} = + Projects.update_project_with_users( + project, + %{ + name: "renamed", + project_users: [ + %{id: owner_project_user.id}, + %{id: member_project_user.id, delete: true} + ] + }, + actor, + false + ) + + assert %{ + item_id: ^project_id, + actor_id: ^actor_id, + changes: %Audit.Changes{ + before: %{"name" => ^old_name}, + after: %{"name" => "renamed"} + } + } = Repo.get_by!(Audit, event: "name_updated") + + assert Repo.get_by!(Audit, event: "collaborator_removed").actor_id == + actor_id + end + test "broadcasts and audits nothing when membership is resubmitted unchanged", %{ project: %{id: project_id} = project, diff --git a/test/lightning/projects/provisioner_test.exs b/test/lightning/projects/provisioner_test.exs index e7d83830db8..7dd63ca89ac 100644 --- a/test/lightning/projects/provisioner_test.exs +++ b/test/lightning/projects/provisioner_test.exs @@ -10,7 +10,6 @@ defmodule Lightning.Projects.ProvisionerTest do import Ecto.Query import Lightning.AdaptorTestHelpers import Lightning.Factories - import LightningWeb.CoreComponents, only: [translate_error: 1] describe "parse_document/2 with a new project" do setup :isolated_adaptors @@ -21,8 +20,8 @@ defmodule Lightning.Projects.ProvisionerTest do changeset = Provisioner.parse_document(%Lightning.Projects.Project{}, %{}) assert flatten_errors(changeset) == %{ - id: ["This field can't be blank."], - name: ["This field can't be blank."] + id: ["can't be blank"], + name: ["can't be blank"] } %{body: body} = valid_document() @@ -56,8 +55,8 @@ defmodule Lightning.Projects.ProvisionerTest do workflows: [ %{ jobs: [ - %{id: ["This field can't be blank."]}, - %{id: ["This field can't be blank."]} + %{id: ["can't be blank"]}, + %{id: ["can't be blank"]} ] } ] @@ -72,10 +71,18 @@ defmodule Lightning.Projects.ProvisionerTest do }) assert flatten_errors(changeset) == %{ - id: ["This field can't be blank."], - name: ["This field can't be blank."], - base: ["extraneous parameters: baz, foo"] + id: ["can't be blank"], + name: ["can't be blank"], + base: ["is not accepted", "is not accepted"] } + + assert Enum.sort(changeset.errors) == + Enum.sort( + id: {"can't be blank", [validation: :required]}, + name: {"can't be blank", [validation: :required]}, + base: {"is not accepted", [key: "foo", code: :unknown_key]}, + base: {"is not accepted", [key: "baz", code: :unknown_key]} + ) end test "with server-owned trigger fields" do @@ -96,10 +103,15 @@ defmodule Lightning.Projects.ProvisionerTest do refute changeset.valid? - assert %{workflows: [%{triggers: [%{base: [error]}]}]} = + assert %{workflows: [%{triggers: [%{base: ["is not accepted"]}]}]} = flatten_errors(changeset) - assert error == "extraneous parameters: #{field}" + assert %{workflows: [%{triggers: [%{base: unknown}]}]} = + raw_errors(changeset) + + assert unknown == [ + {"is not accepted", [key: field, code: :unknown_key]} + ] end end @@ -347,6 +359,51 @@ defmodule Lightning.Projects.ProvisionerTest do assert project_credential.credential_id == credential.id end + test "reports a credential name it can't find at the link's place in the request" do + user = insert(:user) + other = insert(:user) + known = insert(:credential, name: "known", user: user) + insert(:credential, name: "theirs", user: other) + + %{body: body} = valid_document() + + body = + Map.put(body, "project_credentials", [ + %{ + "id" => Ecto.UUID.generate(), + "name" => "theirs", + "owner" => other.email + }, + %{ + "id" => Ecto.UUID.generate(), + "name" => "missing", + "owner" => user.email + }, + %{ + "id" => Ecto.UUID.generate(), + "name" => known.name, + "owner" => user.email + } + ]) + + Mox.stub(Lightning.Extensions.MockUsageLimiter, :limit_action, fn _a, _c -> + :ok + end) + + assert {:error, changeset} = + Provisioner.import_document( + %Lightning.Projects.Project{}, + user, + body + ) + + assert changeset.errors == [ + project_credentials: + {"names nothing you can see", + [key: ["project_credentials", 1, "name"], code: :not_found]} + ] + end + test "with valid data" do Mox.verify_on_exit!() %{id: user_id} = user = insert(:user) @@ -1123,7 +1180,7 @@ defmodule Lightning.Projects.ProvisionerTest do workflows: [ %{ jobs: [ - %{id: ["This value should be unique."]}, + %{id: ["has already been taken"]}, %{}, %{} ] @@ -1146,7 +1203,7 @@ defmodule Lightning.Projects.ProvisionerTest do assert flatten_errors(changeset) == %{ workflows: [ %{ - triggers: [%{id: ["This value should be unique."]}, %{}] + triggers: [%{id: ["has already been taken"]}, %{}] } ] } @@ -1169,7 +1226,7 @@ defmodule Lightning.Projects.ProvisionerTest do workflows: [ %{ edges: [ - %{id: ["This value should be unique."]}, + %{id: ["has already been taken"]}, %{}, %{} ] @@ -1931,14 +1988,16 @@ defmodule Lightning.Projects.ProvisionerTest do assert {:error, changeset} = Provisioner.import_document(project, user, body) - assert %{ - channels: [ - %{destination_auth_method: %{project_credential_id: [msg]}} - ] - } = flatten_errors(changeset) + assert %{channels: [%{destination_credential_id: [msg]}]} = + flatten_errors(changeset) assert msg =~ "isn't available in this project" + assert %{channels: [%{destination_credential_id: [{_, opts}]}]} = + raw_errors(changeset) + + assert opts == [code: :credential_not_allowed] + # Channel was not persisted refute Repo.get(Lightning.Channels.Channel, channel_id) end @@ -1967,14 +2026,19 @@ defmodule Lightning.Projects.ProvisionerTest do assert {:error, changeset} = Provisioner.import_document(project, user, body) - assert %{ - channels: [ - %{destination_auth_method: %{project_credential_id: [msg]}} - ] - } = flatten_errors(changeset) + assert %{channels: [%{destination_credential_id: [msg]}]} = + flatten_errors(changeset) assert msg == Scoping.violation_message(:project_credential_id) + assert %{channels: [%{destination_credential_id: [{_, opts}]}]} = + raw_errors(changeset) + + assert opts == [ + constraint: :foreign, + constraint_name: "channel_auth_methods_project_credential_id_fkey" + ] + # Channel was not persisted refute Repo.get(Lightning.Channels.Channel, channel_id) end @@ -2687,17 +2751,42 @@ defmodule Lightning.Projects.ProvisionerTest do :limit_action, fn %{type: :github_sync}, _context -> - {:error, :upgrade_required, error_message} + {:error, :exceeds_limit, error_message} _action, _context -> :ok end ) - assert {:error, ^error_message} = + assert {:error, :exceeds_limit, ^error_message} = Provisioner.import_document(project, repo_connection, body) end + test "refuses an invalid document for its errors before asking the limiter", + %{project: project} do + repo_connection = insert(:project_repo_connection, project: project) + %{body: body} = valid_document(project.id) + + Mox.stub( + Lightning.Extensions.MockUsageLimiter, + :limit_action, + fn + %{type: :github_sync}, _context -> + flunk("the limiter was asked about github sync") + + _action, _context -> + :ok + end + ) + + assert {:error, %Ecto.Changeset{valid?: false}} = + Provisioner.import_document( + project, + repo_connection, + Map.put(body, "name", nil) + ) + end + test "allows import when actor is a repo_connection and plan allows github sync", %{project: project} do repo_connection = insert(:project_repo_connection, project: project) @@ -2896,8 +2985,10 @@ defmodule Lightning.Projects.ProvisionerTest do } end + defp raw_errors(changeset), do: Ecto.Changeset.traverse_errors(changeset, & &1) + defp flatten_errors(changeset) do - Ecto.Changeset.traverse_errors(changeset, &translate_error/1) + errors_on(changeset) end defp add_entity_to_workflow(document, workflow_id, key, params) do diff --git a/test/lightning/projects_test.exs b/test/lightning/projects_test.exs index c050642f802..51c77024d3b 100644 --- a/test/lightning/projects_test.exs +++ b/test/lightning/projects_test.exs @@ -6,6 +6,7 @@ defmodule Lightning.ProjectsTest do import Lightning.AccountsFixtures import Lightning.Factories import Lightning.ProjectsFixtures + import Lightning.ServiceAccountHelpers import Mox import Swoosh.TestAssertions @@ -122,8 +123,8 @@ defmodule Lightning.ProjectsTest do assert Projects.get_project_user(project_user.id) == project_user end - test "create_project/1 with valid data creates a project" do - %{id: user_id} = insert(:user) + test "create_project/3 with valid data creates a project" do + %{id: user_id} = user = insert(:user) valid_attrs = %{ name: "some-name", @@ -131,7 +132,7 @@ defmodule Lightning.ProjectsTest do } assert {:ok, %Project{id: project_id} = project} = - Projects.create_project(valid_attrs) + Projects.create_project(valid_attrs, user) assert project.name == "some-name" @@ -139,28 +140,34 @@ defmodule Lightning.ProjectsTest do project.project_users end - test "create_project/1 expects project to have exactly one owner" do + test "create_project/3 expects project to have exactly one owner" do user = insert(:user) # creates successfully if there's one assert {:ok, _project} = - Projects.create_project(%{ - name: "some-name", - project_users: [%{user_id: user.id, role: :owner}] - }) + Projects.create_project( + %{ + name: "some-name", + project_users: [%{user_id: user.id, role: :owner}] + }, + user + ) # errors out if there is none for role <- [:admin, :editor, :viewer] do assert {:error, %Ecto.Changeset{errors: errors}} = - Projects.create_project(%{ - name: "some-name", - project_users: [%{user_id: user.id, role: role}] - }) + Projects.create_project( + %{ + name: "some-name", + project_users: [%{user_id: user.id, role: role}] + }, + user + ) assert [ {:owner, {"Every project must have exactly one owner. Please specify one below.", - []}} + [code: :one_owner]}} ] == errors end @@ -169,24 +176,64 @@ defmodule Lightning.ProjectsTest do another_user = insert(:user) assert {:error, %Ecto.Changeset{errors: errors}} = - Projects.create_project(%{ - name: "some-name", - project_users: [ - %{user_id: user.id, role: :owner}, - %{user_id: another_user.id, role: :owner} - ] - }) + Projects.create_project( + %{ + name: "some-name", + project_users: [ + %{user_id: user.id, role: :owner}, + %{user_id: another_user.id, role: :owner} + ] + }, + user + ) - assert [{:owner, {"A project can have only one owner.", []}}] == + assert [ + {:owner, + {"A project can have only one owner.", [code: :one_owner]}} + ] == errors end - test "create_project/1 with invalid data returns error changeset" do + test "update_project_with_users/4 hands ownership over when the promotion is listed before the demotion" do + owner = insert(:user) + editor = insert(:user) + + project = + insert(:project, + project_users: [ + %{user: owner, role: :owner}, + %{user: editor, role: :editor} + ] + ) + + owner_pu = Enum.find(project.project_users, &(&1.user_id == owner.id)) + editor_pu = Enum.find(project.project_users, &(&1.user_id == editor.id)) + + assert {:ok, _project} = + Projects.update_project_with_users( + project, + %{ + project_users: [ + %{id: editor_pu.id, role: :owner}, + %{id: owner_pu.id, role: :editor} + ] + }, + owner, + false + ) + + assert %{role: :editor} = Repo.reload!(owner_pu) + assert %{role: :owner} = Repo.reload!(editor_pu) + end + + test "create_project/3 with invalid data returns error changeset" do + user = insert(:user) + assert {:error, %Ecto.Changeset{}} = - Projects.create_project(@invalid_attrs) + Projects.create_project(@invalid_attrs, user) assert {:error, %Ecto.Changeset{}} = - Projects.create_project(%{"name" => "Can't have spaces!"}) + Projects.create_project(%{"name" => "Can't have spaces!"}, user) end test "set_notification_pref/3 with a changed value updates the field" do @@ -3308,34 +3355,154 @@ defmodule Lightning.ProjectsTest do end end + describe "create_project/3 audit and notification" do + setup do + owner = insert(:user) + admin = insert(:user) + viewer = insert(:user) + + members = [{owner, :owner}, {admin, :admin}, {viewer, :viewer}] + + %{ + members: members, + attrs: %{ + name: "audited", + description: "made by apply", + project_users: + Enum.map(members, fn {user, role} -> + %{user_id: user.id, role: role} + end) + } + } + end + + test "a service account is recorded as creating the project and adding each member", + %{attrs: attrs, members: members} do + {account, _key} = service_account_with_key() + + assert {:ok, project} = Projects.create_project(attrs, account) + + assert_creation_audited(project, members, account.uuid, :service_account) + end + + test "a person is recorded as creating the project and adding each member", + %{attrs: attrs, members: members} do + user = insert(:user) + + assert {:ok, project} = Projects.create_project(attrs, user) + + assert_creation_audited(project, members, user.id, :user) + end + + test "an id another project already holds is an error on id, not a raise", + %{attrs: attrs} do + taken = insert(:project) + + assert {:error, changeset} = + Projects.create_project( + Map.put(attrs, :id, taken.id), + insert(:user) + ) + + assert {_message, opts} = changeset.errors[:id] + assert opts[:constraint] == :unique + end + + test "an audit event it cannot record is an error, and leaves no project", + %{attrs: attrs} do + id = Ecto.UUID.generate() + + assert {:error, %Ecto.Changeset{} = changeset} = + Projects.create_project(Map.put(attrs, :id, id), %User{}) + + assert changeset.errors[:actor_id] + refute Repo.get(Project, id) + end + + test "emails each initial member by default, and none with notify: false", + %{attrs: attrs} do + user = insert(:user) + + Oban.Testing.with_testing_mode(:manual, fn -> + {:ok, quiet} = + Projects.create_project(%{attrs | name: "quiet"}, user, notify: false) + + {:ok, loud} = Projects.create_project(attrs, user) + + notified = + all_enqueued(worker: Lightning.Accounts.UserNotifier) + |> Enum.map(& &1.args["project_user_id"]) + |> Enum.sort() + + assert notified == loud.project_users |> Enum.map(& &1.id) |> Enum.sort() + + refute Enum.any?(quiet.project_users, &(&1.id in notified)) + end) + end + + defp assert_creation_audited(project, members, actor_id, actor_type) do + events = + from(a in Audit, where: a.item_id == ^project.id, order_by: a.event) + |> Repo.all() + |> Enum.map(&{&1.event, &1.actor_id, &1.actor_type, &1.changes}) + + added = + for {user, role} <- members do + {"collaborator_added", actor_id, actor_type, + %Audit.Changes{ + before: nil, + after: %{"user_id" => user.id, "role" => to_string(role)} + }} + end + + created = + {"created", actor_id, actor_type, + %Audit.Changes{ + before: nil, + after: %{"name" => "audited", "description" => "made by apply"} + }} + + assert Enum.sort([created | added]) == Enum.sort(events) + end + end + describe ".update_project/3" do setup do %{user: insert(:user)} end - test "update_project/3 with valid data updates the project" do + test "update_project/3 with valid data updates the project", %{user: user} do project = project_fixture() update_attrs = %{name: "some-updated-name"} assert {:ok, %Project{} = project} = - Projects.update_project(project, update_attrs) + Projects.update_project(project, update_attrs, user) assert project.name == "some-updated-name" end - test "update_project/3 updates the MFA requirement" do + test "update_project/3 on a project another request deleted is a concurrent edit, not a raise", + %{user: user} do + project = insert(:project) + Repo.delete!(project) + + assert Projects.update_project(project, %{name: "renamed"}, user) == + {:error, :concurrent_edit} + end + + test "update_project/3 updates the MFA requirement", %{user: user} do project = insert(:project) refute project.requires_mfa update_attrs = %{requires_mfa: true} assert {:ok, %Project{} = project} = - Projects.update_project(project, update_attrs) + Projects.update_project(project, update_attrs, user) assert project.requires_mfa end - test "update_project/3 updates the data retention periods" do + test "update_project/3 updates the data retention periods", %{user: user} do project = insert(:project, project_users: @@ -3356,7 +3523,7 @@ defmodule Lightning.ProjectsTest do } assert {:ok, %Project{} = updated_project} = - Projects.update_project(project, update_attrs) + Projects.update_project(project, update_attrs, user) # admins and owners receives an email admins = @@ -3400,7 +3567,7 @@ defmodule Lightning.ProjectsTest do # no email is sent when there's no change assert {:ok, updated_project} = - Projects.update_project(updated_project, update_attrs) + Projects.update_project(updated_project, update_attrs, user) for %{user: %{email: email}} <- project.project_users do refute_receive {:email, @@ -3413,10 +3580,14 @@ defmodule Lightning.ProjectsTest do # no email is sent when there's an error in the changeset assert {:error, _changeset} = - Projects.update_project(updated_project, %{ - history_retention_period: "xyz", - dataclip_retention_period: 7 - }) + Projects.update_project( + updated_project, + %{ + history_retention_period: "xyz", + dataclip_retention_period: 7 + }, + user + ) for %{user: %{email: email}} <- project.project_users do refute_receive {:email, @@ -3428,7 +3599,8 @@ defmodule Lightning.ProjectsTest do end end - test "update_project/3 rejects lowering history below existing dataclip retention" do + test "update_project/3 rejects lowering history below existing dataclip retention", + %{user: user} do project = insert(:project, history_retention_period: 30, @@ -3436,23 +3608,29 @@ defmodule Lightning.ProjectsTest do ) assert {:error, changeset} = - Projects.update_project(project, %{history_retention_period: 7}) + Projects.update_project( + project, + %{history_retention_period: 7}, + user + ) assert "dataclip retention period must be less or equal to the history retention period" in errors_on( changeset ).dataclip_retention_period end - test "update_project/3 with invalid data returns error changeset" do + test "update_project/3 with invalid data returns error changeset", %{ + user: user + } do project = project_fixture() |> unload_relation(:project_users) assert {:error, %Ecto.Changeset{}} = - Projects.update_project(project, @invalid_attrs) + Projects.update_project(project, @invalid_attrs, user) assert Repo.preload(project, :parent) == Projects.get_project!(project.id) end - test "update_project/2 calls the validate_changeset hook" do + test "update_project/3 calls the validate_changeset hook", %{user: user} do verify_on_exit!() project = @@ -3474,9 +3652,13 @@ defmodule Lightning.ProjectsTest do ) assert {:error, changeset} = - Projects.update_project(project, %{ - name: "new-name" - }) + Projects.update_project( + project, + %{ + name: "new-name" + }, + user + ) assert errors_on(changeset) == %{name: [error_msg]} end @@ -3578,24 +3760,41 @@ defmodule Lightning.ProjectsTest do } = Repo.get_by!(Audit, event: "requires_mfa_updated") end - test "does not create events if no user was provided" do - project = - insert( - :project, - dataclip_retention_period: 7, - history_retention_period: 30, - retention_policy: :retain_all - ) + test "records a service account renaming the project" do + {%{uuid: account_uuid} = account, _key} = service_account_with_key() + %{id: project_id} = project = insert(:project, name: "before") - update_attrs = %{ - dataclip_retention_period: 14, - history_retention_period: 90, - retention_policy: :retain_with_errors - } + assert {:ok, _} = + Projects.update_project(project, %{name: "after"}, account) - Projects.update_project(project, update_attrs) + assert %{ + item_id: ^project_id, + actor_id: ^account_uuid, + actor_type: :service_account, + changes: %Audit.Changes{ + before: %{"name" => "before"}, + after: %{"name" => "after"} + } + } = Repo.get_by!(Audit, event: "name_updated") + end - assert Audit |> Repo.all() |> Enum.empty?() + test "records a person changing the description", %{ + user: %{id: user_id} = user + } do + %{id: project_id} = project = insert(:project, description: "old") + + assert {:ok, _} = + Projects.update_project(project, %{description: "new"}, user) + + assert %{ + item_id: ^project_id, + actor_id: ^user_id, + actor_type: :user, + changes: %Audit.Changes{ + before: %{"description" => "old"}, + after: %{"description" => "new"} + } + } = Repo.get_by!(Audit, event: "description_updated") end test "does not create events if the project change fails", %{ @@ -3757,7 +3956,7 @@ defmodule Lightning.ProjectsTest do assert Enum.uniq(Enum.map(got, & &1.parent_id)) == [parent.id] end - test "create_sandbox/3 sets parent and creates a normal project (emails off by default)" do + test "create_sandbox/3 sets parent and creates a normal project" do owner = insert(:user) parent = @@ -3768,7 +3967,7 @@ defmodule Lightning.ProjectsTest do project_users: [%{user_id: owner.id, role: :owner}] } - assert {:ok, sandbox} = Projects.create_sandbox(parent, attrs) + assert {:ok, sandbox} = Projects.create_sandbox(parent, attrs, owner) assert sandbox.parent_id == parent.id assert sandbox.name == "sandbox-1" end @@ -5024,6 +5223,27 @@ defmodule Lightning.ProjectsTest do end end + describe "paginate_projects_for_user/2" do + test "pages the projects the user is a member of, by name" do + user = insert(:user) + + insert(:project, name: "b-project", project_users: [%{user_id: user.id}]) + insert(:project, name: "a-project", project_users: [%{user_id: user.id}]) + insert(:project, name: "c-project") + + page = Projects.paginate_projects_for_user(user, page: 2, page_size: 1) + + assert %Scrivener.Page{ + page_number: 2, + page_size: 1, + total_entries: 2, + total_pages: 2 + } = page + + assert Enum.map(page.entries, & &1.name) == ["b-project"] + end + end + describe "subscribe/0" do test "delivers project lifecycle events to the calling process" do assert :ok = Projects.subscribe() @@ -5069,10 +5289,14 @@ defmodule Lightning.ProjectsTest do end defp change_retention_periods(project) do - Projects.update_project(project, %{ - history_retention_period: 14, - dataclip_retention_period: 7 - }) + Projects.update_project( + project, + %{ + history_retention_period: 14, + dataclip_retention_period: 7 + }, + insert(:user) + ) end defp data_retention_email(updated_project) do diff --git a/test/lightning/runs_test.exs b/test/lightning/runs_test.exs index c0f8b42550e..5695742f9a0 100644 --- a/test/lightning/runs_test.exs +++ b/test/lightning/runs_test.exs @@ -324,8 +324,9 @@ defmodule Lightning.RunsTest do "step_id" => Ecto.UUID.generate() }) - assert {:job_id, {"does not exist", []}} in changeset.errors - refute {:run_id, {"does not exist", []}} in changeset.errors + assert {:job_id, {"does not exist", [code: :not_found]}} in changeset.errors + + refute {:run_id, {"does not exist", [code: :not_found]}} in changeset.errors # both run_id and job_id doesn't exist {:error, changeset} = @@ -335,8 +336,9 @@ defmodule Lightning.RunsTest do "step_id" => Ecto.UUID.generate() }) - assert {:job_id, {"does not exist", []}} in changeset.errors - assert {:run_id, {"does not exist", []}} in changeset.errors + assert {:job_id, {"does not exist", [code: :not_found]}} in changeset.errors + + assert {:run_id, {"does not exist", [code: :not_found]}} in changeset.errors Lightning.WorkOrders.subscribe(workflow.project_id) @@ -382,7 +384,7 @@ defmodule Lightning.RunsTest do "step_id" => Ecto.UUID.generate() }) - assert {:job_id, {"does not exist", []}} in changeset.errors + assert {:job_id, {"does not exist", [code: :not_found]}} in changeset.errors {:ok, step} = Runs.start_step(run_1, %{ @@ -621,7 +623,10 @@ defmodule Lightning.RunsTest do work_order_for(trigger, workflow: workflow, dataclip: dataclip) |> insert() - assert {:error, %Ecto.Changeset{errors: [step_id: {"not found", []}]}} = + assert {:error, + %Ecto.Changeset{ + errors: [step_id: {"not found", [code: :not_found]}] + }} = Runs.complete_step(%{ step_id: Ecto.UUID.generate(), reason: "success", @@ -851,7 +856,7 @@ defmodule Lightning.RunsTest do assert {:state, {"cannot mark run success that has not been claimed by a worker", - []}} in changeset.errors + [code: :not_allowed]}} in changeset.errors {:ok, run} = Repo.update(run |> Ecto.Changeset.change(state: :claimed)) @@ -897,7 +902,7 @@ defmodule Lightning.RunsTest do assert changeset.errors == [ state: {"cannot mark run lost that has not been claimed by a worker", - []} + [code: :not_allowed]} ] end @@ -988,7 +993,7 @@ defmodule Lightning.RunsTest do {"should be at most %{count} character(s)", [count: 8, validation: :length, kind: :max, type: :string]}} in changeset.errors - assert {:message, {"can't be blank", []}} in changeset.errors + assert {:message, {"can't be blank", [code: :required]}} in changeset.errors assert {:timestamp, {"can't be blank", [validation: :required]}} in changeset.errors @@ -999,7 +1004,7 @@ defmodule Lightning.RunsTest do timestamp: DateTime.utc_now() |> DateTime.to_unix(:millisecond) }) - assert {:step_id, {"must be associated with the run", []}} in changeset.errors + assert {:step_id, {"must be associated with the run", [code: :not_found]}} in changeset.errors {:ok, _log_line} = Runs.append_run_log(run, %{ @@ -1281,6 +1286,78 @@ defmodule Lightning.RunsTest do end end + describe "paginate_runs_for/3" do + setup do + user = insert(:user) + project = insert(:project, project_users: [%{user_id: user.id}]) + + %{triggers: [trigger]} = + workflow = insert(:simple_workflow, project: project) + + [started, success] = + for state <- [:started, :success] do + insert(:workorder, + workflow: workflow, + trigger: trigger, + dataclip: build(:dataclip), + runs: [ + %{ + state: state, + dataclip: build(:dataclip), + starting_trigger: trigger + } + ] + ).runs + |> hd() + end + + %{triggers: [other_trigger]} = other_workflow = insert(:simple_workflow) + + insert(:workorder, + workflow: other_workflow, + trigger: other_trigger, + dataclip: build(:dataclip), + runs: [ + %{ + state: :success, + dataclip: build(:dataclip), + starting_trigger: other_trigger + } + ] + ) + + %{user: user, project: project, started: started, success: success} + end + + test "narrows a project's runs by the filters", %{ + project: project, + success: success + } do + assert %Scrivener.Page{entries: [%{id: id}], total_entries: 1} = + Runs.paginate_runs_for(project, %{state: ["success"]}, []) + + assert id == success.id + end + + test "pages the runs of every project a user may access", %{ + user: user, + started: started, + success: success + } do + page = Runs.paginate_runs_for(user, %{}, page_size: 1) + + assert %Scrivener.Page{ + page_number: 1, + page_size: 1, + total_entries: 2, + total_pages: 2 + } = page + + assert MapSet.new(Runs.paginate_runs_for(user, %{}, []).entries, & &1.id) == + MapSet.new([started.id, success.id]) + end + end + describe "wipe_dataclips/1" do test "clears the dataclip body and request fields" do project = insert(:project) diff --git a/test/lightning/projects/scope_test.exs b/test/lightning/scope_test.exs similarity index 50% rename from test/lightning/projects/scope_test.exs rename to test/lightning/scope_test.exs index b06af7f2a3a..84ec5d7241e 100644 --- a/test/lightning/projects/scope_test.exs +++ b/test/lightning/scope_test.exs @@ -1,8 +1,8 @@ -defmodule Lightning.Projects.ScopeTest do +defmodule Lightning.ScopeTest do use Lightning.DataCase, async: true alias Lightning.Projects.Project - alias Lightning.Projects.Scope + alias Lightning.Scope setup do editor = insert(:user) @@ -32,23 +32,36 @@ defmodule Lightning.Projects.ScopeTest do assert scope.actor == editor end - test "a scheduled-deleted project yields no scope, and says why", %{ + test "a scope from new/1 gains the project's facts, as the actor would", %{ editor: editor, + live_project: project + } do + assert Scope.fetch(Scope.new(editor), project) == + Scope.fetch(editor, project) + end + + test "a scheduled-deleted project resolves, and says so", %{ + editor: editor, + live_project: live_project, scheduled_deleted_project: project } do - assert {:error, :project_scheduled_for_deletion} = + assert {:ok, %Scope{scheduled_for_deletion?: true, role: :editor}} = Scope.fetch(editor, project) + + assert {:ok, %Scope{scheduled_for_deletion?: false}} = + Scope.fetch(editor, live_project) end - test "a hand-built project struct cannot assert its own liveness", %{ + test "a hand-built project struct cannot hide that it is scheduled", %{ editor: editor, scheduled_deleted_project: project } do # scheduled_deletion is nil on this struct because nothing read it. Without - # the reload it would pass a liveness check. + # the reload the scope would say the project is live. fake = %Project{id: project.id} - assert {:error, :project_scheduled_for_deletion} = Scope.fetch(editor, fake) + assert {:ok, %Scope{scheduled_for_deletion?: true}} = + Scope.fetch(editor, fake) end describe "subject shapes" do @@ -84,6 +97,35 @@ defmodule Lightning.Projects.ScopeTest do end end + describe "fetch_all/2" do + test "gives each project the scope fetch/2 would, leaving out ids it can't resolve", + %{editor: editor, live_project: project} do + elsewhere = insert(:project) + + scopes = + Scope.fetch_all(editor, [ + project.id, + elsewhere.id, + Ecto.UUID.generate(), + "not-a-uuid" + ]) + + assert Map.keys(scopes) |> Enum.sort() == + Enum.sort([project.id, elsewhere.id]) + + for id <- [project.id, elsewhere.id], + do: assert({:ok, scopes[id]} == Scope.fetch(editor, id)) + end + + test "leaves out every project but a repo connection's own" do + [own, other] = insert_pair(:project) + repo_connection = insert(:project_repo_connection, project: own) + + assert Map.keys(Scope.fetch_all(repo_connection, [own.id, other.id])) == + [own.id] + end + end + describe "non-members" do test "get no role but still resolve", %{live_project: project} do stranger = insert(:user) @@ -113,7 +155,7 @@ defmodule Lightning.Projects.ScopeTest do assert scope.support? == false end - test "a support user is refused a scheduled-deleted project outright" do + test "a support user sees a scheduled-deleted project as scheduled" do support_user = insert(:user, support_user: true) project = @@ -122,7 +164,7 @@ defmodule Lightning.Projects.ScopeTest do scheduled_deletion: DateTime.utc_now() |> DateTime.truncate(:second) ) - assert {:error, :project_scheduled_for_deletion} = + assert {:ok, %Scope{scheduled_for_deletion?: true, support?: true}} = Scope.fetch(support_user, project) end end @@ -176,9 +218,125 @@ defmodule Lightning.Projects.ScopeTest do end # Pins the fail-closed default: a scope built without going through - # `build/2` should refuse, not silently pass. - test "defaults to false on a scope that never went through build/2" do - assert %Scope{}.mfa_satisfied? == false + # `new/1` should refuse, not silently pass. + test "defaults to false on a scope that never went through new/1" do + assert %Scope{actor: insert(:user)}.mfa_satisfied? == false + end + end + + describe "new/1" do + test "a person holds every grant and has met no project's MFA rule yet" do + user = insert(:user, support_user: true, mfa_enabled: false) + + assert %Scope{ + actor: ^user, + grants: :all, + instance_role: nil, + support_user?: true, + mfa_satisfied?: true, + project: nil, + role: nil + } = Scope.new(user) + end + + test "a superuser's instance role is :superuser" do + assert Scope.new(insert(:user, role: :superuser)).instance_role == + :superuser + end + + test "a service account holds its token's grants and is an operator" do + {account, _private_key} = + Lightning.ServiceAccountHelpers.service_account_with_key() + + account = %{account | grants: MapSet.new(["projects:write"])} + + assert %Scope{ + actor: ^account, + instance_role: :operator, + mfa_satisfied?: true + } = scope = Scope.new(account) + + assert scope.grants == MapSet.new(["projects:write"]) + end + + test "a repo connection holds only projects:provision and no instance role" do + repo_connection = insert(:project_repo_connection) + + assert %Scope{ + actor: ^repo_connection, + instance_role: nil, + mfa_satisfied?: true + } = scope = Scope.new(repo_connection) + + assert scope.grants == MapSet.new(["projects:provision"]) + end + + test "a hand-built scope holds no grant" do + assert %Scope{actor: insert(:user)}.grants == MapSet.new() + end + + test "only a person can own a credential" do + {account, _private_key} = + Lightning.ServiceAccountHelpers.service_account_with_key() + + assert Scope.new(insert(:user)).can_own_credentials? + refute Scope.new(account).can_own_credentials? + refute Scope.new(insert(:project_repo_connection)).can_own_credentials? + refute %Scope{actor: insert(:user)}.can_own_credentials? + end + end + + describe "allows?/2" do + test "a person is allowed any grant" do + assert Scope.allows?(Scope.new(insert(:user)), "projects:write") + end + + test "a token is allowed only the grants it carries" do + {account, _private_key} = + Lightning.ServiceAccountHelpers.service_account_with_key() + + scope = Scope.new(%{account | grants: MapSet.new(["users:read"])}) + + assert Scope.allows?(scope, "users:read") + refute Scope.allows?(scope, "users:write") + end + + test "a repo connection is allowed nothing" do + refute Scope.allows?( + Scope.new(insert(:project_repo_connection)), + "projects:write" + ) + end + end + + describe "holds_role?/2 and operator?/1" do + test "a member holds their own role and no other", %{ + editor: editor, + live_project: project + } do + {:ok, scope} = Scope.fetch(editor, project) + + assert Scope.holds_role?(scope, [:editor, :admin]) + refute Scope.holds_role?(scope, [:owner, :admin]) + refute Scope.operator?(scope) + end + + test "an operator holds every role", %{live_project: project} do + {account, _private_key} = + Lightning.ServiceAccountHelpers.service_account_with_key() + + {:ok, scope} = Scope.fetch(account, project) + + assert Scope.operator?(scope) + assert Scope.holds_role?(scope, [:owner]) + end + + test "a superuser gets no project role from their instance role" do + superuser = insert(:user, role: :superuser) + {:ok, scope} = Scope.fetch(superuser, insert(:project)) + + refute Scope.operator?(scope) + refute Scope.holds_role?(scope, [:owner, :admin, :editor, :viewer]) end end @@ -213,6 +371,15 @@ defmodule Lightning.Projects.ScopeTest do assert Scope.role_in?(user, project, [:owner, :admin]) end + test "is true for an operator on a live project, false on a scheduled one", + %{live_project: live_project, scheduled_deleted_project: scheduled} do + {account, _private_key} = + Lightning.ServiceAccountHelpers.service_account_with_key() + + assert Scope.role_in?(account, live_project, [:owner]) + refute Scope.role_in?(account, scheduled, [:owner]) + end + test "is false for a scheduled-deleted project", %{ editor: editor, scheduled_deleted_project: project @@ -245,21 +412,21 @@ defmodule Lightning.Projects.ScopeTest do assert scope.mfa_satisfied? == true end - test "is refused a scheduled-deleted project, same as a user", %{ + test "sees its own scheduled-deleted project as scheduled", %{ scheduled_deleted_project: project } do repo_connection = insert(:project_repo_connection, project: project) - assert {:error, :project_scheduled_for_deletion} = + assert {:ok, %Scope{scheduled_for_deletion?: true}} = Scope.fetch(repo_connection, project) end - test "cannot assert liveness with a hand-built project either", %{ + test "cannot hide that with a hand-built project either", %{ scheduled_deleted_project: project } do repo_connection = insert(:project_repo_connection, project: project) - assert {:error, :project_scheduled_for_deletion} = + assert {:ok, %Scope{scheduled_for_deletion?: true}} = Scope.fetch(repo_connection, %Project{id: project.id}) end @@ -270,43 +437,97 @@ defmodule Lightning.Projects.ScopeTest do Scope.fetch(repo_connection, Ecto.UUID.generate()) end - # A connection's standing is which single project it belongs to. Refusing - # here rather than at the call site is what stops a policy forgetting the - # comparison — the same reason a shut-down project yields no scope. - test "is refused a project it does not belong to", %{live_project: project} do + # A connection's standing is which single project it belongs to, so another + # project gets what a user with no membership row is refused with, which the + # API answers as a missing project. + test "has no standing on a project it does not belong to", %{ + live_project: project + } do other_project = insert(:project) repo_connection = insert(:project_repo_connection, project: other_project) - assert {:error, :connection_not_for_this_project} = + assert {:error, :no_standing} = Scope.fetch(repo_connection, project) end - test "is refused another project by id, and by anything carrying its id", %{ - live_project: project - } do + test "has no standing on another project by id, and by anything carrying its id", + %{ + live_project: project + } do other_project = insert(:project) repo_connection = insert(:project_repo_connection, project: other_project) dataclip = insert(:dataclip, project: project) - assert {:error, :connection_not_for_this_project} = + assert {:error, :no_standing} = Scope.fetch(repo_connection, project.id) - assert {:error, :connection_not_for_this_project} = + assert {:error, :no_standing} = Scope.fetch(repo_connection, dataclip) end - # Ownership is checked before liveness so that a connection cannot learn - # whether an unrelated project is being wound down. If this assertion ever - # reads :project_scheduled_for_deletion, the `with` clauses in fetch/2 have - # been reordered and a third party's state is leaking to a stranger. + # A connection is refused someone else's project before it is told the + # project's state, so it can't learn whether that project is being wound + # down. test "learns nothing about the state of a project that is not its own", %{ scheduled_deleted_project: project } do other_project = insert(:project) repo_connection = insert(:project_repo_connection, project: other_project) - assert {:error, :connection_not_for_this_project} = + assert {:error, :no_standing} = Scope.fetch(repo_connection, project) end end + + describe "a service account as the actor" do + setup do + {account, _private_key} = + Lightning.ServiceAccountHelpers.service_account_with_key() + + %{account: account} + end + + test "resolves a live project by project, id, or membership row, with no role", + %{account: account, live_project: project} do + project_user = + hd(Lightning.Repo.preload(project, :project_users).project_users) + + for subject <- [project, project.id, project_user] do + assert {:ok, scope} = Scope.fetch(account, subject) + assert scope.project.id == project.id + assert scope.actor == account + assert scope.instance_role == :operator + assert scope.role == nil + assert scope.project_user == nil + assert scope.support? == false + end + end + + # A machine credential cannot enrol, so the requirement does not apply. + test "meets the MFA requirement of a project that has one", %{ + account: account + } do + project = insert(:project, requires_mfa: true) + + assert {:ok, scope} = Scope.fetch(account, project) + assert scope.mfa_satisfied? == true + end + + test "sees a scheduled-deleted project as scheduled, same as a user", %{ + account: account, + scheduled_deleted_project: project + } do + assert {:ok, %Scope{scheduled_for_deletion?: true}} = + Scope.fetch(account, %Project{id: project.id}) + end + + test "gets no such project for an unknown or malformed id", %{ + account: account + } do + assert {:error, :no_such_project} = + Scope.fetch(account, Ecto.UUID.generate()) + + assert {:error, :no_such_project} = Scope.fetch(account, "not-a-uuid") + end + end end diff --git a/test/lightning/setup_utils_test.exs b/test/lightning/setup_utils_test.exs index f0145509612..01b0e04ab23 100644 --- a/test/lightning/setup_utils_test.exs +++ b/test/lightning/setup_utils_test.exs @@ -66,10 +66,10 @@ defmodule Lightning.SetupUtilsTest do assert viewer.email == "viewer@openfn.org" User.valid_password?(viewer, "welcome12345") - assert Enum.map( - openhie_project.project_users, - fn project_user -> project_user.user_id end - ) == [super_user.id, admin.id, editor.id, viewer.id] + assert openhie_project.project_users + |> Enum.map(fn project_user -> project_user.user_id end) + |> Enum.sort() == + Enum.sort([super_user.id, admin.id, editor.id, viewer.id]) assert Enum.map( dhis2_project.project_users, diff --git a/test/lightning/utils/validators_test.exs b/test/lightning/utils/validators_test.exs index 6cd3e847e1b..28e5e45c355 100644 --- a/test/lightning/utils/validators_test.exs +++ b/test/lightning/utils/validators_test.exs @@ -30,7 +30,7 @@ defmodule Lightning.ValidatorsTest do test "rejects a 16-byte non-hex string (the dump/cast asymmetry)" do cs = changeset("__ID_JOB_Fetch__") refute cs.valid? - assert cs.errors[:ref_id] == {"is not a valid UUID", []} + assert cs.errors[:ref_id] == {"is not a valid UUID", [code: :format]} end test "accepts a canonical UUID" do diff --git a/test/lightning/webhook_auth_methods_test.exs b/test/lightning/webhook_auth_methods_test.exs index 576ecfaf0a4..0c5e0601784 100644 --- a/test/lightning/webhook_auth_methods_test.exs +++ b/test/lightning/webhook_auth_methods_test.exs @@ -490,7 +490,12 @@ defmodule Lightning.WebhookAuthMethodsTest do actor: user ) - assert {:error, _changeset} = result + assert {:error, changeset} = result + + assert changeset.errors == [ + scheduled_deletion: + {"already scheduled for deletion", [code: :already_scheduled]} + ] final_audit_entries_count = Lightning.Auditing.list_all() diff --git a/test/lightning/work_orders_test.exs b/test/lightning/work_orders_test.exs index 63241d77c5d..a3e7379ca87 100644 --- a/test/lightning/work_orders_test.exs +++ b/test/lightning/work_orders_test.exs @@ -482,7 +482,8 @@ defmodule Lightning.WorkOrdersTest do assert changeset.errors == [ {:input_dataclip_id, - {"cannot retry run using a wiped dataclip", []}} + {"cannot retry run using a wiped dataclip", + [code: :dataclip_wiped]}} ] end @@ -527,7 +528,8 @@ defmodule Lightning.WorkOrdersTest do assert changeset.errors == [ {:input_dataclip_id, - {"cannot retry run using a wiped dataclip", []}} + {"cannot retry run using a wiped dataclip", + [code: :dataclip_wiped]}} ] end @@ -3329,6 +3331,62 @@ defmodule Lightning.WorkOrdersTest do end end + describe "paginate_work_orders_for/3" do + setup do + user = insert(:user) + project = insert(:project, project_users: [%{user_id: user.id}]) + workflow = insert(:simple_workflow, project: project) + + [failed, success] = + for state <- [:failed, :success] do + insert(:workorder, + workflow: workflow, + state: state, + dataclip: build(:dataclip) + ) + end + + insert(:workorder, state: :failed, dataclip: build(:dataclip)) + + %{user: user, project: project, failed: failed, success: success} + end + + test "narrows a project's work orders by the filters", %{ + project: project, + failed: failed + } do + assert %Scrivener.Page{entries: [%{id: id}], total_entries: 1} = + WorkOrders.paginate_work_orders_for( + project, + %{state: ["failed"]}, + [] + ) + + assert id == failed.id + end + + test "pages the work orders of every project a user may access", %{ + user: user, + failed: failed, + success: success + } do + page = + WorkOrders.paginate_work_orders_for(user, %{}, page: 2, page_size: 1) + + assert %Scrivener.Page{ + page_number: 2, + page_size: 1, + total_entries: 2, + total_pages: 2 + } = page + + assert MapSet.new( + WorkOrders.paginate_work_orders_for(user, %{}, []).entries, + & &1.id + ) == MapSet.new([failed.id, success.id]) + end + end + describe "cancel_many_async/2" do test "enqueues an Oban job to cancel work orders in the background" do trigger = build(:trigger, type: :webhook) diff --git a/test/lightning/workflows/edge_test.exs b/test/lightning/workflows/edge_test.exs index 76aec97a7b5..577fe65d8a9 100644 --- a/test/lightning/workflows/edge_test.exs +++ b/test/lightning/workflows/edge_test.exs @@ -204,7 +204,9 @@ defmodule Lightning.Workflows.EdgeTest do }) refute changeset.valid? - assert changeset.errors[:source_job_id] == {"is not a valid UUID", []} + + assert changeset.errors[:source_job_id] == + {"is not a valid UUID", [code: :format]} end test "a malformed workflow_id is a changeset error, not an Ecto.ChangeError on save" do @@ -217,7 +219,9 @@ defmodule Lightning.Workflows.EdgeTest do }) refute changeset.valid? - assert changeset.errors[:workflow_id] == {"is not a valid UUID", []} + + assert changeset.errors[:workflow_id] == + {"is not a valid UUID", [code: :format]} end test "edges must have a condition" do @@ -245,7 +249,8 @@ defmodule Lightning.Workflows.EdgeTest do assert changeset.errors == [ source_job_id: - {"source_job_id or source_trigger_id must be present", []} + {"source_job_id or source_trigger_id must be present", + [code: :required]} ] end @@ -275,7 +280,8 @@ defmodule Lightning.Workflows.EdgeTest do refute changeset.valid? assert {:source_job_id, - {"source_job_id and source_trigger_id are mutually exclusive", []}} in changeset.errors, + {"source_job_id and source_trigger_id are mutually exclusive", + [code: :exclusive]}} in changeset.errors, "error on the first change in the case both are set" changeset = @@ -288,7 +294,8 @@ defmodule Lightning.Workflows.EdgeTest do assert { :source_trigger_id, - {"source_job_id and source_trigger_id are mutually exclusive", []} + {"source_job_id and source_trigger_id are mutually exclusive", + [code: :exclusive]} } in changeset.errors end @@ -307,7 +314,8 @@ defmodule Lightning.Workflows.EdgeTest do assert { :target_job_id, - {"target_job_id must be different from source_job_id", []} + {"target_job_id must be different from source_job_id", + [code: :cycle]} } in changeset.errors changeset = diff --git a/test/lightning/workflows/graph_rules_test.exs b/test/lightning/workflows/graph_rules_test.exs new file mode 100644 index 00000000000..61bff8f6fac --- /dev/null +++ b/test/lightning/workflows/graph_rules_test.exs @@ -0,0 +1,344 @@ +defmodule Lightning.Workflows.GraphRulesTest do + use Lightning.DataCase, async: true + + alias Lightning.Workflows + alias Lightning.Workflows.GraphRules + alias Lightning.Workflows.Workflow + + setup do + %{project: insert(:project)} + end + + defp new_workflow(project, params) do + Workflow.changeset( + %Workflow{}, + Map.merge(%{"name" => "graph rules", "project_id" => project.id}, params) + ) + end + + defp change_stored(workflow, params) do + workflow + |> Repo.preload([:jobs, :triggers, :edges], force: true) + |> Workflows.change_workflow(params) + end + + # The errors on the workflow itself, which are the ones on a list as a whole. + defp list_errors(changeset), + do: changeset |> GraphRules.validate() |> Map.fetch!(:errors) |> Enum.sort() + + # The errors on each of `relation`'s children, each with the params of the + # request item it came from, so a rule is held to the item at fault whatever + # order Ecto keeps its list in. + defp item_errors(changeset, relation) do + for child <- Map.get(GraphRules.validate(changeset).changes, relation, []), + child.errors != [], + do: {child.params, Enum.sort(child.errors)} + end + + defp cycle, do: [target_job_id: {"is part of a cycle", [code: :cycle]}] + + defp no_trigger_edge, + do: + {:edges, + {"needs an enabled edge from a trigger", [code: :missing_trigger_edge]}} + + defp job(id), do: %{"id" => id, "name" => "job #{id}", "body" => "fn(s => s)"} + + defp trigger(id, enabled \\ true), + do: %{"id" => id, "type" => "webhook", "enabled" => enabled} + + defp edge({:trigger, from}, to), + do: %{ + "source_trigger_id" => from, + "target_job_id" => to, + "condition_type" => "always" + } + + defp edge(from, to), + do: %{ + "source_job_id" => from, + "target_job_id" => to, + "condition_type" => "on_job_success" + } + + defp ids(count), do: for(_ <- 1..count, do: Ecto.UUID.generate()) + + test "a workflow that keeps every rule passes", %{project: project} do + [t, a, b] = ids(3) + + changeset = + new_workflow(project, %{ + "triggers" => [trigger(t)], + "jobs" => [job(a), job(b)], + "edges" => [edge({:trigger, t}, a), edge(a, b)] + }) + + assert GraphRules.validate(changeset).valid? + end + + describe "cycles" do + test "a cycle through several edges is reported at the edge that closes it", + %{project: project} do + [t, a, b, c] = ids(4) + + changeset = + new_workflow(project, %{ + "triggers" => [trigger(t)], + "jobs" => [job(a), job(b), job(c)], + "edges" => [ + edge({:trigger, t}, a), + edge(a, b), + edge(b, c), + edge(c, a) + ] + }) + + assert item_errors(changeset, :edges) == [{edge(c, a), cycle()}] + assert list_errors(changeset) == [] + refute GraphRules.validate(changeset).valid? + end + + test "the edge's position is the request's, not the stored list's" do + workflow = insert(:simple_workflow) + [trigger] = workflow.triggers + [stored_job] = workflow.jobs + [k] = ids(1) + + # The stored edge is left out, so Ecto puts its removal ahead of the + # request's edges in the changeset. + changeset = + change_stored(workflow, %{ + "jobs" => [job(stored_job.id), job(k)], + "edges" => [ + edge({:trigger, trigger.id}, k), + edge(k, stored_job.id), + edge(stored_job.id, k) + ] + }) + + assert item_errors(changeset, :edges) == [ + {edge(stored_job.id, k), cycle()} + ] + + assert list_errors(changeset) == [] + end + + test "two separate cycles are each reported", %{project: project} do + [t, a, b, c, d] = ids(5) + + changeset = + new_workflow(project, %{ + "triggers" => [trigger(t)], + "jobs" => [job(a), job(b), job(c), job(d)], + "edges" => [ + edge({:trigger, t}, a), + edge(a, b), + edge(b, a), + edge(c, d), + edge(d, c) + ] + }) + + assert item_errors(changeset, :edges) == [ + {edge(b, a), cycle()}, + {edge(d, c), cycle()} + ] + end + + test "an edge from a job to itself is reported once, by the edge", + %{project: project} do + [t, a] = ids(2) + + changeset = + new_workflow(project, %{ + "triggers" => [trigger(t)], + "jobs" => [job(a)], + "edges" => [edge({:trigger, t}, a), edge(a, a)] + }) + + assert item_errors(changeset, :edges) == [ + {edge(a, a), + [ + target_job_id: + {"target_job_id must be different from source_job_id", + [code: :cycle]} + ]} + ] + + assert list_errors(changeset) == [] + end + + test "a stored cycle the request leaves alone is reported at the list" do + workflow = insert(:simple_workflow) + [trigger] = workflow.triggers + [a] = workflow.jobs + b = insert(:job, workflow: workflow) + insert(:edge, workflow: workflow, source_job_id: a.id, target_job_id: b.id) + insert(:edge, workflow: workflow, source_job_id: b.id, target_job_id: a.id) + + assert trigger.enabled + + changeset = change_stored(workflow, %{"name" => "renamed"}) + + assert list_errors(changeset) == [ + edges: {"is part of a cycle", [code: :cycle]} + ] + + assert item_errors(changeset, :edges) == [] + end + end + + describe "edges from triggers" do + test "a workflow with no edge from a trigger", %{project: project} do + [t, a, b] = ids(3) + + changeset = + new_workflow(project, %{ + "triggers" => [trigger(t)], + "jobs" => [job(a), job(b)], + "edges" => [edge(a, b)] + }) + + assert list_errors(changeset) == [no_trigger_edge()] + end + + test "a workflow with no jobs, triggers or edges", %{project: project} do + assert list_errors(new_workflow(project, %{})) == [no_trigger_edge()] + end + + test "edges from two triggers, both enabled", %{project: project} do + [t1, t2, a, b] = ids(4) + + changeset = + new_workflow(project, %{ + "triggers" => [trigger(t1), trigger(t2, false)], + "jobs" => [job(a), job(b)], + "edges" => [edge({:trigger, t1}, a), edge({:trigger, t2}, b)] + }) + + assert list_errors(changeset) == [ + {:edges, + {"only one edge from a trigger may be enabled", + [code: :one_enabled_edge]}} + ] + end + + test "a trigger with two targets", %{project: project} do + [t, a, b] = ids(3) + + changeset = + new_workflow(project, %{ + "triggers" => [trigger(t)], + "jobs" => [job(a), job(b)], + "edges" => [edge({:trigger, t}, a), edge({:trigger, t}, b)] + }) + + assert list_errors(changeset) == [ + {:edges, + {"a trigger must have a single target", [code: :single_target]}}, + {:edges, + {"only one edge from a trigger may be enabled", + [code: :one_enabled_edge]}} + ] + end + + test "an edge naming a trigger as its source job counts as the trigger's", + %{project: project} do + [t, a, b] = ids(3) + + changeset = + new_workflow(project, %{ + "triggers" => [trigger(t)], + "jobs" => [job(a), job(b)], + "edges" => [edge({:trigger, t}, a), edge(t, b)] + }) + + assert list_errors(changeset) == [ + {:edges, + {"a trigger must have a single target", [code: :single_target]}} + ] + end + end + + describe "triggers" do + test "two enabled triggers", %{project: project} do + [t1, t2, a] = ids(3) + + changeset = + new_workflow(project, %{ + "triggers" => [trigger(t1), trigger(t2)], + "jobs" => [job(a)], + "edges" => [edge({:trigger, t1}, a)] + }) + + assert list_errors(changeset) == [ + triggers: + {"only one trigger may be enabled", [code: :one_enabled]} + ] + end + + test "a request that leaves a stored trigger out" do + workflow = insert(:simple_workflow) + [stored_job] = workflow.jobs + [t] = ids(1) + + changeset = + change_stored(workflow, %{ + "triggers" => [trigger(t, false)], + "edges" => [edge({:trigger, t}, stored_job.id)] + }) + + assert list_errors(changeset) == [ + triggers: + {"a trigger cannot be removed, only edited or added", + [code: :trigger_removed]} + ] + end + + # The rule is checked only on the triggers a request sends, so a workflow + # the editor saved with two enabled triggers can still be renamed. + test "a request that leaves the triggers alone isn't checked for them" do + workflow = insert(:simple_workflow) + insert(:trigger, workflow: workflow, enabled: true) + + changeset = change_stored(workflow, %{"name" => "renamed"}) + + assert list_errors(changeset) == [] + assert item_errors(changeset, :triggers) == [] + end + end + + test "a job sent without its id", %{project: project} do + [t, a] = ids(2) + + changeset = + new_workflow(project, %{ + "triggers" => [trigger(t)], + "jobs" => [job(a), %{"name" => "no id", "body" => "fn(s => s)"}], + "edges" => [edge({:trigger, t}, a)] + }) + + assert item_errors(changeset, :jobs) == [ + {%{"name" => "no id", "body" => "fn(s => s)"}, + [id: {"can't be blank", [validation: :required]}]} + ] + end + + test "a trigger sent without its id", %{project: project} do + [a] = ids(1) + + changeset = + new_workflow(project, %{ + "triggers" => [%{"type" => "webhook", "enabled" => true}], + "jobs" => [job(a)], + "edges" => [] + }) + + assert list_errors(changeset) == [no_trigger_edge()] + + assert item_errors(changeset, :triggers) == [ + {%{"type" => "webhook", "enabled" => true}, + [id: {"can't be blank", [validation: :required]}]} + ] + end +end diff --git a/test/lightning/workflows/job_test.exs b/test/lightning/workflows/job_test.exs index 3965b114ad7..3d6d180ffe1 100644 --- a/test/lightning/workflows/job_test.exs +++ b/test/lightning/workflows/job_test.exs @@ -33,7 +33,7 @@ defmodule Lightning.Workflows.JobTest do }) refute changeset.valid? - assert changeset.errors[:id] == {"is not a valid UUID", []} + assert changeset.errors[:id] == {"is not a valid UUID", [code: :format]} end test "malformed FK ids are changeset errors, not Ecto.ChangeError on save" do @@ -49,10 +49,12 @@ defmodule Lightning.Workflows.JobTest do }) refute changeset.valid? - assert changeset.errors[:workflow_id] == {"is not a valid UUID", []} + + assert changeset.errors[:workflow_id] == + {"is not a valid UUID", [code: :format]} assert changeset.errors[:keychain_credential_id] == - {"is not a valid UUID", []} + {"is not a valid UUID", [code: :format]} # project_credential_id in isolation (no keychain set). project_changeset = @@ -66,7 +68,7 @@ defmodule Lightning.Workflows.JobTest do refute project_changeset.valid? assert project_changeset.errors[:project_credential_id] == - {"is not a valid UUID", []} + {"is not a valid UUID", [code: :format]} end test "FKs left unset stay valid" do @@ -125,7 +127,8 @@ defmodule Lightning.Workflows.JobTest do # The validate_exclusive function adds the error to the field that was changed assert changeset.errors[:project_credential_id] == - {"cannot be set when the other credential type is also set", []} + {"cannot be set when the other credential type is also set", + [code: :exclusive]} end test "validates that keychain credential belongs to the same project as the job" do @@ -150,7 +153,8 @@ defmodule Lightning.Workflows.JobTest do refute changeset.valid? assert changeset.errors[:keychain_credential_id] == - {"must belong to the same project as the job", []} + {"must belong to the same project as the job", + [code: :credential_not_allowed]} end test "allows both credential fields to be null" do @@ -589,4 +593,39 @@ defmodule Lightning.Workflows.JobTest do end) end end + + describe "changeset/2 while the catalogue's first load is running" do + setup :isolated_adaptors + + test "refuses the adaptor as not ready instead of waiting for the load", %{ + sup: sup + } do + {:global, scheduler} = + Lightning.Adaptors.Supervisor.global_scheduler_name(sup) + + scheduler = :global.whereis_name(scheduler) + Mox.allow(Lightning.Adaptors.StrategyMock, self(), scheduler) + Ecto.Adapters.SQL.Sandbox.allow(Repo, self(), scheduler) + + Mox.stub(Lightning.Adaptors.StrategyMock, :list_adaptors, fn -> + Process.sleep(2_000) + {:ok, []} + end) + + Mox.stub(Lightning.Adaptors.StrategyMock, :fetch_icons, fn _opts -> + {:ok, %{}} + end) + + changeset = + Job.changeset(%Job{}, %{ + name: "job", + body: "fn(state => state)", + adaptor: "@openfn/language-http@1.0.0" + }) + + assert errors_on(changeset).adaptor == [ + "adaptor catalogue is not ready yet, try again shortly" + ] + end + end end diff --git a/test/lightning/workflows/trigger_test.exs b/test/lightning/workflows/trigger_test.exs index 7f305ee3f21..9ec2b101440 100644 --- a/test/lightning/workflows/trigger_test.exs +++ b/test/lightning/workflows/trigger_test.exs @@ -125,7 +125,9 @@ defmodule Lightning.Workflows.TriggerTest do }) refute changeset.valid? - assert changeset.errors[:workflow_id] == {"is not a valid UUID", []} + + assert changeset.errors[:workflow_id] == + {"is not a valid UUID", [code: :format]} end test "must raise an error when cron expression is invalid" do @@ -240,7 +242,7 @@ defmodule Lightning.Workflows.TriggerTest do refute changeset.valid? assert changeset.errors[:cron_cursor_job_id] == - {"is not a valid UUID", []} + {"is not a valid UUID", [code: :format]} end test "cron_cursor_job_id pointing at a non-existent job is a changeset error, not a raise" do diff --git a/test/lightning/workflows/webhook_lookup_queries_test.exs b/test/lightning/workflows/webhook_lookup_queries_test.exs index 2c38405b7ef..96eaf712c10 100644 --- a/test/lightning/workflows/webhook_lookup_queries_test.exs +++ b/test/lightning/workflows/webhook_lookup_queries_test.exs @@ -8,6 +8,7 @@ defmodule Lightning.Workflows.WebhookLookupQueriesTest do use Lightning.DataCase, async: false import Lightning.Factories + import Lightning.QueryCountHelpers alias Lightning.Workflows alias Lightning.Workflows.Trigger @@ -25,48 +26,26 @@ defmodule Lightning.Workflows.WebhookLookupQueriesTest do {project, trigger} end - defp count_queries(fun) do - ref = make_ref() - parent = self() - name = "count-#{System.unique_integer([:positive])}" - - :telemetry.attach( - name, - [:lightning, :repo, :query], - fn _event, _measures, meta, _config -> - if meta[:source] == "triggers", do: send(parent, {ref, :query}) - end, - nil - ) - - fun.() - :telemetry.detach(name) - - drain(ref, 0) - end - - defp drain(ref, acc) do - receive do - {^ref, :query} -> drain(ref, acc + 1) - after - 0 -> acc - end - end - test "the namespaced URL is one query" do {project, _trigger} = webhook_trigger(custom_path: "facility-001") - assert count_queries(fn -> - Workflows.get_webhook_trigger([project.id, "facility-001"]) - end) == 1 + assert {_, 1} = + count_queries( + fn -> + Workflows.get_webhook_trigger([project.id, "facility-001"]) + end, + "triggers" + ) end test "the generated URL is one query" do {_project, trigger} = webhook_trigger() - assert count_queries(fn -> - Workflows.get_webhook_trigger([trigger.id]) - end) == 1 + assert {_, 1} = + count_queries( + fn -> Workflows.get_webhook_trigger([trigger.id]) end, + "triggers" + ) end test "the generated URL with a trailing segment costs one more" do @@ -74,15 +53,21 @@ defmodule Lightning.Workflows.WebhookLookupQueriesTest do # being tried first, which is the shape this feature creates. {_project, trigger} = webhook_trigger() - assert count_queries(fn -> - Workflows.get_webhook_trigger([trigger.id, "Patient"]) - end) == 2 + assert {_, 2} = + count_queries( + fn -> Workflows.get_webhook_trigger([trigger.id, "Patient"]) end, + "triggers" + ) end test "a miss costs no more than three" do - assert count_queries(fn -> - Workflows.get_webhook_trigger([Ecto.UUID.generate(), "nope"]) - end) <= 3 + {_, queries} = + count_queries( + fn -> Workflows.get_webhook_trigger([Ecto.UUID.generate(), "nope"]) end, + "triggers" + ) + + assert queries <= 3 assert Trigger end diff --git a/test/lightning/workflows_test.exs b/test/lightning/workflows_test.exs index 16ee52892eb..3e986c87184 100644 --- a/test/lightning/workflows_test.exs +++ b/test/lightning/workflows_test.exs @@ -1305,11 +1305,11 @@ defmodule Lightning.WorkflowsTest do assert updated_workflow.lock_version == 2 - # Throws an error because the lock_version is outdated - assert_raise Ecto.StaleEntryError, fn -> - Workflows.change_workflow(workflow, %{jobs: [params_for(:job)]}) - |> Workflows.save_workflow(user) - end + assert {:error, :concurrent_edit} = + Workflows.change_workflow(workflow, %{jobs: [params_for(:job)]}) + |> Workflows.save_workflow(user) + + assert Repo.reload(workflow).lock_version == 2 end test "change_workflow/1 returns a workflow changeset" do @@ -1679,24 +1679,6 @@ defmodule Lightning.WorkflowsTest do refute cs.valid? assert {"contains an invalid value", _} = cs.errors[:base] end - - test "stale lock still raises Ecto.StaleEntryError (not rescued)" do - # Guards that the rescue does NOT swallow optimistic-lock conflicts - # (mirrors the "saving with locks" test above). - user = insert(:user) - valid_attrs = params_with_assocs(:workflow, jobs: [params_for(:job)]) - - {:ok, workflow} = Workflows.save_workflow(valid_attrs, insert(:user)) - - {:ok, _} = - Workflows.change_workflow(workflow, %{jobs: [params_for(:job)]}) - |> Workflows.save_workflow(user) - - assert_raise Ecto.StaleEntryError, fn -> - Workflows.change_workflow(workflow, %{jobs: [params_for(:job)]}) - |> Workflows.save_workflow(user) - end - end end describe "save_workflow/3 credential project scoping" do @@ -2594,6 +2576,44 @@ defmodule Lightning.WorkflowsTest do end end + describe "list_workflows_for_user/2" do + test "lists the live workflows in the user's projects, newest first" do + user = insert(:user) + project = insert(:project, project_users: [%{user: user}]) + + older = + insert(:workflow, + project: project, + inserted_at: ~U[2024-01-01 00:00:00Z] + ) + + newer = + insert(:workflow, + project: project, + inserted_at: ~U[2024-02-01 00:00:00Z] + ) + + insert(:workflow, project: project, deleted_at: DateTime.utc_now()) + insert(:workflow) + + assert Workflows.list_workflows_for_user(user) |> Enum.map(& &1.id) == + [newer.id, older.id] + end + + test "preloads the associations named in include:" do + user = insert(:user) + project = insert(:project, project_users: [%{user: user}]) + workflow = insert(:workflow, project: project) + job = insert(:job, workflow: workflow) + + assert [%{jobs: [%{id: job_id}]} = listed] = + Workflows.list_workflows_for_user(user, include: [:jobs]) + + assert job_id == job.id + refute Ecto.assoc_loaded?(listed.edges) + end + end + describe "project_workflows_using_credentials/1" do test "returns each project's workflow names sorted alphabetically" do project = insert(:project) diff --git a/test/lightning_web/api/before_the_router_test.exs b/test/lightning_web/api/before_the_router_test.exs new file mode 100644 index 00000000000..58ebcfc7921 --- /dev/null +++ b/test/lightning_web/api/before_the_router_test.exs @@ -0,0 +1,396 @@ +defmodule LightningWeb.API.BeforeTheRouterTest do + @moduledoc """ + The answers Lightning gives before a controller runs, on each JSON pipeline + (error-format spec §4, §5, §9.3, §9.5). + """ + use LightningWeb.ConnCase, async: true + use Mimic + + import Lightning.Factories + import Lightning.ServiceAccountHelpers + import LightningWeb.API.ProblemHelpers + + alias Lightning.Extensions.MockRateLimiter + alias Lightning.Projects.Provisioner + alias Lightning.ServiceAccount.AccessToken + alias LightningWeb.API.Codes + + # One route per pipeline that takes a body: access_token_api, + # person_or_service_account_api, api, authenticated_json (no route there + # takes a body, but the endpoint answers before routing) and + # authenticated_api. + @body_routes [ + {:post, "/api/users"}, + {:put, "/api/projects/#{Ecto.UUID.generate()}"}, + {:post, "/api/provision"}, + {:post, "/api/ai_assistant/sessions"}, + {:post, "/collections/a-collection"} + ] + + # authenticated_json has no route that takes a body, so nothing there can + # reach the 415. + @parsed_routes List.delete_at(@body_routes, 3) + + defp send_body(conn, {method, path}, content_type, body) do + conn + |> put_req_header("content-type", content_type) + |> dispatch(@endpoint, method, path, body) + end + + defp too_large_body do + limit = Application.fetch_env!(:lightning, :max_dataclip_size_bytes) + ~s(") <> String.duplicate("a", max(limit, 10_000_000) + 1) <> ~s(") + end + + defp assert_problem_sent({status, headers, body}, code, media_type) do + assert status == Codes.status(code) + + assert {"content-type", "#{media_type}; charset=utf-8"} in headers + + assert %{"status" => ^status, "code" => code_name, "detail" => detail} = + Jason.decode!(body) + + assert code_name == Atom.to_string(code) + assert detail == Codes.detail(code) + end + + describe "a body that isn't JSON" do + test "is 400 malformed on every pipeline", %{conn: conn} do + for route <- @body_routes do + conn + |> send_body(route, "application/json", ~s({"name": )) + |> assert_problem(400, :malformed, []) + end + end + + test "follows Accept", %{conn: conn} do + conn + |> put_req_header("accept", "application/problem+json") + |> send_body(hd(@body_routes), "application/json", "{") + |> assert_problem(400, :malformed, []) + end + end + + describe "a request the parsers can't decode" do + test "is 400 bad_request for a query string, whatever Accept asks", %{ + conn: conn + } do + conn + |> get("/api/runs?a=%FF") + |> assert_problem(400, :bad_request, []) + + conn + |> put_req_header("accept", "application/json") + |> get("/api/runs?a=%FF") + |> assert_problem(400, :bad_request, []) + end + + test "is 414 uri_too_long for a query string over Plug's limit", %{ + conn: conn + } do + conn + |> get("/api/runs?a=" <> String.duplicate("a", 1_000_000)) + |> assert_problem(414, :uri_too_long, []) + end + + test "is 400 bad_request for a form body", %{conn: conn} do + conn + |> send_body( + hd(@body_routes), + "application/x-www-form-urlencoded", + "a=%FF" + ) + |> assert_problem(400, :bad_request, []) + end + end + + describe "a body over the size limit" do + test "is 413 content_too_large on every pipeline", %{conn: conn} do + body = too_large_body() + + for route <- @body_routes do + conn + |> send_body(route, "application/json", body) + |> assert_problem(413, :content_too_large, []) + end + end + end + + describe "an error raised before the router, from an allowed Origin" do + setup %{conn: conn} do + Mox.stub(Lightning.MockConfig, :cors_origin, fn -> + ["https://app.example.com"] + end) + + %{conn: put_req_header(conn, "origin", "https://app.example.com")} + end + + defp assert_cors_and_request_id(conn) do + assert get_resp_header(conn, "access-control-allow-origin") == [ + "https://app.example.com" + ] + + assert [_request_id] = get_resp_header(conn, "x-request-id") + end + + test "an answer from the router carries CORS and a request id", %{ + conn: conn + } do + conn + |> get("/api/nothing-here") + |> tap(&assert_problem(&1, 404, :not_found, [])) + |> assert_cors_and_request_id() + end + + test "still carries CORS and a request id on a 400 or 413", %{conn: conn} do + for path <- ["/api/users", "/collections/a-collection"], + {body, status, code} <- [ + {"{", 400, :malformed}, + {too_large_body(), 413, :content_too_large} + ] do + conn + |> send_body({:post, path}, "application/json", body) + |> tap(&assert_problem(&1, status, code, [])) + |> assert_cors_and_request_id() + end + end + + test "answers a preflight that raises with the error, not a 204", %{ + conn: conn + } do + # The bad query string raises in LiveDashboard's RequestLogger, which runs + # ahead of CORSPlug, so the preflight never reaches CORSPlug's 204. + conn + |> put_req_header("access-control-request-method", "GET") + |> options("/api/runs?a=%FF") + |> tap(&assert_problem(&1, 400, :bad_request, [])) + |> assert_cors_and_request_id() + end + + test "answers with the request id the request's log lines carry", %{ + conn: conn + } do + test_pid = self() + + # CORSPlug reads the allowed origins after Plug.RequestId has set the + # id, so this sees the request's logged id before anything raises. + Mox.stub(Lightning.MockConfig, :cors_origin, fn -> + send(test_pid, {:logged_request_id, Logger.metadata()[:request_id]}) + ["https://app.example.com"] + end) + + conn = send_body(conn, {:post, "/api/users"}, "application/json", "{") + + assert_received {:logged_request_id, logged_request_id} + assert is_binary(logged_request_id) + assert get_resp_header(conn, "x-request-id") == [logged_request_id] + end + + test "carries no CORS for an Origin that isn't allowed", %{conn: conn} do + for path <- ["/api/users", "/collections/a-collection"], + body <- ["{", too_large_body()] do + conn = + conn + |> put_req_header("origin", "https://elsewhere.example.com") + |> send_body({:post, path}, "application/json", body) + + assert get_resp_header(conn, "access-control-allow-origin") == [] + assert [_request_id] = get_resp_header(conn, "x-request-id") + end + end + end + + describe "a body no parser reads" do + test "is 415 unsupported_media_type on every pipeline", %{conn: conn} do + for route <- @parsed_routes do + conn + |> send_body(route, "text/plain", "hello") + |> assert_problem(415, :unsupported_media_type, []) + end + end + + test "is not refused when there is no body, whatever its content type", %{ + conn: conn + } do + for route <- @parsed_routes do + conn + |> send_body(route, "text/plain", "") + |> assert_problem(401, :unauthenticated, []) + end + end + + test "lets a DELETE with no body reach its controller", %{conn: conn} do + user = insert(:user) + project = insert(:project, project_users: [%{user: user}]) + + collection = + insert(:collection, + project: project, + items: [%{key: "foo", value: "bar"}] + ) + + conn = + conn + |> assign_bearer(user) + |> send_body( + {:delete, "/collections/#{collection.name}/foo"}, + "text/plain", + "" + ) + + assert %{"key" => "foo", "deleted" => 1} = json_response(conn, 200) + end + end + + describe "an unmatched route" do + test "is 404 not_found as JSON, not HTML, with no Accept header", %{ + conn: conn + } do + for path <- ["/api/nothing-here", "/api/users/a/b", "/collections/a/b/c"] do + conn = get(conn, path) + + assert_problem(conn, 404, :not_found, []) + refute conn.resp_body =~ "<" + end + end + + test "follows Accept, ahead of authentication", %{conn: conn} do + conn + |> put_req_header("accept", "application/problem+json") + |> delete("/collections/a/b/c") + |> assert_problem(404, :not_found, []) + end + + test "on the OAuth token endpoint stays the router's own 404", %{ + conn: conn + } do + conn = get(conn, "/api/oauth/token") + + assert conn.status == 404 + refute conn.resp_body =~ "not_found" + end + end + + describe "a path id that isn't a UUID" do + test "is 404 not_found on the service-account pipeline", %{conn: conn} do + {account, _private_key} = service_account_with_key() + Mox.stub(Lightning.MockConfig, :service_account, fn -> account end) + + conn + |> assign_bearer(AccessToken.issue(account, ["users:read"])) + |> get("/api/users/not-a-uuid") + |> assert_problem(404, :not_found, []) + end + + test "is 404 not_found on the person-or-service-account pipeline", %{ + conn: conn + } do + conn + |> assign_bearer(insert(:user)) + |> get("/api/projects/not-a-uuid") + |> assert_problem(404, :not_found, []) + end + + test "is 404 not_found where the controller casts the id", %{conn: conn} do + conn + |> assign_bearer(insert(:user)) + |> get("/api/jobs/not-a-uuid") + |> assert_problem(404, :not_found, []) + end + end + + describe "a raise on a problem path" do + setup %{conn: conn} do + stub(Provisioner, :import_document, fn _project, _actor, _document -> + raise "raised for a test" + end) + + %{conn: assign_bearer(conn, insert(:user, role: :superuser))} + end + + defp provision(conn), do: post(conn, "/api/provision", %{"name" => "a"}) + + test "is 500 internal, and still raises", %{conn: conn} do + assert_error_sent(500, fn -> + conn |> put_req_header("accept", "*/*") |> provision() + end) + |> assert_problem_sent(:internal, "application/json") + end + + test "follows Accept", %{conn: conn} do + assert_error_sent(500, fn -> + conn + |> put_req_header("accept", "application/problem+json") + |> provision() + end) + |> assert_problem_sent(:internal, "application/problem+json") + end + + test "answers a 406 with a problem document as JSON", %{conn: conn} do + conn = conn |> put_req_header("accept", "text/html") |> get("/api/users") + + assert_problem(conn, 406, :not_acceptable, []) + + assert ["application/json; charset=utf-8"] = + get_resp_header(conn, "content-type") + end + end + + describe "a path the format doesn't cover" do + test "keeps Phoenix's answer to a 500", %{conn: conn} do + Mox.stub(Lightning.MockConfig, :check_flag?, fn _flag -> + raise "raised for a test" + end) + + {500, headers, body} = + assert_error_sent(500, fn -> get(conn, "/users/log_in") end) + + assert {"content-type", "text/html; charset=utf-8"} in headers + refute body =~ "internal" + end + + test "keeps ErrorView's JSON 500 on a webhook with no Accept", %{ + conn: conn + } do + Mox.stub(MockRateLimiter, :limit_request, fn _conn, _context, _opts -> + raise "raised for a test" + end) + + %{triggers: [trigger]} = + insert(:simple_workflow) |> Lightning.Repo.preload(:triggers) + + {500, headers, body} = + assert_error_sent(500, fn -> post(conn, "/i/#{trigger.id}") end) + + assert {"content-type", "application/json; charset=utf-8"} in headers + assert Jason.decode!(body) == %{"error" => "Internal Server Error"} + end + + test "re-raises a body that isn't JSON", %{conn: conn} do + assert_raise Plug.Parsers.ParseError, fn -> + send_body(conn, {:post, "/users/log_in"}, "application/json", "{") + end + end + + test "re-raises a query string it can't decode", %{conn: conn} do + {400, headers, _body} = + assert_error_sent(400, fn -> get(conn, "/users/log_in?a=%FF") end) + + assert {"content-type", "text/html; charset=utf-8"} in headers + end + + test "keeps phoenix_ecto's 400 for a path id that isn't a UUID", %{ + conn: conn + } do + {400, headers, body} = + assert_error_sent(400, fn -> + conn + |> log_in_user(insert(:user)) + |> get("/project_files/not-a-uuid/download") + end) + + assert {"content-type", "text/html; charset=utf-8"} in headers + refute body =~ "not_found" + end + end +end diff --git a/test/lightning_web/api/codes_test.exs b/test/lightning_web/api/codes_test.exs new file mode 100644 index 00000000000..c15a9e5acb5 --- /dev/null +++ b/test/lightning_web/api/codes_test.exs @@ -0,0 +1,156 @@ +defmodule LightningWeb.API.CodesTest do + use ExUnit.Case, async: true + + alias LightningWeb.API.Codes + + @lib Path.expand("../../../lib", __DIR__) + + defp codes_set_in(dir) do + "#{@lib}/#{dir}/**/*.ex" + |> Path.wildcard() + |> Enum.flat_map(fn file -> + Regex.scan(~r/\bcode: :(\w+)/, File.read!(file), capture: :all_but_first) + end) + |> MapSet.new(fn [code] -> String.to_atom(code) end) + end + + test "every code lib/lightning sets with code: is in a code list" do + # `not_ready` is set but never sent as an entry (spec §6.2). + known = MapSet.new([:not_ready | Codes.entry_codes()]) + + assert MapSet.difference(codes_set_in("lightning"), known) == MapSet.new() + end + + test "every code Lightning is listed as setting is set somewhere in lib" do + # An entry code may be set in lightning_web, where an API controller + # validates the request itself. + set = MapSet.union(codes_set_in("lightning"), codes_set_in("lightning_web")) + + assert MapSet.difference(MapSet.new(Codes.lightning_entry_codes()), set) == + MapSet.new() + end + + test "every response code answers with its status and detail" do + expected = %{ + bad_request: {400, "The request could not be read."}, + invalid_request: {400, "The credentials could not be read."}, + malformed: {400, "The body is not a JSON object."}, + invalid_parameter: {400, "A query parameter is not valid. See errors."}, + invalid_header: {400, "A request header is not valid. See errors."}, + unauthenticated: {401, "Authentication is required."}, + invalid_token: {401, "The token is not valid."}, + exceeds_limit: {402, "This would exceed a limit of your plan."}, + forbidden: {403, "You may not do this."}, + insufficient_scope: {403, "The token lacks the scope this needs."}, + trigger_disabled: {403, "This trigger is disabled."}, + not_found: {404, "Not found, or you don't have access to it."}, + not_acceptable: + {406, "This resource cannot be returned in the requested format."}, + request_timeout: {408, "The request body did not arrive in time."}, + scheduled_for_deletion: + {409, "This is scheduled for deletion. Cancel the deletion first."}, + id_taken: {409, "The id is already in use."}, + taken: {409, "A value that must be unique is already in use. See errors."}, + being_edited: {409, "This workflow is open in the editor."}, + concurrent_edit: + {409, + "Another request changed or deleted this record. Fetch it again and retry."}, + in_use: {409, "This has been used by runs and cannot be deleted."}, + ambiguous_name: + {409, "More than one collection has this name. Add project_id."}, + export_conflict: + {409, "Two items in this project export under the same key."}, + content_too_large: {413, "The body is too large."}, + uri_too_long: {414, "The query string is too long."}, + unsupported_media_type: {415, "The body must be JSON."}, + invalid: {422, "The request is not valid. See errors."}, + too_many_runs: {429, "Too many runs. Try again later."}, + too_many_requests: {429, "Too many requests. Try again later."}, + internal: {500, "Something went wrong."}, + not_implemented: {501, "This is not available on this instance."}, + bad_gateway: {502, "The channel could not reach its destination."}, + service_unavailable: {503, "Try again shortly."}, + not_ready: {503, "This is not ready yet. Try again shortly."}, + timeout: {504, "The workflow did not finish in time."} + } + + actual = + Map.new( + Codes.response_codes(), + &{&1, {Codes.status(&1), Codes.detail(&1)}} + ) + + assert actual == expected + end + + describe "for_exception/2" do + test "every exception with a 4xx status has a code for that status" do + for exception <- client_exceptions() do + status = Plug.Exception.status(exception) + code = Codes.for_exception(:error, exception) + + assert code != :internal, + "#{inspect(exception)} answers #{status} with no code" + + assert Codes.status(code) == status, + "#{inspect(exception)} answers #{status} as #{code}" + end + end + + test "a raise a client did not cause is internal" do + for {kind, reason} <- [ + error: %Ecto.StaleEntryError{}, + error: %RuntimeError{}, + exit: :timeout, + throw: :oops + ] do + assert Codes.for_exception(kind, reason) == :internal + end + end + end + + describe "for_limiter/1" do + test "answers each reason Lightning names with its code" do + assert Codes.for_limiter(:too_many_runs) == :too_many_runs + assert Codes.for_limiter(:too_many_requests) == :too_many_requests + assert Codes.for_limiter(:unknown) == :too_many_requests + end + + test "answers any other reason with exceeds_limit, never a raise" do + for reason <- [:too_many_workflows, :disabled, :runs_cap_exceeded] do + assert Codes.for_limiter(reason) == :exceeds_limit + end + end + end + + # Every exception these apps give a 4xx status, through a `plug_status` + # field or a `Plug.Exception` implementation, so an upgrade that adds one + # fails here until it has a code. CastError and StaleEntryError are left + # out, as `for_exception/2` deliberately answers them by another code. + defp client_exceptions do + {:consolidated, implemented} = Plug.Exception.__protocol__(:impls) + + with_field = + [:lightning, :plug, :phoenix, :phoenix_live_view] + |> Enum.flat_map(&Application.spec(&1, :modules)) + |> Enum.filter(&plug_status_field?/1) + + defaults = + (implemented ++ with_field) + |> Enum.uniq() + |> Enum.reject(&(&1 in [Any, Ecto.Query.CastError, Ecto.StaleEntryError])) + |> Enum.map(&struct/1) + + others = [ + %Plug.Conn.InvalidQueryError{plug_status: 414}, + %Postgrex.Error{postgres: %{code: :character_not_in_repertoire}} + ] + + Enum.filter(defaults ++ others, &(Plug.Exception.status(&1) in 400..499)) + end + + defp plug_status_field?(module) do + Code.ensure_loaded?(module) and function_exported?(module, :exception, 1) and + Map.get(module.__struct__(), :plug_status) != nil + end +end diff --git a/test/lightning_web/api/errors_test.exs b/test/lightning_web/api/errors_test.exs new file mode 100644 index 00000000000..624def82764 --- /dev/null +++ b/test/lightning_web/api/errors_test.exs @@ -0,0 +1,331 @@ +defmodule LightningWeb.API.ErrorsTest do + use Lightning.DataCase, async: true + + import Lightning.AdaptorTestHelpers + import Lightning.Factories + import Mox, only: [stub: 3] + + alias Lightning.Accounts.User + alias Lightning.Extensions.MockUsageLimiter + alias Lightning.Projects.Project + alias Lightning.Projects.ProjectUser + alias Lightning.Projects.Provisioner + alias Lightning.Workflows.Job + alias Lightning.Workflows.Workflow + alias LightningWeb.API.Errors + alias LightningWeb.API.Problem + + @adaptor "@openfn/language-common@latest" + + setup :isolated_adaptors + + setup do + stub(MockUsageLimiter, :limit_action, fn _, _ -> :ok end) + :ok + end + + defp job(id, name) do + %{"id" => id, "name" => name, "adaptor" => @adaptor, "body" => "fn(s => s)"} + end + + defp provision(jobs) do + workflow = %{"id" => Ecto.UUID.generate(), "name" => "w", "jobs" => jobs} + %{"id" => Ecto.UUID.generate(), "name" => "p", "workflows" => [workflow]} + end + + @one_owner "Every project must have exactly one owner. Please specify one below." + + defp project_with(name \\ "members", roles) do + Project.project_with_users_changeset(%Project{}, %{ + "name" => name, + "project_users" => + for( + role <- roles, + do: %{"user_id" => insert(:user).id, "role" => "#{role}"} + ) + }) + end + + defp entry_at(pointer, code, detail), + do: %{pointer: pointer, code: code, detail: detail} + + defp decode_pointer("#" <> path) do + path + |> URI.decode() + |> String.split("/", trim: true) + |> Enum.map(fn key -> + key |> String.replace("~1", "/") |> String.replace("~0", "~") + end) + end + + describe "where an entry is placed" do + setup do + seed_ready_catalogue() + ensure_adaptor(@adaptor) + :ok + end + + test "counts a provisioned child's position among the items the request sent, with one marked delete: true mid-list" do + [kept, deleted, left_out, blank, last] = + for _ <- 1..5, do: Ecto.UUID.generate() + + document = + provision([ + job(kept, "kept"), + job(deleted, "deleted"), + job(left_out, "left out") + ]) + + {:ok, project} = + Provisioner.import_document(%Project{}, insert(:user), document) + + changeset = + Provisioner.parse_document( + project, + put_in(document, ["workflows", Access.at(0), "jobs"], [ + job(kept, "kept"), + %{"id" => deleted, "delete" => true}, + job(blank, ""), + job(last, "last") + ]) + ) + + assert Errors.from_changeset(changeset) == [ + entry_at( + "#/workflows/0/jobs/2/name", + :required, + "job name can't be blank" + ) + ] + end + + test "keeps a list's own error and its items' errors, under the names the request used" do + changeset = project_with([:editor, :boss]) + + assert Errors.from_changeset(changeset, %{ + project_users: :members, + owner: :members + }) == [ + entry_at("#/members", :one_owner, @one_owner), + entry_at("#/members/1/role", :not_allowed, "is invalid") + ] + end + + test "names the path segment for a top-level key the request sent in its path, and only there" do + changeset = project_with("", [:boss]) + + assert Errors.from_changeset(changeset, %{}, path: [:name, :role]) == [ + %{path: "name", code: :required, detail: "can't be blank"}, + entry_at("#/owner", :one_owner, @one_owner), + entry_at("#/project_users/0/role", :not_allowed, "is invalid") + ] + end + + test "points a duplicate at the later of two provisioned children with one id, naming the earlier" do + id = Ecto.UUID.generate() + document = provision([job(id, "first"), job(id, "second")]) + changeset = Provisioner.parse_document(%Project{}, document) + + assert Errors.from_changeset(changeset) == [ + entry_at( + "#/workflows/0/jobs/1/id", + :duplicate, + "names the same id as #/workflows/0/jobs/0" + ) + ] + end + + test "places an error at the keys and positions its key: option lists" do + changeset = + Ecto.Changeset.add_error( + Ecto.Changeset.change({%{}, %{bodies: {:array, :map}}}), + :bodies, + "can't be blank", + code: :required, + key: [:credential_bodies, 1, :body, "host"] + ) + + assert Errors.from_changeset(changeset) == [ + entry_at( + "#/credential_bodies/1/body/host", + :required, + "can't be blank" + ) + ] + end + end + + describe "which code an Ecto error gets" do + test "a value outside an enum's set is not_allowed, and a failed cast to any other type is type" do + user_id = Ecto.UUID.generate() + + for {changeset, entry} <- [ + {ProjectUser.changeset(%ProjectUser{}, %{ + "user_id" => user_id, + "role" => "boss" + }), entry_at("#/role", :not_allowed, "is invalid")}, + {Project.changeset(%Project{}, %{ + "name" => "enum", + "retention_policy" => "keep_forever" + }), entry_at("#/retention_policy", :not_allowed, "is invalid")}, + {Project.changeset(%Project{}, %{ + "name" => "type", + "concurrency" => "x" + }), entry_at("#/concurrency", :type, "is invalid")} + ] do + assert Errors.from_changeset(changeset) == [entry] + end + end + + test "codes an assoc_constraint error not_found at the foreign key the request sent" do + {:error, changeset} = + %Workflow{} + |> Workflow.changeset(%{ + name: "orphan", + project_id: Ecto.UUID.generate() + }) + |> Repo.insert() + + assert Errors.from_changeset(changeset) == [ + entry_at("#/project_id", :not_found, "does not exist") + ] + + assert [%{pointer: "#/parent"}] = + Errors.from_changeset(changeset, %{project: :parent}) + end + + test "raises on an error with no code" do + changeset = + Ecto.Changeset.add_error( + Ecto.Changeset.change({%{}, %{id: :binary_id}}), + :id, + "is not a UUID" + ) + + assert_raise ArgumentError, + ~r/"is not a UUID" at \[:id\] has no code/, + fn -> Errors.from_changeset(changeset) end + end + end + + describe "pointer/1" do + test "escapes a key holding / and ~ so a client decodes it back, and is # alone for the whole body" do + key = "a/b~c d%é" + pointer = Errors.pointer([:workflows, 0, key]) + + assert pointer == "#/workflows/0/a~1b~0c%20d%25%C3%A9" + assert decode_pointer(pointer) == ["workflows", "0", key] + assert Errors.pointer([]) == "#" + end + end + + describe "which status a refused changeset gets" do + test "answers 503 not_ready, with no entries, when the adaptor catalogue isn't loaded" do + changeset = + Job.changeset(%Job{}, %{ + name: "waiting", + adaptor: "@openfn/language-unseeded@1.0.0", + body: "fn(s => s)" + }) + + assert Errors.problem(changeset) == %Problem{ + status: 503, + code: :not_ready, + headers: [{"retry-after", "5"}] + } + end + + test "answers 500 internal, with no entries, when stored files couldn't be deleted" do + changeset = + Ecto.Changeset.add_error( + Ecto.Changeset.change(%Project{}), + :project_files, + "1 stored file(s) could not be deleted from storage", + code: :storage_failed + ) + + {problem, log} = + ExUnit.CaptureLog.with_log(fn -> Errors.problem(changeset) end) + + assert problem == %Problem{status: 500, code: :internal} + assert log =~ "could not delete stored files" + end + + test "answers 402 when every error is a plan limit" do + stub(MockUsageLimiter, :limit_action, fn + %{type: :new_collection}, _context -> + {:error, :exceeds_limit, %{text: "No collections left"}} + + _action, _context -> + :ok + end) + + changeset = + Provisioner.parse_document( + %Project{}, + Map.put(provision([]), "collections", [ + %{"id" => Ecto.UUID.generate(), "name" => "more"} + ]) + ) + + assert Errors.problem(changeset) == + %Problem{ + status: 402, + code: :exceeds_limit, + errors: [ + entry_at("#/id", :exceeds_limit, "No collections left") + ] + } + end + + test "answers 409 taken when every error is taken, whether a unique constraint or an application check found it" do + seed_ready_catalogue() + ensure_adaptor(@adaptor) + job = insert(:job, name: "taken", adaptor: @adaptor) + + {:error, constraint} = + %Job{} + |> Job.changeset(%{ + name: "taken", + adaptor: @adaptor, + body: "fn(s => s)", + workflow_id: job.workflow_id + }) + |> Repo.insert() + + check = + User.changeset(%User{}, %{ + first_name: "Ada", + last_name: "Lovelace", + email: insert(:user).email, + password: "a long enough password" + }) + + for {changeset, entry} <- [ + {constraint, + entry_at("#/name", :taken, "job name has already been taken")}, + {check, entry_at("#/email", :taken, "has already been taken")} + ] do + assert Errors.problem(changeset) == + %Problem{status: 409, code: :taken, errors: [entry]} + end + end + end + + describe "refuse/1 and parameters/1" do + test "passes no entries and refuses any as one 422 invalid" do + entry = entry_at("#/name", :required, "can't be blank") + + assert Errors.refuse([]) == :ok + + assert Errors.refuse([entry]) == + {:error, %Problem{status: 422, code: :invalid, errors: [entry]}} + end + + test "parameters/1 raises on a changeset with no errors" do + changeset = Ecto.Changeset.cast({%{}, %{state: :string}}, %{}, [:state]) + + assert_raise ArgumentError, fn -> Errors.parameters(changeset) end + end + end +end diff --git a/test/lightning_web/api/filters_test.exs b/test/lightning_web/api/filters_test.exs new file mode 100644 index 00000000000..9ef6e78cb24 --- /dev/null +++ b/test/lightning_web/api/filters_test.exs @@ -0,0 +1,140 @@ +defmodule LightningWeb.API.FiltersTest do + use ExUnit.Case, async: true + + alias LightningWeb.API.Errors + alias LightningWeb.API.Filters + + describe "runs/1" do + test "answers the filters it accepts, parsed" do + assert Filters.runs(%{ + "inserted_before" => "2024-01-01T02:00:00+02:00", + "state" => "started,failed", + "page" => "2" + }) == + {:ok, + %{ + inserted_before: ~U[2024-01-01 00:00:00.000000Z], + state: ["started", "failed"] + }} + end + end + + describe "datetime parameters" do + test "a value that is not a datetime is refused, whatever it looks like" do + for {filter, parameters} <- [ + {&Filters.runs/1, + ~w(inserted_after inserted_before updated_after updated_before)}, + {&Filters.work_orders/1, + ~w(inserted_after inserted_before updated_after updated_before)}, + {&Filters.log_lines/1, ~w(timestamp_after timestamp_before)} + ], + parameter <- parameters, + value <- ["not-a-valid-datetime", "2024-13-45", "123456"] do + assert {:error, changeset} = filter.(%{parameter => value}) + + assert Errors.parameters(changeset).errors == [ + %{parameter: parameter, code: :type, detail: "is invalid"} + ], + "#{parameter}=#{value}" + end + end + end + + describe "runs/1 and work_orders/1 states" do + test "a state that is not one of the resource's is refused" do + for filter <- [&Filters.runs/1, &Filters.work_orders/1] do + assert {:error, changeset} = filter.(%{"state" => "started,bogus"}) + + assert Errors.parameters(changeset).errors == [ + %{ + parameter: "state", + code: :not_allowed, + detail: "has an invalid entry" + } + ] + end + end + end + + describe "log_lines/1" do + test "refuses a level that is not a log level" do + assert {:error, changeset} = Filters.log_lines(%{"level" => "loud"}) + + assert Errors.parameters(changeset).errors == [ + %{parameter: "level", code: :not_allowed, detail: "is invalid"} + ] + end + end + + describe "work_orders/1" do + test "refuses an id filter that isn't a UUID" do + assert {:error, changeset} = + Filters.work_orders(%{ + "id" => "#{Ecto.UUID.generate()},nope", + "workflow_id" => "nope" + }) + + assert Enum.sort(Errors.parameters(changeset).errors) == [ + %{parameter: "id", code: :type, detail: "is invalid"}, + %{parameter: "workflow_id", code: :type, detail: "is invalid"} + ] + end + end + + describe "collection_items/1" do + test "counts an empty value as a value: an empty key stays, and an empty limit or cursor is refused" do + Mox.stub(Lightning.MockConfig, :collections_default_stream_limit, fn -> + 25 + end) + + assert Filters.collection_items(%{"key" => ""}) == + {:ok, %{key: "", limit: 25}} + + for parameter <- ["limit", "cursor"] do + assert {:error, changeset} = + Filters.collection_items(%{parameter => ""}) + + assert Errors.parameters(changeset).errors == [ + %{parameter: parameter, code: :type, detail: "is invalid"} + ] + end + end + end + + describe "page/1" do + test "gives Repo.paginate/2 the page it would read from the request itself" do + for params <- [ + %{"page" => "2", "page_size" => "50"}, + %{"page" => "5abc", "page_size" => "2.0"}, + %{"page" => "+3", "page_size" => " 4"}, + %{"page" => "0", "page_size" => "-1"}, + %{"page" => "", "page_size" => "abc"}, + %{"page" => ["2"], "page_size" => %{"a" => "2"}}, + %{"page" => "99999999999999999999", "page_size" => "1000"}, + %{"page" => 3, "page_size" => 0}, + %{"state" => "failed"} + ] do + assert scrivener(Filters.page(params)) == scrivener(params), + inspect(params) + end + end + + test "answers the leading number, leaving out what it can't read" do + assert Filters.page(%{"page" => "5abc", "page_size" => "2.0"}) == + [page: 5, page_size: 2] + + assert Filters.page(%{"page" => "abc", "page_size" => "0"}) == [] + end + end + + defp scrivener(options) do + config = + Scrivener.Config.new( + Lightning.Repo, + Lightning.Repo.scrivener_defaults(), + options + ) + + {config.page_number, config.page_size} + end +end diff --git a/test/lightning_web/api/problem_test.exs b/test/lightning_web/api/problem_test.exs new file mode 100644 index 00000000000..ae8dd117f51 --- /dev/null +++ b/test/lightning_web/api/problem_test.exs @@ -0,0 +1,115 @@ +defmodule LightningWeb.API.ProblemTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProblemHelpers + + alias Lightning.Extensions.Message + alias LightningWeb.API.FallbackController + alias LightningWeb.API.Problem + + defp respond(accept, problem) do + build_conn() + |> then(fn conn -> + if accept, do: put_req_header(conn, "accept", accept), else: conn + end) + |> Phoenix.Controller.accepts(["json"]) + |> Problem.respond(problem) + end + + describe "respond/2" do + test "sends one body, as application/problem+json only when Accept names it, and never refuses it with a 406" do + for {accept, content_type} <- [ + {nil, "application/json"}, + {"*/*", "application/json"}, + {"application/json", "application/json"}, + {"application/problem+json", "application/problem+json"}, + {"application/json, application/problem+json;q=0.5", + "application/problem+json"} + ] do + conn = respond(accept, Problem.new(:not_found)) + + assert get_resp_header(conn, "content-type") == [ + "#{content_type}; charset=utf-8" + ] + + assert_problem(conn, 404, :not_found, []) + end + end + + test "an API route takes a request that accepts only application/problem+json" do + conn = + build_conn() + |> put_req_header("accept", "application/problem+json") + |> get(~p"/api/projects") + + assert get_resp_header(conn, "content-type") == [ + "application/problem+json; charset=utf-8" + ] + + assert_unauthenticated(conn, :unauthenticated, ["Bearer"]) + end + + test "sends the problem's entries, extensions and headers" do + problem = + Problem.new(:not_ready, + headers: [{"retry-after", "5"}], + extensions: %{work_order_id: "wo-1"} + ) + + conn = respond(nil, problem) + + assert get_resp_header(conn, "retry-after") == ["5"] + + assert %{"work_order_id" => "wo-1"} = + assert_problem(conn, 503, :not_ready, []) + end + + test "replaces a header the conn already had, so it is sent once with the problem's value" do + conn = + build_conn() + |> put_resp_header("www-authenticate", "Basic") + |> put_resp_header("retry-after", "1") + |> Problem.respond( + Problem.new(:not_ready, + headers: [{"www-authenticate", "Bearer"}, {"retry-after", "5"}] + ) + ) + + assert get_resp_header(conn, "www-authenticate") == ["Bearer"] + assert get_resp_header(conn, "retry-after") == ["5"] + end + end + + describe "new/2" do + test "takes a status a code may also answer with" do + problem = Problem.new(:exceeds_limit, status: 429) + + assert problem.status == 429 + assert Problem.body(problem).title == "Too Many Requests" + end + + test "refuses a status the code never answers with" do + assert_raise ArgumentError, ":not_found is never sent at 429", fn -> + Problem.new(:not_found, status: 429) + end + end + end + + describe "FallbackController" do + test "answers the results no route shows, and the limiter reasons, with one status and code" do + for {result, status, code} <- [ + {{:error, :no_such_project}, 404, :not_found}, + {{:error, :workflow_deleted}, 404, :not_found}, + {{:error, :exceeds_limit}, 402, :exceeds_limit}, + {{:error, :runs_hard_limit, %Message{}}, 402, :exceeds_limit}, + {{:error, :too_many_runs, %Message{}}, 429, :too_many_runs}, + {{:error, :too_many_workflows, %Message{}}, 402, :exceeds_limit}, + {{:error, :unknown_project, %Message{}}, 404, :not_found} + ] do + build_conn() + |> FallbackController.call(result) + |> assert_problem(status, code, []) + end + end + end +end diff --git a/test/lightning_web/api/request_test.exs b/test/lightning_web/api/request_test.exs new file mode 100644 index 00000000000..96d813a970a --- /dev/null +++ b/test/lightning_web/api/request_test.exs @@ -0,0 +1,130 @@ +defmodule LightningWeb.API.RequestTest do + use ExUnit.Case, async: true + + alias LightningWeb.API.Problem + alias LightningWeb.API.Request + + @id "3f2504e0-4f89-41d3-9a0c-0305e82c3301" + @accepted [:id, :name] + + @mismatch %{ + pointer: "#/id", + code: :mismatch, + detail: "does not match the id in the path" + } + + test "passes a body holding only accepted keys, with the path's id in either case or left out" do + for body <- [ + %{"id" => @id, "name" => "n"}, + %{"name" => "n"}, + %{"id" => String.upcase(@id)} + ] do + assert Request.check(body, @id, @accepted) == {:ok, []}, inspect(body) + end + end + + test "answers 400 malformed for a body that is not a JSON object" do + assert Request.check(%{"_json" => [1, 2]}, @id, @accepted) == + {:error, Problem.new(:malformed)} + end + + test "gives one unknown_key entry per key the route does not accept, in key order" do + body = %{"id" => @id, "name" => "n", "colour" => "red", "a/b" => 1} + + assert Request.check(body, @id, @accepted) == + {:ok, + [ + %{ + pointer: "#/a~1b", + code: :unknown_key, + detail: "is not accepted" + }, + %{ + pointer: "#/colour", + code: :unknown_key, + detail: "is not accepted" + } + ]} + end + + test "gives a mismatch for a body id that is another UUID, is not a UUID, or is a 16-byte string whose raw bytes are the path's id" do + raw = "aaaaaaaaaaaaaaaa" + + for {body_id, path_id} <- [ + {"0b9e0a5e-7d55-4d8e-8f7e-4a0b1c2d3e4f", @id}, + {42, @id}, + {raw, Ecto.UUID.cast!(raw)} + ] do + assert Request.check(%{"id" => body_id}, path_id, @accepted) == + {:ok, [@mismatch]}, + inspect(body_id) + end + end + + test "gives unknown keys and a mismatch together" do + body = %{"id" => "0b9e0a5e-7d55-4d8e-8f7e-4a0b1c2d3e4f", "colour" => "red"} + + assert Request.check(body, @id, @accepted) == + {:ok, + [ + %{ + pointer: "#/colour", + code: :unknown_key, + detail: "is not accepted" + }, + @mismatch + ]} + end + + describe "validate_matches_path/4" do + defp matched(body, path_id) do + {%{}, %{project_id: :binary_id}} + |> Ecto.Changeset.change() + |> Request.validate_matches_path(body, :project_id, path_id) + end + + test "adds a mismatch error on the field when the body names another id there" do + changeset = matched(%{"project_id" => Ecto.UUID.generate()}, @id) + + assert changeset.errors == [ + project_id: {"does not match the id in the path", code: :mismatch} + ] + end + + test "leaves the changeset alone when the body names the path's id there, in any case, or names none" do + for body <- [ + %{"project_id" => @id}, + %{"project_id" => String.upcase(@id)}, + %{} + ] do + assert matched(body, @id).errors == [], inspect(body) + end + end + end + + describe "path_id/1" do + test "takes a UUID written out, in any case" do + assert Request.path_id(String.upcase(@id)) == {:ok, @id} + end + + test "answers not_found for anything else, a 16-byte string included" do + for id <- ["not-a-uuid", "aaaaaaaaaaaaaaaa", ""] do + assert Request.path_id(id) == {:error, :not_found}, inspect(id) + end + end + end + + describe "id_taken?/1" do + defp refused(field, opts) do + {%{}, %{id: :binary_id, name: :string}} + |> Ecto.Changeset.change() + |> Ecto.Changeset.add_error(field, "has already been taken", opts) + end + + test "is true only for the id's unique constraint" do + assert Request.id_taken?(refused(:id, constraint: :unique)) + refute Request.id_taken?(refused(:name, constraint: :unique)) + refute Request.id_taken?(refused(:id, validation: :required)) + end + end +end diff --git a/test/lightning_web/api/scope_test.exs b/test/lightning_web/api/scope_test.exs new file mode 100644 index 00000000000..7bc3c29ee39 --- /dev/null +++ b/test/lightning_web/api/scope_test.exs @@ -0,0 +1,261 @@ +defmodule LightningWeb.API.ScopeTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProblemHelpers + import Lightning.Factories + import Lightning.ServiceAccountHelpers + + alias Lightning.Accounts + alias Lightning.ServiceAccount.AccessToken + + setup %{conn: conn} do + {account, _private_key} = service_account_with_key() + Mox.stub(Lightning.MockConfig, :service_account, fn -> account end) + + user = insert(:user) + project = insert(:project, project_users: [%{user: user, role: :owner}]) + workflow = insert(:simple_workflow, project: project) + + workorder = + insert(:workorder, dataclip: insert(:dataclip), workflow: workflow) + + run = + insert(:run, + work_order: workorder, + dataclip: workorder.dataclip, + starting_trigger: hd(workflow.triggers) + ) + + conn = + conn + |> put_req_header("accept", "application/json") + |> put_req_header("content-type", "application/json") + + %{ + conn: conn, + account: account, + user: user, + project: project, + workflow: workflow, + job: hd(workflow.jobs), + workorder: workorder, + run: run, + repo_connection: insert(:project_repo_connection, project: project) + } + end + + describe "a token without the grant a route needs" do + test "projects:write is needed for projects, provisioning, workflows, jobs, runs, work orders and log lines", + ctx do + %{project: project, workflow: workflow, job: job} = ctx + missing = Ecto.UUID.generate() + new_project_id = Ecto.UUID.generate() + repo_connection = bearer(ctx.conn, ctx.repo_connection.access_token) + short_account = bearer(ctx.conn, issue(ctx, ["users:write"])) + + repo_connection_answers = + for path <- [ + ~p"/api/projects", + ~p"/api/projects/#{project.id}", + ~p"/api/projects/#{missing}", + ~p"/api/jobs", + ~p"/api/jobs?#{%{project_id: project.id}}", + ~p"/api/jobs?#{%{project_id: missing}}", + ~p"/api/projects/#{project.id}/jobs", + ~p"/api/projects/#{missing}/jobs", + ~p"/api/jobs/#{job.id}", + ~p"/api/jobs/#{missing}", + ~p"/api/runs", + ~p"/api/runs?state=sideways", + ~p"/api/projects/#{project.id}/runs", + ~p"/api/projects/#{missing}/runs?state=sideways", + ~p"/api/runs/#{ctx.run.id}", + ~p"/api/runs/#{missing}", + ~p"/api/work_orders", + ~p"/api/work_orders?state=sideways", + ~p"/api/projects/#{project.id}/work_orders", + ~p"/api/projects/#{missing}/work_orders?state=sideways", + ~p"/api/work_orders/#{ctx.workorder.id}", + ~p"/api/work_orders/#{missing}", + ~p"/api/log_lines", + ~p"/api/log_lines?#{%{project_id: project.id}}", + ~p"/api/log_lines?#{%{project_id: missing}}", + ~p"/api/log_lines?level=loud&run_id=run-ish", + ~p"/api/workflows/#{workflow.id}", + ~p"/api/workflows/#{missing}" + ] do + get(repo_connection, path) + end + + assert_refused_for_scope(repo_connection_answers, "projects:write") + + # Each of these is refused before its body is read, so a body that would + # fail on its own does not change the answer. + assert_refused_for_scope( + [ + put(repo_connection, ~p"/api/projects/#{missing}", ~s({"name": "x"})), + post( + repo_connection, + ~p"/api/projects/#{project.id}/workflows", + "[1]" + ), + patch( + repo_connection, + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + ~s({"name": ""}) + ), + post( + repo_connection, + ~p"/api/provision", + ~s({"id": "#{new_project_id}"}) + ) + ], + "projects:write" + ) + + refute Lightning.Repo.get(Lightning.Projects.Project, new_project_id) + refute Lightning.Repo.get(Lightning.Projects.Project, missing) + + assert_refused_for_scope( + [ + get(short_account, ~p"/api/projects/#{project.id}"), + get(short_account, ~p"/api/projects/#{missing}"), + put(short_account, ~p"/api/projects/#{project.id}", ~s({"name": "x"})), + put(short_account, ~p"/api/projects/#{missing}", ~s({"name": "x"})) + ], + "projects:write" + ) + + refute Lightning.Repo.get(Lightning.Projects.Project, missing) + end + + test "collections:write is needed to read or write a collection through the API", + %{project: project} = ctx do + collection = insert(:collection, project: project) + missing = Ecto.UUID.generate() + + for caller <- [ + bearer(ctx.conn, ctx.repo_connection.access_token), + bearer(ctx.conn, issue(ctx, ["projects:write"])) + ] do + assert_refused_for_scope( + [ + get( + caller, + ~p"/api/projects/#{project.id}/collections/#{collection.id}" + ), + get(caller, ~p"/api/projects/#{project.id}/collections/#{missing}"), + put( + caller, + ~p"/api/projects/#{project.id}/collections/#{missing}", + ~s({"name": "patients"}) + ) + ], + "collections:write" + ) + end + + refute Lightning.Repo.get(Lightning.Collections.Collection, missing) + end + + test "credentials:write is needed to list, read, create, replace or delete a credential", + ctx do + credential = insert(:credential, user: ctx.user) + missing = Ecto.UUID.generate() + repo_connection = bearer(ctx.conn, ctx.repo_connection.access_token) + short_account = bearer(ctx.conn, issue(ctx, ["projects:write"])) + + assert_refused_for_scope( + [ + get(repo_connection, ~p"/api/credentials"), + get(repo_connection, ~p"/api/projects/#{ctx.project.id}/credentials"), + get(repo_connection, ~p"/api/projects/#{missing}/credentials"), + post( + repo_connection, + ~p"/api/credentials", + Jason.encode!(%{ + "name" => "Planted", + "schema" => "raw", + "credential_bodies" => [ + %{"name" => "main", "body" => %{"a" => "b"}} + ] + }) + ), + delete(repo_connection, ~p"/api/credentials/#{credential.id}"), + delete(repo_connection, ~p"/api/credentials/#{missing}") + ], + "credentials:write" + ) + + refute Lightning.Repo.get_by(Lightning.Credentials.Credential, + name: "Planted" + ) + + assert Lightning.Repo.get(Lightning.Credentials.Credential, credential.id) + + for caller <- [repo_connection, short_account] do + assert_refused_for_scope( + [ + get(caller, ~p"/api/credentials/#{credential.id}"), + get(caller, ~p"/api/credentials/#{missing}"), + put(caller, ~p"/api/credentials/#{missing}", ~s({"name": "x"})) + ], + "credentials:write" + ) + end + + refute Lightning.Repo.get(Lightning.Credentials.Credential, missing) + end + + test "users:read is needed to list or show users", ctx do + short_account = bearer(ctx.conn, issue(ctx, ["users:write"])) + + assert_refused_for_scope( + [ + get(short_account, ~p"/api/users?email=a@example.com"), + get(short_account, ~p"/api/users/#{insert(:user).id}"), + get(short_account, ~p"/api/users/#{Ecto.UUID.generate()}") + ], + "users:read" + ) + end + + test "users:write is needed to create or change a user", ctx do + short_account = bearer(ctx.conn, issue(ctx, ["users:read"])) + user = insert(:user, first_name: "Ada") + + assert_refused_for_scope( + [ + post( + short_account, + ~p"/api/users", + Jason.encode!(%{ + email: "no@example.com", + password: "a long enough password" + }) + ), + post(short_account, ~p"/api/users", "[1]"), + patch( + short_account, + ~p"/api/users/#{user.id}", + ~s({"first_name": "Grace"}) + ), + patch( + short_account, + ~p"/api/users/#{Ecto.UUID.generate()}", + ~s({"first_name": "Grace"}) + ) + ], + "users:write" + ) + + refute Accounts.get_user_by_email("no@example.com") + assert %{first_name: "Ada"} = Accounts.get_user(user.id) + end + end + + defp issue(%{account: account}, grants), do: AccessToken.issue(account, grants) + + defp bearer(conn, token), + do: put_req_header(conn, "authorization", "Bearer " <> token) +end diff --git a/test/lightning_web/api/sentinel_sweep_test.exs b/test/lightning_web/api/sentinel_sweep_test.exs new file mode 100644 index 00000000000..ec86c06e349 --- /dev/null +++ b/test/lightning_web/api/sentinel_sweep_test.exs @@ -0,0 +1,330 @@ +defmodule LightningWeb.API.SentinelSweepTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProblemHelpers + import Lightning.Factories + + @sentinel plant_sentinel("x") + + @query plant_sentinel(%{ + "project_id" => nil, + "id" => nil, + "page" => nil, + "page_size" => nil, + "state" => nil, + "inserted_after" => nil, + "days" => nil, + "offset" => nil, + "limit" => nil, + "cursor" => nil, + "email" => nil, + "snapshots" => nil + }) + + @body plant_sentinel(%{ + "id" => nil, + "name" => nil, + "project_id" => nil, + "value" => nil, + "items" => [%{"key" => nil, "value" => nil}] + }) + + @unauthenticated {401, :unauthenticated, ["Bearer"]} + @invalid_token {401, :invalid_token, [~s(Bearer error="invalid_token")]} + @session {401, :unauthenticated, [~s(Session realm="Lightning")]} + @not_found {404, :not_found, []} + @malformed {400, :malformed, []} + + # What every in-scope route answers, by the pipelines it goes through. + # + # `planted` is a member with a personal token and a request whose path + # parameters aren't ids. `array` is the same member with real ids and a JSON + # array for a body: `real_ids` for a route whose path parameters are all the + # real ones, `other_ids` for a route with a path id the request doesn't have. + # A route with no path parameter has nothing planted, so it is only held to + # answering with a problem that quotes nothing. + @pipelines %{ + [:access_token_api] => %{ + none: @unauthenticated, + garbage: @invalid_token, + planted: @invalid_token, + array: %{real_ids: @invalid_token, other_ids: @invalid_token} + }, + [:person_or_service_account_api] => %{ + none: @unauthenticated, + garbage: @invalid_token, + planted: @not_found, + array: %{real_ids: @malformed, other_ids: @not_found} + }, + [:api, :authenticate_bearer, :require_authenticated_api_resource] => %{ + none: @unauthenticated, + garbage: @invalid_token, + planted: @not_found, + array: %{real_ids: @malformed, other_ids: @not_found} + }, + [:authenticated_api] => %{ + none: @unauthenticated, + garbage: @invalid_token, + planted: @not_found, + array: %{real_ids: @malformed, other_ids: @not_found} + }, + [:authenticated_json, :require_authenticated_user] => %{ + none: @session, + garbage: @session, + planted: @not_found, + array: nil + } + } + + # Routes that take no token at all, each with the reason it answers the + # same whatever the request carries. + @exceptions %{ + [:api] => %{ + reason: "registration takes no token, and it is switched off here", + none: {501, :not_implemented, []}, + garbage: {501, :not_implemented, []}, + planted: {501, :not_implemented, []}, + array: %{ + real_ids: {501, :not_implemented, []}, + other_ids: {501, :not_implemented, []} + } + }, + [:webhook] => %{ + reason: + "webhooks run before any token check, and no trigger has the planted path", + none: @not_found, + garbage: @not_found, + planted: @not_found, + array: %{real_ids: @not_found, other_ids: @not_found} + }, + [] => %{ + reason: "an unmatched path is answered before any token check", + none: @not_found, + garbage: @not_found, + planted: @not_found, + array: nil + }, + :channel_proxy => %{ + reason: + "the proxy sits above the router and finds the channel before it reads a token", + none: @not_found, + garbage: @not_found, + planted: @not_found, + array: %{real_ids: @not_found, other_ids: @not_found} + } + } + + setup do + user = insert(:user) + project = insert(:project, project_users: [%{user: user, role: :owner}]) + workflow = insert(:workflow, project: project) + token = Lightning.Accounts.generate_api_token(user) + + member = + build_conn() + |> log_in_user(user) + |> put_req_header("authorization", "Bearer " <> token) + + %{ + member: member, + real: %{"project_id" => project.id, "workflow_id" => workflow.id} + } + end + + test "a request with no token gets the answer its pipeline promises" do + for route <- in_scope_routes() do + conn = request(build_conn(), route, %{}) + assert_answer(conn, route, :none) + end + end + + test "a garbage bearer token gets the answer its pipeline promises" do + garbage = + put_req_header(build_conn(), "authorization", "Bearer " <> @sentinel) + + for route <- in_scope_routes() do + conn = request(garbage, route, %{}) + assert_answer(conn, route, :garbage) + end + end + + test "a member who names ids that aren't there gets not found, or the token refusal", + %{ + member: member + } do + for route <- in_scope_routes() do + conn = request(member, route, %{}) + + if has_path_parameter?(route) do + assert_answer(conn, route, :planted) + else + assert_any_problem(conn, route) + end + end + end + + test "a member who names ids that are there gets an answer that quotes nothing", + %{ + member: member, + real: real + } do + for route <- in_scope_routes() do + conn = request(member, route, real) + assert_any_problem(conn, route) + end + end + + test "a member who sends a JSON array to a write route gets malformed, or not found for an id that isn't there", + %{member: member, real: real} do + for %{verb: verb} = route <- in_scope_routes(), + verb in [:post, :put, :patch] do + conn = request(member, route, real, [@sentinel]) + assert_answer(conn, route, :array) + end + end + + defp in_scope_routes do + routes = + for %{verb: verb, path: path} <- + Phoenix.Router.routes(LightningWeb.Router), + in_scope?(path), + do: %{verb: verb, path: path, pipeline: pipeline(verb, path)} + + # The channel proxy sits above the router. + [ + %{ + verb: :post, + path: "/channels/:channel_id/*path", + pipeline: :channel_proxy + } + | routes + ] + end + + defp in_scope?("/api/oauth/" <> _), do: false + defp in_scope?("/api/" <> _), do: true + defp in_scope?("/collections" <> _), do: true + defp in_scope?("/adaptors/catalogue"), do: true + defp in_scope?("/i/" <> _), do: true + defp in_scope?(_path), do: false + + # The router's route list doesn't carry the pipelines, so ask the router + # which route the planted request lands on. + defp pipeline(verb, path) do + method = + if verb == :*, do: "GET", else: verb |> to_string() |> String.upcase() + + case Phoenix.Router.route_info( + LightningWeb.Router, + method, + planted_path(path, %{}), + "localhost" + ) do + %{pipe_through: pipeline} -> pipeline + :error -> :no_route + end + end + + defp has_path_parameter?(%{path: path}), do: String.contains?(path, [":", "*"]) + + # `ids` names the path parameters to fill with real ids, so a request can get + # past the lookup to the checks on its query and body. Every other parameter + # is planted. + defp planted_path(path, ids) do + path + |> String.split("/") + |> Enum.map_join("/", fn + ":" <> param -> Map.get(ids, param, @sentinel) + "*" <> _param -> @sentinel + segment -> segment + end) + end + + defp request(conn, route, ids, body \\ @body) + + defp request(conn, %{verb: verb, path: path}, ids, body) do + conn = + conn + |> put_req_header("accept", "application/json") + |> put_req_header("x-timezone", @sentinel) + + query = Map.merge(@query, Map.take(ids, ["project_id"])) + path = planted_path(path, ids) <> "?" <> URI.encode_query(query) + + case verb do + verb when verb in [:get, :*] -> + get(conn, path) + + verb -> + conn + |> put_req_header("content-type", "application/json") + |> dispatch(@endpoint, verb, path, Jason.encode!(body)) + end + end + + defp expected(%{pipeline: pipeline} = route, attempt) do + row = + Map.get(@pipelines, pipeline) || Map.get(@exceptions, pipeline) || + flunk( + "#{describe(route)} goes through #{inspect(pipeline)}, which neither " <> + "@pipelines nor @exceptions says what to expect from" + ) + + case {Map.fetch!(row, attempt), attempt} do + {%{real_ids: real, other_ids: other}, :array} -> + if all_ids_real?(route), do: real, else: other + + {answer, _attempt} -> + answer || + flunk("#{describe(route)} has no #{attempt} answer in the table") + end + end + + # Provisioning takes the project to replace from an `id` in the query, which + # the sweep plants, so its path has no real id to find either. + defp all_ids_real?(%{path: "/api/provision"}), do: false + + defp all_ids_real?(%{path: path}) do + params = for ":" <> param <- String.split(path, "/"), do: param + + Enum.all?(params, &(&1 in ["project_id", "workflow_id"])) and + not String.contains?(path, "*") + end + + defp assert_answer(conn, route, attempt) do + {status, code, challenges} = expected(route, attempt) + + with_route(route, attempt, fn -> + assert_problem(conn, status, code, []) + assert get_resp_header(conn, "www-authenticate") == challenges + refute_sentinel(conn) + end) + end + + defp assert_any_problem(conn, route) do + with_route(route, :any, fn -> + refute_sentinel(conn) + + if conn.status >= 400 do + %{"code" => code} = Jason.decode!(conn.resp_body) + assert_problem(conn, conn.status, String.to_existing_atom(code)) + end + end) + end + + defp with_route(route, attempt, fun) do + fun.() + rescue + error in ExUnit.AssertionError -> + reraise %{ + error + | message: + "#{describe(route)}, #{attempt}: " <> + (error.message || "assertion failed") + }, + __STACKTRACE__ + end + + defp describe(%{verb: verb, path: path}), + do: "#{verb |> to_string() |> String.upcase()} #{path}" +end diff --git a/test/lightning_web/auth_test.exs b/test/lightning_web/auth_test.exs index 2285976b165..e7c9c3edb88 100644 --- a/test/lightning_web/auth_test.exs +++ b/test/lightning_web/auth_test.exs @@ -1,10 +1,379 @@ defmodule LightningWeb.AuthTest do - use ExUnit.Case, async: true + @moduledoc """ + One route per JSON pipeline, refused for no token, for a bad one and for a + `Bearer` header it can't read, which token kinds each pipeline takes, a + locked-out account on a bearer pipeline, and the bearer scheme read without + regard to case on one route per auth plug. Then the checks of a request + against webhook auth methods. + """ + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProblemHelpers + import Lightning.Factories + import Lightning.ServiceAccountHelpers + import Plug.Test, only: [conn: 2] - alias LightningWeb.Auth + alias Lightning.Accounts + alias Lightning.Config.API, as: ConfigAPI + alias Lightning.Repo + alias Lightning.ServiceAccount.AccessToken alias Lightning.Workflows.WebhookAuthMethod + alias LightningWeb.Auth - import Plug.Test, only: [conn: 2] + @invalid_token [~s(Bearer error="invalid_token")] + + setup %{conn: conn} do + {account, _private_key} = service_account_with_key() + Mox.stub(Lightning.MockConfig, :service_account, fn -> account end) + + %{conn: put_req_header(conn, "accept", "application/json"), account: account} + end + + describe "a lower-case bearer scheme" do + test "authenticates a service account's access token on GET /api/users", + %{conn: conn, account: account} do + conn = + conn + |> put_req_header( + "authorization", + "bearer " <> AccessToken.issue(account, ["users:read"]) + ) + |> get(~p"/api/users") + + assert %{"data" => _users} = json_response(conn, 200) + end + + test "authenticates a personal access token on GET /api/projects", + %{conn: conn} do + project = insert(:project, project_users: [%{user: user = insert(:user)}]) + + conn = + conn + |> put_req_header( + "authorization", + "bearer " <> Accounts.generate_api_token(user) + ) + |> get(~p"/api/projects") + + assert %{"data" => [%{"id" => id}]} = json_response(conn, 200) + assert id == project.id + end + + test "authenticates a personal access token on GET /collections/:name", + %{conn: conn} do + user = insert(:user) + project = insert(:project, project_users: [%{user: user}]) + collection = insert(:collection, project: project) + + conn = + conn + |> put_req_header( + "authorization", + "bearer " <> Accounts.generate_api_token(user) + ) + |> get(~p"/collections/#{collection.name}") + + assert %{"items" => [], "cursor" => nil} = json_response(conn, 200) + end + end + + for {pipeline, path} <- [ + access_token_api: "/api/users", + person_or_service_account_api: + "/api/projects/7b0c1b4c-0c62-4b8e-9a39-0a6c9e1f2d11", + api: "/api/projects", + authenticated_api: "/collections/a-collection" + ] do + describe "the #{pipeline} pipeline, GET #{path}" do + test "answers no token with unauthenticated and a bare Bearer challenge", + %{conn: conn} do + conn = get(conn, unquote(path)) + + assert_unauthenticated(conn, :unauthenticated, ["Bearer"]) + end + + test "answers a bad token with invalid_token", %{conn: conn} do + conn = + conn + |> put_req_header("authorization", "Bearer not-a-token") + |> get(unquote(path)) + + assert_unauthenticated(conn, :invalid_token, @invalid_token) + end + + test "answers another scheme as no token sent", %{conn: conn} do + conn = + conn + |> put_req_header("authorization", "Basic dXNlcjpwYXNz") + |> get(unquote(path)) + + assert_unauthenticated(conn, :unauthenticated, ["Bearer"]) + end + end + + describe "the #{pipeline} pipeline, GET #{path}, a Bearer header Lightning can't read" do + test "answers no token after the scheme with invalid_request", + %{conn: conn} do + conn = + conn + |> put_req_header("authorization", "Bearer") + |> get(unquote(path)) + + assert_invalid_request(conn) + end + + test "answers whitespace inside the token with invalid_request", + %{conn: conn} do + conn = + conn + |> put_req_header("authorization", "Bearer a b") + |> get(unquote(path)) + + assert_invalid_request(conn) + end + + test "answers two Bearer headers with invalid_request", %{conn: conn} do + # Cowboy joins a repeated request header into one value with ", ". + conn = + conn + |> put_req_header("authorization", "Bearer x, Bearer x") + |> get(unquote(path)) + + assert_invalid_request(conn) + end + end + end + + for path <- [ + "/api/ai_assistant/sessions", + "/adaptors/catalogue" + ] do + describe "the authenticated_json pipeline, GET #{path}" do + test "answers no session with unauthenticated and the Session challenge", + %{conn: conn} do + conn = get(conn, unquote(path)) + + assert_session_refused(conn) + end + + test "answers a bearer token with unauthenticated, since it takes only the cookie", + %{conn: conn} do + conn = + conn + |> put_req_header("authorization", "Bearer not-a-token") + |> get(unquote(path)) + + assert_session_refused(conn) + end + end + end + + # Proves the plug is wired into the /api scope, rather than only that the + # plug function itself refuses — see user_auth_test.exs for the unit tests. + describe "with an account past its confirmation deadline" do + test "gets a 401, and the same token works again once confirmed", %{ + conn: conn + } do + Mox.stub(Lightning.MockConfig, :check_flag?, fn + :require_email_verification -> true + flag -> ConfigAPI.check_flag?(flag) + end) + + user = insert(:user) + insert(:project, project_users: [%{user: user}]) + + token = Lightning.Accounts.generate_api_token(user) + + user = + user + |> Ecto.Changeset.change( + confirmed_at: nil, + inserted_at: + DateTime.utc_now() + |> DateTime.add(-50, :hour) + |> DateTime.truncate(:second) + ) + |> Repo.update!() + + refused = conn |> assign_bearer(token) |> get(~p"/api/projects") + + assert_problem(refused, 401, :invalid_token, []) + + user + |> Ecto.Changeset.change( + confirmed_at: DateTime.utc_now() |> DateTime.truncate(:second) + ) + |> Repo.update!() + + allowed = conn |> assign_bearer(token) |> get(~p"/api/projects") + + assert [%{"type" => "projects"}] = json_response(allowed, 200)["data"] + end + + test "gets a 401 on GET /collections/:name too, and the same token works again once confirmed", + %{conn: conn} do + Mox.stub(Lightning.MockConfig, :check_flag?, fn + :require_email_verification -> true + flag -> ConfigAPI.check_flag?(flag) + end) + + user = insert(:user) + project = insert(:project, project_users: [%{user: user}]) + collection = insert(:collection, project: project) + token = Accounts.generate_api_token(user) + + user = + user + |> Ecto.Changeset.change( + confirmed_at: nil, + inserted_at: + DateTime.utc_now() + |> DateTime.add(-50, :hour) + |> DateTime.truncate(:second) + ) + |> Repo.update!() + + path = ~p"/collections/#{collection.name}" + authorization = "Bearer " <> token + + refused = + conn |> put_req_header("authorization", authorization) |> get(path) + + assert_unauthenticated(refused, :invalid_token, @invalid_token) + + user + |> Ecto.Changeset.change( + confirmed_at: DateTime.utc_now() |> DateTime.truncate(:second) + ) + |> Repo.update!() + + allowed = + conn |> put_req_header("authorization", authorization) |> get(path) + + assert %{"items" => []} = json_response(allowed, 200) + end + end + + describe "a token of the wrong kind" do + setup %{account: account} do + user = insert(:user) + project = insert(:project, project_users: [%{user: user, role: :owner}]) + collection = insert(:collection, project: project) + workflow = insert(:simple_workflow, project: project) + + workorder = + insert(:workorder, dataclip: insert(:dataclip), workflow: workflow) + + run = + insert(:run, + work_order: workorder, + dataclip: workorder.dataclip, + starting_trigger: hd(workflow.triggers) + ) + + %{ + project: project, + collection: collection, + tokens: [ + service_account: AccessToken.issue(account, AccessToken.scopes()), + expired_service_account: + expired_access_token(account, "users:read projects:write"), + personal: Accounts.generate_api_token(user), + repo_connection: + insert(:project_repo_connection, project: project).access_token, + run: Lightning.Workers.generate_run_token(run) + ] + } + end + + test "access_token_api takes only a service account's access token", + %{conn: conn, tokens: tokens} do + assert_each_kind(conn, tokens, ~p"/api/users", + service_account: :logs_in, + expired_service_account: :invalid_token, + personal: :invalid_token, + repo_connection: :invalid_token, + run: :invalid_token + ) + end + + test "person_or_service_account_api takes a service account's or a person's token, and not a run token", + %{conn: conn, tokens: tokens, project: project} do + assert_each_kind(conn, tokens, ~p"/api/projects/#{project.id}", + service_account: :logs_in, + expired_service_account: :invalid_token, + personal: :logs_in, + repo_connection: {:insufficient_scope, "projects:write"}, + run: :invalid_token + ) + end + + test "the :api scope takes a personal or repo connection token only", + %{conn: conn, tokens: tokens} do + assert_each_kind(conn, tokens, ~p"/api/projects", + service_account: :invalid_token, + expired_service_account: :invalid_token, + personal: :logs_in, + repo_connection: {:insufficient_scope, "projects:write"}, + run: :invalid_token + ) + end + + test "authenticated_api takes a personal or run token, and not a repo connection's or a service account's", + %{conn: conn, tokens: tokens, collection: collection} do + assert_each_kind(conn, tokens, ~p"/collections/#{collection.name}", + service_account: :invalid_token, + expired_service_account: :invalid_token, + personal: :logs_in, + repo_connection: :invalid_token, + run: :logs_in + ) + end + + test "authenticated_json takes no bearer token of any kind", %{ + conn: conn, + tokens: tokens + } do + for {_kind, token} <- tokens do + conn = + conn + |> put_req_header("authorization", "Bearer " <> token) + |> get(~p"/api/ai_assistant/sessions") + + assert_session_refused(conn) + end + end + end + + defp assert_each_kind(conn, tokens, path, expected) do + assert Enum.sort(Keyword.keys(expected)) == Enum.sort(Keyword.keys(tokens)) + + for {kind, answer} <- expected do + conn = + conn + |> put_req_header("authorization", "Bearer " <> tokens[kind]) + |> get(path) + + case answer do + :logs_in -> + assert conn.status == 200, "#{kind} was answered #{conn.status}" + + :invalid_token -> + assert_unauthenticated(conn, :invalid_token, @invalid_token) + + {:insufficient_scope, grant} -> + assert_refused_for_scope(conn, grant) + end + end + end + + defp assert_invalid_request(conn) do + assert_problem(conn, 400, :invalid_request, []) + + assert get_resp_header(conn, "www-authenticate") == [ + ~s(Bearer error="invalid_request") + ] + end defp api_method(key) do %WebhookAuthMethod{auth_type: :api, api_key: key} diff --git a/test/lightning_web/channels/run_channel_test.exs b/test/lightning_web/channels/run_channel_test.exs index c93d5033d18..52353cd8a4e 100644 --- a/test/lightning_web/channels/run_channel_test.exs +++ b/test/lightning_web/channels/run_channel_test.exs @@ -3624,7 +3624,7 @@ defmodule LightningWeb.RunChannelTest do end test "stops a support user's channel when the project withdraws support access", - %{project: project, run: run} do + %{actor: actor, project: project, run: run} do Mox.stub( Lightning.Extensions.MockProjectHook, :handle_project_validation, @@ -3632,7 +3632,11 @@ defmodule LightningWeb.RunChannelTest do ) {:ok, project} = - Lightning.Projects.update_project(project, %{allow_support_access: true}) + Lightning.Projects.update_project( + project, + %{allow_support_access: true}, + actor + ) support_user = insert(:user, support_user: true) @@ -3644,14 +3648,16 @@ defmodule LightningWeb.RunChannelTest do monitor_ref = Process.monitor(socket.channel_pid) {:ok, _project} = - Lightning.Projects.update_project(project, %{ - allow_support_access: false - }) + Lightning.Projects.update_project( + project, + %{allow_support_access: false}, + actor + ) assert_receive {:DOWN, ^monitor_ref, :process, _pid, :normal} end - # Project-wide, with no user on the event: `Scope` refuses a wound-down + # Project-wide, with no user on the event: the policies refuse a wound-down # project to everybody, so every browser socket streaming its runs goes. test "stops the channel when the project is scheduled for deletion", %{ project: project, diff --git a/test/lightning_web/channels/worker_channel_test.exs b/test/lightning_web/channels/worker_channel_test.exs index df462bc0be5..13e8e6fd324 100644 --- a/test/lightning_web/channels/worker_channel_test.exs +++ b/test/lightning_web/channels/worker_channel_test.exs @@ -316,4 +316,60 @@ defmodule LightningWeb.WorkerChannelTest do assert_reply ref, :ok, %{runs: [%{"id" => ^run_id3}]} end end + + defmodule RefusingRunQueue do + def claim(_demand, _worker_name, _queues) do + {:error, + %Lightning.Run{} + |> Ecto.Changeset.change() + |> Ecto.Changeset.add_error(:worker_name, "can't be blank") + |> Ecto.Changeset.add_error( + :worker_name, + "should be at most %{count} character(s)", + count: 10, + validation: :length, + kind: :max, + type: :string + )} + end + end + + describe "a refused claim" do + # The run queue adapter is resolved once and cached in :persistent_term, so + # the refusing adapter is swapped in there. Safe only because this module + # is async: false. + setup do + key = {Lightning.Services.AdapterHelper, :run_queue} + original = :persistent_term.get(key, nil) + :persistent_term.put(key, RefusingRunQueue) + + on_exit(fn -> + if original, + do: :persistent_term.put(key, original), + else: :persistent_term.erase(key) + end) + + {:ok, _, socket} = + ws_worker_socket() + |> subscribe_and_join(LightningWeb.WorkerChannel, "worker:queue") + + %{socket: socket} + end + + test "replies with the changeset's errors as %{field => [message]}", %{ + socket: socket + } do + ref = + push(socket, "claim", %{"demand" => 1, "worker_name" => "my.pod.name"}) + + assert_reply ref, :error, errors + + assert errors == %{ + worker_name: [ + "should be at most 10 character(s)", + "This field can't be blank." + ] + } + end + end end diff --git a/test/lightning_web/channels/workflow_channel_test.exs b/test/lightning_web/channels/workflow_channel_test.exs index 1445b2dc921..c6f0179c0e9 100644 --- a/test/lightning_web/channels/workflow_channel_test.exs +++ b/test/lightning_web/channels/workflow_channel_test.exs @@ -5,6 +5,7 @@ defmodule LightningWeb.WorkflowChannelTest do import Lightning.CollaborationHelpers import Lightning.Factories import Lightning.ProjectsHelpers + import Lightning.RaceHelpers import Mox import ExUnit.CaptureLog @@ -79,6 +80,25 @@ defmodule LightningWeb.WorkflowChannelTest do assert Lightning.Workflows.get_workflow!(workflow.id).state == :draft end + test "go_live losing a race to another save replies workflow_moved_on", + %{socket: socket, workflow: workflow} do + bump_workflow_lock_mid_save(workflow.id, socket.assigns.session_pid) + + ref = push(socket, "go_live", %{}) + + assert_reply ref, :error, %{ + type: "workflow_moved_on", + errors: %{base: ["Someone else saved this workflow. Try again."]} + } + + assert Process.alive?(socket.channel_pid) + + assert %{state: :draft, lock_version: lock_version} = + Lightning.Repo.reload(workflow) + + assert lock_version == workflow.lock_version + 1 + end + test "going live from a version being read is refused when the plan is at its limit", %{user: user, project: project, workflow: workflow} do insert(:trigger, type: :webhook, workflow: workflow, enabled: false) @@ -2124,7 +2144,7 @@ defmodule LightningWeb.WorkflowChannelTest do channel_pid = socket.channel_pid monitor_ref = Process.monitor(channel_pid) - revoke_support_access(project) + revoke_support_access(project, insert(:user)) assert_receive {:DOWN, ^monitor_ref, :process, ^channel_pid, :normal} end @@ -2143,7 +2163,7 @@ defmodule LightningWeb.WorkflowChannelTest do log = capture_log(fn -> - revoke_support_access(project) + revoke_support_access(project, insert(:user)) ref = push(socket, "validate_workflow_name", %{"workflow" => %{}}) assert_reply ref, :ok, %{workflow: _} @@ -2169,7 +2189,7 @@ defmodule LightningWeb.WorkflowChannelTest do socket = join_as(support_user, project, workflow) assert socket.assigns.project_user - revoke_support_access(project) + revoke_support_access(project, insert(:user)) ref = push(socket, "validate_workflow_name", %{"workflow" => %{}}) assert_reply ref, :ok, %{workflow: _} @@ -2588,21 +2608,23 @@ defmodule LightningWeb.WorkflowChannelTest do test "a concurrent save is refused, not a dropped channel", %{ restore_socket: socket, - workflow: workflow, - user: user + workflow: workflow } do - Mimic.copy(Lightning.Workflows) - - Mimic.stub(Lightning.Workflows, :restore_version, fn _wf, _rel, _actor -> - raise Ecto.StaleEntryError, action: :update, changeset: %Ecto.Changeset{} - end) + bump_workflow_lock_mid_save(workflow.id, socket.channel_pid) ref = push(socket, "restore_version", %{"version_number" => 1}) - assert_reply ref, :error, %{type: "workflow_moved_on"} + assert_reply ref, :error, %{ + type: "workflow_moved_on", + errors: %{ + base: ["Someone else saved this workflow. Try again."] + } + } + assert Process.alive?(socket.channel_pid) - _ = {workflow, user} + assert Lightning.Repo.reload(workflow).lock_version == + workflow.lock_version + 1 end test "the check reports nothing to lose when the triggers are unchanged", %{ @@ -3356,6 +3378,33 @@ defmodule LightningWeb.WorkflowChannelTest do assert saved.lock_version == lock_version end + test "a save losing a race to another save replies workflow_moved_on", + %{socket: socket, workflow: workflow} do + doc = Lightning.Collaboration.Session.get_doc(socket.assigns.session_pid) + workflow_map = Yex.Doc.get_map(doc, "workflow") + + Yex.Doc.transaction(doc, "test_update", fn -> + Yex.Map.set(workflow_map, "name", "Updated via Channel") + end) + + bump_workflow_lock_mid_save(workflow.id, socket.assigns.session_pid) + + ref = push(socket, "save_workflow", %{}) + + assert_reply ref, :error, %{ + type: "workflow_moved_on", + errors: %{base: ["Someone else saved this workflow. Try again."]} + } + + assert Process.alive?(socket.channel_pid) + + assert %{name: name, lock_version: lock_version} = + Lightning.Repo.reload(workflow) + + assert name == workflow.name + assert lock_version == workflow.lock_version + 1 + end + test "replies an error and keeps the channel alive when the session process is dead", %{socket: socket} do # A dead session makes the call exit rather than raise; the reply must diff --git a/test/lightning_web/collections/download_test.exs b/test/lightning_web/collections/download_test.exs new file mode 100644 index 00000000000..3ef7f597be0 --- /dev/null +++ b/test/lightning_web/collections/download_test.exs @@ -0,0 +1,121 @@ +defmodule LightningWeb.CollectionsDownloadTest do + use LightningWeb.ConnCase, async: true + + import Lightning.Factories + + describe "downloading a collection from the browser" do + setup :register_and_log_in_user + setup :create_project_for_current_user + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "text/html,*/*")} + end + + test "downloads collection items as JSON array", %{ + conn: conn, + project: project + } do + collection = insert(:collection, project: project) + + items = + for {key, value} <- [ + {"key-1", ~s({"name": "Alice"})}, + {"key-2", ~s({"name": "Bob"})} + ] do + insert(:collection_item, + collection: collection, + key: key, + value: value + ) + |> Jason.encode!() + |> Jason.decode!() + end + + response = + get(conn, ~p"/download/collections/#{project.id}/#{collection.name}") + + assert response.status == 200 + + assert get_resp_header(response, "content-type") == + ["application/json; charset=utf-8"] + + assert get_resp_header(response, "content-disposition") == + [~s(attachment; filename="#{collection.name}.json")] + + assert Jason.decode!(response.resp_body) == items + end + + test "downloads empty collection as empty JSON array", %{ + conn: conn, + project: project + } do + collection = insert(:collection, project: project) + + response = + get(conn, ~p"/download/collections/#{project.id}/#{collection.name}") + + assert response.status == 200 + assert Jason.decode!(response.resp_body) == [] + end + + test "returns 401 when user is not a project member", %{conn: conn} do + other_project = insert(:project) + collection = insert(:collection, project: other_project) + + response = + get( + conn, + ~p"/download/collections/#{other_project.id}/#{collection.name}" + ) + + assert response.status == 401 + end + + test "returns 404 for non-existent collection", %{ + conn: conn, + project: project + } do + response = get(conn, ~p"/download/collections/#{project.id}/non-existent") + + assert response.status == 404 + end + + test "returns 400 when project_id is not a valid UUID", %{conn: conn} do + response = get(conn, ~p"/download/collections/not-a-uuid/anything") + + assert response.status == 400 + end + + test "does not serve the payload to a locked-out account", + %{conn: conn, user: user, project: project} do + Mox.stub(Lightning.MockConfig, :check_flag?, fn + :require_email_verification -> true + flag -> Lightning.Config.API.check_flag?(flag) + end) + + user + |> Ecto.Changeset.change( + confirmed_at: nil, + inserted_at: + DateTime.utc_now() + |> Timex.shift(hours: -50) + |> DateTime.truncate(:second) + ) + |> Lightning.Repo.update!() + + collection = insert(:collection, project: project) + + insert(:collection_item, + collection: collection, + key: "key-1", + value: ~s({"name": "Alice"}) + ) + + response = + get(conn, ~p"/download/collections/#{project.id}/#{collection.name}") + + assert redirected_to(response) == "/users/confirm-required" + refute response.resp_body =~ "Alice" + end + end +end diff --git a/test/lightning_web/collections/items_test.exs b/test/lightning_web/collections/items_test.exs new file mode 100644 index 00000000000..310a76c75e8 --- /dev/null +++ b/test/lightning_web/collections/items_test.exs @@ -0,0 +1,149 @@ +defmodule LightningWeb.API.CollectionsItemsTest do + use LightningWeb.ConnCase, async: true + + import Lightning.Factories + import LightningWeb.ApiCollectionsHelpers + + alias Lightning.Collections + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + setup :member_with_collection + + describe "reading one item" do + test "answers the item", %{conn: conn, collection: collection} do + item = + insert(:collection_item, + collection: collection, + key: "foo", + value: "bar" + ) + + conn = get(conn, ~p"/collections/#{collection.name}/foo") + + assert json_response(conn, 200) == %{ + "key" => "foo", + "value" => "bar", + "created" => DateTime.to_iso8601(item.inserted_at), + "updated" => DateTime.to_iso8601(item.updated_at) + } + end + + test "answers 204 with no body for a key the collection hasn't got", %{ + conn: conn, + collection: collection + } do + conn = get(conn, ~p"/collections/#{collection.name}/some-unexisting-key") + + assert response(conn, 204) == "" + end + end + + describe "writing one item" do + test "creates a new key and replaces an existing one", %{ + conn: conn, + collection: collection + } do + insert(:collection_item, collection: collection, key: "foo", value: "bar") + + for {key, value} <- [{"baz", "qux"}, {"foo", "qux2"}] do + conn = + put(conn, ~p"/collections/#{collection.name}/#{key}", value: value) + + assert json_response(conn, 200) == %{"upserted" => 1, "error" => nil} + assert Collections.get(collection, key).value == value + end + end + end + + describe "writing many items" do + test "inserts multiple items with same timestamp", %{ + conn: conn, + collection: collection + } do + assert %{"upserted" => 1_000, "error" => nil} = + post(conn, ~p"/collections/#{collection.name}", %{ + items: + Enum.map(1..1_000, &%{key: "foo#{&1}", value: "bar#{&1}"}) + }) + |> json_response(200) + + assert %{"items" => items, "cursor" => nil} = + get(conn, ~p"/collections/#{collection.name}", limit: 1_000) + |> json_response(200) + + assert MapSet.new(items, & &1["created"]) |> MapSet.size() == 1 + assert Enum.count(items) == 1_000 + end + + test "upserts new and existing keys together", %{ + conn: conn, + collection: collection + } do + for n <- 1..3 do + insert(:collection_item, + collection: collection, + key: "foo#{n}", + value: "old#{n}" + ) + end + + conn = + post(conn, ~p"/collections/#{collection.name}", %{ + items: Enum.map(1..10, &%{key: "foo#{&1}", value: "bar#{&1}"}) + }) + + assert json_response(conn, 200) == %{"upserted" => 10, "error" => nil} + + assert Enum.map(1..10, &Collections.get(collection, "foo#{&1}").value) == + Enum.map(1..10, &"bar#{&1}") + end + end + + describe "deleting" do + test "deletes an item", %{conn: conn, collection: collection} do + insert(:collection_item, collection: collection, key: "foo", value: "bar") + + conn = delete(conn, ~p"/collections/#{collection.name}/foo") + + assert json_response(conn, 200) == %{ + "key" => "foo", + "deleted" => 1 + } + end + + test "returns deleted: 0 when deleting a missing key", %{ + conn: conn, + collection: collection + } do + conn = delete(conn, ~p"/collections/#{collection.name}/missing-key") + + assert json_response(conn, 200) == %{ + "key" => "missing-key", + "deleted" => 0 + } + end + + @tag role: :owner + test "deletes matching items", %{conn: conn, collection: collection} do + for {key, value} <- [ + {"foo:123:bar1", "value1"}, + {"foo:234:boo", "value2"}, + {"foo:345:bar2", "value3"} + ] do + insert(:collection_item, collection: collection, key: key, value: value) + end + + conn = delete(conn, ~p"/collections/#{collection.name}", key: "foo:*:bar*") + + assert json_response(conn, 200) == %{ + "key" => "foo:*:bar*", + "deleted" => 2 + } + + assert Collections.get(collection, "foo:234:boo") + end + end +end diff --git a/test/lightning_web/collections/naming_test.exs b/test/lightning_web/collections/naming_test.exs new file mode 100644 index 00000000000..a90da8576fe --- /dev/null +++ b/test/lightning_web/collections/naming_test.exs @@ -0,0 +1,277 @@ +defmodule LightningWeb.API.CollectionsNamingTest do + use LightningWeb.ConnCase, async: true + + import Lightning.Factories + import LightningWeb.API.ProblemHelpers + import LightningWeb.ApiCollectionsHelpers + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + setup :member_with_collection + + describe "a bare name" do + test "answers 404 for a collection the project hasn't got, on every route", + %{conn: conn} do + for request <- [ + &get(&1, ~p"/collections/misspelled-collection/foo"), + &put(&1, ~p"/collections/misspelled-collection/baz", value: "qux"), + &post(&1, ~p"/collections/misspelled-collection", + items: [%{key: "baz", value: "qux"}] + ), + &delete(&1, ~p"/collections/misspelled-collection/foo") + ] do + assert_problem(request.(conn), 404, :not_found, []) + end + end + + test "answers 409 ambiguous_name when two of the member's projects have it, on every route", + %{conn: conn, user: user, project: project} do + other_project = insert(:project, project_users: [%{user: user}]) + name = "shared-collection" + insert(:collection, name: name, project: project) + insert(:collection, name: name, project: other_project) + + for request <- [ + &get(&1, ~p"/collections/#{name}"), + &get(&1, ~p"/collections/#{name}/some-key"), + &put(&1, ~p"/collections/#{name}/some-key", value: "val"), + &post(&1, ~p"/collections/#{name}", items: []), + &delete(&1, ~p"/collections/#{name}/some-key"), + &delete(&1, ~p"/collections/#{name}") + ] do + assert_problem(request.(conn), 409, :ambiguous_name, []) + end + end + + test "a run token counts only its own project's collection", %{ + conn: conn, + project: project, + collection: collection + } do + insert(:collection_item, collection: collection, key: "foo", value: "bar") + insert(:collection, name: collection.name, project: insert(:project)) + + conn = + conn + |> assign_bearer(run_token(project)) + |> get(~p"/collections/#{collection.name}/foo") + + assert json_response(conn, 200)["key"] == "foo" + end + end + + describe "?project_id=" do + @describetag role: :owner + + setup %{collection: collection} do + item = + insert(:collection_item, + collection: collection, + key: "foo", + value: "bar" + ) + + %{item: item} + end + + test "reads one item", %{ + conn: conn, + project: project, + collection: collection, + item: item + } do + conn = + get( + conn, + ~p"/collections/#{collection.name}/foo?project_id=#{project.id}" + ) + + assert json_response(conn, 200) == %{ + "key" => "foo", + "value" => "bar", + "created" => DateTime.to_iso8601(item.inserted_at), + "updated" => DateTime.to_iso8601(item.updated_at) + } + end + + test "answers 204 for a key the collection hasn't got", %{ + conn: conn, + project: project, + collection: collection + } do + conn = + get( + conn, + ~p"/collections/#{collection.name}/nonexistent?project_id=#{project.id}" + ) + + assert response(conn, 204) == "" + end + + test "answers 404 for a collection the project hasn't got", %{ + conn: conn, + project: project + } do + conn = + get( + conn, + ~p"/collections/nonexistent-collection/foo?project_id=#{project.id}" + ) + + assert_problem(conn, 404, :not_found, []) + end + + test "lists the project's items", %{ + conn: conn, + project: project, + collection: collection, + item: item + } do + conn = + get(conn, ~p"/collections/#{collection.name}?project_id=#{project.id}") + + assert json_response(conn, 200) == %{ + "items" => [ + %{ + "key" => "foo", + "value" => "bar", + "created" => DateTime.to_iso8601(item.inserted_at), + "updated" => DateTime.to_iso8601(item.updated_at) + } + ], + "cursor" => nil + } + end + + test "upserts an item", %{ + conn: conn, + project: project, + collection: collection + } do + conn = + put( + conn, + ~p"/collections/#{collection.name}/new-key?project_id=#{project.id}", + value: "new-val" + ) + + assert json_response(conn, 200) == %{"upserted" => 1, "error" => nil} + end + + test "upserts many items", %{ + conn: conn, + project: project, + collection: collection + } do + conn = + post( + conn, + ~p"/collections/#{collection.name}?project_id=#{project.id}", + items: [%{key: "a", value: "1"}, %{key: "b", value: "2"}] + ) + + assert json_response(conn, 200) == %{"upserted" => 2, "error" => nil} + end + + test "deletes an item", %{ + conn: conn, + project: project, + collection: collection + } do + conn = + delete( + conn, + ~p"/collections/#{collection.name}/foo?project_id=#{project.id}" + ) + + assert json_response(conn, 200) == %{ + "key" => "foo", + "deleted" => 1 + } + end + + test "deletes every item", %{ + conn: conn, + project: project, + collection: collection + } do + conn = + delete( + conn, + ~p"/collections/#{collection.name}?project_id=#{project.id}" + ) + + assert json_response(conn, 200)["deleted"] == 1 + end + + test "answers 404 for a project the member doesn't belong to", %{conn: conn} do + other_project = insert(:project) + other_collection = insert(:collection, project: other_project) + + conn = + get( + conn, + ~p"/collections/#{other_collection.name}/foo?project_id=#{other_project.id}" + ) + + assert_problem(conn, 404, :not_found, []) + end + + test "picks the project's collection when another project has the same name", + %{conn: conn, project: project, collection: collection} do + insert(:collection, + name: collection.name, + project: insert(:project, project_users: []) + ) + + conn = + get( + conn, + ~p"/collections/#{collection.name}/foo?project_id=#{project.id}" + ) + + assert json_response(conn, 200)["key"] == "foo" + end + + test "answers 404 for a project_id that is not a UUID", %{ + conn: conn, + collection: collection + } do + conn = + get(conn, ~p"/collections/#{collection.name}?project_id=not-a-uuid") + + assert_problem(conn, 404, :not_found, []) + end + + test "lets a run token of the project read", %{ + conn: conn, + project: project, + collection: collection + } do + conn = + conn + |> assign_bearer(run_token(project)) + |> get(~p"/collections/#{collection.name}/foo?project_id=#{project.id}") + + assert json_response(conn, 200)["key"] == "foo" + end + + test "lets a run token of the project write", %{ + conn: conn, + project: project, + collection: collection + } do + conn = + conn + |> assign_bearer(run_token(project)) + |> put( + ~p"/collections/#{collection.name}/new-key?project_id=#{project.id}", + value: "new-val" + ) + + assert json_response(conn, 200) == %{"upserted" => 1, "error" => nil} + end + end +end diff --git a/test/lightning_web/collections/refusals_test.exs b/test/lightning_web/collections/refusals_test.exs new file mode 100644 index 00000000000..a7e519e4b1f --- /dev/null +++ b/test/lightning_web/collections/refusals_test.exs @@ -0,0 +1,461 @@ +defmodule LightningWeb.API.CollectionsRefusalsTest do + use LightningWeb.ConnCase, async: true + + import Lightning.Factories + import LightningWeb.API.ProblemHelpers + import LightningWeb.ApiCollectionsHelpers + + alias Lightning.Accounts + alias Lightning.Collections + alias Lightning.Extensions.Message + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + describe "who may reach it" do + setup :member_with_collection + + test "answers 404 to a token with no access to the project's collection", %{ + conn: conn, + collection: collection + } do + outsider = Accounts.generate_api_token(insert(:user)) + + for token <- [outsider, run_token()], + path <- [ + ~p"/collections/#{collection.name}", + ~p"/collections/#{collection.name}?project_id=#{collection.project_id}" + ] do + conn = conn |> assign_bearer(token) |> get(path) + + assert_problem(conn, 404, :not_found, []) + end + + conn = + conn + |> assign_bearer(run_token()) + |> put( + ~p"/collections/#{collection.name}/k?project_id=#{collection.project_id}", + value: "v" + ) + + assert_problem(conn, 404, :not_found, []) + assert Collections.get(collection, "k") == nil + end + + test "answers 401 to a token that was deleted, or to a disabled user", + %{conn: conn, user: user, collection: collection} do + token = Accounts.generate_api_token(user) + user_token = Repo.get_by(Accounts.UserToken, token: token) + {:ok, _} = Accounts.delete_token(user_token) + + deleted = + conn |> assign_bearer(token) |> get(~p"/collections/#{collection.name}") + + assert_problem(deleted, 401, :invalid_token, []) + + # A plain disable revokes sessions but not personal access tokens, so the + # blocked-user check is what refuses this one. + token = Accounts.generate_api_token(user) + {:ok, _user} = Accounts.update_user(user, %{disabled: true}) + + disabled = + conn |> assign_bearer(token) |> get(~p"/collections/#{collection.name}") + + assert_problem(disabled, 401, :invalid_token, []) + end + + test "checks the role before the query parameters", %{conn: conn} do + collection = insert(:collection, project: insert(:project)) + + conn = get(conn, ~p"/collections/#{collection.name}", limit: "ten") + + assert_problem(conn, 404, :not_found, []) + end + + test "answers 409 to a member of a project scheduled for deletion", %{ + conn: conn, + project: project, + collection: collection + } do + project + |> Ecto.Changeset.change( + scheduled_deletion: DateTime.utc_now() |> DateTime.truncate(:second) + ) + |> Repo.update!() + + conn = put(conn, ~p"/collections/#{collection.name}/k", value: "v") + + assert_problem(conn, 409, :scheduled_for_deletion, []) + end + end + + describe "who may reach it, by role" do + setup %{conn: conn} do + for role <- [:viewer, :editor, :admin, :owner] do + user = insert(:user) + project = insert(:project, project_users: [%{user: user, role: role}]) + + collection = + insert(:collection, + project: project, + items: [ + %{key: "foo", value: "bar"}, + %{key: "baz", value: "qux"} + ] + ) + + token = Lightning.Accounts.generate_api_token(user) + conn = assign_bearer(conn, token) + + {role, %{conn: conn, collection: collection}} + end + |> Map.new() + |> then(&{:ok, &1}) + end + + test "viewers may read but not write", ctx do + %{conn: conn, collection: collection} = ctx.viewer + + assert conn + |> get(~p"/collections/#{collection.name}/foo") + |> json_response(200) + + assert conn + |> get(~p"/collections/#{collection.name}") + |> json_response(200) + + for request <- [ + &put(&1, ~p"/collections/#{collection.name}/new", value: "v"), + &post(&1, ~p"/collections/#{collection.name}", + items: [%{key: "new", value: "v"}] + ), + &delete(&1, ~p"/collections/#{collection.name}/foo"), + &delete(&1, ~p"/collections/#{collection.name}") + ] do + assert_problem(request.(conn), 403, :forbidden, []) + end + + assert Collections.get(collection, "foo") + end + + test "editors may read and write but not delete every item", ctx do + %{conn: conn, collection: collection} = ctx.editor + + assert conn + |> put(~p"/collections/#{collection.name}/new", value: "v") + |> json_response(200) == %{"upserted" => 1, "error" => nil} + + assert %{"upserted" => 1} = + conn + |> post(~p"/collections/#{collection.name}", %{ + items: [%{key: "another", value: "v"}] + }) + |> json_response(200) + + assert %{"deleted" => 1} = + conn + |> delete(~p"/collections/#{collection.name}/foo") + |> json_response(200) + + conn = delete(conn, ~p"/collections/#{collection.name}") + + assert_problem(conn, 403, :forbidden, []) + + assert Collections.get(collection, "baz") + end + + for role <- [:admin, :owner] do + test "#{role}s may delete every item", ctx do + %{conn: conn, collection: collection} = ctx[unquote(role)] + + assert %{"deleted" => 2} = + conn + |> delete(~p"/collections/#{collection.name}") + |> json_response(200) + + refute Collections.get(collection, "foo") + end + end + end + + describe "refusing a body or a key" do + setup :member_with_collection + + test "PUT answers 422 for a value it cannot take", %{ + conn: conn, + collection: collection + } do + for {body, entry} <- [ + {%{}, + %{ + "pointer" => "#/value", + "code" => "required", + "detail" => "can't be blank" + }}, + {%{value: String.duplicate("x", 1_000_001)}, + %{ + "pointer" => "#/value", + "code" => "too_long", + "detail" => "should be at most 1000000 character(s)" + }} + ] do + conn = put(conn, ~p"/collections/#{collection.name}/key", body) + + assert_problem(conn, 422, :invalid, [entry]) + assert Collections.get(collection, "key") == nil + end + end + + test "POST answers 422 for items it cannot take", %{ + conn: conn, + collection: collection + } do + for {body, entry} <- [ + {%{}, + %{ + "pointer" => "#/items", + "code" => "required", + "detail" => "can't be blank" + }}, + {%{ + items: [ + %{key: "foo1", value: "bar1"}, + %{key: "foo1", value: "bar1"} + ] + }, + %{ + "pointer" => "#/items/1/key", + "code" => "duplicate", + "detail" => "names the same key as #/items/0" + }} + ] do + conn = post(conn, ~p"/collections/#{collection.name}", body) + + assert_problem(conn, 422, :invalid, [entry]) + end + end + + test "PUT with a key too long to store answers 422 with an entry naming the key's path segment", + %{conn: conn, collection: collection} do + key = String.duplicate("k", 256) + + conn = put(conn, ~p"/collections/#{collection.name}/#{key}", value: "v") + + assert_problem(conn, 422, :invalid, [ + %{ + "path" => "key", + "code" => "too_long", + "detail" => "key is too long, please use a shorter one" + } + ]) + + assert Collections.get(collection, key) == nil + end + + test "PUT with a bad key and a bad value answers both entries, the value's first", + %{conn: conn, collection: collection} do + conn = + put( + conn, + ~p"/collections/#{collection.name}/#{String.duplicate("k", 256)}", + value: %{"not" => "a string"} + ) + + entries = [ + %{ + "pointer" => "#/value", + "code" => "type", + "detail" => "is invalid" + }, + %{ + "path" => "key", + "code" => "too_long", + "detail" => "key is too long, please use a shorter one" + } + ] + + body = assert_problem(conn, 422, :invalid, entries) + + assert body["errors"] == entries + end + end + + describe "refusing query parameters" do + setup :member_with_collection + + test "GET answers 400 naming every bad parameter, quoting none", %{ + conn: conn, + collection: collection + } do + for {params, entries} <- [ + {[limit: "-5"], + [ + %{ + "parameter" => "limit", + "code" => "out_of_range", + "detail" => "must be greater than or equal to 0" + } + ]}, + {[ + limit: "ten", + cursor: "not-a-cursor", + colour: "red", + updated_after: "2025-01-01" + ], + [ + %{ + "parameter" => "colour", + "code" => "unknown_key", + "detail" => "is not accepted" + }, + %{ + "parameter" => "cursor", + "code" => "type", + "detail" => "is invalid" + }, + %{ + "parameter" => "limit", + "code" => "type", + "detail" => "is invalid" + }, + %{ + "parameter" => "updated_after", + "code" => "type", + "detail" => "is invalid" + } + ]} + ] do + conn = get(conn, ~p"/collections/#{collection.name}", params) + + assert_problem(conn, 400, :invalid_parameter, entries) + end + end + end + + describe "the plan limit" do + setup :member_with_collection + + test "answers 402 when a write would pass it", %{ + conn: conn, + collection: collection + } do + Mox.stub( + Lightning.Extensions.MockCollectionHook, + :handle_put_items, + fn _collection, _size -> + {:error, :exceeds_limit, %Message{text: "some limit error message"}} + end + ) + + for request <- [ + &put(&1, ~p"/collections/#{collection.name}/foo2", value: "bar22"), + &post(&1, ~p"/collections/#{collection.name}", + items: [ + %{key: "foo4", value: "bar4"}, + %{key: "foo5", value: "bar5"} + ] + ) + ] do + assert_problem(request.(conn), 402, :exceeds_limit, []) + end + end + + test "an invalid PUT over the limit answers 422, not 402", + %{conn: conn, collection: collection} do + Mox.stub(Lightning.Extensions.MockCollectionHook, :handle_put_items, fn _c, + _s -> + {:error, :exceeds_limit, %Message{text: "limit"}} + end) + + conn = + put(conn, ~p"/collections/#{collection.name}/k", + value: String.duplicate("x", 1_000_001) + ) + + assert_problem(conn, 422, :invalid, [ + %{ + "pointer" => "#/value", + "code" => "too_long", + "detail" => "should be at most 1000000 character(s)" + } + ]) + end + end + + describe "never quoting a request value" do + setup :member_with_collection + + test "no answer quotes a value from the request", + %{conn: conn, collection: collection} do + sentinel = plant_sentinel(%{"x" => 1})["x"] + + conn + |> get( + ~p"/collections/#{collection.name}?#{plant_sentinel(%{"limit" => 1, "cursor" => 1, "updated_after" => 1})}" + ) + |> refute_sentinel() + |> assert_problem(400, :invalid_parameter, [ + %{"parameter" => "cursor", "code" => "type", "detail" => "is invalid"}, + %{"parameter" => "limit", "code" => "type", "detail" => "is invalid"}, + %{ + "parameter" => "updated_after", + "code" => "type", + "detail" => "is invalid" + } + ]) + + conn + |> put( + ~p"/collections/#{collection.name}/#{String.duplicate(sentinel, 20)}", + plant_sentinel(%{"value" => 1}) + ) + |> refute_sentinel() + |> assert_problem(422, :invalid, [ + %{ + "path" => "key", + "code" => "too_long", + "detail" => "key is too long, please use a shorter one" + } + ]) + + conn + |> post( + ~p"/collections/#{collection.name}", + plant_sentinel(%{ + "items" => [%{"key" => 1, "value" => 1}, %{"key" => 1, "value" => 1}] + }) + ) + |> refute_sentinel() + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/items/1/key", + "code" => "duplicate", + "detail" => "names the same key as #/items/0" + } + ]) + + conn + |> post( + ~p"/collections/#{collection.name}", + %{"items" => [%{"key" => 1, "value" => %{"a" => 1}}]} + |> plant_sentinel() + |> put_in(["items", Access.at(0), "value"], %{"a" => sentinel}) + ) + |> refute_sentinel() + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/items/0/value", + "code" => "type", + "detail" => "is invalid" + } + ]) + + conn + |> get(~p"/collections/#{collection.name}?#{%{project_id: sentinel}}") + |> refute_sentinel() + |> assert_problem(404, :not_found, []) + end + end +end diff --git a/test/lightning_web/collections/streaming_test.exs b/test/lightning_web/collections/streaming_test.exs new file mode 100644 index 00000000000..f39da44e1f9 --- /dev/null +++ b/test/lightning_web/collections/streaming_test.exs @@ -0,0 +1,348 @@ +defmodule LightningWeb.API.CollectionsStreamingTest do + use LightningWeb.ConnCase, async: true + + import Lightning.Factories + import LightningWeb.ApiCollectionsHelpers + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + setup :member_with_collection + + describe "listing a collection" do + test "answers no items and no cursor for an empty collection", %{ + conn: conn, + collection: collection + } do + conn = get(conn, ~p"/collections/#{collection.name}") + + assert json_response(conn, 200) == %{"items" => [], "cursor" => nil} + end + + test "streams the answer in chunks", %{conn: conn, collection: collection} do + conn = get(conn, ~p"/collections/#{collection.name}") + + assert conn.state == :chunked + end + + test "narrows by key pattern", %{conn: conn, collection: collection} do + insert(:collection_item, collection: collection, key: "foo:bar:baz") + insert(:collection_item, collection: collection, key: "foo:bar:baz:quux") + + for {pattern, keys} <- [ + {"foo:bar:*", ["foo:bar:baz", "foo:bar:baz:quux"]}, + {"foo:*:baz", ["foo:bar:baz"]} + ] do + conn = get(conn, ~p"/collections/#{collection.name}?#{%{key: pattern}}") + + assert %{"items" => items, "cursor" => nil} = json_response(conn, 200) + assert Enum.map(items, & &1["key"]) == keys + end + end + + test "narrows by key pattern and creation time", %{ + conn: conn, + collection: collection + } do + before_insert = DateTime.utc_now() + + insert(:collection_item, collection: collection, key: "foo:bar:baz") + insert(:collection_item, collection: collection, key: "foo:moon:baz") + insert(:collection_item, collection: collection, key: "foo:bar:baz:out") + + after_insert = DateTime.utc_now() |> DateTime.add(10, :millisecond) + + for {params, keys} <- [ + {[created_after: DateTime.to_iso8601(after_insert)], []}, + {[created_before: DateTime.to_iso8601(before_insert)], []}, + {[ + created_after: DateTime.to_iso8601(before_insert), + created_before: DateTime.to_iso8601(after_insert) + ], ["foo:bar:baz", "foo:moon:baz"]} + ] do + conn = + get( + conn, + ~p"/collections/#{collection.name}", + [key: "foo:*:baz"] ++ params + ) + + assert %{"items" => items, "cursor" => nil} = json_response(conn, 200) + assert Enum.map(items, & &1["key"]) == keys + end + end + + test "narrows by key pattern and update time", %{ + conn: conn, + collection: collection + } do + before_insert = DateTime.utc_now() + + insert(:collection_item, collection: collection, key: "foo:bar:baz") + insert(:collection_item, collection: collection, key: "foo:moon:baz") + insert(:collection_item, collection: collection, key: "foo:bar:baz:out") + + after_insert = DateTime.utc_now() |> DateTime.add(10, :millisecond) + + insert(:collection_item, + collection: collection, + key: "foo:in:baz", + updated_at: DateTime.add(after_insert, 1, :millisecond) + ) + + for {params, keys} <- [ + {[updated_after: DateTime.to_iso8601(after_insert)], ["foo:in:baz"]}, + {[updated_before: DateTime.to_iso8601(before_insert)], []}, + {[ + updated_after: DateTime.to_iso8601(before_insert), + updated_before: DateTime.to_iso8601(after_insert) + ], ["foo:bar:baz", "foo:moon:baz"]} + ] do + conn = + get( + conn, + ~p"/collections/#{collection.name}", + [key: "foo:*:baz"] ++ params + ) + + assert %{"items" => items, "cursor" => nil} = json_response(conn, 200) + assert Enum.map(items, & &1["key"]) == keys + end + end + + test "narrows by creation time", %{conn: conn, collection: collection} do + before_insert = DateTime.utc_now() + items = insert_list(3, :collection_item, collection: collection) + after_insert = DateTime.utc_now() |> DateTime.add(10, :millisecond) + + items = Enum.map(items, &encode_decode/1) + + for params <- [ + [created_after: DateTime.to_iso8601(before_insert)], + [created_before: DateTime.to_iso8601(after_insert)], + [] + ] do + conn = get(conn, ~p"/collections/#{collection.name}", params) + + assert json_response(conn, 200) == %{"items" => items, "cursor" => nil} + end + end + + test "narrows by update time", %{conn: conn, collection: collection} do + old_item = + insert(:collection_item, collection: collection, key: "old-key") + |> encode_decode() + + before_insert_list = DateTime.utc_now() + + items = + insert_list(3, :collection_item, collection: collection) + |> Enum.map(&encode_decode/1) + + after_insert = DateTime.utc_now() |> DateTime.add(10, :millisecond) + + for {params, expected} <- [ + {[updated_after: DateTime.to_iso8601(before_insert_list)], items}, + {[updated_before: DateTime.to_iso8601(after_insert)], + [old_item | items]}, + {[], [old_item | items]} + ] do + conn = get(conn, ~p"/collections/#{collection.name}", params) + + assert json_response(conn, 200) == %{ + "items" => expected, + "cursor" => nil + } + end + end + end + + describe "page size and cursors" do + test "up to the default limit", %{conn: conn, collection: collection} do + default_limit = 25 + + Mox.stub(Lightning.MockConfig, :collections_default_stream_limit, fn -> + default_limit + end) + + items = insert_list(3, :collection_item, collection: collection) + + conn = get(conn, ~p"/collections/#{collection.name}") + + items = Enum.map(items, &encode_decode/1) + + assert json_response(conn, 200) == %{"items" => items, "cursor" => nil} + + # Insert more to bring the total to the stream limit + items = + Enum.concat( + items, + insert_list( + default_limit - length(items), + :collection_item, + collection: collection + ) + |> Enum.map(&encode_decode/1) + ) + + conn = get(conn, ~p"/collections/#{collection.name}") + + assert json_response(conn, 200) == %{"items" => items, "cursor" => nil} + end + + test "up to a limit from params", %{conn: conn, collection: collection} do + page_size = 15 + + Mox.stub(Lightning.MockConfig, :collections_stream_page_size, fn -> + page_size + end) + + limit = page_size + 10 + + all_items = + insert_list(limit + 1, :collection_item, collection: collection) + |> Enum.map(&encode_decode/1) + + conn = get(conn, ~p"/collections/#{collection.name}", limit: limit) + + assert %{"items" => items, "cursor" => cursor} = json_response(conn, 200) + assert items == Enum.take(all_items, limit) + assert cursor + + conn = + get(conn, ~p"/collections/#{collection.name}", + limit: limit, + cursor: cursor + ) + + assert json_response(conn, 200) == %{ + "items" => Enum.drop(all_items, limit), + "cursor" => nil + } + + # A limit below the page size still leaves a cursor to the rest + half_limit = div(page_size, 2) + + conn = get(conn, ~p"/collections/#{collection.name}", limit: half_limit) + + assert %{"items" => items, "cursor" => cursor} = json_response(conn, 200) + assert items == Enum.take(all_items, half_limit) + assert cursor + + # Request everything, shouldn't be getting a cursor + conn = get(conn, ~p"/collections/#{collection.name}", limit: limit + 1) + + assert json_response(conn, 200) == %{"items" => all_items, "cursor" => nil} + end + + test "answers no items for a limit of 0", %{ + conn: conn, + collection: collection + } do + insert(:collection_item, collection: collection) + + conn = get(conn, ~p"/collections/#{collection.name}", limit: "0") + + assert json_response(conn, 200) == %{"items" => [], "cursor" => nil} + end + + test "up to the default limit and returning a cursor", %{ + conn: conn, + collection: collection + } do + default_limit = 25 + + Mox.stub(Lightning.MockConfig, :collections_default_stream_limit, fn -> + default_limit + end) + + all_items = + insert_list(default_limit + 1, :collection_item, collection: collection) + |> Enum.map(&encode_decode/1) + + conn = get(conn, ~p"/collections/#{collection.name}") + + assert %{"items" => items, "cursor" => cursor} = json_response(conn, 200) + assert items == Enum.take(all_items, default_limit) + assert cursor + + conn = get(conn, ~p"/collections/#{collection.name}", cursor: cursor) + + assert json_response(conn, 200) == %{ + "items" => Enum.drop(all_items, default_limit), + "cursor" => nil + } + end + + test "up to the limit from a cursor returning a cursor", %{ + conn: conn, + collection: collection + } do + limit = 100 + + all_items = + insert_list(2 * limit + 1, :collection_item, collection: collection) + |> Enum.map(&encode_decode/1) + + conn = get(conn, ~p"/collections/#{collection.name}", limit: limit) + + assert %{"items" => items, "cursor" => cursor} = json_response(conn, 200) + assert items == Enum.take(all_items, limit) + + conn = + get(conn, ~p"/collections/#{collection.name}", + cursor: cursor, + limit: limit + ) + + assert %{"items" => items, "cursor" => cursor} = json_response(conn, 200) + assert items == all_items |> Enum.drop(limit) |> Enum.take(limit) + assert cursor + + conn = + get(conn, ~p"/collections/#{collection.name}", + cursor: cursor, + limit: limit + ) + + assert json_response(conn, 200) == %{ + "items" => Enum.drop(all_items, 2 * limit), + "cursor" => nil + } + end + + test "up exactly to the limit from a cursor", %{ + conn: conn, + collection: collection + } do + limit = 25 + + all_items = + insert_list(2 * limit, :collection_item, collection: collection) + |> Enum.map(&encode_decode/1) + + conn = get(conn, ~p"/collections/#{collection.name}", limit: limit) + + assert %{"items" => items, "cursor" => cursor} = json_response(conn, 200) + assert items == Enum.take(all_items, limit) + + conn = + get(conn, ~p"/collections/#{collection.name}", + cursor: cursor, + limit: limit + ) + + assert json_response(conn, 200) == %{ + "items" => Enum.drop(all_items, limit), + "cursor" => nil + } + end + end + + defp encode_decode(item) do + item + |> Jason.encode!() + |> Jason.decode!() + end +end diff --git a/test/lightning_web/collections_controller_test.exs b/test/lightning_web/collections_controller_test.exs deleted file mode 100644 index 371cb6e367b..00000000000 --- a/test/lightning_web/collections_controller_test.exs +++ /dev/null @@ -1,1760 +0,0 @@ -defmodule LightningWeb.API.CollectionsControllerTest do - use LightningWeb.ConnCase, async: true - - import Lightning.Factories - - alias Lightning.Collections - alias Lightning.Extensions.Message - - @limits Application.compile_env!( - :lightning, - LightningWeb.CollectionsController - ) - - @default_stream_limit @limits[:default_stream_limit] - @max_database_limit @limits[:max_database_limit] - - setup %{conn: conn} do - {:ok, conn: put_req_header(conn, "accept", "application/json")} - end - - test "without a token", %{conn: conn} do - conn = get(conn, ~p"/collections/foo") - - assert %{"error" => "Unauthorized"} == json_response(conn, 401) - end - - describe "authenticating with a run token" do - test "with a token that is invalid", %{conn: conn} do - workflow = insert(:simple_workflow) - workorder = insert(:workorder, dataclip: insert(:dataclip)) - - collection = insert(:collection, project: workflow.project) - - run = - insert(:run, - work_order: workorder, - dataclip: workorder.dataclip, - starting_trigger: workflow.triggers |> hd() - ) - - token = Lightning.Workers.generate_run_token(run) - - conn = - conn - |> assign_bearer(token) - |> get(~p"/collections/#{collection.name}") - - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - end - - describe "authenticating with a personal access token" do - test "with a project they don't have access to", %{conn: conn} do - user = insert(:user) - - project = - insert(:project, project_users: []) - - collection = insert(:collection, project: project) - - token = Lightning.Accounts.generate_api_token(user) - - conn = - conn - |> assign_bearer(token) - |> get(~p"/collections/#{collection.name}") - - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - - test "with a live token and project access", %{conn: conn} do - user = insert(:user) - project = insert(:project, project_users: [%{user: user}]) - collection = insert(:collection, project: project) - - token = Lightning.Accounts.generate_api_token(user) - - conn = - conn - |> assign_bearer(token) - |> get(~p"/collections/#{collection.name}") - - assert json_response(conn, 200) == %{"items" => [], "cursor" => nil} - end - - # The user keeps project access, so revocation is the only thing that can - # cause the 401. - test "with a deleted token, despite project access", %{conn: conn} do - user = insert(:user) - project = insert(:project, project_users: [%{user: user}]) - collection = insert(:collection, project: project) - - token = Lightning.Accounts.generate_api_token(user) - - user_token = Repo.get_by(Lightning.Accounts.UserToken, token: token) - {:ok, _} = Lightning.Accounts.delete_token(user_token) - - conn = - conn - |> assign_bearer(token) - |> get(~p"/collections/#{collection.name}") - - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - - # A plain disable revokes only sessions, not personal access tokens, so - # this PAT stays valid — the verify/1 blocked-user gate is what returns the - # 401. - test "with a disabled user, despite project access", %{conn: conn} do - user = insert(:user) - project = insert(:project, project_users: [%{user: user}]) - collection = insert(:collection, project: project) - - token = Lightning.Accounts.generate_api_token(user) - - {:ok, _user} = - Lightning.Accounts.update_user(user, %{disabled: true}) - - conn = - conn - |> assign_bearer(token) - |> get(~p"/collections/#{collection.name}") - - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - - # The token is untouched and the project access is unchanged, so the - # confirmation deadline is the only thing that can cause the 401 — and - # confirming the address is enough to bring the same token back. - test "with an account past its confirmation deadline, despite project access", - %{conn: conn} do - Mox.stub(Lightning.MockConfig, :check_flag?, fn - :require_email_verification -> true - flag -> Lightning.Config.API.check_flag?(flag) - end) - - user = insert(:user) - project = insert(:project, project_users: [%{user: user}]) - collection = insert(:collection, project: project) - - token = Lightning.Accounts.generate_api_token(user) - - user = - user - |> Ecto.Changeset.change( - confirmed_at: nil, - inserted_at: - DateTime.utc_now() - |> Timex.shift(hours: -50) - |> DateTime.truncate(:second) - ) - |> Lightning.Repo.update!() - - refused = - conn - |> assign_bearer(token) - |> get(~p"/collections/#{collection.name}") - - assert json_response(refused, 401) == %{"error" => "Unauthorized"} - - user - |> Ecto.Changeset.change( - confirmed_at: DateTime.utc_now() |> DateTime.truncate(:second) - ) - |> Lightning.Repo.update!() - - allowed = - conn - |> assign_bearer(token) - |> get(~p"/collections/#{collection.name}") - - assert json_response(allowed, 200) == %{"items" => [], "cursor" => nil} - end - end - - describe "GET /collections/:name/:key" do - test "returns the item", %{conn: conn} do - user = insert(:user) - - project = - insert(:project, project_users: [%{user: user}]) - - collection = - insert(:collection, - project: project, - items: [%{key: "foo", value: "bar"}] - ) - - token = Lightning.Accounts.generate_api_token(user) - - conn = - conn - |> assign_bearer(token) - |> get(~p"/collections/#{collection.name}/foo") - - item = hd(collection.items) - - assert json_response(conn, 200) == %{ - "key" => item.key, - "value" => item.value, - "created" => DateTime.to_iso8601(item.inserted_at), - "updated" => DateTime.to_iso8601(item.updated_at) - } - end - - test "returns 404 when the collection doesn't exist", %{conn: conn} do - user = insert(:user) - - project = - insert(:project, project_users: [%{user: user}]) - - _another_collection = insert(:collection, project: project) - - token = Lightning.Accounts.generate_api_token(user) - - conn = - conn - |> assign_bearer(token) - |> get(~p"/collections/misspelled-collection/foo") - - assert json_response(conn, 404) == %{"error" => "Not Found"} - end - - test "returns 404 when the item doesn't exist", %{conn: conn} do - user = insert(:user) - - project = - insert(:project, project_users: [%{user: user}]) - - collection = insert(:collection, project: project) - - token = Lightning.Accounts.generate_api_token(user) - - conn = - conn - |> assign_bearer(token) - |> get(~p"/collections/#{collection.name}/some-unexisting-key") - - assert response(conn, 204) == "" - end - end - - describe "PUT /collections/:name/:key" do - test "inserts an item", %{conn: conn} do - user = insert(:user) - - project = - insert(:project, project_users: [%{user: user}]) - - collection = - insert(:collection, - project: project, - items: [%{key: "foo", value: "bar"}] - ) - - token = Lightning.Accounts.generate_api_token(user) - - conn = - conn - |> assign_bearer(token) - |> put(~p"/collections/#{collection.name}/baz", value: "qux") - - assert json_response(conn, 200) == %{ - "upserted" => 1, - "error" => nil - } - end - - test "updates an item", %{conn: conn} do - user = insert(:user) - - project = - insert(:project, project_users: [%{user: user}]) - - collection = - insert(:collection, - project: project, - items: [%{key: "foo", value: "bar"}] - ) - - token = Lightning.Accounts.generate_api_token(user) - - conn = - conn - |> assign_bearer(token) - |> put(~p"/collections/#{collection.name}/foo", value: "qux2") - - assert json_response(conn, 200) == %{ - "upserted" => 1, - "error" => nil - } - end - - test "returns 404 when the collection doesn't exist", %{conn: conn} do - user = insert(:user) - - project = - insert(:project, project_users: [%{user: user}]) - - _another_collection = insert(:collection, project: project) - - token = Lightning.Accounts.generate_api_token(user) - - conn = - conn - |> assign_bearer(token) - |> put(~p"/collections/misspelled-collection/baz", value: "qux") - - assert json_response(conn, 404) == %{"error" => "Not Found"} - end - - test "returns 422 when request exceeds the limit", %{conn: conn} do - user = insert(:user) - - project = - insert(:project, project_users: [%{user: user}]) - - collection = - insert(:collection, - project: project, - items: Enum.map(1..2, &%{key: "foo#{&1}", value: "bar#{&1}"}) - ) - - Mox.stub( - Lightning.Extensions.MockCollectionHook, - :handle_put_items, - fn _collection, _size -> - {:error, :exceeds_limit, %Message{text: "some limit error message"}} - end - ) - - conn = - conn - |> assign_bearer(Lightning.Accounts.generate_api_token(user)) - |> put(~p"/collections/#{collection.name}/foo2", value: "bar22") - - assert json_response(conn, 422) == %{ - "upserted" => 0, - "error" => "some limit error message" - } - end - - test "returns 'Format error' when the value fails validation", %{conn: conn} do - user = insert(:user) - project = insert(:project, project_users: [%{user: user}]) - collection = insert(:collection, project: project) - token = Lightning.Accounts.generate_api_token(user) - - oversized = String.duplicate("x", 1_000_001) - - conn = - conn - |> assign_bearer(token) - |> put(~p"/collections/#{collection.name}/key", value: oversized) - - assert json_response(conn, 200) == %{ - "upserted" => 0, - "error" => "Format error" - } - end - end - - describe "POST /collections/:name" do - test "inserts multiple items with same timestamp", %{conn: conn} do - user = insert(:user) - - project = - insert(:project, project_users: [%{user: user}]) - - collection = insert(:collection, project: project) - - token = Lightning.Accounts.generate_api_token(user) - - conn = assign_bearer(conn, token) - - assert %{"upserted" => 1_000, "error" => nil} = - post(conn, ~p"/collections/#{collection.name}", %{ - items: - Enum.map(1..1_000, &%{key: "foo#{&1}", value: "bar#{&1}"}) - }) - |> json_response(200) - - assert %{"items" => items, "cursor" => nil} = - get(conn, ~p"/collections/#{collection.name}", limit: 1_000) - |> json_response(200) - - assert MapSet.new(items, & &1["created"]) |> MapSet.size() == 1 - assert Enum.count(items) == 1_000 - end - - test "upserted multiple items", %{conn: conn} do - user = insert(:user) - - project = - insert(:project, project_users: [%{user: user}]) - - collection = - insert(:collection, - project: project, - items: Enum.map(1..3, &%{key: "foo#{&1}", value: "bar#{&1}"}) - ) - - token = Lightning.Accounts.generate_api_token(user) - - conn = - conn - |> assign_bearer(token) - |> post(~p"/collections/#{collection.name}", %{ - items: Enum.map(1..10, &%{key: "foo#{&1}", value: "bar#{&1}"}) - }) - - assert json_response(conn, 200) == %{ - "upserted" => 10, - "error" => nil - } - end - - test "returns 404 when the collection doesn't exist", %{conn: conn} do - user = insert(:user) - - project = - insert(:project, project_users: [%{user: user}]) - - _another_collection = insert(:collection, project: project) - - token = Lightning.Accounts.generate_api_token(user) - - conn = - conn - |> assign_bearer(token) - |> post(~p"/collections/misspelled-collection", %{ - items: [%{key: "baz", value: "qux"}] - }) - - assert json_response(conn, 404) == %{"error" => "Not Found"} - end - - test "returns 422 when request exceeds the limit", %{conn: conn} do - user = insert(:user) - - project = - insert(:project, project_users: [%{user: user}]) - - collection = - insert(:collection, - project: project, - items: Enum.map(1..3, &%{key: "foo#{&1}", value: "bar#{&1}"}) - ) - - Mox.stub( - Lightning.Extensions.MockCollectionHook, - :handle_put_items, - fn _collection, _size -> - {:error, :exceeds_limit, %Message{text: "some limit error message"}} - end - ) - - conn = - conn - |> assign_bearer(Lightning.Accounts.generate_api_token(user)) - |> post(~p"/collections/#{collection.name}", %{ - items: [%{key: "foo4", value: "bar4"}, %{key: "foo5", value: "bar5"}] - }) - - assert json_response(conn, 422) == %{ - "upserted" => 0, - "error" => "some limit error message" - } - end - - test "returns 422 when a key is referenced twice", %{conn: conn} do - user = insert(:user) - - project = - insert(:project, project_users: [%{user: user}]) - - collection = - insert(:collection, - project: project, - items: Enum.map(1..3, &%{key: "foo#{&1}", value: "bar#{&1}"}) - ) - - token = Lightning.Accounts.generate_api_token(user) - - conn = - conn - |> assign_bearer(token) - |> post(~p"/collections/#{collection.name}", %{ - items: [%{key: "foo1", value: "bar1"}, %{key: "foo1", value: "bar1"}] - }) - - assert json_response(conn, 422) == %{ - "upserted" => 0, - "error" => "Duplicate key found" - } - end - end - - describe "DELETE" do - test "deletes an item", %{conn: conn} do - user = insert(:user) - - project = - insert(:project, project_users: [%{user: user}]) - - collection = - insert(:collection, - project: project, - items: [%{key: "foo", value: "bar"}] - ) - - token = Lightning.Accounts.generate_api_token(user) - - conn = - conn - |> assign_bearer(token) - |> delete(~p"/collections/#{collection.name}/foo") - - assert json_response(conn, 200) == %{ - "key" => "foo", - "deleted" => 1, - "error" => nil - } - end - - test "returns 'Item Not Found' when deleting a missing key", %{conn: conn} do - user = insert(:user) - project = insert(:project, project_users: [%{user: user}]) - collection = insert(:collection, project: project) - token = Lightning.Accounts.generate_api_token(user) - - conn = - conn - |> assign_bearer(token) - |> delete(~p"/collections/#{collection.name}/missing-key") - - assert json_response(conn, 200) == %{ - "key" => "missing-key", - "deleted" => 0, - "error" => "Item Not Found" - } - end - - test "deletes matching items", %{conn: conn} do - user = insert(:user) - - project = - insert(:project, project_users: [%{user: user, role: :owner}]) - - collection = - insert(:collection, - project: project, - items: [ - %{key: "foo:123:bar1", value: "value1"}, - %{key: "foo:234:boo", value: "value2"}, - %{key: "foo:345:bar2", value: "value3"} - ] - ) - - token = Lightning.Accounts.generate_api_token(user) - - conn = - conn - |> assign_bearer(token) - |> delete(~p"/collections/#{collection.name}", key: "foo:*:bar*") - - assert json_response(conn, 200) == %{ - "key" => "foo:*:bar*", - "deleted" => 2, - "error" => nil - } - end - - test "returns 404 when the collection doesn't exist", %{conn: conn} do - user = insert(:user) - - project = - insert(:project, project_users: [%{user: user}]) - - _another_collection = insert(:collection, project: project) - - token = Lightning.Accounts.generate_api_token(user) - - conn = - conn - |> assign_bearer(token) - |> delete(~p"/collections/misspelled-collection/foo") - - assert json_response(conn, 404) == %{"error" => "Not Found"} - end - end - - describe "authorization by project role" do - setup %{conn: conn} do - for role <- [:viewer, :editor, :admin, :owner] do - user = insert(:user) - project = insert(:project, project_users: [%{user: user, role: role}]) - - collection = - insert(:collection, - project: project, - items: [ - %{key: "foo", value: "bar"}, - %{key: "baz", value: "qux"} - ] - ) - - token = Lightning.Accounts.generate_api_token(user) - conn = assign_bearer(conn, token) - - {role, %{conn: conn, collection: collection}} - end - |> Map.new() - |> then(&{:ok, &1}) - end - - test "viewers may read but not write", ctx do - %{conn: conn, collection: collection} = ctx.viewer - - # reads are allowed - assert conn - |> get(~p"/collections/#{collection.name}/foo") - |> json_response(200) - - assert conn - |> get(~p"/collections/#{collection.name}") - |> json_response(200) - - # writes are rejected - assert conn - |> put(~p"/collections/#{collection.name}/new", value: "v") - |> json_response(401) == %{"error" => "Unauthorized"} - - assert conn - |> post(~p"/collections/#{collection.name}", %{ - items: [%{key: "new", value: "v"}] - }) - |> json_response(401) == %{"error" => "Unauthorized"} - - assert conn - |> delete(~p"/collections/#{collection.name}/foo") - |> json_response(401) == %{"error" => "Unauthorized"} - - # delete_all (no key) must not wipe the collection - assert conn - |> delete(~p"/collections/#{collection.name}") - |> json_response(401) == %{"error" => "Unauthorized"} - - assert Collections.get(collection, "foo") - end - - test "editors may read and write but not delete_all", ctx do - %{conn: conn, collection: collection} = ctx.editor - - assert conn - |> put(~p"/collections/#{collection.name}/new", value: "v") - |> json_response(200) == %{"upserted" => 1, "error" => nil} - - assert %{"upserted" => 1} = - conn - |> post(~p"/collections/#{collection.name}", %{ - items: [%{key: "another", value: "v"}] - }) - |> json_response(200) - - assert %{"deleted" => 1} = - conn - |> delete(~p"/collections/#{collection.name}/foo") - |> json_response(200) - - # wiping the whole collection is reserved for owners/admins - assert conn - |> delete(~p"/collections/#{collection.name}") - |> json_response(401) == %{"error" => "Unauthorized"} - - assert Collections.get(collection, "baz") - end - - for role <- [:admin, :owner] do - test "#{role}s may delete_all", ctx do - %{conn: conn, collection: collection} = ctx[unquote(role)] - - assert %{"deleted" => 2} = - conn - |> delete(~p"/collections/#{collection.name}") - |> json_response(200) - - refute Collections.get(collection, "foo") - end - end - end - - describe "GET /collections/:name" do - test "with no results", %{conn: conn} do - user = insert(:user) - - project = - insert(:project, project_users: [%{user: user}]) - - collection = insert(:collection, project: project) - - token = Lightning.Accounts.generate_api_token(user) - - conn = - conn - |> assign_bearer(token) - |> get(~p"/collections/#{collection.name}") - - assert json_response(conn, 200) == %{"items" => [], "cursor" => nil} - end - - test "using a key pattern", %{conn: conn} do - user = insert(:user) - - project = - insert(:project, project_users: [%{user: user}]) - - collection = insert(:collection, project: project) - - insert(:collection_item, collection: collection, key: "foo:bar:baz") - insert(:collection_item, collection: collection, key: "foo:bar:baz:quux") - - token = Lightning.Accounts.generate_api_token(user) - - conn = assign_bearer(conn, token) - - assert %{state: :chunked} = - conn = - get( - conn, - ~p"/collections/#{collection.name}?#{%{key: "foo:bar:*"}}" - ) - - assert %{ - "items" => [ - %{"key" => "foo:bar:baz", "value" => _}, - %{"key" => "foo:bar:baz:quux", "value" => _} - ], - "cursor" => nil - } = json_response(conn, 200) - - assert %{state: :chunked} = - conn = - get( - conn, - ~p"/collections/#{collection.name}?#{%{key: "foo:*:baz"}}" - ) - - assert %{ - "items" => [%{"key" => "foo:bar:baz", "value" => _}], - "cursor" => nil - } = json_response(conn, 200) - end - - test "using a key pattern and creation filters", %{conn: conn} do - user = insert(:user) - - project = - insert(:project, project_users: [%{user: user}]) - - collection = insert(:collection, project: project) - - before_insert = DateTime.utc_now() - - insert(:collection_item, collection: collection, key: "foo:bar:baz") - insert(:collection_item, collection: collection, key: "foo:moon:baz") - insert(:collection_item, collection: collection, key: "foo:bar:baz:out") - - after_insert = DateTime.utc_now() |> DateTime.add(10, :millisecond) - - token = Lightning.Accounts.generate_api_token(user) - - conn = assign_bearer(conn, token) - - assert %{ - "items" => [], - "cursor" => nil - } = - conn - |> get(~p"/collections/#{collection.name}", - key: "foo:*:baz", - created_after: DateTime.to_iso8601(after_insert) - ) - |> json_response(200) - - assert %{ - "items" => [], - "cursor" => nil - } = - conn - |> get(~p"/collections/#{collection.name}", - key: "foo:*:baz", - created_before: DateTime.to_iso8601(before_insert) - ) - |> json_response(200) - - assert %{ - "items" => [ - %{"key" => "foo:bar:baz", "value" => _}, - %{"key" => "foo:moon:baz", "value" => _} - ], - "cursor" => nil - } = - conn - |> get(~p"/collections/#{collection.name}", - key: "foo:*:baz", - created_after: DateTime.to_iso8601(before_insert), - created_before: DateTime.to_iso8601(after_insert) - ) - |> json_response(200) - end - - test "using a key pattern and update filters", %{conn: conn} do - user = insert(:user) - - project = - insert(:project, project_users: [%{user: user}]) - - collection = insert(:collection, project: project) - - before_insert = DateTime.utc_now() - - insert(:collection_item, collection: collection, key: "foo:bar:baz") - insert(:collection_item, collection: collection, key: "foo:moon:baz") - insert(:collection_item, collection: collection, key: "foo:bar:baz:out") - - after_insert = DateTime.utc_now() |> DateTime.add(10, :millisecond) - - insert(:collection_item, - collection: collection, - key: "foo:in:baz", - updated_at: DateTime.add(after_insert, 1, :millisecond) - ) - - token = Lightning.Accounts.generate_api_token(user) - - conn = assign_bearer(conn, token) - - assert %{ - "items" => [%{"key" => "foo:in:baz", "value" => _}], - "cursor" => nil - } = - conn - |> get(~p"/collections/#{collection.name}", - key: "foo:*:baz", - updated_after: DateTime.to_iso8601(after_insert) - ) - |> json_response(200) - - assert %{ - "items" => [], - "cursor" => nil - } = - conn - |> get(~p"/collections/#{collection.name}", - key: "foo:*:baz", - updated_before: DateTime.to_iso8601(before_insert) - ) - |> json_response(200) - - assert %{ - "items" => [ - %{"key" => "foo:bar:baz", "value" => _}, - %{"key" => "foo:moon:baz", "value" => _} - ], - "cursor" => nil - } = - conn - |> get(~p"/collections/#{collection.name}", - key: "foo:*:baz", - updated_after: DateTime.to_iso8601(before_insert), - updated_before: DateTime.to_iso8601(after_insert) - ) - |> json_response(200) - end - - test "using creation filters", %{conn: conn} do - user = insert(:user) - project = insert(:project, project_users: [%{user: user}]) - - before_insert = DateTime.utc_now() - - collection = - insert(:collection, - project: project, - items: insert_list(3, :collection_item) - ) - - after_insert = DateTime.utc_now() |> DateTime.add(10, :millisecond) - - token = Lightning.Accounts.generate_api_token(user) - - conn = assign_bearer(conn, token) - - items = Enum.map(collection.items, &encode_decode/1) - - assert %{ - "items" => ^items, - "cursor" => nil - } = - conn - |> get(~p"/collections/#{collection.name}", - created_after: DateTime.to_iso8601(before_insert) - ) - |> json_response(200) - - assert %{ - "items" => ^items, - "cursor" => nil - } = - conn - |> get(~p"/collections/#{collection.name}", - created_before: DateTime.to_iso8601(after_insert) - ) - |> json_response(200) - - items = Enum.map(collection.items, &encode_decode/1) - - assert %{ - "items" => ^items, - "cursor" => nil - } = - conn - |> get(~p"/collections/#{collection.name}") - |> json_response(200) - end - - test "using update filters", %{conn: conn} do - user = insert(:user) - project = insert(:project, project_users: [%{user: user}]) - - collection = insert(:collection, project: project) - - old_item = - insert(:collection_item, collection: collection, key: "old-key") - |> encode_decode() - - before_insert_list = DateTime.utc_now() - - items = - insert_list(3, :collection_item, collection: collection) - |> Enum.map(&encode_decode/1) - - after_insert = DateTime.utc_now() |> DateTime.add(10, :millisecond) - - token = Lightning.Accounts.generate_api_token(user) - - conn = assign_bearer(conn, token) - - assert conn - |> get(~p"/collections/#{collection.name}", - updated_after: - DateTime.to_date(before_insert_list) |> Date.to_iso8601() - ) - |> json_response(400) - - assert %{ - "items" => ^items, - "cursor" => nil - } = - conn - |> get(~p"/collections/#{collection.name}", - updated_after: DateTime.to_iso8601(before_insert_list) - ) - |> json_response(200) - - expected_items = [old_item | items] - - assert %{ - "items" => ^expected_items, - "cursor" => nil - } = - conn - |> get(~p"/collections/#{collection.name}", - updated_before: DateTime.to_iso8601(after_insert) - ) - |> json_response(200) - - assert %{ - "items" => ^expected_items, - "cursor" => nil - } = - conn - |> get(~p"/collections/#{collection.name}") - |> json_response(200) - end - - test "up to the default limit", %{conn: conn} do - user = insert(:user) - project = insert(:project, project_users: [%{user: user}]) - - collection = - insert(:collection, - project: project, - items: insert_list(3, :collection_item) - ) - - token = Lightning.Accounts.generate_api_token(user) - - conn = - conn - |> assign_bearer(token) - |> get(~p"/collections/#{collection.name}") - - assert conn.state == :chunked - - items = - collection.items - |> Enum.map(&encode_decode/1) - - assert json_response(conn, 200) == %{ - "items" => items, - "cursor" => nil - } - - # Insert more to bring the total to the stream limit - items = - Enum.concat( - items, - insert_list( - @default_stream_limit - length(collection.items), - :collection_item, - collection: collection - ) - |> Enum.map(&encode_decode/1) - ) - - conn = conn |> get(~p"/collections/#{collection.name}") - - assert json_response(conn, 200) == %{"items" => items, "cursor" => nil} - end - - test "up to a limit from params", %{ - conn: conn - } do - user = insert(:user) - project = insert(:project, project_users: [%{user: user}]) - limit = @max_database_limit + 10 - - collection = - insert(:collection, - project: project, - items: insert_list(limit + 1, :collection_item) - ) - - all_items = - collection.items - |> Enum.map(&encode_decode/1) - - expected_items = - all_items |> Enum.take(limit) - - last_item = - collection.items - |> Enum.take(limit) - |> List.last() - - token = Lightning.Accounts.generate_api_token(user) - - conn = - conn - |> assign_bearer(token) - |> get(~p"/collections/#{collection.name}", limit: limit) - - assert conn.state == :chunked - - assert json_response(conn, 200) == %{ - "items" => expected_items, - "cursor" => Base.encode64(to_string(last_item.id)) - } - - # Test for the existence of a cursor when the limit is less than the - # database limit - half_limit = (@max_database_limit / 2) |> floor() - - expected_items = - all_items |> Enum.take(half_limit) - - last_item = - collection.items - |> Enum.take(half_limit) - |> List.last() - - conn = - conn - |> get(~p"/collections/#{collection.name}", - limit: half_limit - ) - - assert json_response(conn, 200) == %{ - "items" => expected_items, - "cursor" => Base.encode64(to_string(last_item.id)) - } - - # Request everything, shouldn't be getting a cursor - conn = - conn |> get(~p"/collections/#{collection.name}", limit: limit + 1) - - assert json_response(conn, 200) == %{ - "items" => all_items, - "cursor" => nil - } - end - end - - describe "GET /collections/:name with cursors" do - test "up to the default limit and returning a cursor", %{conn: conn} do - user = insert(:user) - project = insert(:project, project_users: [%{user: user}]) - collection = insert(:collection, project: project) - - items = - insert_list(@default_stream_limit + 1, :collection_item, - collection: collection - ) - - token = Lightning.Accounts.generate_api_token(user) - - conn = - conn - |> assign_bearer(token) - |> get(~p"/collections/#{collection.name}") - - assert conn.state == :chunked - - expected_items = Enum.take(items, @default_stream_limit) - last_item = List.last(expected_items) - - assert %{ - "items" => items, - "cursor" => cursor - } = json_response(conn, 200) - - assert items == - expected_items - |> Enum.map(&encode_decode/1) - - assert cursor == Base.encode64(to_string(last_item.id)) - end - - test "up to the limit from a cursor returning a cursor", %{conn: conn} do - user = insert(:user) - project = insert(:project, project_users: [%{user: user}]) - collection = insert(:collection, project: project) - limit = 100 - - all_items = - insert_list(2 * limit + 1, :collection_item, collection: collection) - |> Enum.map(&encode_decode/1) - - token = Lightning.Accounts.generate_api_token(user) - - conn = assign_bearer(conn, token) - - conn = - conn - |> assign_bearer(token) - |> get(~p"/collections/#{collection.name}", limit: limit) - - assert conn.state == :chunked - - expected_items = all_items |> Enum.take(limit) - - assert %{ - "items" => ^expected_items, - "cursor" => cursor - } = json_response(conn, 200) - - conn = - conn - |> get(~p"/collections/#{collection.name}", cursor: cursor, limit: limit) - - assert conn.state == :chunked - - expected_items = all_items |> Enum.drop(limit) |> Enum.take(limit) - - assert %{ - "items" => ^expected_items, - "cursor" => cursor - } = json_response(conn, 200) - - %{id: id} = - Repo.get_by(Collections.Item, - collection_id: collection.id, - key: List.last(expected_items)["key"] - ) - - assert {^id, ""} = - cursor |> Base.decode64!() |> Integer.parse() - end - - test "up exactly to the limit from a cursor", %{conn: conn} do - user = insert(:user) - project = insert(:project, project_users: [%{user: user}]) - collection = insert(:collection, project: project) - limit = @default_stream_limit - - all_items = - insert_list(2 * limit, :collection_item, collection: collection) - - token = Lightning.Accounts.generate_api_token(user) - - conn = assign_bearer(conn, token) - - assert %{state: :chunked} = - conn = - get(conn, ~p"/collections/#{collection.name}", limit: limit) - - expected_items = - all_items - |> Enum.take(limit) - |> Enum.map(&encode_decode/1) - - assert %{ - "items" => response_items, - "cursor" => cursor - } = json_response(conn, 200) - - assert response_items == expected_items - - assert %{state: :chunked} = - conn = - get(conn, ~p"/collections/#{collection.name}", - cursor: cursor, - limit: limit - ) - - expected_items = - all_items - |> Enum.drop(limit) - |> Enum.map(&encode_decode/1) - - assert %{ - "items" => ^expected_items, - "cursor" => nil - } = json_response(conn, 200) - end - end - - describe "GET /download/collections/:project_id/:name" do - setup :register_and_log_in_user - setup :create_project_for_current_user - - setup %{conn: conn} do - {:ok, conn: put_req_header(conn, "accept", "text/html,*/*")} - end - - test "downloads collection items as JSON array", %{ - conn: conn, - project: project - } do - collection = insert(:collection, project: project) - - insert(:collection_item, - collection: collection, - key: "key-1", - value: ~s({"name": "Alice"}) - ) - - insert(:collection_item, - collection: collection, - key: "key-2", - value: ~s({"name": "Bob"}) - ) - - response = - get(conn, ~p"/download/collections/#{project.id}/#{collection.name}") - - assert response.status == 200 - - assert {"content-type", "application/json; charset=utf-8"} in response.resp_headers - - assert Enum.any?(response.resp_headers, fn - {"content-disposition", value} -> - String.contains?(value, "attachment") and - String.contains?(value, collection.name) - - _ -> - false - end) - - items = Jason.decode!(response.resp_body) - assert length(items) == 2 - assert Enum.any?(items, fn item -> item["key"] == "key-1" end) - assert Enum.any?(items, fn item -> item["key"] == "key-2" end) - end - - test "downloads empty collection as empty JSON array", %{ - conn: conn, - project: project - } do - collection = insert(:collection, project: project) - - response = - get(conn, ~p"/download/collections/#{project.id}/#{collection.name}") - - assert response.status == 200 - assert Jason.decode!(response.resp_body) == [] - end - - test "returns 401 when user is not a project member", %{conn: conn} do - other_project = insert(:project) - collection = insert(:collection, project: other_project) - - response = - get( - conn, - ~p"/download/collections/#{other_project.id}/#{collection.name}" - ) - - assert response.status == 401 - end - - test "returns 404 for non-existent collection", %{ - conn: conn, - project: project - } do - response = get(conn, ~p"/download/collections/#{project.id}/non-existent") - - assert response.status == 404 - end - - test "returns 400 when project_id is not a valid UUID", %{conn: conn} do - response = get(conn, ~p"/download/collections/not-a-uuid/anything") - - assert response.status == 400 - end - - test "does not serve the payload to a locked-out account", - %{conn: conn, user: user, project: project} do - Mox.stub(Lightning.MockConfig, :check_flag?, fn - :require_email_verification -> true - flag -> Lightning.Config.API.check_flag?(flag) - end) - - user - |> Ecto.Changeset.change( - confirmed_at: nil, - inserted_at: - DateTime.utc_now() - |> Timex.shift(hours: -50) - |> DateTime.truncate(:second) - ) - |> Lightning.Repo.update!() - - collection = insert(:collection, project: project) - - insert(:collection_item, - collection: collection, - key: "key-1", - value: ~s({"name": "Alice"}) - ) - - response = - get(conn, ~p"/download/collections/#{project.id}/#{collection.name}") - - assert redirected_to(response) == "/users/confirm-required" - refute response.resp_body =~ "Alice" - end - end - - describe "malformed request bodies" do - setup %{conn: conn} do - user = insert(:user) - project = insert(:project, project_users: [%{user: user}]) - collection = insert(:collection, project: project) - token = Lightning.Accounts.generate_api_token(user) - conn = assign_bearer(conn, token) - {:ok, conn: conn, project: project, collection: collection} - end - - test "PUT with no value returns 422", %{conn: conn, collection: collection} do - conn = put(conn, ~p"/collections/#{collection.name}/some-key", %{}) - - assert json_response(conn, 422)["error"] =~ "Missing required field: value" - end - - test "POST with no items returns 422", %{conn: conn, collection: collection} do - conn = post(conn, ~p"/collections/#{collection.name}", %{}) - - assert json_response(conn, 422)["error"] =~ "Missing required field: items" - end - end - - describe "ambiguous name without ?project_id" do - setup %{conn: conn} do - user = insert(:user) - project_1 = insert(:project, project_users: [%{user: user}]) - project_2 = insert(:project, project_users: [%{user: user}]) - name = "shared-collection" - insert(:collection, name: name, project: project_1) - insert(:collection, name: name, project: project_2) - token = Lightning.Accounts.generate_api_token(user) - conn = assign_bearer(conn, token) - {:ok, conn: conn, name: name} - end - - test "GET /:name returns 409", %{conn: conn, name: name} do - conn = get(conn, ~p"/collections/#{name}") - assert json_response(conn, 409)["error"] =~ "Add ?project_id=" - end - - test "GET /:name/:key returns 409", %{conn: conn, name: name} do - conn = get(conn, ~p"/collections/#{name}/some-key") - assert json_response(conn, 409)["error"] =~ "Add ?project_id=" - end - - test "PUT /:name/:key returns 409", %{conn: conn, name: name} do - conn = put(conn, ~p"/collections/#{name}/some-key", value: "val") - assert json_response(conn, 409)["error"] =~ "Add ?project_id=" - end - - test "POST /:name returns 409", %{conn: conn, name: name} do - conn = post(conn, ~p"/collections/#{name}", items: []) - assert json_response(conn, 409)["error"] =~ "Add ?project_id=" - end - - test "DELETE /:name/:key returns 409", %{conn: conn, name: name} do - conn = delete(conn, ~p"/collections/#{name}/some-key") - assert json_response(conn, 409)["error"] =~ "Add ?project_id=" - end - - test "DELETE /:name returns 409", %{conn: conn, name: name} do - conn = delete(conn, ~p"/collections/#{name}") - assert json_response(conn, 409)["error"] =~ "Add ?project_id=" - end - end - - describe "?project_id= query param" do - setup %{conn: conn} do - user = insert(:user) - project = insert(:project, project_users: [%{user: user, role: :owner}]) - - collection = - insert(:collection, - project: project, - items: [%{key: "foo", value: "bar"}] - ) - - token = Lightning.Accounts.generate_api_token(user) - conn = assign_bearer(conn, token) - {:ok, conn: conn, project: project, collection: collection} - end - - test "GET /:name/:key returns the item", %{ - conn: conn, - project: project, - collection: collection - } do - item = hd(collection.items) - - conn = - get( - conn, - ~p"/collections/#{collection.name}/foo?project_id=#{project.id}" - ) - - assert json_response(conn, 200) == %{ - "key" => item.key, - "value" => item.value, - "created" => DateTime.to_iso8601(item.inserted_at), - "updated" => DateTime.to_iso8601(item.updated_at) - } - end - - test "GET /:name/:key returns 204 when item not found", %{ - conn: conn, - project: project, - collection: collection - } do - conn = - get( - conn, - ~p"/collections/#{collection.name}/nonexistent?project_id=#{project.id}" - ) - - assert response(conn, 204) == "" - end - - test "GET /:name/:key returns 404 when collection not found", %{ - conn: conn, - project: project - } do - conn = - get( - conn, - ~p"/collections/nonexistent-collection/foo?project_id=#{project.id}" - ) - - assert json_response(conn, 404) - end - - test "GET /:name streams items scoped by project", %{ - conn: conn, - project: project, - collection: collection - } do - conn = - get(conn, ~p"/collections/#{collection.name}?project_id=#{project.id}") - - body = json_response(conn, 200) - assert length(body["items"]) == 1 - assert hd(body["items"])["key"] == "foo" - end - - test "PUT /:name/:key upserts an item", %{ - conn: conn, - project: project, - collection: collection - } do - conn = - put( - conn, - ~p"/collections/#{collection.name}/new-key?project_id=#{project.id}", - value: "new-val" - ) - - assert json_response(conn, 200) == %{"upserted" => 1, "error" => nil} - end - - test "POST /:name upserts multiple items", %{ - conn: conn, - project: project, - collection: collection - } do - conn = - post( - conn, - ~p"/collections/#{collection.name}?project_id=#{project.id}", - items: [%{key: "a", value: "1"}, %{key: "b", value: "2"}] - ) - - assert json_response(conn, 200) == %{"upserted" => 2, "error" => nil} - end - - test "DELETE /:name/:key deletes an item", %{ - conn: conn, - project: project, - collection: collection - } do - conn = - delete( - conn, - ~p"/collections/#{collection.name}/foo?project_id=#{project.id}" - ) - - assert json_response(conn, 200) == %{ - "key" => "foo", - "deleted" => 1, - "error" => nil - } - end - - test "DELETE /:name deletes all items", %{ - conn: conn, - project: project, - collection: collection - } do - conn = - delete( - conn, - ~p"/collections/#{collection.name}?project_id=#{project.id}" - ) - - assert json_response(conn, 200)["deleted"] == 1 - end - - test "returns 401 when user does not have access to the project", %{ - conn: conn - } do - other_project = insert(:project) - other_collection = insert(:collection, project: other_project) - - conn = - get( - conn, - ~p"/collections/#{other_collection.name}/foo?project_id=#{other_project.id}" - ) - - assert json_response(conn, 401) - end - - test "resolves correctly when same name exists in multiple projects", %{ - conn: conn, - project: project, - collection: collection - } do - other_project = insert(:project, project_users: []) - insert(:collection, name: collection.name, project: other_project) - - conn = - get( - conn, - ~p"/collections/#{collection.name}/foo?project_id=#{project.id}" - ) - - assert json_response(conn, 200)["key"] == "foo" - end - - test "returns 400 when project_id is not a valid UUID", %{ - conn: conn, - collection: collection - } do - conn = - get(conn, ~p"/collections/#{collection.name}?project_id=not-a-uuid") - - assert response(conn, 400) - end - - test "with a valid run token, can access own project's collection", %{ - conn: conn, - project: project, - collection: collection - } do - workflow = insert(:simple_workflow, project: project) - - workorder = - insert(:workorder, workflow: workflow, dataclip: insert(:dataclip)) - - run = - insert(:run, - work_order: workorder, - dataclip: workorder.dataclip, - starting_trigger: hd(workflow.triggers) - ) - - token = Lightning.Workers.generate_run_token(run) - - conn = - conn - |> assign_bearer(token) - |> get(~p"/collections/#{collection.name}/foo?project_id=#{project.id}") - - assert json_response(conn, 200)["key"] == "foo" - end - - test "with a valid run token, can write to own project's collection", %{ - conn: conn, - project: project, - collection: collection - } do - workflow = insert(:simple_workflow, project: project) - - workorder = - insert(:workorder, workflow: workflow, dataclip: insert(:dataclip)) - - run = - insert(:run, - work_order: workorder, - dataclip: workorder.dataclip, - starting_trigger: hd(workflow.triggers) - ) - - token = Lightning.Workers.generate_run_token(run) - - conn = - conn - |> assign_bearer(token) - |> put( - ~p"/collections/#{collection.name}/new-key?project_id=#{project.id}", - value: "new-val" - ) - - assert json_response(conn, 200) == %{"upserted" => 1, "error" => nil} - end - - test "with a run token, cannot access a different project's collection", %{ - conn: conn - } do - other_project = insert(:project) - other_collection = insert(:collection, project: other_project) - - workflow = insert(:simple_workflow) - - workorder = - insert(:workorder, workflow: workflow, dataclip: insert(:dataclip)) - - run = - insert(:run, - work_order: workorder, - dataclip: workorder.dataclip, - starting_trigger: hd(workflow.triggers) - ) - - token = Lightning.Workers.generate_run_token(run) - - conn = - conn - |> assign_bearer(token) - |> get( - ~p"/collections/#{other_collection.name}/foo?project_id=#{other_project.id}" - ) - - assert json_response(conn, 401) - end - - test "with a run token, cannot write to a different project's collection", - %{conn: conn} do - other_project = insert(:project) - other_collection = insert(:collection, project: other_project) - - workflow = insert(:simple_workflow) - - workorder = - insert(:workorder, workflow: workflow, dataclip: insert(:dataclip)) - - run = - insert(:run, - work_order: workorder, - dataclip: workorder.dataclip, - starting_trigger: hd(workflow.triggers) - ) - - token = Lightning.Workers.generate_run_token(run) - - conn = - conn - |> assign_bearer(token) - |> put( - ~p"/collections/#{other_collection.name}/k?project_id=#{other_project.id}", - value: "v" - ) - - assert json_response(conn, 401) - end - - test "with a run token, bare ambiguous name returns 409 before authorization", - %{conn: conn, project: project, collection: collection} do - other_project = insert(:project) - insert(:collection, name: collection.name, project: other_project) - - workflow = insert(:simple_workflow, project: project) - - workorder = - insert(:workorder, workflow: workflow, dataclip: insert(:dataclip)) - - run = - insert(:run, - work_order: workorder, - dataclip: workorder.dataclip, - starting_trigger: hd(workflow.triggers) - ) - - token = Lightning.Workers.generate_run_token(run) - - conn = - conn - |> assign_bearer(token) - |> get(~p"/collections/#{collection.name}/foo") - - assert json_response(conn, 409)["error"] =~ "Add ?project_id=" - end - end - - defp encode_decode(item) do - item - |> Jason.encode!() - |> Jason.decode!() - end -end diff --git a/test/lightning_web/controllers/adaptor_controller_test.exs b/test/lightning_web/controllers/adaptor_controller_test.exs index cbad90a0356..59bd8a3e070 100644 --- a/test/lightning_web/controllers/adaptor_controller_test.exs +++ b/test/lightning_web/controllers/adaptor_controller_test.exs @@ -2,6 +2,7 @@ defmodule LightningWeb.AdaptorControllerTest do use LightningWeb.ConnCase, async: true import Lightning.Factories + import LightningWeb.API.ProblemHelpers import Mimic alias Lightning.Adaptors @@ -128,23 +129,14 @@ defmodule LightningWeb.AdaptorControllerTest do end end - test "an unauthenticated request is rejected with a JSON 401", %{conn: conn} do - conn = get(conn, ~p"/adaptors/catalogue") - - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - - test "returns a 503 with a JSON body when the store fails", %{conn: conn} do + test "a store failure is a 503 not_ready", %{conn: conn} do conn = log_in_user(conn, insert(:user)) stub(Adaptors, :catalogue, fn -> {:error, :unavailable} end) conn = get(conn, ~p"/adaptors/catalogue") - assert json_response(conn, 503) == %{ - "error" => "adaptor catalogue unavailable" - } - + assert_problem(conn, 503, :not_ready, []) assert get_resp_header(conn, "retry-after") == ["5"] end @@ -157,7 +149,7 @@ defmodule LightningWeb.AdaptorControllerTest do conn = get(conn, ~p"/adaptors/catalogue") - assert json_response(conn, 503) + assert_problem(conn, 503, :not_ready, []) assert get_resp_header(conn, "retry-after") == ["5"] end diff --git a/test/lightning_web/controllers/api/ai_assistant/job_code_sessions_test.exs b/test/lightning_web/controllers/api/ai_assistant/job_code_sessions_test.exs new file mode 100644 index 00000000000..e250d076124 --- /dev/null +++ b/test/lightning_web/controllers/api/ai_assistant/job_code_sessions_test.exs @@ -0,0 +1,426 @@ +defmodule LightningWeb.API.AiAssistantJobCodeSessionsTest do + use LightningWeb.ConnCase, async: true + + @moduletag :capture_log + + import Lightning.Factories + import LightningWeb.API.ProblemHelpers + import LightningWeb.ApiAiAssistantHelpers + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + describe "listing the sessions of a saved job" do + setup [:register_and_log_in_user] + + test "answers each member of the project with the session document", %{ + conn: conn, + user: user + } do + for role <- [:viewer, :editor, :admin, :owner] do + project = + insert(:project, project_users: [%{user_id: user.id, role: role}]) + + workflow = insert(:workflow, project: project, name: "Test Workflow") + job = insert(:job, workflow: workflow, name: "Test Job") + + session = + insert(:chat_session, + user: user, + session_type: "job_code", + job: job, + title: "A #{role} session" + ) + + conn = + get( + conn, + ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{job.id}" + ) + + assert json_response(conn, 200) == %{ + "sessions" => [ + %{ + "id" => session.id, + "title" => "A #{role} session", + "session_type" => "job_code", + "job_name" => "Test Job", + "workflow_name" => "Test Workflow", + "message_count" => 0, + "updated_at" => updated_at(session) + } + ], + "pagination" => pagination(1) + }, + "a #{role} is answered the session" + end + end + + test "counts the messages of each session", %{conn: conn, user: user} do + project = insert(:project, project_users: [%{user_id: user.id}]) + workflow = insert(:workflow, project: project) + job = insert(:job, workflow: workflow) + + messages = + for i <- 1..5 do + insert(:chat_message, + role: if(rem(i, 2) == 0, do: "user", else: "assistant"), + content: "Message #{i}", + status: "success" + ) + end + + session = + insert(:chat_session, + user: user, + session_type: "job_code", + job: job, + title: "Session with messages", + messages: messages + ) + + conn = + get( + conn, + ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{job.id}" + ) + + assert %{"sessions" => [returned_session]} = json_response(conn, 200) + assert returned_session["id"] == session.id + assert returned_session["message_count"] == 5 + end + + test "lists the sessions of the job asked for and no others", %{ + conn: conn, + user: user + } do + project = insert(:project, project_users: [%{user_id: user.id}]) + workflow = insert(:workflow, project: project) + job = insert(:job, workflow: workflow) + + insert(:chat_session, + user: user, + session_type: "job_code", + job_id: nil, + meta: %{ + "unsaved_job" => %{ + "id" => Ecto.UUID.generate(), + "name" => "Unsaved Job", + "workflow_id" => workflow.id + } + } + ) + + insert(:chat_session, + user: user, + session_type: "job_code", + job_id: nil, + meta: %{} + ) + + conn = + get( + conn, + ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{job.id}" + ) + + assert json_response(conn, 200) == %{ + "sessions" => [], + "pagination" => pagination(0) + } + end + end + + describe "paging the sessions of a saved job" do + setup [:register_and_log_in_user, :create_job_with_sessions] + + test "lists every session when no paging is asked for", %{ + conn: conn, + job: job, + sessions: sessions + } do + conn = + get( + conn, + ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{job.id}" + ) + + assert %{"sessions" => returned_sessions, "pagination" => page} = + json_response(conn, 200) + + assert MapSet.new(returned_sessions, & &1["id"]) == + MapSet.new(sessions, & &1.id) + + assert page == pagination(3) + end + + test "respects pagination offset and limit", %{conn: conn, job: job} do + conn = + get( + conn, + ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{job.id}&offset=1&limit=1" + ) + + assert %{"sessions" => [_one], "pagination" => page} = + json_response(conn, 200) + + assert page == pagination(3, next: true, prev: true) + end + + test "returns an empty list when the offset exceeds the total", %{ + conn: conn, + job: job + } do + conn = + get( + conn, + ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{job.id}&offset=100" + ) + + assert json_response(conn, 200) == %{ + "sessions" => [], + "pagination" => pagination(3, prev: true) + } + end + + test "returns 404 when the caller has no access to the job", %{job: job} do + conn = + build_conn() + |> put_req_header("accept", "application/json") + |> log_in_user(insert(:user)) + |> get( + ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{job.id}" + ) + + assert_problem(conn, 404, :not_found, []) + end + end + + describe "listing the sessions of an unsaved job" do + setup [:register_and_log_in_user] + + test "returns an empty list when the job has no sessions", %{conn: conn} do + conn = + get( + conn, + ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{Ecto.UUID.generate()}" + ) + + assert json_response(conn, 200) == %{ + "sessions" => [], + "pagination" => pagination(0) + } + end + + test "returns the session of a workflow the caller can reach", %{ + conn: conn, + user: user + } do + project = insert(:project, project_users: [%{user_id: user.id}]) + workflow = insert(:workflow, project: project, name: "Test Workflow") + unsaved_job_id = Ecto.UUID.generate() + + session = + insert(:chat_session, + user: user, + session_type: "job_code", + title: "Unsaved Job Session", + workflow_id: workflow.id, + meta: %{ + "unsaved_job" => %{ + "id" => unsaved_job_id, + "name" => "Unsaved Job", + "workflow_id" => workflow.id + } + } + ) + + conn = + get( + conn, + ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{unsaved_job_id}" + ) + + assert json_response(conn, 200) == %{ + "sessions" => [ + %{ + "id" => session.id, + "title" => "Unsaved Job Session", + "session_type" => "job_code", + "job_name" => "Unsaved Job", + "workflow_name" => "Test Workflow", + "is_unsaved" => true, + "message_count" => 0, + "updated_at" => updated_at(session) + } + ], + "pagination" => pagination(1) + } + end + + test "returns the owner their session when it names no workflow", %{ + conn: conn, + user: user + } do + unsaved_job_id = Ecto.UUID.generate() + + session = + insert(:chat_session, + user: user, + session_type: "job_code", + job_id: nil, + title: "No-workflow unsaved session", + meta: %{"unsaved_job" => %{"id" => unsaved_job_id, "name" => "X"}} + ) + + conn = + get( + conn, + ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{unsaved_job_id}" + ) + + assert json_response(conn, 200) == %{ + "sessions" => [ + %{ + "id" => session.id, + "title" => "No-workflow unsaved session", + "session_type" => "job_code", + "job_name" => "X", + "workflow_name" => nil, + "is_unsaved" => true, + "message_count" => 0, + "updated_at" => updated_at(session) + } + ], + "pagination" => pagination(1) + } + end + + test "returns 404 when the caller has no access to the unsaved job's workflow", + %{conn: _conn} do + other_user = insert(:user) + project = insert(:project, project_users: [%{user_id: other_user.id}]) + + workflow = + insert(:workflow, project: project, name: "Other User Workflow") + + unsaved_job_id = Ecto.UUID.generate() + + insert(:chat_session, + user: other_user, + session_type: "job_code", + title: "Unsaved Job Session", + meta: %{ + "unsaved_job" => %{ + "id" => unsaved_job_id, + "name" => "Unsaved Job", + "workflow_id" => workflow.id + } + } + ) + + conn = + build_conn() + |> put_req_header("accept", "application/json") + |> log_in_user(insert(:user)) + |> get( + ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{unsaved_job_id}" + ) + + assert_problem(conn, 404, :not_found, []) + end + + test "returns 404 for a non-owner when the session names no workflow", %{ + conn: _conn + } do + unsaved_job_id = Ecto.UUID.generate() + + insert(:chat_session, + user: insert(:user), + session_type: "job_code", + job_id: nil, + title: "No-workflow unsaved session", + meta: %{"unsaved_job" => %{"id" => unsaved_job_id, "name" => "X"}} + ) + + conn = + build_conn() + |> put_req_header("accept", "application/json") + |> log_in_user(insert(:user)) + |> get( + ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{unsaved_job_id}" + ) + + assert_problem(conn, 404, :not_found, []) + end + end + + describe "listing the sessions of a deleted job" do + setup [:register_and_log_in_user] + + test "returns an empty list once the job is deleted", %{ + conn: conn, + user: user + } do + project = insert(:project, project_users: [%{user_id: user.id}]) + workflow = insert(:workflow, project: project) + job = insert(:job, workflow: workflow, name: "Job to Delete") + job_id = job.id + + insert(:chat_session, + user: user, + session_type: "job_code", + job: job, + title: "Session for deleted job" + ) + + # Deleting the job nilifies the session's job_id, so nothing matches it. + Lightning.Repo.delete!(job) + + conn = + get( + conn, + ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{job_id}" + ) + + assert json_response(conn, 200) == %{ + "sessions" => [], + "pagination" => pagination(0) + } + end + end + + defp create_job_with_sessions(%{user: user}) do + project = insert(:project, project_users: [%{user_id: user.id}]) + + workflow = + insert(:workflow, project: project, name: "Test Workflow") + + job = + insert(:job, + workflow: workflow, + name: "Test Job", + body: "fn(state => state);", + adaptor: "@openfn/language-common@latest" + ) + + sessions = + for i <- 3..1//-1 do + session = + insert(:chat_session, + user: user, + session_type: "job_code", + job: job, + title: "Test Session #{i}" + ) + + # Distinct updated_at values, so the order the route sorts by is stable. + Process.sleep(50) + + session + end + + %{job: job, project: project, workflow: workflow, sessions: sessions} + end +end diff --git a/test/lightning_web/controllers/api/ai_assistant/refusals_test.exs b/test/lightning_web/controllers/api/ai_assistant/refusals_test.exs new file mode 100644 index 00000000000..02a47fb4631 --- /dev/null +++ b/test/lightning_web/controllers/api/ai_assistant/refusals_test.exs @@ -0,0 +1,212 @@ +defmodule LightningWeb.API.AiAssistantRefusalsTest do + use LightningWeb.ConnCase, async: true + + @moduletag :capture_log + + import Lightning.Factories + import LightningWeb.API.ProblemHelpers + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + setup :register_and_log_in_user + + describe "naming the sessions to list" do + test "names a session type or target that is missing or not allowed", %{ + conn: conn + } do + for {query, parameter, code, detail} <- [ + {"job_id=123", "session_type", "required", "can't be blank"}, + {"session_type=invalid&job_id=123", "session_type", "not_allowed", + "is invalid"}, + {"session_type=job_code", "job_id", "required", "can't be blank"}, + {"session_type=workflow_template", "project_id", "required", + "can't be blank"} + ] do + conn + |> get("/api/ai_assistant/sessions?" <> query) + |> assert_problem(400, :invalid_parameter, [ + %{"parameter" => parameter, "code" => code, "detail" => detail} + ]) + end + end + + test "every bad parameter is reported at once", %{conn: conn, user: user} do + project = insert(:project, project_users: [%{user_id: user.id}]) + + conn + |> get(~p"/api/ai_assistant/sessions", + session_type: "workflow_template", + project_id: project.id, + offset: "-1", + limit: "0", + workflow_id: "nope" + ) + |> assert_problem(400, :invalid_parameter, [ + %{ + "parameter" => "offset", + "code" => "out_of_range", + "detail" => "must be greater than or equal to 0" + }, + %{ + "parameter" => "limit", + "code" => "out_of_range", + "detail" => "must be greater than 0" + }, + %{ + "parameter" => "workflow_id", + "code" => "type", + "detail" => "is invalid" + } + ]) + + conn + |> get(~p"/api/ai_assistant/sessions", offset: "abc", limit: "x") + |> assert_problem(400, :invalid_parameter, [ + %{ + "parameter" => "session_type", + "code" => "required", + "detail" => "can't be blank" + }, + %{"parameter" => "offset", "code" => "type", "detail" => "is invalid"}, + %{"parameter" => "limit", "code" => "type", "detail" => "is invalid"} + ]) + end + + test "no answer quotes a session_type, job_id or workflow_id it was sent", + %{conn: conn, user: user} do + project = insert(:project, project_users: [%{user_id: user.id}]) + + params = + plant_sentinel(%{ + "session_type" => "", + "job_id" => "", + "workflow_id" => "" + }) + + conn + |> get(~p"/api/ai_assistant/sessions", params) + |> refute_sentinel() + |> assert_problem(400, :invalid_parameter, [ + %{ + "parameter" => "workflow_id", + "code" => "type", + "detail" => "is invalid" + }, + %{ + "parameter" => "session_type", + "code" => "not_allowed", + "detail" => "is invalid" + } + ]) + + conn + |> get( + ~p"/api/ai_assistant/sessions", + Map.merge(params, %{ + "session_type" => "workflow_template", + "project_id" => project.id + }) + ) + |> refute_sentinel() + |> assert_problem(400, :invalid_parameter, [ + %{ + "parameter" => "workflow_id", + "code" => "type", + "detail" => "is invalid" + } + ]) + + conn + |> get( + ~p"/api/ai_assistant/sessions", + Map.put(params, "session_type", "job_code") + ) + |> refute_sentinel() + |> assert_problem(404, :not_found, []) + end + end + + describe "who may reach it" do + test "a job_id or project_id that isn't a UUID answers 404", %{conn: conn} do + conn + |> get(~p"/api/ai_assistant/sessions?session_type=job_code&job_id=nope") + |> assert_problem(404, :not_found, []) + + conn + |> get( + ~p"/api/ai_assistant/sessions?session_type=workflow_template&project_id=nope" + ) + |> assert_problem(404, :not_found, []) + end + + test "a project that doesn't exist answers 404", %{conn: conn} do + conn + |> get( + ~p"/api/ai_assistant/sessions?session_type=workflow_template&project_id=#{Ecto.UUID.generate()}" + ) + |> assert_problem(404, :not_found, []) + end + + test "a caller who can't see the project gets 404 whatever the other parameters say", + %{conn: conn} do + project = insert(:project) + + conn + |> get( + ~p"/api/ai_assistant/sessions?session_type=workflow_template&project_id=#{project.id}" + ) + |> assert_problem(404, :not_found, []) + + conn + |> get(~p"/api/ai_assistant/sessions", + session_type: "workflow_template", + project_id: project.id, + offset: "abc", + workflow_id: "nope" + ) + |> assert_problem(404, :not_found, []) + end + + test "a member who hasn't met MFA is refused 403", %{conn: conn, user: user} do + project = + insert(:project, + requires_mfa: true, + project_users: [%{user_id: user.id, role: :owner}] + ) + + workflow = insert(:simple_workflow, project: project) + job = hd(workflow.jobs) + + conn + |> get( + ~p"/api/ai_assistant/sessions?session_type=workflow_template&project_id=#{project.id}" + ) + |> assert_problem(403, :forbidden, []) + + conn + |> get( + ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{job.id}" + ) + |> assert_problem(403, :forbidden, []) + end + + test "a project scheduled for deletion answers 409 to a member", %{ + conn: conn, + user: user + } do + project = + insert(:project, + scheduled_deletion: ~U[2026-01-01 00:00:00Z], + project_users: [%{user_id: user.id, role: :owner}] + ) + + conn + |> get( + ~p"/api/ai_assistant/sessions?session_type=workflow_template&project_id=#{project.id}" + ) + |> assert_problem(409, :scheduled_for_deletion, []) + end + end +end diff --git a/test/lightning_web/controllers/api/ai_assistant/workflow_template_sessions_test.exs b/test/lightning_web/controllers/api/ai_assistant/workflow_template_sessions_test.exs new file mode 100644 index 00000000000..a7701b2d277 --- /dev/null +++ b/test/lightning_web/controllers/api/ai_assistant/workflow_template_sessions_test.exs @@ -0,0 +1,153 @@ +defmodule LightningWeb.API.AiAssistantWorkflowTemplateSessionsTest do + use LightningWeb.ConnCase, async: true + + @moduletag :capture_log + + import Lightning.Factories + import LightningWeb.ApiAiAssistantHelpers + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + describe "listing the workflow template sessions of a project" do + setup [:register_and_log_in_user] + + test "answers each member of the project with the session document", %{ + conn: conn, + user: user + } do + for role <- [:viewer, :editor, :admin, :owner] do + project = + insert(:project, + name: "Project of a #{role}", + project_users: [%{user_id: user.id, role: role}] + ) + + session = + insert(:workflow_chat_session, + user: user, + project: project, + title: "A #{role} session" + ) + + conn = + get( + conn, + ~p"/api/ai_assistant/sessions?session_type=workflow_template&project_id=#{project.id}" + ) + + assert json_response(conn, 200) == %{ + "sessions" => [ + %{ + "id" => session.id, + "title" => "A #{role} session", + "session_type" => "workflow_template", + "project_name" => "Project of a #{role}", + "workflow_name" => nil, + "message_count" => 0, + "updated_at" => updated_at(session) + } + ], + "pagination" => pagination(1) + }, + "a #{role} is answered the session" + end + end + + test "lists every session of the project", %{conn: conn, user: user} do + project = insert(:project, project_users: [%{user_id: user.id}]) + + sessions = + for i <- 1..3 do + insert(:workflow_chat_session, + user: user, + project: project, + title: "Workflow Template Session #{i}" + ) + end + + conn = + get( + conn, + ~p"/api/ai_assistant/sessions?session_type=workflow_template&project_id=#{project.id}" + ) + + assert %{"sessions" => returned_sessions, "pagination" => page} = + json_response(conn, 200) + + assert MapSet.new(returned_sessions, & &1["id"]) == + MapSet.new(sessions, & &1.id) + + assert page == pagination(3) + end + + test "narrows the sessions to one workflow when workflow_id is given", %{ + conn: conn, + user: user + } do + project = + insert(:project, + name: "My Project", + project_users: [%{user_id: user.id}] + ) + + workflow1 = insert(:workflow, project: project, name: "Workflow 1") + workflow2 = insert(:workflow, project: project, name: "Workflow 2") + + session1 = + insert(:workflow_chat_session, + user: user, + project: project, + workflow: workflow1, + title: "Workflow 1 Session" + ) + + insert(:workflow_chat_session, + user: user, + project: project, + workflow: workflow2, + title: "Workflow 2 Session" + ) + + conn = + get( + conn, + ~p"/api/ai_assistant/sessions?session_type=workflow_template&project_id=#{project.id}&workflow_id=#{workflow1.id}" + ) + + assert json_response(conn, 200) == %{ + "sessions" => [ + %{ + "id" => session1.id, + "title" => "Workflow 1 Session", + "session_type" => "workflow_template", + "project_name" => "My Project", + "workflow_name" => "Workflow 1", + "message_count" => 0, + "updated_at" => updated_at(session1) + } + ], + "pagination" => pagination(1) + } + end + + test "returns an empty list when the project has no sessions", %{ + conn: conn, + user: user + } do + project = insert(:project, project_users: [%{user_id: user.id}]) + + conn = + get( + conn, + ~p"/api/ai_assistant/sessions?session_type=workflow_template&project_id=#{project.id}" + ) + + assert json_response(conn, 200) == %{ + "sessions" => [], + "pagination" => pagination(0) + } + end + end +end diff --git a/test/lightning_web/controllers/api/ai_assistant_controller_test.exs b/test/lightning_web/controllers/api/ai_assistant_controller_test.exs deleted file mode 100644 index c13fbcd2704..00000000000 --- a/test/lightning_web/controllers/api/ai_assistant_controller_test.exs +++ /dev/null @@ -1,928 +0,0 @@ -defmodule LightningWeb.API.AiAssistantControllerTest do - use LightningWeb.ConnCase, async: true - - @moduletag :capture_log - - import Mox - import Lightning.Factories - - setup :verify_on_exit! - - setup %{conn: conn} do - {:ok, conn: put_req_header(conn, "accept", "application/json")} - end - - setup do - # Mock Apollo configuration to prevent real HTTP calls - Mox.stub(Lightning.MockConfig, :apollo, fn key -> - case key do - :endpoint -> "http://localhost:3000" - :ai_assistant_api_key -> "test_api_key" - :connect_timeout -> 1_000 - :idle_timeout -> 5_000 - :request_timeout -> 5_000 - end - end) - - # Mock Tesla HTTP client to prevent real HTTP calls - Mox.stub( - Lightning.Tesla.Mock, - :call, - Lightning.AiAssistantHelpers.streaming_or_sync_response(%{ - "response" => "This is a test AI response.", - "history" => [ - %{"role" => "user", "content" => "test message"}, - %{ - "role" => "assistant", - "content" => "This is a test AI response." - } - ] - }) - ) - - :ok - end - - describe "list_sessions without authentication" do - test "returns 401 without token", %{conn: conn} do - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=123" - ) - - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - - test "returns 401 with invalid token", %{conn: conn} do - conn = - conn - |> Plug.Conn.put_req_header("authorization", "Bearer invalid_token") - - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=123" - ) - - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - end - - describe "list_sessions with an email-confirmation lockout" do - setup [:register_and_log_in_user] - - test "returns 401 to a locked-out account", %{conn: conn, user: user} do - Mox.stub(Lightning.MockConfig, :check_flag?, fn - :require_email_verification -> true - flag -> Lightning.Config.API.check_flag?(flag) - end) - - user - |> Ecto.Changeset.change( - confirmed_at: nil, - inserted_at: - DateTime.utc_now() - |> Timex.shift(hours: -50) - |> DateTime.truncate(:second) - ) - |> Lightning.Repo.update!() - - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{Ecto.UUID.generate()}" - ) - - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - end - - describe "list_sessions validation" do - setup [:register_and_log_in_user] - - test "returns 400 when session_type is missing", %{conn: conn} do - conn = get(conn, ~p"/api/ai_assistant/sessions?job_id=123") - - assert json_response(conn, 400) == %{"error" => "Bad Request"} - end - - test "returns 400 when session_type is invalid", %{conn: conn} do - conn = - get(conn, ~p"/api/ai_assistant/sessions?session_type=invalid&job_id=123") - - assert json_response(conn, 400) == %{"error" => "Bad Request"} - end - - test "returns 400 when job_code session_type missing job_id", %{conn: conn} do - conn = get(conn, ~p"/api/ai_assistant/sessions?session_type=job_code") - - assert json_response(conn, 400) == %{"error" => "Bad Request"} - end - - test "returns 400 when workflow_template session_type missing project_id", - %{conn: conn} do - conn = - get(conn, ~p"/api/ai_assistant/sessions?session_type=workflow_template") - - assert json_response(conn, 400) == %{"error" => "Bad Request"} - end - - test "returns 404 when workflow_template project does not exist", %{ - conn: conn - } do - nonexistent_uuid = Ecto.UUID.generate() - - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=workflow_template&project_id=#{nonexistent_uuid}" - ) - - assert json_response(conn, 404) == %{"error" => "Not Found"} - end - end - - describe "list_sessions for job_code with saved job" do - setup [:register_and_log_in_user, :create_job_with_sessions] - - test "returns sessions for job user has access to", %{ - conn: conn, - job: job, - sessions: sessions - } do - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{job.id}" - ) - - response = json_response(conn, 200) - - assert %{ - "sessions" => returned_sessions, - "pagination" => pagination - } = response - - assert length(returned_sessions) == 3 - - assert pagination["total_count"] == 3 - assert pagination["has_next_page"] == false - assert pagination["has_prev_page"] == false - - # Verify all sessions are returned (don't check order due to flaky timestamps) - returned_ids = Enum.map(returned_sessions, & &1["id"]) |> MapSet.new() - created_ids = Enum.map(sessions, & &1.id) |> MapSet.new() - assert MapSet.equal?(returned_ids, created_ids) - - [session1 | _] = returned_sessions - assert session1["session_type"] == "job_code" - assert session1["job_name"] == job.name - assert is_binary(session1["workflow_name"]) - assert session1["message_count"] >= 0 - assert is_binary(session1["updated_at"]) - end - - test "respects pagination offset and limit", %{conn: conn, job: job} do - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{job.id}&offset=1&limit=1" - ) - - response = json_response(conn, 200) - - assert %{ - "sessions" => returned_sessions, - "pagination" => pagination - } = response - - assert length(returned_sessions) == 1 - assert pagination["total_count"] == 3 - assert pagination["has_next_page"] == true - assert pagination["has_prev_page"] == true - end - - test "returns empty list when offset exceeds total", %{conn: conn, job: job} do - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{job.id}&offset=100" - ) - - response = json_response(conn, 200) - - assert %{ - "sessions" => returned_sessions, - "pagination" => pagination - } = response - - assert returned_sessions == [] - assert pagination["total_count"] == 3 - assert pagination["has_next_page"] == false - end - - test "returns 403 when user does not have access to job", %{ - conn: _conn, - job: job - } do - # Create another user without access - other_user = insert(:user) - - conn = build_conn() - conn = put_req_header(conn, "accept", "application/json") - conn = log_in_user(conn, other_user) - - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{job.id}" - ) - - assert json_response(conn, 403) == %{"error" => "Forbidden"} - end - end - - describe "list_sessions for job_code with unsaved job" do - setup [:register_and_log_in_user] - - test "returns empty list for new unsaved job with no sessions", %{ - conn: conn - } do - unsaved_job_id = Ecto.UUID.generate() - - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{unsaved_job_id}" - ) - - response = json_response(conn, 200) - - assert %{ - "sessions" => returned_sessions, - "pagination" => pagination - } = response - - assert returned_sessions == [] - assert pagination["total_count"] == 0 - end - - test "returns sessions for unsaved job user has access to", %{ - conn: conn, - user: user - } do - # Create workflow and unsaved job session - project = insert(:project, project_users: [%{user_id: user.id}]) - workflow = insert(:workflow, project: project, name: "Test Workflow") - unsaved_job_id = Ecto.UUID.generate() - - # Create session with unsaved job metadata - # Note: workflow_id is set on the session (as create_session_for_unsaved_job does) - # to enable preloading and avoid N+1 queries - session = - insert(:chat_session, - user: user, - session_type: "job_code", - title: "Unsaved Job Session", - workflow_id: workflow.id, - meta: %{ - "unsaved_job" => %{ - "id" => unsaved_job_id, - "name" => "Unsaved Job", - "workflow_id" => workflow.id - } - } - ) - - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{unsaved_job_id}" - ) - - response = json_response(conn, 200) - - assert %{ - "sessions" => returned_sessions, - "pagination" => _pagination - } = response - - assert length(returned_sessions) == 1 - - [returned_session] = returned_sessions - - assert returned_session["id"] == session.id - assert returned_session["title"] == session.title - assert returned_session["session_type"] == "job_code" - assert returned_session["job_name"] == "Unsaved Job" - assert returned_session["workflow_name"] == workflow.name - assert returned_session["is_unsaved"] == true - end - - test "returns 403 when user does not have access to unsaved job's workflow", - %{conn: _conn, user: _user} do - # Create workflow owned by another user - other_user = insert(:user) - project = insert(:project, project_users: [%{user_id: other_user.id}]) - - workflow = - insert(:workflow, project: project, name: "Other User Workflow") - - unsaved_job_id = Ecto.UUID.generate() - - # Create session with unsaved job metadata - _session = - insert(:chat_session, - user: other_user, - session_type: "job_code", - title: "Unsaved Job Session", - meta: %{ - "unsaved_job" => %{ - "id" => unsaved_job_id, - "name" => "Unsaved Job", - "workflow_id" => workflow.id - } - } - ) - - # Create a new user with no access - requesting_user = insert(:user) - - conn = build_conn() - conn = put_req_header(conn, "accept", "application/json") - conn = log_in_user(conn, requesting_user) - - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{unsaved_job_id}" - ) - - assert json_response(conn, 403) == %{"error" => "Forbidden"} - end - - test "returns 403 for a non-owner on a matching session with no workflow_id", - %{conn: _conn} do - # Matching session with no workflow to authorise against. - owner = insert(:user) - unsaved_job_id = Ecto.UUID.generate() - - _session = - insert(:chat_session, - user: owner, - session_type: "job_code", - job_id: nil, - title: "No-workflow unsaved session", - meta: %{"unsaved_job" => %{"id" => unsaved_job_id, "name" => "X"}} - ) - - requester = insert(:user) - - conn = - build_conn() - |> put_req_header("accept", "application/json") - |> log_in_user(requester) - - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{unsaved_job_id}" - ) - - assert json_response(conn, 403) == %{"error" => "Forbidden"} - end - - test "owner can list a matching session with no workflow_id", %{ - conn: conn, - user: user - } do - unsaved_job_id = Ecto.UUID.generate() - - session = - insert(:chat_session, - user: user, - session_type: "job_code", - job_id: nil, - title: "No-workflow unsaved session", - meta: %{"unsaved_job" => %{"id" => unsaved_job_id, "name" => "X"}} - ) - - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{unsaved_job_id}" - ) - - response = json_response(conn, 200) - assert %{"sessions" => sessions} = response - assert Enum.any?(sessions, &(&1["id"] == session.id)) - end - end - - describe "list_sessions for job_code with deleted job" do - setup [:register_and_log_in_user] - - test "returns empty list when job is deleted (job_id becomes nil)", %{ - conn: conn, - user: user - } do - # Create job and session, then delete job - project = insert(:project, project_users: [%{user_id: user.id}]) - workflow = insert(:workflow, project: project) - job = insert(:job, workflow: workflow, name: "Job to Delete") - job_id = job.id - - _session = - insert(:chat_session, - user: user, - session_type: "job_code", - job: job, - title: "Session for deleted job" - ) - - # Delete the job - this sets job_id to nil due to on_delete: :nilify_all - Lightning.Repo.delete!(job) - - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{job_id}" - ) - - response = json_response(conn, 200) - - # Session won't be found because job_id is now nil - assert %{"sessions" => [], "pagination" => %{"total_count" => 0}} = - response - end - end - - describe "list_sessions for workflow_template" do - setup [:register_and_log_in_user] - - test "returns sessions for project user has access to", %{ - conn: conn, - user: user - } do - project = insert(:project, project_users: [%{user_id: user.id}]) - - # Create multiple workflow template sessions - sessions = - for i <- 1..3 do - session = - insert(:workflow_chat_session, - user: user, - project: project, - title: "Workflow Template Session #{i}" - ) - - # Add delay to ensure different updated_at timestamps - Process.sleep(50) - - session - end - - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=workflow_template&project_id=#{project.id}" - ) - - response = json_response(conn, 200) - - assert %{ - "sessions" => returned_sessions, - "pagination" => pagination - } = response - - assert length(returned_sessions) == 3 - assert pagination["total_count"] == 3 - - # Verify all sessions are returned (don't check order due to flaky timestamps) - returned_ids = Enum.map(returned_sessions, & &1["id"]) |> MapSet.new() - created_ids = Enum.map(sessions, & &1.id) |> MapSet.new() - assert MapSet.equal?(returned_ids, created_ids) - - # Verify first session has correct structure - [session1 | _] = returned_sessions - assert session1["session_type"] == "workflow_template" - assert session1["project_name"] == project.name - end - - test "filters sessions by workflow_id when provided", %{ - conn: conn, - user: user - } do - project = insert(:project, project_users: [%{user_id: user.id}]) - - workflow1 = - insert(:workflow, project: project, name: "Workflow 1") - - workflow2 = - insert(:workflow, project: project, name: "Workflow 2") - - # Create sessions for different workflows - session1 = - insert(:workflow_chat_session, - user: user, - project: project, - workflow: workflow1, - title: "Workflow 1 Session" - ) - - _session2 = - insert(:workflow_chat_session, - user: user, - project: project, - workflow: workflow2, - title: "Workflow 2 Session" - ) - - # Request sessions filtered by workflow1 - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=workflow_template&project_id=#{project.id}&workflow_id=#{workflow1.id}" - ) - - response = json_response(conn, 200) - - assert %{"sessions" => returned_sessions} = response - - assert length(returned_sessions) == 1 - assert hd(returned_sessions)["id"] == session1.id - assert hd(returned_sessions)["workflow_name"] == workflow1.name - end - - test "includes workflow_name when session has workflow_id", %{ - conn: conn, - user: user - } do - project = insert(:project, project_users: [%{user_id: user.id}]) - workflow = insert(:workflow, project: project, name: "My Workflow") - - # Create session with workflow - must filter by workflow_id in query - # because API interprets missing workflow_id as "filter by nil" - session = - insert(:workflow_chat_session, - user: user, - project: project, - workflow: workflow, - title: "Workflow Session" - ) - - # Must include workflow_id in query to get sessions with that workflow - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=workflow_template&project_id=#{project.id}&workflow_id=#{workflow.id}" - ) - - response = json_response(conn, 200) - - assert %{"sessions" => [returned_session]} = response - - assert returned_session["id"] == session.id - assert returned_session["workflow_name"] == workflow.name - assert returned_session["project_name"] == project.name - end - - test "returns 403 when user does not have access to project", %{conn: _conn} do - # Create project owned by another user - other_user = insert(:user) - project = insert(:project, project_users: [%{user_id: other_user.id}]) - - # Create a new user with no access - requesting_user = insert(:user) - - conn = build_conn() - conn = put_req_header(conn, "accept", "application/json") - conn = log_in_user(conn, requesting_user) - - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=workflow_template&project_id=#{project.id}" - ) - - assert json_response(conn, 403) == %{"error" => "Forbidden"} - end - - test "returns empty list when no sessions exist", %{conn: conn, user: user} do - project = insert(:project, project_users: [%{user_id: user.id}]) - - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=workflow_template&project_id=#{project.id}" - ) - - response = json_response(conn, 200) - - assert %{ - "sessions" => [], - "pagination" => %{"total_count" => 0} - } = response - end - end - - describe "list_sessions authorization edge cases" do - setup [:register_and_log_in_user] - - test "viewer can list sessions for job they have read access to", %{ - conn: _conn - } do - # Create viewer user - viewer = insert(:user) - - project = - insert(:project, project_users: [%{user_id: viewer.id, role: :viewer}]) - - workflow = insert(:workflow, project: project) - job = insert(:job, workflow: workflow) - - # Create session - _session = - insert(:chat_session, - user: viewer, - session_type: "job_code", - job: job, - title: "Viewer Session" - ) - - conn = build_conn() - conn = put_req_header(conn, "accept", "application/json") - conn = log_in_user(conn, viewer) - - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{job.id}" - ) - - response = json_response(conn, 200) - - assert %{"sessions" => [_session]} = response - end - - test "admin can list sessions for project", %{conn: _conn} do - # Create admin user - admin = insert(:user) - - project = - insert(:project, project_users: [%{user_id: admin.id, role: :admin}]) - - _session = - %Lightning.AiAssistant.ChatSession{id: Ecto.UUID.generate()} - |> Lightning.AiAssistant.ChatSession.changeset(%{ - user_id: admin.id, - session_type: "workflow_template", - project_id: project.id, - title: "Admin Session", - meta: %{} - }) - |> Lightning.Repo.insert!() - - conn = build_conn() - conn = put_req_header(conn, "accept", "application/json") - conn = log_in_user(conn, admin) - - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=workflow_template&project_id=#{project.id}" - ) - - response = json_response(conn, 200) - - assert %{"sessions" => [_session]} = response - end - end - - describe "list_sessions message_count" do - setup [:register_and_log_in_user] - - test "includes accurate message_count for sessions", %{ - conn: conn, - user: user - } do - project = insert(:project, project_users: [%{user_id: user.id}]) - workflow = insert(:workflow, project: project) - job = insert(:job, workflow: workflow) - - # Create messages - messages = - for i <- 1..5 do - insert(:chat_message, - role: if(rem(i, 2) == 0, do: "user", else: "assistant"), - content: "Message #{i}", - status: "success" - ) - end - - session = - insert(:chat_session, - user: user, - session_type: "job_code", - job: job, - title: "Session with messages", - messages: messages - ) - - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{job.id}" - ) - - response = json_response(conn, 200) - - assert %{"sessions" => [returned_session]} = response - assert returned_session["id"] == session.id - assert returned_session["message_count"] == 5 - end - end - - # Helper functions - - defp create_job_with_sessions(%{user: user}) do - project = insert(:project, project_users: [%{user_id: user.id}]) - - workflow = - insert(:workflow, project: project, name: "Test Workflow") - - job = - insert(:job, - workflow: workflow, - name: "Test Job", - body: "fn(state => state);", - adaptor: "@openfn/language-common@latest" - ) - - # Create multiple sessions in reverse order so most recent is first - sessions = - for i <- 3..1//-1 do - session = - insert(:chat_session, - user: user, - session_type: "job_code", - job: job, - title: "Test Session #{i}" - ) - - # Add delay to ensure different updated_at timestamps - Process.sleep(50) - - session - end - - %{job: job, project: project, workflow: workflow, sessions: sessions} - end - - describe "list_sessions job_code validation errors" do - setup [:register_and_log_in_user] - - test "returns 400 when job_id is missing for job_code", %{conn: conn} do - conn = get(conn, ~p"/api/ai_assistant/sessions?session_type=job_code") - - assert json_response(conn, 400) == %{"error" => "Bad Request"} - end - end - - describe "list_sessions workflow_template validation errors" do - setup [:register_and_log_in_user] - - test "returns 400 when project_id is missing for workflow_template", %{ - conn: conn - } do - conn = - get(conn, ~p"/api/ai_assistant/sessions?session_type=workflow_template") - - assert json_response(conn, 400) == %{"error" => "Bad Request"} - end - - test "returns 404 when project doesn't exist for workflow_template", %{ - conn: conn - } do - non_existent_project_id = Ecto.UUID.generate() - - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=workflow_template&project_id=#{non_existent_project_id}" - ) - - assert json_response(conn, 404) == %{"error" => "Not Found"} - end - end - - describe "list_sessions with unsaved jobs" do - setup [:register_and_log_in_user] - - test "includes sessions with unsaved_job metadata", %{ - conn: conn, - user: user - } do - project = insert(:project, project_users: [%{user_id: user.id}]) - workflow = insert(:workflow, project: project) - job = insert(:job, workflow: workflow) - # Create session with unsaved_job in meta - unsaved_job_id = Ecto.UUID.generate() - - _session = - insert(:chat_session, - user: user, - session_type: "job_code", - job_id: nil, - meta: %{ - "unsaved_job" => %{ - "id" => unsaved_job_id, - "name" => "Unsaved Job", - "body" => "console.log('test');", - "adaptor" => "@openfn/language-common@1.0.0", - "workflow_id" => workflow.id - } - } - ) - - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{job.id}" - ) - - response = json_response(conn, 200) - assert %{"sessions" => sessions, "pagination" => _pagination} = response - # Verify sessions can be returned - assert is_list(sessions) - end - end - - describe "format_session edge cases" do - setup [:register_and_log_in_user] - - test "formats session with deleted job correctly", %{ - conn: conn, - user: user - } do - project = insert(:project, project_users: [%{user_id: user.id}]) - workflow = insert(:workflow, project: project) - # Create a job - job = - insert(:job, - workflow: workflow, - body: "console.log('test');", - adaptor: "@openfn/language-common@1.0.0" - ) - - # Create session for the job - session = insert(:chat_session, user: user, job: job) - - # Delete the job - Lightning.Repo.delete!(job) - - # List sessions should still work - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{session.job_id}" - ) - - response = json_response(conn, 200) - assert %{"sessions" => sessions, "pagination" => _pagination} = response - # Should return sessions even with deleted job - assert is_list(sessions) - end - - test "formats session without job_id or unsaved_job", %{ - conn: conn, - user: user - } do - project = insert(:project, project_users: [%{user_id: user.id}]) - workflow = insert(:workflow, project: project) - job = insert(:job, workflow: workflow) - - # Create a minimal session without job context - _session = - insert(:chat_session, - user: user, - session_type: "job_code", - job_id: nil, - meta: %{} - ) - - # List with different job_id (should not match) - conn = - get( - conn, - ~p"/api/ai_assistant/sessions?session_type=job_code&job_id=#{job.id}" - ) - - # Should return response with sessions list - response = json_response(conn, 200) - assert %{"sessions" => sessions, "pagination" => _pagination} = response - assert is_list(sessions) - end - end -end diff --git a/test/lightning_web/controllers/api/collection_controller_test.exs b/test/lightning_web/controllers/api/collection_controller_test.exs new file mode 100644 index 00000000000..9856662ebc9 --- /dev/null +++ b/test/lightning_web/controllers/api/collection_controller_test.exs @@ -0,0 +1,492 @@ +defmodule LightningWeb.API.CollectionControllerTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProblemHelpers + import Lightning.RaceHelpers + import Ecto.Query + import Lightning.Factories + import Lightning.ServiceAccountHelpers + + alias Lightning.Collections + alias Lightning.Collections.Collection + alias Lightning.Repo + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + defp service_account(%{conn: conn}) do + {conn, account} = with_service_account(conn, ["collections:write"]) + %{conn: conn, account: account, project: insert(:project)} + end + + defp entry(pointer, code, detail), + do: %{"pointer" => pointer, "code" => code, "detail" => detail} + + defp scheduled_for_deletion do + DateTime.utc_now() |> DateTime.truncate(:second) + end + + defp put_collection(conn, project_id, id, body) do + conn + |> put_req_header("content-type", "application/json") + |> put( + ~p"/api/projects/#{project_id}/collections/#{id}", + Jason.encode!(body) + ) + end + + defp with_pat(conn, user) do + assign_bearer(conn, Lightning.Accounts.generate_api_token(user)) + end + + defp audits(collection_id) do + Repo.all( + from a in Lightning.Collections.Audit.base_query(), + where: a.item_id == ^collection_id, + order_by: a.inserted_at + ) + end + + describe "PUT creates or renames" do + setup :service_account + + test "creates the collection under the path id, recording the account", %{ + conn: conn, + account: account, + project: project + } do + id = Ecto.UUID.generate() + + conn = put_collection(conn, project.id, id, %{"name" => "patients"}) + + assert json_response(conn, 201) == %{ + "data" => %{ + "id" => id, + "project_id" => project.id, + "name" => "patients" + } + } + + assert %Collection{project_id: project_id, name: "patients"} = + Repo.get(Collection, id) + + assert project_id == project.id + + assert [%{event: "created", actor_type: :service_account} = audit] = + audits(id) + + assert audit.actor_id == account.uuid + end + + test "re-sending what is stored answers 200 and records nothing", %{ + conn: conn, + project: project + } do + collection = insert(:collection, project: project, name: "patients") + + conn = + put_collection(conn, project.id, collection.id, %{"name" => "patients"}) + + assert json_response(conn, 200)["data"]["name"] == "patients" + assert audits(collection.id) == [] + end + + test "renames the collection, keeping its items", %{ + conn: conn, + project: project + } do + collection = insert(:collection, project: project, name: "patients") + insert(:collection_item, collection: collection, key: "k1", value: "v1") + + conn = + put_collection(conn, project.id, collection.id, %{"name" => "people"}) + + assert json_response(conn, 200)["data"] == %{ + "id" => collection.id, + "project_id" => project.id, + "name" => "people" + } + + assert Collections.get(collection, "k1").value == "v1" + + assert [%{event: "updated"} = audit] = audits(collection.id) + assert audit.changes.after == %{"name" => "people"} + end + + test "an owner or admin creates and renames, recorded as the actor", %{ + conn: conn + } do + for role <- [:owner, :admin] do + user = insert(:user) + project = insert(:project, project_users: [%{user: user, role: role}]) + id = Ecto.UUID.generate() + conn = with_pat(conn, user) + + assert json_response( + put_collection(conn, project.id, id, %{"name" => "patients"}), + 201 + ) + + assert json_response( + put_collection(conn, project.id, id, %{"name" => "people"}), + 200 + ) + + assert [ + %{event: "created", actor_type: :user} = created, + %{event: "updated", actor_type: :user} = updated + ] = audits(id) + + assert created.actor_id == user.id and updated.actor_id == user.id + end + end + end + + describe "PUT refuses the body" do + setup :service_account + + test "refuses a body it cannot take with one 422, each error at its key", %{ + conn: conn, + project: project + } do + unknown = &entry("#/" <> &1, "unknown_key", "is not accepted") + + blank = entry("#/name", "required", "can't be blank") + + unsafe = entry("#/name", "format", "Collection name must be URL safe") + + refused = [ + unknown_key: + {%{"name" => "patients", "items" => []}, [unknown.("items")]}, + project_key: + {%{"name" => "patients", "project_id" => insert(:project).id}, + [unknown.("project_id")]}, + other_id: + {%{"name" => "patients", "id" => Ecto.UUID.generate()}, + [ + entry("#/id", "mismatch", "does not match the id in the path") + ]}, + missing_name: {%{}, [blank]}, + blank_name: {%{"name" => ""}, [blank]}, + unsafe_name: {%{"name" => "Not Safe"}, [unsafe]}, + name_not_string: + {%{"name" => 7}, [entry("#/name", "type", "is invalid")]}, + all_at_once: + {%{"name" => "Not Safe", "colour" => "red"}, + [unknown.("colour"), unsafe]} + ] + + for {case_name, {body, entries}} <- refused do + id = Ecto.UUID.generate() + conn = put_collection(conn, project.id, id, body) + + assert conn.status == 422, "#{case_name}: #{conn.resp_body}" + assert_problem(conn, 422, :invalid, entries) + refute Repo.get(Collection, id), "#{case_name} created a collection" + end + end + + test "refuses a renaming body the same way, leaving the name", %{ + conn: conn, + project: project + } do + collection = insert(:collection, project: project, name: "patients") + + conn = + put_collection(conn, project.id, collection.id, %{ + "name" => "Not Safe", + "id" => Ecto.UUID.generate() + }) + + assert_problem(conn, 422, :invalid, [ + entry("#/id", "mismatch", "does not match the id in the path"), + entry("#/name", "format", "Collection name must be URL safe") + ]) + + assert Repo.reload!(collection).name == "patients" + end + + test "answers 400 malformed for a body that is not a JSON object", %{ + conn: conn, + project: project + } do + for body <- [["patients"], "patients", 7] do + id = Ecto.UUID.generate() + conn = put_collection(conn, project.id, id, body) + + assert_problem(conn, 400, :malformed, []) + refute Repo.get(Collection, id) + end + end + + test "never quotes a value from the request", %{conn: conn, project: project} do + collection = insert(:collection, project: project, name: "patients") + + for {body, entry} <- [ + {%{"name" => "Not Safe", "id" => "x"}, + %{ + "pointer" => "#/id", + "code" => "mismatch", + "detail" => "does not match the id in the path" + }}, + {%{"name" => "x", "colour" => "x"}, + %{ + "pointer" => "#/colour", + "code" => "unknown_key", + "detail" => "is not accepted" + }}, + {%{"name" => ["x"]}, + %{ + "pointer" => "#/name", + "code" => "type", + "detail" => "is invalid" + }} + ], + id <- [Ecto.UUID.generate(), collection.id] do + conn + |> put_collection(project.id, id, plant_sentinel(body)) + |> refute_sentinel() + |> assert_problem(422, :invalid, [entry]) + end + end + end + + describe "PUT refuses a name or id already held" do + setup :service_account + + test "answers 409 taken at the name when another collection in the project has it", + %{conn: conn, project: project} do + insert(:collection, project: project, name: "patients") + other = insert(:collection, project: project, name: "visits") + id = Ecto.UUID.generate() + + taken = [ + entry("#/name", "taken", "A collection with this name already exists") + ] + + created = put_collection(conn, project.id, id, %{"name" => "patients"}) + assert_problem(created, 409, :taken, taken) + refute Repo.get(Collection, id) + + renamed = + put_collection(conn, project.id, other.id, %{"name" => "patients"}) + + assert_problem(renamed, 409, :taken, taken) + assert Repo.reload!(other).name == "visits" + end + + test "answers 409 id_taken for an id held by a collection in another project, whatever the body", + %{conn: conn, project: project} do + elsewhere = insert(:collection, name: "patients") + + for body <- [%{"name" => "patients"}, %{"name" => "Not Safe", "x" => 1}] do + conn = put_collection(conn, project.id, elsewhere.id, body) + assert_problem(conn, 409, :id_taken, []) + end + + assert Repo.reload!(elsewhere).project_id != project.id + assert audits(elsewhere.id) == [] + end + end + + describe "PUT refuses a write it cannot land" do + setup :service_account + + test "a collection another request deletes mid-rename answers 409 concurrent_edit", + %{conn: conn, project: project} do + collection = insert(:collection, project: project, name: "patients") + + after_first_query("collections", collection.id, fn id -> + Repo.query!("DELETE FROM collections WHERE id = $1", [id]) + end) + + conn + |> put_collection(project.id, collection.id, %{"name" => "people"}) + |> assert_problem(409, :concurrent_edit, []) + + refute Repo.reload(collection) + end + + test "answers exactly 402 exceeds_limit when the hook refuses, writing nothing", + %{conn: conn, project: project} do + Mox.expect( + Lightning.Extensions.MockCollectionHook, + :handle_create, + fn %{"project_id" => project_id} -> + assert project_id == project.id + + {:error, :exceeds_limit, + %Lightning.Extensions.Message{text: "Upgrade to the PLANTED plan"}} + end + ) + + id = Ecto.UUID.generate() + conn = put_collection(conn, project.id, id, %{"name" => "patients"}) + + assert_problem(conn, 402, :exceeds_limit, []) + refute Repo.get(Collection, id) + assert audits(id) == [] + end + + test "answers 422 rather than 402 for a refused body, when the plan limit is also reached", + %{conn: conn, project: project} do + Mox.stub( + Lightning.Extensions.MockCollectionHook, + :handle_create, + fn _attrs -> + {:error, :exceeds_limit, %Lightning.Extensions.Message{text: "full"}} + end + ) + + conn = put_collection(conn, project.id, Ecto.UUID.generate(), %{}) + + assert_problem(conn, 422, :invalid, [ + entry("#/name", "required", "can't be blank") + ]) + end + end + + describe "answering 404 for what is not there" do + setup :service_account + + test "PUT answers 404 for a project or a path id that does not exist", %{ + conn: conn, + project: project + } do + for {project_id, id} <- [ + {Ecto.UUID.generate(), Ecto.UUID.generate()}, + {"not-a-uuid", Ecto.UUID.generate()}, + {project.id, "not-a-uuid"} + ] do + conn = put_collection(conn, project_id, id, %{"name" => "patients"}) + + assert_problem(conn, 404, :not_found, []) + end + end + + test "GET answers 404 for a collection of another project or none", %{ + conn: conn, + project: project + } do + elsewhere = insert(:collection) + + for id <- [elsewhere.id, Ecto.UUID.generate(), "not-a-uuid"] do + conn = get(conn, ~p"/api/projects/#{project.id}/collections/#{id}") + assert_problem(conn, 404, :not_found, []) + end + end + end + + describe "GET" do + setup :service_account + + test "a service account with the scope reads a project's collection", %{ + conn: conn + } do + collection = insert(:collection, name: "patients") + + conn = + get( + conn, + ~p"/api/projects/#{collection.project_id}/collections/#{collection.id}" + ) + + assert json_response(conn, 200) == %{ + "data" => %{ + "id" => collection.id, + "project_id" => collection.project_id, + "name" => "patients" + } + } + end + end + + describe "who may reach it" do + test "an editor or a viewer is refused with 403, an outsider with 404, an outsider on a project scheduled for deletion with 404", + %{conn: conn} do + outsider = insert(:user) + + for {role, deletion, status, code} <- [ + {:editor, nil, 403, :forbidden}, + {:viewer, nil, 403, :forbidden}, + {nil, nil, 404, :not_found}, + {nil, scheduled_for_deletion(), 404, :not_found} + ] do + user = if role, do: insert(:user), else: outsider + + project_users = if role, do: [%{user: user, role: role}], else: [] + + project = + insert(:project, + project_users: project_users, + scheduled_deletion: deletion + ) + + id = Ecto.UUID.generate() + + conn = + conn + |> with_pat(user) + |> put_collection(project.id, id, %{"name" => "patients"}) + + assert_problem(conn, status, code, []) + refute Repo.get(Collection, id) + end + end + + test "an owner of a project scheduled for deletion gets 409 writing or reading", + %{conn: conn} do + user = insert(:user) + + project = + insert(:project, + project_users: [%{user: user, role: :owner}], + scheduled_deletion: scheduled_for_deletion() + ) + + collection = insert(:collection, project: project) + id = Ecto.UUID.generate() + conn = with_pat(conn, user) + + written = put_collection(conn, project.id, id, %{"name" => "patients"}) + assert_problem(written, 409, :scheduled_for_deletion, []) + refute Repo.get(Collection, id) + + read = + get(conn, ~p"/api/projects/#{project.id}/collections/#{collection.id}") + + assert_problem(read, 409, :scheduled_for_deletion, []) + end + + test "reading needs :manage_collection on the project, checked after the collection exists", + %{conn: conn} do + admin = insert(:user) + editor = insert(:user) + + project = + insert(:project, + project_users: [ + %{user: admin, role: :admin}, + %{user: editor, role: :editor} + ] + ) + + collection = insert(:collection, project: project) + path = ~p"/api/projects/#{project.id}/collections/#{collection.id}" + + assert json_response(conn |> with_pat(admin) |> get(path), 200) + + assert_problem(conn |> with_pat(editor) |> get(path), 403, :forbidden, []) + + for id <- [Ecto.UUID.generate(), "not-a-uuid"] do + conn = + conn + |> with_pat(editor) + |> get(~p"/api/projects/#{project.id}/collections/#{id}") + + assert_problem(conn, 404, :not_found, []) + end + end + end +end diff --git a/test/lightning_web/controllers/api/credential_controller_test.exs b/test/lightning_web/controllers/api/credential_controller_test.exs deleted file mode 100644 index ae705acc64d..00000000000 --- a/test/lightning_web/controllers/api/credential_controller_test.exs +++ /dev/null @@ -1,964 +0,0 @@ -defmodule LightningWeb.API.CredentialControllerTest do - use LightningWeb.ConnCase, async: true - - import Lightning.Factories - - # Sample curl requests: - # - # # List your credentials - # curl http://localhost:4000/api/credentials \ - # -H "Authorization: Bearer $TOKEN" -H "Accept: application/json" - # - # # List credentials for a project - # curl http://localhost:4000/api/projects/$PROJECT_ID/credentials \ - # -H "Authorization: Bearer $TOKEN" -H "Accept: application/json" - # - # # Create a credential - # curl -X POST http://localhost:4000/api/credentials \ - # -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ - # -d '{"name":"My Cred","schema":"raw","credential_bodies":[{"name":"main","body":{"key":"val"}}]}' - # - # # Delete a credential - # curl -X DELETE http://localhost:4000/api/credentials/$CREDENTIAL_ID \ - # -H "Authorization: Bearer $TOKEN" - - setup %{conn: conn} do - {:ok, conn: put_req_header(conn, "accept", "application/json")} - end - - describe "without a token" do - test "index gets a 401", %{conn: conn} do - conn = get(conn, ~p"/api/credentials") - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - - test "project credentials index gets a 401", %{conn: conn} do - conn = get(conn, ~p"/api/projects/#{Ecto.UUID.generate()}/credentials") - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - - test "create gets a 401", %{conn: conn} do - conn = post(conn, ~p"/api/credentials", %{}) - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - - test "delete gets a 401", %{conn: conn} do - conn = delete(conn, ~p"/api/credentials/#{Ecto.UUID.generate()}") - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - end - - describe "with invalid token" do - test "index gets a 401", %{conn: conn} do - token = "InvalidToken" - conn = conn |> Plug.Conn.put_req_header("authorization", "Bearer #{token}") - conn = get(conn, ~p"/api/credentials") - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - - test "project credentials index gets a 401", %{conn: conn} do - token = "InvalidToken" - conn = conn |> Plug.Conn.put_req_header("authorization", "Bearer #{token}") - conn = get(conn, ~p"/api/projects/#{Ecto.UUID.generate()}/credentials") - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - - test "create gets a 401", %{conn: conn} do - token = "InvalidToken" - conn = conn |> Plug.Conn.put_req_header("authorization", "Bearer #{token}") - conn = post(conn, ~p"/api/credentials", %{}) - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - - test "delete gets a 401", %{conn: conn} do - token = "InvalidToken" - conn = conn |> Plug.Conn.put_req_header("authorization", "Bearer #{token}") - conn = delete(conn, ~p"/api/credentials/#{Ecto.UUID.generate()}") - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - end - - describe "index" do - setup [:assign_bearer_for_api] - - test "lists all credentials owned by the user", %{conn: conn, user: user} do - # Create some credentials for the user - _credential1 = - insert(:credential, user: user, name: "First Credential", schema: "raw") - - _credential2 = - insert(:credential, user: user, name: "Second Credential", schema: "raw") - - # Create a credential for another user (should not appear) - other_user = insert(:user) - - _other_credential = - insert(:credential, - user: other_user, - name: "Other User Credential", - schema: "raw" - ) - - conn = get(conn, ~p"/api/credentials") - response = json_response(conn, 200) - - assert %{ - "credentials" => credentials, - "errors" => %{} - } = response - - assert length(credentials) == 2 - - # Check that credentials are returned without body field - returned_names = Enum.map(credentials, & &1["name"]) |> Enum.sort() - assert returned_names == ["First Credential", "Second Credential"] - - # Verify body field is excluded for security - Enum.each(credentials, fn credential -> - refute Map.has_key?(credential, "body") - assert credential["user_id"] == user.id - end) - - # Check specific credential structure - first_credential = - Enum.find(credentials, &(&1["name"] == "First Credential")) - - assert %{ - "id" => _, - "name" => "First Credential", - "schema" => _, - "external_id" => _, - "user_id" => user_id, - "project_credentials" => _, - "projects" => _, - "inserted_at" => _, - "updated_at" => _ - } = first_credential - - assert user_id == user.id - # production field no longer exists - refute Map.has_key?(first_credential, "production") - end - - test "returns empty list when user has no credentials", %{ - conn: conn, - user: _user - } do - conn = get(conn, ~p"/api/credentials") - response = json_response(conn, 200) - - assert %{ - "credentials" => [], - "errors" => %{} - } = response - end - - test "includes project associations in the response", %{ - conn: conn, - user: user - } do - project = - insert(:project, project_users: [%{user_id: user.id, role: :owner}]) - - _credential = - insert(:credential, - user: user, - name: "Project Credential", - schema: "raw", - project_credentials: [%{project_id: project.id}] - ) - - conn = get(conn, ~p"/api/credentials") - response = json_response(conn, 200) - - assert %{"credentials" => [credential_data]} = response - assert credential_data["name"] == "Project Credential" - assert length(credential_data["project_credentials"]) == 1 - assert length(credential_data["projects"]) == 1 - - project_data = List.first(credential_data["projects"]) - assert project_data["id"] == project.id - assert project_data["name"] == project.name - end - end - - describe "index for specific project" do - setup [:assign_bearer_for_api] - - test "lists all credentials in a project when user has access", %{ - conn: conn, - user: user - } do - project = - insert(:project, project_users: [%{user_id: user.id, role: :editor}]) - - # Create credentials for different users but in the same project - _user_credential = - insert(:credential, - user: user, - name: "User Credential", - schema: "raw", - project_credentials: [%{project_id: project.id}] - ) - - other_user = insert(:user) - - _other_user_credential = - insert(:credential, - user: other_user, - name: "Other User Credential", - schema: "raw", - project_credentials: [%{project_id: project.id}] - ) - - # Create a credential not in this project (should not appear) - _unrelated_credential = - insert(:credential, - user: user, - name: "Unrelated Credential", - schema: "raw" - ) - - conn = get(conn, ~p"/api/projects/#{project.id}/credentials") - response = json_response(conn, 200) - - assert %{ - "credentials" => credentials, - "errors" => %{} - } = response - - assert length(credentials) == 2 - - # Should include credentials from all users that have access to this project - returned_names = Enum.map(credentials, & &1["name"]) |> Enum.sort() - assert returned_names == ["Other User Credential", "User Credential"] - - # Verify body field is excluded for security - Enum.each(credentials, fn credential -> - refute Map.has_key?(credential, "body") - end) - end - - test "does not disclose sibling projects the caller is not a member of", %{ - conn: conn, - user: user - } do - project_a = - insert(:project, - name: "project-a", - project_users: [%{user_id: user.id, role: :viewer}] - ) - - # A different tenant's project the caller has no membership in. - other_project = insert(:project, name: "other-tenant-secret") - - owner = insert(:user) - - _shared_credential = - insert(:credential, - user: owner, - name: "Shared Credential", - schema: "raw", - project_credentials: [ - %{project_id: project_a.id}, - %{project_id: other_project.id} - ] - ) - - conn = get(conn, ~p"/api/projects/#{project_a.id}/credentials") - - assert %{"credentials" => [credential]} = json_response(conn, 200) - - rendered_project_ids = Enum.map(credential["projects"], & &1["id"]) - assert project_a.id in rendered_project_ids - refute other_project.id in rendered_project_ids - - refute Enum.any?( - credential["projects"], - &(&1["name"] == "other-tenant-secret") - ) - - # The sibling project's UUID must not leak via project_credentials either. - pc_project_ids = - Enum.map(credential["project_credentials"], & &1["project_id"]) - - assert project_a.id in pc_project_ids - refute other_project.id in pc_project_ids - end - - test "shows the caller's own association when the project is a sandbox", %{ - conn: conn, - user: user - } do - root = insert(:project) - - sandbox = - insert(:project, - parent: root, - project_users: [%{user_id: user.id, role: :viewer}] - ) - - owner = insert(:user) - - _credential = - insert(:credential, - user: owner, - name: "Sandbox Credential", - schema: "raw", - project_credentials: [%{project_id: sandbox.id}] - ) - - conn = get(conn, ~p"/api/projects/#{sandbox.id}/credentials") - - assert %{"credentials" => [credential]} = json_response(conn, 200) - assert Enum.map(credential["projects"], & &1["id"]) == [sandbox.id] - - assert Enum.map(credential["project_credentials"], & &1["project_id"]) == - [sandbox.id] - end - - test "returns 403 when user lacks access to project", %{ - conn: conn, - user: _user - } do - other_user = insert(:user) - - project = - insert(:project, - project_users: [%{user_id: other_user.id, role: :owner}] - ) - - conn = get(conn, ~p"/api/projects/#{project.id}/credentials") - assert json_response(conn, 403) == %{"error" => "Forbidden"} - end - - test "returns 404 when project does not exist", %{conn: conn, user: _user} do - non_existent_id = Ecto.UUID.generate() - - conn = get(conn, ~p"/api/projects/#{non_existent_id}/credentials") - assert json_response(conn, 404) == %{"error" => "Not Found"} - end - - test "returns empty list when project has no credentials", %{ - conn: conn, - user: user - } do - project = - insert(:project, project_users: [%{user_id: user.id, role: :owner}]) - - conn = get(conn, ~p"/api/projects/#{project.id}/credentials") - response = json_response(conn, 200) - - assert %{ - "credentials" => [], - "errors" => %{} - } = response - end - - test "allows access for support users with project access", %{conn: conn} do - support_user = insert(:user, support_user: true) - project = insert(:project, allow_support_access: true) - - _credential = - insert(:credential, - user: support_user, - name: "Support Credential", - schema: "raw", - project_credentials: [%{project_id: project.id}] - ) - - token = Lightning.Accounts.generate_api_token(support_user) - conn = conn |> Plug.Conn.put_req_header("authorization", "Bearer #{token}") - - conn = get(conn, ~p"/api/projects/#{project.id}/credentials") - response = json_response(conn, 200) - - assert %{"credentials" => [credential_data]} = response - assert credential_data["name"] == "Support Credential" - end - - test "includes project information in credential data", %{ - conn: conn, - user: user - } do - project = - insert(:project, project_users: [%{user_id: user.id, role: :admin}]) - - _credential = - insert(:credential, - user: user, - name: "Project Credential", - schema: "raw", - project_credentials: [%{project_id: project.id}] - ) - - conn = get(conn, ~p"/api/projects/#{project.id}/credentials") - response = json_response(conn, 200) - - assert %{"credentials" => [credential_data]} = response - assert length(credential_data["projects"]) == 1 - assert length(credential_data["project_credentials"]) == 1 - - project_data = List.first(credential_data["projects"]) - assert project_data["id"] == project.id - assert project_data["name"] == project.name - end - end - - describe "create" do - setup [:assign_bearer_for_api] - - test "refuses a repo connection token instead of crashing", %{conn: conn} do - # `require_authenticated_api_resource` lets these through the pipeline, - # and a credential belongs to a person, so there is no sensible owner to - # give it. It used to fall over on the user association and come back a - # 422; it must not become a 500 now that the actor is threaded through. - repo_connection = - insert(:project_repo_connection, project: insert(:project)) - - conn = - Plug.Conn.put_req_header( - conn, - "authorization", - "Bearer #{repo_connection.access_token}" - ) - - assert conn - |> post(~p"/api/credentials", %{ - "name" => "Planted", - "schema" => "raw", - "credential_bodies" => [ - %{"name" => "main", "body" => %{"a" => "b"}} - ] - }) - |> json_response(403) - - refute Lightning.Repo.get_by(Lightning.Credentials.Credential, - name: "Planted" - ) - - # Listing crashed the same way, for the same reason, on both routes. - assert conn |> get(~p"/api/credentials") |> json_response(403) - - project = insert(:project) - - assert conn - |> get(~p"/api/projects/#{project.id}/credentials") - |> json_response(403) - end - - test "creates a basic credential without project associations", %{ - conn: conn, - user: user - } do - credential_attrs = %{ - "name" => "Test Credential", - "schema" => "raw", - "credential_bodies" => [ - %{ - "name" => "main", - "body" => %{"username" => "test", "password" => "secret"} - } - ] - } - - conn = post(conn, ~p"/api/credentials", credential_attrs) - response = json_response(conn, 201) - - assert %{ - "credential" => %{ - "id" => id, - "name" => "Test Credential", - "schema" => "raw", - "external_id" => nil, - "user_id" => user_id, - "project_credentials" => [], - "projects" => [] - }, - "errors" => %{} - } = response - - assert is_binary(id) - assert user_id == user.id - # body should be excluded - refute Map.has_key?(response["credential"], "body") - # production field no longer exists - refute Map.has_key?(response["credential"], "production") - end - - test "creates a credential with project associations when user has access", - %{conn: conn, user: user} do - project = - insert(:project, project_users: [%{user_id: user.id, role: :editor}]) - - credential_attrs = %{ - "name" => "Project Credential", - "schema" => "raw", - "credential_bodies" => [ - %{ - "name" => "main", - "body" => %{"api_key" => "secret"} - } - ], - "project_credentials" => [ - %{"project_id" => project.id} - ] - } - - conn = post(conn, ~p"/api/credentials", credential_attrs) - response = json_response(conn, 201) - - assert %{ - "credential" => %{ - "id" => id, - "name" => "Project Credential", - "user_id" => user_id, - "project_credentials" => [project_credential], - "projects" => [project_data] - } - } = response - - assert is_binary(id) - assert user_id == user.id - assert project_credential["project_id"] == project.id - assert project_data["id"] == project.id - assert project_data["name"] == project.name - end - - test "creates a credential with multiple project associations", %{ - conn: conn, - user: user - } do - project1 = - insert(:project, project_users: [%{user_id: user.id, role: :admin}]) - - project2 = - insert(:project, project_users: [%{user_id: user.id, role: :owner}]) - - credential_attrs = %{ - "name" => "Multi-Project Credential", - "schema" => "raw", - "credential_bodies" => [ - %{ - "name" => "main", - "body" => %{"token" => "abc123"} - } - ], - "project_credentials" => [ - %{"project_id" => project1.id}, - %{"project_id" => project2.id} - ] - } - - conn = post(conn, ~p"/api/credentials", credential_attrs) - response = json_response(conn, 201) - - assert %{ - "credential" => %{ - "project_credentials" => project_credentials, - "projects" => projects - } - } = response - - assert length(project_credentials) == 2 - assert length(projects) == 2 - - project_ids = Enum.map(projects, & &1["id"]) - assert project1.id in project_ids - assert project2.id in project_ids - end - - test "creates a credential with multiple environment bodies", %{ - conn: conn, - user: user - } do - credential_attrs = %{ - "name" => "Multi-Environment Credential", - "schema" => "raw", - "credential_bodies" => [ - %{ - "name" => "main", - "body" => %{"api_key" => "main_key"} - }, - %{ - "name" => "production", - "body" => %{"api_key" => "prod_key"} - }, - %{ - "name" => "staging", - "body" => %{"api_key" => "staging_key"} - } - ] - } - - conn = post(conn, ~p"/api/credentials", credential_attrs) - response = json_response(conn, 201) - - assert %{ - "credential" => %{ - "id" => id, - "name" => "Multi-Environment Credential", - "user_id" => user_id - } - } = response - - assert is_binary(id) - assert user_id == user.id - - # Verify bodies were created - credential = Lightning.Credentials.get_credential(id) - credential = Lightning.Repo.preload(credential, :credential_bodies) - assert length(credential.credential_bodies) == 3 - - env_names = - Enum.map(credential.credential_bodies, & &1.name) |> Enum.sort() - - assert env_names == ["main", "production", "staging"] - end - - test "fails when user lacks access to project", %{conn: conn, user: _user} do - other_user = insert(:user) - - project = - insert(:project, - project_users: [%{user_id: other_user.id, role: :owner}] - ) - - credential_attrs = %{ - "name" => "Unauthorized Credential", - "schema" => "raw", - "credential_bodies" => [ - %{ - "name" => "main", - "body" => %{"secret" => "value"} - } - ], - "project_credentials" => [ - %{"project_id" => project.id} - ] - } - - conn = post(conn, ~p"/api/credentials", credential_attrs) - assert json_response(conn, 403) == %{"error" => "Forbidden"} - end - - test "fails when user has insufficient role (viewer) on project", %{ - conn: conn, - user: user - } do - project = - insert(:project, project_users: [%{user_id: user.id, role: :viewer}]) - - credential_attrs = %{ - "name" => "Viewer Credential", - "schema" => "raw", - "credential_bodies" => [ - %{ - "name" => "main", - "body" => %{"secret" => "value"} - } - ], - "project_credentials" => [ - %{"project_id" => project.id} - ] - } - - conn = post(conn, ~p"/api/credentials", credential_attrs) - assert json_response(conn, 403) == %{"error" => "Forbidden"} - end - - test "fails when project does not exist", %{conn: conn, user: _user} do - credential_attrs = %{ - "name" => "Nonexistent Project Credential", - "schema" => "raw", - "credential_bodies" => [ - %{ - "name" => "main", - "body" => %{"secret" => "value"} - } - ], - "project_credentials" => [ - %{"project_id" => Ecto.UUID.generate()} - ] - } - - conn = post(conn, ~p"/api/credentials", credential_attrs) - assert json_response(conn, 403) == %{"error" => "Forbidden"} - end - - test "allows access when user is support user with project access", %{ - conn: conn - } do - support_user = insert(:user, support_user: true) - project = insert(:project, allow_support_access: true) - - token = Lightning.Accounts.generate_api_token(support_user) - conn = conn |> Plug.Conn.put_req_header("authorization", "Bearer #{token}") - - credential_attrs = %{ - "name" => "Support User Credential", - "schema" => "raw", - "credential_bodies" => [ - %{ - "name" => "main", - "body" => %{"secret" => "value"} - } - ], - "project_credentials" => [ - %{"project_id" => project.id} - ] - } - - conn = post(conn, ~p"/api/credentials", credential_attrs) - response = json_response(conn, 201) - - assert %{ - "credential" => %{ - "name" => "Support User Credential", - "user_id" => user_id - } - } = response - - assert user_id == support_user.id - end - - test "fails with invalid credential data", %{conn: conn, user: _user} do - credential_attrs = %{ - # Invalid: empty name - "name" => "", - "schema" => "raw", - "credential_bodies" => [ - %{ - "name" => "main", - "body" => %{"secret" => "value"} - } - ] - } - - conn = post(conn, ~p"/api/credentials", credential_attrs) - response = json_response(conn, 422) - - assert %{"errors" => errors} = response - assert Map.has_key?(errors, "name") - end - - test "fails when missing required fields", %{conn: conn, user: _user} do - credential_attrs = %{ - "name" => "" - } - - conn = post(conn, ~p"/api/credentials", credential_attrs) - response = json_response(conn, 422) - - assert %{"errors" => _errors} = response - end - - test "cannot override user_id in request", %{conn: conn, user: user} do - other_user = insert(:user) - - credential_attrs = %{ - "name" => "Test Credential", - "schema" => "raw", - "credential_bodies" => [ - %{ - "name" => "main", - "body" => %{"secret" => "value"} - } - ], - # This should be ignored - "user_id" => other_user.id - } - - conn = post(conn, ~p"/api/credentials", credential_attrs) - response = json_response(conn, 201) - - # Should use the authenticated user, not the provided user_id - assert response["credential"]["user_id"] == user.id - refute response["credential"]["user_id"] == other_user.id - end - - test "returns 201 created status", %{conn: conn, user: _user} do - credential_attrs = %{ - "name" => "Status Test", - "schema" => "raw", - "credential_bodies" => [ - %{ - "name" => "main", - "body" => %{"secret" => "value"} - } - ] - } - - conn = post(conn, ~p"/api/credentials", credential_attrs) - assert json_response(conn, 201) - end - - test "handles partial project access - allows accessible, denies inaccessible", - %{conn: conn, user: user} do - accessible_project = - insert(:project, project_users: [%{user_id: user.id, role: :editor}]) - - # No user access - inaccessible_project = insert(:project) - - credential_attrs = %{ - "name" => "Mixed Access Credential", - "schema" => "raw", - "credential_bodies" => [ - %{ - "name" => "main", - "body" => %{"secret" => "value"} - } - ], - "project_credentials" => [ - %{"project_id" => accessible_project.id}, - %{"project_id" => inaccessible_project.id} - ] - } - - conn = post(conn, ~p"/api/credentials", credential_attrs) - assert json_response(conn, 403) == %{"error" => "Forbidden"} - end - end - - describe "editor role permissions" do - setup [:assign_bearer_for_api] - - test "editor can create project credentials", %{conn: conn, user: user} do - project = - insert(:project, project_users: [%{user_id: user.id, role: :editor}]) - - credential_attrs = %{ - "name" => "Editor Credential", - "schema" => "raw", - "credential_bodies" => [ - %{ - "name" => "main", - "body" => %{"secret" => "value"} - } - ], - "project_credentials" => [%{"project_id" => project.id}] - } - - conn = post(conn, ~p"/api/credentials", credential_attrs) - assert json_response(conn, 201) - end - end - - describe "admin role permissions" do - setup [:assign_bearer_for_api] - - test "admin can create project credentials", %{conn: conn, user: user} do - project = - insert(:project, project_users: [%{user_id: user.id, role: :admin}]) - - credential_attrs = %{ - "name" => "Admin Credential", - "schema" => "raw", - "credential_bodies" => [ - %{ - "name" => "main", - "body" => %{"secret" => "value"} - } - ], - "project_credentials" => [%{"project_id" => project.id}] - } - - conn = post(conn, ~p"/api/credentials", credential_attrs) - assert json_response(conn, 201) - end - end - - describe "owner role permissions" do - setup [:assign_bearer_for_api] - - test "owner can create project credentials", %{conn: conn, user: user} do - project = - insert(:project, project_users: [%{user_id: user.id, role: :owner}]) - - credential_attrs = %{ - "name" => "Owner Credential", - "schema" => "raw", - "credential_bodies" => [ - %{ - "name" => "main", - "body" => %{"secret" => "value"} - } - ], - "project_credentials" => [%{"project_id" => project.id}] - } - - conn = post(conn, ~p"/api/credentials", credential_attrs) - assert json_response(conn, 201) - end - end - - describe "delete" do - setup [:assign_bearer_for_api] - - test "deletes a credential owned by the user", %{conn: conn, user: user} do - credential = - insert(:credential, user: user, name: "To Be Deleted", schema: "raw") - - conn = delete(conn, ~p"/api/credentials/#{credential.id}") - assert response(conn, 204) == "" - - # Verify credential is deleted - refute Lightning.Credentials.get_credential(credential.id) - end - - test "returns 404 when credential does not exist", %{conn: conn, user: _user} do - non_existent_id = Ecto.UUID.generate() - - conn = delete(conn, ~p"/api/credentials/#{non_existent_id}") - assert json_response(conn, 404) == %{"error" => "Not Found"} - end - - test "returns 403 when trying to delete credential owned by another user", %{ - conn: conn, - user: _user - } do - other_user = insert(:user) - - other_credential = - insert(:credential, - user: other_user, - name: "Other User Credential", - schema: "raw" - ) - - conn = delete(conn, ~p"/api/credentials/#{other_credential.id}") - assert json_response(conn, 403) == %{"error" => "Forbidden"} - - # Verify credential still exists - assert Lightning.Credentials.get_credential(other_credential.id) - end - - test "deletes credential with project associations", %{ - conn: conn, - user: user - } do - project = - insert(:project, project_users: [%{user_id: user.id, role: :owner}]) - - credential = - insert(:credential, - user: user, - name: "Credential with Projects", - schema: "raw", - project_credentials: [%{project_id: project.id}] - ) - - conn = delete(conn, ~p"/api/credentials/#{credential.id}") - assert response(conn, 204) == "" - - # Verify credential and associations are deleted - refute Lightning.Credentials.get_credential(credential.id) - end - - test "handles invalid UUID format", %{conn: conn, user: _user} do - conn = delete(conn, ~p"/api/credentials/invalid-uuid") - assert json_response(conn, 404) == %{"error" => "Not Found"} - end - end -end diff --git a/test/lightning_web/controllers/api/credentials/reading_test.exs b/test/lightning_web/controllers/api/credentials/reading_test.exs new file mode 100644 index 00000000000..6d9a7eac506 --- /dev/null +++ b/test/lightning_web/controllers/api/credentials/reading_test.exs @@ -0,0 +1,508 @@ +defmodule LightningWeb.API.Credentials.ReadingTest do + use LightningWeb.ConnCase, async: true + + import Lightning.Factories + import Lightning.QueryCountHelpers + import LightningWeb.API.ProblemHelpers + import LightningWeb.ApiCredentialsHelpers + import Lightning.ServiceAccountHelpers + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + describe "listing what the caller owns" do + setup [:assign_bearer_for_api] + + test "lists all credentials owned by the user", %{conn: conn, user: user} do + # Create some credentials for the user + _credential1 = + insert(:credential, user: user, name: "First Credential", schema: "raw") + + _credential2 = + insert(:credential, user: user, name: "Second Credential", schema: "raw") + + # Create a credential for another user (should not appear) + other_user = insert(:user) + + _other_credential = + insert(:credential, + user: other_user, + name: "Other User Credential", + schema: "raw" + ) + + conn = get(conn, ~p"/api/credentials") + response = json_response(conn, 200) + + assert %{ + "credentials" => credentials, + "errors" => %{} + } = response + + assert length(credentials) == 2 + + # Check that credentials are returned without body field + returned_names = Enum.map(credentials, & &1["name"]) |> Enum.sort() + assert returned_names == ["First Credential", "Second Credential"] + + # Verify body field is excluded for security + Enum.each(credentials, fn credential -> + refute Map.has_key?(credential, "body") + assert credential["user_id"] == user.id + end) + + # Check specific credential structure + first_credential = + Enum.find(credentials, &(&1["name"] == "First Credential")) + + assert %{ + "id" => _, + "name" => "First Credential", + "schema" => _, + "external_id" => _, + "user_id" => user_id, + "project_credentials" => _, + "projects" => _, + "inserted_at" => _, + "updated_at" => _ + } = first_credential + + assert user_id == user.id + refute Map.has_key?(first_credential, "production") + end + + test "names a linked project only to a caller who may access it", %{ + conn: conn, + user: user + } do + # A member who has not enrolled in MFA may not access this project, so the + # response does not name it, just as showing one credential doesn't. + mfa_project = + insert(:project, + requires_mfa: true, + project_users: [%{user_id: user.id, role: :owner}] + ) + + open_project = + insert(:project, project_users: [%{user_id: user.id, role: :owner}]) + + insert(:credential, + user: user, + schema: "raw", + project_credentials: [ + %{project_id: mfa_project.id}, + %{project_id: open_project.id} + ] + ) + + assert %{"credentials" => [credential]} = + conn |> get(~p"/api/credentials") |> json_response(200) + + assert Enum.map(credential["projects"], & &1["id"]) == [open_project.id] + + assert Enum.map(credential["project_credentials"], & &1["project_id"]) == + [open_project.id] + end + + test "returns empty list when user has no credentials", %{ + conn: conn, + user: _user + } do + conn = get(conn, ~p"/api/credentials") + response = json_response(conn, 200) + + assert %{ + "credentials" => [], + "errors" => %{} + } = response + end + + test "includes project associations in the response", %{ + conn: conn, + user: user + } do + project = + insert(:project, project_users: [%{user_id: user.id, role: :owner}]) + + _credential = + insert(:credential, + user: user, + name: "Project Credential", + schema: "raw", + project_credentials: [%{project_id: project.id}] + ) + + conn = get(conn, ~p"/api/credentials") + response = json_response(conn, 200) + + assert %{"credentials" => [credential_data]} = response + assert credential_data["name"] == "Project Credential" + assert length(credential_data["project_credentials"]) == 1 + assert length(credential_data["projects"]) == 1 + + project_data = List.first(credential_data["projects"]) + assert project_data["id"] == project.id + assert project_data["name"] == project.name + end + end + + describe "listing a project's credentials" do + setup [:assign_bearer_for_api] + + test "lists all credentials in a project when user has access", %{ + conn: conn, + user: user + } do + project = + insert(:project, project_users: [%{user_id: user.id, role: :editor}]) + + # Create credentials for different users but in the same project + _user_credential = + insert(:credential, + user: user, + name: "User Credential", + schema: "raw", + project_credentials: [%{project_id: project.id}] + ) + + other_user = insert(:user) + + _other_user_credential = + insert(:credential, + user: other_user, + name: "Other User Credential", + schema: "raw", + project_credentials: [%{project_id: project.id}] + ) + + # Create a credential not in this project (should not appear) + _unrelated_credential = + insert(:credential, + user: user, + name: "Unrelated Credential", + schema: "raw" + ) + + conn = get(conn, ~p"/api/projects/#{project.id}/credentials") + response = json_response(conn, 200) + + assert %{ + "credentials" => credentials, + "errors" => %{} + } = response + + assert length(credentials) == 2 + + # Should include credentials from all users that have access to this project + returned_names = Enum.map(credentials, & &1["name"]) |> Enum.sort() + assert returned_names == ["Other User Credential", "User Credential"] + + # Verify body field is excluded for security + Enum.each(credentials, fn credential -> + refute Map.has_key?(credential, "body") + end) + end + + test "does not disclose sibling projects the caller is not a member of", %{ + conn: conn, + user: user + } do + project_a = + insert(:project, + name: "project-a", + project_users: [%{user_id: user.id, role: :viewer}] + ) + + # A different tenant's project the caller has no membership in. + other_project = insert(:project, name: "other-tenant-secret") + + owner = insert(:user) + + _shared_credential = + insert(:credential, + user: owner, + name: "Shared Credential", + schema: "raw", + project_credentials: [ + %{project_id: project_a.id}, + %{project_id: other_project.id} + ] + ) + + conn = get(conn, ~p"/api/projects/#{project_a.id}/credentials") + + assert %{"credentials" => [credential]} = json_response(conn, 200) + + rendered_project_ids = Enum.map(credential["projects"], & &1["id"]) + assert project_a.id in rendered_project_ids + refute other_project.id in rendered_project_ids + + refute Enum.any?( + credential["projects"], + &(&1["name"] == "other-tenant-secret") + ) + + # The sibling project's UUID must not leak via project_credentials either. + pc_project_ids = + Enum.map(credential["project_credentials"], & &1["project_id"]) + + assert project_a.id in pc_project_ids + refute other_project.id in pc_project_ids + end + + test "shows the caller's own association when the project is a sandbox", %{ + conn: conn, + user: user + } do + root = insert(:project) + + sandbox = + insert(:project, + parent: root, + project_users: [%{user_id: user.id, role: :viewer}] + ) + + owner = insert(:user) + + _credential = + insert(:credential, + user: owner, + name: "Sandbox Credential", + schema: "raw", + project_credentials: [%{project_id: sandbox.id}] + ) + + conn = get(conn, ~p"/api/projects/#{sandbox.id}/credentials") + + assert %{"credentials" => [credential]} = json_response(conn, 200) + assert Enum.map(credential["projects"], & &1["id"]) == [sandbox.id] + + assert Enum.map(credential["project_credentials"], & &1["project_id"]) == + [sandbox.id] + end + + test "answers 404 for a project the caller isn't a member of, or that doesn't exist", + %{conn: conn} do + project = + insert(:project, + project_users: [%{user_id: insert(:user).id, role: :owner}] + ) + + for id <- [project.id, Ecto.UUID.generate()] do + conn + |> get(~p"/api/projects/#{id}/credentials") + |> assert_problem(404, :not_found, []) + end + end + + test "tells a member, and only a member, that the project is scheduled for deletion", + %{conn: conn, user: user} do + scheduled_deletion = DateTime.utc_now() |> DateTime.truncate(:second) + + member_of = + insert(:project, + project_users: [%{user_id: user.id, role: :editor}], + scheduled_deletion: scheduled_deletion + ) + + stranger_to = insert(:project, scheduled_deletion: scheduled_deletion) + + conn + |> get(~p"/api/projects/#{member_of.id}/credentials") + |> assert_problem(409, :scheduled_for_deletion, []) + + conn + |> get(~p"/api/projects/#{stranger_to.id}/credentials") + |> assert_problem(404, :not_found, []) + end + + test "returns empty list when project has no credentials", %{ + conn: conn, + user: user + } do + project = + insert(:project, project_users: [%{user_id: user.id, role: :owner}]) + + conn = get(conn, ~p"/api/projects/#{project.id}/credentials") + response = json_response(conn, 200) + + assert %{ + "credentials" => [], + "errors" => %{} + } = response + end + + test "allows access for support users with project access", %{conn: conn} do + support_user = insert(:user, support_user: true) + project = insert(:project, allow_support_access: true) + + _credential = + insert(:credential, + user: support_user, + name: "Support Credential", + schema: "raw", + project_credentials: [%{project_id: project.id}] + ) + + token = Lightning.Accounts.generate_api_token(support_user) + conn = conn |> Plug.Conn.put_req_header("authorization", "Bearer #{token}") + + conn = get(conn, ~p"/api/projects/#{project.id}/credentials") + response = json_response(conn, 200) + + assert %{"credentials" => [credential_data]} = response + assert credential_data["name"] == "Support Credential" + end + + test "includes project information in credential data", %{ + conn: conn, + user: user + } do + project = + insert(:project, project_users: [%{user_id: user.id, role: :admin}]) + + _credential = + insert(:credential, + user: user, + name: "Project Credential", + schema: "raw", + project_credentials: [%{project_id: project.id}] + ) + + conn = get(conn, ~p"/api/projects/#{project.id}/credentials") + response = json_response(conn, 200) + + assert %{"credentials" => [credential_data]} = response + assert length(credential_data["projects"]) == 1 + assert length(credential_data["project_credentials"]) == 1 + + project_data = List.first(credential_data["projects"]) + assert project_data["id"] == project.id + assert project_data["name"] == project.name + end + end + + describe "showing one credential" do + test "a service account with the scope reads any credential, never its body", + %{conn: conn} do + {conn, _account} = with_service_account(conn, ["credentials:write"]) + owner = insert(:user) + project = insert(:project) + credential = existing_credential(owner, project) + + conn = get(conn, ~p"/api/credentials/#{credential.id}") + + assert %{ + "credential" => %{ + "id" => id, + "user_id" => user_id, + "projects" => [] + }, + "errors" => %{} + } = response = json_response(conn, 200) + + assert {id, user_id} == {credential.id, owner.id} + assert linked_project_ids(response) == [project.id] + refute conn.resp_body =~ "password" + + conn + |> get(~p"/api/credentials/#{Ecto.UUID.generate()}") + |> assert_problem(404, :not_found, []) + end + + test "a service account also holding projects:write sees the projects by name", + %{conn: conn} do + {conn, _account} = + with_service_account(conn, ["credentials:write", "projects:write"]) + + project = insert(:project) + credential = existing_credential(insert(:user), project) + + response = + conn + |> get(~p"/api/credentials/#{credential.id}") + |> json_response(200) + + assert [%{"id" => project_id}] = response["credential"]["projects"] + assert project_id == project.id + end + + test "a person reads only their own, seeing links only to projects they stand in", + %{conn: conn} do + owner = insert(:user) + theirs = insert(:project, project_users: [%{user: owner, role: :owner}]) + + winding_down = + insert(:project, + project_users: [%{user: owner, role: :owner}], + scheduled_deletion: DateTime.utc_now() |> DateTime.truncate(:second) + ) + + credential = existing_credential(owner, theirs) + + for project <- [insert(:project), winding_down], + do: + insert(:project_credential, credential: credential, project: project) + + response = + conn + |> with_pat(owner) + |> get(~p"/api/credentials/#{credential.id}") + |> json_response(200) + + assert linked_project_ids(response) == [theirs.id] + assert [%{"id" => theirs_id}] = response["credential"]["projects"] + assert theirs_id == theirs.id + + conn + |> with_pat(insert(:user)) + |> get(~p"/api/credentials/#{credential.id}") + |> assert_problem(404, :not_found, []) + end + + test "a person reads someone else's credential linked to a project they stand in", + %{conn: conn} do + viewer = insert(:user) + shared = insert(:project, project_users: [%{user: viewer, role: :viewer}]) + credential = existing_credential(insert(:user), shared) + + insert(:project_credential, + credential: credential, + project: insert(:project) + ) + + response = + conn + |> with_pat(viewer) + |> get(~p"/api/credentials/#{credential.id}") + |> json_response(200) + + assert response["credential"]["id"] == credential.id + assert linked_project_ids(response) == [shared.id] + refute Jason.encode!(response) =~ "password" + end + end + + test "GET costs the same number of queries however many projects it links", + %{conn: conn} do + owner = insert(:user) + + [first | rest] = + insert_list(3, :project, project_users: [%{user: owner, role: :owner}]) + + credential = existing_credential(owner, first) + conn = with_pat(conn, owner) + + show = fn -> + conn |> get(~p"/api/credentials/#{credential.id}") |> json_response(200) + end + + {_, one_link} = count_queries(show) + + for project <- rest, + do: insert(:project_credential, credential: credential, project: project) + + assert {_, ^one_link} = count_queries(show) + end +end diff --git a/test/lightning_web/controllers/api/credentials/refusals_test.exs b/test/lightning_web/controllers/api/credentials/refusals_test.exs new file mode 100644 index 00000000000..8048ace9fdf --- /dev/null +++ b/test/lightning_web/controllers/api/credentials/refusals_test.exs @@ -0,0 +1,833 @@ +defmodule LightningWeb.API.Credentials.RefusalsTest do + use LightningWeb.ConnCase, async: true + + import Lightning.Factories + import LightningWeb.API.ProblemHelpers + import LightningWeb.ApiCredentialsHelpers + import Lightning.ServiceAccountHelpers + import Lightning.AdaptorTestHelpers + import Lightning.RaceHelpers + import Ecto.Query + + alias Lightning.Credentials + alias Lightning.Credentials.Credential + alias Lightning.Repo + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + describe "a body it cannot parse" do + setup [:assign_bearer_for_api] + + test "answers 400 malformed for a body that is not a JSON object", %{ + conn: conn, + user: user + } do + for body <- ["Listed", 7] do + conn + |> put_req_header("content-type", "application/json") + |> post(~p"/api/credentials", Jason.encode!(body)) + |> assert_problem(400, :malformed, []) + end + + assert Lightning.Credentials.list_credentials(user) == [] + end + end + + describe "a body its keys refuse" do + setup [:assign_bearer_for_api] + + test "refuses a blank name, and a body without a schema, at the key", %{ + conn: conn, + user: user + } do + required = fn pointer -> + %{ + "pointer" => pointer, + "code" => "required", + "detail" => "can't be blank" + } + end + + for {attrs, entries} <- [ + {credential_body(%{"name" => ""}), [required.("#/name")]}, + {%{"name" => ""}, [required.("#/name"), required.("#/schema")]} + ] do + conn + |> post(~p"/api/credentials", attrs) + |> assert_problem(422, :invalid, entries) + end + + assert Credentials.list_credentials(user) == [] + end + end + + describe "a link it cannot make" do + test "refuses a link it can't make, at the link, on POST and PUT", %{ + conn: conn + } do + user = insert(:user) + unseen = insert(:project) + + member = + insert(:project, project_users: [%{user: user, role: :editor}]) + + not_found = fn position -> + %{ + "pointer" => "#/project_credentials/#{position}/project_id", + "code" => "not_found", + "detail" => "names nothing you can see" + } + end + + not_valid = fn pointer, code, detail -> + %{"pointer" => pointer, "code" => code, "detail" => detail} + end + + refused = [ + {[%{"project_id" => unseen.id}], [not_found.(0)]}, + {[%{"project_id" => Ecto.UUID.generate()}], [not_found.(0)]}, + {[%{"project_id" => "not-a-uuid"}], [not_found.(0)]}, + {[%{"project_id" => 42}], + [ + not_valid.( + "#/project_credentials/0/project_id", + "type", + "is invalid" + ) + ]}, + {[%{}], + [ + not_valid.( + "#/project_credentials/0/project_id", + "required", + "can't be blank" + ) + ]}, + {["acme"], + [not_valid.("#/project_credentials/0", "type", "is invalid")]}, + {%{"project_id" => Ecto.UUID.generate()}, + [not_valid.("#/project_credentials", "type", "is invalid")]}, + {[ + %{"project_id" => unseen.id}, + %{"project_id" => member.id}, + %{"project_id" => unseen.id} + ], [not_found.(0), not_found.(2)]} + ] + + conn = with_pat(conn, user) + + for {link_list, entries} <- refused do + body = credential_body(%{"project_credentials" => link_list}) + + conn + |> post(~p"/api/credentials", body) + |> assert_problem(422, :invalid, entries) + + conn + |> put_credential(Ecto.UUID.generate(), body) + |> assert_problem(422, :invalid, entries) + end + + assert Credentials.list_credentials(user) == [] + end + + test "answers no_access in the words for whoever is asking", %{conn: conn} do + person = insert(:user) + viewing = insert(:project, project_users: [%{user: person, role: :viewer}]) + elsewhere = insert(:project) + + {account_conn, _account} = + with_service_account(conn, ["credentials:write"]) + + requests = [ + {fn -> + conn + |> with_pat(person) + |> post( + ~p"/api/credentials", + credential_body(%{"project_credentials" => links([viewing.id])}) + ) + end, "you may not link a credential to this project"}, + {fn -> + conn + |> with_pat(person) + |> put_credential( + Ecto.UUID.generate(), + credential_body(%{"project_credentials" => links([viewing.id])}) + ) + end, "you may not link a credential to this project"}, + {fn -> + put_credential( + account_conn, + Ecto.UUID.generate(), + credential_body(%{ + "owner" => insert(:user).email, + "project_credentials" => links([elsewhere.id]) + }) + ) + end, "the credential's owner is not a member of this project"} + ] + + for {request, detail} <- requests do + assert_problem(request.(), 422, :invalid, [ + %{ + "pointer" => "#/project_credentials/0/project_id", + "code" => "no_access", + "detail" => detail + } + ]) + end + + assert Repo.aggregate(Credential, :count) == 0 + end + end + + describe "a name already held" do + test "answers 409 taken for a name the caller already holds, on POST and PUT", + %{conn: conn} do + user = insert(:user) + held = insert(:credential, user: user, name: "Taken") + renamed = insert(:credential, user: user, name: "Mine") + new_id = Ecto.UUID.generate() + conn = with_pat(conn, user) + + taken = [ + %{ + "pointer" => "#/name", + "code" => "taken", + "detail" => "you have another credential with the same name" + } + ] + + conn + |> post(~p"/api/credentials", credential_body(%{"name" => "Taken"})) + |> assert_problem(409, :taken, taken) + + conn + |> put_credential(new_id, credential_body(%{"name" => "Taken"})) + |> assert_problem(409, :taken, taken) + + refute Repo.get(Credential, new_id) + + conn + |> put_credential(renamed.id, credential_body(%{"name" => held.name})) + |> assert_problem(409, :taken, taken) + + assert Repo.reload!(renamed).name == "Mine" + end + end + + describe "PUT, for a service account with credentials:write" do + setup %{conn: conn} do + {conn, account} = with_service_account(conn, ["credentials:write"]) + %{conn: conn, account: account} + end + + test "refuses a body it cannot take with 422, at each key it refuses", %{ + conn: conn + } do + owner = insert(:user) + project_id = insert(:project).id + main = %{"name" => "main", "body" => %{"a" => "b"}} + + entry = fn pointer, code, detail -> + %{"pointer" => pointer, "code" => code, "detail" => detail} + end + + one_body = + entry.( + "#/credential_bodies", + "one_body", + "must be one body named main, holding an object" + ) + + refused = [ + unknown_key: + {%{"external_id" => "x"}, + [entry.("#/external_id", "unknown_key", "is not accepted")]}, + no_bodies: {%{"credential_bodies" => []}, [one_body]}, + missing_bodies: {%{"credential_bodies" => nil}, [one_body]}, + two_bodies: + {%{"credential_bodies" => [main, %{main | "name" => "staging"}]}, + [one_body]}, + not_main: + {%{"credential_bodies" => [%{main | "name" => "staging"}]}, [one_body]}, + body_not_object: + {%{"credential_bodies" => [%{main | "body" => "text"}]}, [one_body]}, + bodies_not_list: {%{"credential_bodies" => main}, [one_body]}, + link_without_project: + {%{"project_credentials" => [%{"id" => project_id}]}, + [ + entry.( + "#/project_credentials/0/project_id", + "required", + "can't be blank" + ), + entry.( + "#/project_credentials/0/id", + "unknown_key", + "is not accepted" + ) + ]}, + link_not_uuid: + {%{"project_credentials" => [%{"project_id" => "acme"}]}, + [ + entry.( + "#/project_credentials/0/project_id", + "not_found", + "names nothing you can see" + ) + ]}, + links_not_list: + {%{"project_credentials" => %{"project_id" => project_id}}, + [entry.("#/project_credentials", "type", "is invalid")]}, + unknown_owner: + {%{"owner" => "nobody@example.com"}, + [entry.("#/owner", "not_found", "names nothing you can see")]}, + missing_owner: + {%{"owner" => nil}, [entry.("#/owner", "required", "can't be blank")]}, + other_id: + {%{"id" => Ecto.UUID.generate()}, + [ + entry.("#/id", "mismatch", "does not match the id in the path") + ]}, + blank_name: + {%{"name" => ""}, [entry.("#/name", "required", "can't be blank")]}, + missing_schema: + {%{"schema" => nil}, + [entry.("#/schema", "required", "can't be blank")]} + ] + + for {case_name, {overrides, entries}} <- refused do + id = Ecto.UUID.generate() + + body = + %{"owner" => owner.email} + |> credential_body() + |> Map.merge(overrides) + |> Map.reject(fn {_key, value} -> is_nil(value) end) + + conn + |> put_credential(id, body) + |> assert_problem(422, :invalid, entries) + + refute Repo.get(Credential, id), "#{case_name} created a credential" + end + end + + test "reports every refused key, link and owner in one answer, links by their position in the request", + %{conn: conn} do + owner = insert(:user) + held = insert(:project, project_users: [%{user: owner, role: :owner}]) + member = insert(:project, project_users: [%{user: owner, role: :editor}]) + elsewhere = insert(:project) + + credential = + insert(:credential, + name: "Acme DHIS2", + schema: "raw", + user: owner, + project_credentials: [%{project_id: held.id}] + ) + + body = + credential_body(%{ + "owner" => owner.email, + "colour" => "blue", + "credential_bodies" => [], + "project_credentials" => + links([ + member.id, + elsewhere.id, + held.id, + Ecto.UUID.generate() + ]) + }) + + conn + |> put_credential(credential.id, body) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/colour", + "code" => "unknown_key", + "detail" => "is not accepted" + }, + %{ + "pointer" => "#/credential_bodies", + "code" => "one_body", + "detail" => "must be one body named main, holding an object" + }, + %{ + "pointer" => "#/project_credentials/1/project_id", + "code" => "no_access", + "detail" => "the credential's owner is not a member of this project" + }, + %{ + "pointer" => "#/project_credentials/3/project_id", + "code" => "not_found", + "detail" => "names nothing you can see" + } + ]) + + assert Repo.all( + from pc in Lightning.Projects.ProjectCredential, + where: pc.credential_id == ^credential.id, + select: pc.project_id + ) == [held.id] + end + + test "a credential another request deletes mid-replace answers 409 concurrent_edit", + %{conn: conn} do + owner = insert(:user) + credential = existing_credential(owner, insert(:project)) + + after_first_query("credentials", credential.id, fn id -> + for child <- ~w(credential_bodies project_credentials) do + Repo.query!("DELETE FROM #{child} WHERE credential_id = $1", [id]) + end + + Repo.query!("DELETE FROM credentials WHERE id = $1", [id]) + end) + + conn + |> put_credential( + credential.id, + credential_body(%{"name" => "Renamed", "owner" => owner.email}) + ) + |> assert_problem(409, :concurrent_edit, []) + + refute Repo.reload(credential) + end + + test "will not rewrite another owner's credential through a new owner", %{ + conn: conn + } do + owner = insert(:user) + credential = existing_credential(owner, insert(:project)) + other = insert(:user) + + conn = + put_credential( + conn, + credential.id, + credential_body(%{"name" => "Hijacked", "owner" => other.email}) + ) + + assert_problem(conn, 409, :id_taken, []) + + assert %Credential{name: "Acme DHIS2", user_id: owner_id} = + Repo.reload!(credential) + + assert owner_id == owner.id + assert main_body(credential.id) == %{"password" => "old"} + end + + test "answers scheduled_for_deletion and writes nothing to a credential being deleted", + %{conn: conn} do + owner = insert(:user) + [earlier, requested] = insert_pair(:project) + credential = scheduled_for_deletion(existing_credential(owner, earlier)) + audits_before = audit_ids(credential.id) + + conn = + put_credential( + conn, + credential.id, + credential_body(%{ + "name" => "Renamed", + "owner" => owner.email, + "project_credentials" => links([requested.id]) + }) + ) + + assert_problem(conn, 409, :scheduled_for_deletion, []) + assert Repo.reload!(credential).name == "Acme DHIS2" + assert main_body(credential.id) == %{"password" => "old"} + + assert Repo.all( + from pc in Lightning.Projects.ProjectCredential, + where: pc.credential_id == ^credential.id, + select: pc.project_id + ) == [earlier.id] + + assert audit_ids(credential.id) == audits_before + end + end + + describe "PUT, for a person's token" do + test "cannot rewrite a credential someone else owns", %{conn: conn} do + credential = insert(:credential, user: insert(:user), name: "Theirs") + + conn = + conn + |> with_pat(insert(:user)) + |> put_credential(credential.id, credential_body(%{})) + + assert_problem(conn, 409, :id_taken, []) + assert Repo.reload!(credential).name == "Theirs" + end + + test "is refused with 409 on their own credential while it is being deleted", + %{conn: conn} do + user = insert(:user) + + credential = + scheduled_for_deletion(existing_credential(user, insert(:project))) + + conn = + conn + |> with_pat(user) + |> put_credential(credential.id, credential_body(%{"name" => "Renamed"})) + + assert_problem(conn, 409, :scheduled_for_deletion, []) + assert Repo.reload!(credential).name == credential.name + end + + test "learns only that the id is taken, not that someone else's credential is being deleted", + %{conn: conn} do + credential = + scheduled_for_deletion( + existing_credential(insert(:user), insert(:project)) + ) + + conn = + conn + |> with_pat(insert(:user)) + |> put_credential(credential.id, credential_body(%{})) + + assert_problem(conn, 409, :id_taken, []) + end + end + + describe "a body its schema refuses, on POST and PUT" do + setup [:assign_bearer_for_api] + + defp oauth_token(overrides) do + Map.merge( + %{ + "access_token" => "access", + "refresh_token" => "refresh", + "token_type" => "Bearer", + "scope" => "read write", + "expires_in" => 3600 + }, + overrides + ) + end + + test "refuses an OAuth token at the key it refuses, on POST and PUT", %{ + conn: conn, + user: user + } do + conn + |> post(~p"/api/credentials", %{ + "name" => "OAuth", + "schema" => "oauth", + "credential_bodies" => [ + %{"name" => "main", "body" => oauth_token(%{})}, + %{ + "name" => "staging", + "body" => oauth_token(%{"token_type" => "Basic"}) + } + ] + }) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/credential_bodies/1/body/token_type", + "code" => "not_allowed", + "detail" => "Unsupported token type. Expected 'Bearer'" + } + ]) + + for {body, key, code, detail} <- [ + {oauth_token(%{"token_type" => "Basic"}), "token_type", + "not_allowed", "Unsupported token type. Expected 'Bearer'"}, + {Map.delete(oauth_token(%{}), "access_token"), "access_token", + "required", "Missing required OAuth field: access_token"} + ] do + attrs = %{ + "name" => "OAuth", + "schema" => "oauth", + "credential_bodies" => [%{"name" => "main", "body" => body}] + } + + entries = [ + %{ + "pointer" => "#/credential_bodies/0/body/#{key}", + "code" => code, + "detail" => detail + } + ] + + conn + |> post(~p"/api/credentials", attrs) + |> assert_problem(422, :invalid, entries) + + conn + |> put_credential(Ecto.UUID.generate(), attrs) + |> assert_problem(422, :invalid, entries) + end + + assert Credentials.list_credentials(user) == [] + end + end + + describe "PUT, a credential body that is refused" do + setup :isolated_adaptors + + setup %{conn: conn} do + seed_credential_schema("postgresql") + {conn, _account} = with_service_account(conn, ["credentials:write"]) + %{conn: conn, owner: insert(:user)} + end + + defp typed_body(owner, body) do + credential_body(%{ + "owner" => owner.email, + "schema" => "postgresql", + "credential_bodies" => [%{"name" => "main", "body" => body}] + }) + end + + test "is refused at each key its schema refuses", %{conn: conn, owner: owner} do + id = Ecto.UUID.generate() + + conn + |> put_credential( + id, + typed_body(owner, %{"host" => "acme", "port" => 5432}) + ) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/credential_bodies/0/body/host", + "code" => "type", + "detail" => "expected to be a URI or an IPv4 address" + }, + %{ + "pointer" => "#/credential_bodies/0/body/database", + "code" => "required", + "detail" => "can't be blank" + } + ]) + + refute Repo.get(Credential, id) + end + + test "is refused at each key on a credential it replaces", %{ + conn: conn, + owner: owner + } do + credential = existing_credential(owner, insert(:project)) + + conn + |> put_credential( + credential.id, + typed_body(owner, %{ + "host" => "https://db.example", + "port" => "many", + "database" => "acme" + }) + ) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/credential_bodies/0/body/port", + "code" => "type", + "detail" => "is invalid" + }, + # The failed cast leaves no port, so the required check reports it too. + %{ + "pointer" => "#/credential_bodies/0/body/port", + "code" => "required", + "detail" => "can't be blank" + } + ]) + + assert main_body(credential.id) == %{"password" => "old"} + end + + test "answers 503 not_ready while the adaptor catalogue can't answer", %{ + conn: conn, + owner: owner + } do + Mimic.stub(Lightning.Adaptors, :resolve_name, fn name -> {:ok, name} end) + + Mimic.stub(Lightning.Adaptors, :schema, fn _name -> + {:error, :not_ready} + end) + + id = Ecto.UUID.generate() + + conn = + put_credential( + conn, + id, + typed_body(owner, %{"host" => "https://db.example"}) + ) + + assert_problem(conn, 503, :not_ready, []) + assert get_resp_header(conn, "retry-after") == ["5"] + refute Repo.get(Credential, id) + end + end + + describe "DELETE, refused" do + setup [:assign_bearer_for_api] + + test "answers 404 for a credential that doesn't exist, or that the caller can't see", + %{conn: conn} do + other_credential = + insert(:credential, user: insert(:user), schema: "raw") + + for id <- [Ecto.UUID.generate(), other_credential.id] do + conn + |> delete(~p"/api/credentials/#{id}") + |> assert_problem(404, :not_found, []) + end + + assert Lightning.Credentials.get_credential(other_credential.id) + end + + test "answers 403 for another user's credential the caller sees through a project", + %{conn: conn, user: user} do + project = + insert(:project, project_users: [%{user_id: user.id, role: :owner}]) + + other_credential = + insert(:credential, + user: insert(:user), + schema: "raw", + project_credentials: [%{project_id: project.id}] + ) + + conn + |> delete(~p"/api/credentials/#{other_credential.id}") + |> assert_problem(403, :forbidden, []) + + assert Lightning.Credentials.get_credential(other_credential.id) + end + + test "answers 409 in_use for a credential runs have used", %{ + conn: conn, + user: user + } do + credential = insert(:credential, user: user, schema: "raw") + insert(:step, credential: credential) + + conn + |> delete(~p"/api/credentials/#{credential.id}") + |> assert_problem(409, :in_use, []) + + assert Lightning.Credentials.get_credential(credential.id) + end + end + + describe "error answers on every credentials route" do + setup [:assign_bearer_for_api] + + test "never quotes a value planted in the credential's own keys, on POST and PUT", + %{conn: conn, user: user} do + planted = + plant_sentinel(%{ + "id" => nil, + "name" => nil, + "schema" => nil, + "owner" => nil, + "colour" => nil, + "credential_bodies" => [%{"name" => nil, "body" => %{"key" => nil}}], + "project_credentials" => [%{"project_id" => nil, "extra" => nil}] + }) + + conn + |> post(~p"/api/credentials", planted) + |> refute_sentinel() + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/project_credentials/0/project_id", + "code" => "not_found", + "detail" => "names nothing you can see" + } + ]) + + conn + |> post(~p"/api/credentials", %{ + "name" => "OAuth", + "schema" => "oauth", + "credential_bodies" => + plant_sentinel([ + %{ + "name" => "main", + "body" => %{ + "access_token" => [nil], + "refresh_token" => nil, + "token_type" => nil, + "scope" => nil, + "expires_in" => nil + } + } + ]) + }) + |> refute_sentinel() + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/credential_bodies/0/body/access_token", + "code" => "type", + "detail" => "access_token must be a non-empty string" + } + ]) + + {account_conn, _account} = + with_service_account(conn, ["credentials:write"]) + + base = [ + %{ + "pointer" => "#/colour", + "code" => "unknown_key", + "detail" => "is not accepted" + }, + %{ + "pointer" => "#/id", + "code" => "mismatch", + "detail" => "does not match the id in the path" + }, + %{ + "pointer" => "#/credential_bodies", + "code" => "one_body", + "detail" => "must be one body named main, holding an object" + } + ] + + rest = [ + %{ + "pointer" => "#/project_credentials/0/project_id", + "code" => "not_found", + "detail" => "names nothing you can see" + }, + %{ + "pointer" => "#/project_credentials/0/extra", + "code" => "unknown_key", + "detail" => "is not accepted" + } + ] + + for {owner, entries} <- [ + {planted["owner"], + base ++ + [ + %{ + "pointer" => "#/owner", + "code" => "not_found", + "detail" => "names nothing you can see" + } + ] ++ rest}, + {user.email, base ++ rest} + ] do + account_conn + |> put_credential(Ecto.UUID.generate(), Map.put(planted, "owner", owner)) + |> refute_sentinel() + |> assert_problem(422, :invalid, entries) + end + end + end +end diff --git a/test/lightning_web/controllers/api/credentials/writing_test.exs b/test/lightning_web/controllers/api/credentials/writing_test.exs new file mode 100644 index 00000000000..fb2b7d65fae --- /dev/null +++ b/test/lightning_web/controllers/api/credentials/writing_test.exs @@ -0,0 +1,498 @@ +defmodule LightningWeb.API.Credentials.WritingTest do + use LightningWeb.ConnCase, async: true + + import Lightning.Factories + import LightningWeb.API.ProblemHelpers + import LightningWeb.ApiCredentialsHelpers + import Lightning.ServiceAccountHelpers + + alias Lightning.Credentials + alias Lightning.Credentials.Credential + alias Lightning.Repo + + @secret LightningWeb.ApiCredentialsHelpers.secret() + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + describe "creating with POST" do + setup [:assign_bearer_for_api] + + test "creates a basic credential without project associations", %{ + conn: conn, + user: user + } do + credential_attrs = %{ + "name" => "Test Credential", + "schema" => "raw", + "credential_bodies" => [ + %{ + "name" => "main", + "body" => %{"username" => "test", "password" => "secret"} + } + ] + } + + conn = post(conn, ~p"/api/credentials", credential_attrs) + response = json_response(conn, 201) + + assert %{ + "credential" => %{ + "id" => id, + "name" => "Test Credential", + "schema" => "raw", + "external_id" => nil, + "user_id" => user_id, + "project_credentials" => [], + "projects" => [] + }, + "errors" => %{} + } = response + + assert is_binary(id) + assert user_id == user.id + refute Map.has_key?(response["credential"], "body") + refute Map.has_key?(response["credential"], "production") + end + + test "creates a credential with multiple project associations", %{ + conn: conn, + user: user + } do + project1 = + insert(:project, project_users: [%{user_id: user.id, role: :admin}]) + + project2 = + insert(:project, project_users: [%{user_id: user.id, role: :owner}]) + + credential_attrs = %{ + "name" => "Multi-Project Credential", + "schema" => "raw", + "credential_bodies" => [ + %{ + "name" => "main", + "body" => %{"token" => "abc123"} + } + ], + "project_credentials" => [ + %{"project_id" => project1.id}, + %{"project_id" => project2.id} + ] + } + + conn = post(conn, ~p"/api/credentials", credential_attrs) + response = json_response(conn, 201) + + assert %{ + "credential" => %{ + "project_credentials" => project_credentials, + "projects" => projects + } + } = response + + assert length(project_credentials) == 2 + assert length(projects) == 2 + + project_ids = Enum.map(projects, & &1["id"]) + assert project1.id in project_ids + assert project2.id in project_ids + end + + test "links a project named twice once", %{conn: conn, user: user} do + project = + insert(:project, project_users: [%{user_id: user.id, role: :editor}]) + + credential_attrs = %{ + "name" => "Twice Linked", + "schema" => "raw", + "credential_bodies" => [%{"name" => "main", "body" => %{"a" => "b"}}], + "project_credentials" => [ + %{"project_id" => project.id}, + %{"project_id" => project.id} + ] + } + + assert %{"credential" => %{"project_credentials" => [link]}} = + conn + |> post(~p"/api/credentials", credential_attrs) + |> json_response(201) + + assert link["project_id"] == project.id + end + + test "creates a credential with multiple environment bodies", %{ + conn: conn, + user: user + } do + credential_attrs = %{ + "name" => "Multi-Environment Credential", + "schema" => "raw", + "credential_bodies" => [ + %{ + "name" => "main", + "body" => %{"api_key" => "main_key"} + }, + %{ + "name" => "production", + "body" => %{"api_key" => "prod_key"} + }, + %{ + "name" => "staging", + "body" => %{"api_key" => "staging_key"} + } + ] + } + + conn = post(conn, ~p"/api/credentials", credential_attrs) + response = json_response(conn, 201) + + assert %{ + "credential" => %{ + "id" => id, + "name" => "Multi-Environment Credential", + "user_id" => user_id + } + } = response + + assert is_binary(id) + assert user_id == user.id + + # Verify bodies were created + credential = Lightning.Credentials.get_credential(id) + credential = Lightning.Repo.preload(credential, :credential_bodies) + assert length(credential.credential_bodies) == 3 + + env_names = + Enum.map(credential.credential_bodies, & &1.name) |> Enum.sort() + + assert env_names == ["main", "production", "staging"] + end + + test "allows access when user is support user with project access", %{ + conn: conn + } do + support_user = insert(:user, support_user: true) + project = insert(:project, allow_support_access: true) + + token = Lightning.Accounts.generate_api_token(support_user) + conn = conn |> Plug.Conn.put_req_header("authorization", "Bearer #{token}") + + credential_attrs = %{ + "name" => "Support User Credential", + "schema" => "raw", + "credential_bodies" => [ + %{ + "name" => "main", + "body" => %{"secret" => "value"} + } + ], + "project_credentials" => [ + %{"project_id" => project.id} + ] + } + + conn = post(conn, ~p"/api/credentials", credential_attrs) + response = json_response(conn, 201) + + assert %{ + "credential" => %{ + "name" => "Support User Credential", + "user_id" => user_id + } + } = response + + assert user_id == support_user.id + end + + test "cannot override user_id in request", %{conn: conn, user: user} do + other_user = insert(:user) + + credential_attrs = %{ + "name" => "Test Credential", + "schema" => "raw", + "credential_bodies" => [ + %{ + "name" => "main", + "body" => %{"secret" => "value"} + } + ], + # This should be ignored + "user_id" => other_user.id + } + + conn = post(conn, ~p"/api/credentials", credential_attrs) + response = json_response(conn, 201) + + # Should use the authenticated user, not the provided user_id + assert response["credential"]["user_id"] == user.id + refute response["credential"]["user_id"] == other_user.id + end + + test "links the project, whatever the caller's role in it", %{ + conn: conn, + user: user + } do + for role <- [:editor, :admin, :owner] do + project = + insert(:project, project_users: [%{user_id: user.id, role: role}]) + + assert %{ + "credential" => %{ + "user_id" => user_id, + "projects" => [%{"id" => project_id, "name" => project_name}], + "project_credentials" => [%{"project_id" => linked_id}] + } + } = + conn + |> post( + ~p"/api/credentials", + credential_body(%{ + "name" => "#{role} Credential", + "project_credentials" => links([project.id]) + }) + ) + |> json_response(201) + + assert {user_id, project_id, project_name, linked_id} == + {user.id, project.id, project.name, project.id} + end + end + end + + describe "PUT, for a service account with credentials:write" do + setup %{conn: conn} do + {conn, account} = with_service_account(conn, ["credentials:write"]) + %{conn: conn, account: account} + end + + test "creates the credential under the path id for the owner it names", %{ + conn: conn + } do + owner = insert(:user, email: "alice@example.com") + + [project_a, project_b] = + insert_pair(:project, project_users: [%{user: owner}]) + + id = Ecto.UUID.generate() + + conn = + put_credential( + conn, + id, + credential_body(%{ + "owner" => "ALICE@example.com", + "project_credentials" => links([project_a.id, project_b.id]) + }) + ) + + response = json_response(conn, 201) + + # Without projects:write the account sees the links, but no project + # names. + assert %{ + "credential" => %{ + "id" => ^id, + "name" => "Acme DHIS2", + "schema" => "raw", + "user_id" => user_id, + "projects" => [] + }, + "errors" => %{} + } = response + + assert user_id == owner.id + + assert linked_project_ids(response) == + Enum.sort([project_a.id, project_b.id]) + + refute conn.resp_body =~ @secret + assert main_body(id) == %{"password" => @secret} + end + + test "replaces name, schema and main body; keeps other bodies and every link", + %{conn: conn} do + owner = insert(:user) + earlier = insert(:project) + requested = insert(:project, project_users: [%{user: owner}]) + credential = existing_credential(owner, earlier) + + conn = + put_credential( + conn, + credential.id, + credential_body(%{ + "name" => "Renamed", + "owner" => owner.email, + "credential_bodies" => [ + %{"name" => "main", "body" => %{"password" => @secret}} + ], + "project_credentials" => links([requested.id]) + }) + ) + + response = json_response(conn, 200) + + assert response["credential"]["name"] == "Renamed" + + assert linked_project_ids(response) == + Enum.sort([earlier.id, requested.id]) + + [earlier_link] = + Enum.filter( + response["credential"]["project_credentials"], + &(&1["project_id"] == earlier.id) + ) + + assert earlier_link["id"] == + Repo.get_by!(Lightning.Projects.ProjectCredential, + project_id: earlier.id, + credential_id: credential.id + ).id + + refute conn.resp_body =~ @secret + assert main_body(credential.id) == %{"password" => @secret} + + assert Credentials.get_credential_body(credential.id, "staging").body == + %{"password" => "staging"} + end + + test "re-sending what is stored answers 200 and changes nothing", %{ + conn: conn + } do + owner = insert(:user) + project = insert(:project, project_users: [%{user: owner}]) + id = Ecto.UUID.generate() + + body = + credential_body(%{ + "owner" => owner.email, + "project_credentials" => links([project.id]) + }) + + first = conn |> put_credential(id, body) |> json_response(201) + second = conn |> put_credential(id, body) |> json_response(200) + + assert second["credential"]["project_credentials"] == + first["credential"]["project_credentials"] + end + + test "links a support user's credential to a project allowing support access", + %{conn: conn} do + owner = insert(:user, support_user: true) + project = insert(:project, allow_support_access: true) + id = Ecto.UUID.generate() + + conn = + put_credential( + conn, + id, + credential_body(%{ + "owner" => owner.email, + "project_credentials" => links([project.id]) + }) + ) + + assert linked_project_ids(json_response(conn, 201)) == [project.id] + end + end + + describe "PUT, for a person's token" do + test "creates a credential they own without naming an owner", %{conn: conn} do + user = insert(:user) + project = insert(:project, project_users: [%{user: user, role: :editor}]) + id = Ecto.UUID.generate() + + response = + conn + |> with_pat(user) + |> put_credential( + id, + credential_body(%{"project_credentials" => links([project.id])}) + ) + |> json_response(201) + + assert response["credential"]["user_id"] == user.id + assert linked_project_ids(response) == [project.id] + end + + test "may name themselves as owner, but no one else", %{conn: conn} do + user = insert(:user) + other = insert(:user) + conn = with_pat(conn, user) + + assert conn + |> put_credential( + Ecto.UUID.generate(), + credential_body(%{"owner" => String.upcase(user.email)}) + ) + |> json_response(201) + + id = Ecto.UUID.generate() + + conn + |> put_credential(id, credential_body(%{"owner" => other.email})) + |> assert_problem(403, :forbidden, []) + + conn + |> put_credential( + id, + credential_body(%{"owner" => "nobody@example.com"}) + ) + |> assert_problem(403, :forbidden, []) + + refute Repo.get(Credential, id) + end + + test "keeps a link they can't see, and leaves it out of the response", + %{conn: conn} do + owner = insert(:user) + theirs = insert(:project, project_users: [%{user: owner, role: :owner}]) + unseen = insert(:project) + credential = existing_credential(owner, unseen) + + response = + conn + |> with_pat(owner) + |> put_credential( + credential.id, + credential_body(%{"project_credentials" => links([theirs.id])}) + ) + |> json_response(200) + + assert linked_project_ids(response) == [theirs.id] + refute inspect(response) =~ unseen.id + + assert Repo.get_by(Lightning.Projects.ProjectCredential, + project_id: unseen.id, + credential_id: credential.id + ) + end + end + + describe "deleting what the caller owns" do + setup [:assign_bearer_for_api] + + test "deletes a credential the caller owns, with its project links", %{ + conn: conn, + user: user + } do + project = + insert(:project, project_users: [%{user_id: user.id, role: :owner}]) + + credential = + insert(:credential, + user: user, + name: "Credential with Projects", + schema: "raw", + project_credentials: [%{project_id: project.id}] + ) + + conn = delete(conn, ~p"/api/credentials/#{credential.id}") + assert response(conn, 204) == "" + + # Verify credential and associations are deleted + refute Lightning.Credentials.get_credential(credential.id) + end + end +end diff --git a/test/lightning_web/controllers/api/helpers_test.exs b/test/lightning_web/controllers/api/helpers_test.exs new file mode 100644 index 00000000000..e789b44b625 --- /dev/null +++ b/test/lightning_web/controllers/api/helpers_test.exs @@ -0,0 +1,72 @@ +defmodule LightningWeb.API.HelpersTest do + use ExUnit.Case, async: true + + alias LightningWeb.API.Helpers + + setup do + conn = + Plug.Test.conn(:get, "/api/runs") + |> Plug.Conn.put_private(:phoenix_endpoint, LightningWeb.Endpoint) + + %{conn: conn} + end + + @url "#{LightningWeb.Endpoint.url()}/api/runs" + + describe "pagination_links/3" do + test "the first of several pages has no previous page", %{conn: conn} do + assert links(conn, 1, 3) == %{ + self: "#{@url}?page=1&page_size=10", + first: "#{@url}?page=1&page_size=10", + last: "#{@url}?page=3&page_size=10", + next: "#{@url}?page=2&page_size=10", + prev: nil + } + end + + test "the last of several pages has no next page", %{conn: conn} do + assert links(conn, 3, 3) == %{ + self: "#{@url}?page=3&page_size=10", + first: "#{@url}?page=1&page_size=10", + last: "#{@url}?page=3&page_size=10", + next: nil, + prev: "#{@url}?page=2&page_size=10" + } + end + + test "a single page has neither", %{conn: conn} do + assert links(conn, 1, 1) == %{ + self: "#{@url}?page=1&page_size=10", + first: "#{@url}?page=1&page_size=10", + last: "#{@url}?page=1&page_size=10", + next: nil, + prev: nil + } + end + + test "carries the filters, as cast, except one the path already names", %{ + conn: conn + } do + conn = %{conn | path_params: %{"project_id" => "p1"}} + + filters = %{ + project_id: "p1", + state: ["failed", "crashed"], + inserted_after: ~U[2024-01-01 00:00:00.000000Z] + } + + assert Helpers.pagination_links(conn, page(1, 1), filters).self == + "#{@url}?inserted_after=2024-01-01T00%3A00%3A00.000000Z&page=1&page_size=10&state=failed%2Ccrashed" + end + end + + defp links(conn, number, total_pages), + do: Helpers.pagination_links(conn, page(number, total_pages), %{}) + + defp page(number, total_pages), + do: %Scrivener.Page{ + page_number: number, + page_size: 10, + total_pages: total_pages + } +end diff --git a/test/lightning_web/controllers/api/job_controller_test.exs b/test/lightning_web/controllers/api/job_controller_test.exs index 82de783537b..a760bc1f49a 100644 --- a/test/lightning_web/controllers/api/job_controller_test.exs +++ b/test/lightning_web/controllers/api/job_controller_test.exs @@ -1,98 +1,128 @@ defmodule LightningWeb.API.JobControllerTest do use LightningWeb.ConnCase, async: true + import LightningWeb.API.ProblemHelpers + import Lightning.Factories import Lightning.JobsFixtures - import Lightning.ProjectsFixtures setup %{conn: conn} do {:ok, conn: put_req_header(conn, "accept", "application/json")} end - test "without a token", %{conn: conn} do - conn = get(conn, ~p"/api/projects/#{Ecto.UUID.generate()}/jobs") - - assert %{"error" => "Unauthorized"} == json_response(conn, 401) - end - - describe "with invalid token" do - test "gets a 401", %{conn: conn} do - token = "Oooops" - conn = conn |> Plug.Conn.put_req_header("authorization", "Bearer #{token}") - conn = get(conn, ~p"/api/projects/#{Ecto.UUID.generate()}/jobs") - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - end - describe "index" do setup [:assign_bearer_for_api, :create_project_for_current_user, :create_job] - test "lists all jobs for project I belong to", %{conn: conn, job: job} do - conn = get(conn, ~p"/api/projects/#{job.workflow.project_id}/jobs") - - response = json_response(conn, 200) - - assert response["data"] == [ - %{ - "attributes" => %{"name" => "some name"}, - "id" => job.id, - "links" => %{ - "self" => "#{LightningWeb.Endpoint.url()}/api/jobs/#{job.id}" - }, - "relationships" => %{}, - "type" => "jobs" - } - ] + test "lists the project's jobs, on the nested route and the plain one", %{ + conn: conn, + job: job + } do + for path <- [ + ~p"/api/projects/#{job.workflow.project_id}/jobs", + ~p"/api/jobs" + ] do + assert conn |> get(path) |> json_response(200) |> Map.fetch!("data") == [ + %{ + "attributes" => %{"name" => "some name"}, + "id" => job.id, + "links" => %{ + "self" => + "#{LightningWeb.Endpoint.url()}/api/jobs/#{job.id}" + }, + "relationships" => %{}, + "type" => "jobs" + } + ] + end end - test "responds with a 401 when I don't have access", %{conn: conn} do - other_project = project_fixture() - - conn = get(conn, ~p"/api/projects/#{other_project.id}/jobs") - - response = json_response(conn, 401) + test "reads the number a page param starts with, and falls back to the default for one it can't read", + %{conn: conn, project: project} do + workflow = insert(:workflow, project: project) + insert_list(11, :job, workflow: workflow) + + for {query, entries, page} <- [ + {"page=5abc&page_size=2.0", 2, %{"page" => "5", "page_size" => "2"}}, + {"page=abc&page_size=xyz", 10, + %{"page" => "1", "page_size" => "10"}}, + {"page[]=2&page_size[]=2", 10, + %{"page" => "1", "page_size" => "10"}}, + {"page=-1&page_size=0", 10, %{"page" => "1", "page_size" => "10"}} + ] do + assert %{"data" => data, "links" => %{"self" => self}} = + conn |> get("/api/jobs?" <> query) |> json_response(200) + + assert {length(data), URI.decode_query(URI.parse(self).query)} == + {entries, page}, + query + end + end - assert response == %{"error" => "Unauthorized"} + test "pagination links carry the project filter applied and nothing else", + %{conn: conn, project: project} do + for {path, request_path, query} <- [ + {~p"/api/jobs?#{%{project_id: project.id, page_size: 1, foo: "bar"}}", + "/api/jobs", %{"project_id" => project.id}}, + {~p"/api/projects/#{project.id}/jobs?page_size=1&foo=bar", + "/api/projects/#{project.id}/jobs", %{}}, + {~p"/api/jobs?page_size=1&foo=bar", "/api/jobs", %{}} + ] do + %{"links" => links} = conn |> get(path) |> json_response(200) + + for key <- ["self", "first"] do + uri = URI.parse(links[key]) + assert uri.path == request_path + + assert URI.decode_query(uri.query) == + Map.merge(query, %{"page" => "1", "page_size" => "1"}) + end + end end + end + + describe "show" do + setup [:assign_bearer_for_api, :create_project_for_current_user, :create_job] - test "lists all jobs", %{conn: conn, job: job} do - conn = get(conn, ~p"/api/jobs") - response = json_response(conn, 200) + test "answers the job, and links to it without the request's query", %{ + conn: conn, + job: job + } do + url = "#{LightningWeb.Endpoint.url()}/api/jobs/#{job.id}" - assert response["data"] == [ + assert conn |> get(~p"/api/jobs/#{job}?foo=bar") |> json_response(200) == %{ - "attributes" => %{"name" => "some name"}, - "id" => job.id, - "links" => %{ - "self" => "#{LightningWeb.Endpoint.url()}/api/jobs/#{job.id}" + "data" => %{ + "attributes" => %{"name" => "some name"}, + "id" => job.id, + "links" => %{"self" => url}, + "relationships" => %{}, + "type" => "jobs" }, - "relationships" => %{}, - "type" => "jobs" + "included" => [], + "links" => %{"self" => url} } - ] end end - describe "show" do + describe "who may reach it" do setup [:assign_bearer_for_api, :create_project_for_current_user, :create_job] - test "returns 404 for non-existent job", %{conn: conn} do - conn = get(conn, ~p"/api/jobs/#{Ecto.UUID.generate()}") - assert json_response(conn, 404) - end - - test "shows the job", %{conn: conn, job: job} do - conn = get(conn, ~p"/api/jobs/#{job}") - response = json_response(conn, 200) - - assert response["data"] == %{ - "attributes" => %{"name" => "some name"}, - "id" => job.id, - "links" => %{ - "self" => "#{LightningWeb.Endpoint.url()}/api/jobs/#{job.id}" - }, - "relationships" => %{}, - "type" => "jobs" - } + test "answers 404 for what the caller has no standing on or that is missing, and 403 to a member who has not met the MFA requirement", + %{conn: conn, project: project, job: job} do + elsewhere = insert(:project) + %{job: unseen} = create_job(%{project: elsewhere}) + Lightning.Repo.update!(Ecto.Changeset.change(project, requires_mfa: true)) + + for {path, status, code} <- [ + {~p"/api/projects/#{elsewhere.id}/jobs", 404, :not_found}, + {~p"/api/projects/#{Ecto.UUID.generate()}/jobs", 404, :not_found}, + {~p"/api/jobs?#{%{project_id: elsewhere.id}}", 404, :not_found}, + {~p"/api/jobs/#{unseen.id}", 404, :not_found}, + {~p"/api/jobs/#{Ecto.UUID.generate()}", 404, :not_found}, + {~p"/api/jobs?#{%{project_id: project.id}}", 403, :forbidden}, + {~p"/api/jobs/#{job.id}", 403, :forbidden} + ] do + conn |> get(path) |> assert_problem(status, code, []) + end end end diff --git a/test/lightning_web/controllers/api/log_lines_controller_test.exs b/test/lightning_web/controllers/api/log_lines_controller_test.exs index fe2a329590f..9050c063d94 100644 --- a/test/lightning_web/controllers/api/log_lines_controller_test.exs +++ b/test/lightning_web/controllers/api/log_lines_controller_test.exs @@ -1,526 +1,177 @@ defmodule LightningWeb.API.LogLinesControllerTest do use LightningWeb.ConnCase, async: true + import LightningWeb.API.ProblemHelpers + import LightningWeb.ApiReadHelpers import Lightning.Factories setup %{conn: conn} do {:ok, conn: put_req_header(conn, "accept", "application/json")} end - test "without a token", %{conn: conn} do - conn = get(conn, ~p"/api/log_lines") - - assert %{"error" => "Unauthorized"} == json_response(conn, 401) + defp ids(conn) do + conn |> json_response(200) |> Map.fetch!("data") |> Enum.map(& &1["id"]) end - describe "with invalid token" do - test "gets a 401", %{conn: conn} do - token = "InvalidToken" - conn = conn |> Plug.Conn.put_req_header("authorization", "Bearer #{token}") - conn = get(conn, ~p"/api/log_lines") - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end + defp insert_log_line(run, attrs \\ []) do + insert( + :log_line, + [run: run, timestamp: ~U[2024-01-01 10:00:00.000000Z]] ++ attrs + ) end describe "index" do setup [:assign_bearer_for_api, :create_project_for_current_user] - test "lists log lines for runs in projects I have access to", %{ - conn: conn, - project: project - } do - # Create a workflow and run in my project - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) + test "lists the log lines of the projects the caller can reach, newest first", + %{conn: conn, project: project} do + run = insert_run(project) - run = List.first(workorder.runs) - - # Create log lines for this run - log1 = - insert(:log_line, - run: run, + older = + insert_log_line(run, message: "First log message", level: :info, - timestamp: DateTime.utc_now() + timestamp: ~U[2024-01-01 10:00:00.000000Z] ) - log2 = - insert(:log_line, - run: run, + newer = + insert_log_line(run, message: "Second log message", level: :error, - timestamp: DateTime.utc_now() |> DateTime.add(1, :second) - ) - - # Create a run in another project (should not be accessible) - other_project = insert(:project) - other_workflow = insert(:workflow, project: other_project) - other_trigger = insert(:trigger, workflow: other_workflow) - - other_workorder = - insert(:workorder, - workflow: other_workflow, - trigger: other_trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: other_trigger - } - ] - ) - - other_run = List.first(other_workorder.runs) - - other_log = - insert(:log_line, - run: other_run, - message: "Other project log", - level: :info, - timestamp: DateTime.utc_now() - ) - - conn = get(conn, ~p"/api/log_lines") - - response = json_response(conn, 200) - - # Should only return logs from my project - assert length(response["data"]) == 2 - - log_ids = Enum.map(response["data"], & &1["id"]) - assert log1.id in log_ids - assert log2.id in log_ids - refute other_log.id in log_ids - - # Verify pagination metadata - assert response["meta"]["total_entries"] == 2 - assert response["meta"]["page_number"] == 1 + timestamp: ~U[2024-01-01 11:00:00.000000Z] + ) + + insert_log_line(insert_run(insert(:project))) + + page = "#{LightningWeb.Endpoint.url()}/api/log_lines?page=1&page_size=10" + + assert conn |> get(~p"/api/log_lines") |> json_response(200) == %{ + "data" => [ + %{ + "type" => "log_lines", + "id" => newer.id, + "attributes" => %{ + "source" => nil, + "level" => "error", + "message" => "Second log message", + "timestamp" => "2024-01-01T11:00:00.000000Z", + "step_id" => nil, + "run_id" => run.id + } + }, + %{ + "type" => "log_lines", + "id" => older.id, + "attributes" => %{ + "source" => nil, + "level" => "info", + "message" => "First log message", + "timestamp" => "2024-01-01T10:00:00.000000Z", + "step_id" => nil, + "run_id" => run.id + } + } + ], + "included" => [], + "meta" => %{ + "total_entries" => 2, + "total_pages" => 1, + "page_number" => 1, + "page_size" => 10 + }, + "links" => %{ + "self" => page, + "first" => page, + "last" => page, + "next" => nil, + "prev" => nil + } + } end - test "filters log lines by timestamp_after", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - run = List.first(workorder.runs) + test "narrows by timestamp_after", %{conn: conn, project: project} do + run = insert_run(project) + insert_log_line(run, timestamp: ~U[2024-01-01 10:00:00.000000Z]) + new = insert_log_line(run, timestamp: ~U[2024-01-01 12:00:00.000000Z]) - old_time = ~U[2024-01-01 10:00:00Z] - new_time = ~U[2024-01-01 12:00:00Z] + conn = get(conn, ~p"/api/log_lines?timestamp_after=2024-01-01T11:00:00Z") - _old_log = - insert(:log_line, - run: run, - message: "Old log", - timestamp: old_time - ) - - new_log = - insert(:log_line, - run: run, - message: "New log", - timestamp: new_time - ) - - conn = - get( - conn, - ~p"/api/log_lines?timestamp_after=2024-01-01T11:00:00Z" - ) - - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == new_log.id + assert ids(conn) == [new.id] end - test "filters log lines by timestamp_before", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - run = List.first(workorder.runs) + test "narrows by timestamp_before", %{conn: conn, project: project} do + run = insert_run(project) + old = insert_log_line(run, timestamp: ~U[2024-01-01 10:00:00.000000Z]) + insert_log_line(run, timestamp: ~U[2024-01-01 12:00:00.000000Z]) - old_time = ~U[2024-01-01 10:00:00Z] - new_time = ~U[2024-01-01 12:00:00Z] + conn = get(conn, ~p"/api/log_lines?timestamp_before=2024-01-01T11:00:00Z") - old_log = - insert(:log_line, - run: run, - message: "Old log", - timestamp: old_time - ) - - _new_log = - insert(:log_line, - run: run, - message: "New log", - timestamp: new_time - ) - - conn = - get( - conn, - ~p"/api/log_lines?timestamp_before=2024-01-01T11:00:00Z" - ) - - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == old_log.id + assert ids(conn) == [old.id] end - test "filters log lines by level", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - run = List.first(workorder.runs) - - _info_log = - insert(:log_line, - run: run, - message: "Info message", - level: :info, - timestamp: DateTime.utc_now() - ) - - error_log = - insert(:log_line, - run: run, - message: "Error message", - level: :error, - timestamp: DateTime.utc_now() - ) - - conn = get(conn, ~p"/api/log_lines?level=error") - - response = json_response(conn, 200) + test "narrows by level", %{conn: conn, project: project} do + run = insert_run(project) + insert_log_line(run, level: :info) + error = insert_log_line(run, level: :error) - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == error_log.id - assert List.first(response["data"])["attributes"]["level"] == "error" + assert ids(get(conn, ~p"/api/log_lines?level=error")) == [error.id] end - test "filters log lines by run_id", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - # Create two runs - workorder1 = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - workorder2 = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - run1 = List.first(workorder1.runs) - run2 = List.first(workorder2.runs) + test "narrows by run_id", %{conn: conn, project: project} do + run1 = insert_run(project) + log1 = insert_log_line(run1) + insert_log_line(insert_run(project)) - log1 = insert(:log_line, run: run1, message: "Run 1 log") - _log2 = insert(:log_line, run: run2, message: "Run 2 log") - - conn = get(conn, ~p"/api/log_lines?run_id=#{run1.id}") - - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == log1.id + assert ids(get(conn, ~p"/api/log_lines?run_id=#{run1.id}")) == [log1.id] end - test "filters log lines by work_order_id", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - workorder1 = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - workorder2 = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) + test "narrows by work_order_id", %{conn: conn, project: project} do + run1 = insert_run(project) + log1 = insert_log_line(run1) + insert_log_line(insert_run(project)) - run1 = List.first(workorder1.runs) - run2 = List.first(workorder2.runs) + conn = get(conn, ~p"/api/log_lines?work_order_id=#{run1.work_order_id}") - log1 = insert(:log_line, run: run1, message: "WorkOrder 1 log") - _log2 = insert(:log_line, run: run2, message: "WorkOrder 2 log") - - conn = get(conn, ~p"/api/log_lines?work_order_id=#{workorder1.id}") - - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == log1.id + assert ids(conn) == [log1.id] end - test "filters log lines by workflow_id", %{ - conn: conn, - project: project - } do + test "narrows by workflow_id", %{conn: conn, project: project} do workflow1 = insert(:workflow, project: project) - workflow2 = insert(:workflow, project: project) - trigger1 = insert(:trigger, workflow: workflow1) - trigger2 = insert(:trigger, workflow: workflow2) - - workorder1 = - insert(:workorder, - workflow: workflow1, - trigger: trigger1, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger1 - } - ] - ) - - workorder2 = - insert(:workorder, - workflow: workflow2, - trigger: trigger2, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger2 - } - ] - ) - - run1 = List.first(workorder1.runs) - run2 = List.first(workorder2.runs) + log1 = workflow1 |> insert_run() |> insert_log_line() + insert_log_line(insert_run(project)) - log1 = insert(:log_line, run: run1, message: "Workflow 1 log") - _log2 = insert(:log_line, run: run2, message: "Workflow 2 log") - - conn = get(conn, ~p"/api/log_lines?workflow_id=#{workflow1.id}") - - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == log1.id + assert ids(get(conn, ~p"/api/log_lines?workflow_id=#{workflow1.id}")) == + [log1.id] end - test "filters log lines by project_id", %{ - conn: conn, - user: user - } do - # User has access to both projects + test "narrows by project_id", %{conn: conn, user: user} do project1 = insert(:project, project_users: [%{user: user, role: :owner}]) project2 = insert(:project, project_users: [%{user: user, role: :owner}]) + log1 = project1 |> insert_run() |> insert_log_line() + insert_log_line(insert_run(project2)) - workflow1 = insert(:workflow, project: project1) - workflow2 = insert(:workflow, project: project2) - trigger1 = insert(:trigger, workflow: workflow1) - trigger2 = insert(:trigger, workflow: workflow2) - - workorder1 = - insert(:workorder, - workflow: workflow1, - trigger: trigger1, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger1 - } - ] - ) - - workorder2 = - insert(:workorder, - workflow: workflow2, - trigger: trigger2, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger2 - } - ] - ) - - run1 = List.first(workorder1.runs) - run2 = List.first(workorder2.runs) - - log1 = insert(:log_line, run: run1, message: "Project 1 log") - _log2 = insert(:log_line, run: run2, message: "Project 2 log") - - conn = get(conn, ~p"/api/log_lines?project_id=#{project1.id}") - - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == log1.id + assert ids(get(conn, ~p"/api/log_lines?project_id=#{project1.id}")) == + [log1.id] end - test "combines multiple filters", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - run = List.first(workorder.runs) - - old_time = ~U[2024-01-01 10:00:00Z] - new_time = ~U[2024-01-01 12:00:00Z] + test "narrows by every filter at once", %{conn: conn, project: project} do + run = insert_run(project) - # Should match filters - _match_log = - insert(:log_line, - run: run, - message: "Match", + match = + insert_log_line(run, level: :error, - timestamp: new_time + timestamp: ~U[2024-01-01 12:00:00.000000Z] ) - # Wrong level - _wrong_level = - insert(:log_line, - run: run, - message: "Wrong level", - level: :info, - timestamp: new_time - ) + insert_log_line(run, + level: :info, + timestamp: ~U[2024-01-01 12:00:00.000000Z] + ) - # Wrong time - _wrong_time = - insert(:log_line, - run: run, - message: "Wrong time", - level: :error, - timestamp: old_time - ) + insert_log_line(run, + level: :error, + timestamp: ~U[2024-01-01 10:00:00.000000Z] + ) conn = get( @@ -528,237 +179,169 @@ defmodule LightningWeb.API.LogLinesControllerTest do ~p"/api/log_lines?level=error×tamp_after=2024-01-01T11:00:00Z" ) - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["attributes"]["message"] == "Match" + assert ids(conn) == [match.id] end - test "supports pagination", %{ + test "pages with exact links to the neighbouring pages", %{ conn: conn, project: project } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] + run = insert_run(project) + + for n <- 1..25 do + insert_log_line(run, + timestamp: DateTime.add(~U[2024-01-01 00:00:00Z], n) ) + end - run = List.first(workorder.runs) + url = "#{LightningWeb.Endpoint.url()}/api/log_lines" - # Create 25 log lines - Enum.each(1..25, fn i -> - insert(:log_line, - run: run, - message: "Log #{i}", - timestamp: DateTime.utc_now() |> DateTime.add(i, :second) - ) - end) + assert %{"data" => data, "meta" => meta, "links" => links} = + conn + |> get(~p"/api/log_lines?page=2&page_size=10") + |> json_response(200) - # Get first page - conn = get(conn, ~p"/api/log_lines?page=1&page_size=10") - response = json_response(conn, 200) + assert length(data) == 10 - assert length(response["data"]) == 10 - assert response["meta"]["total_entries"] == 25 - assert response["meta"]["total_pages"] == 3 - assert response["meta"]["page_number"] == 1 - assert response["meta"]["page_size"] == 10 + assert meta == %{ + "total_entries" => 25, + "total_pages" => 3, + "page_number" => 2, + "page_size" => 10 + } - # Verify pagination links exist - assert Map.has_key?(response["links"], "first") - assert Map.has_key?(response["links"], "next") - assert Map.has_key?(response["links"], "last") + assert links == %{ + "self" => "#{url}?page=2&page_size=10", + "first" => "#{url}?page=1&page_size=10", + "last" => "#{url}?page=3&page_size=10", + "next" => "#{url}?page=3&page_size=10", + "prev" => "#{url}?page=1&page_size=10" + } end test "returns logs ordered by timestamp desc (newest first)", %{ conn: conn, project: project } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - run = List.first(workorder.runs) - - _log1 = - insert(:log_line, - run: run, - message: "First", - timestamp: ~U[2024-01-01 10:00:00Z] - ) + run = insert_run(project) + insert_log_line(run, message: "First", timestamp: ~U[2024-01-01 10:00:00Z]) log2 = - insert(:log_line, - run: run, + insert_log_line(run, message: "Second", timestamp: ~U[2024-01-01 11:00:00Z] ) - _log3 = - insert(:log_line, - run: run, - message: "Third", - timestamp: ~U[2024-01-01 12:00:00Z] - ) + insert_log_line(run, message: "Third", timestamp: ~U[2024-01-01 12:00:00Z]) conn = get(conn, ~p"/api/log_lines?page_size=1") response = json_response(conn, 200) - # Should get the newest log first assert List.first(response["data"])["attributes"]["message"] == "Third" - # Get second page conn = get(conn, ~p"/api/log_lines?page=2&page_size=1") response = json_response(conn, 200) assert List.first(response["data"])["id"] == log2.id end - test "returns error for invalid timestamp_after format", %{ + test "pagination links carry the filters applied and nothing else", %{ conn: conn, project: project } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - run = List.first(workorder.runs) - - insert(:log_line, - run: run, - message: "Test log", - timestamp: DateTime.utc_now() - ) - - conn = - get( - conn, - ~p"/api/log_lines?timestamp_after=not-a-valid-datetime" - ) - - response = json_response(conn, 400) - - assert %{"error" => error_message} = response - assert error_message =~ "Invalid datetime format for 'timestamp_after'" - assert error_message =~ "not-a-valid-datetime" + run = insert_run(project) + insert_log_line(run, level: :error) + insert_log_line(run, level: :error) + + %{"links" => links} = + conn + |> get( + ~p"/api/log_lines?page_size=1&foo=bar&level=error×tamp_after=2024-01-01T02:00:00%2B02:00" + ) + |> json_response(200) + + uri = URI.parse(links["next"]) + assert uri.path == "/api/log_lines" + + assert URI.decode_query(uri.query) == %{ + "level" => "error", + "timestamp_after" => "2024-01-01T00:00:00.000000Z", + "page" => "2", + "page_size" => "1" + } + + self_uri = URI.parse(links["self"]) + assert self_uri.path == "/api/log_lines" + + assert URI.decode_query(self_uri.query) == %{ + "level" => "error", + "timestamp_after" => "2024-01-01T00:00:00.000000Z", + "page" => "1", + "page_size" => "1" + } end - test "returns error for invalid timestamp_before format", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - run = List.first(workorder.runs) - - insert(:log_line, - run: run, - message: "Test log", - timestamp: DateTime.utc_now() - ) - - conn = - get( - conn, - ~p"/api/log_lines?timestamp_before=2024-13-45" - ) - - response = json_response(conn, 400) - - assert %{"error" => error_message} = response - assert error_message =~ "Invalid datetime format for 'timestamp_before'" - assert error_message =~ "2024-13-45" + test "every bad query parameter is reported, none quoted", %{conn: conn} do + bad = plant_sentinel(%{"v" => "x"})["v"] + + query = + URI.encode_query(%{ + "timestamp_after" => bad, + "timestamp_before" => bad, + "level" => bad, + "run_id" => bad, + "job_id" => bad, + "workflow_id" => bad, + "work_order_id" => bad + }) + + conn + |> get("/api/log_lines?" <> query) + |> refute_sentinel() + |> assert_problem(400, :invalid_parameter, [ + %{ + "parameter" => "timestamp_after", + "code" => "type", + "detail" => "is invalid" + }, + %{ + "parameter" => "timestamp_before", + "code" => "type", + "detail" => "is invalid" + }, + %{ + "parameter" => "level", + "code" => "not_allowed", + "detail" => "is invalid" + }, + %{"parameter" => "run_id", "code" => "type", "detail" => "is invalid"}, + %{"parameter" => "job_id", "code" => "type", "detail" => "is invalid"}, + %{ + "parameter" => "workflow_id", + "code" => "type", + "detail" => "is invalid" + }, + %{ + "parameter" => "work_order_id", + "code" => "type", + "detail" => "is invalid" + } + ]) end + end - test "returns error for invalid datetime with integer", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - run = List.first(workorder.runs) - - insert(:log_line, - run: run, - message: "Test log", - timestamp: DateTime.utc_now() - ) - - conn = get(conn, ~p"/api/log_lines?timestamp_after=123456") + describe "who may reach it" do + setup [:assign_bearer_for_api, :create_project_for_current_user] - response = json_response(conn, 400) + test "filtering by a project the caller has no standing on lists nothing", + %{conn: conn} do + elsewhere = insert(:project) + insert_log_line(insert_run(elsewhere), level: :error) - assert %{"error" => error_message} = response - assert error_message =~ "Invalid datetime format for 'timestamp_after'" - assert error_message =~ "123456" + assert %{"data" => []} = + conn + |> get(~p"/api/log_lines?#{%{project_id: elsewhere.id}}") + |> json_response(200) end end end diff --git a/test/lightning_web/controllers/api/project_controller_test.exs b/test/lightning_web/controllers/api/project_controller_test.exs deleted file mode 100644 index f5a9dd0a661..00000000000 --- a/test/lightning_web/controllers/api/project_controller_test.exs +++ /dev/null @@ -1,158 +0,0 @@ -defmodule LightningWeb.API.ProjectControllerTest do - use LightningWeb.ConnCase, async: true - - import Lightning.Factories - - setup %{conn: conn} do - {:ok, conn: put_req_header(conn, "accept", "application/json")} - end - - describe "without a token" do - test "gets a 401", %{conn: conn} do - conn = get(conn, ~p"/api/projects") - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - end - - describe "with invalid token" do - test "gets a 401", %{conn: conn} do - token = "Oooops" - conn = conn |> Plug.Conn.put_req_header("authorization", "Bearer #{token}") - conn = get(conn, ~p"/api/projects") - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - end - - # Proves the plug is wired into the /api scope, rather than only that the - # plug function itself refuses — see user_auth_test.exs for the unit tests. - describe "with an account past its confirmation deadline" do - test "gets a 401, and the same token works again once confirmed", %{ - conn: conn - } do - Mox.stub(Lightning.MockConfig, :check_flag?, fn - :require_email_verification -> true - flag -> Lightning.Config.API.check_flag?(flag) - end) - - user = insert(:user) - insert(:project, project_users: [%{user: user}]) - - token = Lightning.Accounts.generate_api_token(user) - - user = - user - |> Ecto.Changeset.change( - confirmed_at: nil, - inserted_at: - DateTime.utc_now() - |> DateTime.add(-50, :hour) - |> DateTime.truncate(:second) - ) - |> Lightning.Repo.update!() - - refused = conn |> assign_bearer(token) |> get(~p"/api/projects") - - assert json_response(refused, 401) == %{"error" => "Unauthorized"} - - user - |> Ecto.Changeset.change( - confirmed_at: DateTime.utc_now() |> DateTime.truncate(:second) - ) - |> Lightning.Repo.update!() - - allowed = conn |> assign_bearer(token) |> get(~p"/api/projects") - - assert [%{"type" => "projects"}] = json_response(allowed, 200)["data"] - end - end - - describe "index" do - setup [:assign_bearer_for_api, :create_project_for_current_user] - - test "lists all projects i belong to", %{conn: conn, project: project} do - conn = get(conn, ~p"/api/projects") - response = json_response(conn, 200) - - assert response["data"] == [ - %{ - "attributes" => %{ - "name" => project.name, - "description" => nil - }, - "id" => project.id, - "links" => %{ - "self" => - "#{LightningWeb.Endpoint.url()}/api/projects/#{project.id}" - }, - "relationships" => %{}, - "type" => "projects" - } - ] - end - - test "Other user don't have access to user project", %{ - conn: conn, - project: project - } do - other_user = insert(:user) - - token = - other_user - |> Lightning.Accounts.generate_api_token() - - conn = conn |> Plug.Conn.put_req_header("authorization", "Bearer #{token}") - - insert(:project, project_users: [%{user_id: other_user.id}]) - - conn = get(conn, ~p"/api/projects") - response = json_response(conn, 200) - - refute response["data"] == [ - %{ - "attributes" => %{"name" => "a-test-project"}, - "id" => project.id, - "links" => %{ - "self" => - "#{LightningWeb.Endpoint.url()}/api/projects/#{project.id}" - }, - "relationships" => %{}, - "type" => "projects" - } - ] - end - end - - describe "show" do - setup [:assign_bearer_for_api, :create_project_for_current_user] - - test "returns 404 for non-existent project", %{conn: conn} do - conn = get(conn, ~p"/api/projects/#{Ecto.UUID.generate()}") - assert json_response(conn, 404) - end - - test "with token for other project", %{conn: conn} do - other_project = insert(:project) - conn = get(conn, ~p"/api/projects/#{other_project.id}") - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - - test "shows the project", %{conn: conn, project: project} do - conn = get(conn, Routes.api_project_path(conn, :show, project)) - response = json_response(conn, 200) - - assert response["data"] == %{ - "attributes" => %{ - "name" => project.name, - "description" => nil - }, - "id" => project.id, - "links" => %{ - "self" => - "#{LightningWeb.Endpoint.url()}/api/projects/#{project.id}" - }, - "relationships" => %{}, - "type" => "projects" - } - end - end -end diff --git a/test/lightning_web/controllers/api/projects/reading_test.exs b/test/lightning_web/controllers/api/projects/reading_test.exs new file mode 100644 index 00000000000..57d4c757bb0 --- /dev/null +++ b/test/lightning_web/controllers/api/projects/reading_test.exs @@ -0,0 +1,125 @@ +defmodule LightningWeb.API.Projects.ReadingTest do + use LightningWeb.ConnCase, async: true + + import Lightning.Factories + import Lightning.ServiceAccountHelpers + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + describe "listing" do + setup [:assign_bearer_for_api, :create_project_for_current_user] + + test "page links carry the page size applied and no other query parameter", + %{conn: conn, project: project} do + page = + conn + |> get(~p"/api/projects?page=1&page_size=500&foo=bar") + |> json_response(200) + + first = + "#{LightningWeb.Endpoint.url()}/api/projects?page=1&page_size=100" + + assert page == %{ + "data" => [ + %{ + "attributes" => %{ + "name" => project.name, + "description" => nil + }, + "id" => project.id, + "links" => %{ + "self" => + "#{LightningWeb.Endpoint.url()}/api/projects/#{project.id}" + }, + "relationships" => %{}, + "type" => "projects" + } + ], + "included" => [], + "links" => %{ + "first" => first, + "self" => first, + "last" => first, + "next" => nil, + "prev" => nil + } + } + end + + test "lists only the projects the token's user belongs to", %{conn: conn} do + other_user = insert(:user) + + other_project = + insert(:project, project_users: [%{user_id: other_user.id}]) + + ids = + conn + |> assign_bearer(other_user) + |> get(~p"/api/projects") + |> json_response(200) + |> Map.fetch!("data") + |> Enum.map(& &1["id"]) + + assert ids == [other_project.id] + end + end + + describe "showing, with a person's token" do + setup [:assign_bearer_for_api, :create_project_for_current_user] + + test "shows the project with its members, linking to it without the request's query", + %{conn: conn, project: project, user: user} do + response = + conn + |> get(~p"/api/projects/#{project.id}?foo=bar") + |> json_response(200) + + url = "#{LightningWeb.Endpoint.url()}/api/projects/#{project.id}" + + assert response == %{ + "data" => %{ + "attributes" => %{ + "name" => project.name, + "description" => nil, + "members" => [%{"email" => user.email, "role" => "editor"}] + }, + "id" => project.id, + "links" => %{"self" => url}, + "relationships" => %{}, + "type" => "projects" + }, + "included" => [], + "links" => %{"self" => url} + } + end + end + + describe "showing, with a service account that has projects:write" do + setup %{conn: conn} do + {conn, _account} = with_service_account(conn, ["projects:write"]) + %{conn: conn} + end + + test "shows a project it is not a member of, with its members", %{ + conn: conn + } do + owner = insert(:user) + project = insert(:project, project_users: [%{user: owner, role: :owner}]) + + conn = get(conn, ~p"/api/projects/#{project.id}") + + assert %{"id" => id, "attributes" => attributes} = + json_response(conn, 200)["data"] + + assert id == project.id + + assert attributes == %{ + "name" => project.name, + "description" => project.description, + "members" => [%{"email" => owner.email, "role" => "owner"}] + } + end + end +end diff --git a/test/lightning_web/controllers/api/projects/refusing_a_body_test.exs b/test/lightning_web/controllers/api/projects/refusing_a_body_test.exs new file mode 100644 index 00000000000..b1376b0df9b --- /dev/null +++ b/test/lightning_web/controllers/api/projects/refusing_a_body_test.exs @@ -0,0 +1,364 @@ +defmodule LightningWeb.API.Projects.RefusingABodyTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProblemHelpers + import LightningWeb.ApiProjectsHelpers + import Lightning.Factories + import Lightning.ServiceAccountHelpers + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + describe "refusing a body, with a service account that has projects:write" do + setup %{conn: conn} do + {conn, account} = with_service_account(conn, ["projects:write"]) + %{conn: conn, account: account} + end + + test "refuses a blank name on replace as it does on create", %{conn: conn} do + project = insert(:project, name: "before") + + blank_name = [ + %{ + "pointer" => "#/name", + "code" => "required", + "detail" => "can't be blank" + } + ] + + conn + |> put_project(project.id, %{"name" => ""}) + |> assert_problem(422, :invalid, blank_name) + + conn + |> put_project(Ecto.UUID.generate(), %{ + "name" => "", + "members" => four_members() + }) + |> assert_problem(422, :invalid, blank_name) + + assert Lightning.Repo.reload!(project).name == "before" + end + + test "refuses members it cannot write, each at its place in members", %{ + conn: conn + } do + [owner, admin | _] = four_members() + + one_owner = fn detail -> + %{"pointer" => "#/members", "code" => "one_owner", "detail" => detail} + end + + refused = [ + unknown_email: { + [owner, %{"email" => "nobody@example.com", "role" => "admin"}], + [ + %{ + "pointer" => "#/members/1/email", + "code" => "not_found", + "detail" => "names nothing you can see" + } + ] + }, + missing_email_and_unknown_role: { + [owner, %{"role" => "admin"}, %{admin | "role" => "boss"}], + [ + %{ + "pointer" => "#/members/1/email", + "code" => "required", + "detail" => "can't be blank" + }, + %{ + "pointer" => "#/members/2/role", + "code" => "not_allowed", + "detail" => "is invalid" + } + ] + }, + two_owners: { + [owner, %{admin | "role" => "owner"}], + [one_owner.("A project can have only one owner.")] + }, + no_owner: { + [admin], + [ + one_owner.( + "Every project must have exactly one owner. Please specify one below." + ) + ] + }, + no_members: { + [], + [ + %{ + "pointer" => "#/members", + "code" => "required", + "detail" => "can't be blank" + }, + one_owner.( + "Every project must have exactly one owner. Please specify one below." + ) + ] + }, + not_a_list: { + %{"email" => owner["email"], "role" => "owner"}, + [ + %{ + "pointer" => "#/members", + "code" => "type", + "detail" => "is invalid" + } + ] + } + ] + + for {case_name, {members, entries}} <- refused do + id = Ecto.UUID.generate() + + response = + conn + |> put_project(id, %{"name" => "acme", "members" => members}) + |> assert_problem(422, :invalid, entries) + + refute Jason.encode!(response) =~ "@", + "#{case_name} echoes an email: #{inspect(response)}" + + refute Lightning.Repo.get(Lightning.Projects.Project, id) + end + end + + test "places a member's error by its position in the request, not the owner-last order members are written in", + %{conn: conn} do + [owner, admin, editor, _viewer] = four_members() + + conn + |> put_project(Ecto.UUID.generate(), %{ + "name" => "acme", + "members" => [ + owner, + admin, + %{"email" => "nobody@example.com", "role" => "editor"}, + %{admin | "email" => String.upcase(admin["email"])}, + editor, + %{admin | "role" => "viewer"} + ] + }) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/members/2/email", + "code" => "not_found", + "detail" => "names nothing you can see" + }, + %{ + "pointer" => "#/members/3/email", + "code" => "duplicate", + "detail" => "names the same user as #/members/1" + }, + %{ + "pointer" => "#/members/5/email", + "code" => "duplicate", + "detail" => "names the same user as #/members/1" + } + ]) + end + + test "answers 402 with the members list when the project refuses a member for want of a seat", + %{conn: conn} do + # An extension refusing a member the way a seat limit does. + Mox.expect(Lightning.Extensions.MockProjectHook, :handle_create_project, fn + attrs -> + %Lightning.Projects.Project{} + |> Lightning.Projects.Project.project_with_users_changeset(attrs) + |> Ecto.Changeset.add_error(:project_users, "has no seat left", + code: :exceeds_limit + ) + |> Ecto.Changeset.apply_action(:insert) + end) + + conn + |> put_project(Ecto.UUID.generate(), %{ + "name" => "acme", + "members" => four_members() + }) + |> assert_problem(402, :exceeds_limit, [ + %{ + "pointer" => "#/members", + "code" => "exceeds_limit", + "detail" => "has no seat left" + } + ]) + end + + test "reports every error in the body in one answer, an unknown key and a mismatched id among them", + %{conn: conn} do + [owner | _] = four_members() + id = Ecto.UUID.generate() + + conn + |> put_project(id, %{ + "id" => Ecto.UUID.generate(), + "name" => "Not Allowed", + "requires_mfa" => true, + "notify" => "sometimes", + "members" => [ + owner, + %{"email" => "nobody@example.com", "role" => "admin"} + ] + }) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/requires_mfa", + "code" => "unknown_key", + "detail" => "is not accepted" + }, + %{ + "pointer" => "#/id", + "code" => "mismatch", + "detail" => "does not match the id in the path" + }, + %{"pointer" => "#/notify", "code" => "type", "detail" => "is invalid"}, + %{ + "pointer" => "#/members/1/email", + "code" => "not_found", + "detail" => "names nothing you can see" + }, + %{ + "pointer" => "#/name", + "code" => "format", + "detail" => "has invalid format" + } + ]) + + refute Lightning.Repo.get(Lightning.Projects.Project, id) + end + + test "refuses a notify of null, under notify", %{conn: conn} do + id = Ecto.UUID.generate() + + conn + |> put_project(id, %{ + "name" => "acme", + "members" => four_members(), + "notify" => nil + }) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/notify", + "code" => "required", + "detail" => "can't be blank" + } + ]) + + refute Lightning.Repo.get(Lightning.Projects.Project, id) + end + + test "refuses a body that is not a JSON object", %{conn: conn} do + conn = put_project(conn, Ecto.UUID.generate(), [%{"name" => "acme"}]) + + assert_problem(conn, 400, :malformed, []) + end + + test "refuses a name or description the project cannot hold", %{conn: conn} do + conn_name = + put_project(conn, Ecto.UUID.generate(), %{ + "name" => "Not Allowed", + "members" => four_members() + }) + + assert_problem(conn_name, 422, :invalid, [ + %{ + "pointer" => "#/name", + "code" => "format", + "detail" => "has invalid format" + } + ]) + + conn_description = + put_project(conn, Ecto.UUID.generate(), %{ + "name" => "acme", + "description" => String.duplicate("a", 241), + "members" => four_members() + }) + + assert_problem(conn_description, 422, :invalid, [ + %{ + "pointer" => "#/description", + "code" => "too_long", + "detail" => "should be at most 240 character(s)" + } + ]) + end + end + + describe "answers that quote nothing from the request" do + setup %{conn: conn} do + {conn, _account} = with_service_account(conn, ["projects:write"]) + + %{ + conn: conn, + planted: + plant_sentinel(%{ + "id" => nil, + "name" => nil, + "description" => nil, + "notify" => nil, + "members" => [%{"email" => nil, "role" => nil}], + "colour" => nil + }) + } + end + + test "never quote a value from the request, a member's email and role among them", + %{conn: conn, planted: planted} do + conn + |> get(~p"/api/projects/#{planted["name"]}") + |> refute_sentinel() + |> assert_problem(404, :not_found, []) + + conn + |> put_project(planted["name"], planted) + |> refute_sentinel() + |> assert_problem(404, :not_found, []) + + conn + |> put_project(Ecto.UUID.generate(), planted) + |> refute_sentinel() + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/colour", + "code" => "unknown_key", + "detail" => "is not accepted" + }, + %{ + "pointer" => "#/id", + "code" => "mismatch", + "detail" => "does not match the id in the path" + }, + %{"pointer" => "#/notify", "code" => "type", "detail" => "is invalid"}, + %{ + "pointer" => "#/members/0/role", + "code" => "not_allowed", + "detail" => "is invalid" + } + ]) + + conn + |> put_project(insert(:project).id, planted) + |> refute_sentinel() + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/colour", + "code" => "unknown_key", + "detail" => "is not accepted" + }, + %{ + "pointer" => "#/id", + "code" => "mismatch", + "detail" => "does not match the id in the path" + }, + %{"pointer" => "#/notify", "code" => "type", "detail" => "is invalid"} + ]) + end + end +end diff --git a/test/lightning_web/controllers/api/projects/replacing_test.exs b/test/lightning_web/controllers/api/projects/replacing_test.exs new file mode 100644 index 00000000000..f76c99ae7c7 --- /dev/null +++ b/test/lightning_web/controllers/api/projects/replacing_test.exs @@ -0,0 +1,187 @@ +defmodule LightningWeb.API.Projects.ReplacingTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProblemHelpers + import LightningWeb.ApiProjectsHelpers + import Lightning.RaceHelpers + import Lightning.Factories + import Ecto.Query + import Lightning.ServiceAccountHelpers + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + describe "replacing or creating, with a service account that has projects:write" do + setup %{conn: conn} do + {conn, account} = with_service_account(conn, ["projects:write"]) + %{conn: conn, account: account} + end + + test "creates the project with its members and links to it without the request's query, and is not one of them", + %{ + conn: conn, + account: account + } do + id = Ecto.UUID.generate() + members = four_members() + + conn = + conn + |> put_req_header("content-type", "application/json") + |> put( + ~p"/api/projects/#{id}?foo=bar", + Jason.encode!(%{ + "name" => "acme", + "description" => "The acme project", + "members" => members, + "notify" => false + }) + ) + + response = json_response(conn, 201) + self_link = "#{LightningWeb.Endpoint.url()}/api/projects/#{id}" + + assert %{ + "id" => ^id, + "type" => "projects", + "attributes" => %{ + "name" => "acme", + "description" => "The acme project" + }, + "links" => %{ + "self" => ^self_link + } + } = response["data"] + + assert response["links"] == %{"self" => self_link} + assert sorted_members(response) == Enum.sort_by(members, & &1["email"]) + + assert members_of(id) == + members + |> Enum.map(&{&1["email"], String.to_existing_atom(&1["role"])}) + |> Enum.sort() + + actors = + from(a in Lightning.Auditing.Audit, + where: a.item_id == ^id, + select: {a.event, a.actor_id, a.actor_type} + ) + |> Lightning.Repo.all() + + assert Enum.count(actors, &match?({"collaborator_added", _, _}, &1)) == 4 + assert {"created", account.uuid, :service_account} in actors + + assert Enum.all?(actors, fn {_, actor_id, actor_type} -> + actor_id == account.uuid and actor_type == :service_account + end) + end + + test "emails the members it adds unless told not to", %{conn: conn} do + Oban.Testing.with_testing_mode(:manual, fn -> + quiet_id = Ecto.UUID.generate() + + put_project(conn, quiet_id, %{ + "name" => "quiet", + "members" => four_members(), + "notify" => false + }) + |> json_response(201) + + assert all_enqueued(worker: Lightning.Accounts.UserNotifier) == [] + + put_project(conn, Ecto.UUID.generate(), %{ + "name" => "loud", + "members" => four_members() + }) + |> json_response(201) + + assert length(all_enqueued(worker: Lightning.Accounts.UserNotifier)) == + 4 + end) + end + + test "matches a member's email without regard to case", %{conn: conn} do + owner = insert(:user, email: "mixed.case@example.com") + id = Ecto.UUID.generate() + + conn = + put_project(conn, id, %{ + "name" => "acme", + "members" => [ + %{"email" => "Mixed.CASE@example.com", "role" => "owner"} + ] + }) + + assert json_response(conn, 201) + assert members_of(id) == [{owner.email, :owner}] + end + + test "a project another request deletes mid-replace answers 409 concurrent_edit", + %{conn: conn} do + owner = insert(:user) + + project = + insert(:project, + name: "before", + project_users: [%{user: owner, role: :owner}] + ) + + delete_project_mid_save(project.id) + + conn + |> put_project(project.id, %{ + "name" => "after", + "members" => [%{"email" => owner.email, "role" => "owner"}] + }) + |> assert_problem(409, :concurrent_edit, []) + + refute Lightning.Repo.reload(project) + end + + test "replaces the name and description and keeps the members", %{ + conn: conn + } do + owner = insert(:user) + + project = + insert(:project, + name: "before", + description: "old", + project_users: [%{user: owner, role: :owner}] + ) + + conn = + put_project(conn, project.id, %{ + "name" => "after", + "description" => "new", + "members" => [%{"email" => insert(:user).email, "role" => "owner"}] + }) + + response = json_response(conn, 200) + + assert response["data"]["attributes"] == %{ + "name" => "after", + "description" => "new", + "members" => [%{"email" => owner.email, "role" => "owner"}] + } + + assert %{name: "after", description: "new"} = + Lightning.Repo.reload!(project) + + assert members_of(project.id) == [{owner.email, :owner}] + end + end + + describe "creating, with a person's token" do + test "a superuser creates a project", %{conn: conn} do + conn = with_pat(conn, insert(:user, role: :superuser)) + id = Ecto.UUID.generate() + + conn = + put_project(conn, id, %{"name" => "acme", "members" => four_members()}) + + assert %{"id" => ^id} = json_response(conn, 201)["data"] + end + end +end diff --git a/test/lightning_web/controllers/api/projects/who_may_reach_it_test.exs b/test/lightning_web/controllers/api/projects/who_may_reach_it_test.exs new file mode 100644 index 00000000000..b72fe05ba96 --- /dev/null +++ b/test/lightning_web/controllers/api/projects/who_may_reach_it_test.exs @@ -0,0 +1,93 @@ +defmodule LightningWeb.API.Projects.WhoMayReachItTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProblemHelpers + import LightningWeb.ApiProjectsHelpers + import Lightning.Factories + import Lightning.ServiceAccountHelpers + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + describe "who may reach it" do + test "an editor or a viewer may not replace a project, and a stranger may not see or replace it", + %{conn: conn} do + refusals = [ + {:editor, [:put], 403, :forbidden}, + {:viewer, [:put], 403, :forbidden}, + {:stranger, [:get, :put], 404, :not_found} + ] + + for {role, verbs, status, code} <- refusals, verb <- verbs do + user = insert(:user) + members = if role == :stranger, do: [], else: [%{user: user, role: role}] + project = insert(:project, name: "before", project_users: members) + + conn + |> with_pat(user) + |> reach(verb, project.id) + |> assert_problem(status, code, []) + + assert Lightning.Repo.reload!(project).name == "before", + "#{role} changed the project by #{verb}" + end + end + + test "an owner or an admin may replace a project", %{conn: conn} do + for role <- [:owner, :admin] do + user = insert(:user) + project = insert(:project, project_users: [%{user: user, role: role}]) + + conn + |> with_pat(user) + |> put_project(project.id, %{"name" => "after"}) + |> json_response(200) + + assert Lightning.Repo.reload!(project).name == "after" + end + end + + test "an id that names no project is 404 to read, and creating it is refused unless the token's user is a superuser", + %{conn: conn} do + conn = with_pat(conn, insert(:user)) + id = Ecto.UUID.generate() + + conn |> reach(:get, id) |> assert_problem(404, :not_found, []) + conn |> reach(:put, id) |> assert_problem(403, :forbidden, []) + + refute Lightning.Repo.get(Lightning.Projects.Project, id) + end + + test "a project scheduled for deletion is 404 to a stranger and 409 to a member or a service account, on both routes", + %{conn: conn} do + {service_account_conn, _account} = + with_service_account(conn, ["projects:write"]) + + owner = insert(:user) + stranger = insert(:user) + + callers = [ + {with_pat(conn, stranger), 404, :not_found}, + {with_pat(conn, owner), 409, :scheduled_for_deletion}, + {service_account_conn, 409, :scheduled_for_deletion} + ] + + for {caller, status, code} <- callers, verb <- [:get, :put] do + project = scheduled_project([%{user: owner, role: :owner}]) + + caller + |> reach(verb, project.id) + |> assert_problem(status, code, []) + + assert Lightning.Repo.reload!(project).name == project.name + end + end + end + + # A body no route would accept, so a refusal that comes first is visible. + defp reach(conn, :get, id), do: get(conn, ~p"/api/projects/#{id}") + + defp reach(conn, :put, id), + do: put_project(conn, id, %{"name" => "renamed", "nonsense" => 1}) +end diff --git a/test/lightning_web/controllers/api/provisioning/importing_test.exs b/test/lightning_web/controllers/api/provisioning/importing_test.exs new file mode 100644 index 00000000000..2ef7bf42b5b --- /dev/null +++ b/test/lightning_web/controllers/api/provisioning/importing_test.exs @@ -0,0 +1,327 @@ +defmodule LightningWeb.API.ProvisioningImportingTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProvisioningHelpers + import LightningWeb.API.ProblemHelpers + import Ecto.Query + import Lightning.Factories + + alias Lightning.Workflows.Workflow + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + describe "importing a document" do + setup [:assign_bearer_for_api] + + test "fails with 402 when repo_connection is limited by github_sync", %{ + conn: conn + } do + %{id: project_id} = project = insert(:project) + + repo_connection = + insert(:project_repo_connection, project: project) + + %{body: body} = valid_payload(project.id) + + conn = + put_bearer(conn, repo_connection.access_token) + + error_text = "Upgrade to use Github Sync" + + Lightning.Extensions.MockUsageLimiter + |> Mox.expect(:limit_action, fn %{type: :github_sync}, + %{project_id: ^project_id} -> + {:error, :disabled, %Lightning.Extensions.Message{text: error_text}} + end) + + conn = post(conn, ~p"/api/provision", body) + assert_problem(conn, 402, :exceeds_limit, []) + refute conn.resp_body =~ error_text + end + + test "an owner replaces a project's name and jobs by posting a changed document", + %{ + conn: conn, + user: user + } do + project = + insert(:project, + project_users: [%{user_id: user.id, role: :owner}] + ) + + %{ + project_id: project_id, + body: body, + second_job_id: second_job_id + } = valid_payload(project.id) + + conn = post(conn, ~p"/api/provision", body) + response = json_response(conn, 201) + + assert %{ + "id" => ^project_id, + "name" => "test-project", + "workflows" => [_] + } = response["data"] + + # - - - + third_job_id = Ecto.UUID.generate() + + body = + body + |> Map.put("name", "test-project-renamed") + |> add_job_to_document(%{ + "id" => third_job_id, + "name" => "third-job", + "adaptor" => "@openfn/language-common@latest", + "body" => "console.log('hello world');" + }) + + conn = post(conn, ~p"/api/provision", body) + response = json_response(conn, 201) + + assert %{ + "id" => ^project_id, + "name" => "test-project-renamed", + "workflows" => workflows + } = response["data"] + + workflow_job_ids = + workflows |> Enum.at(0) |> Map.get("jobs") |> Enum.into([], & &1["id"]) + + assert third_job_id in workflow_job_ids + + body = body |> remove_job_from_document(second_job_id) + + conn = post(conn, ~p"/api/provision", body) + response = json_response(conn, 201) + + assert %{ + "id" => ^project_id, + "name" => "test-project-renamed", + "workflows" => workflows + } = response["data"] + + workflow_job_ids = + workflows |> Enum.at(0) |> Map.get("jobs") |> Enum.into([], & &1["id"]) + + refute second_job_id in workflow_job_ids + assert third_job_id in workflow_job_ids + + assert workflows |> Enum.at(0) |> Map.get("edges") == [], + "The edge associated with the deleted job should be removed" + end + + test "records workflow version with 'cli' source when workflow is created via provisioner", + %{conn: conn, user: user} do + project = + insert(:project, + project_users: [%{user_id: user.id, role: :owner}] + ) + + %{body: body, workflow_id: workflow_id} = valid_payload(project.id) + + conn = post(conn, ~p"/api/provision", body) + assert json_response(conn, 201) + + workflow = Lightning.Repo.get!(Workflow, workflow_id) + version_history = Lightning.WorkflowVersions.history_for(workflow) + + assert length(version_history) == 1 + assert [version] = version_history + assert String.starts_with?(version, "cli:") + end + + test "records new workflow version when workflow is updated via provisioner", + %{conn: conn, user: user} do + project = + insert(:project, + project_users: [%{user_id: user.id, role: :owner}] + ) + + # First provision - creates workflow + %{body: body, workflow_id: workflow_id} = valid_payload(project.id) + + conn = post(conn, ~p"/api/provision", body) + assert json_response(conn, 201) + + workflow = Lightning.Repo.get!(Workflow, workflow_id) + initial_history = Lightning.WorkflowVersions.history_for(workflow) + assert length(initial_history) == 1 + assert [initial_version] = initial_history + assert String.starts_with?(initial_version, "cli:") + + # Update workflow by changing job body + updated_body = + body + |> Map.update!("workflows", fn workflows -> + Enum.at(workflows, 0) + |> Map.update!("jobs", fn jobs -> + Enum.map(jobs, fn job -> + Map.put(job, "body", "console.log('updated');") + end) + end) + |> then(fn workflow -> + List.replace_at(workflows, 0, workflow) + end) + end) + + conn = post(conn, ~p"/api/provision", updated_body) + assert json_response(conn, 201) + + workflow = Lightning.Repo.get!(Workflow, workflow_id) + updated_history = Lightning.WorkflowVersions.history_for(workflow) + + # we nolonger squash the first version + assert length(updated_history) == 2 + assert [^initial_version, latest_version] = updated_history + assert String.starts_with?(latest_version, "cli:") + + # Verify the hash changed after the update + refute initial_version == latest_version + end + + test "does not create duplicate version when provisioning same workflow content", + %{conn: conn, user: user} do + project = + insert(:project, + project_users: [%{user_id: user.id, role: :owner}] + ) + + # First provision - creates workflow + %{body: body, workflow_id: workflow_id} = valid_payload(project.id) + + conn = post(conn, ~p"/api/provision", body) + assert json_response(conn, 201) + + workflow = Lightning.Repo.get!(Workflow, workflow_id) + initial_history = Lightning.WorkflowVersions.history_for(workflow) + assert length(initial_history) == 1 + + # Provision again with the exact same content + conn = post(conn, ~p"/api/provision", body) + assert json_response(conn, 201) + + workflow = Lightning.Repo.get!(Workflow, workflow_id) + duplicate_history = Lightning.WorkflowVersions.history_for(workflow) + + # Should still have only 1 version since content didn't change + assert length(duplicate_history) == 1 + assert duplicate_history == initial_history + end + + test "handles provisioning with no workflow changes gracefully", + %{conn: conn, user: user} do + project = + insert(:project, + project_users: [%{user_id: user.id, role: :owner}] + ) + + # Create a project with a workflow that has an initial version + %{body: body, workflow_id: workflow_id} = valid_payload(project.id) + + # First provision to create the workflow + conn = post(conn, ~p"/api/provision", body) + assert json_response(conn, 201) + + # Get the workflow and its initial version history + workflow = Lightning.Repo.get!(Workflow, workflow_id) + initial_history = Lightning.WorkflowVersions.history_for(workflow) + assert length(initial_history) == 1 + + # Update only project metadata, keeping workflow unchanged + updated_body = Map.put(body, "name", "updated-project-name") + + conn = post(conn, ~p"/api/provision", updated_body) + assert json_response(conn, 201) + + # Verify version history unchanged since workflow content didn't change + workflow = Lightning.Repo.get!(Workflow, workflow_id) + version_history = Lightning.WorkflowVersions.history_for(workflow) + assert version_history == initial_history + + # Verify project name was updated + project = Lightning.Repo.get!(Lightning.Projects.Project, project.id) + assert project.name == "updated-project-name" + end + + test "returns 201 for an existing project with workflows marked for deletion", + %{ + conn: conn + } do + project = insert(:project) + + _deleted_workflow = + insert(:workflow, + project: project, + name: "Deleted workflow", + deleted_at: DateTime.utc_now() + ) + + repo_connection = + insert(:project_repo_connection, project: project) + + %{body: body} = valid_payload(project.id) + + conn = + put_bearer(conn, repo_connection.access_token) + + assert post(conn, ~p"/api/provision", body) |> json_response(201) + end + + @tag login_as: "superuser" + test "allows a superuser to create a new project", %{conn: conn} do + %{ + body: body, + project_id: project_id, + first_job_id: first_job_id, + second_job_id: second_job_id, + trigger_id: trigger_id, + workflow_id: workflow_id, + job_edge_id: job_edge_id + } = valid_payload() + + conn = post(conn, ~p"/api/provision", body) + response = json_response(conn, 201) + + assert %{ + "id" => ^project_id, + "name" => "test-project", + "workflows" => workflows + } = response["data"] + + project = Lightning.Projects.get_project!(project_id) + + assert project.name == "test-project" + + workflow_ids = Enum.map(workflows, fn w -> w["id"] end) + + parent_project_ids = + from(w in Workflow, + where: w.id in ^workflow_ids + ) + |> Repo.all() + |> Enum.map(fn w -> w.project_id end) + + assert parent_project_ids |> Enum.all?(&match?(^project_id, &1)), + "All workflows should belong to the same project" + + workflow = + from(w in Workflow, + preload: [:jobs, :triggers, :edges], + where: w.id == ^workflow_id + ) + |> Lightning.Repo.one!() + + assert workflow.name == "default" + assert workflow.edges |> MapSet.new(& &1.id) == MapSet.new([job_edge_id]) + + assert workflow.jobs |> MapSet.new(& &1.id) == + MapSet.new([first_job_id, second_job_id]) + + assert workflow.triggers |> MapSet.new(& &1.id) == MapSet.new([trigger_id]) + end + end +end diff --git a/test/lightning_web/controllers/api/provisioning/invalid_documents_test.exs b/test/lightning_web/controllers/api/provisioning/invalid_documents_test.exs new file mode 100644 index 00000000000..321676c70e7 --- /dev/null +++ b/test/lightning_web/controllers/api/provisioning/invalid_documents_test.exs @@ -0,0 +1,301 @@ +defmodule LightningWeb.API.ProvisioningInvalidDocumentsTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProvisioningHelpers + import LightningWeb.API.ProblemHelpers + import Lightning.Factories + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + describe "a document that breaks a rule" do + setup [:assign_bearer_for_api] + + setup %{user: user} do + %{ + project: + insert(:project, project_users: [%{user_id: user.id, role: :owner}]) + } + end + + test "a blank project name and a workflow without an id answer 422 required at each pointer", + %{conn: conn, project: project} do + body = %{ + "id" => project.id, + "name" => "", + "workflows" => [ + %{"name" => "default"}, + %{"id" => Ecto.UUID.generate(), "name" => "Valid Workflow"} + ] + } + + conn + |> post(~p"/api/provision", body) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/name", + "code" => "required", + "detail" => "can't be blank" + }, + %{ + "pointer" => "#/workflows/0/id", + "code" => "required", + "detail" => "can't be blank" + } + ]) + end + + test "jobs missing an id, a name or a body answer 422 required at each pointer", + %{conn: conn, project: project} do + body = %{ + "id" => project.id, + "name" => "test-project", + "workflows" => [ + %{ + "name" => "default", + "jobs" => [ + %{ + "name" => "first-job", + "adaptor" => "@openfn/language-common@latest" + }, + %{ + "adaptor" => "@openfn/language-common@latest", + "body" => "console.log('hello world');" + }, + %{ + "id" => Ecto.UUID.generate(), + "name" => "valid job", + "adaptor" => "@openfn/language-common@latest", + "body" => "console.log('hello world');" + } + ] + } + ] + } + + conn + |> post(~p"/api/provision", body) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/workflows/0/id", + "code" => "required", + "detail" => "can't be blank" + }, + %{ + "pointer" => "#/workflows/0/jobs/0/id", + "code" => "required", + "detail" => "can't be blank" + }, + %{ + "pointer" => "#/workflows/0/jobs/0/body", + "code" => "required", + "detail" => "job body can't be blank" + }, + %{ + "pointer" => "#/workflows/0/jobs/1/id", + "code" => "required", + "detail" => "can't be blank" + }, + %{ + "pointer" => "#/workflows/0/jobs/1/name", + "code" => "required", + "detail" => "job name can't be blank" + } + ]) + end + + test "an edge without a condition type answers 422 required, whatever its trigger's type", + %{conn: conn, project: project} do + job_id = Ecto.UUID.generate() + trigger_id = Ecto.UUID.generate() + + for trigger_type <- ["cron", "webhook"] do + body = %{ + "id" => project.id, + "name" => "test-project", + "workflows" => [ + %{ + "id" => Ecto.UUID.generate(), + "name" => "default", + "jobs" => [ + %{ + "id" => job_id, + "name" => "first-job", + "adaptor" => "@openfn/language-common@latest", + "body" => "console.log('hello world')" + } + ], + "triggers" => [ + %{"id" => trigger_id, "type" => trigger_type} + ], + "edges" => [ + %{ + "id" => Ecto.UUID.generate(), + "source_trigger_id" => trigger_id, + "target_job_id" => job_id + } + ] + } + ] + } + + conn + |> post(~p"/api/provision", body) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/workflows/0/edges/0/condition_type", + "code" => "required", + "detail" => "can't be blank" + } + ]) + end + end + + test "a key the server doesn't accept answers 422 unknown_key at its own pointer", + %{conn: conn, project: project} do + %{body: body} = valid_payload(project.id) + + body = + body + |> Map.put("foo", "bar") + |> put_in( + ["workflows", Access.at(0), "triggers", Access.at(0), "project_id"], + "anything" + ) + + conn + |> post(~p"/api/provision", body) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/foo", + "code" => "unknown_key", + "detail" => "is not accepted" + }, + %{ + "pointer" => "#/workflows/0/triggers/0/project_id", + "code" => "unknown_key", + "detail" => "is not accepted" + } + ]) + end + + test "a credential name the caller can't see answers 422 not_found at that link's name", + %{conn: conn, project: project, user: user} do + known = insert(:credential, name: "known", user: user) + %{body: body} = valid_payload(project.id) + + body = + Map.put(body, "project_credentials", [ + %{ + "id" => Ecto.UUID.generate(), + "name" => known.name, + "owner" => user.email + }, + %{ + "id" => Ecto.UUID.generate(), + "name" => "missing", + "owner" => user.email + } + ]) + + conn + |> post(~p"/api/provision", body) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/project_credentials/1/name", + "code" => "not_found", + "detail" => "names nothing you can see" + } + ]) + end + + test "a job marked for deletion keeps its place, so later errors point past it", + %{conn: conn, project: project} do + %{body: body, first_job_id: first_job_id} = valid_payload(project.id) + + conn |> post(~p"/api/provision", body) |> json_response(201) + + body = + body + |> remove_job_from_document(first_job_id) + |> update_in(["workflows", Access.at(0), "jobs"], fn jobs -> + jobs ++ [%{"id" => Ecto.UUID.generate(), "body" => "fn(s => s)"}] + end) + + conn + |> post(~p"/api/provision", body) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/workflows/0/jobs/2/name", + "code" => "required", + "detail" => "job name can't be blank" + } + ]) + + assert Lightning.Repo.get(Lightning.Workflows.Job, first_job_id), + "a refused document deleted a job" + end + + test "two jobs with one name answer 409 taken, with one entry", %{ + conn: conn, + project: project + } do + %{body: body} = valid_payload(project.id) + + body = + update_in(body, ["workflows", Access.at(0), "jobs", Access.at(1)], fn + job -> Map.put(job, "name", "first-job") + end) + + conn + |> post(~p"/api/provision", body) + |> assert_problem(409, :taken, [ + %{ + "pointer" => "#/workflows/0/jobs/1/name", + "code" => "taken", + "detail" => "job name has already been taken" + } + ]) + end + + test "a project credential, collection or channel id another project uses answers 409 taken at its position", + %{conn: conn, project: project, user: user} do + insert(:credential, user: user, name: "mine") + %{body: body} = valid_payload(project.id) + + for {key, taken_id, item} <- [ + {"project_credentials", insert(:project_credential).id, + fn id, _name -> + %{"id" => id, "name" => "mine", "owner" => user.email} + end}, + {"collections", insert(:collection).id, + fn id, name -> %{"id" => id, "name" => name} end}, + {"channels", insert(:channel).id, + fn id, name -> + %{ + "id" => id, + "name" => name, + "destination_url" => "https://example.com/#{name}" + } + end} + ] do + body = + Map.put(body, key, [ + item.(Ecto.UUID.generate(), "first"), + item.(taken_id, "second") + ]) + + conn + |> post(~p"/api/provision", body) + |> assert_problem(409, :taken, [ + %{ + "pointer" => "#/#{key}/1/id", + "code" => "taken", + "detail" => "has already been taken" + } + ]) + end + end + end +end diff --git a/test/lightning_web/controllers/api/provisioning/parameters_test.exs b/test/lightning_web/controllers/api/provisioning/parameters_test.exs new file mode 100644 index 00000000000..b3c76987a8e --- /dev/null +++ b/test/lightning_web/controllers/api/provisioning/parameters_test.exs @@ -0,0 +1,144 @@ +defmodule LightningWeb.API.ProvisioningParametersTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProvisioningHelpers + import LightningWeb.API.ProblemHelpers + import Lightning.Factories + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + describe "parameters and identifiers" do + setup [:assign_bearer_for_api] + + setup %{user: user} do + %{ + project: + insert(:project, project_users: [%{user_id: user.id, role: :owner}]) + } + end + + test "a project id in the body that isn't a UUID is not found", %{ + conn: conn + } do + %{body: body} = valid_payload("not-a-uuid") + + conn + |> post(~p"/api/provision", body) + |> assert_problem(404, :not_found, []) + end + + test "GET answers an id that isn't there, or a yaml id that isn't a UUID, as not found", + %{conn: conn} do + for path <- [ + ~p"/api/provision/#{Ecto.UUID.generate()}", + ~p"/api/provision/yaml?#{%{id: Ecto.UUID.generate()}}", + ~p"/api/provision/yaml?#{%{id: "not-a-uuid"}}" + ] do + conn |> get(path) |> assert_problem(404, :not_found, []) + end + end + + test "GET yaml without an id names the parameter", %{conn: conn} do + conn + |> get(~p"/api/provision/yaml") + |> assert_problem(400, :invalid_parameter, [ + %{ + "parameter" => "id", + "code" => "required", + "detail" => "can't be blank" + } + ]) + end + + test "no answer quotes a value planted in the project credentials and channels of a document", + %{ + conn: conn, + project: project, + user: user + } do + %{body: body} = valid_payload(project.id) + + body = + body + |> Map.merge(%{ + "project_credentials" => [ + %{"id" => Ecto.UUID.generate(), "name" => "x", "owner" => user.email} + ], + "channels" => [ + %{ + "id" => Ecto.UUID.generate(), + "name" => "x", + "destination_url" => "x", + "destination_credential_id" => "x" + } + ] + }) + |> plant_sentinel() + |> put_in(["project_credentials", Access.at(0), "owner"], user.email) + |> Map.put("id", project.id) + + conn + |> post(~p"/api/provision", body) + |> refute_sentinel() + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/project_credentials/0/name", + "code" => "not_found", + "detail" => "names nothing you can see" + }, + %{ + "pointer" => "#/channels/0/destination_url", + "code" => "format", + "detail" => "must be either a http or https URL" + }, + %{ + "pointer" => "#/workflows/0/edges/0/condition_type", + "code" => "not_allowed", + "detail" => "is invalid" + }, + %{ + "pointer" => "#/workflows/0/edges/0/id", + "code" => "format", + "detail" => "is not a valid UUID" + }, + %{ + "pointer" => "#/workflows/0/edges/0/source_job_id", + "code" => "format", + "detail" => "is not a valid UUID" + }, + %{ + "pointer" => "#/workflows/0/edges/0/target_job_id", + "code" => "format", + "detail" => "is not a valid UUID" + }, + %{ + "pointer" => "#/workflows/0/edges/0/target_job_id", + "code" => "cycle", + "detail" => "target_job_id must be different from source_job_id" + }, + %{ + "pointer" => "#/workflows/0/triggers/0/id", + "code" => "format", + "detail" => "is not a valid UUID" + }, + %{ + "pointer" => "#/workflows/0/jobs/0/id", + "code" => "format", + "detail" => "is not a valid UUID" + }, + %{ + "pointer" => "#/workflows/0/jobs/1/id", + "code" => "format", + "detail" => "is not a valid UUID" + }, + %{ + "pointer" => "#/workflows/0/jobs/1/id", + "code" => "duplicate", + "detail" => "names the same id as #/workflows/0/jobs/0" + } + ]) + end + end +end diff --git a/test/lightning_web/controllers/api/provisioning/refusals_test.exs b/test/lightning_web/controllers/api/provisioning/refusals_test.exs new file mode 100644 index 00000000000..ab41cdbd14e --- /dev/null +++ b/test/lightning_web/controllers/api/provisioning/refusals_test.exs @@ -0,0 +1,198 @@ +defmodule LightningWeb.API.ProvisioningRefusalsTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProvisioningHelpers + import LightningWeb.API.ProblemHelpers + import Lightning.Factories + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + describe "who may reach it, reading" do + setup [:assign_bearer_for_api] + + test "every role reads the project and a non-member is told it doesn't exist", + %{conn: conn, user: user} do + for role <- [:owner, :admin, :editor, :viewer], + path <- [:state, :yaml] do + project = + insert(:project, project_users: [%{user_id: user.id, role: role}]) + + assert get(conn, read_path(path, project)).status == 200, + "a #{role} was refused the #{path}" + end + + project = insert(:project) + + for path <- [:state, :yaml] do + conn + |> get(read_path(path, project)) + |> assert_problem(404, :not_found, []) + end + end + + test "a repo connection token reads its own project and no other", %{ + conn: conn + } do + project = insert(:project) + other_project = insert(:project) + own = insert(:project_repo_connection, project: project) + other = insert(:project_repo_connection, project: other_project) + + for path <- [:state, :yaml] do + assert conn + |> put_bearer(own.access_token) + |> get(read_path(path, project)) + |> Map.fetch!(:status) == 200 + + conn + |> put_bearer(other.access_token) + |> get(read_path(path, project)) + |> assert_problem(404, :not_found, []) + end + end + end + + describe "who may reach it, importing" do + setup [:assign_bearer_for_api] + + test "owners and admins import; editors and viewers are forbidden; a non-member is told the project doesn't exist", + %{conn: conn, user: user} do + for {member_role, refusal} <- [ + {:owner, nil}, + {:admin, nil}, + {:editor, {403, :forbidden}}, + {:viewer, {403, :forbidden}}, + {nil, {404, :not_found}} + ] do + project = + insert(:project, + project_users: + if(member_role, + do: [%{user_id: user.id, role: member_role}], + else: [] + ) + ) + + %{body: body} = valid_payload(project.id) + response = post(conn, ~p"/api/provision", body) + + case refusal do + nil -> + assert response.status == 201, "a #{member_role} was refused" + + {status, code} -> + assert_problem(response, status, code, []) + + assert Lightning.Repo.reload!(project).name == project.name, + "a refused #{member_role} still renamed the project" + end + end + end + + test "a repo connection token imports into its own project and not another", + %{conn: conn} do + project = insert(:project) + other_project = insert(:project) + repo_connection = insert(:project_repo_connection, project: project) + conn = put_bearer(conn, repo_connection.access_token) + + %{body: body} = valid_payload(project.id) + assert post(conn, ~p"/api/provision", body).status == 201 + + %{body: other_body} = valid_payload(other_project.id) + + conn + |> post(~p"/api/provision", other_body) + |> assert_problem(404, :not_found, []) + end + + test "a user who isn't a superuser can't create a new project", %{ + conn: conn + } do + %{body: body} = valid_payload() + + conn + |> post(~p"/api/provision", body) + |> assert_problem(403, :forbidden, []) + end + end + + describe "a project scheduled for deletion" do + setup [:assign_bearer_for_api] + + # Scheduling deletion removes no membership row and revokes no + # repo-connection token, so both callers below still exist during the grace + # window. They get the same 409 that the projects API gives. + setup %{conn: conn, user: user} do + project = + insert(:project, project_users: [%{user_id: user.id, role: :owner}]) + + workflow = + insert(:simple_workflow, project: project) + |> Lightning.Repo.preload([:jobs, :triggers, :edges]) + + {:ok, project} = Lightning.Projects.schedule_project_deletion(project) + + [trigger] = workflow.triggers + refute Lightning.Repo.reload!(trigger).enabled + + repo_connection = insert(:project_repo_connection, project: project) + + %{ + project: project, + workflow: workflow, + trigger: trigger, + callers: [ + member: conn, + repo_connection: put_bearer(build_conn(), repo_connection.access_token) + ] + } + end + + # A refusal that still wrote would turn the triggers back on, which project + # shutdown had disabled on purpose. + test "a member and a repo connection token are refused on every route, and the trigger stays disabled", + %{ + callers: callers, + project: project, + workflow: workflow, + trigger: trigger + } do + requests = [ + fn conn -> get(conn, read_path(:state, project)) end, + fn conn -> get(conn, read_path(:yaml, project)) end, + fn conn -> + post( + conn, + ~p"/api/provision", + reenable_triggers_payload(project, workflow) + ) + end + ] + + for {caller, conn} <- callers, request <- requests do + conn |> request.() |> assert_problem(409, :scheduled_for_deletion, []) + + assert Lightning.Repo.reload!(trigger).enabled == false, + "the #{caller} re-enabled a trigger that project shutdown had disabled" + end + end + + test "a non-member is told nothing about the deletion", %{project: project} do + %{conn: outsider} = assign_bearer_for_api(%{conn: build_conn()}) + + for path <- [:state, :yaml] do + outsider + |> get(read_path(path, project)) + |> assert_problem(404, :not_found, []) + end + end + end + + defp read_path(:state, project), do: ~p"/api/provision/#{project.id}" + + defp read_path(:yaml, project), + do: ~p"/api/provision/yaml?#{%{id: project.id}}" +end diff --git a/test/lightning_web/controllers/api/provisioning/snapshots_test.exs b/test/lightning_web/controllers/api/provisioning/snapshots_test.exs new file mode 100644 index 00000000000..14543e12b7e --- /dev/null +++ b/test/lightning_web/controllers/api/provisioning/snapshots_test.exs @@ -0,0 +1,194 @@ +defmodule LightningWeb.API.ProvisioningSnapshotsTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProvisioningHelpers + import LightningWeb.API.ProblemHelpers + import Lightning.Factories + + alias Lightning.Workflows.Snapshot + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + describe "exporting only the snapshots asked for" do + setup [:assign_bearer_for_api] + + test "returns a project only with the specified snapshots", %{ + conn: conn, + user: user + } do + %{id: project_id, name: project_name} = + project = + insert(:project, + project_users: [%{user_id: user.id}] + ) + + workflow_1 = + insert(:simple_workflow, + project: project, + name: "workflow 1" + ) + + {:ok, snapshot_1} = Snapshot.create(workflow_1) + + {:ok, updated_workflow_1} = + workflow_1 + |> Ecto.Changeset.change(%{name: "updated-workflow-name"}) + |> Lightning.Repo.update() + + workflow_2 = + insert(:simple_workflow, + project: project, + name: "workflow 2" + ) + + {:ok, snapshot_2} = Snapshot.create(workflow_2) + + conn = + get(conn, ~p"/api/provision/#{project_id}", snapshots: [snapshot_1.id]) + + response = json_response(conn, 200) + + assert %{ + "id" => ^project_id, + "name" => ^project_name, + "workflows" => [workflow_resp] + } = response["data"] + + # Only the first workflow is returned because its snapshot was specified + assert workflow_resp["id"] == workflow_1.id + # The name of the workflow is the original name, not the updated name + assert workflow_resp["name"] == workflow_1.name + assert updated_workflow_1.name != workflow_1.name + + # Now we specify both snapshots + conn = + get(conn, ~p"/api/provision/#{project_id}", + snapshots: [snapshot_1.id, snapshot_2.id] + ) + + response = json_response(conn, 200) + + assert %{"workflows" => workflows} = response["data"] + assert Enum.count(workflows) == 2 + end + + test "does not return snapshots from another project", %{ + conn: conn, + user: user + } do + %{id: project_id} = + insert(:project, project_users: [%{user_id: user.id}]) + + # A snapshot in a different project the user has no access to. + foreign_workflow = insert(:simple_workflow, project: insert(:project)) + {:ok, foreign_snapshot} = Snapshot.create(foreign_workflow) + + conn = + get(conn, ~p"/api/provision/#{project_id}", + snapshots: [foreign_snapshot.id] + ) + + # The foreign snapshot's workflow is not disclosed. + assert %{"workflows" => []} = json_response(conn, 200)["data"] + end + + test "returns valid project yaml for snapshots provided" do + project = insert(:project) + repo_connection = insert(:project_repo_connection, project: project) + + workflow_1 = + insert(:simple_workflow, + project: project, + name: "workflow 1" + ) + + {:ok, snapshot_1} = Snapshot.create(workflow_1) + + {:ok, updated_workflow_1} = + workflow_1 + |> Ecto.Changeset.change(%{name: "updated-workflow-name"}) + |> Lightning.Repo.update() + + workflow_2 = + insert(:simple_workflow, + project: project, + name: "workflow 2" + ) + + {:ok, snapshot_2} = Snapshot.create(workflow_2) + + conn = + put_bearer(build_conn(), repo_connection.access_token) + + response = + get( + conn, + ~p"/api/provision/yaml?#{%{id: project.id, snapshots: [snapshot_1.id]}}" + ) + |> response(200) + + assert response =~ workflow_1.name + refute response =~ updated_workflow_1.name + refute response =~ workflow_2.name + + response = + get( + conn, + ~p"/api/provision/yaml?#{%{id: project.id, snapshots: [snapshot_1.id, snapshot_2.id]}}" + ) + |> response(200) + + assert response =~ workflow_1.name + refute response =~ updated_workflow_1.name + assert response =~ workflow_2.name + end + + test "does not include another project's snapshots in the yaml" do + project = insert(:project) + repo_connection = insert(:project_repo_connection, project: project) + + foreign_workflow = + insert(:simple_workflow, + project: insert(:project), + name: "foreign-workflow-name" + ) + + {:ok, foreign_snapshot} = Snapshot.create(foreign_workflow) + + conn = + put_bearer(build_conn(), repo_connection.access_token) + + response = + get( + conn, + ~p"/api/provision/yaml?#{%{id: project.id, snapshots: [foreign_snapshot.id]}}" + ) + |> response(200) + + refute response =~ "foreign-workflow-name" + end + + test "answers 400 invalid_parameter for snapshots that aren't a list of UUIDs, on both routes", + %{conn: conn, user: user} do + project = insert(:project, project_users: [%{user_id: user.id}]) + + for path <- [ + ~p"/api/provision/#{project.id}", + ~p"/api/provision/yaml?id=#{project.id}" + ], + snapshots <- ["true", ["not-a-uuid"]] do + conn + |> get(path, snapshots: snapshots) + |> assert_problem(400, :invalid_parameter, [ + %{ + "parameter" => "snapshots", + "code" => "type", + "detail" => "is invalid" + } + ]) + end + end + end +end diff --git a/test/lightning_web/controllers/api/provisioning/stale_writes_test.exs b/test/lightning_web/controllers/api/provisioning/stale_writes_test.exs new file mode 100644 index 00000000000..77d57bb92cc --- /dev/null +++ b/test/lightning_web/controllers/api/provisioning/stale_writes_test.exs @@ -0,0 +1,65 @@ +defmodule LightningWeb.API.ProvisioningStaleWritesTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProvisioningHelpers + import LightningWeb.API.ProblemHelpers + import Ecto.Query + import Lightning.Factories + + alias Lightning.Workflows.Workflow + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + describe "a stale write" do + setup [:assign_bearer_for_api] + + setup %{user: user} do + %{ + project: + insert(:project, project_users: [%{user_id: user.id, role: :owner}]) + } + end + + test "a workflow another request changes mid-import answers 409 concurrent_edit and keeps that request's change", + %{conn: conn, project: project} do + %{body: body, workflow_id: workflow_id, trigger_id: trigger_id} = + valid_payload(project.id) + + conn |> post(~p"/api/provision", body) |> json_response(201) + before = Lightning.Repo.get!(Workflow, workflow_id) + + # Enabling the trigger asks the usage limiter, which runs after the + # document is read and before it is written. The workflows are read by + # parallel preload tasks, so no query in the request's own process falls + # between the read and the write. + Mox.expect( + Lightning.Extensions.MockUsageLimiter, + :limit_action, + fn %{type: :activate_workflow}, _context -> + from(w in Workflow, where: w.id == ^workflow_id) + |> Lightning.Repo.update_all(inc: [lock_version: 1]) + + :ok + end + ) + + body = + put_in( + body, + ["workflows", Access.at(0), "triggers", Access.at(0), "enabled"], + true + ) + + conn + |> post(~p"/api/provision", body) + |> assert_problem(409, :concurrent_edit, []) + + assert Lightning.Repo.get!(Workflow, workflow_id).lock_version == + before.lock_version + 1 + + refute Lightning.Repo.get!(Lightning.Workflows.Trigger, trigger_id).enabled + end + end +end diff --git a/test/lightning_web/controllers/api/provisioning/state_export_test.exs b/test/lightning_web/controllers/api/provisioning/state_export_test.exs new file mode 100644 index 00000000000..a6d5b918d56 --- /dev/null +++ b/test/lightning_web/controllers/api/provisioning/state_export_test.exs @@ -0,0 +1,505 @@ +defmodule LightningWeb.API.ProvisioningStateExportTest do + use LightningWeb.ConnCase, async: true + + import Lightning.Factories + + alias Lightning.Workflows.Workflow + alias LightningWeb.API.ProvisioningJSON + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + describe "exporting a project" do + setup [:assign_bearer_for_api] + + test "returns a new empty project", %{ + conn: conn, + user: user + } do + %{id: project_id, name: project_name} = + insert(:project, + project_users: [%{user_id: user.id}] + ) + + conn = get(conn, ~p"/api/provision/#{project_id}") + response = json_response(conn, 200) + + assert %{ + "id" => ^project_id, + "name" => ^project_name, + "workflows" => [] + } = response["data"] + end + + test "returns a project with collections", %{ + conn: conn, + user: user + } do + %{id: project_id, name: project_name} = + project = + insert(:project, + project_users: [%{user_id: user.id}] + ) + + %{id: collection_id, name: collection_name} = + insert(:collection, project: project) + + conn = get(conn, ~p"/api/provision/#{project_id}") + response = json_response(conn, 200) + + assert %{ + "id" => ^project_id, + "name" => ^project_name, + "workflows" => [], + "collections" => collections_resp + } = response["data"] + + assert collections_resp == [ + %{"id" => collection_id, "name" => collection_name} + ] + end + + test "returns a project with channels", %{ + conn: conn, + user: user + } do + %{id: project_id} = + project = + insert(:project, + project_users: [%{user_id: user.id}] + ) + + project_credential = + insert(:project_credential, + credential: %{name: "dest-cred", body: %{}, user_id: user.id}, + project: project + ) + + %{id: channel_with_cred_id} = + channel_with_cred = + insert(:channel, + project: project, + name: "with-cred", + destination_url: "https://example.com/a", + enabled: true + ) + + insert(:channel_auth_method, + channel: channel_with_cred, + role: :destination, + webhook_auth_method: nil, + project_credential: project_credential + ) + + %{id: channel_without_cred_id} = + insert(:channel, + project: project, + name: "without-cred", + destination_url: "https://example.com/b", + enabled: false + ) + + conn = get(conn, ~p"/api/provision/#{project_id}") + response = json_response(conn, 200) + + assert %{"channels" => channels_resp} = response["data"] + + expected_pc_id = project_credential.id + + assert [ + %{ + "id" => ^channel_with_cred_id, + "name" => "with-cred", + "destination_url" => "https://example.com/a", + "enabled" => true, + "destination_credential_id" => ^expected_pc_id + }, + %{ + "id" => ^channel_without_cred_id, + "name" => "without-cred", + "destination_url" => "https://example.com/b", + "enabled" => false, + "destination_credential_id" => nil + } + ] = Enum.sort_by(channels_resp, & &1["name"]) + end + + test "returns a non empty project without credentials", %{ + conn: conn, + user: user + } do + %{id: project_id, name: project_name} = + project = + insert(:project, + project_users: [%{user_id: user.id}] + ) + + project_credential = + insert(:project_credential, + credential: %{ + name: "test credential", + body: %{"username" => "quux", "password" => "immasecret"}, + user_id: user.id + }, + project: project + ) + + %{ + triggers: [%{id: trigger_id}], + edges: [%{id: edge_1_id} = edge_1], + jobs: [%{id: job_1_id} = job_1] + } = + workflow = + insert(:simple_workflow, project: project, name: "Workflow123") + + %{id: job_2_id} = + job_2 = + insert(:job, + workflow: workflow, + name: "Second Step", + adaptor: "@openfn/language-http@latest", + body: "fn(state => state.references)", + workflow: workflow, + project_credential: project_credential + ) + + %{id: edge_2_id} = + edge_2 = + insert(:edge, + workflow: workflow, + source_job: job_1, + target_job: job_2, + condition_type: :js_expression, + condition_label: "sick", + condition_expression: "data.illness === true" + ) + + %{ + "edges" => [edge_1_json, edge_2_json], + "jobs" => [_job_1, job_2_json], + "triggers" => [trigger_json] + } = + workflow_json = + workflow + |> Map.merge(%{jobs: [job_1, job_2], edges: [edge_1, edge_2]}) + |> ProvisioningJSON.as_json() + |> Jason.encode!() + |> Jason.decode!() + + assert %{ + "id" => ^edge_1_id, + "condition_type" => "always", + "source_trigger_id" => ^trigger_id, + "target_job_id" => ^job_1_id, + "enabled" => true + } = + edge_1_json + + refute Map.has_key?(edge_1_json, "source_job_id") + refute Map.has_key?(edge_1_json, "condition_label") + refute Map.has_key?(edge_1_json, "condition_expression") + + assert %{ + "id" => ^edge_2_id, + "condition_type" => "js_expression", + "condition_label" => "sick", + "condition_expression" => "data.illness === true", + "source_job_id" => ^job_1_id, + "target_job_id" => ^job_2_id, + "enabled" => true + } = + edge_2_json + + assert Map.has_key?(job_2_json, "project_credential_id") + + assert %{ + "id" => ^job_2_id, + "name" => "Second Step", + "adaptor" => "@openfn/language-http@latest", + "body" => "fn(state => state.references)" + } = job_2_json + + assert %{ + "id" => ^trigger_id, + "type" => "webhook", + "enabled" => true + } = trigger_json + + conn = get(conn, ~p"/api/provision/#{project_id}") + response = json_response(conn, 200) + + expected_hash = + workflow + |> Repo.preload([:jobs, :edges, :triggers], force: true) + |> Lightning.WorkflowVersions.generate_hash() + + workflow_json = %{ + workflow_json + | "version_history" => ["app:#{expected_hash}"] + } + + assert %{ + "id" => ^project_id, + "name" => ^project_name, + "workflows" => [^workflow_json] + } = response["data"] + end + + test "returns a non empty project without credentials for support user", %{ + conn: conn, + user: user + } do + _user = Repo.update!(Ecto.Changeset.change(user, %{support_user: true})) + + %{id: project_id, name: project_name} = + project = + insert(:project, + allow_support_access: true, + project_users: [%{user: build(:user), role: :owner}] + ) + + project_credential = + insert(:project_credential, + credential: %{ + name: "test credential", + body: %{"username" => "quux", "password" => "immasecret"}, + user_id: user.id + }, + project: project + ) + + %{ + triggers: [%{id: trigger_id}], + edges: [%{id: edge_1_id} = edge_1], + jobs: [%{id: job_1_id} = job_1] + } = + workflow = + insert(:simple_workflow, project: project, name: "Workflow123") + + %{id: job_2_id} = + job_2 = + insert(:job, + workflow: workflow, + name: "Second Step", + adaptor: "@openfn/language-http@latest", + body: "fn(state => state.references)", + workflow: workflow, + project_credential: project_credential + ) + + %{id: edge_2_id} = + edge_2 = + insert(:edge, + workflow: workflow, + source_job: job_1, + target_job: job_2, + condition_type: :js_expression, + condition_label: "sick", + condition_expression: "data.illness === true" + ) + + %{ + "edges" => [edge_1_json, edge_2_json], + "jobs" => [_job_1, job_2_json], + "triggers" => [trigger_json] + } = + workflow_json = + workflow + |> Map.merge(%{jobs: [job_1, job_2], edges: [edge_1, edge_2]}) + |> ProvisioningJSON.as_json() + |> Jason.encode!() + |> Jason.decode!() + + assert %{ + "id" => ^edge_1_id, + "condition_type" => "always", + "source_trigger_id" => ^trigger_id, + "target_job_id" => ^job_1_id, + "enabled" => true + } = + edge_1_json + + refute Map.has_key?(edge_1_json, "source_job_id") + refute Map.has_key?(edge_1_json, "condition_label") + refute Map.has_key?(edge_1_json, "condition_expression") + + assert %{ + "id" => ^edge_2_id, + "condition_type" => "js_expression", + "condition_label" => "sick", + "condition_expression" => "data.illness === true", + "source_job_id" => ^job_1_id, + "target_job_id" => ^job_2_id, + "enabled" => true + } = + edge_2_json + + assert Map.has_key?(job_2_json, "project_credential_id") + + assert %{ + "id" => ^job_2_id, + "name" => "Second Step", + "adaptor" => "@openfn/language-http@latest", + "body" => "fn(state => state.references)" + } = job_2_json + + assert %{ + "id" => ^trigger_id, + "type" => "webhook", + "enabled" => true + } = trigger_json + + conn = get(conn, ~p"/api/provision/#{project_id}") + response = json_response(conn, 200) + + expected_hash = + workflow + |> Repo.preload([:jobs, :edges, :triggers], force: true) + |> Lightning.WorkflowVersions.generate_hash() + + workflow_json = %{ + workflow_json + | "version_history" => ["app:#{expected_hash}"] + } + + assert %{ + "id" => ^project_id, + "name" => ^project_name, + "workflows" => [^workflow_json] + } = response["data"] + end + + test "returns a project without deleted workflows", %{ + conn: conn, + user: user + } do + %{id: project_id, name: project_name} = + project = + insert(:project, + project_users: [%{user_id: user.id}] + ) + + _deleted_workflow = + insert(:workflow, + project: project, + name: "Deleted workflow", + deleted_at: DateTime.utc_now() + ) + + existing_workflow = + insert(:workflow, + project: project, + name: "Existing workflow", + deleted_at: nil + ) + + conn = get(conn, ~p"/api/provision/#{project_id}") + response = json_response(conn, 200) + + assert %{ + "id" => ^project_id, + "name" => ^project_name, + "workflows" => [workflow_resp] + } = response["data"] + + assert workflow_resp["id"] == existing_workflow.id + end + + test "includes version_history in workflow response", %{ + conn: conn, + user: user + } do + %{id: project_id} = + project = + insert(:project, + project_users: [%{user_id: user.id}] + ) + + workflow = + insert(:simple_workflow, project: project, name: "Test Workflow") + + # Record some version history + {:ok, _v1} = + Lightning.WorkflowVersions.record_version( + workflow, + "aabbccddeeff", + "app" + ) + + {:ok, _v2} = + Lightning.WorkflowVersions.record_version( + workflow, + "112233445566", + "cli" + ) + + conn = get(conn, ~p"/api/provision/#{project_id}") + response = json_response(conn, 200) + + assert %{"workflows" => [workflow_json]} = response["data"] + assert workflow_json["id"] == workflow.id + + # Verify version_history is included and has the expected values + assert workflow_json["version_history"] == [ + "app:aabbccddeeff", + "cli:112233445566" + ] + end + + test "automatically ensures version history for workflows without versions", + %{ + conn: conn, + user: user + } do + %{id: project_id} = + project = + insert(:project, + project_users: [%{user_id: user.id}] + ) + + workflow = + insert(:simple_workflow, + project: project, + name: "Workflow Without Version" + ) + + # Verify workflow has no version history initially + initial_history = Lightning.WorkflowVersions.history_for(workflow) + assert initial_history == [] + + conn = get(conn, ~p"/api/provision/#{project_id}") + response = json_response(conn, 200) + + assert %{"workflows" => [workflow_json]} = response["data"] + assert workflow_json["id"] == workflow.id + + # Verify version was automatically created + workflow = Lightning.Repo.get!(Workflow, workflow.id) + version_history = Lightning.WorkflowVersions.history_for(workflow) + assert length(version_history) == 1 + assert [version] = version_history + assert String.starts_with?(version, "app:") + + # Verify version_history is included in response + assert workflow_json["version_history"] == version_history + end + + test "returns the project yaml to an owner's personal access token", %{ + conn: conn, + user: user + } do + project = + insert(:project, project_users: [%{user_id: user.id, role: :owner}]) + + workflow = insert(:simple_workflow, project: project) + + response = + conn + |> get(~p"/api/provision/yaml?#{%{id: project.id}}") + |> response(200) + + assert response =~ workflow.name + end + end +end diff --git a/test/lightning_web/controllers/api/provisioning/trigger_export_test.exs b/test/lightning_web/controllers/api/provisioning/trigger_export_test.exs new file mode 100644 index 00000000000..2ea0f1f1e04 --- /dev/null +++ b/test/lightning_web/controllers/api/provisioning/trigger_export_test.exs @@ -0,0 +1,209 @@ +defmodule LightningWeb.API.ProvisioningTriggerExportTest do + use LightningWeb.ConnCase, async: true + + import Ecto.Query + import Lightning.Factories + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + describe "exporting triggers" do + setup [:assign_bearer_for_api] + + test "returns a webhook trigger with webhook_reply in the response", %{ + conn: conn, + user: user + } do + project = insert(:project, project_users: [%{user_id: user.id}]) + + trigger = + build(:trigger, + type: :webhook, + webhook_reply: :after_completion + ) + + job = build(:job) + + %{triggers: [%{id: trigger_id}]} = + build(:workflow, project: project) + |> with_trigger(trigger) + |> with_job(job) + |> with_edge({trigger, job}, condition_type: :always) + |> insert() + + conn = get(conn, ~p"/api/provision/#{project.id}") + response = json_response(conn, 200) + + assert %{ + "workflows" => [ + %{ + "triggers" => [trigger_json] + } + ] + } = response["data"] + + assert %{ + "id" => ^trigger_id, + "type" => "webhook", + "enabled" => true, + "webhook_reply" => "after_completion" + } = trigger_json + end + + test "returns a webhook trigger with webhook_response_config in the response", + %{ + conn: conn, + user: user + } do + project = insert(:project, project_users: [%{user_id: user.id}]) + + trigger = + build(:trigger, + type: :webhook, + webhook_reply: :after_completion, + webhook_response_config: + build(:webhook_response_config, + success_code: 200, + error_code: 500 + ) + ) + + job = build(:job) + + %{triggers: [%{id: trigger_id}]} = + build(:workflow, project: project) + |> with_trigger(trigger) + |> with_job(job) + |> with_edge({trigger, job}, condition_type: :always) + |> insert() + + conn = get(conn, ~p"/api/provision/#{project.id}") + response = json_response(conn, 200) + + assert %{ + "workflows" => [ + %{ + "triggers" => [trigger_json] + } + ] + } = response["data"] + + assert %{ + "id" => ^trigger_id, + "type" => "webhook", + "webhook_reply" => "after_completion", + "webhook_response_config" => %{ + "success_code" => 200, + "error_code" => 500 + } + } = trigger_json + end + + test "omits webhook_response_config when it is nil", %{ + conn: conn, + user: user + } do + project = insert(:project, project_users: [%{user_id: user.id}]) + + trigger = build(:trigger, type: :webhook) + job = build(:job) + + build(:workflow, project: project) + |> with_trigger(trigger) + |> with_job(job) + |> with_edge({trigger, job}, condition_type: :always) + |> insert() + + conn = get(conn, ~p"/api/provision/#{project.id}") + response = json_response(conn, 200) + + assert %{ + "workflows" => [%{"triggers" => [trigger_json]}] + } = response["data"] + + refute Map.has_key?(trigger_json, "webhook_response_config") + end + + test "returns a cron trigger with cron_cursor_job_id in the response", %{ + conn: conn, + user: user + } do + project = insert(:project, project_users: [%{user_id: user.id}]) + + %{triggers: [trigger], jobs: [%{id: job_id}]} = + insert(:simple_workflow, project: project) + + trigger_id = trigger.id + + {:ok, _trigger} = + trigger + |> Ecto.Changeset.change( + type: :cron, + cron_expression: "0 * * * *", + cron_cursor_job_id: job_id + ) + |> Lightning.Repo.update() + + conn = get(conn, ~p"/api/provision/#{project.id}") + response = json_response(conn, 200) + + assert %{ + "workflows" => [ + %{ + "triggers" => [trigger_json] + } + ] + } = response["data"] + + assert %{ + "id" => ^trigger_id, + "type" => "cron", + "cron_expression" => "0 * * * *", + "cron_cursor_job_id" => ^job_id + } = trigger_json + end + + test "a custom path is exported only when the server would accept it", %{ + conn: conn, + user: user + } do + project = insert(:project, project_users: [%{user_id: user.id}]) + + cases = [ + {"facility-001", {:ok, "facility-001"}}, + {"some/legacy/path", :error}, + {nil, :error} + ] + + expected_by_trigger_id = + for {path, expected} <- cases, into: %{} do + trigger = + insert(:trigger, + type: :webhook, + workflow: insert(:workflow, project: project), + custom_path: nil + ) + + Lightning.Repo.update_all( + from(t in Lightning.Workflows.Trigger, where: t.id == ^trigger.id), + set: [custom_path: path] + ) + + {trigger.id, expected} + end + + conn = get(conn, ~p"/api/provision/#{project.id}") + + exported = + for %{"triggers" => triggers} <- + json_response(conn, 200)["data"]["workflows"], + trigger <- triggers, + into: %{} do + {trigger["id"], Map.fetch(trigger, "custom_path")} + end + + assert exported == expected_by_trigger_id + end + end +end diff --git a/test/lightning_web/controllers/api/provisioning_controller_test.exs b/test/lightning_web/controllers/api/provisioning_controller_test.exs deleted file mode 100644 index 20043924051..00000000000 --- a/test/lightning_web/controllers/api/provisioning_controller_test.exs +++ /dev/null @@ -1,1791 +0,0 @@ -defmodule LightningWeb.API.ProvisioningControllerTest do - use LightningWeb.ConnCase, async: true - - import Ecto.Query - import Lightning.Factories - - alias Lightning.Workflows.Snapshot - alias Lightning.Workflows.Workflow - alias LightningWeb.API.ProvisioningJSON - - setup %{conn: conn} do - {:ok, conn: put_req_header(conn, "accept", "application/json")} - end - - describe "without a token" do - test "get returns a 401", %{conn: conn} do - conn = get(conn, Routes.api_project_path(conn, :index)) - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - - test "post returns a 401", %{conn: conn} do - body = %{"id" => "abc123", "workflows" => [%{"name" => "default"}]} - - conn = post(conn, ~p"/api/provision", body) - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - end - - describe "get (with an API token)" do - setup [:assign_bearer_for_api] - - test "returns a new empty project", %{ - conn: conn, - user: user - } do - %{id: project_id, name: project_name} = - insert(:project, - project_users: [%{user_id: user.id}] - ) - - conn = get(conn, ~p"/api/provision/#{project_id}") - response = json_response(conn, 200) - - assert %{ - "id" => ^project_id, - "name" => ^project_name, - "workflows" => [] - } = response["data"] - end - - test "returns a project with collections", %{ - conn: conn, - user: user - } do - %{id: project_id, name: project_name} = - project = - insert(:project, - project_users: [%{user_id: user.id}] - ) - - %{id: collection_id, name: collection_name} = - insert(:collection, project: project) - - conn = get(conn, ~p"/api/provision/#{project_id}") - response = json_response(conn, 200) - - assert %{ - "id" => ^project_id, - "name" => ^project_name, - "workflows" => [], - "collections" => collections_resp - } = response["data"] - - assert collections_resp == [ - %{"id" => collection_id, "name" => collection_name} - ] - end - - test "returns a project with channels", %{ - conn: conn, - user: user - } do - %{id: project_id} = - project = - insert(:project, - project_users: [%{user_id: user.id}] - ) - - project_credential = - insert(:project_credential, - credential: %{name: "dest-cred", body: %{}, user_id: user.id}, - project: project - ) - - %{id: channel_with_cred_id} = - channel_with_cred = - insert(:channel, - project: project, - name: "with-cred", - destination_url: "https://example.com/a", - enabled: true - ) - - insert(:channel_auth_method, - channel: channel_with_cred, - role: :destination, - webhook_auth_method: nil, - project_credential: project_credential - ) - - %{id: channel_without_cred_id} = - insert(:channel, - project: project, - name: "without-cred", - destination_url: "https://example.com/b", - enabled: false - ) - - conn = get(conn, ~p"/api/provision/#{project_id}") - response = json_response(conn, 200) - - assert %{"channels" => channels_resp} = response["data"] - - expected_pc_id = project_credential.id - - assert [ - %{ - "id" => ^channel_with_cred_id, - "name" => "with-cred", - "destination_url" => "https://example.com/a", - "enabled" => true, - "destination_credential_id" => ^expected_pc_id - }, - %{ - "id" => ^channel_without_cred_id, - "name" => "without-cred", - "destination_url" => "https://example.com/b", - "enabled" => false, - "destination_credential_id" => nil - } - ] = channels_resp - end - - test "returns a non empty project without credentials", %{ - conn: conn, - user: user - } do - %{id: project_id, name: project_name} = - project = - insert(:project, - project_users: [%{user_id: user.id}] - ) - - project_credential = - insert(:project_credential, - credential: %{ - name: "test credential", - body: %{"username" => "quux", "password" => "immasecret"}, - user_id: user.id - }, - project: project - ) - - %{ - triggers: [%{id: trigger_id}], - edges: [%{id: edge_1_id} = edge_1], - jobs: [%{id: job_1_id} = job_1] - } = - workflow = - insert(:simple_workflow, project: project, name: "Workflow123") - - %{id: job_2_id} = - job_2 = - insert(:job, - workflow: workflow, - name: "Second Step", - adaptor: "@openfn/language-http@latest", - body: "fn(state => state.references)", - workflow: workflow, - project_credential: project_credential - ) - - %{id: edge_2_id} = - edge_2 = - insert(:edge, - workflow: workflow, - source_job: job_1, - target_job: job_2, - condition_type: :js_expression, - condition_label: "sick", - condition_expression: "data.illness === true" - ) - - %{ - "edges" => [edge_1_json, edge_2_json], - "jobs" => [_job_1, job_2_json], - "triggers" => [trigger_json] - } = - workflow_json = - workflow - |> Map.merge(%{jobs: [job_1, job_2], edges: [edge_1, edge_2]}) - |> ProvisioningJSON.as_json() - |> Jason.encode!() - |> Jason.decode!() - - assert %{ - "id" => ^edge_1_id, - "condition_type" => "always", - "source_trigger_id" => ^trigger_id, - "target_job_id" => ^job_1_id, - "enabled" => true - } = - edge_1_json - - refute Map.has_key?(edge_1_json, "source_job_id") - refute Map.has_key?(edge_1_json, "condition_label") - refute Map.has_key?(edge_1_json, "condition_expression") - - assert %{ - "id" => ^edge_2_id, - "condition_type" => "js_expression", - "condition_label" => "sick", - "condition_expression" => "data.illness === true", - "source_job_id" => ^job_1_id, - "target_job_id" => ^job_2_id, - "enabled" => true - } = - edge_2_json - - assert Map.has_key?(job_2_json, "project_credential_id") - - assert %{ - "id" => ^job_2_id, - "name" => "Second Step", - "adaptor" => "@openfn/language-http@latest", - "body" => "fn(state => state.references)" - } = job_2_json - - assert %{ - "id" => ^trigger_id, - "type" => "webhook", - "enabled" => true - } = trigger_json - - conn = get(conn, ~p"/api/provision/#{project_id}") - response = json_response(conn, 200) - - expected_hash = - workflow - |> Repo.preload([:jobs, :edges, :triggers], force: true) - |> Lightning.WorkflowVersions.generate_hash() - - workflow_json = %{ - workflow_json - | "version_history" => ["app:#{expected_hash}"] - } - - assert %{ - "id" => ^project_id, - "name" => ^project_name, - "workflows" => [^workflow_json] - } = response["data"] - end - - test "returns a non empty project without credentials for support user", %{ - conn: conn, - user: user - } do - _user = Repo.update!(Ecto.Changeset.change(user, %{support_user: true})) - - %{id: project_id, name: project_name} = - project = - insert(:project, - allow_support_access: true, - project_users: [%{user: build(:user), role: :owner}] - ) - - project_credential = - insert(:project_credential, - credential: %{ - name: "test credential", - body: %{"username" => "quux", "password" => "immasecret"}, - user_id: user.id - }, - project: project - ) - - %{ - triggers: [%{id: trigger_id}], - edges: [%{id: edge_1_id} = edge_1], - jobs: [%{id: job_1_id} = job_1] - } = - workflow = - insert(:simple_workflow, project: project, name: "Workflow123") - - %{id: job_2_id} = - job_2 = - insert(:job, - workflow: workflow, - name: "Second Step", - adaptor: "@openfn/language-http@latest", - body: "fn(state => state.references)", - workflow: workflow, - project_credential: project_credential - ) - - %{id: edge_2_id} = - edge_2 = - insert(:edge, - workflow: workflow, - source_job: job_1, - target_job: job_2, - condition_type: :js_expression, - condition_label: "sick", - condition_expression: "data.illness === true" - ) - - %{ - "edges" => [edge_1_json, edge_2_json], - "jobs" => [_job_1, job_2_json], - "triggers" => [trigger_json] - } = - workflow_json = - workflow - |> Map.merge(%{jobs: [job_1, job_2], edges: [edge_1, edge_2]}) - |> ProvisioningJSON.as_json() - |> Jason.encode!() - |> Jason.decode!() - - assert %{ - "id" => ^edge_1_id, - "condition_type" => "always", - "source_trigger_id" => ^trigger_id, - "target_job_id" => ^job_1_id, - "enabled" => true - } = - edge_1_json - - refute Map.has_key?(edge_1_json, "source_job_id") - refute Map.has_key?(edge_1_json, "condition_label") - refute Map.has_key?(edge_1_json, "condition_expression") - - assert %{ - "id" => ^edge_2_id, - "condition_type" => "js_expression", - "condition_label" => "sick", - "condition_expression" => "data.illness === true", - "source_job_id" => ^job_1_id, - "target_job_id" => ^job_2_id, - "enabled" => true - } = - edge_2_json - - assert Map.has_key?(job_2_json, "project_credential_id") - - assert %{ - "id" => ^job_2_id, - "name" => "Second Step", - "adaptor" => "@openfn/language-http@latest", - "body" => "fn(state => state.references)" - } = job_2_json - - assert %{ - "id" => ^trigger_id, - "type" => "webhook", - "enabled" => true - } = trigger_json - - conn = get(conn, ~p"/api/provision/#{project_id}") - response = json_response(conn, 200) - - expected_hash = - workflow - |> Repo.preload([:jobs, :edges, :triggers], force: true) - |> Lightning.WorkflowVersions.generate_hash() - - workflow_json = %{ - workflow_json - | "version_history" => ["app:#{expected_hash}"] - } - - assert %{ - "id" => ^project_id, - "name" => ^project_name, - "workflows" => [^workflow_json] - } = response["data"] - end - - test "returns a project without deleted workflows", %{ - conn: conn, - user: user - } do - %{id: project_id, name: project_name} = - project = - insert(:project, - project_users: [%{user_id: user.id}] - ) - - _deleted_workflow = - insert(:workflow, - project: project, - name: "Deleted workflow", - deleted_at: DateTime.utc_now() - ) - - existing_workflow = - insert(:workflow, - project: project, - name: "Existing workflow", - deleted_at: nil - ) - - conn = get(conn, ~p"/api/provision/#{project_id}") - response = json_response(conn, 200) - - assert %{ - "id" => ^project_id, - "name" => ^project_name, - "workflows" => [workflow_resp] - } = response["data"] - - assert workflow_resp["id"] == existing_workflow.id - end - - test "includes version_history in workflow response", %{ - conn: conn, - user: user - } do - %{id: project_id} = - project = - insert(:project, - project_users: [%{user_id: user.id}] - ) - - workflow = - insert(:simple_workflow, project: project, name: "Test Workflow") - - # Record some version history - {:ok, _v1} = - Lightning.WorkflowVersions.record_version( - workflow, - "aabbccddeeff", - "app" - ) - - {:ok, _v2} = - Lightning.WorkflowVersions.record_version( - workflow, - "112233445566", - "cli" - ) - - conn = get(conn, ~p"/api/provision/#{project_id}") - response = json_response(conn, 200) - - assert %{"workflows" => [workflow_json]} = response["data"] - assert workflow_json["id"] == workflow.id - - # Verify version_history is included and has the expected values - assert workflow_json["version_history"] == [ - "app:aabbccddeeff", - "cli:112233445566" - ] - end - - test "automatically ensures version history for workflows without versions", - %{ - conn: conn, - user: user - } do - %{id: project_id} = - project = - insert(:project, - project_users: [%{user_id: user.id}] - ) - - workflow = - insert(:simple_workflow, - project: project, - name: "Workflow Without Version" - ) - - # Verify workflow has no version history initially - initial_history = Lightning.WorkflowVersions.history_for(workflow) - assert initial_history == [] - - conn = get(conn, ~p"/api/provision/#{project_id}") - response = json_response(conn, 200) - - assert %{"workflows" => [workflow_json]} = response["data"] - assert workflow_json["id"] == workflow.id - - # Verify version was automatically created - workflow = Lightning.Repo.get!(Workflow, workflow.id) - version_history = Lightning.WorkflowVersions.history_for(workflow) - assert length(version_history) == 1 - assert [version] = version_history - assert String.starts_with?(version, "app:") - - # Verify version_history is included in response - assert workflow_json["version_history"] == version_history - end - - test "returns a project only with the specified snapshots", %{ - conn: conn, - user: user - } do - %{id: project_id, name: project_name} = - project = - insert(:project, - project_users: [%{user_id: user.id}] - ) - - workflow_1 = - insert(:simple_workflow, - project: project, - name: "workflow 1" - ) - - {:ok, snapshot_1} = Snapshot.create(workflow_1) - - {:ok, updated_workflow_1} = - workflow_1 - |> Ecto.Changeset.change(%{name: "updated-workflow-name"}) - |> Lightning.Repo.update() - - workflow_2 = - insert(:simple_workflow, - project: project, - name: "workflow 2" - ) - - {:ok, snapshot_2} = Snapshot.create(workflow_2) - - conn = - get(conn, ~p"/api/provision/#{project_id}", snapshots: [snapshot_1.id]) - - response = json_response(conn, 200) - - assert %{ - "id" => ^project_id, - "name" => ^project_name, - "workflows" => [workflow_resp] - } = response["data"] - - # Only the first workflow is returned because its snapshot was specified - assert workflow_resp["id"] == workflow_1.id - # The name of the workflow is the original name, not the updated name - assert workflow_resp["name"] == workflow_1.name - assert updated_workflow_1.name != workflow_1.name - - # Now we specify both snapshots - conn = - get(conn, ~p"/api/provision/#{project_id}", - snapshots: [snapshot_1.id, snapshot_2.id] - ) - - response = json_response(conn, 200) - - assert %{"workflows" => workflows} = response["data"] - assert Enum.count(workflows) == 2 - end - - test "does not return snapshots from another project", %{ - conn: conn, - user: user - } do - %{id: project_id} = - insert(:project, project_users: [%{user_id: user.id}]) - - # A snapshot in a different project the user has no access to. - foreign_workflow = insert(:simple_workflow, project: insert(:project)) - {:ok, foreign_snapshot} = Snapshot.create(foreign_workflow) - - conn = - get(conn, ~p"/api/provision/#{project_id}", - snapshots: [foreign_snapshot.id] - ) - - # The foreign snapshot's workflow is not disclosed. - assert %{"workflows" => []} = json_response(conn, 200)["data"] - end - - test "returns a webhook trigger with webhook_reply in the response", %{ - conn: conn, - user: user - } do - project = insert(:project, project_users: [%{user_id: user.id}]) - - trigger = - build(:trigger, - type: :webhook, - webhook_reply: :after_completion - ) - - job = build(:job) - - %{triggers: [%{id: trigger_id}]} = - build(:workflow, project: project) - |> with_trigger(trigger) - |> with_job(job) - |> with_edge({trigger, job}, condition_type: :always) - |> insert() - - conn = get(conn, ~p"/api/provision/#{project.id}") - response = json_response(conn, 200) - - assert %{ - "workflows" => [ - %{ - "triggers" => [trigger_json] - } - ] - } = response["data"] - - assert %{ - "id" => ^trigger_id, - "type" => "webhook", - "enabled" => true, - "webhook_reply" => "after_completion" - } = trigger_json - end - - test "returns a webhook trigger with webhook_response_config in the response", - %{ - conn: conn, - user: user - } do - project = insert(:project, project_users: [%{user_id: user.id}]) - - trigger = - build(:trigger, - type: :webhook, - webhook_reply: :after_completion, - webhook_response_config: - build(:webhook_response_config, - success_code: 200, - error_code: 500 - ) - ) - - job = build(:job) - - %{triggers: [%{id: trigger_id}]} = - build(:workflow, project: project) - |> with_trigger(trigger) - |> with_job(job) - |> with_edge({trigger, job}, condition_type: :always) - |> insert() - - conn = get(conn, ~p"/api/provision/#{project.id}") - response = json_response(conn, 200) - - assert %{ - "workflows" => [ - %{ - "triggers" => [trigger_json] - } - ] - } = response["data"] - - assert %{ - "id" => ^trigger_id, - "type" => "webhook", - "webhook_reply" => "after_completion", - "webhook_response_config" => %{ - "success_code" => 200, - "error_code" => 500 - } - } = trigger_json - end - - test "omits webhook_response_config when it is nil", %{ - conn: conn, - user: user - } do - project = insert(:project, project_users: [%{user_id: user.id}]) - - trigger = build(:trigger, type: :webhook) - job = build(:job) - - build(:workflow, project: project) - |> with_trigger(trigger) - |> with_job(job) - |> with_edge({trigger, job}, condition_type: :always) - |> insert() - - conn = get(conn, ~p"/api/provision/#{project.id}") - response = json_response(conn, 200) - - assert %{ - "workflows" => [%{"triggers" => [trigger_json]}] - } = response["data"] - - refute Map.has_key?(trigger_json, "webhook_response_config") - end - - test "returns a cron trigger with cron_cursor_job_id in the response", %{ - conn: conn, - user: user - } do - project = insert(:project, project_users: [%{user_id: user.id}]) - - %{triggers: [trigger], jobs: [%{id: job_id}]} = - insert(:simple_workflow, project: project) - - trigger_id = trigger.id - - {:ok, _trigger} = - trigger - |> Ecto.Changeset.change( - type: :cron, - cron_expression: "0 * * * *", - cron_cursor_job_id: job_id - ) - |> Lightning.Repo.update() - - conn = get(conn, ~p"/api/provision/#{project.id}") - response = json_response(conn, 200) - - assert %{ - "workflows" => [ - %{ - "triggers" => [trigger_json] - } - ] - } = response["data"] - - assert %{ - "id" => ^trigger_id, - "type" => "cron", - "cron_expression" => "0 * * * *", - "cron_cursor_job_id" => ^job_id - } = trigger_json - end - - test "returns a project if user has owner access", %{ - conn: conn, - user: user - } do - %{id: project_id, name: project_name} = - insert(:project, - project_users: [%{user_id: user.id, role: :owner}] - ) - - conn = get(conn, ~p"/api/provision/#{project_id}") - response = json_response(conn, 200) - - assert %{ - "id" => ^project_id, - "name" => ^project_name, - "workflows" => workflows - } = response["data"] - - assert workflows |> Enum.all?(&match?(%{"project_id" => ^project_id}, &1)) - end - - test "returns a 200 if user has admin access", %{ - conn: conn, - user: user - } do - project = - insert(:project, - project_users: [%{user_id: user.id, role: :admin}] - ) - - response = get(conn, ~p"/api/provision/#{project.id}") - assert response.status == 200 - end - - test "returns a 200 if user has editor access", %{ - conn: conn, - user: user - } do - project = - insert(:project, - project_users: [%{user_id: user.id, role: :editor}] - ) - - response = get(conn, ~p"/api/provision/#{project.id}") - assert response.status == 200 - end - - test "returns a 200 if user has viewer access", %{ - conn: conn, - user: user - } do - project = - insert(:project, - project_users: [%{user_id: user.id, role: :viewer}] - ) - - response = get(conn, ~p"/api/provision/#{project.id}") - assert response.status == 200 - end - - test "returns a 403 if user does not have access", %{ - conn: conn - } do - %{id: project_id} = insert(:project) - - conn = get(conn, ~p"/api/provision/#{project_id}") - response = json_response(conn, 403) - - assert response == %{"error" => "Forbidden"} - end - - test "returns a 200 if a valid repo conenction token is provided for the project state" do - project = insert(:project) - - repo_connection = insert(:project_repo_connection, project: project) - - conn = - Plug.Conn.put_req_header( - build_conn(), - "authorization", - "Bearer #{repo_connection.access_token}" - ) - - response = get(conn, ~p"/api/provision/#{project.id}") - assert response.status == 200 - end - - test "returns a 200 if a valid repo conenction token is provided for the project yaml" do - project = insert(:project) - - repo_connection = insert(:project_repo_connection, project: project) - - conn = - Plug.Conn.put_req_header( - build_conn(), - "authorization", - "Bearer #{repo_connection.access_token}" - ) - - response = get(conn, ~p"/api/provision/yaml?#{%{id: project.id}}") - assert response.status == 200 - end - - test "returns valid project yaml for snapshots provided" do - project = insert(:project) - repo_connection = insert(:project_repo_connection, project: project) - - workflow_1 = - insert(:simple_workflow, - project: project, - name: "workflow 1" - ) - - {:ok, snapshot_1} = Snapshot.create(workflow_1) - - {:ok, updated_workflow_1} = - workflow_1 - |> Ecto.Changeset.change(%{name: "updated-workflow-name"}) - |> Lightning.Repo.update() - - workflow_2 = - insert(:simple_workflow, - project: project, - name: "workflow 2" - ) - - {:ok, snapshot_2} = Snapshot.create(workflow_2) - - conn = - Plug.Conn.put_req_header( - build_conn(), - "authorization", - "Bearer #{repo_connection.access_token}" - ) - - response = - get( - conn, - ~p"/api/provision/yaml?#{%{id: project.id, snapshots: [snapshot_1.id]}}" - ) - |> response(200) - - assert response =~ workflow_1.name - refute response =~ updated_workflow_1.name - refute response =~ workflow_2.name - - response = - get( - conn, - ~p"/api/provision/yaml?#{%{id: project.id, snapshots: [snapshot_1.id, snapshot_2.id]}}" - ) - |> response(200) - - assert response =~ workflow_1.name - refute response =~ updated_workflow_1.name - assert response =~ workflow_2.name - end - - test "does not include another project's snapshots in the yaml" do - project = insert(:project) - repo_connection = insert(:project_repo_connection, project: project) - - foreign_workflow = - insert(:simple_workflow, - project: insert(:project), - name: "foreign-workflow-name" - ) - - {:ok, foreign_snapshot} = Snapshot.create(foreign_workflow) - - conn = - Plug.Conn.put_req_header( - build_conn(), - "authorization", - "Bearer #{repo_connection.access_token}" - ) - - response = - get( - conn, - ~p"/api/provision/yaml?#{%{id: project.id, snapshots: [foreign_snapshot.id]}}" - ) - |> response(200) - - refute response =~ "foreign-workflow-name" - end - - test "returns a 403 if an invalid repo conenction token is provided" do - project_1 = insert(:project) - project_2 = insert(:project) - - wrong_repo_connection = - insert(:project_repo_connection, project: project_2) - - conn = - Plug.Conn.put_req_header( - build_conn(), - "authorization", - "Bearer #{wrong_repo_connection.access_token}" - ) - - conn = get(conn, ~p"/api/provision/#{project_1.id}") - - response = json_response(conn, 403) - - assert response == %{"error" => "Forbidden"} - end - end - - describe "as_json/1 and a trigger's custom path" do - setup do - project = insert(:project) - workflow = insert(:workflow, project: project) - - trigger = - insert(:trigger, type: :webhook, workflow: workflow, custom_path: nil) - - %{trigger: trigger, workflow: workflow} - end - - defp rendered_trigger(trigger, path) do - Lightning.Repo.update_all( - from(t in Lightning.Workflows.Trigger, where: t.id == ^trigger.id), - set: [custom_path: path] - ) - - trigger - |> Lightning.Repo.reload!() - |> ProvisioningJSON.as_json() - end - - test "is rendered when it is one the server would accept", %{ - trigger: trigger - } do - assert %{custom_path: "facility-001"} = - rendered_trigger(trigger, "facility-001") - end - - test "is dropped when it predates the naming rules", %{trigger: trigger} do - # Deploying this document into another project would fail validation and - # take the whole import with it. - refute Map.has_key?( - rendered_trigger(trigger, "some/legacy/path"), - :custom_path - ) - end - - test "is absent when there is none", %{trigger: trigger} do - refute Map.has_key?(rendered_trigger(trigger, nil), :custom_path) - end - end - - describe "post (with an API token)" do - setup [:assign_bearer_for_api] - - test "is forbidden for a viewer", %{conn: conn, user: user} do - project = - insert(:project, - project_users: [%{user_id: user.id, role: :viewer}] - ) - - body = %{ - "id" => project.id, - "workflows" => [%{"name" => "default"}] - } - - response = post(conn, ~p"/api/provision", body) - assert response.status == 403 - end - - test "is forbidden for an editor", %{conn: conn, user: user} do - project = - insert(:project, - project_users: [%{user_id: user.id, role: :editor}] - ) - - body = %{ - "id" => project.id, - "workflows" => [%{"name" => "default"}] - } - - response = post(conn, ~p"/api/provision", body) - assert response.status == 403 - end - - test "is forbidden for an invalid PRC access_token", %{conn: conn} do - project = insert(:project) - wrong_project = insert(:project) - - wrong_repo_connection = - insert(:project_repo_connection, project: wrong_project) - - %{body: body} = valid_payload(project.id) - - conn = - Plug.Conn.put_req_header( - conn, - "authorization", - "Bearer #{wrong_repo_connection.access_token}" - ) - - response = post(conn, ~p"/api/provision", body) - assert response.status == 403 - end - - test "fails with 403 when repo_connection is limited by github_sync", %{ - conn: conn - } do - %{id: project_id} = project = insert(:project) - - repo_connection = - insert(:project_repo_connection, project: project) - - %{body: body} = valid_payload(project.id) - - conn = - Plug.Conn.put_req_header( - conn, - "authorization", - "Bearer #{repo_connection.access_token}" - ) - - error_text = "Upgrade to use Github Sync" - - Lightning.Extensions.MockUsageLimiter - |> Mox.expect(:limit_action, fn %{type: :github_sync}, - %{project_id: ^project_id} -> - {:error, :upgrade_required, - %Lightning.Extensions.Message{text: error_text}} - end) - - assert post(conn, ~p"/api/provision", body) |> json_response(403) == - %{"error" => error_text} - end - - test "fails with a 422 on validation errors", %{conn: conn, user: user} do - project = - insert(:project, - project_users: [%{user_id: user.id, role: :owner}] - ) - - body = %{ - "id" => project.id, - "name" => "", - "workflows" => [ - %{"name" => "default"}, - %{"id" => Ecto.UUID.generate(), "name" => "Valid Workflow"} - ] - } - - conn = post(conn, ~p"/api/provision", body) - response = json_response(conn, 422) - - assert response == %{ - "errors" => %{ - "name" => ["This field can't be blank."], - "workflows" => %{ - "default" => %{"id" => ["This field can't be blank."]} - } - } - } - - body = %{ - "id" => project.id, - "name" => "test-project", - "workflows" => [ - %{ - "name" => "default", - "jobs" => [ - %{ - "name" => "first-job", - "adaptor" => "@openfn/language-common@latest" - }, - %{ - "adaptor" => "@openfn/language-common@latest", - "body" => "console.log('hello world');" - }, - %{ - "id" => Ecto.UUID.generate(), - "name" => "valid job", - "adaptor" => "@openfn/language-common@latest", - "body" => "console.log('hello world');" - } - ] - } - ] - } - - conn = post(conn, ~p"/api/provision", body) - response = json_response(conn, 422) - - assert response == %{ - "errors" => %{ - "workflows" => %{ - "default" => %{ - "jobs" => %{ - "first-job" => %{ - "id" => ["This field can't be blank."], - "body" => ["Code editor cannot be empty."] - }, - "" => %{ - "name" => ["Job name can't be blank."], - "id" => ["This field can't be blank."] - } - }, - "id" => ["This field can't be blank."] - } - } - } - } - - job_id = Ecto.UUID.generate() - trigger_id = Ecto.UUID.generate() - - for trigger_type <- ["cron", "webhook"] do - body = %{ - "id" => project.id, - "name" => "test-project", - "workflows" => [ - %{ - "id" => Ecto.UUID.generate(), - "name" => "default", - "jobs" => [ - %{ - "id" => job_id, - "name" => "first-job", - "adaptor" => "@openfn/language-common@latest", - "body" => "console.log('hello world')" - } - ], - "triggers" => [ - %{"id" => trigger_id, "type" => trigger_type} - ], - "edges" => [ - %{ - "id" => Ecto.UUID.generate(), - "source_trigger_id" => trigger_id, - "target_job_id" => job_id - } - ] - } - ] - } - - conn = post(conn, ~p"/api/provision", body) - response = json_response(conn, 422) - - assert response == %{ - "errors" => %{ - "workflows" => %{ - "default" => %{ - "edges" => %{ - "#{trigger_type}->first-job" => %{ - "condition_type" => ["This field can't be blank."] - } - } - } - } - } - } - end - end - - test "allows an owner to update an existing project", %{ - conn: conn, - user: user - } do - project = - insert(:project, - project_users: [%{user_id: user.id, role: :owner}] - ) - - %{ - project_id: project_id, - body: body, - second_job_id: second_job_id - } = valid_payload(project.id) - - conn = post(conn, ~p"/api/provision", body) - response = json_response(conn, 201) - - assert %{ - "id" => ^project_id, - "name" => "test-project", - "workflows" => [_] - } = response["data"] - - # - - - - third_job_id = Ecto.UUID.generate() - - body = - body - |> Map.put("name", "test-project-renamed") - |> add_job_to_document(%{ - "id" => third_job_id, - "name" => "third-job", - "adaptor" => "@openfn/language-common@latest", - "body" => "console.log('hello world');" - }) - - conn = post(conn, ~p"/api/provision", body) - response = json_response(conn, 201) - - assert %{ - "id" => ^project_id, - "name" => "test-project-renamed", - "workflows" => workflows - } = response["data"] - - workflow_job_ids = - workflows |> Enum.at(0) |> Map.get("jobs") |> Enum.into([], & &1["id"]) - - assert third_job_id in workflow_job_ids - - body = body |> remove_job_from_document(second_job_id) - - conn = post(conn, ~p"/api/provision", body) - response = json_response(conn, 201) - - assert %{ - "id" => ^project_id, - "name" => "test-project-renamed", - "workflows" => workflows - } = response["data"] - - workflow_job_ids = - workflows |> Enum.at(0) |> Map.get("jobs") |> Enum.into([], & &1["id"]) - - refute second_job_id in workflow_job_ids - assert third_job_id in workflow_job_ids - - assert workflows |> Enum.at(0) |> Map.get("edges") == [], - "The edge associated with the deleted job should be removed" - end - - test "allows an admin to update an existing project", %{ - conn: conn, - user: user - } do - project = - insert(:project, - project_users: [%{user_id: user.id, role: :admin}] - ) - - %{body: body} = valid_payload(project.id) - - assert post(conn, ~p"/api/provision", body) |> json_response(201) - end - - test "allows a valid PRC token to update an existing project", %{ - conn: conn - } do - project = insert(:project) - - repo_connection = - insert(:project_repo_connection, project: project) - - %{body: body} = valid_payload(project.id) - - conn = - Plug.Conn.put_req_header( - conn, - "authorization", - "Bearer #{repo_connection.access_token}" - ) - - assert post(conn, ~p"/api/provision", body) |> json_response(201) - end - - test "records workflow version with 'cli' source when workflow is created via provisioner", - %{conn: conn, user: user} do - project = - insert(:project, - project_users: [%{user_id: user.id, role: :owner}] - ) - - %{body: body, workflow_id: workflow_id} = valid_payload(project.id) - - conn = post(conn, ~p"/api/provision", body) - assert json_response(conn, 201) - - workflow = Lightning.Repo.get!(Workflow, workflow_id) - version_history = Lightning.WorkflowVersions.history_for(workflow) - - assert length(version_history) == 1 - assert [version] = version_history - assert String.starts_with?(version, "cli:") - end - - test "records new workflow version when workflow is updated via provisioner", - %{conn: conn, user: user} do - project = - insert(:project, - project_users: [%{user_id: user.id, role: :owner}] - ) - - # First provision - creates workflow - %{body: body, workflow_id: workflow_id} = valid_payload(project.id) - - conn = post(conn, ~p"/api/provision", body) - assert json_response(conn, 201) - - workflow = Lightning.Repo.get!(Workflow, workflow_id) - initial_history = Lightning.WorkflowVersions.history_for(workflow) - assert length(initial_history) == 1 - assert [initial_version] = initial_history - assert String.starts_with?(initial_version, "cli:") - - # Update workflow by changing job body - updated_body = - body - |> Map.update!("workflows", fn workflows -> - Enum.at(workflows, 0) - |> Map.update!("jobs", fn jobs -> - Enum.map(jobs, fn job -> - Map.put(job, "body", "console.log('updated');") - end) - end) - |> then(fn workflow -> - List.replace_at(workflows, 0, workflow) - end) - end) - - conn = post(conn, ~p"/api/provision", updated_body) - assert json_response(conn, 201) - - workflow = Lightning.Repo.get!(Workflow, workflow_id) - updated_history = Lightning.WorkflowVersions.history_for(workflow) - - # we nolonger squash the first version - assert length(updated_history) == 2 - assert [^initial_version, latest_version] = updated_history - assert String.starts_with?(latest_version, "cli:") - - # Verify the hash changed after the update - refute initial_version == latest_version - end - - test "does not create duplicate version when provisioning same workflow content", - %{conn: conn, user: user} do - project = - insert(:project, - project_users: [%{user_id: user.id, role: :owner}] - ) - - # First provision - creates workflow - %{body: body, workflow_id: workflow_id} = valid_payload(project.id) - - conn = post(conn, ~p"/api/provision", body) - assert json_response(conn, 201) - - workflow = Lightning.Repo.get!(Workflow, workflow_id) - initial_history = Lightning.WorkflowVersions.history_for(workflow) - assert length(initial_history) == 1 - - # Provision again with the exact same content - conn = post(conn, ~p"/api/provision", body) - assert json_response(conn, 201) - - workflow = Lightning.Repo.get!(Workflow, workflow_id) - duplicate_history = Lightning.WorkflowVersions.history_for(workflow) - - # Should still have only 1 version since content didn't change - assert length(duplicate_history) == 1 - assert duplicate_history == initial_history - end - - test "handles provisioning with no workflow changes gracefully", - %{conn: conn, user: user} do - project = - insert(:project, - project_users: [%{user_id: user.id, role: :owner}] - ) - - # Create a project with a workflow that has an initial version - %{body: body, workflow_id: workflow_id} = valid_payload(project.id) - - # First provision to create the workflow - conn = post(conn, ~p"/api/provision", body) - assert json_response(conn, 201) - - # Get the workflow and its initial version history - workflow = Lightning.Repo.get!(Workflow, workflow_id) - initial_history = Lightning.WorkflowVersions.history_for(workflow) - assert length(initial_history) == 1 - - # Update only project metadata, keeping workflow unchanged - updated_body = Map.put(body, "name", "updated-project-name") - - conn = post(conn, ~p"/api/provision", updated_body) - assert json_response(conn, 201) - - # Verify version history unchanged since workflow content didn't change - workflow = Lightning.Repo.get!(Workflow, workflow_id) - version_history = Lightning.WorkflowVersions.history_for(workflow) - assert version_history == initial_history - - # Verify project name was updated - project = Lightning.Repo.get!(Lightning.Projects.Project, project.id) - assert project.name == "updated-project-name" - end - - test "returns 201 for an existing project with workflows marked for deletion", - %{ - conn: conn - } do - project = insert(:project) - - _deleted_workflow = - insert(:workflow, - project: project, - name: "Deleted workflow", - deleted_at: DateTime.utc_now() - ) - - repo_connection = - insert(:project_repo_connection, project: project) - - %{body: body} = valid_payload(project.id) - - conn = - Plug.Conn.put_req_header( - conn, - "authorization", - "Bearer #{repo_connection.access_token}" - ) - - assert post(conn, ~p"/api/provision", body) |> json_response(201) - end - - @tag login_as: "superuser" - test "allows a superuser to create a new project", %{conn: conn} do - %{ - body: body, - project_id: project_id, - first_job_id: first_job_id, - second_job_id: second_job_id, - trigger_id: trigger_id, - workflow_id: workflow_id, - job_edge_id: job_edge_id - } = valid_payload() - - conn = post(conn, ~p"/api/provision", body) - response = json_response(conn, 201) - - assert %{ - "id" => ^project_id, - "name" => "test-project", - "workflows" => workflows - } = response["data"] - - project = Lightning.Projects.get_project!(project_id) - - assert project.name == "test-project" - - workflow_ids = Enum.map(workflows, fn w -> w["id"] end) - - parent_project_ids = - from(w in Workflow, - where: w.id in ^workflow_ids - ) - |> Repo.all() - |> Enum.map(fn w -> w.project_id end) - - assert parent_project_ids |> Enum.all?(&match?(^project_id, &1)), - "All workflows should belong to the same project" - - workflow = - from(w in Workflow, - preload: [:jobs, :triggers, :edges], - where: w.id == ^workflow_id - ) - |> Lightning.Repo.one!() - - assert workflow.name == "default" - assert workflow.edges |> MapSet.new(& &1.id) == MapSet.new([job_edge_id]) - - assert workflow.jobs |> MapSet.new(& &1.id) == - MapSet.new([first_job_id, second_job_id]) - - assert workflow.triggers |> MapSet.new(& &1.id) == MapSet.new([trigger_id]) - end - - test "doesn't let a normal user create a new project", %{conn: conn} do - %{body: body} = valid_payload() - - response = post(conn, ~p"/api/provision", body) - assert response.status == 403 - end - end - - # `Policies.Provisioning` never mentions - # `scheduled_deletion`. Its `%User{}` clauses decide on - # `Projects.get_project_user_role/2` alone, and its - # `%ProjectRepoConnection{}` clauses check only - # `repo_connection.project_id == project.id`, which is trivially true. - # - # Scheduling deletion removes no membership row and revokes no repo-connection - # token, so both principals below still exist during the grace window. - describe "a project scheduled for deletion" do - setup [:assign_bearer_for_api] - - setup %{conn: conn, user: user} do - project = - insert(:project, project_users: [%{user_id: user.id, role: :owner}]) - - workflow = - insert(:simple_workflow, project: project) - |> Lightning.Repo.preload([:jobs, :triggers, :edges]) - - # The real offboarding path: stamps `scheduled_deletion` AND disables - # every trigger in the project. - {:ok, project} = Lightning.Projects.schedule_project_deletion(project) - - [trigger] = workflow.triggers - refute Lightning.Repo.reload!(trigger).enabled - - repo_connection = insert(:project_repo_connection, project: project) - - %{ - conn: conn, - project: project, - workflow: workflow, - trigger: trigger, - prc_conn: - Plug.Conn.put_req_header( - build_conn(), - "authorization", - "Bearer #{repo_connection.access_token}" - ) - } - end - - test "refuses GET /api/provision/:id for a project member", %{ - conn: conn, - project: project, - workflow: workflow - } do - response = get(conn, ~p"/api/provision/#{project.id}") - - assert response.status == 403, - "a project member read the shut-down project's state" - - refute response.resp_body =~ workflow.name - end - - test "refuses GET /api/provision/:id for a repo-connection token", %{ - prc_conn: prc_conn, - project: project, - workflow: workflow - } do - response = get(prc_conn, ~p"/api/provision/#{project.id}") - - assert response.status == 403, - "a repo-connection token read the shut-down project's state" - - refute response.resp_body =~ workflow.name - end - - test "refuses GET /api/provision/yaml for a project member", %{ - conn: conn, - project: project, - workflow: workflow - } do - response = get(conn, ~p"/api/provision/yaml?#{%{id: project.id}}") - - assert response.status == 403, - "a project member exported the shut-down project as YAML" - - refute response.resp_body =~ workflow.name - end - - test "refuses GET /api/provision/yaml for a repo-connection token", %{ - prc_conn: prc_conn, - project: project, - workflow: workflow - } do - response = get(prc_conn, ~p"/api/provision/yaml?#{%{id: project.id}}") - - assert response.status == 403, - "a repo-connection token exported the shut-down project as YAML" - - refute response.resp_body =~ workflow.name - end - - # The status code is the lesser assertion. The damage in #287 is that the - # import turns the triggers back on, so a fix that returns 403 and still - # writes must fail here too. - test "refuses POST /api/provision for a project member, leaving the trigger disabled", - %{conn: conn, project: project, workflow: workflow, trigger: trigger} do - response = - post( - conn, - ~p"/api/provision", - reenable_triggers_payload(project, workflow) - ) - - assert Lightning.Repo.reload!(trigger).enabled == false, - "the import re-enabled a trigger that project shutdown had disabled" - - assert response.status == 403, - "a project member wrote to the shut-down project" - end - - test "refuses POST /api/provision for a repo-connection token, leaving the trigger disabled", - %{ - prc_conn: prc_conn, - project: project, - workflow: workflow, - trigger: trigger - } do - response = - post( - prc_conn, - ~p"/api/provision", - reenable_triggers_payload(project, workflow) - ) - - assert Lightning.Repo.reload!(trigger).enabled == false, - "the import re-enabled a trigger that project shutdown had disabled" - - assert response.status == 403, - "a repo-connection token wrote to the shut-down project" - end - end - - # A document that names the project's existing workflow, job, trigger and - # edge, and asks for the trigger to be turned back on. - defp reenable_triggers_payload(project, workflow) do - [job] = workflow.jobs - [trigger] = workflow.triggers - [edge] = workflow.edges - - %{ - "id" => project.id, - "name" => project.name, - "workflows" => [ - %{ - "id" => workflow.id, - "name" => workflow.name, - "jobs" => [ - %{ - "id" => job.id, - "name" => job.name, - "adaptor" => job.adaptor, - "body" => job.body - } - ], - "triggers" => [ - %{"id" => trigger.id, "type" => "webhook", "enabled" => true} - ], - "edges" => [ - %{ - "id" => edge.id, - "source_trigger_id" => trigger.id, - "target_job_id" => job.id, - "condition_type" => "always", - "enabled" => true - } - ] - } - ] - } - end - - defp valid_payload(project_id \\ nil) do - project_id = project_id || Ecto.UUID.generate() - first_job_id = Ecto.UUID.generate() - second_job_id = Ecto.UUID.generate() - trigger_id = Ecto.UUID.generate() - workflow_id = Ecto.UUID.generate() - job_edge_id = Ecto.UUID.generate() - - body = %{ - "id" => project_id, - "name" => "test-project", - "workflows" => [ - %{ - "id" => workflow_id, - "name" => "default", - "jobs" => [ - %{ - "id" => first_job_id, - "name" => "first-job", - "adaptor" => "@openfn/language-common@latest", - "body" => "console.log('hello world');" - }, - %{ - "id" => second_job_id, - "name" => "second-job", - "adaptor" => "@openfn/language-common@latest", - "body" => "console.log('hello world');" - } - ], - "triggers" => [ - %{ - "id" => trigger_id - } - ], - "edges" => [ - %{ - "id" => job_edge_id, - "source_job_id" => first_job_id, - "condition_type" => "on_job_success", - "target_job_id" => second_job_id - } - ] - } - ] - } - - %{ - body: body, - project_id: project_id, - workflow_id: workflow_id, - first_job_id: first_job_id, - second_job_id: second_job_id, - trigger_id: trigger_id, - job_edge_id: job_edge_id - } - end - - defp add_job_to_document(document, job_params) do - document - |> Map.update!("workflows", fn workflows -> - Enum.at(workflows, 0) - |> Map.update!("jobs", fn jobs -> - [job_params | jobs] - end) - |> then(fn workflow -> - List.replace_at(workflows, 0, workflow) - end) - end) - end - - defp remove_job_from_document(document, id) do - document - |> Map.update!("workflows", fn workflows -> - Enum.at(workflows, 0) - |> Map.update!("jobs", fn jobs -> - jobs - |> Enum.map(fn job -> - if job["id"] == id do - Map.put(job, "delete", true) - else - job - end - end) - end) - |> then(fn workflow -> - List.replace_at(workflows, 0, workflow) - end) - end) - end -end diff --git a/test/lightning_web/controllers/api/registration_controller_test.exs b/test/lightning_web/controllers/api/registration_controller_test.exs index 2bfbbb471d6..bd6c3001cea 100644 --- a/test/lightning_web/controllers/api/registration_controller_test.exs +++ b/test/lightning_web/controllers/api/registration_controller_test.exs @@ -1,16 +1,17 @@ defmodule LightningWeb.API.RegistrationControllerTest do use LightningWeb.ConnCase, async: true + import LightningWeb.API.ProblemHelpers import Mox setup %{conn: conn} do {:ok, conn: put_req_header(conn, "accept", "application/json")} end - test "create", %{conn: conn} do + test "create with no extension is 501 not_implemented", %{conn: conn} do conn = post(conn, ~p"/api/users/register") - assert %{"error" => "Not Implemented"} == json_response(conn, 501) + assert_problem(conn, 501, :not_implemented, []) end test "create with valid extension", %{conn: conn} do diff --git a/test/lightning_web/controllers/api/run_controller_index_test.exs b/test/lightning_web/controllers/api/run_controller_index_test.exs new file mode 100644 index 00000000000..ab657dc7e62 --- /dev/null +++ b/test/lightning_web/controllers/api/run_controller_index_test.exs @@ -0,0 +1,312 @@ +defmodule LightningWeb.API.RunControllerIndexTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProblemHelpers + import LightningWeb.ApiReadHelpers + import Lightning.Factories + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + defp ids(conn), + do: conn |> json_response(200) |> Map.fetch!("data") |> Enum.map(& &1["id"]) + + describe "index" do + setup [:assign_bearer_for_api, :create_project_for_current_user] + + test "lists the runs of the projects the caller can reach", %{ + conn: conn, + project: project + } do + older = + insert_run(project, + state: :started, + inserted_at: ~U[2024-01-01 10:00:00Z] + ) + + newer = + insert_run(project, + state: :success, + inserted_at: ~U[2024-01-01 12:00:00Z] + ) + + insert_run(insert(:project)) + + url = LightningWeb.Endpoint.url() + page = "#{url}/api/runs?page=1&page_size=10" + + assert conn |> get(~p"/api/runs") |> json_response(200) == %{ + "data" => + for run <- [newer, older] do + %{ + "type" => "runs", + "id" => run.id, + "relationships" => %{}, + "links" => %{"self" => "#{url}/api/runs/#{run.id}"}, + "attributes" => %{ + "state" => to_string(run.state), + "started_at" => nil, + "finished_at" => nil, + "priority" => "normal", + "error_type" => nil + } + } + end, + "included" => [], + "meta" => %{ + "total_entries" => 2, + "total_pages" => 1, + "page_number" => 1, + "page_size" => 10 + }, + "links" => %{ + "self" => page, + "first" => page, + "last" => page, + "next" => nil, + "prev" => nil + } + } + end + + test "narrows by inserted_after", %{conn: conn, project: project} do + insert_run(project, inserted_at: ~U[2024-01-01 10:00:00Z]) + new = insert_run(project, inserted_at: ~U[2024-01-01 12:00:00Z]) + + conn = get(conn, ~p"/api/runs?inserted_after=2024-01-01T11:00:00Z") + + assert ids(conn) == [new.id] + end + + test "narrows by inserted_before", %{conn: conn, project: project} do + old = insert_run(project, inserted_at: ~U[2024-01-01 10:00:00Z]) + insert_run(project, inserted_at: ~U[2024-01-01 12:00:00Z]) + + conn = get(conn, ~p"/api/runs?inserted_before=2024-01-01T11:00:00Z") + + assert ids(conn) == [old.id] + end + + test "narrows by project_id", %{conn: conn, user: user} do + project1 = insert(:project, project_users: [%{user: user, role: :owner}]) + project2 = insert(:project, project_users: [%{user: user, role: :owner}]) + run1 = insert_run(project1) + insert_run(project2) + + conn = get(conn, ~p"/api/runs?project_id=#{project1.id}") + + assert ids(conn) == [run1.id] + end + + test "narrows by workflow_id", %{conn: conn, project: project} do + run1 = insert_run(project) + insert_run(project) + + conn = get(conn, ~p"/api/runs?workflow_id=#{workflow_of(run1).id}") + + assert ids(conn) == [run1.id] + end + + test "narrows by work_order_id", %{conn: conn, project: project} do + run1 = insert_run(project) + insert_run(project) + + conn = get(conn, ~p"/api/runs?work_order_id=#{run1.work_order_id}") + + assert ids(conn) == [run1.id] + end + + test "narrows by one state, or by several comma-separated", %{ + conn: conn, + project: project + } do + started = insert_run(project, state: :started) + failed = insert_run(project, state: :failed) + insert_run(project, state: :success) + + assert ids(get(conn, ~p"/api/runs?state=started")) == [started.id] + + assert conn + |> get(~p"/api/runs?state=started,failed") + |> ids() + |> Enum.sort() == + Enum.sort([started.id, failed.id]) + end + + test "narrows by every filter at once", %{conn: conn, project: project} do + mid = + insert_run(project, + state: :started, + inserted_at: ~U[2024-01-01 12:00:00Z] + ) + + insert_run(project, state: :started, inserted_at: ~U[2024-01-01 10:00:00Z]) + + insert_run(project, state: :started, inserted_at: ~U[2024-01-01 14:00:00Z]) + + insert_run(project, state: :failed, inserted_at: ~U[2024-01-01 12:00:00Z]) + + conn = + get( + conn, + ~p"/api/runs?inserted_after=2024-01-01T11:00:00Z&inserted_before=2024-01-01T13:00:00Z&state=started" + ) + + assert ids(conn) == [mid.id] + end + + test "lists one project's runs on the nested route, narrowed by the same filters", + %{conn: conn, user: user} do + project1 = insert(:project, project_users: [%{user: user, role: :owner}]) + project2 = insert(:project, project_users: [%{user: user, role: :owner}]) + insert_run(project1, inserted_at: ~U[2024-01-01 10:00:00Z]) + new = insert_run(project1, inserted_at: ~U[2024-01-01 12:00:00Z]) + insert_run(project2, inserted_at: ~U[2024-01-01 12:00:00Z]) + + assert conn + |> get(~p"/api/projects/#{project1.id}/runs") + |> ids() + |> length() == 2 + + conn = + get( + conn, + ~p"/api/projects/#{project1.id}/runs?inserted_after=2024-01-01T11:00:00Z" + ) + + assert ids(conn) == [new.id] + end + + test "pages with exact links to the neighbouring pages", %{ + conn: conn, + project: project + } do + for n <- 1..25 do + insert_run(project, + inserted_at: DateTime.add(~U[2024-01-01 00:00:00Z], n) + ) + end + + url = "#{LightningWeb.Endpoint.url()}/api/runs" + + assert %{"data" => data, "meta" => meta, "links" => links} = + conn + |> get(~p"/api/runs?page=2&page_size=10") + |> json_response(200) + + assert length(data) == 10 + + assert meta == %{ + "total_entries" => 25, + "total_pages" => 3, + "page_number" => 2, + "page_size" => 10 + } + + assert links == %{ + "self" => "#{url}?page=2&page_size=10", + "first" => "#{url}?page=1&page_size=10", + "last" => "#{url}?page=3&page_size=10", + "next" => "#{url}?page=3&page_size=10", + "prev" => "#{url}?page=1&page_size=10" + } + end + + test "pagination links carry the filters applied and nothing else", %{ + conn: conn, + project: project + } do + for {path, request_path} <- [ + {~p"/api/runs?page_size=1&foo=bar&state=failed,crashed&inserted_after=2024-01-01T02:00:00%2B02:00", + "/api/runs"}, + {~p"/api/projects/#{project.id}/runs?page_size=1&foo=bar&state=failed,crashed&inserted_after=2024-01-01T02:00:00%2B02:00", + "/api/projects/#{project.id}/runs"} + ] do + %{"links" => links} = conn |> get(path) |> json_response(200) + + for key <- ["self", "first"] do + uri = URI.parse(links[key]) + assert uri.path == request_path + + assert URI.decode_query(uri.query) == %{ + "inserted_after" => "2024-01-01T00:00:00.000000Z", + "page" => "1", + "page_size" => "1", + "state" => "failed,crashed" + } + end + end + end + + test "every bad query parameter is reported, none quoted", %{ + conn: conn, + project: project + } do + bad = plant_sentinel(%{"v" => "x"})["v"] + + query = + URI.encode_query(%{ + "inserted_after" => bad, + "inserted_before" => bad, + "updated_after" => bad, + "updated_before" => bad, + "workflow_id" => bad, + "work_order_id" => bad, + "state" => "started,#{bad}" + }) + + for path <- [ + "/api/runs?" <> query, + "/api/projects/#{project.id}/runs?" <> query + ] do + conn + |> get(path) + |> refute_sentinel() + |> assert_problem(400, :invalid_parameter, [ + %{ + "parameter" => "inserted_after", + "code" => "type", + "detail" => "is invalid" + }, + %{ + "parameter" => "inserted_before", + "code" => "type", + "detail" => "is invalid" + }, + %{ + "parameter" => "updated_after", + "code" => "type", + "detail" => "is invalid" + }, + %{ + "parameter" => "updated_before", + "code" => "type", + "detail" => "is invalid" + }, + %{ + "parameter" => "workflow_id", + "code" => "type", + "detail" => "is invalid" + }, + %{ + "parameter" => "work_order_id", + "code" => "type", + "detail" => "is invalid" + }, + %{ + "parameter" => "state", + "code" => "not_allowed", + "detail" => "has an invalid entry" + } + ]) + end + end + end + + defp workflow_of(run) do + Lightning.Repo.get!(Lightning.WorkOrder, run.work_order_id) + |> Lightning.Repo.preload(:workflow) + |> Map.fetch!(:workflow) + end +end diff --git a/test/lightning_web/controllers/api/run_controller_test.exs b/test/lightning_web/controllers/api/run_controller_test.exs index 3d17b1ad7fc..d3a3928f418 100644 --- a/test/lightning_web/controllers/api/run_controller_test.exs +++ b/test/lightning_web/controllers/api/run_controller_test.exs @@ -1,896 +1,74 @@ defmodule LightningWeb.API.RunControllerTest do use LightningWeb.ConnCase, async: true + import LightningWeb.API.ProblemHelpers + import LightningWeb.ApiReadHelpers import Lightning.Factories setup %{conn: conn} do {:ok, conn: put_req_header(conn, "accept", "application/json")} end - test "without a token", %{conn: conn} do - conn = get(conn, ~p"/api/runs") - - assert %{"error" => "Unauthorized"} == json_response(conn, 401) - end - - describe "with invalid token" do - test "gets a 401", %{conn: conn} do - token = "InvalidToken" - conn = conn |> Plug.Conn.put_req_header("authorization", "Bearer #{token}") - conn = get(conn, ~p"/api/runs") - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - end - - describe "index" do + describe "show" do setup [:assign_bearer_for_api, :create_project_for_current_user] - test "lists runs for projects I have access to", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - workorder1 = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - workorder2 = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :success, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - run1 = List.first(workorder1.runs) - run2 = List.first(workorder2.runs) - - # Create a run in another project (should not be accessible) - other_project = insert(:project) - other_workflow = insert(:workflow, project: other_project) - other_trigger = insert(:trigger, workflow: other_workflow) - - other_workorder = - insert(:workorder, - workflow: other_workflow, - trigger: other_trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: other_trigger - } - ] - ) - - other_run = List.first(other_workorder.runs) - - conn = get(conn, ~p"/api/runs") - - response = json_response(conn, 200) - - # Should only return runs from my project - assert length(response["data"]) == 2 - - run_ids = Enum.map(response["data"], & &1["id"]) - assert run1.id in run_ids - assert run2.id in run_ids - refute other_run.id in run_ids - - # Verify pagination metadata - assert response["meta"]["total_entries"] == 2 - assert response["meta"]["page_number"] == 1 - end - - test "filters runs by inserted_after", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - old_time = ~U[2024-01-01 10:00:00Z] - new_time = ~U[2024-01-01 12:00:00Z] - - # Create old run - _old_workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - inserted_at: old_time, - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger, - inserted_at: old_time - } - ] - ) - - # Create new run - new_workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - inserted_at: new_time, - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger, - inserted_at: new_time - } - ] - ) - - new_run = List.first(new_workorder.runs) - - conn = get(conn, ~p"/api/runs?inserted_after=2024-01-01T11:00:00Z") - - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == new_run.id - end - - test "filters runs by inserted_before", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - old_time = ~U[2024-01-01 10:00:00Z] - new_time = ~U[2024-01-01 12:00:00Z] - - # Create old run - old_workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - inserted_at: old_time, - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger, - inserted_at: old_time - } - ] - ) - - # Create new run - _new_workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - inserted_at: new_time, - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger, - inserted_at: new_time - } - ] - ) - - old_run = List.first(old_workorder.runs) - - conn = get(conn, ~p"/api/runs?inserted_before=2024-01-01T11:00:00Z") - - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == old_run.id - end - - test "combines multiple filters", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - old_time = ~U[2024-01-01 10:00:00Z] - mid_time = ~U[2024-01-01 12:00:00Z] - new_time = ~U[2024-01-01 14:00:00Z] - - # Create runs at different times - _old_workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - inserted_at: old_time, - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger, - inserted_at: old_time - } - ] - ) - - mid_workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - inserted_at: mid_time, - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger, - inserted_at: mid_time - } - ] - ) - - _new_workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - inserted_at: new_time, - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger, - inserted_at: new_time - } - ] - ) - - mid_run = List.first(mid_workorder.runs) - - conn = - get( - conn, - ~p"/api/runs?inserted_after=2024-01-01T11:00:00Z&inserted_before=2024-01-01T13:00:00Z" - ) - - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == mid_run.id - end - - test "supports pagination", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - # Create 25 runs - Enum.each(1..25, fn i -> - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - inserted_at: DateTime.utc_now() |> DateTime.add(i, :second), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger, - inserted_at: DateTime.utc_now() |> DateTime.add(i, :second) - } - ] - ) - end) - - # Get first page - conn = get(conn, ~p"/api/runs?page=1&page_size=10") - response = json_response(conn, 200) - - assert length(response["data"]) == 10 - assert response["meta"]["total_entries"] == 25 - assert response["meta"]["total_pages"] == 3 - assert response["meta"]["page_number"] == 1 - assert response["meta"]["page_size"] == 10 - - # Verify pagination links exist - assert Map.has_key?(response["links"], "first") - assert Map.has_key?(response["links"], "next") - assert Map.has_key?(response["links"], "last") - end - - test "filters runs by project_id via query parameter", %{ - conn: conn, - user: user - } do - # User has access to both projects - project1 = insert(:project, project_users: [%{user: user, role: :owner}]) - project2 = insert(:project, project_users: [%{user: user, role: :owner}]) - - workflow1 = insert(:workflow, project: project1) - workflow2 = insert(:workflow, project: project2) - trigger1 = insert(:trigger, workflow: workflow1) - trigger2 = insert(:trigger, workflow: workflow2) - - workorder1 = - insert(:workorder, - workflow: workflow1, - trigger: trigger1, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger1 - } - ] - ) - - workorder2 = - insert(:workorder, - workflow: workflow2, - trigger: trigger2, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger2 - } - ] - ) - - run1 = List.first(workorder1.runs) - run2 = List.first(workorder2.runs) - - conn = get(conn, ~p"/api/runs?project_id=#{project1.id}") - - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == run1.id - refute Enum.any?(response["data"], fn r -> r["id"] == run2.id end) - end - - test "filters runs by workflow_id via query parameter", %{ - conn: conn, - project: project - } do - workflow1 = insert(:workflow, project: project) - workflow2 = insert(:workflow, project: project) - trigger1 = insert(:trigger, workflow: workflow1) - trigger2 = insert(:trigger, workflow: workflow2) - - workorder1 = - insert(:workorder, - workflow: workflow1, - trigger: trigger1, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger1 - } - ] - ) - - workorder2 = - insert(:workorder, - workflow: workflow2, - trigger: trigger2, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger2 - } - ] - ) - - run1 = List.first(workorder1.runs) - run2 = List.first(workorder2.runs) - - conn = get(conn, ~p"/api/runs?workflow_id=#{workflow1.id}") - - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == run1.id - refute Enum.any?(response["data"], fn r -> r["id"] == run2.id end) - end - - test "filters runs by work_order_id via query parameter", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - workorder1 = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - workorder2 = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - run1 = List.first(workorder1.runs) - run2 = List.first(workorder2.runs) - - conn = get(conn, ~p"/api/runs?work_order_id=#{workorder1.id}") - - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == run1.id - refute Enum.any?(response["data"], fn r -> r["id"] == run2.id end) - end - - test "filters runs by state", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - started_wo = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - _success_wo = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :success, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - started_run = List.first(started_wo.runs) - - conn = get(conn, ~p"/api/runs?state=started") - - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == started_run.id - end - - test "filters runs by multiple comma-separated states", %{ + test "answers the run, and links to it without the request's query", %{ conn: conn, project: project } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - started_wo = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - failed_wo = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :failed, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - _success_wo = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :success, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - started_run = List.first(started_wo.runs) - failed_run = List.first(failed_wo.runs) - - conn = get(conn, ~p"/api/runs?state=started,failed") - - response = json_response(conn, 200) - - assert length(response["data"]) == 2 - ids = Enum.map(response["data"], & &1["id"]) - assert started_run.id in ids - assert failed_run.id in ids - end - - test "returns error for invalid state filter", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - conn = get(conn, ~p"/api/runs?state=bogus") - - response = json_response(conn, 400) - - assert %{"error" => error_message} = response - assert error_message =~ "Invalid state filter" - assert error_message =~ "bogus" - end - - test "nested route: lists runs for specific project", %{ - conn: conn, - user: user - } do - # User has access to both projects - project1 = insert(:project, project_users: [%{user: user, role: :owner}]) - project2 = insert(:project, project_users: [%{user: user, role: :owner}]) - - workflow1 = insert(:workflow, project: project1) - workflow2 = insert(:workflow, project: project2) - trigger1 = insert(:trigger, workflow: workflow1) - trigger2 = insert(:trigger, workflow: workflow2) - - workorder1 = - insert(:workorder, - workflow: workflow1, - trigger: trigger1, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger1 - } - ] - ) - - workorder2 = - insert(:workorder, - workflow: workflow2, - trigger: trigger2, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger2 - } - ] - ) - - run1 = List.first(workorder1.runs) - run2 = List.first(workorder2.runs) - - conn = get(conn, ~p"/api/projects/#{project1.id}/runs") - - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == run1.id - refute Enum.any?(response["data"], fn r -> r["id"] == run2.id end) - end - - test "nested route: filters runs by datetime in project scope", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - old_time = ~U[2024-01-01 10:00:00Z] - new_time = ~U[2024-01-01 12:00:00Z] - - # Create old run - _old_workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - inserted_at: old_time, - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger, - inserted_at: old_time - } - ] - ) - - # Create new run - new_workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - inserted_at: new_time, - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger, - inserted_at: new_time - } - ] - ) - - new_run = List.first(new_workorder.runs) - - conn = - get( - conn, - ~p"/api/projects/#{project.id}/runs?inserted_after=2024-01-01T11:00:00Z" - ) - - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == new_run.id - end - - test "returns error for invalid inserted_after format", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - _run = List.first(workorder.runs) - - conn = - get( - conn, - ~p"/api/runs?inserted_after=not-a-valid-datetime" - ) - - response = json_response(conn, 400) - - assert %{"error" => error_message} = response - assert error_message =~ "Invalid datetime format for 'inserted_after'" - assert error_message =~ "not-a-valid-datetime" - end - - test "returns error for invalid updated_before format", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - _run = List.first(workorder.runs) - - conn = - get( - conn, - ~p"/api/runs?updated_before=2024-13-45" - ) - - response = json_response(conn, 400) - - assert %{"error" => error_message} = response - assert error_message =~ "Invalid datetime format for 'updated_before'" - assert error_message =~ "2024-13-45" - end - - test "returns error for invalid datetime with integer", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - _run = List.first(workorder.runs) - - conn = get(conn, ~p"/api/runs?inserted_after=123456") - - response = json_response(conn, 400) - - assert %{"error" => error_message} = response - assert error_message =~ "Invalid datetime format for 'inserted_after'" - assert error_message =~ "123456" + run = + insert_run(project, + state: :success, + started_at: ~U[2024-01-01 10:00:00.000000Z], + finished_at: ~U[2024-01-01 10:05:00.000000Z] + ) + + assert conn |> get(~p"/api/runs/#{run.id}?foo=bar") |> json_response(200) == + %{ + "data" => %{ + "type" => "runs", + "id" => run.id, + "relationships" => %{}, + "links" => %{ + "self" => + "#{LightningWeb.Endpoint.url()}/api/runs/#{run.id}" + }, + "attributes" => %{ + "state" => "success", + "started_at" => "2024-01-01T10:00:00.000000Z", + "finished_at" => "2024-01-01T10:05:00.000000Z", + "priority" => "normal", + "error_type" => nil + } + }, + "included" => [], + "links" => %{ + "self" => "#{LightningWeb.Endpoint.url()}/api/runs/#{run.id}" + } + } end end - describe "show" do + describe "who may reach it" do setup [:assign_bearer_for_api, :create_project_for_current_user] - test "returns a single run by id", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: trigger - } - ] - ) - - run = List.first(workorder.runs) - - conn = get(conn, ~p"/api/runs/#{run.id}") - - response = json_response(conn, 200) - - assert response["data"]["id"] == run.id - assert response["data"]["type"] == "runs" - assert Map.has_key?(response["data"]["attributes"], "state") - assert Map.has_key?(response["data"]["attributes"], "started_at") - assert Map.has_key?(response["data"]["attributes"], "finished_at") - end - - test "returns 404 for non-existent run", %{conn: conn} do - conn = get(conn, ~p"/api/runs/#{Ecto.UUID.generate()}") - assert json_response(conn, 404) - end - - test "returns 401 for run in project user cannot access", %{conn: conn} do - other_project = insert(:project) - other_workflow = insert(:workflow, project: other_project) - other_trigger = insert(:trigger, workflow: other_workflow) - - other_workorder = - insert(:workorder, - workflow: other_workflow, - trigger: other_trigger, - dataclip: build(:dataclip), - runs: [ - %{ - state: :started, - dataclip: build(:dataclip), - starting_trigger: other_trigger - } - ] - ) - - other_run = List.first(other_workorder.runs) - - conn = get(conn, ~p"/api/runs/#{other_run.id}") - - assert json_response(conn, 401) + test "answers 404 for what the caller has no standing on or that is missing, before the query is read, and 403 to a member who has not met the MFA requirement", + %{conn: conn, project: project} do + own = insert_run(project) + elsewhere = insert(:project) + unseen = insert_run(elsewhere) + Lightning.Repo.update!(Ecto.Changeset.change(project, requires_mfa: true)) + + for {path, status, code} <- [ + {~p"/api/projects/#{elsewhere.id}/runs?state=sideways", 404, + :not_found}, + {~p"/api/projects/#{Ecto.UUID.generate()}/runs", 404, :not_found}, + {~p"/api/runs/#{unseen.id}", 404, :not_found}, + {~p"/api/runs/#{Ecto.UUID.generate()}", 404, :not_found}, + {~p"/api/runs/#{own.id}", 403, :forbidden} + ] do + conn |> get(path) |> assert_problem(status, code, []) + end end end end diff --git a/test/lightning_web/controllers/api/unexpected_errors_test.exs b/test/lightning_web/controllers/api/unexpected_errors_test.exs new file mode 100644 index 00000000000..4561b18ae14 --- /dev/null +++ b/test/lightning_web/controllers/api/unexpected_errors_test.exs @@ -0,0 +1,86 @@ +defmodule LightningWeb.API.UnexpectedErrorsTest do + use LightningWeb.ConnCase, async: true + + import ExUnit.CaptureLog + import Lightning.Factories + import Lightning.ServiceAccountHelpers + import LightningWeb.API.ProblemHelpers + import LightningWeb.ApiProjectsHelpers, only: [put_project: 3] + + setup %{conn: conn} do + {:ok, conn: put_req_header(conn, "accept", "application/json")} + end + + describe "a workflow write whose snapshot can't be saved" do + setup [:assign_bearer_for_api, :create_project_for_current_user] + + test "answers 500 internal, logs it, and leaves the workflow as it was", %{ + conn: conn, + project: project + } do + workflow = insert(:simple_workflow, name: "before", project: project) + + # The snapshot of a save carries the lock_version the save produces, so a + # snapshot already holding it makes the save's own snapshot collide. + insert(:snapshot, + workflow: workflow, + lock_version: workflow.lock_version + 1 + ) + + log = + capture_log(fn -> + conn + |> put_req_header("content-type", "application/json") + |> patch( + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(%{name: "after"}) + ) + |> assert_problem(500, :internal, []) + end) + + assert log =~ "An API action could not save a workflow snapshot" + assert Repo.reload!(workflow).name == "before" + end + end + + describe "a project change whose audit events can't be written" do + setup %{conn: conn} do + {conn, account} = with_service_account(conn, ["projects:write"]) + + # An account with no uuid has no actor id to record, so its audit events + # are refused. + Mox.stub(Lightning.MockConfig, :service_account, fn -> + %{account | uuid: nil} + end) + + %{conn: conn} + end + + test "answers 500 internal, logs it, and leaves the project as it was", %{ + conn: conn + } do + owner = insert(:user) + + project = + insert(:project, + name: "before", + project_users: [%{user: owner, role: :owner}] + ) + + log = + capture_log(fn -> + conn + |> put_project(project.id, %{ + "name" => "after", + "members" => [%{"email" => owner.email, "role" => "owner"}] + }) + |> assert_problem(500, :internal, []) + end) + + assert log =~ + "An API action could not write the audit events of a project change" + + assert Repo.reload!(project).name == "before" + end + end +end diff --git a/test/lightning_web/controllers/api/user_controller_test.exs b/test/lightning_web/controllers/api/user_controller_test.exs deleted file mode 100644 index cb08b634554..00000000000 --- a/test/lightning_web/controllers/api/user_controller_test.exs +++ /dev/null @@ -1,810 +0,0 @@ -defmodule LightningWeb.API.UserControllerTest do - use LightningWeb.ConnCase, async: true - - import Lightning.Factories - import Lightning.ServiceAccountHelpers - - alias Lightning.Accounts - alias Lightning.Accounts.User - alias Lightning.ServiceAccount.AccessToken - alias LightningWeb.UserAuth - - @password "a long enough password" - - setup %{conn: conn} do - {account, _private_key} = service_account_with_key() - Mox.stub(Lightning.MockConfig, :service_account, fn -> account end) - - %{ - conn: put_req_header(conn, "accept", "application/json"), - account: account - } - end - - defp with_scopes(conn, account, scopes) do - put_req_header( - conn, - "authorization", - "Bearer " <> AccessToken.issue(account, scopes) - ) - end - - defp user_url(id), do: "#{LightningWeb.Endpoint.url()}/api/users/#{id}" - - defp watch_sockets(user) do - LightningWeb.Endpoint.subscribe(UserAuth.user_socket_topic(user)) - LightningWeb.Endpoint.subscribe(UserAuth.live_socket_topic(user)) - end - - describe "POST /api/users" do - setup %{conn: conn, account: account} do - %{conn: with_scopes(conn, account, ["users:write"])} - end - - test "creates a user with names and answers 201", %{conn: conn} do - conn = - post(conn, ~p"/api/users", %{ - email: "Ada@Example.com", - password: @password, - first_name: "Ada", - last_name: "Lovelace" - }) - - assert %{"data" => %{"id" => id} = data} = json_response(conn, 201) - - assert data == %{ - "type" => "users", - "id" => id, - "attributes" => %{ - "email" => "ada@example.com", - "first_name" => "Ada", - "last_name" => "Lovelace", - "role" => "user" - }, - "links" => %{"self" => user_url(id)} - } - - assert %User{role: :user, confirmed_at: nil} = - Accounts.get_user_by_email("ada@example.com") - end - - test "creates a confirmed superuser", %{conn: conn} do - conn = - post(conn, ~p"/api/users", %{ - email: "root@example.com", - password: @password, - first_name: "Root", - last_name: "Admin", - role: "superuser", - confirmed: true - }) - - assert %{ - "data" => %{ - "attributes" => %{ - "email" => "root@example.com", - "role" => "superuser" - } - } - } = json_response(conn, 201) - - user = Accounts.get_user_by_email("root@example.com") - assert %User{role: :superuser} = user - assert user.confirmed_at - assert Accounts.get_user_by_email_and_password(user.email, @password) - end - - test "ignores keys beyond the six it takes", %{conn: conn} do - conn = - post(conn, ~p"/api/users", %{ - email: "sneaky@example.com", - password: @password, - first_name: "Sneaky", - last_name: "User", - support_user: true, - disabled: true, - hashed_password: "not-a-hash", - confirmed_at: "2020-01-01T00:00:00Z" - }) - - assert json_response(conn, 201) - - assert %User{support_user: false, disabled: false, confirmed_at: nil} = - user = Accounts.get_user_by_email("sneaky@example.com") - - assert Accounts.get_user_by_email_and_password(user.email, @password) - end - - test "answers 409 with the existing user when the email is taken in any case", - %{conn: conn} do - existing = - insert(:user, - email: "taken@example.com", - first_name: "Tak", - last_name: "En", - role: :superuser - ) - - conn = - post(conn, ~p"/api/users", %{ - email: "TAKEN@example.com", - password: @password, - first_name: "Other", - last_name: "Person", - role: "user" - }) - - assert json_response(conn, 409) == %{ - "data" => %{ - "type" => "users", - "id" => existing.id, - "attributes" => %{ - "email" => "taken@example.com", - "first_name" => "Tak", - "last_name" => "En", - "role" => "superuser" - }, - "links" => %{"self" => user_url(existing.id)} - } - } - end - - test "answers 422 keyed by the request's field names", %{conn: conn} do - assert json_response( - post(conn, ~p"/api/users", %{ - email: "not-an-email", - password: "short", - role: "admin", - confirmed: "perhaps", - first_name: String.duplicate("a", 256), - last_name: "Long" - }), - 422 - ) == %{ - "errors" => %{ - "first_name" => ["should be at most 255 character(s)"], - "email" => ["Email address not valid."], - "password" => ["Password minimum length is 12 characters."], - "role" => ["is invalid"], - "confirmed" => ["is invalid"] - } - } - - assert json_response( - post(conn, ~p"/api/users", %{ - email: "nul@example.com", - password: "abcdefghijkl\0anything", - first_name: "Nul", - last_name: "a\0b" - }), - 422 - ) == %{ - "errors" => %{ - "password" => ["can't contain a NUL character"], - "last_name" => ["can't contain control characters"] - } - } - - assert json_response(post(conn, ~p"/api/users", %{}), 422) == %{ - "errors" => %{ - "email" => ["This field can't be blank."], - "password" => ["This field can't be blank."], - "first_name" => ["This field can't be blank."], - "last_name" => ["This field can't be blank."] - } - } - end - - test "answers 422 for a user without both names, for either role", %{ - conn: conn - } do - for role <- ["user", "superuser"], - {names, blank} <- [ - {%{}, ["first_name", "last_name"]}, - {%{first_name: "", last_name: " "}, ["first_name", "last_name"]}, - {%{first_name: "Ada"}, ["last_name"]}, - {%{first_name: "\u200B", last_name: "Lovelace"}, ["first_name"]} - ] do - email = "nameless-#{System.unique_integer([:positive])}@example.com" - - conn = - post( - conn, - ~p"/api/users", - Map.merge(%{email: email, password: @password, role: role}, names) - ) - - assert %{"errors" => errors} = json_response(conn, 422) - assert errors |> Map.keys() |> Enum.sort() == blank - refute Accounts.get_user_by_email(email) - end - end - - test "records an audit event whose actor is the service account", %{ - conn: conn, - account: account - } do - conn = - post(conn, ~p"/api/users", %{ - email: "audited@example.com", - password: @password, - first_name: "Audi", - last_name: "Ted" - }) - - assert %{"data" => %{"id" => id}} = json_response(conn, 201) - - assert %{entries: [audit]} = Lightning.Auditing.list_all() - - assert %{ - item_type: "user", - event: "created", - item_id: ^id, - actor_type: :service_account, - actor_display: %{label: "Service account", identifier: identifier} - } = audit - - assert identifier == account.id - - assert audit.actor_id == account.uuid - refute Map.has_key?(audit.changes.after, "hashed_password") - refute inspect(audit.changes) =~ @password - end - - test "needs users:write", %{conn: conn, account: account} do - conn = - conn - |> with_scopes(account, ["users:read"]) - |> post(~p"/api/users", %{email: "no@example.com", password: @password}) - - assert json_response(conn, 403) == %{"error" => "insufficient_scope"} - refute Accounts.get_user_by_email("no@example.com") - end - - test "refuses a personal access token", %{conn: conn} do - token = insert(:user, role: :superuser) |> Accounts.generate_api_token() - - conn = - conn - |> put_req_header("authorization", "Bearer " <> token) - |> post(~p"/api/users", %{email: "no@example.com", password: @password}) - - assert json_response(conn, 401) == %{"error" => "invalid_token"} - end - end - - describe "PATCH /api/users/:id" do - setup %{conn: conn, account: account} do - user = - insert(:user, - email: "patched@example.com", - first_name: "Ada", - last_name: "Lovelace" - ) - - %{conn: with_scopes(conn, account, ["users:write"]), user: user} - end - - test "changes the names and role and answers 200 as a create would", %{ - conn: conn, - user: user - } do - conn = - patch(conn, ~p"/api/users/#{user.id}", %{ - first_name: "Grace", - last_name: "Hopper", - role: "superuser" - }) - - assert json_response(conn, 200) == %{ - "data" => %{ - "type" => "users", - "id" => user.id, - "attributes" => %{ - "email" => "patched@example.com", - "first_name" => "Grace", - "last_name" => "Hopper", - "role" => "superuser" - }, - "links" => %{"self" => user_url(user.id)} - } - } - - assert %User{first_name: "Grace", last_name: "Hopper", role: :superuser} = - Accounts.get_user(user.id) - end - - test "leaves the password alone unless one is given", %{ - conn: conn, - user: user - } do - assert conn - |> patch(~p"/api/users/#{user.id}", %{first_name: "Grace"}) - |> json_response(200) - - assert Accounts.get_user_by_email_and_password(user.email, "hello world!") - end - - test "changes the password and signs the user out of its sessions only", %{ - conn: conn, - user: user - } do - session = Accounts.generate_user_session_token(user) - api_token = Accounts.generate_api_token(user) - watch_sockets(user) - - assert conn - |> patch(~p"/api/users/#{user.id}", %{password: @password}) - |> json_response(200) - - assert Accounts.get_user_by_email_and_password(user.email, @password) - refute Accounts.get_user_by_email_and_password(user.email, "hello world!") - refute Accounts.get_user_by_session_token(session) - assert Accounts.get_user_by_api_token(api_token) - assert_receive %Phoenix.Socket.Broadcast{event: "disconnect"} - assert_receive %Phoenix.Socket.Broadcast{event: "disconnect"} - end - - test "changes nothing when given the password the user already has", %{ - conn: conn, - user: user - } do - session = Accounts.generate_user_session_token(user) - watch_sockets(user) - - for _ <- 1..2 do - assert conn - |> patch(~p"/api/users/#{user.id}", %{password: "hello world!"}) - |> json_response(200) - end - - assert Accounts.get_user_by_session_token(session) - assert Accounts.get_user(user.id).hashed_password == user.hashed_password - assert %{entries: []} = Lightning.Auditing.list_all() - refute_received %Phoenix.Socket.Broadcast{event: "disconnect"} - end - - test "refuses a password bcrypt would read as the current one", %{ - conn: conn, - user: user - } do - assert json_response( - patch(conn, ~p"/api/users/#{user.id}", %{ - password: "hello world!\0anything" - }), - 422 - ) == %{ - "errors" => %{"password" => ["can't contain a NUL character"]} - } - end - - test "signs the user out when its role changes", %{conn: conn, user: user} do - session = Accounts.generate_user_session_token(user) - - assert conn - |> patch(~p"/api/users/#{user.id}", %{role: "superuser"}) - |> json_response(200) - - refute Accounts.get_user_by_session_token(session) - end - - test "keeps the sessions when only the names change", %{ - conn: conn, - user: user - } do - session = Accounts.generate_user_session_token(user) - watch_sockets(user) - - assert conn - |> patch(~p"/api/users/#{user.id}", %{ - first_name: "Grace", - role: "user" - }) - |> json_response(200) - - assert Accounts.get_user_by_session_token(session) - refute_received %Phoenix.Socket.Broadcast{event: "disconnect"} - end - - test "confirms an unconfirmed user and never unconfirms one", %{ - conn: conn, - user: user - } do - assert conn - |> patch(~p"/api/users/#{user.id}", %{confirmed: false}) - |> json_response(200) - - refute Accounts.get_user(user.id).confirmed_at - - assert conn - |> patch(~p"/api/users/#{user.id}", %{confirmed: true}) - |> json_response(200) - - assert %{confirmed_at: confirmed_at} = Accounts.get_user(user.id) - assert confirmed_at - - confirmed = - insert(:user, confirmed_at: ~U[2020-01-01 00:00:00Z]) - - for confirmed_param <- [true, false] do - assert conn - |> patch(~p"/api/users/#{confirmed.id}", %{ - confirmed: confirmed_param - }) - |> json_response(200) - - assert Accounts.get_user(confirmed.id).confirmed_at == - ~U[2020-01-01 00:00:00Z] - end - end - - test "ignores the email and keys it doesn't take", %{conn: conn, user: user} do - assert conn - |> patch(~p"/api/users/#{user.id}", %{ - email: "moved@example.com", - support_user: true, - disabled: true, - scheduled_deletion: "2030-01-01T00:00:00Z", - hashed_password: "not-a-hash", - confirmed_at: "2020-01-01T00:00:00Z" - }) - |> json_response(200) - - assert %User{ - email: "patched@example.com", - support_user: false, - disabled: false, - scheduled_deletion: nil, - confirmed_at: nil - } = Accounts.get_user(user.id) - - assert Accounts.get_user_by_email_and_password(user.email, "hello world!") - end - - test "answers 422 keyed by the request's field names", %{ - conn: conn, - user: user - } do - assert json_response( - patch(conn, ~p"/api/users/#{user.id}", %{ - first_name: " ", - last_name: "a\0b", - password: "short", - role: "admin", - confirmed: "perhaps" - }), - 422 - ) == %{ - "errors" => %{ - "first_name" => ["This field can't be blank."], - "last_name" => ["can't contain control characters"], - "password" => ["Password minimum length is 12 characters."], - "role" => ["is invalid"], - "confirmed" => ["is invalid"] - } - } - - assert json_response( - patch(conn, ~p"/api/users/#{user.id}", %{ - last_name: "", - password: "abcdefghijkl\0anything" - }), - 422 - ) == %{ - "errors" => %{ - "last_name" => ["This field can't be blank."], - "password" => ["can't contain a NUL character"] - } - } - - assert %User{first_name: "Ada", last_name: "Lovelace", role: :user} = - Accounts.get_user(user.id) - - assert Accounts.get_user_by_email_and_password(user.email, "hello world!") - assert %{entries: []} = Lightning.Auditing.list_all() - end - - test "answers 404 for a user that doesn't exist", %{conn: conn} do - assert conn - |> patch(~p"/api/users/#{Ecto.UUID.generate()}", %{first_name: "A"}) - |> json_response(404) - - assert conn - |> patch(~p"/api/users/not-a-uuid", %{first_name: "A"}) - |> json_response(404) - end - - test "records what changed, with the service account as the actor", %{ - conn: conn, - account: account, - user: user - } do - assert conn - |> patch(~p"/api/users/#{user.id}", %{ - first_name: "Grace", - last_name: "Lovelace", - password: @password - }) - |> json_response(200) - - assert %{entries: [audit]} = Lightning.Auditing.list_all() - - assert %{ - item_type: "user", - event: "updated", - item_id: item_id, - actor_type: :service_account, - metadata: %{"password_changed" => true} - } = audit - - assert item_id == user.id - assert audit.actor_id == account.uuid - assert audit.changes.before == %{"first_name" => "Ada"} - assert audit.changes.after == %{"first_name" => "Grace"} - refute inspect(audit) =~ @password - refute inspect(audit) =~ "$2b$" - end - - test "records nothing when nothing changes", %{conn: conn, user: user} do - assert conn - |> patch(~p"/api/users/#{user.id}", %{ - first_name: "Ada", - role: "user", - confirmed: false - }) - |> json_response(200) - - assert %{entries: []} = Lightning.Auditing.list_all() - end - - test "needs users:write", %{conn: conn, account: account, user: user} do - conn = - conn - |> with_scopes(account, ["users:read"]) - |> patch(~p"/api/users/#{user.id}", %{first_name: "Grace"}) - - assert json_response(conn, 403) == %{"error" => "insufficient_scope"} - assert %User{first_name: "Ada"} = Accounts.get_user(user.id) - end - - test "refuses a personal access token", %{conn: conn, user: user} do - token = insert(:user, role: :superuser) |> Accounts.generate_api_token() - - conn = - conn - |> put_req_header("authorization", "Bearer " <> token) - |> patch(~p"/api/users/#{user.id}", %{first_name: "Grace"}) - - assert json_response(conn, 401) == %{"error" => "invalid_token"} - assert %User{first_name: "Ada"} = Accounts.get_user(user.id) - end - end - - describe "superuser changes" do - setup %{conn: conn, account: account} do - ref = - :telemetry_test.attach_event_handlers(self(), [ - [:lightning, :accounts, :superuser_changed] - ]) - - %{conn: with_scopes(conn, account, ["users:write"]), ref: ref} - end - - defp create_user(conn, role) do - conn - |> post(~p"/api/users", %{ - email: "#{role}-#{System.unique_integer([:positive])}@example.com", - password: @password, - first_name: "Some", - last_name: "One", - role: role - }) - |> json_response(201) - |> get_in(["data", "id"]) - end - - defp assert_superuser_change(ref, change, user_id, account) do - assert_receive {[:lightning, :accounts, :superuser_changed], ^ref, - %{count: 1}, - %{ - change: ^change, - user_id: ^user_id, - actor_type: :service_account, - actor_id: actor_id - }} - - assert actor_id == account.id - end - - test "report a superuser created", %{conn: conn, account: account, ref: ref} do - id = create_user(conn, "superuser") - - assert_superuser_change(ref, :created, id, account) - end - - test "report a user made a superuser", %{ - conn: conn, - account: account, - ref: ref - } do - user = insert(:user) - - conn - |> patch(~p"/api/users/#{user.id}", %{role: "superuser"}) - |> json_response(200) - - assert_superuser_change(ref, :granted, user.id, account) - end - - test "report a superuser's password changing", %{ - conn: conn, - account: account, - ref: ref - } do - user = insert(:user, role: :superuser) - - conn - |> patch(~p"/api/users/#{user.id}", %{password: @password}) - |> json_response(200) - - assert_superuser_change(ref, :password_changed, user.id, account) - end - - test "say nothing for an ordinary user, or a superuser's names", %{ - conn: conn, - ref: ref - } do - user = insert(:user) - superuser = insert(:user, role: :superuser) - - create_user(conn, "user") - - conn - |> patch(~p"/api/users/#{user.id}", %{password: @password}) - |> json_response(200) - - conn - |> patch(~p"/api/users/#{superuser.id}", %{first_name: "Renamed"}) - |> json_response(200) - - refute_received {[:lightning, :accounts, :superuser_changed], ^ref, _, _} - end - end - - describe "GET /api/users" do - setup %{conn: conn, account: account} do - %{conn: with_scopes(conn, account, ["users:read"])} - end - - test "finds a user by email without regard to case", %{conn: conn} do - user = insert(:user, email: "findme@example.com", first_name: "Find") - insert(:user) - - assert %{ - "data" => [ - %{ - "id" => id, - "attributes" => %{ - "email" => "findme@example.com", - "first_name" => "Find" - } - } - ], - "links" => %{"self" => _} - } = - conn - |> get(~p"/api/users?email=FindMe@Example.COM") - |> json_response(200) - - assert id == user.id - - assert %{"data" => []} = - conn - |> get(~p"/api/users?email=nobody@example.com") - |> json_response(200) - end - - test "never includes a password or its hash", %{conn: conn} do - insert(:user, email: "secret@example.com") - - body = - conn |> get(~p"/api/users?email=secret@example.com") |> response(200) - - refute body =~ "password" - refute body =~ "$2b$" - end - - test "is paginated", %{conn: conn} do - insert_list(3, :user) - - assert %{"data" => [_, _], "links" => %{"next" => _}} = - conn |> get(~p"/api/users?page_size=2") |> json_response(200) - end - - test "never answers with more than 100 users a page", %{conn: conn} do - insert_list(101, :user) - - assert %{"data" => data, "links" => %{"next" => next}} = - conn - |> get(~p"/api/users?page_size=100000000") - |> json_response(200) - - assert length(data) == 100 - assert next =~ "page=2" - end - - test "falls back to the default page for page params it can't read", %{ - conn: conn - } do - insert_list(11, :user) - - for query <- [ - "page=abc&page_size=xyz", - "page[]=2&page_size[]=2", - "page=-1&page_size=0" - ] do - assert %{"data" => data, "links" => %{"prev" => nil}} = - conn |> get("/api/users?" <> query) |> json_response(200) - - assert length(data) == 10, query - end - end - - test "answers a page number past the end with the last page", %{conn: conn} do - insert_list(11, :user) - - assert %{"data" => [_ | _], "links" => %{"next" => nil}} = - conn - |> get(~p"/api/users?page=99999999999999999999") - |> json_response(200) - end - - test "answers 422 for an email filter that isn't a string", %{conn: conn} do - insert(:user, email: "x@example.com") - - assert %{"errors" => %{"email" => ["is invalid"]}} = - conn - |> get(~p"/api/users?email[]=x@example.com") - |> json_response(422) - end - - test "needs users:read", %{conn: conn, account: account} do - conn = - conn - |> with_scopes(account, ["users:write"]) - |> get(~p"/api/users?email=a@example.com") - - assert json_response(conn, 403) == %{"error" => "insufficient_scope"} - end - - test "refuses a personal access token", %{conn: conn} do - token = insert(:user, role: :superuser) |> Accounts.generate_api_token() - - conn = - conn - |> put_req_header("authorization", "Bearer " <> token) - |> get(~p"/api/users?email=a@example.com") - - assert json_response(conn, 401) == %{"error" => "invalid_token"} - end - end - - describe "GET /api/users/:id" do - test "shows the user its self link names", %{conn: conn, account: account} do - user = insert(:user, email: "shown@example.com") - conn = with_scopes(conn, account, ["users:read"]) - - assert %{"data" => %{"id" => id, "links" => %{"self" => self}}} = - conn |> get(~p"/api/users/#{user.id}") |> json_response(200) - - assert id == user.id - assert self == user_url(user.id) - - assert conn - |> get(~p"/api/users/#{Ecto.UUID.generate()}") - |> json_response(404) - - assert conn |> get(~p"/api/users/not-a-uuid") |> json_response(404) - end - end -end diff --git a/test/lightning_web/controllers/api/users/creating_test.exs b/test/lightning_web/controllers/api/users/creating_test.exs new file mode 100644 index 00000000000..fef87933bbd --- /dev/null +++ b/test/lightning_web/controllers/api/users/creating_test.exs @@ -0,0 +1,389 @@ +defmodule LightningWeb.API.UsersCreatingTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProblemHelpers + import Lightning.Factories + import LightningWeb.ApiUsersHelpers + + alias Lightning.Accounts + alias Lightning.Accounts.User + + setup :json_conn_with_service_account + + describe "POST /api/users" do + setup %{conn: conn, account: account} do + %{conn: with_scopes(conn, account, ["users:write"])} + end + + test "creates a user with names and answers 201", %{conn: conn} do + conn = + post(conn, ~p"/api/users", %{ + email: "Ada@Example.com", + password: password(), + first_name: "Ada", + last_name: "Lovelace" + }) + + assert %{"data" => %{"id" => id} = data} = json_response(conn, 201) + + assert data == %{ + "type" => "users", + "id" => id, + "attributes" => %{ + "email" => "ada@example.com", + "first_name" => "Ada", + "last_name" => "Lovelace", + "role" => "user" + }, + "links" => %{"self" => user_url(id)} + } + + assert %User{role: :user, confirmed_at: nil} = + Accounts.get_user_by_email("ada@example.com") + end + + test "creates a confirmed superuser", %{conn: conn} do + conn = + post(conn, ~p"/api/users", %{ + email: "root@example.com", + password: password(), + first_name: "Root", + last_name: "Admin", + role: "superuser", + confirmed: true + }) + + assert %{ + "data" => %{ + "attributes" => %{ + "email" => "root@example.com", + "role" => "superuser" + } + } + } = json_response(conn, 201) + + user = Accounts.get_user_by_email("root@example.com") + assert %User{role: :superuser} = user + assert user.confirmed_at + assert Accounts.get_user_by_email_and_password(user.email, password()) + end + + test "refuses a key beyond the six it takes, creating no user", %{ + conn: conn + } do + conn + |> post(~p"/api/users", %{ + email: "colourful@example.com", + password: password(), + first_name: "Colour", + last_name: "Ful", + colour: "blue" + }) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/colour", + "code" => "unknown_key", + "detail" => "is not accepted" + } + ]) + + refute Accounts.get_user_by_email("colourful@example.com") + end + + test "more than one error is a 422 with every entry, never a 409", %{ + conn: conn + } do + insert(:user, email: "taken@example.com") + + unknown = fn key -> + %{ + "pointer" => "#/" <> key, + "code" => "unknown_key", + "detail" => "is not accepted" + } + end + + for {body, entries} <- [ + {%{ + email: "nopass@example.com", + first_name: "No", + last_name: "Pass", + colour: "blue" + }, + [ + unknown.("colour"), + %{ + "pointer" => "#/password", + "code" => "required", + "detail" => "can't be blank" + } + ]}, + {%{ + email: "taken@example.com", + password: password(), + first_name: "Other", + last_name: "Person", + support_user: true + }, + [ + unknown.("support_user"), + %{ + "pointer" => "#/email", + "code" => "taken", + "detail" => "has already been taken" + } + ]}, + {%{ + email: "taken@example.com", + password: "short", + first_name: "Other", + last_name: "Person" + }, + [ + %{ + "pointer" => "#/email", + "code" => "taken", + "detail" => "has already been taken" + }, + %{ + "pointer" => "#/password", + "code" => "too_short", + "detail" => "should be at least 12 character(s)" + } + ]} + ] do + body = + conn + |> post(~p"/api/users", body) + |> assert_problem(422, :invalid, entries) + + refute Map.has_key?(body, "existing") + end + + refute Accounts.get_user_by_email("nopass@example.com") + end + + test "answers 409 taken with only the existing user's id when the email is taken in any case", + %{conn: conn} do + existing = + insert(:user, + email: "taken@example.com", + first_name: "Tak", + last_name: "En", + role: :superuser + ) + + conn = + post(conn, ~p"/api/users", %{ + email: "TAKEN@example.com", + password: password(), + first_name: "Other", + last_name: "Person", + role: "user" + }) + + body = + assert_problem(conn, 409, :taken, [ + %{ + "pointer" => "#/email", + "code" => "taken", + "detail" => "has already been taken" + } + ]) + + assert body["existing"] == %{"id" => existing.id} + + assert body |> Map.keys() |> Enum.sort() == + ~w(code detail errors existing status title type) + end + + test "answers 409 taken without existing when the holder is gone by the time it is looked up", + %{conn: conn} do + attrs = %{ + "email" => "taken@example.com", + "password" => password(), + "first_name" => "Other", + "last_name" => "Person" + } + + holder = insert(:user, email: "taken@example.com") + + {:error, refused} = + Lightning.Extensions.AccountHook.handle_create_user(attrs) + + Lightning.Repo.delete!(holder) + + Mox.expect(Lightning.Extensions.MockAccountHook, :handle_create_user, fn + ^attrs -> {:error, refused} + end) + + body = + conn + |> post(~p"/api/users", attrs) + |> assert_problem(409, :taken, [ + %{ + "pointer" => "#/email", + "code" => "taken", + "detail" => "has already been taken" + } + ]) + + refute Map.has_key?(body, "existing") + end + + test "answers 422 with an entry at each field that breaks a rule", %{ + conn: conn + } do + conn + |> post(~p"/api/users", %{ + email: "not-an-email", + password: "short", + role: "admin", + confirmed: "perhaps", + first_name: String.duplicate("a", 256), + last_name: "Long" + }) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/first_name", + "code" => "too_long", + "detail" => "should be at most 255 character(s)" + }, + %{ + "pointer" => "#/email", + "code" => "format", + "detail" => "must be a valid email address" + }, + %{ + "pointer" => "#/password", + "code" => "too_short", + "detail" => "should be at least 12 character(s)" + }, + %{ + "pointer" => "#/role", + "code" => "not_allowed", + "detail" => "is invalid" + }, + %{"pointer" => "#/confirmed", "code" => "type", "detail" => "is invalid"} + ]) + end + + test "answers 422 for a NUL character in a password and a control character in a name", + %{conn: conn} do + conn + |> post(~p"/api/users", %{ + email: "nul@example.com", + password: "abcdefghijkl\0anything", + first_name: "Nul", + last_name: "a\0b" + }) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/password", + "code" => "format", + "detail" => "can't contain a NUL character" + }, + %{ + "pointer" => "#/last_name", + "code" => "format", + "detail" => "can't contain control characters" + } + ]) + end + + test "answers 422 for every field a body leaves out", %{conn: conn} do + conn + |> post(~p"/api/users", %{}) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/email", + "code" => "required", + "detail" => "can't be blank" + }, + %{ + "pointer" => "#/password", + "code" => "required", + "detail" => "can't be blank" + }, + %{ + "pointer" => "#/first_name", + "code" => "required", + "detail" => "can't be blank" + }, + %{ + "pointer" => "#/last_name", + "code" => "required", + "detail" => "can't be blank" + } + ]) + end + + test "answers 422 for a user without both names, for either role", %{ + conn: conn + } do + for role <- ["user", "superuser"], + {names, blank} <- [ + {%{}, ["first_name", "last_name"]}, + {%{first_name: "", last_name: " "}, ["first_name", "last_name"]}, + {%{first_name: "Ada"}, ["last_name"]}, + {%{first_name: "\u200B", last_name: "Lovelace"}, ["first_name"]} + ] do + email = "nameless-#{System.unique_integer([:positive])}@example.com" + + conn = + post( + conn, + ~p"/api/users", + Map.merge(%{email: email, password: password(), role: role}, names) + ) + + assert_problem( + conn, + 422, + :invalid, + Enum.map( + blank, + &%{ + "pointer" => "#/" <> &1, + "code" => "required", + "detail" => "can't be blank" + } + ) + ) + + refute Accounts.get_user_by_email(email) + end + end + + test "records an audit event whose actor is the service account", %{ + conn: conn, + account: account + } do + conn = + post(conn, ~p"/api/users", %{ + email: "audited@example.com", + password: password(), + first_name: "Audi", + last_name: "Ted" + }) + + assert %{"data" => %{"id" => id}} = json_response(conn, 201) + + assert %{entries: [audit]} = Lightning.Auditing.list_all() + + assert %{ + item_type: "user", + event: "created", + item_id: ^id, + actor_type: :service_account, + actor_display: %{label: "Service account", identifier: identifier} + } = audit + + assert identifier == account.id + + assert audit.actor_id == account.uuid + refute Map.has_key?(audit.changes.after, "hashed_password") + refute inspect(audit.changes) =~ password() + end + end +end diff --git a/test/lightning_web/controllers/api/users/listing_test.exs b/test/lightning_web/controllers/api/users/listing_test.exs new file mode 100644 index 00000000000..750c690cd5e --- /dev/null +++ b/test/lightning_web/controllers/api/users/listing_test.exs @@ -0,0 +1,187 @@ +defmodule LightningWeb.API.UsersListingTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProblemHelpers + import Lightning.Factories + import LightningWeb.ApiUsersHelpers + + setup :json_conn_with_service_account + + describe "GET /api/users" do + setup %{conn: conn, account: account} do + %{conn: with_scopes(conn, account, ["users:read"])} + end + + test "finds a user by email without regard to case", %{conn: conn} do + user = insert(:user, email: "findme@example.com", first_name: "Find") + insert(:user) + + assert %{ + "data" => [ + %{ + "id" => id, + "attributes" => %{ + "email" => "findme@example.com", + "first_name" => "Find" + } + } + ], + "links" => %{"self" => _} + } = + conn + |> get(~p"/api/users?email=FindMe@Example.COM") + |> json_response(200) + + assert id == user.id + + assert %{"data" => []} = + conn + |> get(~p"/api/users?email=nobody@example.com") + |> json_response(200) + end + + test "never includes a password or its hash", %{conn: conn} do + insert(:user, email: "secret@example.com") + + body = + conn |> get(~p"/api/users?email=secret@example.com") |> response(200) + + refute body =~ "password" + refute body =~ "$2b$" + end + + test "page links carry the email filter and no other query parameter", %{ + conn: conn + } do + insert(:user, email: "linked@example.com") + + assert %{"links" => links} = + conn + |> get( + ~p"/api/users?email=linked@example.com&page_size=5&foo=bar" + ) + |> json_response(200) + + first = + "#{LightningWeb.Endpoint.url()}/api/users?email=linked%40example.com&page=1&page_size=5" + + assert links == %{ + "self" => first, + "first" => first, + "last" => first, + "next" => nil, + "prev" => nil + } + end + + test "page links carry only the page and page size without a filter", %{ + conn: conn + } do + insert_list(3, :user) + + base = "#{LightningWeb.Endpoint.url()}/api/users" + + for {query, expected} <- [ + {"page=1&page_size=2&foo=bar", + %{ + "self" => base <> "?page=1&page_size=2", + "first" => base <> "?page=1&page_size=2", + "last" => base <> "?page=2&page_size=2", + "next" => base <> "?page=2&page_size=2", + "prev" => nil + }}, + {"page=2&page_size=2&foo=bar", + %{ + "self" => base <> "?page=2&page_size=2", + "first" => base <> "?page=1&page_size=2", + "last" => base <> "?page=2&page_size=2", + "next" => nil, + "prev" => base <> "?page=1&page_size=2" + }} + ] do + assert %{"links" => links} = + conn |> get("/api/users?" <> query) |> json_response(200) + + assert links == expected, query + end + end + + test "never answers with more than 100 users a page", %{conn: conn} do + insert_list(101, :user) + + assert %{"data" => data, "links" => %{"next" => next}} = + conn + |> get(~p"/api/users?page_size=100000000") + |> json_response(200) + + assert length(data) == 100 + assert next =~ "page=2" + end + + test "falls back to the default page for page params it can't read", %{ + conn: conn + } do + insert_list(11, :user) + + for query <- [ + "page=abc&page_size=xyz", + "page[]=2&page_size[]=2", + "page=-1&page_size=0" + ] do + assert %{"data" => data, "links" => %{"prev" => nil}} = + conn |> get("/api/users?" <> query) |> json_response(200) + + assert length(data) == 10, query + end + end + + test "answers a page number past the end with the last page", %{conn: conn} do + insert_list(11, :user) + + assert %{"data" => [_ | _], "links" => %{"next" => nil}} = + conn + |> get(~p"/api/users?page=99999999999999999999") + |> json_response(200) + end + + test "answers 400 invalid_parameter for an email filter that isn't a string", + %{conn: conn} do + insert(:user, email: "x@example.com") + + conn + |> get(~p"/api/users?email[]=x@example.com") + |> assert_problem(400, :invalid_parameter, [ + %{"parameter" => "email", "code" => "type", "detail" => "is invalid"} + ]) + end + end + + describe "GET /api/users/:id" do + test "shows a user, with a self link that names it", %{ + conn: conn, + account: account + } do + user = insert(:user, email: "shown@example.com") + conn = with_scopes(conn, account, ["users:read"]) + + assert %{"data" => %{"id" => id, "links" => %{"self" => self}}} = + conn |> get(~p"/api/users/#{user.id}") |> json_response(200) + + assert id == user.id + assert self == user_url(user.id) + end + + test "answers 404 for a user that doesn't exist or an id that isn't a UUID", + %{conn: conn, account: account} do + conn = with_scopes(conn, account, ["users:read"]) + + conn + |> get(~p"/api/users/#{Ecto.UUID.generate()}") + |> assert_problem(404, :not_found, []) + + conn + |> get(~p"/api/users/not-a-uuid") + |> assert_problem(404, :not_found, []) + end + end +end diff --git a/test/lightning_web/controllers/api/users/request_reading_test.exs b/test/lightning_web/controllers/api/users/request_reading_test.exs new file mode 100644 index 00000000000..2be0646fc03 --- /dev/null +++ b/test/lightning_web/controllers/api/users/request_reading_test.exs @@ -0,0 +1,135 @@ +defmodule LightningWeb.API.UsersRequestReadingTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProblemHelpers + import Lightning.Factories + import LightningWeb.ApiUsersHelpers + + alias Lightning.Accounts + alias Lightning.Accounts.User + + setup :json_conn_with_service_account + + describe "reading a request" do + setup %{conn: conn, account: account} do + %{ + conn: with_scopes(conn, account, ["users:read", "users:write"]), + planted: + plant_sentinel(%{ + "email" => nil, + "password" => nil, + "first_name" => nil, + "last_name" => nil, + "role" => nil, + "confirmed" => nil + }) + } + end + + test "answers 400 malformed for a body that is not a JSON object, creating or changing nothing", + %{conn: conn} do + user = insert(:user, first_name: "Ada") + + for body <- [ + [%{"first_name" => "Grace", "email" => "array@example.com"}], + "Grace", + 7 + ], + path <- [~p"/api/users", ~p"/api/users/#{user.id}"] do + conn = put_req_header(conn, "content-type", "application/json") + json = Jason.encode!(body) + + conn = + if path == ~p"/api/users", + do: post(conn, path, json), + else: patch(conn, path, json) + + assert_problem(conn, 400, :malformed, []) + end + + assert Repo.reload!(user).first_name == "Ada" + refute Accounts.get_user_by_email("array@example.com") + end + + test "takes the fields of a create or an update from the body, never the query string", + %{conn: conn} do + user = insert(:user, first_name: "Ada") + + conn + |> post(~p"/api/users?role=superuser", %{ + email: "query@example.com", + password: password(), + first_name: "Query", + last_name: "String" + }) + |> json_response(201) + + assert %User{role: :user} = Accounts.get_user_by_email("query@example.com") + + conn + |> patch(~p"/api/users/#{user.id}?role=superuser", %{first_name: "Grace"}) + |> json_response(200) + + assert %User{first_name: "Grace", role: :user} = Accounts.get_user(user.id) + end + + test "no error answer quotes a value from the request, on any users route", + %{conn: conn, planted: planted} do + user = insert(:user) + + conn + |> post(~p"/api/users", planted) + |> refute_sentinel() + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/confirmed", + "code" => "type", + "detail" => "is invalid" + }, + %{ + "pointer" => "#/role", + "code" => "not_allowed", + "detail" => "is invalid" + }, + %{ + "pointer" => "#/email", + "code" => "format", + "detail" => "must be a valid email address" + } + ]) + + conn + |> patch(~p"/api/users/#{user.id}", Map.delete(planted, "email")) + |> refute_sentinel() + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/confirmed", + "code" => "type", + "detail" => "is invalid" + }, + %{ + "pointer" => "#/role", + "code" => "not_allowed", + "detail" => "is invalid" + } + ]) + + conn + |> patch(~p"/api/users/#{planted["email"]}", %{first_name: "A"}) + |> refute_sentinel() + |> assert_problem(404, :not_found, []) + + conn + |> get("/api/users?email[]=" <> planted["email"]) + |> refute_sentinel() + |> assert_problem(400, :invalid_parameter, [ + %{"parameter" => "email", "code" => "type", "detail" => "is invalid"} + ]) + + conn + |> get(~p"/api/users/#{planted["email"]}") + |> refute_sentinel() + |> assert_problem(404, :not_found, []) + end + end +end diff --git a/test/lightning_web/controllers/api/users/updating_test.exs b/test/lightning_web/controllers/api/users/updating_test.exs new file mode 100644 index 00000000000..7f21312e36f --- /dev/null +++ b/test/lightning_web/controllers/api/users/updating_test.exs @@ -0,0 +1,447 @@ +defmodule LightningWeb.API.UsersUpdatingTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProblemHelpers + import Lightning.RaceHelpers + import Lightning.Factories + import LightningWeb.ApiUsersHelpers + + alias Lightning.Accounts + alias Lightning.Accounts.User + + setup :json_conn_with_service_account + + describe "PATCH /api/users/:id" do + setup %{conn: conn, account: account} do + user = + insert(:user, + email: "patched@example.com", + first_name: "Ada", + last_name: "Lovelace" + ) + + %{conn: with_scopes(conn, account, ["users:write"]), user: user} + end + + test "changes the names and role and answers 200 as a create would", %{ + conn: conn, + user: user + } do + conn = + patch(conn, ~p"/api/users/#{user.id}", %{ + first_name: "Grace", + last_name: "Hopper", + role: "superuser" + }) + + assert json_response(conn, 200) == %{ + "data" => %{ + "type" => "users", + "id" => user.id, + "attributes" => %{ + "email" => "patched@example.com", + "first_name" => "Grace", + "last_name" => "Hopper", + "role" => "superuser" + }, + "links" => %{"self" => user_url(user.id)} + } + } + + assert %User{first_name: "Grace", last_name: "Hopper", role: :superuser} = + Accounts.get_user(user.id) + end + + test "a user another request deletes mid-update answers 409 concurrent_edit", + %{conn: conn, user: user} do + after_first_query("users", user.id, fn id -> + Repo.query!("DELETE FROM users WHERE id = $1", [id]) + end) + + conn + |> patch(~p"/api/users/#{user.id}", %{first_name: "Grace"}) + |> assert_problem(409, :concurrent_edit, []) + + refute Repo.reload(user) + end + + test "leaves the password alone unless one is given", %{ + conn: conn, + user: user + } do + assert conn + |> patch(~p"/api/users/#{user.id}", %{first_name: "Grace"}) + |> json_response(200) + + assert Accounts.get_user_by_email_and_password(user.email, "hello world!") + end + + test "changes the password and signs the user out of its sessions only", %{ + conn: conn, + user: user + } do + session = Accounts.generate_user_session_token(user) + api_token = Accounts.generate_api_token(user) + watch_sockets(user) + + assert conn + |> patch(~p"/api/users/#{user.id}", %{password: password()}) + |> json_response(200) + + assert Accounts.get_user_by_email_and_password(user.email, password()) + refute Accounts.get_user_by_email_and_password(user.email, "hello world!") + refute Accounts.get_user_by_session_token(session) + assert Accounts.get_user_by_api_token(api_token) + assert_receive %Phoenix.Socket.Broadcast{event: "disconnect"} + assert_receive %Phoenix.Socket.Broadcast{event: "disconnect"} + end + + test "changes nothing when given the password the user already has", %{ + conn: conn, + user: user + } do + session = Accounts.generate_user_session_token(user) + watch_sockets(user) + + for _ <- 1..2 do + assert conn + |> patch(~p"/api/users/#{user.id}", %{password: "hello world!"}) + |> json_response(200) + end + + assert Accounts.get_user_by_session_token(session) + assert Accounts.get_user(user.id).hashed_password == user.hashed_password + assert %{entries: []} = Lightning.Auditing.list_all() + refute_received %Phoenix.Socket.Broadcast{event: "disconnect"} + end + + test "refuses a password bcrypt would read as the current one", %{ + conn: conn, + user: user + } do + conn + |> patch(~p"/api/users/#{user.id}", %{password: "hello world!\0anything"}) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/password", + "code" => "format", + "detail" => "can't contain a NUL character" + } + ]) + end + + test "signs the user out when its role changes", %{conn: conn, user: user} do + session = Accounts.generate_user_session_token(user) + + assert conn + |> patch(~p"/api/users/#{user.id}", %{role: "superuser"}) + |> json_response(200) + + refute Accounts.get_user_by_session_token(session) + end + + test "keeps the sessions when only the names change", %{ + conn: conn, + user: user + } do + session = Accounts.generate_user_session_token(user) + watch_sockets(user) + + assert conn + |> patch(~p"/api/users/#{user.id}", %{ + first_name: "Grace", + role: "user" + }) + |> json_response(200) + + assert Accounts.get_user_by_session_token(session) + refute_received %Phoenix.Socket.Broadcast{event: "disconnect"} + end + + test "confirms an unconfirmed user and never unconfirms one", %{ + conn: conn, + user: user + } do + assert conn + |> patch(~p"/api/users/#{user.id}", %{confirmed: false}) + |> json_response(200) + + refute Accounts.get_user(user.id).confirmed_at + + assert conn + |> patch(~p"/api/users/#{user.id}", %{confirmed: true}) + |> json_response(200) + + assert %{confirmed_at: confirmed_at} = Accounts.get_user(user.id) + assert confirmed_at + + confirmed = + insert(:user, confirmed_at: ~U[2020-01-01 00:00:00Z]) + + for confirmed_param <- [true, false] do + assert conn + |> patch(~p"/api/users/#{confirmed.id}", %{ + confirmed: confirmed_param + }) + |> json_response(200) + + assert Accounts.get_user(confirmed.id).confirmed_at == + ~U[2020-01-01 00:00:00Z] + end + end + + test "refuses a key beyond the five it takes, the email included, with the body's other errors, changing nothing", + %{conn: conn, user: user} do + unknown = fn key -> + %{ + "pointer" => "#/" <> key, + "code" => "unknown_key", + "detail" => "is not accepted" + } + end + + for {body, entries} <- [ + {%{email: "new@example.com"}, [unknown.("email")]}, + {%{first_name: "Grace", password: "short", disabled: true}, + [ + unknown.("disabled"), + %{ + "pointer" => "#/password", + "code" => "too_short", + "detail" => "should be at least 12 character(s)" + } + ]} + ] do + conn + |> patch(~p"/api/users/#{user.id}", body) + |> assert_problem(422, :invalid, entries) + end + + assert %User{first_name: "Ada", email: "patched@example.com"} = + Accounts.get_user(user.id) + + assert Accounts.get_user_by_email_and_password(user.email, "hello world!") + assert %{entries: []} = Lightning.Auditing.list_all() + end + + test "answers 422 with an entry at each field that breaks a rule, changing nothing", + %{conn: conn, user: user} do + conn + |> patch(~p"/api/users/#{user.id}", %{ + first_name: " ", + last_name: "a\0b", + password: "short", + role: "admin", + confirmed: "perhaps" + }) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/first_name", + "code" => "required", + "detail" => "can't be blank" + }, + %{ + "pointer" => "#/last_name", + "code" => "format", + "detail" => "can't contain control characters" + }, + %{ + "pointer" => "#/password", + "code" => "too_short", + "detail" => "should be at least 12 character(s)" + }, + %{ + "pointer" => "#/role", + "code" => "not_allowed", + "detail" => "is invalid" + }, + %{"pointer" => "#/confirmed", "code" => "type", "detail" => "is invalid"} + ]) + + assert %User{first_name: "Ada", last_name: "Lovelace", role: :user} = + Accounts.get_user(user.id) + + assert Accounts.get_user_by_email_and_password(user.email, "hello world!") + assert %{entries: []} = Lightning.Auditing.list_all() + end + + test "answers 422 for a blank last name and a NUL character in a password, changing nothing", + %{conn: conn, user: user} do + conn + |> patch(~p"/api/users/#{user.id}", %{ + last_name: "", + password: "abcdefghijkl\0anything" + }) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/last_name", + "code" => "required", + "detail" => "can't be blank" + }, + %{ + "pointer" => "#/password", + "code" => "format", + "detail" => "can't contain a NUL character" + } + ]) + + assert %User{first_name: "Ada", last_name: "Lovelace", role: :user} = + Accounts.get_user(user.id) + + assert Accounts.get_user_by_email_and_password(user.email, "hello world!") + assert %{entries: []} = Lightning.Auditing.list_all() + end + + test "answers 404 for a user that doesn't exist", %{conn: conn} do + conn + |> patch(~p"/api/users/#{Ecto.UUID.generate()}", %{first_name: "A"}) + |> assert_problem(404, :not_found, []) + + conn + |> patch(~p"/api/users/not-a-uuid", %{first_name: "A"}) + |> assert_problem(404, :not_found, []) + end + + test "records what changed, with the service account as the actor", %{ + conn: conn, + account: account, + user: user + } do + assert conn + |> patch(~p"/api/users/#{user.id}", %{ + first_name: "Grace", + last_name: "Lovelace", + password: password() + }) + |> json_response(200) + + assert %{entries: [audit]} = Lightning.Auditing.list_all() + + assert %{ + item_type: "user", + event: "updated", + item_id: item_id, + actor_type: :service_account, + metadata: %{"password_changed" => true} + } = audit + + assert item_id == user.id + assert audit.actor_id == account.uuid + assert audit.changes.before == %{"first_name" => "Ada"} + assert audit.changes.after == %{"first_name" => "Grace"} + refute inspect(audit) =~ password() + refute inspect(audit) =~ "$2b$" + end + + test "records nothing when nothing changes", %{conn: conn, user: user} do + assert conn + |> patch(~p"/api/users/#{user.id}", %{ + first_name: "Ada", + role: "user", + confirmed: false + }) + |> json_response(200) + + assert %{entries: []} = Lightning.Auditing.list_all() + end + end + + describe "superuser changes" do + setup %{conn: conn, account: account} do + ref = + :telemetry_test.attach_event_handlers(self(), [ + [:lightning, :accounts, :superuser_changed] + ]) + + on_exit(fn -> :telemetry.detach(ref) end) + + %{conn: with_scopes(conn, account, ["users:write"]), ref: ref} + end + + defp create_user(conn, role) do + conn + |> post(~p"/api/users", %{ + email: "#{role}-#{System.unique_integer([:positive])}@example.com", + password: password(), + first_name: "Some", + last_name: "One", + role: role + }) + |> json_response(201) + |> get_in(["data", "id"]) + end + + defp assert_superuser_change(ref, change, user_id, account) do + assert_receive {[:lightning, :accounts, :superuser_changed], ^ref, + %{count: 1}, + %{ + change: ^change, + user_id: ^user_id, + actor_type: :service_account, + actor_id: actor_id + }} + + assert actor_id == account.id + end + + test "report a superuser created", %{conn: conn, account: account, ref: ref} do + id = create_user(conn, "superuser") + + assert_superuser_change(ref, :created, id, account) + end + + test "report a user made a superuser", %{ + conn: conn, + account: account, + ref: ref + } do + user = insert(:user) + + conn + |> patch(~p"/api/users/#{user.id}", %{role: "superuser"}) + |> json_response(200) + + assert_superuser_change(ref, :granted, user.id, account) + end + + test "report a superuser's password changing", %{ + conn: conn, + account: account, + ref: ref + } do + user = insert(:user, role: :superuser) + + conn + |> patch(~p"/api/users/#{user.id}", %{password: password()}) + |> json_response(200) + + assert_superuser_change(ref, :password_changed, user.id, account) + end + + test "say nothing for an ordinary user, or a superuser's names", %{ + conn: conn, + account: account, + ref: ref + } do + user = insert(:user) + superuser = insert(:user, role: :superuser) + + create_user(conn, "user") + + conn + |> patch(~p"/api/users/#{user.id}", %{password: password()}) + |> json_response(200) + + conn + |> patch(~p"/api/users/#{superuser.id}", %{first_name: "Renamed"}) + |> json_response(200) + + # The handler hears every test running at the same time, so only this + # test's own service account counts. + actor_id = account.id + + refute_received {[:lightning, :accounts, :superuser_changed], ^ref, _, + %{actor_id: ^actor_id}} + end + end +end diff --git a/test/lightning_web/controllers/api/work_orders_controller_test.exs b/test/lightning_web/controllers/api/work_orders_controller_test.exs index 453bd843400..40119e630a9 100644 --- a/test/lightning_web/controllers/api/work_orders_controller_test.exs +++ b/test/lightning_web/controllers/api/work_orders_controller_test.exs @@ -1,711 +1,418 @@ defmodule LightningWeb.API.WorkOrdersControllerTest do use LightningWeb.ConnCase, async: true + import LightningWeb.API.ProblemHelpers + import LightningWeb.ApiReadHelpers import Lightning.Factories setup %{conn: conn} do {:ok, conn: put_req_header(conn, "accept", "application/json")} end - test "without a token", %{conn: conn} do - conn = get(conn, ~p"/api/work_orders") - - assert %{"error" => "Unauthorized"} == json_response(conn, 401) - end - - describe "with invalid token" do - test "gets a 401", %{conn: conn} do - token = "InvalidToken" - conn = conn |> Plug.Conn.put_req_header("authorization", "Bearer #{token}") - conn = get(conn, ~p"/api/work_orders") - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end + defp ids(conn) do + conn |> json_response(200) |> Map.fetch!("data") |> Enum.map(& &1["id"]) end describe "index" do setup [:assign_bearer_for_api, :create_project_for_current_user] - test "lists work orders for projects I have access to", %{ + test "lists the work orders of the projects the caller can reach", %{ conn: conn, project: project } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - workorder1 = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip) - ) - - workorder2 = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip) - ) - - # Create a work order in another project (should not be accessible) - other_project = insert(:project) - other_workflow = insert(:workflow, project: other_project) - other_trigger = insert(:trigger, workflow: other_workflow) - - other_workorder = - insert(:workorder, - workflow: other_workflow, - trigger: other_trigger, - dataclip: build(:dataclip) - ) - - conn = get(conn, ~p"/api/work_orders") - - response = json_response(conn, 200) - - # Should only return work orders from my project - assert length(response["data"]) == 2 - - workorder_ids = Enum.map(response["data"], & &1["id"]) - assert workorder1.id in workorder_ids - assert workorder2.id in workorder_ids - refute other_workorder.id in workorder_ids - - # Verify pagination metadata - assert response["meta"]["total_entries"] == 2 - assert response["meta"]["page_number"] == 1 + older = + insert_work_order(project, + state: :failed, + inserted_at: ~U[2024-01-01 10:00:00.000000Z], + updated_at: ~U[2024-01-01 10:30:00.000000Z], + last_activity: ~U[2024-01-01 10:20:00.000000Z] + ) + + newer = + insert_work_order(project, + state: :success, + inserted_at: ~U[2024-01-01 12:00:00.000000Z], + updated_at: ~U[2024-01-01 12:30:00.000000Z], + last_activity: ~U[2024-01-01 12:20:00.000000Z] + ) + + insert_work_order(insert(:project)) + + url = LightningWeb.Endpoint.url() + page = "#{url}/api/work_orders?page=1&page_size=10" + + assert conn |> get(~p"/api/work_orders") |> json_response(200) == %{ + "data" => + for {work_order, state, last_activity, inserted_at, updated_at} <- + [ + {newer, "success", "2024-01-01T12:20:00.000000Z", + "2024-01-01T12:00:00.000000Z", + "2024-01-01T12:30:00.000000Z"}, + {older, "failed", "2024-01-01T10:20:00.000000Z", + "2024-01-01T10:00:00.000000Z", + "2024-01-01T10:30:00.000000Z"} + ] do + %{ + "type" => "work_orders", + "id" => work_order.id, + "relationships" => %{}, + "links" => %{ + "self" => "#{url}/api/work_orders/#{work_order.id}" + }, + "attributes" => %{ + "state" => state, + "last_activity" => last_activity, + "inserted_at" => inserted_at, + "updated_at" => updated_at + } + } + end, + "included" => [], + "meta" => %{ + "total_entries" => 2, + "total_pages" => 1, + "page_number" => 1, + "page_size" => 10 + }, + "links" => %{ + "self" => page, + "first" => page, + "last" => page, + "next" => nil, + "prev" => nil + } + } end - test "filters work orders by inserted_after", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - old_time = ~U[2024-01-01 10:00:00Z] - new_time = ~U[2024-01-01 12:00:00Z] - - # Create old work order - _old_workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - inserted_at: old_time - ) - - # Create new work order - new_workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - inserted_at: new_time - ) + test "narrows by inserted_after", %{conn: conn, project: project} do + insert_work_order(project, inserted_at: ~U[2024-01-01 10:00:00Z]) + new = insert_work_order(project, inserted_at: ~U[2024-01-01 12:00:00Z]) conn = get(conn, ~p"/api/work_orders?inserted_after=2024-01-01T11:00:00Z") - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == new_workorder.id + assert ids(conn) == [new.id] end - test "filters work orders by inserted_before", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - old_time = ~U[2024-01-01 10:00:00Z] - new_time = ~U[2024-01-01 12:00:00Z] - - # Create old work order - old_workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - inserted_at: old_time - ) - - # Create new work order - _new_workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - inserted_at: new_time - ) + test "narrows by inserted_before", %{conn: conn, project: project} do + old = insert_work_order(project, inserted_at: ~U[2024-01-01 10:00:00Z]) + insert_work_order(project, inserted_at: ~U[2024-01-01 12:00:00Z]) conn = get(conn, ~p"/api/work_orders?inserted_before=2024-01-01T11:00:00Z") - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == old_workorder.id + assert ids(conn) == [old.id] end - test "combines multiple filters", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - old_time = ~U[2024-01-01 10:00:00Z] - mid_time = ~U[2024-01-01 12:00:00Z] - new_time = ~U[2024-01-01 14:00:00Z] - - # Create work orders at different times - _old_workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - inserted_at: old_time - ) - - mid_workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - inserted_at: mid_time - ) - - _new_workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - inserted_at: new_time - ) - - conn = - get( - conn, - ~p"/api/work_orders?inserted_after=2024-01-01T11:00:00Z&inserted_before=2024-01-01T13:00:00Z" - ) - - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == mid_workorder.id - end - - test "supports pagination", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - # Create 25 work orders - Enum.each(1..25, fn i -> - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - inserted_at: DateTime.utc_now() |> DateTime.add(i, :second) - ) - end) - - # Get first page - conn = get(conn, ~p"/api/work_orders?page=1&page_size=10") - response = json_response(conn, 200) - - assert length(response["data"]) == 10 - assert response["meta"]["total_entries"] == 25 - assert response["meta"]["total_pages"] == 3 - assert response["meta"]["page_number"] == 1 - assert response["meta"]["page_size"] == 10 - - # Verify pagination links exist - assert Map.has_key?(response["links"], "first") - assert Map.has_key?(response["links"], "next") - assert Map.has_key?(response["links"], "last") - end - - test "filters work orders by project_id via query parameter", %{ - conn: conn, - user: user - } do - # User has access to both projects + test "narrows by project_id", %{conn: conn, user: user} do project1 = insert(:project, project_users: [%{user: user, role: :owner}]) project2 = insert(:project, project_users: [%{user: user, role: :owner}]) - - workflow1 = insert(:workflow, project: project1) - workflow2 = insert(:workflow, project: project2) - trigger1 = insert(:trigger, workflow: workflow1) - trigger2 = insert(:trigger, workflow: workflow2) - - workorder1 = - insert(:workorder, - workflow: workflow1, - trigger: trigger1, - dataclip: build(:dataclip) - ) - - workorder2 = - insert(:workorder, - workflow: workflow2, - trigger: trigger2, - dataclip: build(:dataclip) - ) + work_order1 = insert_work_order(project1) + insert_work_order(project2) conn = get(conn, ~p"/api/work_orders?project_id=#{project1.id}") - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == workorder1.id - refute Enum.any?(response["data"], fn wo -> wo["id"] == workorder2.id end) + assert ids(conn) == [work_order1.id] end - test "filters work orders by workflow_id via query parameter", %{ - conn: conn, - project: project - } do - workflow1 = insert(:workflow, project: project) - workflow2 = insert(:workflow, project: project) - trigger1 = insert(:trigger, workflow: workflow1) - trigger2 = insert(:trigger, workflow: workflow2) - - workorder1 = - insert(:workorder, - workflow: workflow1, - trigger: trigger1, - dataclip: build(:dataclip) - ) + test "narrows by workflow_id", %{conn: conn, project: project} do + work_order1 = insert_work_order(project) + insert_work_order(project) - workorder2 = - insert(:workorder, - workflow: workflow2, - trigger: trigger2, - dataclip: build(:dataclip) - ) - - conn = get(conn, ~p"/api/work_orders?workflow_id=#{workflow1.id}") - - response = json_response(conn, 200) + conn = + get(conn, ~p"/api/work_orders?workflow_id=#{work_order1.workflow_id}") - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == workorder1.id - refute Enum.any?(response["data"], fn wo -> wo["id"] == workorder2.id end) + assert ids(conn) == [work_order1.id] end - test "nested route: lists work orders for specific project", %{ + test "narrows by one state, or by several comma-separated", %{ conn: conn, - user: user + project: project } do - # User has access to both projects - project1 = insert(:project, project_users: [%{user: user, role: :owner}]) - project2 = insert(:project, project_users: [%{user: user, role: :owner}]) - - workflow1 = insert(:workflow, project: project1) - workflow2 = insert(:workflow, project: project2) - trigger1 = insert(:trigger, workflow: workflow1) - trigger2 = insert(:trigger, workflow: workflow2) - - workorder1 = - insert(:workorder, - workflow: workflow1, - trigger: trigger1, - dataclip: build(:dataclip) - ) - - workorder2 = - insert(:workorder, - workflow: workflow2, - trigger: trigger2, - dataclip: build(:dataclip) - ) + pending = insert_work_order(project, state: :pending) + failed = insert_work_order(project, state: :failed) + insert_work_order(project, state: :success) - conn = get(conn, ~p"/api/projects/#{project1.id}/work_orders") + assert ids(get(conn, ~p"/api/work_orders?state=pending")) == [pending.id] - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == workorder1.id - refute Enum.any?(response["data"], fn wo -> wo["id"] == workorder2.id end) + assert conn + |> get(~p"/api/work_orders?state=pending,failed") + |> ids() + |> Enum.sort() == Enum.sort([pending.id, failed.id]) end - test "nested route: filters work orders by datetime in project scope", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - old_time = ~U[2024-01-01 10:00:00Z] - new_time = ~U[2024-01-01 12:00:00Z] - - # Create old work order - _old_workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - inserted_at: old_time - ) + test "filters work orders by a single id", %{conn: conn, project: project} do + work_order1 = insert_work_order(project) + insert_work_order(project) - # Create new work order - new_workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - inserted_at: new_time - ) + conn = get(conn, ~p"/api/work_orders?id=#{work_order1.id}") - conn = - get( - conn, - ~p"/api/projects/#{project.id}/work_orders?inserted_after=2024-01-01T11:00:00Z" - ) - - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == new_workorder.id + assert ids(conn) == [work_order1.id] end - test "returns error for invalid inserted_after format", %{ + test "filters work orders by an array of ids", %{ conn: conn, project: project } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip) - ) + work_order1 = insert_work_order(project) + work_order2 = insert_work_order(project) + insert_work_order(project) conn = get( conn, - ~p"/api/work_orders?inserted_after=not-a-valid-datetime" + "/api/work_orders?id[]=#{work_order1.id}&id[]=#{work_order2.id}" ) - response = json_response(conn, 400) - - assert %{"error" => error_message} = response - assert error_message =~ "Invalid datetime format for 'inserted_after'" - assert error_message =~ "not-a-valid-datetime" + assert Enum.sort(ids(conn)) == Enum.sort([work_order1.id, work_order2.id]) end - test "returns error for invalid updated_before format", %{ + test "filters work orders by comma-separated ids", %{ conn: conn, project: project } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip) - ) + work_order1 = insert_work_order(project) + work_order2 = insert_work_order(project) + insert_work_order(project) conn = - get( - conn, - ~p"/api/work_orders?updated_before=2024-13-45" - ) - - response = json_response(conn, 400) + get(conn, "/api/work_orders?id=#{work_order1.id},#{work_order2.id}") - assert %{"error" => error_message} = response - assert error_message =~ "Invalid datetime format for 'updated_before'" - assert error_message =~ "2024-13-45" + assert Enum.sort(ids(conn)) == Enum.sort([work_order1.id, work_order2.id]) end - test "returns error for invalid datetime with integer", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) + test "narrows by every filter at once", %{conn: conn, project: project} do + mid = + insert_work_order(project, + state: :failed, + inserted_at: ~U[2024-01-01 12:00:00Z] + ) - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip) + insert_work_order(project, + state: :failed, + inserted_at: ~U[2024-01-01 10:00:00Z] ) - conn = get(conn, ~p"/api/work_orders?inserted_after=123456") - - response = json_response(conn, 400) - - assert %{"error" => error_message} = response - assert error_message =~ "Invalid datetime format for 'inserted_after'" - assert error_message =~ "123456" - end + insert_work_order(project, + state: :failed, + inserted_at: ~U[2024-01-01 14:00:00Z] + ) - test "filters work orders by state", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - pending_wo = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - state: :pending - ) + insert_work_order(project, + state: :pending, + inserted_at: ~U[2024-01-01 12:00:00Z] + ) - _failed_wo = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - state: :failed + conn = + get( + conn, + ~p"/api/work_orders?inserted_after=2024-01-01T11:00:00Z&inserted_before=2024-01-01T13:00:00Z&state=failed" ) - conn = get(conn, ~p"/api/work_orders?state=pending") - - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == pending_wo.id + assert ids(conn) == [mid.id] end - test "filters work orders by multiple comma-separated states", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - pending_wo = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - state: :pending - ) + test "lists one project's work orders on the nested route, narrowed by the same filters", + %{conn: conn, user: user} do + project1 = insert(:project, project_users: [%{user: user, role: :owner}]) + project2 = insert(:project, project_users: [%{user: user, role: :owner}]) + insert_work_order(project1, inserted_at: ~U[2024-01-01 10:00:00Z]) + new = insert_work_order(project1, inserted_at: ~U[2024-01-01 12:00:00Z]) + insert_work_order(project2, inserted_at: ~U[2024-01-01 12:00:00Z]) - failed_wo = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - state: :failed - ) + assert conn + |> get(~p"/api/projects/#{project1.id}/work_orders") + |> ids() + |> length() == 2 - _success_wo = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip), - state: :success + conn = + get( + conn, + ~p"/api/projects/#{project1.id}/work_orders?inserted_after=2024-01-01T11:00:00Z" ) - conn = get(conn, ~p"/api/work_orders?state=pending,failed") - - response = json_response(conn, 200) - - assert length(response["data"]) == 2 - ids = Enum.map(response["data"], & &1["id"]) - assert pending_wo.id in ids - assert failed_wo.id in ids + assert ids(conn) == [new.id] end - test "returns error for invalid state filter", %{ + test "pages with exact links to the neighbouring pages", %{ conn: conn, project: project } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip) - ) - - conn = get(conn, ~p"/api/work_orders?state=bogus") - - response = json_response(conn, 400) - - assert %{"error" => error_message} = response - assert error_message =~ "Invalid state filter" - assert error_message =~ "bogus" + for n <- 1..25 do + insert_work_order(project, + inserted_at: DateTime.add(~U[2024-01-01 00:00:00Z], n) + ) + end + + url = "#{LightningWeb.Endpoint.url()}/api/work_orders" + + assert %{"data" => data, "meta" => meta, "links" => links} = + conn + |> get(~p"/api/work_orders?page=2&page_size=10") + |> json_response(200) + + assert length(data) == 10 + + assert meta == %{ + "total_entries" => 25, + "total_pages" => 3, + "page_number" => 2, + "page_size" => 10 + } + + assert links == %{ + "self" => "#{url}?page=2&page_size=10", + "first" => "#{url}?page=1&page_size=10", + "last" => "#{url}?page=3&page_size=10", + "next" => "#{url}?page=3&page_size=10", + "prev" => "#{url}?page=1&page_size=10" + } end - test "filters work orders by a single id", %{ + test "pagination links carry the filters applied and nothing else", %{ conn: conn, project: project } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - workorder1 = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip) - ) - - _workorder2 = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip) - ) - - conn = get(conn, ~p"/api/work_orders?id=#{workorder1.id}") - - response = json_response(conn, 200) - - assert length(response["data"]) == 1 - assert List.first(response["data"])["id"] == workorder1.id + for {path, request_path} <- [ + {~p"/api/work_orders?page_size=1&foo=bar&state=failed,crashed&inserted_after=2024-01-01T02:00:00%2B02:00", + "/api/work_orders"}, + {~p"/api/projects/#{project.id}/work_orders?page_size=1&foo=bar&state=failed,crashed&inserted_after=2024-01-01T02:00:00%2B02:00", + "/api/projects/#{project.id}/work_orders"} + ] do + %{"links" => links} = conn |> get(path) |> json_response(200) + + for key <- ["self", "first"] do + uri = URI.parse(links[key]) + assert uri.path == request_path + + assert URI.decode_query(uri.query) == %{ + "inserted_after" => "2024-01-01T00:00:00.000000Z", + "page" => "1", + "page_size" => "1", + "state" => "failed,crashed" + } + end + end end - test "filters work orders by an array of ids", %{ + test "every bad query parameter is reported, none quoted", %{ conn: conn, project: project } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - workorder1 = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip) - ) - - workorder2 = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip) - ) - - _workorder3 = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip) - ) - - conn = - get( - conn, - "/api/work_orders?id[]=#{workorder1.id}&id[]=#{workorder2.id}" - ) - - response = json_response(conn, 200) - - assert length(response["data"]) == 2 - ids = Enum.map(response["data"], & &1["id"]) - assert workorder1.id in ids - assert workorder2.id in ids - end - - test "filters work orders by comma-separated ids", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - workorder1 = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip) - ) - - workorder2 = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip) - ) - - _workorder3 = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip) - ) - - conn = - get( - conn, - "/api/work_orders?id=#{workorder1.id},#{workorder2.id}" - ) - - response = json_response(conn, 200) - - assert length(response["data"]) == 2 - ids = Enum.map(response["data"], & &1["id"]) - assert workorder1.id in ids - assert workorder2.id in ids + bad = plant_sentinel(%{"v" => "x"})["v"] + + query = + URI.encode_query(%{ + "id" => bad, + "inserted_after" => bad, + "inserted_before" => bad, + "updated_after" => bad, + "updated_before" => bad, + "workflow_id" => bad, + "state" => "pending,#{bad}" + }) + + for path <- [ + "/api/work_orders?" <> query, + "/api/projects/#{project.id}/work_orders?" <> query + ] do + conn + |> get(path) + |> refute_sentinel() + |> assert_problem(400, :invalid_parameter, [ + %{"parameter" => "id", "code" => "type", "detail" => "is invalid"}, + %{ + "parameter" => "inserted_after", + "code" => "type", + "detail" => "is invalid" + }, + %{ + "parameter" => "inserted_before", + "code" => "type", + "detail" => "is invalid" + }, + %{ + "parameter" => "updated_after", + "code" => "type", + "detail" => "is invalid" + }, + %{ + "parameter" => "updated_before", + "code" => "type", + "detail" => "is invalid" + }, + %{ + "parameter" => "workflow_id", + "code" => "type", + "detail" => "is invalid" + }, + %{ + "parameter" => "state", + "code" => "not_allowed", + "detail" => "has an invalid entry" + } + ]) + end end end describe "show" do setup [:assign_bearer_for_api, :create_project_for_current_user] - test "returns a single work order by id", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project) - trigger = insert(:trigger, workflow: workflow) - - workorder = - insert(:workorder, - workflow: workflow, - trigger: trigger, - dataclip: build(:dataclip) - ) - - conn = get(conn, ~p"/api/work_orders/#{workorder}") - - response = json_response(conn, 200) - - assert response["data"]["id"] == workorder.id - assert response["data"]["type"] == "work_orders" - assert Map.has_key?(response["data"]["attributes"], "state") - assert Map.has_key?(response["data"]["attributes"], "last_activity") - assert Map.has_key?(response["data"]["attributes"], "inserted_at") - assert Map.has_key?(response["data"]["attributes"], "updated_at") - end - - test "returns 404 for non-existent work order", %{conn: conn} do - conn = get(conn, ~p"/api/work_orders/#{Ecto.UUID.generate()}") - - assert json_response(conn, 404) + test "answers the work order, and links to it without the request's query", + %{conn: conn, project: project} do + work_order = + insert_work_order(project, + state: :success, + last_activity: ~U[2024-01-01 11:00:00.000000Z], + inserted_at: ~U[2024-01-01 10:00:00.000000Z], + updated_at: ~U[2024-01-01 12:00:00.000000Z] + ) + + url = LightningWeb.Endpoint.url() + + assert conn + |> get(~p"/api/work_orders/#{work_order}?foo=bar") + |> json_response(200) == %{ + "data" => %{ + "type" => "work_orders", + "id" => work_order.id, + "relationships" => %{}, + "links" => %{ + "self" => "#{url}/api/work_orders/#{work_order.id}" + }, + "attributes" => %{ + "state" => "success", + "last_activity" => "2024-01-01T11:00:00.000000Z", + "inserted_at" => "2024-01-01T10:00:00.000000Z", + "updated_at" => "2024-01-01T12:00:00.000000Z" + } + }, + "included" => [], + "links" => %{"self" => "#{url}/api/work_orders/#{work_order.id}"} + } end + end - test "returns 401 for work order in project user cannot access", %{ - conn: conn - } do - other_project = insert(:project) - other_workflow = insert(:workflow, project: other_project) - other_trigger = insert(:trigger, workflow: other_workflow) - - other_workorder = - insert(:workorder, - workflow: other_workflow, - trigger: other_trigger, - dataclip: build(:dataclip) - ) - - conn = get(conn, ~p"/api/work_orders/#{other_workorder}") + describe "who may reach it" do + setup [:assign_bearer_for_api, :create_project_for_current_user] - assert json_response(conn, 401) + test "answers 404 for what the caller has no standing on or that is missing, before the query is read, and 403 to a member who has not met the MFA requirement", + %{conn: conn, project: project} do + own = insert_work_order(project) + elsewhere = insert(:project) + unseen = insert_work_order(elsewhere) + Lightning.Repo.update!(Ecto.Changeset.change(project, requires_mfa: true)) + + for {path, status, code} <- [ + {~p"/api/projects/#{elsewhere.id}/work_orders?state=sideways", 404, + :not_found}, + {~p"/api/projects/#{Ecto.UUID.generate()}/work_orders", 404, + :not_found}, + {~p"/api/work_orders/#{unseen.id}", 404, :not_found}, + {~p"/api/work_orders/#{Ecto.UUID.generate()}", 404, :not_found}, + {~p"/api/work_orders/#{own.id}", 403, :forbidden} + ] do + conn |> get(path) |> assert_problem(status, code, []) + end end end end diff --git a/test/lightning_web/controllers/api/workflow_health_controller_test.exs b/test/lightning_web/controllers/api/workflow_health_controller_test.exs index 441270072c6..f166911a967 100644 --- a/test/lightning_web/controllers/api/workflow_health_controller_test.exs +++ b/test/lightning_web/controllers/api/workflow_health_controller_test.exs @@ -2,6 +2,7 @@ defmodule LightningWeb.API.WorkflowHealthControllerTest do use LightningWeb.ConnCase, async: true import Lightning.Factories + import LightningWeb.API.ProblemHelpers alias Lightning.Workflows.Snapshot @@ -38,19 +39,19 @@ defmodule LightningWeb.API.WorkflowHealthControllerTest do user, project_id, workflow_id, - params \\ %{}, + params, headers \\ [] ) do - Enum.reduce(headers, log_in_user(conn, user), fn {name, value}, conn -> - put_req_header(conn, name, value) - end) + conn + |> log_in_user(user) + |> Map.update!(:req_headers, &(headers ++ &1)) |> get( ~p"/api/projects/#{project_id}/workflows/#{workflow_id}/health/runs", params ) end - describe "authorization" do + describe "who may reach it" do test "a project member is served", %{ conn: conn, user: user, @@ -60,62 +61,28 @@ defmodule LightningWeb.API.WorkflowHealthControllerTest do assert %{status: 200} = get_outcomes(conn, user, project.id, workflow.id) end - test "an anonymous request is refused", %{ - conn: conn, - project: project, - workflow: workflow - } do - conn = - get( - conn, - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}/health/outcomes" - ) - - assert conn.status == 401 - end - - test "a non-member is refused", %{ - conn: conn, - project: project, - workflow: workflow - } do - assert %{status: 404} = - get_outcomes(conn, insert(:user), project.id, workflow.id) - end - - # :validate_days runs after :authorize_workflow, so a bad `days` value - # never gets the chance to produce a 400 for someone who can't see the - # workflow in the first place. - test "a non-member is refused with 404, not 400, even with an invalid days value", - %{ - conn: conn, - project: project, - workflow: workflow - } do - assert %{status: 404} = - get_outcomes(conn, insert(:user), project.id, workflow.id, %{ - "days" => "banana" - }) - end - - test "a member of another project cannot read this workflow", %{ - conn: conn, - user: user, - project: project - } do + # `days` is read only once the caller is authorised, so a bad value never + # gets the chance to produce a 400 for someone who can't see the workflow. + test "a caller with no standing on the project or workflow gets 404 on every slice, even with a bad days", + %{conn: conn, user: user, project: project, workflow: workflow} do + stranger = insert(:user) other_workflow = insert(:simple_workflow) - assert %{status: 404} = - get_outcomes(conn, user, project.id, other_workflow.id) - - # Nor by naming the other workflow's own project, which they aren't in. - assert %{status: 404} = - get_outcomes( - conn, - user, - other_workflow.project_id, - other_workflow.id - ) + for {get_slice, caller, project_id, workflow_id, params} <- [ + {&get_outcomes/5, stranger, project.id, workflow.id, %{}}, + {&get_outcomes/5, stranger, project.id, workflow.id, + %{"days" => "banana"}}, + {&get_failures/5, stranger, project.id, workflow.id, %{}}, + {&get_runs(&1, &2, &3, &4, &5), stranger, project.id, workflow.id, + %{}}, + {&get_outcomes/5, user, project.id, other_workflow.id, %{}}, + {&get_outcomes/5, user, other_workflow.project_id, other_workflow.id, + %{}} + ] do + refused = get_slice.(conn, caller, project_id, workflow_id, params) + + assert_problem(refused, 404, :not_found, []) + end end # Support users have no `project_users` row, so a membership check alone @@ -137,13 +104,15 @@ defmodule LightningWeb.API.WorkflowHealthControllerTest do ) # Not a blanket pass: a project that hasn't opted in still refuses them. - assert %{status: 404} = - get_outcomes( - conn, - support_user, - workflow.project_id, - workflow.id - ) + refused = + get_outcomes( + conn, + support_user, + workflow.project_id, + workflow.id + ) + + assert_problem(refused, 404, :not_found, []) end test "a member cannot read a project scheduled for deletion", %{ @@ -156,11 +125,14 @@ defmodule LightningWeb.API.WorkflowHealthControllerTest do |> Ecto.Changeset.change(scheduled_deletion: ~U[2026-01-01 00:00:00Z]) |> Lightning.Repo.update!() - assert %{status: 404} = get_outcomes(conn, user, project.id, workflow.id) + refused = + get_outcomes(conn, user, project.id, workflow.id) + + assert_problem(refused, 409, :scheduled_for_deletion, []) end - # The LiveView redirects this person to `/mfa_required`; a JSON client has - # nothing to do with that, and 404 doesn't confirm the project exists. + # The LiveView redirects this person to `/mfa_required`. A JSON client gets + # 403 forbidden: it can see the project but hasn't met its MFA requirement. test "a member who has not enrolled is refused a project requiring MFA", %{ conn: conn } do @@ -178,8 +150,10 @@ defmodule LightningWeb.API.WorkflowHealthControllerTest do workflow = insert(:simple_workflow, project: project) - assert %{status: 404} = - get_outcomes(conn, unenrolled, project.id, workflow.id) + refused = + get_outcomes(conn, unenrolled, project.id, workflow.id) + + assert_problem(refused, 403, :forbidden, []) # Not unreadable in general: an enrolled member is still served. assert %{status: 200} = @@ -198,34 +172,32 @@ defmodule LightningWeb.API.WorkflowHealthControllerTest do |> Ecto.Changeset.change(deleted_at: ~U[2026-01-01 00:00:00Z]) |> Lightning.Repo.update!() - assert %{status: 404} = get_outcomes(conn, user, project.id, workflow.id) - assert %{status: 404} = get_failures(conn, user, project.id, workflow.id) - end + refused = + get_outcomes(conn, user, project.id, workflow.id) - test "a malformed id is refused rather than crashing the request", %{ - conn: conn, - user: user, - project: project, - workflow: workflow - } do - assert %{status: 404} = get_outcomes(conn, user, "not-a-uuid", workflow.id) + assert_problem(refused, 404, :not_found, []) + + refused = + get_failures(conn, user, project.id, workflow.id) - assert %{status: 404} = get_outcomes(conn, user, project.id, "not-a-uuid") + assert_problem(refused, 404, :not_found, []) end - # Every action shares one plug, so this only has to prove the plug runs on - # the others too. - test "the other slices are guarded by the same check", %{ - conn: conn, - project: project, - workflow: workflow - } do - stranger = insert(:user) + test "a member who hasn't met MFA is told a missing workflow is missing" do + unenrolled = insert(:user, mfa_enabled: false) - assert %{status: 404} = - get_failures(conn, stranger, project.id, workflow.id) + project = + insert(:project, + requires_mfa: true, + project_users: [%{user: unenrolled, role: :owner}] + ) - assert %{status: 404} = get_runs(conn, stranger, project.id, workflow.id) + conn = + build_conn() + |> put_req_header("accept", "application/json") + |> get_outcomes(unenrolled, project.id, Ecto.UUID.generate()) + + assert_problem(conn, 404, :not_found, []) end end @@ -471,27 +443,53 @@ defmodule LightningWeb.API.WorkflowHealthControllerTest do assert get_resp_header(conn, "vary") == ["x-timezone"] end - # Unknown, empty, and long — the cache key has to stay bounded to the tz - # database whatever arrives. + # Unknown, empty, long and repeated: the cache key has to stay bounded to + # the tz database whatever arrives. test "rejects a value it cannot use", %{ conn: conn, user: user, project: project, workflow: workflow } do - for value <- ["Mars/Olympus", "", String.duplicate("x", 5_000)] do - response = - conn - |> get_runs(user, project.id, workflow.id, %{"days" => "1"}, [ - {"x-timezone", value} - ]) - |> json_response(400) - - assert response == %{ - "error" => "x-timezone must be an IANA timezone name" - } + for headers <- [ + [{"x-timezone", "Mars/Olympus"}], + [{"x-timezone", ""}], + [{"x-timezone", String.duplicate("x", 5_000)}], + [{"x-timezone", "Etc/UTC"}, {"x-timezone", "Etc/UTC"}], + [{"x-timezone", plant_sentinel("tz")}] + ] do + conn + |> get_runs(user, project.id, workflow.id, %{"days" => "1"}, headers) + |> refute_sentinel() + |> assert_problem(400, :invalid_header, [ + %{ + "header" => "x-timezone", + "code" => "not_allowed", + "detail" => "is invalid" + } + ]) end end + + test "a bad days is reported before a bad x-timezone", %{ + conn: conn, + user: user, + project: project, + workflow: workflow + } do + conn = + get_runs(conn, user, project.id, workflow.id, %{"days" => "2"}, [ + {"x-timezone", "Mars/Olympus"} + ]) + + assert_problem(conn, 400, :invalid_parameter, [ + %{ + "parameter" => "days", + "code" => "not_allowed", + "detail" => "is invalid" + } + ]) + end end describe "?days=" do @@ -536,16 +534,17 @@ defmodule LightningWeb.API.WorkflowHealthControllerTest do project: project, workflow: workflow } do - for bad_days <- ["banana", "36500", "0", "7.5", "007"] do - assert %{status: 400} = - get_outcomes(conn, user, project.id, workflow.id, %{ - "days" => bad_days - }) - - assert %{status: 400} = - get_failures(conn, user, project.id, workflow.id, %{ - "days" => bad_days - }) + for bad_days <- ["banana", "36500", "0", "7.5", "007", plant_sentinel("d")] do + conn + |> get_outcomes(user, project.id, workflow.id, %{"days" => bad_days}) + |> refute_sentinel() + |> assert_problem(400, :invalid_parameter, [ + %{ + "parameter" => "days", + "code" => "not_allowed", + "detail" => "is invalid" + } + ]) end end end diff --git a/test/lightning_web/controllers/api/workflows/creating_test.exs b/test/lightning_web/controllers/api/workflows/creating_test.exs new file mode 100644 index 00000000000..f822f4d92e8 --- /dev/null +++ b/test/lightning_web/controllers/api/workflows/creating_test.exs @@ -0,0 +1,440 @@ +defmodule LightningWeb.API.Workflows.CreatingTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProblemHelpers + import Lightning.Factories + import LightningWeb.ApiWorkflowsHelpers + + alias Lightning.Extensions.Message + + setup %{conn: conn} do + conn = + conn + |> put_req_header("accept", "application/json") + |> put_req_header("content-type", "application/json") + + {:ok, conn: conn} + end + + describe "POST /workflows/:project_id" do + setup [:assign_bearer_for_api, :create_project_for_current_user] + + test "inserts a workflow", %{conn: conn, project: project} do + workflow = + build(:simple_workflow, name: "work1", project_id: project.id) + + conn = + post( + conn, + ~p"/api/projects/#{project.id}/workflows/", + Jason.encode!(workflow) + ) + + assert %{"workflow" => %{"id" => id}} = body = json_response(conn, 201) + + assert %{ + edges: [edge], + jobs: [job], + triggers: [trigger] + } = + saved_workflow = get_saved_workflow(id) + + assert body == %{"workflow" => encode_decode(saved_workflow)} + + assert Map.take(hd(workflow.edges), [:condition_type, :enabled]) == + Map.take(edge, [:condition_type, :enabled]) + + assert Map.take(hd(workflow.jobs), [:name, :adaptor, :body]) == + Map.take(job, [:name, :adaptor, :body]) + + assert Map.take(hd(workflow.triggers), [:type, :enabled]) == + Map.take(trigger, [:type, :enabled]) + + assert Map.take(workflow, [:name, :project_id]) == + Map.take(saved_workflow, [:name, :project_id]) + end + + test "inserts a workflow with disconnected job", %{ + conn: conn, + project: project + } do + workflow = + build(:simple_workflow, name: "work1", project_id: project.id) + |> then(fn %{jobs: jobs} = workflow -> + %{workflow | jobs: [build(:job) | jobs]} + end) + + conn = + post( + conn, + ~p"/api/projects/#{project.id}/workflows/", + Jason.encode!(workflow) + ) + + assert %{"workflow" => response_workflow} = + json_response(conn, 201) + + assert saved_workflow = get_saved_workflow(response_workflow["id"]) + + assert_response(response_workflow) + + assert pluck_to_mapset(workflow.edges, [:condition_type, :enabled]) == + pluck_to_mapset(saved_workflow.edges, [:condition_type, :enabled]) + + assert pluck_to_mapset(workflow.jobs, [:name, :adaptor, :body]) == + pluck_to_mapset(saved_workflow.jobs, [:name, :adaptor, :body]) + + assert pluck_to_mapset(workflow.triggers, [:type, :enabled]) == + pluck_to_mapset(saved_workflow.triggers, [:type, :enabled]) + + assert Map.take(workflow, [:name, :project_id]) == + Map.take(saved_workflow, [:name, :project_id]) + end + + test "creates UUIDs on insert based on user arbitrary ids", %{ + conn: conn, + project: project + } do + workflow = + build(:complex_workflow, project_id: project.id) + |> then(fn %{ + jobs: [job1, job2 | jobs], + edges: edges, + triggers: [trigger] + } = workflow -> + old_job1_id = job1.id + old_job2_id = job2.id + job1 = Map.put(job1, :id, 11) + job2 = Map.put(job2, :id, 12) + old_trigger_id = trigger.id + trigger = Map.put(trigger, :id, "trigger1") + + ids_map = %{ + old_job1_id => 11, + old_job2_id => 12, + old_trigger_id => "trigger1" + } + + edges + |> Enum.map(fn edge -> + edge + |> Map.update(:source_trigger_id, nil, &Map.get(ids_map, &1, &1)) + |> Map.update(:source_job_id, nil, &Map.get(ids_map, &1, &1)) + |> Map.update(:target_job_id, nil, &Map.get(ids_map, &1, &1)) + end) + |> then(fn edges -> + %{ + workflow + | jobs: [job1, job2 | jobs], + edges: edges, + triggers: [trigger] + } + end) + end) + + assert %{ + "workflow" => response_workflow + } = + conn + |> post( + ~p"/api/projects/#{project.id}/workflows", + Jason.encode!(workflow) + ) + |> json_response(201) + + saved_workflow = get_saved_workflow(response_workflow["id"]) + + merged = encode_decode(response_workflow) |> remove_timestamps() + saved = encode_decode(saved_workflow) |> remove_timestamps() + + assert MapSet.new(merged["jobs"]) == MapSet.new(saved["jobs"]) + assert MapSet.new(merged["edges"]) == MapSet.new(saved["edges"]) + assert MapSet.new(merged["triggers"]) == MapSet.new(saved["triggers"]) + + assert Map.drop(merged, ["jobs", "edges", "triggers"]) == + Map.drop(saved, ["jobs", "edges", "triggers"]) + end + + test "returns 422 when an edge has invalid condition_type", %{ + conn: conn, + project: project + } do + workflow = + build(:simple_workflow, name: "work1", project_id: project.id) + |> then(fn %{edges: [edge | edges]} = workflow -> + %{workflow | edges: [%{edge | condition_type: "on_failures"} | edges]} + end) + + conn = + post( + conn, + ~p"/api/projects/#{project.id}/workflows/", + Jason.encode!(workflow) + ) + + assert_problem(conn, 422, :invalid, [ + %{ + "pointer" => "#/edges/0/condition_type", + "code" => "not_allowed", + "detail" => "is invalid" + } + ]) + end + + test "returns 422 when a job has invalid value", %{ + conn: conn, + project: project + } do + workflow = + build(:complex_workflow, name: "work1", project_id: project.id) + |> then(fn %{jobs: [job1, job2 | jobs]} = workflow -> + %{ + workflow + | jobs: [ + %{job1 | body: ["mistake as list"]}, + %{job2 | adaptor: ["mistake as list"]} | jobs + ] + } + end) + + conn = + post( + conn, + ~p"/api/projects/#{project.id}/workflows/", + Jason.encode!(workflow) + ) + + assert_problem(conn, 422, :invalid, [ + %{ + "pointer" => "#/jobs/0/body", + "code" => "type", + "detail" => "is invalid" + }, + %{ + "pointer" => "#/jobs/1/adaptor", + "code" => "type", + "detail" => "is invalid" + } + ]) + end + + test "returns 422 when a job references another project's credential", %{ + conn: conn, + project: project + } do + other_project = insert(:project) + project_credential = insert(:project_credential, project: other_project) + + workflow = + build(:simple_workflow, name: "work1", project_id: project.id) + |> then(fn %{jobs: [job]} = workflow -> + %{ + workflow + | jobs: [%{job | project_credential_id: project_credential.id}] + } + end) + + conn = + post( + conn, + ~p"/api/projects/#{project.id}/workflows/", + Jason.encode!(workflow) + ) + + assert_problem(conn, 422, :invalid, [ + %{ + "pointer" => "#/jobs/0/project_credential_id", + "code" => "credential_not_allowed", + "detail" => + "credential doesn't exist or isn't available in this project" + } + ]) + end + + test "returns 422 when a trigger has invalid value", %{ + conn: conn, + project: project + } do + workflow = + build(:complex_workflow, name: "work1", project_id: project.id) + |> then(fn %{triggers: [trigger]} = workflow -> + %{workflow | triggers: [%{trigger | enabled: "always"}]} + end) + + conn = + post( + conn, + ~p"/api/projects/#{project.id}/workflows/", + Jason.encode!(workflow) + ) + + assert_problem(conn, 422, :invalid, [ + %{ + "pointer" => "#/triggers/0/enabled", + "code" => "type", + "detail" => "is invalid" + } + ]) + end + + test "returns 402 when workflow limit has been reached", %{ + conn: conn, + project: project + } do + insert(:simple_workflow, name: "work1", project: project) + + workflow = + build(:simple_workflow, name: "work2", project_id: project.id) + + Mox.stub( + Lightning.Extensions.MockUsageLimiter, + :limit_action, + fn + %{type: :activate_workflow}, _context -> + {:error, :too_many_workflows, %Message{text: "some limit error msg"}} + end + ) + + conn = + post( + conn, + ~p"/api/projects/#{project.id}/workflows/", + Jason.encode!(workflow) + ) + + assert_problem(conn, 402, :exceeds_limit, []) + end + + test "returns 422 on project id mismatch", %{ + conn: conn, + user: user, + project: project + } do + other_project = + insert(:project, project_users: [%{user: user, role: :owner}]) + + workflow = + build(:simple_workflow, name: "work1", project_id: project.id) + + conn = + post( + conn, + ~p"/api/projects/#{other_project.id}/workflows", + Jason.encode!(workflow) + ) + + assert_problem(conn, 422, :invalid, [ + %{ + "pointer" => "#/project_id", + "code" => "mismatch", + "detail" => "does not match the id in the path" + } + ]) + end + + test "returns 409 when there is a duplicated id", %{ + conn: conn, + project: project + } do + %{id: edge_id} = insert(:edge) + + workflow = + build(:simple_workflow, + name: "workflow", + project_id: project.id + ) + |> then(fn %{edges: [edge]} = workflow -> + %{workflow | edges: [Map.put(edge, :id, edge_id)]} + end) + + conn = + post( + conn, + ~p"/api/projects/#{project.id}/workflows/", + Jason.encode!(workflow) + ) + + assert_problem(conn, 409, :taken, [ + %{ + "pointer" => "#/edges/0/id", + "code" => "taken", + "detail" => "has already been taken" + } + ]) + end + + test "reports a cycle at the edge that closes it", %{ + conn: conn, + project: project + } do + body = + graph_body(project.id, [{"t", "a"}, {"a", "b"}, {"b", "c"}, {"c", "a"}]) + + conn + |> post(~p"/api/projects/#{project.id}/workflows", Jason.encode!(body)) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/edges/3/target_job_id", + "code" => "cycle", + "detail" => "is part of a cycle" + } + ]) + end + + test "reports an id that is missing or names no job at the place it was sent", + %{conn: conn, project: project} do + missing_job_id = + project.id + |> graph_body([{"t", "a"}, {"a", "b"}]) + |> update_in(["jobs"], fn [a, b, c] -> [a, b, Map.delete(c, "id")] end) + + for {body, entry} <- [ + {missing_job_id, + %{ + "pointer" => "#/jobs/2/id", + "code" => "required", + "detail" => "can't be blank" + }}, + {graph_body(project.id, [{"t", "a"}, {"a", "nowhere"}]), + %{ + "pointer" => "#/edges/1/target_job_id", + "code" => "format", + "detail" => "is not a valid UUID" + }}, + {graph_body(project.id, [{"t", "a"}, {"a", Ecto.UUID.generate()}]), + %{ + "pointer" => "#/edges/1/target_job_id", + "code" => "not_found", + "detail" => "job doesn't exist, or is not in the same workflow" + }} + ] do + conn + |> post(~p"/api/projects/#{project.id}/workflows", Jason.encode!(body)) + |> assert_problem(422, :invalid, [entry]) + end + end + + test "a body that is invalid and over the plan limit answers 422, not 402", + %{conn: conn, project: project} do + Mox.stub( + Lightning.Extensions.MockUsageLimiter, + :limit_action, + fn %{type: :activate_workflow}, _context -> + {:error, :exceeds_limit, %Message{text: "upgrade"}} + end + ) + + cyclic = graph_body(project.id, [{"t", "a"}, {"a", "b"}, {"b", "a"}]) + + conn + |> post(~p"/api/projects/#{project.id}/workflows", Jason.encode!(cyclic)) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/edges/2/target_job_id", + "code" => "cycle", + "detail" => "is part of a cycle" + } + ]) + end + end +end diff --git a/test/lightning_web/controllers/api/workflows/patching_test.exs b/test/lightning_web/controllers/api/workflows/patching_test.exs new file mode 100644 index 00000000000..07c79bbced3 --- /dev/null +++ b/test/lightning_web/controllers/api/workflows/patching_test.exs @@ -0,0 +1,437 @@ +defmodule LightningWeb.API.Workflows.PatchingTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProblemHelpers + import Lightning.Factories + import LightningWeb.ApiWorkflowsHelpers + + alias Lightning.Extensions.Message + + setup %{conn: conn} do + conn = + conn + |> put_req_header("accept", "application/json") + |> put_req_header("content-type", "application/json") + + {:ok, conn: conn} + end + + describe "PATCH /workflows/:workflow_id" do + setup [:assign_bearer_for_api, :create_project_for_current_user] + + test "updates a workflow trigger", %{conn: conn, project: project} do + %{edges: [edge1 | other_edges], triggers: [trigger]} = + workflow = + insert(:simple_workflow, name: "work1.0", project: project) + |> Repo.reload() + |> Repo.preload([:edges, :jobs, :triggers]) + + patch = + build(:trigger, type: :cron, cron_expression: "0 0 * * *", enabled: true) + |> then( + &%{ + name: "work1.1", + edges: [%{edge1 | source_trigger_id: &1.id} | other_edges], + triggers: [%{trigger | enabled: false}, &1] + } + ) + + conn = + patch( + conn, + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(patch) + ) + + assert %{"workflow" => response_workflow} = + json_response(conn, 200) + + saved_workflow = get_saved_workflow(workflow) + + assert_response(response_workflow) + + assert workflow + |> Map.merge(patch) + |> encode_decode() + |> remove_timestamps() == + saved_workflow + |> encode_decode() + |> remove_timestamps() + end + + test "adds some jobs to a workflow", %{conn: conn, project: project} do + %{edges: edges, jobs: jobs} = + workflow = + insert(:simple_workflow, name: "work1.0", project: project) + |> Repo.reload() + |> Repo.preload([:edges, :jobs, :triggers]) + + patch = + build(:job) + |> then(fn job -> + %{ + name: "work1.1", + edges: + edges ++ + [ + build(:edge, + source_job_id: List.last(jobs).id, + target_job_id: job.id, + condition_type: :on_job_success + ) + ], + jobs: jobs ++ [job] + } + end) + + conn = + patch( + conn, + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(patch) + ) + + assert %{"workflow" => response_workflow} = + json_response(conn, 200) + + saved_workflow = get_saved_workflow(workflow) + + assert_response(response_workflow) + + merged = + workflow |> Map.merge(patch) |> encode_decode() |> remove_timestamps() + + saved = saved_workflow |> encode_decode() |> remove_timestamps() + + assert MapSet.new(merged["jobs"]) == MapSet.new(saved["jobs"]) + assert MapSet.new(merged["edges"]) == MapSet.new(saved["edges"]) + assert MapSet.new(merged["triggers"]) == MapSet.new(saved["triggers"]) + + assert Map.drop(merged, ["jobs", "edges", "triggers"]) == + Map.drop(saved, ["jobs", "edges", "triggers"]) + end + + test "Adds a disconnected/orphan job", %{conn: conn, project: project} do + %{jobs: jobs} = + workflow = + insert(:simple_workflow, name: "work1.0", project: project) + |> Repo.preload([:jobs]) + + job = build(:job) + + patch = + %{ + name: "work1.1", + jobs: jobs ++ [job] + } + + conn = + patch( + conn, + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(patch) + ) + + assert %{"workflow" => response_workflow} = + json_response(conn, 200) + + saved_workflow = get_saved_workflow(workflow) + + # assert both response_workflow and saved_workflow has the same jobs + # regardless of order + assert MapSet.new( + response_workflow["jobs"], + &Map.take(&1, ["id", "name", "adaptor", "body"]) + ) == + MapSet.new( + saved_workflow.jobs, + &(Map.take(&1, [:id, :name, :adaptor, :body]) + |> Map.new(fn {k, v} -> {Atom.to_string(k), v} end)) + ) + + assert workflow + |> Map.merge(patch) + |> encode_decode() + |> remove_timestamps() == + saved_workflow + |> encode_decode() + |> remove_timestamps() + end + + test "returns 422 for invalid triggers patch", %{ + conn: conn, + project: project + } do + %{triggers: [trigger]} = + workflow = + insert(:simple_workflow, name: "work1.0", project: project) + |> Repo.reload() + |> Repo.preload([:edges, :jobs, :triggers]) + + patch = + %{ + name: "work1.1", + triggers: [%{trigger | comment: ["invalid comment in list"]}] + } + + conn = + patch( + conn, + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(patch) + ) + + assert_problem(conn, 422, :invalid, [ + %{ + "pointer" => "#/triggers/0/comment", + "code" => "type", + "detail" => "is invalid" + } + ]) + end + + test "accepts a triggers patch that echoes an existing custom_path", %{ + conn: conn, + project: project + } do + # The CLI round-trips whole documents, so a trigger that already has a + # custom_path sends the same value back on every deploy. That must not + # 422. + %{triggers: [trigger]} = + workflow = + insert(:simple_workflow, name: "work1.0", project: project) + |> Repo.reload() + |> Repo.preload([:edges, :jobs, :triggers]) + + trigger = + trigger + |> Ecto.Changeset.change(custom_path: "partner-feed") + |> Repo.update!() + + patch = %{ + name: "work1.1", + triggers: [%{trigger | comment: "untouched routing"}] + } + + conn = + patch( + conn, + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(patch) + ) + + json_response(conn, 200) + assert Repo.reload!(trigger).custom_path == "partner-feed" + end + + test "returns 422 for invalid jobs patch", %{ + conn: conn, + project: project + } do + %{jobs: [job | other_jobs]} = + workflow = + insert(:simple_workflow, name: "work1.0", project: project) + |> Repo.reload() + |> Repo.preload([:edges, :jobs, :triggers]) + + patch = + %{ + name: "work1.1", + jobs: [%{job | body: ["invalid body in list"]} | other_jobs] + } + + conn = + patch( + conn, + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(patch) + ) + + assert_problem(conn, 422, :invalid, [ + %{ + "pointer" => "#/jobs/0/body", + "code" => "type", + "detail" => "is invalid" + } + ]) + end + + test "returns 422 for invalid edges patch", %{ + conn: conn, + project: project + } do + %{edges: [edge]} = + workflow = + insert(:simple_workflow, name: "work1.0", project: project) + |> Repo.reload() + |> Repo.preload([:edges, :jobs, :edges]) + + patch = + %{ + name: "work1.1", + edges: [%{edge | condition_type: "on_faillllure"}] + } + + conn = + patch( + conn, + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(patch) + ) + + assert_problem(conn, 422, :invalid, [ + %{ + "pointer" => "#/edges/0/condition_type", + "code" => "not_allowed", + "detail" => "is invalid" + } + ]) + end + + test "returns 422 on project id mismatch", %{ + conn: conn, + project: project + } do + workflow = + insert(:simple_workflow, name: "work1", project: project) + |> Repo.reload() + |> Repo.preload([:edges, :jobs, :triggers]) + + patch = %{project_id: Ecto.UUID.generate()} + + conn = + patch( + conn, + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(patch) + ) + + assert_problem(conn, 422, :invalid, [ + %{ + "pointer" => "#/project_id", + "code" => "mismatch", + "detail" => "does not match the id in the path" + } + ]) + end + + test "returns 422 when trying to replace the triggers", %{ + conn: conn, + project: project + } do + workflow = + insert(:simple_workflow, name: "work1", project: project) + |> Repo.reload() + |> Repo.preload([:edges, :jobs, :triggers]) + + patch = + %{ + triggers: [ + build(:trigger, + type: :cron, + cron_expression: "0 0 * * *", + enabled: true + ) + ] + } + + conn = + patch( + conn, + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(patch) + ) + + assert_problem(conn, 422, :invalid, [ + %{ + "pointer" => "#/triggers", + "code" => "trigger_removed", + "detail" => "a trigger cannot be removed, only edited or added" + }, + %{ + "pointer" => "#/edges", + "code" => "missing_trigger_edge", + "detail" => "needs an enabled edge from a trigger" + } + ]) + end + + test "returns 402 when an update enables a trigger over the workflow limit", + %{conn: conn, project: project} do + workflow = + insert(:simple_workflow, name: "work2", project: project) + |> Repo.preload([:edges, :jobs, :triggers]) + + [trigger] = workflow.triggers + trigger |> Ecto.Changeset.change(enabled: false) |> Repo.update!() + + Mox.expect( + Lightning.Extensions.MockUsageLimiter, + :limit_action, + fn %{type: :activate_workflow}, _context -> + {:error, :too_many_workflows, + %Message{text: "some limit error message"}} + end + ) + + patch = %{triggers: [%{id: trigger.id, enabled: true}]} + + conn + |> patch( + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(patch) + ) + |> assert_problem(402, :exceeds_limit, []) + end + + test "an update that leaves an enabled trigger enabled isn't held to the workflow limit", + %{conn: conn, project: project} do + workflow = + insert(:simple_workflow, name: "work2", project: project) + |> Repo.preload([:edges, :jobs, :triggers]) + + [trigger] = workflow.triggers + assert trigger.enabled + + Mox.stub( + Lightning.Extensions.MockUsageLimiter, + :limit_action, + fn %{type: :activate_workflow}, _context -> + {:error, :exceeds_limit, %Message{text: "some limit error message"}} + end + ) + + patch = %{name: "renamed", triggers: [%{id: trigger.id, enabled: true}]} + + assert %{"workflow" => %{"name" => "renamed"}} = + conn + |> patch( + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(patch) + ) + |> json_response(200) + end + + test "answers 404 for a workflow that is missing or deleted, whatever the body holds", + %{conn: conn, project: project} do + deleted = + insert(:simple_workflow, + project: project, + deleted_at: DateTime.utc_now() + ) + + for {id, body} <- [ + {Ecto.UUID.generate(), %{name: "renamed"}}, + {deleted.id, %{name: "renamed"}}, + {deleted.id, %{name: ""}} + ] do + conn + |> patch( + ~p"/api/projects/#{project.id}/workflows/#{id}", + Jason.encode!(body) + ) + |> assert_problem(404, :not_found, []) + end + end + end +end diff --git a/test/lightning_web/controllers/api/workflows/putting_test.exs b/test/lightning_web/controllers/api/workflows/putting_test.exs new file mode 100644 index 00000000000..32b7ca7b0d6 --- /dev/null +++ b/test/lightning_web/controllers/api/workflows/putting_test.exs @@ -0,0 +1,357 @@ +defmodule LightningWeb.API.Workflows.PuttingTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProblemHelpers + import Lightning.Factories + import LightningWeb.ApiWorkflowsHelpers + + setup %{conn: conn} do + conn = + conn + |> put_req_header("accept", "application/json") + |> put_req_header("content-type", "application/json") + + {:ok, conn: conn} + end + + describe "PUT /workflows/:workflow_id" do + setup [:assign_bearer_for_api, :create_project_for_current_user] + + test "updates completely a workflow", %{conn: conn, project: project} do + %{triggers: [trigger]} = + workflow = + insert(:simple_workflow, name: "work1.0", project: project) + |> Repo.reload() + |> Repo.preload([:edges, :jobs, :triggers]) + + complete_update = + build(:simple_workflow, name: "work1.1", project: project) + |> then(fn %{ + edges: [new_edge | other_new_edges], + jobs: [new_job1 | _other_jobs] = new_jobs, + triggers: [new_trigger] + } -> + Map.merge(workflow, %{ + edges: [ + Map.merge(new_edge, %{ + source_trigger_id: new_trigger.id, + target_job_id: new_job1.id, + condition_expression: "state.age > 18", + condition_label: "adult_age" + }) + | other_new_edges + ], + jobs: new_jobs, + triggers: [%{trigger | enabled: false}, new_trigger] + }) + end) + + assert %{"workflow" => response_workflow} = + conn + |> put( + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(complete_update) + ) + |> json_response(200) + + assert_response(response_workflow) + + saved = + get_saved_workflow(response_workflow["id"]) + |> encode_decode() + |> remove_timestamps() + + merged = + workflow + |> Map.merge(complete_update) + |> encode_decode() + |> remove_timestamps() + + assert MapSet.new(merged["jobs"]) == MapSet.new(saved["jobs"]) + assert MapSet.new(merged["edges"]) == MapSet.new(saved["edges"]) + assert MapSet.new(merged["triggers"]) == MapSet.new(saved["triggers"]) + + assert Map.drop(merged, ["jobs", "edges", "triggers"]) == + Map.drop(saved, ["jobs", "edges", "triggers"]) + end + + test "updates completely a workflow with disconnected job", %{ + conn: conn, + project: project + } do + %{triggers: [trigger]} = + workflow = + insert(:simple_workflow, name: "work1.0", project: project) + |> Repo.reload() + |> Repo.preload([:edges, :jobs, :triggers]) + + complete_update = + build(:simple_workflow, name: "work1.1", project: project) + |> then(fn %{ + edges: [new_edge | other_new_edges], + jobs: [new_job1 | _other_jobs], + triggers: [new_trigger] + } -> + Map.merge(workflow, %{ + edges: [ + %{ + new_edge + | source_trigger_id: new_trigger.id, + target_job_id: new_job1.id + } + | other_new_edges + ], + jobs: [build(:job), new_job1 | workflow.jobs], + triggers: [%{trigger | enabled: false}, new_trigger] + }) + end) + + conn = + put( + conn, + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(complete_update) + ) + + assert %{"workflow" => response_workflow} = + json_response(conn, 200) + + previous_workflow_jobs_ids = Enum.map(workflow.jobs, & &1.id) + + saved_workflow = get_saved_workflow(workflow) + + assert Enum.count( + response_workflow["jobs"], + &(&1["id"] not in previous_workflow_jobs_ids) + ) == 2 + + assert Enum.count( + saved_workflow.jobs, + &(&1.id not in previous_workflow_jobs_ids) + ) == 2 + end + + test "updates completely a workflow removing a job", %{ + conn: conn, + project: project + } do + workflow = + insert(:complex_workflow, name: "work1.0", project: project) + |> Repo.reload() + |> Repo.preload([:edges, :jobs, :triggers]) + + complete_update = + workflow + |> then(fn %{ + edges: edges, + jobs: jobs + } -> + last_job = List.last(jobs) + last_edge = List.last(edges) + + Map.merge(workflow, %{ + name: "work1.1", + edges: List.delete(edges, last_edge), + jobs: List.delete(jobs, last_job) + }) + end) + + assert %{"workflow" => response_workflow} = + conn + |> put( + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(complete_update) + ) + |> json_response(200) + + assert_response(response_workflow) + + saved_workflow = get_saved_workflow(workflow.id) + + refute MapSet.new(complete_update.jobs, & &1.id) == + MapSet.new(workflow.jobs, & &1.id) + + refute MapSet.new(complete_update.edges, & &1.id) == + MapSet.new(workflow.edges, & &1.id) + + assert MapSet.new(complete_update.jobs, & &1.id) == + MapSet.new(saved_workflow.jobs, & &1.id) + + assert MapSet.new(complete_update.edges, & &1.id) == + MapSet.new(saved_workflow.edges, & &1.id) + end + + test "updates workflow ignoring workflow_id", %{ + conn: conn, + project: project + } do + %{triggers: [trigger]} = + workflow = + insert(:simple_workflow, name: "work1.0", project: project) + |> Repo.reload() + |> Repo.preload([:edges, :jobs, :triggers]) + + %{id: other_workflow_id} = insert(:simple_workflow) + + complete_update_external_ref = + build(:simple_workflow, name: "work1.1", project: project) + |> then(fn %{ + edges: [new_edge | other_new_edges], + jobs: [new_job | other_jobs], + triggers: [new_trigger] + } -> + Map.merge(workflow, %{ + edges: [ + Map.merge(new_edge, %{ + source_trigger_id: new_trigger.id, + target_job_id: new_job.id, + workflow_id: other_workflow_id + }) + | other_new_edges + ], + jobs: [ + %{new_job | workflow_id: other_workflow_id} + | other_jobs + ], + triggers: [ + %{trigger | enabled: false}, + %{new_trigger | workflow_id: other_workflow_id} + ] + }) + end) + + assert %{ + "workflow" => response_workflow + } = + conn + |> put( + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(complete_update_external_ref) + ) + |> json_response(200) + + assert_response(response_workflow) + + for {schema, children} <- [ + {Lightning.Workflows.Job, complete_update_external_ref.jobs}, + {Lightning.Workflows.Edge, complete_update_external_ref.edges}, + {Lightning.Workflows.Trigger, complete_update_external_ref.triggers} + ], + %{workflow_id: ^other_workflow_id, id: id} <- children do + assert Repo.get!(schema, id).workflow_id == workflow.id + end + end + + test "returns 422 when the body's id doesn't match the one on the path", %{ + conn: conn, + project: project + } do + workflow = + insert(:simple_workflow, project: project) + |> Repo.preload([:edges, :jobs, :triggers]) + + conn + |> put( + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(%{workflow | id: Ecto.UUID.generate()}) + ) + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/id", + "code" => "mismatch", + "detail" => "does not match the id in the path" + } + ]) + end + + test "returns 409 when one id belongs to another workflow", %{ + conn: conn, + project: project + } do + %{triggers: [trigger]} = + workflow = + insert(:simple_workflow, name: "work1.0", project: project) + |> Repo.reload() + |> Repo.preload([:edges, :jobs, :triggers]) + + %{jobs: [external_job | _jobs]} = insert(:simple_workflow) + + complete_update_external_id = + build(:simple_workflow, name: "work1.1", project: project) + |> then(fn %{ + edges: [new_edge | other_new_edges], + jobs: [new_job | other_jobs], + triggers: [new_trigger] + } -> + Map.merge(workflow, %{ + edges: [ + %{ + new_edge + | source_trigger_id: new_trigger.id, + target_job_id: external_job.id + } + | other_new_edges + ], + jobs: [ + %{new_job | id: external_job.id} | other_jobs + ], + triggers: [ + %{trigger | enabled: false}, + new_trigger + ] + }) + end) + + conn = + put( + conn, + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(complete_update_external_id) + ) + + assert_problem(conn, 409, :taken, [ + %{ + "pointer" => "#/jobs/0/id", + "code" => "taken", + "detail" => "has already been taken" + } + ]) + end + + test "answers 404 for a workflow that is missing or deleted, whatever id the body gives", + %{conn: conn, project: project} do + deleted = + insert(:simple_workflow, + project: project, + deleted_at: DateTime.utc_now() + ) + + missing_id = Ecto.UUID.generate() + document = build(:simple_workflow, project_id: project.id) + + for {id, body} <- [ + {missing_id, document}, + {missing_id, Map.put(document, :id, missing_id)}, + {deleted.id, Map.put(document, :id, deleted.id)} + ] do + conn + |> put( + ~p"/api/projects/#{project.id}/workflows/#{id}", + Jason.encode!(body) + ) + |> assert_problem(404, :not_found, []) + end + end + + test "a body that isn't a JSON object answers 400", %{ + conn: conn, + project: project + } do + workflow = insert(:simple_workflow, project: project) + + conn + |> put(~p"/api/projects/#{project.id}/workflows/#{workflow.id}", "[1]") + |> assert_problem(400, :malformed, []) + end + end +end diff --git a/test/lightning_web/controllers/api/workflows/reading_test.exs b/test/lightning_web/controllers/api/workflows/reading_test.exs new file mode 100644 index 00000000000..67d9d229370 --- /dev/null +++ b/test/lightning_web/controllers/api/workflows/reading_test.exs @@ -0,0 +1,151 @@ +defmodule LightningWeb.API.Workflows.ReadingTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProblemHelpers + import Lightning.Factories + import LightningWeb.ApiWorkflowsHelpers + + setup %{conn: conn} do + conn = + conn + |> put_req_header("accept", "application/json") + |> put_req_header("content-type", "application/json") + + {:ok, conn: conn} + end + + describe "listing workflows" do + setup [:assign_bearer_for_api, :create_project_for_current_user] + + test "answers the whole workflow, without its lifecycle state, on either route", + %{conn: conn, project: project} do + workflow = + insert(:simple_workflow, + name: "exact-check", + project: project, + state: :live + ) + + for path <- [ + ~p"/api/workflows", + ~p"/api/projects/#{project.id}/workflows" + ] do + assert conn |> get(path) |> json_response(200) == %{ + "workflows" => [encode_decode(workflow)] + } + end + end + + test "lists the workflows of the projects I belong to, and no others", %{ + conn: conn, + project: project + } do + workflow1 = insert(:workflow, project: project, name: "Workflow A") + insert(:trigger, workflow: workflow1) + insert(:job, workflow: workflow1) + + workflow2 = insert(:workflow, project: project, name: "Workflow B") + insert(:trigger, workflow: workflow2) + + other_workflow = insert(:workflow, project: insert(:project)) + insert(:trigger, workflow: other_workflow) + + response = conn |> get(~p"/api/workflows") |> json_response(200) + + assert length(response["workflows"]) == 2 + + workflow_ids = Enum.map(response["workflows"], & &1["id"]) + assert workflow1.id in workflow_ids + assert workflow2.id in workflow_ids + refute other_workflow.id in workflow_ids + end + + test "narrows to one project by the project_id query parameter or by the nested route", + %{conn: conn, user: user} do + project1 = + insert(:project, project_users: [%{user: user, role: :owner}]) + + project2 = + insert(:project, project_users: [%{user: user, role: :owner}]) + + workflow1 = insert(:workflow, project: project1, name: "P1 Workflow") + insert(:trigger, workflow: workflow1) + + workflow2 = insert(:workflow, project: project2, name: "P2 Workflow") + insert(:trigger, workflow: workflow2) + + for path <- [ + ~p"/api/workflows?project_id=#{project1.id}", + ~p"/api/projects/#{project1.id}/workflows" + ] do + assert [%{"id" => id}] = + conn + |> get(path) + |> json_response(200) + |> Map.fetch!("workflows") + + assert id == workflow1.id + end + end + + test "answers 404 for a project I can't see", %{conn: conn} do + other_project = insert(:project) + + conn = get(conn, ~p"/api/projects/#{other_project.id}/workflows") + + assert_problem(conn, 404, :not_found, []) + end + end + + describe "showing a workflow" do + setup [:assign_bearer_for_api, :create_project_for_current_user] + + test "answers the whole workflow, without its lifecycle state, on either route", + %{conn: conn, project: project} do + workflow = insert(:simple_workflow, project: project, state: :live) + + for path <- [ + ~p"/api/workflows/#{workflow.id}", + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}" + ] do + assert conn |> get(path) |> json_response(200) == %{ + "workflow" => encode_decode(workflow) + } + end + end + + test "answers 404 for a workflow that is missing, deleted, in a project I can't see, or under another project", + %{conn: conn, user: user, project: project} do + deleted = + insert(:simple_workflow, + project: project, + deleted_at: DateTime.utc_now() + ) + + other_project = + insert(:project, project_users: [%{user: user, role: :owner}]) + + mine = insert(:simple_workflow, project: project) + secret = insert(:simple_workflow, name: "Secret") + + for path <- [ + ~p"/api/workflows/#{Ecto.UUID.generate()}", + ~p"/api/workflows/#{deleted.id}", + ~p"/api/projects/#{project.id}/workflows/#{deleted.id}", + ~p"/api/workflows/#{secret.id}", + ~p"/api/projects/#{project.id}/workflows/#{secret.id}", + ~p"/api/projects/#{other_project.id}/workflows/#{mine.id}" + ] do + assert_problem(get(conn, path), 404, :not_found, []) + end + end + + test "has no edit route", %{conn: conn, project: project} do + workflow = insert(:simple_workflow, project: project) + + conn + |> get(~p"/api/projects/#{project.id}/workflows/#{workflow.id}/edit") + |> assert_problem(404, :not_found, []) + end + end +end diff --git a/test/lightning_web/controllers/api/workflows/refusals_test.exs b/test/lightning_web/controllers/api/workflows/refusals_test.exs new file mode 100644 index 00000000000..a02a6bc5153 --- /dev/null +++ b/test/lightning_web/controllers/api/workflows/refusals_test.exs @@ -0,0 +1,460 @@ +defmodule LightningWeb.API.Workflows.RefusalsTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProblemHelpers + import Lightning.Factories + import Lightning.RaceHelpers + import Lightning.WorkflowsFixtures + import LightningWeb.ApiWorkflowsHelpers + + alias Lightning.Workflows.Presence + + setup %{conn: conn} do + conn = + conn + |> put_req_header("accept", "application/json") + |> put_req_header("content-type", "application/json") + + {:ok, conn: conn} + end + + describe "who may reach it" do + setup [:assign_bearer_for_api, :create_project_for_current_user] + + @tag role: :viewer + test "viewer cannot create a workflow", %{conn: conn, project: project} do + workflow = + build(:simple_workflow, + name: "viewer-created-workflow", + project_id: project.id + ) + + conn = + post( + conn, + ~p"/api/projects/#{project.id}/workflows/", + Jason.encode!(workflow) + ) + + assert_problem(conn, 403, :forbidden, []) + + refute Lightning.Repo.get_by(Lightning.Workflows.Workflow, + name: "viewer-created-workflow" + ) + end + + @tag role: :viewer + test "viewer hears 404 for a workflow the project hasn't got, and 403 for one it has, and changes nothing", + %{conn: conn, project: project} do + workflow = + insert(:simple_workflow, name: "original-name", project: project) + + body = Jason.encode!(%{name: "hacked-by-viewer"}) + + conn + |> patch( + ~p"/api/projects/#{project.id}/workflows/#{Ecto.UUID.generate()}", + body + ) + |> assert_problem(404, :not_found, []) + + conn + |> patch(~p"/api/projects/#{project.id}/workflows/#{workflow.id}", body) + |> assert_problem(403, :forbidden, []) + + assert Repo.reload(workflow).name == "original-name" + end + + test "admin, editor and owner can create a workflow", %{conn: conn} do + for role <- [:admin, :editor, :owner] do + user = insert(:user) + project = insert(:project, project_users: [%{user: user, role: role}]) + + workflow = + build(:simple_workflow, + name: "#{role}-created-workflow", + project_id: project.id + ) + + resp_conn = + conn + |> assign_bearer(user) + |> post( + ~p"/api/projects/#{project.id}/workflows/", + Jason.encode!(workflow) + ) + + assert %{"workflow" => %{"id" => id}} = + json_response(resp_conn, 201), + "expected #{role} to be allowed to create a workflow" + + assert Lightning.Repo.get(Lightning.Workflows.Workflow, id) + end + end + + test "admin, editor and owner can update a workflow", %{conn: conn} do + for role <- [:admin, :editor, :owner] do + user = insert(:user) + project = insert(:project, project_users: [%{user: user, role: role}]) + + workflow = + insert(:simple_workflow, name: "#{role}-original", project: project) + |> Repo.reload() + + resp_conn = + conn + |> assign_bearer(user) + |> patch( + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(%{name: "#{role}-updated"}) + ) + + assert %{"workflow" => %{}} = + json_response(resp_conn, 200), + "expected #{role} to be allowed to update a workflow" + + assert Repo.reload(workflow).name == "#{role}-updated" + end + end + end + + describe "refusals and what they leave alone" do + setup [:assign_bearer_for_api, :create_project_for_current_user] + + test "a caller with no standing gets 404, whether or not the workflow is open in the editor", + %{conn: conn} do + editor = insert(:user) + project = insert(:project, project_users: [%{user: editor}]) + workflow = insert(:simple_workflow, project: project) + path = ~p"/api/projects/#{project.id}/workflows/#{workflow.id}" + + conn + |> patch(path, Jason.encode!(%{"name" => "renamed"})) + |> assert_problem(404, :not_found, []) + + Presence.track_user_presence(editor, workflow, self()) + assert Presence.has_any_presence?(workflow) + + conn + |> patch(path, Jason.encode!(%{"name" => "renamed"})) + |> assert_problem(404, :not_found, []) + end + + test "a member short of the project's MFA requirement hears 404 for a workflow the project hasn't got, and 403 for one it has", + %{conn: conn, project: project} do + workflow = insert(:simple_workflow, project: project) + Repo.update!(Ecto.Changeset.change(project, requires_mfa: true)) + + conn + |> get(~p"/api/projects/#{project.id}/workflows/#{Ecto.UUID.generate()}") + |> assert_problem(404, :not_found, []) + + conn + |> get(~p"/api/projects/#{project.id}/workflows/#{workflow.id}") + |> assert_problem(403, :forbidden, []) + end + + test "never quotes a value from the request, on any route", %{ + conn: conn, + project: project + } do + workflow = insert(:simple_workflow, project: project) + sentinel = plant_sentinel(%{"v" => "x"})["v"] + + planted = + project.id + |> graph_body([{"t", "a"}, {"a", "b"}, {"b", "a"}]) + |> Map.delete("project_id") + |> plant_sentinel() + + for path <- [ + ~p"/api/workflows/#{sentinel}", + ~p"/api/workflows?project_id=#{sentinel}", + ~p"/api/projects/#{sentinel}/workflows", + ~p"/api/projects/#{sentinel}/workflows/#{workflow.id}", + ~p"/api/projects/#{project.id}/workflows/#{sentinel}" + ] do + conn + |> get(path) + |> refute_sentinel() + |> assert_problem(404, :not_found, []) + end + + conn + |> post(~p"/api/projects/#{project.id}/workflows", Jason.encode!(planted)) + |> refute_sentinel() + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/edges", + "code" => "single_target", + "detail" => "a trigger must have a single target" + }, + %{ + "pointer" => "#/edges/0/condition_type", + "code" => "not_allowed", + "detail" => "is invalid" + }, + %{ + "pointer" => "#/edges/1/condition_type", + "code" => "not_allowed", + "detail" => "is invalid" + }, + %{ + "pointer" => "#/edges/1/target_job_id", + "code" => "cycle", + "detail" => "target_job_id must be different from source_job_id" + }, + %{ + "pointer" => "#/edges/2/condition_type", + "code" => "not_allowed", + "detail" => "is invalid" + }, + %{ + "pointer" => "#/edges/2/target_job_id", + "code" => "cycle", + "detail" => "target_job_id must be different from source_job_id" + }, + %{ + "pointer" => "#/triggers/0/type", + "code" => "not_allowed", + "detail" => "is invalid" + }, + %{ + "pointer" => "#/triggers/0/enabled", + "code" => "type", + "detail" => "is invalid" + } + ]) + + for request <- [&patch/3, &put/3] do + conn + |> request.( + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(Map.put(planted, "id", sentinel)) + ) + |> refute_sentinel() + |> assert_problem(422, :invalid, [ + %{ + "pointer" => "#/id", + "code" => "mismatch", + "detail" => "does not match the id in the path" + }, + %{ + "pointer" => "#/triggers", + "code" => "trigger_removed", + "detail" => "a trigger cannot be removed, only edited or added" + }, + %{ + "pointer" => "#/edges", + "code" => "single_target", + "detail" => "a trigger must have a single target" + }, + %{ + "pointer" => "#/edges/0/condition_type", + "code" => "not_allowed", + "detail" => "is invalid" + }, + %{ + "pointer" => "#/edges/0/source_trigger_id", + "code" => "format", + "detail" => "is not a valid UUID" + }, + %{ + "pointer" => "#/edges/0/target_job_id", + "code" => "format", + "detail" => "is not a valid UUID" + }, + %{ + "pointer" => "#/edges/1/condition_type", + "code" => "not_allowed", + "detail" => "is invalid" + }, + %{ + "pointer" => "#/edges/1/source_job_id", + "code" => "format", + "detail" => "is not a valid UUID" + }, + %{ + "pointer" => "#/edges/1/target_job_id", + "code" => "format", + "detail" => "is not a valid UUID" + }, + %{ + "pointer" => "#/edges/1/target_job_id", + "code" => "cycle", + "detail" => "target_job_id must be different from source_job_id" + }, + %{ + "pointer" => "#/edges/2/condition_type", + "code" => "not_allowed", + "detail" => "is invalid" + }, + %{ + "pointer" => "#/edges/2/source_job_id", + "code" => "format", + "detail" => "is not a valid UUID" + }, + %{ + "pointer" => "#/edges/2/target_job_id", + "code" => "format", + "detail" => "is not a valid UUID" + }, + %{ + "pointer" => "#/edges/2/target_job_id", + "code" => "cycle", + "detail" => "target_job_id must be different from source_job_id" + }, + %{ + "pointer" => "#/triggers/0/type", + "code" => "not_allowed", + "detail" => "is invalid" + }, + %{ + "pointer" => "#/triggers/0/enabled", + "code" => "type", + "detail" => "is invalid" + }, + %{ + "pointer" => "#/triggers/0/id", + "code" => "format", + "detail" => "is not a valid UUID" + }, + %{ + "pointer" => "#/jobs/0/id", + "code" => "format", + "detail" => "is not a valid UUID" + }, + %{ + "pointer" => "#/jobs/1/id", + "code" => "format", + "detail" => "is not a valid UUID" + }, + %{ + "pointer" => "#/jobs/1/id", + "code" => "duplicate", + "detail" => "names the same id as #/jobs/0" + }, + %{ + "pointer" => "#/jobs/2/id", + "code" => "format", + "detail" => "is not a valid UUID" + }, + %{ + "pointer" => "#/jobs/2/id", + "code" => "duplicate", + "detail" => "names the same id as #/jobs/0" + } + ]) + + conn + |> request.( + ~p"/api/projects/#{project.id}/workflows/#{sentinel}", + Jason.encode!(planted) + ) + |> refute_sentinel() + |> assert_problem(404, :not_found, []) + end + end + end + + describe "PATCH of a workflow that is changing elsewhere" do + setup [:assign_bearer_for_api, :create_project_for_current_user] + + test "returns 409 when the workflow is being edited on the UI" do + %{conn: conn, user: user} = + register_and_log_in_user(%{conn: Phoenix.ConnTest.build_conn()}) + + project = + insert(:project, project_users: [%{user: user}]) + + workflow = workflow_fixture(name: "work1.0", project_id: project.id) + + refute Presence.has_any_presence?(workflow) + + # Simulate a user editing the workflow in the collaborative editor by + # tracking their presence on the workflow. + Presence.track_user_presence(user, workflow, self()) + + patch = %{name: "work1.1"} + + conn = + conn + |> put_req_header("content-type", "application/json") + |> assign_bearer(user) + |> patch( + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(patch) + ) + + assert_problem(conn, 409, :being_edited, []) + + assert Presence.has_any_presence?(workflow) + end + + test "a workflow another request saves mid-update answers 409 concurrent_edit", + %{conn: conn, project: project} do + workflow = insert(:simple_workflow, name: "work1.0", project: project) + + bump_workflow_lock_mid_save(workflow.id) + + conn + |> patch( + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(%{name: "work1.1"}) + ) + |> assert_problem(409, :concurrent_edit, []) + + assert %{name: "work1.0", lock_version: 1} = Repo.reload(workflow) + end + + test "a workflow open in the editor answers 409 before its body is read", + %{conn: conn, project: project, user: user} do + workflow = insert(:simple_workflow, project: project) + Presence.track_user_presence(user, workflow, self()) + + conn + |> patch( + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + "[1]" + ) + |> assert_problem(409, :being_edited, []) + end + end + + describe "PUT of a workflow that is changing elsewhere" do + setup [:assign_bearer_for_api, :create_project_for_current_user] + + test "returns 409 when the workflow is being edited on the UI" do + %{conn: conn, user: user} = + register_and_log_in_user(%{conn: Phoenix.ConnTest.build_conn()}) + + project = + insert(:project, project_users: [%{user: user}]) + + workflow = + workflow_fixture(name: "work1.0", project_id: project.id) + |> Repo.preload([:edges, :jobs, :triggers]) + + refute Presence.has_any_presence?(workflow) + + # Simulate a user editing the workflow in the collaborative editor by + # tracking their presence on the workflow. + Presence.track_user_presence(user, workflow, self()) + + workflow_update = %{workflow | name: "work1.1"} + + conn = + conn + |> put_req_header("content-type", "application/json") + |> assign_bearer(user) + |> put( + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + Jason.encode!(workflow_update) + ) + + assert_problem(conn, 409, :being_edited, []) + + assert Presence.has_any_presence?(workflow) + end + end +end diff --git a/test/lightning_web/controllers/api/workflows_controller_test.exs b/test/lightning_web/controllers/api/workflows_controller_test.exs deleted file mode 100644 index 82f2db92cbd..00000000000 --- a/test/lightning_web/controllers/api/workflows_controller_test.exs +++ /dev/null @@ -1,1988 +0,0 @@ -defmodule LightningWeb.API.WorkflowsControllerTest do - use LightningWeb.ConnCase, async: true - - import Lightning.Factories - import Lightning.WorkflowsFixtures - - alias Lightning.Extensions.Message - alias Lightning.Workflows - alias Lightning.Workflows.Presence - - setup %{conn: conn} do - conn = - conn - |> put_req_header("accept", "application/json") - |> put_req_header("content-type", "application/json") - - {:ok, conn: conn} - end - - describe "index" do - setup [:assign_bearer_for_api, :create_project_for_current_user] - - test "does not publish the workflow lifecycle", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project, state: :live) - insert(:trigger, workflow: workflow) - - index = get(conn, ~p"/api/workflows") |> json_response(200) - - for rendered <- index["workflows"] do - refute Map.has_key?(rendered, "state") - assert Map.has_key?(rendered, "name") - end - - show = - get(conn, ~p"/api/workflows/#{workflow.id}") |> json_response(200) - - refute Map.has_key?(show["workflow"], "state") - assert Map.has_key?(show["workflow"], "name") - end - - test "lists workflows for projects I have access to", %{ - conn: conn, - project: project - } do - workflow1 = insert(:workflow, project: project, name: "Workflow A") - insert(:trigger, workflow: workflow1) - insert(:job, workflow: workflow1) - - workflow2 = insert(:workflow, project: project, name: "Workflow B") - insert(:trigger, workflow: workflow2) - - # Create a workflow in another project (should not be accessible) - other_project = insert(:project) - other_workflow = insert(:workflow, project: other_project) - insert(:trigger, workflow: other_workflow) - - conn = get(conn, ~p"/api/workflows") - - response = json_response(conn, 200) - - assert length(response["workflows"]) == 2 - assert %{"errors" => %{}} = response - - workflow_ids = Enum.map(response["workflows"], & &1["id"]) - assert workflow1.id in workflow_ids - assert workflow2.id in workflow_ids - refute other_workflow.id in workflow_ids - end - - test "filters workflows by project_id query parameter", %{ - conn: conn, - user: user - } do - project1 = - insert(:project, project_users: [%{user: user, role: :owner}]) - - project2 = - insert(:project, project_users: [%{user: user, role: :owner}]) - - workflow1 = insert(:workflow, project: project1, name: "P1 Workflow") - insert(:trigger, workflow: workflow1) - - workflow2 = insert(:workflow, project: project2, name: "P2 Workflow") - insert(:trigger, workflow: workflow2) - - conn = get(conn, ~p"/api/workflows?project_id=#{project1.id}") - - response = json_response(conn, 200) - - assert length(response["workflows"]) == 1 - assert List.first(response["workflows"])["id"] == workflow1.id - end - - test "nested route: lists workflows for specific project", %{ - conn: conn, - user: user - } do - project1 = - insert(:project, project_users: [%{user: user, role: :owner}]) - - project2 = - insert(:project, project_users: [%{user: user, role: :owner}]) - - workflow1 = insert(:workflow, project: project1, name: "P1 Workflow") - insert(:trigger, workflow: workflow1) - - workflow2 = insert(:workflow, project: project2, name: "P2 Workflow") - insert(:trigger, workflow: workflow2) - - conn = get(conn, ~p"/api/projects/#{project1.id}/workflows") - - response = json_response(conn, 200) - - assert length(response["workflows"]) == 1 - assert List.first(response["workflows"])["id"] == workflow1.id - refute Enum.any?(response["workflows"], &(&1["id"] == workflow2.id)) - end - - test "returns workflows with jobs, triggers, and edges", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project, name: "Full Workflow") - trigger = insert(:trigger, workflow: workflow, type: :webhook) - job = insert(:job, workflow: workflow) - - insert(:edge, - workflow: workflow, - source_trigger: trigger, - target_job: job, - condition_type: :always - ) - - conn = get(conn, ~p"/api/workflows") - - response = json_response(conn, 200) - - assert [wf] = response["workflows"] - assert wf["id"] == workflow.id - assert wf["name"] == "Full Workflow" - - assert length(wf["triggers"]) == 1 - assert List.first(wf["triggers"])["id"] == trigger.id - - assert length(wf["jobs"]) == 1 - assert List.first(wf["jobs"])["id"] == job.id - - assert length(wf["edges"]) == 1 - end - - test "returns exact workflow JSON for nested route", %{ - conn: conn, - project: project - } do - workflow = insert(:simple_workflow, name: "exact-check", project: project) - - conn = get(conn, ~p"/api/projects/#{project.id}/workflows") - - assert json_response(conn, 200) == %{ - "errors" => %{}, - "workflows" => [encode_decode(workflow)] - } - end - - test "returns 401 for project user cannot access", %{conn: conn} do - other_project = insert(:project) - - conn = get(conn, ~p"/api/projects/#{other_project.id}/workflows") - - assert json_response(conn, 401) - end - - test "returns 401 without a token" do - conn = - build_conn() - |> put_req_header("accept", "application/json") - |> get(~p"/api/workflows") - - assert %{"error" => "Unauthorized"} == json_response(conn, 401) - end - - test "returns 401 with an invalid token" do - conn = - build_conn() - |> put_req_header("accept", "application/json") - |> Plug.Conn.put_req_header("authorization", "Bearer InvalidToken") - |> get(~p"/api/workflows") - - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - - test "returns 401 with a run token instead of API token" do - workflow = insert(:simple_workflow) - workorder = insert(:workorder, dataclip: insert(:dataclip)) - - run = - insert(:run, - work_order: workorder, - dataclip: workorder.dataclip, - starting_trigger: workflow.triggers |> hd() - ) - - token = Lightning.Workers.generate_run_token(run) - - conn = - build_conn() - |> put_req_header("accept", "application/json") - |> assign_bearer(token) - |> get(~p"/api/projects/#{workflow.project_id}/workflows/#{workflow.id}") - - assert json_response(conn, 401) == %{"error" => "Unauthorized"} - end - end - - describe "show" do - setup [:assign_bearer_for_api, :create_project_for_current_user] - - test "returns a single workflow by id", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project, name: "My Workflow") - trigger = insert(:trigger, workflow: workflow, type: :webhook) - job = insert(:job, workflow: workflow, name: "Step One") - - insert(:edge, - workflow: workflow, - source_trigger: trigger, - target_job: job, - condition_type: :always - ) - - conn = get(conn, ~p"/api/workflows/#{workflow}") - - response = json_response(conn, 200) - - assert %{"workflow" => wf, "errors" => %{}} = response - assert wf["id"] == workflow.id - assert wf["name"] == "My Workflow" - assert wf["project_id"] == project.id - assert length(wf["triggers"]) == 1 - assert length(wf["jobs"]) == 1 - assert length(wf["edges"]) == 1 - end - - test "returns a workflow via nested project route", %{ - conn: conn, - project: project - } do - workflow = insert(:workflow, project: project, name: "Nested Show") - insert(:trigger, workflow: workflow) - - conn = - get(conn, ~p"/api/projects/#{project.id}/workflows/#{workflow}") - - response = json_response(conn, 200) - - assert %{"workflow" => wf, "errors" => %{}} = response - assert wf["id"] == workflow.id - assert wf["name"] == "Nested Show" - end - - test "returns exact workflow JSON for nested route", %{ - conn: conn, - project: project - } do - workflow = insert(:simple_workflow, project: project) - - conn = - get(conn, ~p"/api/projects/#{project.id}/workflows/#{workflow.id}") - - assert json_response(conn, 200) == %{ - "errors" => %{}, - "workflow" => encode_decode(workflow) - } - end - - test "returns 404 for non-existent workflow", %{conn: conn} do - conn = get(conn, ~p"/api/workflows/#{Ecto.UUID.generate()}") - - assert json_response(conn, 404) - end - - test "returns 401 for workflow in project user cannot access", %{ - conn: conn - } do - other_project = insert(:project) - - other_workflow = - insert(:workflow, project: other_project, name: "Secret") - - insert(:trigger, workflow: other_workflow) - - conn = get(conn, ~p"/api/workflows/#{other_workflow}") - - assert json_response(conn, 401) - end - - test "returns 400 for project_id mismatch on nested route", %{ - conn: conn, - user: user, - project: project - } do - other_project = - insert(:project, project_users: [%{user: user, role: :owner}]) - - workflow = insert(:workflow, project: project, name: "Wrong Project") - insert(:trigger, workflow: workflow) - - conn = - get( - conn, - ~p"/api/projects/#{other_project.id}/workflows/#{workflow}" - ) - - response = json_response(conn, 400) - assert response["error"] == "Bad Request" - end - - test "returns 422 for invalid project_id", %{conn: conn, project: project} do - workflow = insert(:workflow, project: project) - insert(:trigger, workflow: workflow) - - conn = get(conn, ~p"/api/projects/foo/workflows/#{workflow.id}") - - assert %{ - "errors" => %{"workflow" => ["Id foo should be a UUID."]}, - "id" => nil - } == json_response(conn, 422) - end - - test "returns 401 without a token" do - %{id: workflow_id, project_id: project_id} = insert(:simple_workflow) - - conn = - build_conn() - |> put_req_header("accept", "application/json") - |> get(~p"/api/projects/#{project_id}/workflows/#{workflow_id}") - - assert %{"error" => "Unauthorized"} == json_response(conn, 401) - end - end - - describe "POST /workflows/:project_id" do - setup [:assign_bearer_for_api, :create_project_for_current_user] - - test "inserts a workflow", %{conn: conn, project: project} do - workflow = - build(:simple_workflow, name: "work1", project_id: project.id) - - conn = - post( - conn, - ~p"/api/projects/#{project.id}/workflows/", - Jason.encode!(workflow) - ) - - assert %{"workflow" => response_workflow, "errors" => %{}} = - json_response(conn, 201) - - assert %{ - edges: [edge], - jobs: [job], - triggers: [trigger] - } = - saved_workflow = get_saved_workflow(response_workflow["id"]) - - assert_response(response_workflow) - - assert Map.take(hd(workflow.edges), [:condition_type, :enabled]) == - Map.take(edge, [:condition_type, :enabled]) - - assert Map.take(hd(workflow.jobs), [:name, :adaptor, :body]) == - Map.take(job, [:name, :adaptor, :body]) - - assert Map.take(hd(workflow.triggers), [:type, :enabled]) == - Map.take(trigger, [:type, :enabled]) - - assert Map.take(workflow, [:name, :project_id]) == - Map.take(saved_workflow, [:name, :project_id]) - end - - test "inserts a workflow with disconnected job", %{ - conn: conn, - project: project - } do - workflow = - build(:simple_workflow, name: "work1", project_id: project.id) - |> then(fn %{jobs: jobs} = workflow -> - %{workflow | jobs: [build(:job) | jobs]} - end) - - conn = - post( - conn, - ~p"/api/projects/#{project.id}/workflows/", - Jason.encode!(workflow) - ) - - assert %{"workflow" => response_workflow, "errors" => %{}} = - json_response(conn, 201) - - assert saved_workflow = get_saved_workflow(response_workflow["id"]) - - assert_response(response_workflow) - - assert pluck_to_mapset(workflow.edges, [:condition_type, :enabled]) == - pluck_to_mapset(saved_workflow.edges, [:condition_type, :enabled]) - - assert pluck_to_mapset(workflow.jobs, [:name, :adaptor, :body]) == - pluck_to_mapset(saved_workflow.jobs, [:name, :adaptor, :body]) - - assert pluck_to_mapset(workflow.triggers, [:type, :enabled]) == - pluck_to_mapset(saved_workflow.triggers, [:type, :enabled]) - - assert Map.take(workflow, [:name, :project_id]) == - Map.take(saved_workflow, [:name, :project_id]) - end - - test "creates UUIDs on insert based on user arbitrary ids", %{ - conn: conn, - project: project - } do - workflow = - build(:complex_workflow, project_id: project.id) - |> then(fn %{ - jobs: [job1, job2 | jobs], - edges: edges, - triggers: [trigger] - } = workflow -> - old_job1_id = job1.id - old_job2_id = job2.id - job1 = Map.put(job1, :id, 11) - job2 = Map.put(job2, :id, 12) - old_trigger_id = trigger.id - trigger = Map.put(trigger, :id, "trigger1") - - ids_map = %{ - old_job1_id => 11, - old_job2_id => 12, - old_trigger_id => "trigger1" - } - - edges - |> Enum.map(fn edge -> - edge - |> Map.update(:source_trigger_id, nil, &Map.get(ids_map, &1, &1)) - |> Map.update(:source_job_id, nil, &Map.get(ids_map, &1, &1)) - |> Map.update(:target_job_id, nil, &Map.get(ids_map, &1, &1)) - end) - |> then(fn edges -> - %{ - workflow - | jobs: [job1, job2 | jobs], - edges: edges, - triggers: [trigger] - } - end) - end) - - assert %{ - "workflow" => response_workflow, - "errors" => %{} - } = - conn - |> post( - ~p"/api/projects/#{project.id}/workflows", - Jason.encode!(workflow) - ) - |> json_response(201) - - saved_workflow = get_saved_workflow(response_workflow["id"]) - - merged = encode_decode(response_workflow) |> remove_timestamps() - saved = encode_decode(saved_workflow) |> remove_timestamps() - - assert MapSet.new(merged["jobs"]) == MapSet.new(saved["jobs"]) - assert MapSet.new(merged["edges"]) == MapSet.new(saved["edges"]) - assert MapSet.new(merged["triggers"]) == MapSet.new(saved["triggers"]) - - assert Map.drop(merged, ["jobs", "edges", "triggers"]) == - Map.drop(saved, ["jobs", "edges", "triggers"]) - end - - test "returns 422 when an edge has invalid condition_type", %{ - conn: conn, - project: project - } do - %{edges: [edge | _edges]} = - workflow = - build(:simple_workflow, name: "work1", project_id: project.id) - |> then(fn %{edges: [edge | edges]} = workflow -> - %{workflow | edges: [%{edge | condition_type: "on_failures"} | edges]} - end) - - conn = - post( - conn, - ~p"/api/projects/#{project.id}/workflows/", - Jason.encode!(workflow) - ) - - assert json_response(conn, 422) == %{ - "id" => nil, - "errors" => %{ - "edges" => [ - "Edge #{edge.id} has the errors: [condition_type: is invalid]" - ] - } - } - end - - test "returns 422 when a job has invalid value", %{ - conn: conn, - project: project - } do - %{jobs: [job1, job2 | _jobs]} = - workflow = - build(:complex_workflow, name: "work1", project_id: project.id) - |> then(fn %{jobs: [job1, job2 | jobs]} = workflow -> - %{ - workflow - | jobs: [ - %{job1 | body: ["mistake as list"]}, - %{job2 | adaptor: ["mistake as list"]} | jobs - ] - } - end) - - conn = - post( - conn, - ~p"/api/projects/#{project.id}/workflows/", - Jason.encode!(workflow) - ) - - assert %{ - "id" => nil, - "errors" => %{ - "jobs" => jobs_errors - } - } = json_response(conn, 422) - - assert Enum.sort(jobs_errors) == - Enum.sort([ - "Job #{job1.id} has the errors: [body: is invalid]", - "Job #{job2.id} has the errors: [adaptor: is invalid]" - ]) - end - - test "returns 422 when a job references another project's credential", %{ - conn: conn, - project: project - } do - other_project = insert(:project) - project_credential = insert(:project_credential, project: other_project) - - %{jobs: [job]} = - workflow = - build(:simple_workflow, name: "work1", project_id: project.id) - |> then(fn %{jobs: [job]} = workflow -> - %{ - workflow - | jobs: [%{job | project_credential_id: project_credential.id}] - } - end) - - conn = - post( - conn, - ~p"/api/projects/#{project.id}/workflows/", - Jason.encode!(workflow) - ) - - assert %{ - "id" => nil, - "errors" => %{ - "jobs" => jobs_errors - } - } = json_response(conn, 422) - - assert [job_error] = jobs_errors - assert job_error =~ "Job #{job.id} has the errors:" - assert job_error =~ "project_credential_id:" - assert job_error =~ "isn't available in this project" - end - - test "returns 422 when a trigger has invalid value", %{ - conn: conn, - project: project - } do - %{triggers: [trigger]} = - workflow = - build(:complex_workflow, name: "work1", project_id: project.id) - |> then(fn %{triggers: [trigger]} = workflow -> - %{workflow | triggers: [%{trigger | enabled: "always"}]} - end) - - conn = - post( - conn, - ~p"/api/projects/#{project.id}/workflows/", - Jason.encode!(workflow) - ) - - assert %{ - "id" => nil, - "errors" => %{ - "triggers" => [ - "Trigger #{trigger.id} has the errors: [enabled: is invalid]" - ] - } - } == json_response(conn, 422) - end - - test "returns 422 when workflow limit has been reached", %{ - conn: conn, - project: project - } do - insert(:simple_workflow, name: "work1", project: project) - - workflow = - build(:simple_workflow, name: "work2", project_id: project.id) - - Mox.stub( - Lightning.Extensions.MockUsageLimiter, - :limit_action, - fn - %{type: :activate_workflow}, _context -> - {:error, :too_many_workflows, %Message{text: "some limit error msg"}} - end - ) - - conn = - post( - conn, - ~p"/api/projects/#{project.id}/workflows/", - Jason.encode!(workflow) - ) - - assert %{ - "id" => nil, - "errors" => %{ - "project_id" => ["some limit error msg"] - } - } = json_response(conn, 422) - end - - test "returns 422 when there are too many active triggers", %{ - conn: conn, - project: project - } do - workflow = - build(:simple_workflow, - name: "workflow", - triggers: build_list(2, :trigger), - project_id: project.id - ) - - conn = - post( - conn, - ~p"/api/projects/#{project.id}/workflows/", - Jason.encode!(workflow) - ) - - assert %{ - "id" => nil, - "errors" => %{ - "triggers" => [ - "A workflow can have only one trigger enabled at a time." - ] - } - } = json_response(conn, 422) - end - - test "returns 422 on project id mismatch", %{ - conn: conn, - user: user, - project: project - } do - other_project = - insert(:project, project_users: [%{user: user, role: :owner}]) - - workflow = - build(:simple_workflow, name: "work1", project_id: project.id) - - conn = - post( - conn, - ~p"/api/projects/#{other_project.id}/workflows", - Jason.encode!(workflow) - ) - - assert json_response(conn, 422) == %{ - "id" => workflow.id, - "errors" => %{ - "project_id" => [ - "The project_id of the body does not match the one the path." - ] - } - } - end - - test "returns 422 when graph has a cycle", %{ - conn: conn, - project: project - } do - workflow = - build(:complex_workflow, name: "work1", project_id: project.id) - |> then(fn %{jobs: jobs, edges: edges} = workflow -> - job0 = Enum.at(jobs, 0) - job3 = Enum.at(jobs, 3) - - %{ - workflow - | edges: - edges ++ - [build(:edge, source_job_id: job3.id, target_job_id: job0.id)] - } - end) - - conn = - post( - conn, - ~p"/api/projects/#{project.id}/workflows", - Jason.encode!(workflow) - ) - - job0 = Enum.at(workflow.jobs, 0) - - assert json_response(conn, 422) == %{ - "id" => nil, - "errors" => %{ - "edges" => ["Cycle detected on job #{job0.id}."] - } - } - end - - test "returns 422 when there is a duplicated id", %{ - conn: conn, - project: project - } do - %{id: edge_id} = insert(:edge) - - workflow = - build(:simple_workflow, - name: "workflow", - project_id: project.id - ) - |> then(fn %{edges: [edge]} = workflow -> - %{workflow | edges: [Map.put(edge, :id, edge_id)]} - end) - - conn = - post( - conn, - ~p"/api/projects/#{project.id}/workflows/", - Jason.encode!(workflow) - ) - - assert %{ - "id" => nil, - "errors" => %{ - "edges" => [ - "Edge #{edge_id} has the errors: [id: This value should be unique.]" - ] - } - } == json_response(conn, 422) - end - - test "returns 422 when edges misses a source trigger", %{ - conn: conn, - project: project - } do - trigger = - build(:trigger, type: :webhook, enabled: true) - - workflow = - build(:workflow, name: "workflow 1", project_id: project.id) - |> with_trigger(trigger) - |> then(fn workflow -> - job1 = build(:job) - job2 = build(:job) - - %{ - workflow - | jobs: [job1, job2], - edges: [ - build(:edge, source_job_id: job1.id, target_job_id: job2.id) - ] - } - end) - - conn = - post( - conn, - ~p"/api/projects/#{project.id}/workflows", - Jason.encode!(workflow) - ) - - assert json_response(conn, 422) == %{ - "id" => workflow.id, - "errors" => %{ - "edges" => ["Missing edge with source_trigger_id."] - } - } - end - - test "returns 422 when edges, jobs and triggers are omitted", %{ - conn: conn, - project: project - } do - conn = - post( - conn, - ~p"/api/projects/#{project.id}/workflows", - Jason.encode!(%{name: "workflow without children"}) - ) - - assert json_response(conn, 422) == %{ - "id" => nil, - "errors" => %{ - "edges" => ["Missing edge with source_trigger_id."] - } - } - end - - test "returns 422 when edges has multiple source triggers", %{ - conn: conn, - project: project - } do - trigger = - build(:trigger, type: :webhook, enabled: true) - - workflow = - build(:workflow, name: "workflow 1", project_id: project.id) - |> with_trigger(trigger) - |> then(fn %{triggers: [trigger]} = workflow -> - job1 = build(:job) - job2 = build(:job) - - %{ - workflow - | jobs: [job1, job2], - edges: [ - build(:edge, - source_trigger_id: trigger.id, - target_job_id: job1.id - ), - build(:edge, source_job_id: job1.id, target_job_id: job2.id), - build(:edge, - source_trigger_id: trigger.id, - target_job_id: job2.id - ) - ] - } - end) - - conn = - post( - conn, - ~p"/api/projects/#{project.id}/workflows", - Jason.encode!(workflow) - ) - - assert json_response(conn, 422) == %{ - "id" => workflow.id, - "errors" => %{ - "edges" => [ - "There should be only one enabled edge with source_trigger_id." - ] - } - } - end - - test "returns 422 when an edge source_job_id points to a trigger", %{ - conn: conn, - project: project - } do - workflow = - build(:simple_workflow, project_id: project.id) - |> then(fn %{jobs: [job1], triggers: [trigger]} = workflow -> - job2 = build(:job) - - %{ - workflow - | jobs: [job1, job2], - edges: [ - build(:edge, - source_trigger_id: trigger.id, - target_job_id: job1.id - ), - build(:edge, source_job_id: trigger.id, target_job_id: job2.id) - ] - } - end) - - conn = - post( - conn, - ~p"/api/projects/#{project.id}/workflows", - Jason.encode!(workflow) - ) - - assert json_response(conn, 422) == %{ - "id" => nil, - "errors" => %{ - "edges" => [ - "source_trigger_id must have a single target." - ] - } - } - end - - test "returns 401 without a token", %{project: project} do - build(:simple_workflow, name: "work1", project: project) - - conn = - build_conn() - |> put_req_header("accept", "application/json") - |> post(~p"/api/projects/#{project.id}/workflows/") - - assert %{"error" => "Unauthorized"} == json_response(conn, 401) - end - end - - describe "PATCH /workflows/:workflow_id" do - setup [:assign_bearer_for_api, :create_project_for_current_user] - - test "updates a workflow trigger", %{conn: conn, project: project} do - %{edges: [edge1 | other_edges], triggers: [trigger]} = - workflow = - insert(:simple_workflow, name: "work1.0", project: project) - |> Repo.reload() - |> Repo.preload([:edges, :jobs, :triggers]) - - patch = - build(:trigger, type: :cron, cron_expression: "0 0 * * *", enabled: true) - |> then( - &%{ - name: "work1.1", - edges: [%{edge1 | source_trigger_id: &1.id} | other_edges], - triggers: [%{trigger | enabled: false}, &1] - } - ) - - conn = - patch( - conn, - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(patch) - ) - - assert %{"workflow" => response_workflow, "errors" => %{}} = - json_response(conn, 200) - - saved_workflow = get_saved_workflow(workflow) - - assert_response(response_workflow) - - assert workflow - |> Map.merge(patch) - |> encode_decode() - |> remove_timestamps() == - saved_workflow - |> encode_decode() - |> remove_timestamps() - end - - test "adds some jobs to a workflow", %{conn: conn, project: project} do - %{edges: edges, jobs: jobs} = - workflow = - insert(:simple_workflow, name: "work1.0", project: project) - |> Repo.reload() - |> Repo.preload([:edges, :jobs, :triggers]) - - patch = - build(:job) - |> then(fn job -> - %{ - name: "work1.1", - edges: - edges ++ - [ - build(:edge, - source_job_id: List.last(jobs).id, - target_job_id: job.id, - condition_type: :on_job_success - ) - ], - jobs: jobs ++ [job] - } - end) - - conn = - patch( - conn, - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(patch) - ) - - assert %{"workflow" => response_workflow, "errors" => %{}} = - json_response(conn, 200) - - saved_workflow = get_saved_workflow(workflow) - - assert_response(response_workflow) - - merged = - workflow |> Map.merge(patch) |> encode_decode() |> remove_timestamps() - - saved = saved_workflow |> encode_decode() |> remove_timestamps() - - assert MapSet.new(merged["jobs"]) == MapSet.new(saved["jobs"]) - assert MapSet.new(merged["edges"]) == MapSet.new(saved["edges"]) - assert MapSet.new(merged["triggers"]) == MapSet.new(saved["triggers"]) - - assert Map.drop(merged, ["jobs", "edges", "triggers"]) == - Map.drop(saved, ["jobs", "edges", "triggers"]) - end - - test "Adds a disconnected/orphan job", %{conn: conn, project: project} do - %{jobs: jobs} = - workflow = - insert(:simple_workflow, name: "work1.0", project: project) - |> Repo.preload([:jobs]) - - job = build(:job) - - patch = - %{ - name: "work1.1", - jobs: jobs ++ [job] - } - - conn = - patch( - conn, - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(patch) - ) - - assert %{"workflow" => response_workflow, "errors" => %{}} = - json_response(conn, 200) - - saved_workflow = get_saved_workflow(workflow) - - # assert both response_workflow and saved_workflow has the same jobs - # regardless of order - assert MapSet.new( - response_workflow["jobs"], - &Map.take(&1, ["id", "name", "adaptor", "body"]) - ) == - MapSet.new( - saved_workflow.jobs, - &(Map.take(&1, [:id, :name, :adaptor, :body]) - |> Map.new(fn {k, v} -> {Atom.to_string(k), v} end)) - ) - - assert workflow - |> Map.merge(patch) - |> encode_decode() - |> remove_timestamps() == - saved_workflow - |> encode_decode() - |> remove_timestamps() - end - - test "returns 404 when the workflow doesn't exist", %{ - conn: conn, - project: project - } do - unexisting_id = Ecto.UUID.generate() - patch = %{name: "work1.1"} - - assert %{"id" => ^unexisting_id, "errors" => ["Not Found"]} = - conn - |> patch( - ~p"/api/projects/#{project.id}/workflows/#{unexisting_id}", - Jason.encode!(patch) - ) - |> json_response(404) - end - - test "returns 409 when the workflow is being edited on the UI" do - %{conn: conn, user: user} = - register_and_log_in_user(%{conn: Phoenix.ConnTest.build_conn()}) - - project = - insert(:project, project_users: [%{user: user}]) - - workflow = workflow_fixture(name: "work1.0", project_id: project.id) - - refute Presence.has_any_presence?(workflow) - - # Simulate a user editing the workflow in the collaborative editor by - # tracking their presence on the workflow. - Presence.track_user_presence(user, workflow, self()) - - patch = %{name: "work1.1"} - - conn = - conn - |> put_req_header("content-type", "application/json") - |> assign_bearer(user) - |> patch( - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(patch) - ) - - assert json_response(conn, 409) == %{ - "id" => workflow.id, - "errors" => %{ - "workflow" => [ - "Cannot save a workflow (work1.0) while it is being edited on the App UI" - ] - } - } - - assert Presence.has_any_presence?(workflow) - end - - test "returns 422 for invalid triggers patch", %{ - conn: conn, - project: project - } do - %{triggers: [trigger]} = - workflow = - insert(:simple_workflow, name: "work1.0", project: project) - |> Repo.reload() - |> Repo.preload([:edges, :jobs, :triggers]) - - patch = - %{ - name: "work1.1", - triggers: [%{trigger | comment: ["invalid comment in list"]}] - } - - conn = - patch( - conn, - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(patch) - ) - - assert json_response(conn, 422) == %{ - "id" => workflow.id, - "errors" => %{ - "triggers" => [ - "Trigger #{trigger.id} has the errors: [comment: is invalid]" - ] - } - } - end - - test "accepts a triggers patch that echoes an existing custom_path", %{ - conn: conn, - project: project - } do - # The CLI round-trips whole documents, so a trigger that already has a - # custom_path sends the same value back on every deploy. That must not - # 422. - %{triggers: [trigger]} = - workflow = - insert(:simple_workflow, name: "work1.0", project: project) - |> Repo.reload() - |> Repo.preload([:edges, :jobs, :triggers]) - - trigger = - trigger - |> Ecto.Changeset.change(custom_path: "partner-feed") - |> Repo.update!() - - patch = %{ - name: "work1.1", - triggers: [%{trigger | comment: "untouched routing"}] - } - - conn = - patch( - conn, - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(patch) - ) - - assert %{"errors" => %{}} = json_response(conn, 200) - assert Repo.reload!(trigger).custom_path == "partner-feed" - end - - test "returns 422 for invalid jobs patch", %{ - conn: conn, - project: project - } do - %{jobs: [job | other_jobs]} = - workflow = - insert(:simple_workflow, name: "work1.0", project: project) - |> Repo.reload() - |> Repo.preload([:edges, :jobs, :triggers]) - - patch = - %{ - name: "work1.1", - jobs: [%{job | body: ["invalid body in list"]} | other_jobs] - } - - conn = - patch( - conn, - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(patch) - ) - - assert json_response(conn, 422) == %{ - "id" => workflow.id, - "errors" => %{ - "jobs" => ["Job #{job.id} has the errors: [body: is invalid]"] - } - } - end - - test "returns 422 for invalid edges patch", %{ - conn: conn, - project: project - } do - %{edges: [edge]} = - workflow = - insert(:simple_workflow, name: "work1.0", project: project) - |> Repo.reload() - |> Repo.preload([:edges, :jobs, :edges]) - - patch = - %{ - name: "work1.1", - edges: [%{edge | condition_type: "on_faillllure"}] - } - - conn = - patch( - conn, - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(patch) - ) - - assert json_response(conn, 422) == %{ - "id" => workflow.id, - "errors" => %{ - "edges" => [ - "Edge #{edge.id} has the errors: [condition_type: is invalid]" - ] - } - } - end - - test "returns 422 on project id mismatch", %{ - conn: conn, - project: project - } do - workflow = - insert(:simple_workflow, name: "work1", project: project) - |> Repo.reload() - |> Repo.preload([:edges, :jobs, :triggers]) - - patch = %{project_id: Ecto.UUID.generate()} - - conn = - patch( - conn, - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(patch) - ) - - assert json_response(conn, 422) == %{ - "id" => workflow.id, - "errors" => %{ - "project_id" => [ - "The project_id of the body does not match the one the path." - ] - } - } - end - - test "returns 422 when trying to replace the triggers", %{ - conn: conn, - project: project - } do - workflow = - insert(:simple_workflow, name: "work1", project: project) - |> Repo.reload() - |> Repo.preload([:edges, :jobs, :triggers]) - - patch = - %{ - triggers: [ - build(:trigger, - type: :cron, - cron_expression: "0 0 * * *", - enabled: true - ) - ] - } - - conn = - patch( - conn, - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(patch) - ) - - assert json_response(conn, 422) == %{ - "id" => workflow.id, - "errors" => %{ - "triggers" => [ - "A trigger cannot be replaced, only edited or added." - ] - } - } - end - - test "returns 422 when workflow limit has been reached", %{ - conn: conn, - project: project - } do - insert(:simple_workflow, name: "work1", project: project) - - trigger = build(:trigger, enabled: false) - - workflow = - insert(:simple_workflow, name: "work2", project: project) - |> with_trigger(trigger) - - patch = %{triggers: [%{(workflow.triggers |> hd()) | enabled: true}]} - - Mox.expect( - Lightning.Extensions.MockUsageLimiter, - :limit_action, - fn - %{type: :activate_workflow}, _context -> - {:error, :too_many_workflows, - %Message{text: "some limit error message"}} - end - ) - - conn = - patch( - conn, - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(patch) - ) - - assert %{ - "id" => workflow.id, - "errors" => %{ - "project_id" => ["some limit error message"] - } - } == json_response(conn, 422) - end - - test "returns 422 when there are too many enabled triggers", %{ - conn: conn, - project: project - } do - workflow = - insert(:simple_workflow, name: "work1", project: project) - |> Repo.reload() - |> Repo.preload([:edges, :jobs, :triggers]) - - patch = - %{ - triggers: [ - build(:trigger, - type: :cron, - cron_expression: "0 0 * * *", - enabled: true - ) - | workflow.triggers - ] - } - - conn = - patch( - conn, - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(patch) - ) - - assert json_response(conn, 422) == %{ - "id" => workflow.id, - "errors" => %{ - "triggers" => [ - "A workflow can have only one trigger enabled at a time." - ] - } - } - end - - test "returns 401 without a token", %{project: project} do - workflow = insert(:simple_workflow, name: "work1", project: project) - - conn = - build_conn() - |> put_req_header("accept", "application/json") - |> put_req_header("content-type", "application/json") - |> patch( - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(%{name: "work-2"}) - ) - - assert %{"error" => "Unauthorized"} == json_response(conn, 401) - end - end - - describe "authorization by project role" do - setup [:assign_bearer_for_api, :create_project_for_current_user] - - @tag role: :viewer - test "viewer cannot create a workflow", %{conn: conn, project: project} do - workflow = - build(:simple_workflow, - name: "viewer-created-workflow", - project_id: project.id - ) - - conn = - post( - conn, - ~p"/api/projects/#{project.id}/workflows/", - Jason.encode!(workflow) - ) - - assert %{"error" => "Unauthorized"} == json_response(conn, 401) - - refute Lightning.Repo.get_by(Lightning.Workflows.Workflow, - name: "viewer-created-workflow" - ) - end - - @tag role: :viewer - test "viewer cannot update a workflow", %{conn: conn, project: project} do - workflow = - insert(:simple_workflow, name: "original-name", project: project) - |> Repo.reload() - - conn = - patch( - conn, - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(%{name: "hacked-by-viewer"}) - ) - - assert %{"error" => "Unauthorized"} == json_response(conn, 401) - - assert Repo.reload(workflow).name == "original-name" - end - - test "admin, editor and owner can create a workflow", %{conn: conn} do - for role <- [:admin, :editor, :owner] do - user = insert(:user) - project = insert(:project, project_users: [%{user: user, role: role}]) - - workflow = - build(:simple_workflow, - name: "#{role}-created-workflow", - project_id: project.id - ) - - resp_conn = - conn - |> assign_bearer(user) - |> post( - ~p"/api/projects/#{project.id}/workflows/", - Jason.encode!(workflow) - ) - - assert %{"workflow" => %{"id" => id}, "errors" => %{}} = - json_response(resp_conn, 201), - "expected #{role} to be allowed to create a workflow" - - assert Lightning.Repo.get(Lightning.Workflows.Workflow, id) - end - end - - test "admin, editor and owner can update a workflow", %{conn: conn} do - for role <- [:admin, :editor, :owner] do - user = insert(:user) - project = insert(:project, project_users: [%{user: user, role: role}]) - - workflow = - insert(:simple_workflow, name: "#{role}-original", project: project) - |> Repo.reload() - - resp_conn = - conn - |> assign_bearer(user) - |> patch( - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(%{name: "#{role}-updated"}) - ) - - assert %{"workflow" => %{}, "errors" => %{}} = - json_response(resp_conn, 200), - "expected #{role} to be allowed to update a workflow" - - assert Repo.reload(workflow).name == "#{role}-updated" - end - end - end - - describe "PUT /workflows/:workflow_id" do - setup [:assign_bearer_for_api, :create_project_for_current_user] - - test "updates completely a workflow", %{conn: conn, project: project} do - %{triggers: [trigger]} = - workflow = - insert(:simple_workflow, name: "work1.0", project: project) - |> Repo.reload() - |> Repo.preload([:edges, :jobs, :triggers]) - - complete_update = - build(:simple_workflow, name: "work1.1", project: project) - |> then(fn %{ - edges: [new_edge | other_new_edges], - jobs: [new_job1 | _other_jobs] = new_jobs, - triggers: [new_trigger] - } -> - Map.merge(workflow, %{ - edges: [ - Map.merge(new_edge, %{ - source_trigger_id: new_trigger.id, - target_job_id: new_job1.id, - condition_expression: "state.age > 18", - condition_label: "adult_age" - }) - | other_new_edges - ], - jobs: new_jobs, - triggers: [%{trigger | enabled: false}, new_trigger] - }) - end) - - assert %{"workflow" => response_workflow, "errors" => %{}} = - conn - |> put( - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(complete_update) - ) - |> json_response(200) - - assert_response(response_workflow) - - saved = - get_saved_workflow(response_workflow["id"]) - |> encode_decode() - |> remove_timestamps() - - merged = - workflow - |> Map.merge(complete_update) - |> encode_decode() - |> remove_timestamps() - - assert MapSet.new(merged["jobs"]) == MapSet.new(saved["jobs"]) - assert MapSet.new(merged["edges"]) == MapSet.new(saved["edges"]) - assert MapSet.new(merged["triggers"]) == MapSet.new(saved["triggers"]) - - assert Map.drop(merged, ["jobs", "edges", "triggers"]) == - Map.drop(saved, ["jobs", "edges", "triggers"]) - end - - test "updates completely a workflow with disconnected job", %{ - conn: conn, - project: project - } do - %{triggers: [trigger]} = - workflow = - insert(:simple_workflow, name: "work1.0", project: project) - |> Repo.reload() - |> Repo.preload([:edges, :jobs, :triggers]) - - complete_update = - build(:simple_workflow, name: "work1.1", project: project) - |> then(fn %{ - edges: [new_edge | other_new_edges], - jobs: [new_job1 | _other_jobs], - triggers: [new_trigger] - } -> - Map.merge(workflow, %{ - edges: [ - %{ - new_edge - | source_trigger_id: new_trigger.id, - target_job_id: new_job1.id - } - | other_new_edges - ], - jobs: [build(:job), new_job1 | workflow.jobs], - triggers: [%{trigger | enabled: false}, new_trigger] - }) - end) - - conn = - put( - conn, - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(complete_update) - ) - - assert %{"workflow" => response_workflow, "errors" => %{}} = - json_response(conn, 200) - - previous_workflow_jobs_ids = Enum.map(workflow.jobs, & &1.id) - - saved_workflow = get_saved_workflow(workflow) - - assert Enum.filter( - response_workflow["jobs"], - &(&1["id"] not in previous_workflow_jobs_ids) - ) - |> Enum.count() == 2 - - assert Enum.filter( - saved_workflow.jobs, - &(&1.id not in previous_workflow_jobs_ids) - ) - |> Enum.count() == 2 - end - - test "updates completely a workflow removing a job", %{ - conn: conn, - project: project - } do - workflow = - insert(:complex_workflow, name: "work1.0", project: project) - |> Repo.reload() - |> Repo.preload([:edges, :jobs, :triggers]) - - complete_update = - workflow - |> then(fn %{ - edges: edges, - jobs: jobs - } -> - last_job = List.last(jobs) - last_edge = List.last(edges) - - Map.merge(workflow, %{ - name: "work1.1", - edges: List.delete(edges, last_edge), - jobs: List.delete(jobs, last_job) - }) - end) - - assert %{"workflow" => response_workflow, "errors" => %{}} = - conn - |> put( - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(complete_update) - ) - |> json_response(200) - - assert_response(response_workflow) - - saved_workflow = get_saved_workflow(workflow.id) - - refute MapSet.new(complete_update.jobs, & &1.id) == - MapSet.new(workflow.jobs, & &1.id) - - refute MapSet.new(complete_update.edges, & &1.id) == - MapSet.new(workflow.edges, & &1.id) - - assert MapSet.new(complete_update.jobs, & &1.id) == - MapSet.new(saved_workflow.jobs, & &1.id) - - assert MapSet.new(complete_update.edges, & &1.id) == - MapSet.new(saved_workflow.edges, & &1.id) - end - - test "updates workflow ignoring workflow_id", %{ - conn: conn, - project: project - } do - %{triggers: [trigger]} = - workflow = - insert(:simple_workflow, name: "work1.0", project: project) - |> Repo.reload() - |> Repo.preload([:edges, :jobs, :triggers]) - - %{id: other_workflow_id} = insert(:simple_workflow) - - complete_update_external_ref = - build(:simple_workflow, name: "work1.1", project: project) - |> then(fn %{ - edges: [new_edge | other_new_edges], - jobs: [new_job | other_jobs], - triggers: [new_trigger] - } -> - Map.merge(workflow, %{ - edges: [ - Map.merge(new_edge, %{ - source_trigger_id: new_trigger.id, - target_job_id: new_job.id, - workflow_id: other_workflow_id - }) - | other_new_edges - ], - jobs: [ - %{new_job | workflow_id: other_workflow_id} - | other_jobs - ], - triggers: [ - %{trigger | enabled: false}, - %{new_trigger | workflow_id: other_workflow_id} - ] - }) - end) - - assert %{ - "workflow" => response_workflow, - "errors" => %{} - } = - conn - |> put( - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(complete_update_external_ref) - ) - |> json_response(200) - - assert_response(response_workflow) - end - - test "returns 404 when the workflow doesn't exist", %{ - conn: conn, - project: project - } do - unexisting_id = Ecto.UUID.generate() - workflow = build(:simple_workflow, project_id: project.id) - - assert %{"id" => ^unexisting_id, "errors" => ["Not Found"]} = - conn - |> put( - ~p"/api/projects/#{project.id}/workflows/#{unexisting_id}", - Jason.encode!(Map.put(workflow, :id, unexisting_id)) - ) - |> json_response(404) - end - - test "returns 409 when the workflow is being edited on the UI" do - %{conn: conn, user: user} = - register_and_log_in_user(%{conn: Phoenix.ConnTest.build_conn()}) - - project = - insert(:project, project_users: [%{user: user}]) - - workflow = - workflow_fixture(name: "work1.0", project_id: project.id) - |> Repo.preload([:edges, :jobs, :triggers]) - - refute Presence.has_any_presence?(workflow) - - # Simulate a user editing the workflow in the collaborative editor by - # tracking their presence on the workflow. - Presence.track_user_presence(user, workflow, self()) - - workflow_update = %{workflow | name: "work1.1"} - - conn = - conn - |> put_req_header("content-type", "application/json") - |> assign_bearer(user) - |> put( - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(workflow_update) - ) - - assert json_response(conn, 409) == %{ - "id" => workflow.id, - "errors" => %{ - "workflow" => [ - "Cannot save a workflow (work1.0) while it is being edited on the App UI" - ] - } - } - - assert Presence.has_any_presence?(workflow) - end - - test "returns 404 when the workflow id doesn't match the one on the path", %{ - conn: conn, - project: project - } do - unexisting_id = Ecto.UUID.generate() - workflow = build(:simple_workflow, project_id: project.id) - - assert %{ - "id" => ^unexisting_id, - "errors" => %{ - "id" => ["Workflow ID doesn't match with the one on the path."] - } - } = - conn - |> put( - ~p"/api/projects/#{project.id}/workflows/#{unexisting_id}", - Jason.encode!(workflow) - ) - |> json_response(422) - end - - test "returns 422 when one id belongs to another workflow", %{ - conn: conn, - project: project - } do - %{triggers: [trigger]} = - workflow = - insert(:simple_workflow, name: "work1.0", project: project) - |> Repo.reload() - |> Repo.preload([:edges, :jobs, :triggers]) - - %{jobs: [external_job | _jobs]} = insert(:simple_workflow) - - complete_update_external_id = - build(:simple_workflow, name: "work1.1", project: project) - |> then(fn %{ - edges: [new_edge | other_new_edges], - jobs: [new_job | other_jobs], - triggers: [new_trigger] - } -> - Map.merge(workflow, %{ - edges: [ - %{ - new_edge - | source_trigger_id: new_trigger.id, - target_job_id: external_job.id - } - | other_new_edges - ], - jobs: [ - %{new_job | id: external_job.id} | other_jobs - ], - triggers: [ - %{trigger | enabled: false}, - new_trigger - ] - }) - end) - - conn = - put( - conn, - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(complete_update_external_id) - ) - - assert json_response(conn, 422) == %{ - "id" => workflow.id, - "errors" => %{ - "jobs" => [ - "Job #{external_job.id} has the errors: [id: This value should be unique.]" - ] - } - } - end - - test "returns 422 when trying to replace the triggers", %{ - conn: conn, - project: project - } do - workflow = - insert(:simple_workflow, name: "work1.0", project: project) - |> Repo.reload() - |> Repo.preload([:edges, :jobs, :triggers]) - - invalid_update = - build(:simple_workflow, name: "work1.1", project: project) - |> then(fn %{ - edges: [new_edge | other_new_edges], - jobs: [new_job1 | _other_jobs] = new_jobs, - triggers: [new_trigger] - } -> - Map.merge(workflow, %{ - edges: [ - %{ - new_edge - | source_trigger_id: new_trigger.id, - target_job_id: new_job1.id - } - | other_new_edges - ], - jobs: new_jobs, - triggers: [new_trigger] - }) - end) - - conn = - put( - conn, - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(invalid_update) - ) - - assert json_response(conn, 422) == %{ - "id" => workflow.id, - "errors" => %{ - "triggers" => [ - "A trigger cannot be replaced, only edited or added." - ] - } - } - end - - test "returns 401 without a token", %{project: project} do - workflow = insert(:simple_workflow, name: "work1", project: project) - - conn = - build_conn() - |> put_req_header("accept", "application/json") - |> put_req_header("content-type", "application/json") - |> put( - ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - Jason.encode!(%{workflow | name: "work2"}) - ) - - assert %{"error" => "Unauthorized"} == json_response(conn, 401) - end - end - - defp encode_decode(item) do - item - |> Jason.encode!() - |> Jason.decode!() - |> drop_workflow_state() - end - - defp drop_workflow_state(%{"state" => _} = workflow), - do: Map.delete(workflow, "state") - - defp drop_workflow_state(other), do: other - - defp remove_timestamps([%{"edges" => _el} | _workflows] = list) - when is_list(list) do - Enum.map(list, &Map.drop(&1, ["inserted_at", "updated_at"])) - end - - defp remove_timestamps(list) when is_list(list) do - Enum.map(list, &Map.drop(&1, ["inserted_at", "updated_at"])) - end - - defp remove_timestamps(workflow) do - Map.merge(workflow, %{ - "inserted_at" => nil, - "updated_at" => nil, - "edges" => remove_timestamps(workflow["edges"]), - "jobs" => remove_timestamps(workflow["jobs"]), - "triggers" => remove_timestamps(workflow["triggers"]) - }) - end - - defp get_saved_workflow(%{id: workflow_id}), - do: get_saved_workflow(workflow_id) - - defp get_saved_workflow(workflow_id), - do: Workflows.get_workflow(workflow_id, include: [:edges, :jobs, :triggers]) - - defp assert_response(response_workflow) do - response_workflow = encode_decode(response_workflow) - - saved_workflow = - response_workflow["id"] |> get_saved_workflow() |> encode_decode() - - assert MapSet.new(response_workflow["jobs"]) == - MapSet.new(saved_workflow["jobs"]) - - assert MapSet.new(response_workflow["edges"]) == - MapSet.new(saved_workflow["edges"]) - - assert MapSet.new(response_workflow["triggers"]) == - MapSet.new(saved_workflow["triggers"]) - end - - defp pluck_to_mapset(map, keys) do - map - |> Enum.into(MapSet.new(), fn e -> - Map.take(e, keys) - end) - end -end diff --git a/test/lightning_web/controllers/project_mfa_enforcement_test.exs b/test/lightning_web/controllers/project_mfa_enforcement_test.exs index 3fe12fad4a0..57a475772f7 100644 --- a/test/lightning_web/controllers/project_mfa_enforcement_test.exs +++ b/test/lightning_web/controllers/project_mfa_enforcement_test.exs @@ -9,8 +9,8 @@ defmodule LightningWeb.ProjectMFAEnforcementControllerTest do instead, which is the only place it is written. Nearly every assertion here funnels through the same `ProjectUsers` - `:access_project` check and so fires the same clause — - `permitted?(_action, %Scope{mfa_satisfied?: false})`. Another route adds no + `:access_project` check and so hits the same MFA refusal in + `Lightning.Policies.Decision.project/4`. Another route adds no MFA coverage on top of that. What a route proves, and the policy unit tests cannot, is that it reaches the policy at all rather than authorising off a raw query: this file is here to catch a bypass. @@ -37,6 +37,7 @@ defmodule LightningWeb.ProjectMFAEnforcementControllerTest do use LightningWeb.ConnCase, async: true import Lightning.Factories + import LightningWeb.API.ProblemHelpers # The member the policy is aimed at: a real member of an MFA-required # project who has never set up a second factor. @@ -209,10 +210,8 @@ defmodule LightningWeb.ProjectMFAEnforcementControllerTest do } end - # `:access_collection` is the read half of `Policies.Collections`, and the - # only half that delegates to `:access_project`. The write actions go - # through `Scope.role_in?/3` instead and are covered by the scope unit - # tests, so this is the clause those do not reach. + # These routes only read, so they reach `:access_collection`. The write + # actions are held to MFA by the same check in `Policies.Decision`. test "are refused for a member who has not enrolled", %{ conn: conn, user: user, @@ -221,11 +220,19 @@ defmodule LightningWeb.ProjectMFAEnforcementControllerTest do } do api_conn = bearer_conn(user) - assert get(api_conn, ~p"/collections/#{collection.name}").status == 401, - "an unenrolled member streamed a collection" + assert_problem( + get(api_conn, ~p"/collections/#{collection.name}"), + 403, + :forbidden, + [] + ) - assert get(api_conn, ~p"/collections/#{collection.name}/a-key").status == - 401 + assert_problem( + get(api_conn, ~p"/collections/#{collection.name}/a-key"), + 403, + :forbidden, + [] + ) assert get( conn, @@ -284,23 +291,20 @@ defmodule LightningWeb.ProjectMFAEnforcementControllerTest do } do workflow = insert(:workflow, project: project) |> with_snapshot() - for {path, refusals} <- [ - {~p"/api/projects/#{project.id}", [401, 403]}, - {~p"/api/projects/#{project.id}/workflows", [401, 403]}, - {~p"/api/projects/#{project.id}/workflows/#{workflow.id}", - [401, 403]}, - {~p"/api/projects/#{project.id}/credentials", [401, 403]}, - {~p"/api/projects/#{project.id}/work_orders", [401, 403]}, - {~p"/api/projects/#{project.id}/jobs", [401]}, - {~p"/api/projects/#{project.id}/runs", [401]}, - {~p"/api/provision/#{project.id}", [401, 403]}, - {~p"/api/provision/yaml?#{%{id: project.id}}", [401, 403]} + for path <- [ + ~p"/api/projects/#{project.id}", + ~p"/api/projects/#{project.id}/workflows", + ~p"/api/projects/#{project.id}/workflows/#{workflow.id}", + ~p"/api/projects/#{project.id}/credentials", + ~p"/api/projects/#{project.id}/work_orders", + ~p"/api/projects/#{project.id}/jobs", + ~p"/api/projects/#{project.id}/runs", + ~p"/api/provision/#{project.id}", + ~p"/api/provision/yaml?#{%{id: project.id}}" ] do - status = get(conn, path).status + conn = get(conn, path) - assert status in refusals, - "#{path} answered #{status} to an unenrolled member, not " <> - "#{inspect(refusals)}" + assert_problem(conn, 403, :forbidden, []) end end diff --git a/test/lightning_web/controllers/token_exchange_controller_test.exs b/test/lightning_web/controllers/token_exchange_controller_test.exs index 7709b863719..6eaf1892f31 100644 --- a/test/lightning_web/controllers/token_exchange_controller_test.exs +++ b/test/lightning_web/controllers/token_exchange_controller_test.exs @@ -31,7 +31,8 @@ defmodule LightningWeb.TokenExchangeControllerTest do "grant_type" => "client_credentials", "client_assertion_type" => @assertion_type, "client_assertion" => - sign_assertion(private_key, assertion_claims(account, audience)) + sign_assertion(private_key, assertion_claims(account, audience)), + "scope" => "users:read" }, extra ) @@ -52,7 +53,13 @@ defmodule LightningWeb.TokenExchangeControllerTest do "grant_types_supported" => ["client_credentials"], "token_endpoint_auth_methods_supported" => ["private_key_jwt"], "token_endpoint_auth_signing_alg_values_supported" => ["RS256"], - "scopes_supported" => ["users:read", "users:write"], + "scopes_supported" => [ + "users:read", + "users:write", + "projects:write", + "credentials:write", + "collections:write" + ], "response_types_supported" => [] } @@ -69,14 +76,18 @@ defmodule LightningWeb.TokenExchangeControllerTest do account: account, private_key: private_key } do - conn = - exchange(conn, valid_params(account, private_key, token_endpoint(conn))) + params = + valid_params(account, private_key, token_endpoint(conn), %{ + "scope" => "users:read users:write projects:write" + }) + + conn = exchange(conn, params) assert %{ "access_token" => access_token, "token_type" => "Bearer", "expires_in" => 300, - "scope" => "users:read users:write" + "scope" => "users:read users:write projects:write" } = json_response(conn, 200) assert get_resp_header(conn, "cache-control") == ["no-store"] @@ -95,7 +106,7 @@ defmodule LightningWeb.TokenExchangeControllerTest do "aud" => aud, "sub" => sub, "client_id" => sub, - "scope" => "users:read users:write", + "scope" => "users:read users:write projects:write", "iat" => iat, "exp" => exp, "jti" => jti @@ -166,6 +177,34 @@ defmodule LightningWeb.TokenExchangeControllerTest do conn |> exchange(params) |> json_response(400) end + test "refuses projects:provision, which only a repo connection holds", %{ + conn: conn, + account: account, + private_key: private_key + } do + params = + valid_params(account, private_key, token_endpoint(conn), %{ + "scope" => "projects:provision" + }) + + assert %{"error" => "invalid_scope"} = + conn |> exchange(params) |> json_response(400) + end + + test "refuses a request that names no scope with invalid_scope", %{ + conn: conn, + account: account, + private_key: private_key + } do + params = valid_params(account, private_key, token_endpoint(conn)) + + for params <- [Map.delete(params, "scope"), Map.put(params, "scope", " ")] do + assert conn |> exchange(params) |> json_response(400) == %{ + "error" => "invalid_scope" + } + end + end + test "refuses a failed assertion with invalid_client", %{ conn: conn, account: account, diff --git a/test/lightning_web/controllers/user_auth_test.exs b/test/lightning_web/controllers/user_auth_test.exs index c6be6abb86c..2cf58b4dee9 100644 --- a/test/lightning_web/controllers/user_auth_test.exs +++ b/test/lightning_web/controllers/user_auth_test.exs @@ -5,6 +5,7 @@ defmodule LightningWeb.UserAuthTest do alias LightningWeb.UserAuth import Lightning.AccountsFixtures import Lightning.Factories + import LightningWeb.API.ProblemHelpers @remember_me_cookie "_lightning_web_user_remember_me" @@ -87,16 +88,11 @@ defmodule LightningWeb.UserAuthTest do end describe "require_authenticated_api_resource/2" do - setup %{conn: conn} do - # The :api pipeline accepts JSON only, so the refusal renders as JSON. - %{conn: Phoenix.Controller.put_format(conn, "json")} - end - test "refuses a request with no resource", %{conn: conn} do conn = UserAuth.require_authenticated_api_resource(conn, []) assert conn.halted - assert json_response(conn, 401) == %{"error" => "Unauthorized"} + assert_unauthenticated(conn, :unauthenticated, ["Bearer"]) end test "refuses a user past their confirmation deadline", %{conn: conn} do @@ -104,11 +100,15 @@ defmodule LightningWeb.UserAuthTest do conn = conn + |> put_req_header("authorization", "Bearer a-locked-users-token") |> assign(:current_resource, unconfirmed_user(hours_old: 50)) |> UserAuth.require_authenticated_api_resource([]) assert conn.halted - assert json_response(conn, 401) == %{"error" => "Unauthorized"} + + assert_unauthenticated(conn, :invalid_token, [ + ~s(Bearer error="invalid_token") + ]) end test "passes a confirmed user, a user inside the grace period, and an unconfirmed user with the flag off", @@ -494,8 +494,7 @@ defmodule LightningWeb.UserAuthTest do |> UserAuth.require_authenticated_user([]) assert conn.halted - assert conn.status == 401 - assert conn.resp_body |> Jason.decode!() == %{"error" => "Unauthorized"} + assert_session_refused(conn) end test "stores the path to redirect to on GET", %{conn: conn} do @@ -576,8 +575,7 @@ defmodule LightningWeb.UserAuthTest do |> UserAuth.require_authenticated_user([]) assert conn.halted - assert conn.status == 401 - assert conn.resp_body |> Jason.decode!() == %{"error" => "Unauthorized"} + assert_session_refused(conn) end end @@ -626,8 +624,7 @@ defmodule LightningWeb.UserAuthTest do |> UserAuth.require_authenticated_user([]) assert conn.halted - assert conn.status == 401 - assert Jason.decode!(conn.resp_body) == %{"error" => "Unauthorized"} + assert_session_refused(conn) end test "serves the paths a locked-out account still needs", %{ @@ -911,21 +908,6 @@ defmodule LightningWeb.UserAuthTest do assert conn |> redirected_to() == "/projects" end - test "returns 403 if user is not superuser and json request ", %{ - conn: conn, - user: user - } do - conn = - conn - |> assign(:current_user, user) - |> Phoenix.Controller.put_format("json") - |> UserAuth.require_superuser([]) - - assert conn.halted - assert conn.status == 403 - assert conn.resp_body |> Jason.decode!() == %{"error" => "Forbidden"} - end - test "allows the request to proceed if user is a superuser", %{conn: conn} do user = superuser_fixture() diff --git a/test/lightning_web/controllers/webhooks_controller_dataclip_test.exs b/test/lightning_web/controllers/webhooks_controller_dataclip_test.exs new file mode 100644 index 00000000000..3f764da027e --- /dev/null +++ b/test/lightning_web/controllers/webhooks_controller_dataclip_test.exs @@ -0,0 +1,292 @@ +defmodule LightningWeb.WebhooksControllerDataclipTest do + use LightningWeb.ConnCase, async: false + + import Lightning.Factories + import Mox + + alias Lightning.Repo + alias Lightning.Runs + alias Lightning.WorkOrders + + @moduletag capture_log: true + + setup :set_mox_from_context + setup :verify_on_exit! + + setup do + Mox.stub(Lightning.MockConfig, :cors_origin, fn -> "*" end) + :ok + end + + describe "the request stored with the dataclip" do + setup [:stub_rate_limiter_ok, :stub_usage_limiter_ok] + + test "creates a pending workorder with a valid trigger and an additional path", + %{conn: conn} do + %{triggers: [%{id: trigger_id}]} = + insert(:simple_workflow) + |> Lightning.Repo.preload(:triggers) + |> with_snapshot() + + message = %{"foo" => "bar"} + conn = post(conn, "/i/#{trigger_id}/Patient", message) + + assert %{"work_order_id" => work_order_id} = + json_response(conn, 200) + + assert %{trigger: %{id: ^trigger_id}, runs: [run], state: :pending} = + WorkOrders.get(work_order_id, + include: [:runs, :dataclip, :trigger] + ) + + assert %{starting_trigger_id: ^trigger_id} = run + + assert Repo.all(Lightning.Invocation.Dataclip) |> Enum.count() == 1 + + assert Runs.get_dataclip_body(run) == ~s({"foo": "bar"}) + + assert Runs.get_dataclip_request(run) == + ~s({\"path\": [\"i\", \"#{trigger_id}\", \"Patient\"], \"method\": \"POST\", \"headers\": {\"content-type\": \"multipart/mixed; boundary=plug_conn_test\"}, \"query_params\": {}}) + end + + test "creates a pending workorder with a valid trigger and some query params", + %{conn: conn} do + %{triggers: [%{id: trigger_id}]} = + insert(:simple_workflow) + |> Lightning.Repo.preload(:triggers) + |> with_snapshot() + + message = %{"foo" => "bar"} + conn = post(conn, "/i/#{trigger_id}?extra=stuff&moar=things", message) + + assert %{"work_order_id" => work_order_id} = + json_response(conn, 200) + + assert %{trigger: %{id: ^trigger_id}, runs: [run], state: :pending} = + WorkOrders.get(work_order_id, + include: [:runs, :dataclip, :trigger] + ) + + assert %{starting_trigger_id: ^trigger_id} = run + + assert Repo.all(Lightning.Invocation.Dataclip) |> Enum.count() == 1 + + assert Runs.get_dataclip_body(run) == ~s({"foo": "bar"}) + + assert Runs.get_dataclip_request(run) == + ~s({\"path\": [\"i\", \"#{trigger_id}\"], \"method\": \"POST\", \"headers\": {\"content-type\": \"multipart/mixed; boundary=plug_conn_test\"}, \"query_params\": {\"moar\": \"things\", \"extra\": \"stuff\"}}) + end + end + + describe "sensitive request headers" do + setup [:stub_rate_limiter_ok, :stub_usage_limiter_ok] + + test "the x-api-key consumed by webhook auth is not persisted in dataclip.request", + %{conn: conn} do + %{triggers: [trigger]} = + insert(:simple_workflow) + |> Lightning.Repo.preload(:triggers) + |> with_snapshot() + + auth_method = + insert(:webhook_auth_method, + auth_type: :api, + api_key: "sup3r-s3cret-api-key" + ) + + associate_auth_methods(trigger, [auth_method]) + + conn = + conn + |> put_req_header("x-api-key", "sup3r-s3cret-api-key") + |> post("/i/#{trigger.id}", %{"foo" => "bar"}) + + assert %{"work_order_id" => work_order_id} = json_response(conn, 200) + + %{runs: [run]} = WorkOrders.get(work_order_id, include: [:runs]) + + request = Runs.get_dataclip_request(run) |> Jason.decode!() + + assert request["headers"]["x-api-key"] == "[REDACTED]", + "the api key Lightning already consumed to authenticate must not be retained" + + refute Runs.get_dataclip_request(run) =~ "sup3r-s3cret-api-key" + end + + test "the Basic Authorization consumed by webhook auth is redacted from dataclip.request", + %{conn: conn} do + %{triggers: [trigger]} = + insert(:simple_workflow) + |> Lightning.Repo.preload(:triggers) + |> with_snapshot() + + auth_method = + insert(:webhook_auth_method, + auth_type: :basic, + username: "caller", + password: "sup3r-s3cret-password" + ) + + associate_auth_methods(trigger, [auth_method]) + + credentials = Base.encode64("caller:sup3r-s3cret-password") + + conn = + conn + |> put_req_header("authorization", "Basic #{credentials}") + |> post("/i/#{trigger.id}", %{"foo" => "bar"}) + + assert %{"work_order_id" => work_order_id} = json_response(conn, 200) + + %{runs: [run]} = WorkOrders.get(work_order_id, include: [:runs]) + + request = Runs.get_dataclip_request(run) |> Jason.decode!() + + assert request["headers"]["authorization"] == "Basic [REDACTED]", + "the Basic credentials Lightning already consumed must not be " <> + "retained; the scheme is not a secret, so it stays legible" + + raw = Runs.get_dataclip_request(run) + refute raw =~ credentials + refute raw =~ "sup3r-s3cret-password" + end + + test "a caller's own x-api-key survives when it is not the trigger's secret", + %{conn: conn} do + %{triggers: [trigger]} = + insert(:simple_workflow) + |> Lightning.Repo.preload(:triggers) + |> with_snapshot() + + auth_method = + insert(:webhook_auth_method, + auth_type: :basic, + username: "caller", + password: "sup3r-s3cret-password" + ) + + associate_auth_methods(trigger, [auth_method]) + + credentials = Base.encode64("caller:sup3r-s3cret-password") + + conn = + conn + |> put_req_header("authorization", "Basic #{credentials}") + |> put_req_header("x-api-key", "a-token-for-the-job-to-forward") + |> post("/i/#{trigger.id}", %{"foo" => "bar"}) + + assert %{"work_order_id" => work_order_id} = json_response(conn, 200) + + %{runs: [run]} = WorkOrders.get(work_order_id, include: [:runs]) + + request = Runs.get_dataclip_request(run) |> Jason.decode!() + + assert request["headers"]["authorization"] == "Basic [REDACTED]" + + assert request["headers"]["x-api-key"] == + "a-token-for-the-job-to-forward", + "redaction is by value, not by header name: a token Lightning " <> + "never consumed belongs to the caller and their job" + end + + test "our api key is redacted from both headers webhook auth reads", + %{conn: conn} do + %{triggers: [trigger]} = + insert(:simple_workflow) + |> Lightning.Repo.preload(:triggers) + |> with_snapshot() + + auth_method = + insert(:webhook_auth_method, + auth_type: :api, + api_key: "sup3r-s3cret-api-key" + ) + + associate_auth_methods(trigger, [auth_method]) + + conn = + conn + |> put_req_header("x-api-key", "sup3r-s3cret-api-key") + |> put_req_header("authorization", "Bearer sup3r-s3cret-api-key") + |> put_req_header("x-copy-of-it", "sup3r-s3cret-api-key") + |> post("/i/#{trigger.id}", %{"foo" => "bar"}) + + assert %{"work_order_id" => work_order_id} = json_response(conn, 200) + + %{runs: [run]} = WorkOrders.get(work_order_id, include: [:runs]) + + request = Runs.get_dataclip_request(run) |> Jason.decode!() + + assert request["headers"]["x-api-key"] == "[REDACTED]" + + assert request["headers"]["authorization"] == "Bearer [REDACTED]", + "auth matched on x-api-key, but the copy in the other header we " <> + "read would otherwise be persisted intact" + + assert request["headers"]["x-copy-of-it"] == "sup3r-s3cret-api-key", + "a header webhook auth does not read is passed through verbatim, " <> + "secret-shaped or not" + end + + test "cookie and proxy-authorization are redacted from dataclip.request", + %{conn: conn} do + %{triggers: [trigger]} = + insert(:simple_workflow) + |> Lightning.Repo.preload(:triggers) + |> with_snapshot() + + conn = + conn + |> put_req_header("cookie", "session=a-third-party-session-cookie") + |> put_req_header("proxy-authorization", "Basic cHJveHk6c2VjcmV0") + |> post("/i/#{trigger.id}", %{"foo" => "bar"}) + + assert %{"work_order_id" => work_order_id} = json_response(conn, 200) + + %{runs: [run]} = WorkOrders.get(work_order_id, include: [:runs]) + + request = Runs.get_dataclip_request(run) |> Jason.decode!() + + assert request["headers"]["cookie"] == "[REDACTED]", + "caller session cookies are a load-balancer/front-end concern, not run state" + + assert request["headers"]["proxy-authorization"] == "[REDACTED]" + + raw = Runs.get_dataclip_request(run) + refute raw =~ "a-third-party-session-cookie" + refute raw =~ "cHJveHk6c2VjcmV0" + end + + test "non-sensitive caller headers are still passed through verbatim", %{ + conn: conn + } do + %{triggers: [trigger]} = + insert(:simple_workflow) + |> Lightning.Repo.preload(:triggers) + |> with_snapshot() + + conn = + conn + |> put_req_header("x-request-id", "caller-supplied-id") + |> put_req_header("user-agent", "acme-integration/1.2") + |> post("/i/#{trigger.id}", %{"foo" => "bar"}) + + assert %{"work_order_id" => work_order_id} = json_response(conn, 200) + + %{runs: [run]} = WorkOrders.get(work_order_id, include: [:runs]) + + request = Runs.get_dataclip_request(run) |> Jason.decode!() + + assert request["headers"]["x-request-id"] == "caller-supplied-id" + assert request["headers"]["user-agent"] == "acme-integration/1.2" + end + end + + defp associate_auth_methods(trigger, auth_methods) do + trigger + |> Repo.preload(:webhook_auth_methods) + |> Ecto.Changeset.change() + |> Ecto.Changeset.put_assoc(:webhook_auth_methods, auth_methods) + |> Repo.update!() + end +end diff --git a/test/lightning_web/controllers/webhooks_controller_reply_test.exs b/test/lightning_web/controllers/webhooks_controller_reply_test.exs new file mode 100644 index 00000000000..e88542143c9 --- /dev/null +++ b/test/lightning_web/controllers/webhooks_controller_reply_test.exs @@ -0,0 +1,285 @@ +defmodule LightningWeb.WebhooksControllerReplyTest do + use LightningWeb.ConnCase, async: false + + import Lightning.Factories + import Mox + import LightningWeb.API.ProblemHelpers + + alias Lightning.Extensions.MockUsageLimiter + alias Lightning.Extensions.StubUsageLimiter + + alias Lightning.Repo + alias Lightning.WorkOrders + + @moduletag capture_log: true + + setup :set_mox_from_context + setup :verify_on_exit! + + setup do + Mox.stub(Lightning.MockConfig, :cors_origin, fn -> "*" end) + :ok + end + + describe "delayed webhook response (webhook_reply: :after_completion)" do + setup [:stub_rate_limiter_ok, :stub_usage_limiter_ok] + + test "waits for and returns the broadcast response body and status code", %{ + conn: conn + } do + %{triggers: [trigger], project_id: project_id} = + insert(:simple_workflow) + |> Lightning.Repo.preload(:triggers) + |> with_snapshot() + + trigger = + trigger + |> Ecto.Changeset.change(webhook_reply: :after_completion) + |> Repo.update!() + + Lightning.WorkOrders.Events.subscribe(project_id) + + test_pid = self() + + task = + Task.async(fn -> + conn = post(conn, "/i/#{trigger.id}", %{"foo" => "bar"}) + send(test_pid, {:response, conn}) + end) + + assert_receive %Lightning.WorkOrders.Events.WorkOrderCreated{ + work_order: work_order + } + + assert_receive %Lightning.WorkOrders.Events.RunCreated{run: run} + + body = %{"result" => "success", "value" => 42} + + reply_to_webhook(work_order, 200, body) + + assert_receive {:response, response_conn}, 5_000 + + assert json_response(response_conn, 200) == body + + assert get_resp_header(response_conn, "x-meta-work-order-id") == [ + work_order.id + ] + + assert get_resp_header(response_conn, "x-meta-run-id") == [run.id] + + Task.await(task) + end + + test "passes through any status code from the broadcast message", %{ + conn: conn + } do + %{triggers: [trigger], project_id: project_id} = + insert(:simple_workflow) + |> Lightning.Repo.preload(:triggers) + |> with_snapshot() + + trigger = + trigger + |> Ecto.Changeset.change(webhook_reply: :after_completion) + |> Repo.update!() + + Lightning.WorkOrders.Events.subscribe(project_id) + + test_pid = self() + + task = + Task.async(fn -> + conn = post(conn, "/i/#{trigger.id}", %{"foo" => "bar"}) + send(test_pid, {:response, conn}) + end) + + assert_receive %Lightning.WorkOrders.Events.WorkOrderCreated{ + work_order: work_order + } + + assert_receive %Lightning.WorkOrders.Events.RunCreated{run: run} + + body = %{"message" => "run failed"} + + reply_to_webhook(work_order, 422, body) + + assert_receive {:response, response_conn}, 5_000 + + assert json_response(response_conn, 422) == body + + assert get_resp_header(response_conn, "x-meta-work-order-id") == [ + work_order.id + ] + + assert get_resp_header(response_conn, "x-meta-run-id") == [run.id] + + Task.await(task) + end + + test "returns timeout if workflow doesn't complete within timeout period", %{ + conn: conn + } do + expect(Lightning.MockConfig, :webhook_response_timeout_ms, fn -> 2_000 end) + + %{triggers: [trigger]} = + insert(:simple_workflow) + |> Lightning.Repo.preload(:triggers) + |> with_snapshot() + + trigger = + trigger + |> Ecto.Changeset.change(webhook_reply: :after_completion) + |> Repo.update!() + + conn = post(conn, "/i/#{trigger.id}", %{"foo" => "bar"}) + + [work_order_id] = get_resp_header(conn, "x-meta-work-order-id") + + body = assert_problem(conn, 504, :timeout, []) + assert body["work_order_id"] == work_order_id + + assert WorkOrders.get(work_order_id) + end + + for status_code <- [204, 304] do + test "sends empty body when broadcast status is #{status_code}", %{ + conn: conn + } do + %{triggers: [trigger], project_id: project_id} = + insert(:simple_workflow) + |> Lightning.Repo.preload(:triggers) + |> with_snapshot() + + trigger = + trigger + |> Ecto.Changeset.change(webhook_reply: :after_completion) + |> Repo.update!() + + Lightning.WorkOrders.Events.subscribe(project_id) + + test_pid = self() + + task = + Task.async(fn -> + conn = post(conn, "/i/#{trigger.id}", %{"foo" => "bar"}) + send(test_pid, {:response, conn}) + end) + + assert_receive %Lightning.WorkOrders.Events.WorkOrderCreated{ + work_order: work_order + } + + assert_receive %Lightning.WorkOrders.Events.RunCreated{run: _run} + + reply_to_webhook(work_order, unquote(status_code), nil) + + assert_receive {:response, response_conn}, 5_000 + + assert response_conn.status == unquote(status_code) + assert response_conn.resp_body == "" + + Task.await(task) + end + end + + test "replies with the rejection instead of waiting when no run was created", + %{conn: conn} do + Mox.stub(MockUsageLimiter, :limit_action, &StubUsageLimiter.limit_action/2) + + Mox.stub(Lightning.MockConfig, :webhook_response_timeout_ms, fn -> + 60_000 + end) + + %{triggers: [trigger]} = + insert(:simple_workflow) + |> Lightning.Repo.preload(:triggers) + |> with_snapshot() + + trigger + |> Ecto.Changeset.change(webhook_reply: :after_completion) + |> Repo.update!() + + {elapsed_us, conn} = + :timer.tc(fn -> post(conn, "/i/#{trigger.id}", %{"foo" => "bar"}) end) + + assert elapsed_us < 5_000_000, + "request waited on a response nothing can send" + + response = assert_problem(conn, 429, :too_many_runs, []) + work_order_id = response["work_order_id"] + + refute Map.has_key?(response, "message"), + "the limiter's project-facing copy must not reach /i/*" + + work_order = WorkOrders.get(work_order_id, include: [:runs]) + assert work_order.state == :rejected + assert work_order.runs == [] + end + + test "returns immediately when webhook_reply is :custom, which has no publisher", + %{conn: conn} do + Mox.stub(Lightning.MockConfig, :webhook_response_timeout_ms, fn -> + 60_000 + end) + + %{triggers: [trigger]} = + insert(:simple_workflow) + |> Lightning.Repo.preload(:triggers) + |> with_snapshot() + + trigger + |> Ecto.Changeset.change(webhook_reply: :custom) + |> Repo.update!() + + {elapsed_us, conn} = + :timer.tc(fn -> post(conn, "/i/#{trigger.id}", %{"foo" => "bar"}) end) + + assert elapsed_us < 5_000_000, + "request waited on a response nothing can send" + + [work_order_id] = get_resp_header(conn, "x-meta-work-order-id") + assert %{"work_order_id" => ^work_order_id} = json_response(conn, 200) + end + + test "returns immediately when webhook_reply is before_start (default)", %{ + conn: conn + } do + %{triggers: [trigger]} = + insert(:simple_workflow) + |> Lightning.Repo.preload(:triggers) + |> with_snapshot() + + assert trigger.webhook_reply == :before_start + + conn = post(conn, "/i/#{trigger.id}", %{"foo" => "bar"}) + + [work_order_id] = get_resp_header(conn, "x-meta-work-order-id") + assert %{"work_order_id" => ^work_order_id} = json_response(conn, 200) + assert WorkOrders.get(work_order_id) + end + end + + # The controller subscribes only after the work order and run exist, so a + # reply sent on RunCreated can arrive before anyone is listening. + defp reply_to_webhook(work_order, status, body) do + topic = "work_order:#{work_order.id}:webhook_response" + await_subscriber(topic, 100) + + Phoenix.PubSub.broadcast( + Lightning.PubSub, + topic, + {:webhook_response, status, body} + ) + end + + defp await_subscriber(topic, 0), + do: flunk("no subscriber on #{topic} after 1s") + + defp await_subscriber(topic, attempts) do + # Phoenix.PubSub keeps subscribers in a Registry named after the server. + if Registry.lookup(Lightning.PubSub, topic) == [] do + Process.sleep(10) + await_subscriber(topic, attempts - 1) + end + end +end diff --git a/test/lightning_web/controllers/webhooks_controller_test.exs b/test/lightning_web/controllers/webhooks_controller_test.exs index 762cb3e34a4..58ef0877502 100644 --- a/test/lightning_web/controllers/webhooks_controller_test.exs +++ b/test/lightning_web/controllers/webhooks_controller_test.exs @@ -3,11 +3,11 @@ defmodule LightningWeb.WebhooksControllerTest do import Lightning.Factories import Mox + import LightningWeb.API.ProblemHelpers alias Lightning.Extensions.MockRateLimiter - alias Lightning.Extensions.StubRateLimiter alias Lightning.Extensions.MockUsageLimiter - alias Lightning.Extensions.StubUsageLimiter + alias Lightning.Extensions.StubRateLimiter alias Lightning.Repo alias Lightning.Runs @@ -26,52 +26,9 @@ defmodule LightningWeb.WebhooksControllerTest do describe "a POST request to '/i'" do setup [:stub_rate_limiter_ok, :stub_usage_limiter_ok] - test "returns 200 and the rejection reason when run soft limit has been reached", - %{conn: conn} do - Mox.stub(MockUsageLimiter, :limit_action, &StubUsageLimiter.limit_action/2) - - %{triggers: [trigger]} = - insert(:simple_workflow) - |> Lightning.Repo.preload(:triggers) - |> with_snapshot() - - conn = post(conn, "/i/#{trigger.id}") - - response = json_response(conn, 200) - - assert %{"work_order_id" => work_order_id, "error" => "too_many_runs"} = - response - - assert Ecto.UUID.dump(work_order_id) - - refute Map.has_key?(response, "message"), - "the limiter's project-facing copy must not reach /i/*" - - work_order = WorkOrders.get(work_order_id, include: [:runs]) - assert work_order.state == :rejected - assert work_order.runs == [] - end - - test "returns 429 when run limit has been reached", %{conn: conn} do - Mox.stub(MockUsageLimiter, :limit_action, fn _action, _ctx -> - {:error, :runs_hard_limit, - %Lightning.Extensions.Message{text: "Runs limit exceeded"}} - end) - - %{triggers: [trigger]} = - insert(:simple_workflow) |> Lightning.Repo.preload(:triggers) - - conn = post(conn, "/i/#{trigger.id}") - - assert json_response(conn, 429) == %{ - "error" => "runs_hard_limit", - "message" => "Runs limit exceeded" - } - end - test "returns 404 when trigger does not exist", %{conn: conn} do conn = post(conn, "/i/bar") - assert json_response(conn, 404) == %{"error" => "Webhook not found"} + assert_problem(conn, 404, :not_found, []) end test "returns 413 with a body exceeding the limit", %{conn: conn} do @@ -90,7 +47,11 @@ defmodule LightningWeb.WebhooksControllerTest do exceeding_body = %{"data" => %{a: String.duplicate("a", 2_000_000)}} - assert {:ok, %Tesla.Env{status: 413, body: "Request Entity Too Large"}} = + assert {:ok, + %Tesla.Env{ + status: 413, + body: %{"status" => 413, "code" => "content_too_large"} + }} = [ {Tesla.Middleware.BaseUrl, LightningWeb.Endpoint.url()}, Tesla.Middleware.JSON @@ -102,30 +63,17 @@ defmodule LightningWeb.WebhooksControllerTest do ) end - test "returns 429 on rate limiting", %{conn: conn} do - Mox.stub(MockRateLimiter, :limit_request, &StubRateLimiter.limit_request/3) - + test "returns a 200 and the trigger-found message when a valid GET is sent", + %{conn: conn} do %{triggers: [trigger]} = - insert(:simple_workflow) |> Lightning.Repo.preload(:triggers) + insert(:simple_workflow) |> Repo.preload(:triggers) - conn = post(conn, "/i/#{trigger.id}") - - assert json_response(conn, 429) == %{ - "error" => "too_many_requests", - "message" => "Too many runs in the last minute" - } - end - - test "returns a 200 when a valid GET is sent", %{conn: conn} do - %{triggers: [%{id: trigger_id}]} = - insert(:simple_workflow) |> Lightning.Repo.preload(:triggers) + conn = get(conn, "/i/#{trigger.id}") - conn = get(conn, "/i/#{trigger_id}") + assert conn.status == 200 - assert json_response(conn, 200) == %{ - "message" => - "OpenFn webhook trigger found. Make a POST request to execute this workflow." - } + assert conn.resp_body == + ~s({"message":"OpenFn webhook trigger found. Make a POST request to execute this workflow."}) end test "returns 404 when trigger does not exist for GET request", %{conn: conn} do @@ -133,7 +81,7 @@ defmodule LightningWeb.WebhooksControllerTest do conn = get(conn, "/i/#{non_existent_trigger_id}") - assert json_response(conn, 404) == %{"error" => "Webhook not found"} + assert_problem(conn, 404, :not_found, []) end test "returns 404 when trigger exists but is of type cron", %{conn: conn} do @@ -147,7 +95,7 @@ defmodule LightningWeb.WebhooksControllerTest do conn = get(conn, "/i/#{trigger_id}") - assert json_response(conn, 404) == %{"error" => "Webhook not found"} + assert_problem(conn, 404, :not_found, []) end test "creates a pending workorder with a valid trigger", %{conn: conn} do @@ -159,8 +107,10 @@ defmodule LightningWeb.WebhooksControllerTest do message = %{"foo" => "bar"} conn = post(conn, "/i/#{trigger_id}", message) - assert %{"work_order_id" => work_order_id} = - json_response(conn, 200) + assert conn.status == 200 + + [work_order_id] = get_resp_header(conn, "x-meta-work-order-id") + assert conn.resp_body == ~s({"work_order_id":"#{work_order_id}"}) assert %{trigger: %{id: ^trigger_id}, runs: [run], state: :pending} = WorkOrders.get(work_order_id, @@ -228,73 +178,39 @@ defmodule LightningWeb.WebhooksControllerTest do WorkOrders.get(work_order_id, include: [:trigger]) end - test "creates a pending workorder with a valid trigger and an additional path", - %{conn: conn} do + test "returns 415 when client sends xml", %{conn: conn} do %{triggers: [%{id: trigger_id}]} = - insert(:simple_workflow) - |> Lightning.Repo.preload(:triggers) - |> with_snapshot() - - message = %{"foo" => "bar"} - conn = post(conn, "/i/#{trigger_id}/Patient", message) - - assert %{"work_order_id" => work_order_id} = - json_response(conn, 200) - - assert %{trigger: %{id: ^trigger_id}, runs: [run], state: :pending} = - WorkOrders.get(work_order_id, - include: [:runs, :dataclip, :trigger] - ) - - assert %{starting_trigger_id: ^trigger_id} = run - - assert Repo.all(Lightning.Invocation.Dataclip) |> Enum.count() == 1 + insert(:simple_workflow) |> Lightning.Repo.preload(:triggers) - assert Runs.get_dataclip_body(run) == ~s({"foo": "bar"}) + conn = + conn + |> put_req_header("content-type", "text/xml") + |> put_req_header("accepts", "*/*") + |> post("/i/#{trigger_id}", "{}") - assert Runs.get_dataclip_request(run) == - ~s({\"path\": [\"i\", \"#{trigger_id}\", \"Patient\"], \"method\": \"POST\", \"headers\": {\"content-type\": \"multipart/mixed; boundary=plug_conn_test\"}, \"query_params\": {}}) + assert_problem(conn, 415, :unsupported_media_type, []) end - test "creates a pending workorder with a valid trigger and some query params", - %{conn: conn} do + test "returns 400 malformed for a body that isn't JSON", %{conn: conn} do %{triggers: [%{id: trigger_id}]} = - insert(:simple_workflow) - |> Lightning.Repo.preload(:triggers) - |> with_snapshot() - - message = %{"foo" => "bar"} - conn = post(conn, "/i/#{trigger_id}?extra=stuff&moar=things", message) - - assert %{"work_order_id" => work_order_id} = - json_response(conn, 200) - - assert %{trigger: %{id: ^trigger_id}, runs: [run], state: :pending} = - WorkOrders.get(work_order_id, - include: [:runs, :dataclip, :trigger] - ) - - assert %{starting_trigger_id: ^trigger_id} = run - - assert Repo.all(Lightning.Invocation.Dataclip) |> Enum.count() == 1 - - assert Runs.get_dataclip_body(run) == ~s({"foo": "bar"}) + insert(:simple_workflow) |> Lightning.Repo.preload(:triggers) - assert Runs.get_dataclip_request(run) == - ~s({\"path\": [\"i\", \"#{trigger_id}\"], \"method\": \"POST\", \"headers\": {\"content-type\": \"multipart/mixed; boundary=plug_conn_test\"}, \"query_params\": {\"moar\": \"things\", \"extra\": \"stuff\"}}) + conn + |> put_req_header("content-type", "application/json") + |> post("/i/#{trigger_id}", ~s({"foo": )) + |> assert_problem(400, :malformed, []) end - test "returns 415 when client sends xml", %{conn: conn} do + test "returns 400 bad_request for a form body that isn't UTF-8", %{ + conn: conn + } do %{triggers: [%{id: trigger_id}]} = insert(:simple_workflow) |> Lightning.Repo.preload(:triggers) - conn = - conn - |> put_req_header("content-type", "text/xml") - |> put_req_header("accepts", "*/*") - |> post("/i/#{trigger_id}", "{}") - - assert response(conn, 415) == ~s({"error":"Unsupported Media Type"}) + conn + |> put_req_header("content-type", "application/x-www-form-urlencoded") + |> post("/i/#{trigger_id}", "a=%FF") + |> assert_problem(400, :bad_request, []) end end @@ -312,10 +228,7 @@ defmodule LightningWeb.WebhooksControllerTest do } do conn = post(conn, "/i/#{trigger_id}", message) - assert %{"message" => response_message} = json_response(conn, 403) - - assert response_message =~ - "Unable to process request, trigger is disabled." + assert_problem(conn, 403, :trigger_disabled, []) end end @@ -356,13 +269,7 @@ defmodule LightningWeb.WebhooksControllerTest do conn = post(conn, "/i/#{trigger.id}") - assert json_response(conn, 503) == %{ - "error" => "service_unavailable", - "message" => - "Unable to process request due to temporary database issues. Please try again in 1s.", - "retry_after" => 1 - } - + assert_problem(conn, 503, :service_unavailable, []) assert get_resp_header(conn, "retry-after") == ["1"] end @@ -406,48 +313,18 @@ defmodule LightningWeb.WebhooksControllerTest do describe "create/2 controller error branches (422 + nil fallback)" do setup [:stub_rate_limiter_ok, :stub_usage_limiter_ok] - test "returns 422 invalid_request with details when WorkOrders.create_for returns a changeset error", + test "returns 422 invalid with one entry at the body when the work order can't be saved", %{conn: conn} do %{triggers: [trigger]} = - insert(:simple_workflow) - |> Lightning.Repo.preload(:triggers) - |> with_snapshot() - - bad_changeset = - %Lightning.WorkOrder{} - |> Ecto.Changeset.change() - |> Ecto.Changeset.add_error(:dataclip, "is invalid") - - Mimic.copy(Lightning.WorkOrders) - - Mimic.expect(Lightning.WorkOrders, :create_for, fn _trigger, _opts -> - {:error, bad_changeset} - end) + insert(:simple_workflow) |> Lightning.Repo.preload(:triggers) conn = post(conn, "/i/#{trigger.id}", %{"foo" => "bar"}) - assert %{"error" => "invalid_request", "details" => details} = - json_response(conn, 422) + assert_problem(conn, 422, :invalid, [ + %{"pointer" => "#", "code" => "invalid", "detail" => "is invalid"} + ]) - assert Map.has_key?(details, "dataclip") - end - - test "returns 422 with atom reason when WorkOrders.create_for returns {:error, reason}", - %{conn: conn} do - %{triggers: [trigger]} = - insert(:simple_workflow) - |> Lightning.Repo.preload(:triggers) - |> with_snapshot() - - Mimic.copy(Lightning.WorkOrders) - - Mimic.expect(Lightning.WorkOrders, :create_for, fn _trigger, _opts -> - {:error, :bad_payload} - end) - - conn = post(conn, "/i/#{trigger.id}", %{"foo" => "bar"}) - - assert json_response(conn, 422) == %{"error" => "bad_payload"} + assert Repo.aggregate(Lightning.WorkOrder, :count) == 0 end test "returns 404 when controller receives nil trigger assign (fallback path)" do @@ -456,467 +333,118 @@ defmodule LightningWeb.WebhooksControllerTest do conn = Phoenix.ConnTest.build_conn(:post, "/i/nonexistent") conn = LightningWeb.WebhooksController.create(conn, %{}) - assert conn.status == 404 - assert Jason.decode!(conn.resp_body) == %{"error" => "Webhook not found"} + assert_problem(conn, 404, :not_found, []) end end - describe "delayed webhook response (webhook_reply: :after_completion)" do - setup [:stub_rate_limiter_ok, :stub_usage_limiter_ok] + describe "the challenges on a 401" do + for {types, challenges} <- [ + {[:basic], []}, + {[:api], [~s(ApiKey header="x-api-key")]}, + {[:api, :basic], [~s(ApiKey header="x-api-key")]} + ] do + test "offer an API key, never Basic, on a trigger with #{inspect(types)} methods", + %{conn: conn} do + %{triggers: [trigger]} = + insert(:simple_workflow) |> Repo.preload(:triggers) - test "waits for and returns the broadcast response body and status code", %{ - conn: conn - } do - %{triggers: [trigger], project_id: project_id} = - insert(:simple_workflow) - |> Lightning.Repo.preload(:triggers) - |> with_snapshot() - - trigger = - trigger - |> Ecto.Changeset.change(webhook_reply: :after_completion) - |> Repo.update!() - - Lightning.WorkOrders.Events.subscribe(project_id) - - test_pid = self() - - task = - Task.async(fn -> - conn = post(conn, "/i/#{trigger.id}", %{"foo" => "bar"}) - send(test_pid, {:response, conn}) - end) - - assert_receive %Lightning.WorkOrders.Events.WorkOrderCreated{ - work_order: work_order - } - - assert_receive %Lightning.WorkOrders.Events.RunCreated{run: run} - - body = %{"result" => "success", "value" => 42} - - Phoenix.PubSub.broadcast( - Lightning.PubSub, - "work_order:#{work_order.id}:webhook_response", - {:webhook_response, 200, body} - ) - - assert_receive {:response, response_conn}, 5_000 - - assert json_response(response_conn, 200) == body - - assert get_resp_header(response_conn, "x-meta-work-order-id") == [ - work_order.id - ] - - assert get_resp_header(response_conn, "x-meta-run-id") == [run.id] - - Task.await(task) - end - - test "passes through any status code from the broadcast message", %{ - conn: conn - } do - %{triggers: [trigger], project_id: project_id} = - insert(:simple_workflow) - |> Lightning.Repo.preload(:triggers) - |> with_snapshot() - - trigger = - trigger - |> Ecto.Changeset.change(webhook_reply: :after_completion) - |> Repo.update!() - - Lightning.WorkOrders.Events.subscribe(project_id) - - test_pid = self() - - task = - Task.async(fn -> - conn = post(conn, "/i/#{trigger.id}", %{"foo" => "bar"}) - send(test_pid, {:response, conn}) - end) - - assert_receive %Lightning.WorkOrders.Events.WorkOrderCreated{ - work_order: work_order - } - - assert_receive %Lightning.WorkOrders.Events.RunCreated{run: run} + methods = + for type <- unquote(types), + do: insert(:webhook_auth_method, auth_type: type, api_key: "key") - body = %{"message" => "run failed"} + associate_auth_methods(trigger, methods) - Phoenix.PubSub.broadcast( - Lightning.PubSub, - "work_order:#{work_order.id}:webhook_response", - {:webhook_response, 422, body} - ) + conn = post(conn, "/i/#{trigger.id}", %{"foo" => "bar"}) - assert_receive {:response, response_conn}, 5_000 - - assert json_response(response_conn, 422) == body - - assert get_resp_header(response_conn, "x-meta-work-order-id") == [ - work_order.id - ] - - assert get_resp_header(response_conn, "x-meta-run-id") == [run.id] - - Task.await(task) + assert_unauthenticated(conn, :unauthenticated, unquote(challenges)) + end end + end - test "returns timeout if workflow doesn't complete within timeout period", %{ - conn: conn - } do - expect(Lightning.MockConfig, :webhook_response_timeout_ms, fn -> 2_000 end) + describe "a limiter's refusal" do + setup [:stub_rate_limiter_ok, :stub_usage_limiter_ok] + setup do %{triggers: [trigger]} = - insert(:simple_workflow) - |> Lightning.Repo.preload(:triggers) - |> with_snapshot() + insert(:simple_workflow) |> Repo.preload(:triggers) |> with_snapshot() - trigger = - trigger - |> Ecto.Changeset.change(webhook_reply: :after_completion) - |> Repo.update!() - - conn = post(conn, "/i/#{trigger.id}", %{"foo" => "bar"}) - - [work_order_id] = get_resp_header(conn, "x-meta-work-order-id") - - assert json_response(conn, 504) == %{ - "error" => "timeout", - "message" => "Workflow did not complete within timeout period", - "work_order_id" => work_order_id - } - - assert WorkOrders.get(work_order_id) + %{trigger: trigger} end - for status_code <- [204, 304] do - test "sends empty body when broadcast status is #{status_code}", %{ - conn: conn - } do - %{triggers: [trigger], project_id: project_id} = - insert(:simple_workflow) - |> Lightning.Repo.preload(:triggers) - |> with_snapshot() - - trigger = - trigger - |> Ecto.Changeset.change(webhook_reply: :after_completion) - |> Repo.update!() - - Lightning.WorkOrders.Events.subscribe(project_id) - - test_pid = self() - - task = - Task.async(fn -> - conn = post(conn, "/i/#{trigger.id}", %{"foo" => "bar"}) - send(test_pid, {:response, conn}) - end) - - assert_receive %Lightning.WorkOrders.Events.WorkOrderCreated{ - work_order: work_order - } - - assert_receive %Lightning.WorkOrders.Events.RunCreated{run: _run} - - Phoenix.PubSub.broadcast( - Lightning.PubSub, - "work_order:#{work_order.id}:webhook_response", - {:webhook_response, unquote(status_code), nil} - ) - - assert_receive {:response, response_conn}, 5_000 + test "gets the usage limiter's 429, with the code by reason", + %{conn: conn, trigger: trigger} do + for reason <- [ + :runs_hard_limit, + :exceeds_limit, + :unknown_project, + :runs_cap_exceeded + ] do + Mox.stub(MockUsageLimiter, :limit_action, fn _action, _context -> + {:error, reason, %Lightning.Extensions.Message{text: "upgrade"}} + end) - assert response_conn.status == unquote(status_code) - assert response_conn.resp_body == "" + conn = post(conn, "/i/#{trigger.id}") - Task.await(task) + body = assert_problem(conn, 429, :exceeds_limit, []) + refute Map.has_key?(body, "message") end end - test "replies with the rejection instead of waiting when no run was created", - %{conn: conn} do - Mox.stub(MockUsageLimiter, :limit_action, &StubUsageLimiter.limit_action/2) - - Mox.stub(Lightning.MockConfig, :webhook_response_timeout_ms, fn -> - 60_000 - end) - - %{triggers: [trigger]} = - insert(:simple_workflow) - |> Lightning.Repo.preload(:triggers) - |> with_snapshot() - - trigger - |> Ecto.Changeset.change(webhook_reply: :after_completion) - |> Repo.update!() - - {elapsed_us, conn} = - :timer.tc(fn -> post(conn, "/i/#{trigger.id}", %{"foo" => "bar"}) end) - - assert elapsed_us < 5_000_000, - "request waited on a response nothing can send" - - response = json_response(conn, 429) - - assert %{"error" => "too_many_runs", "work_order_id" => work_order_id} = - response + test "gets the rate limiter's 429, with no Retry-After", + %{conn: conn, trigger: trigger} do + Mox.stub(MockRateLimiter, :limit_request, &StubRateLimiter.limit_request/3) - refute Map.has_key?(response, "message"), - "the limiter's project-facing copy must not reach /i/*" + conn = post(conn, "/i/#{trigger.id}") - work_order = WorkOrders.get(work_order_id, include: [:runs]) - assert work_order.state == :rejected - assert work_order.runs == [] + body = assert_problem(conn, 429, :too_many_requests, []) + refute Map.has_key?(body, "message") + assert get_resp_header(conn, "retry-after") == [] end - test "returns immediately when webhook_reply is :custom, which has no publisher", - %{conn: conn} do - Mox.stub(Lightning.MockConfig, :webhook_response_timeout_ms, fn -> - 60_000 + test "gets a synchronous trigger's 429, with its work order", + %{conn: conn, trigger: trigger} do + Mox.stub(MockUsageLimiter, :limit_action, fn _action, _context -> + {:error, :too_many_runs, %Lightning.Extensions.Message{text: "wait"}} end) - %{triggers: [trigger]} = - insert(:simple_workflow) - |> Lightning.Repo.preload(:triggers) - |> with_snapshot() - trigger - |> Ecto.Changeset.change(webhook_reply: :custom) + |> Ecto.Changeset.change(webhook_reply: :after_completion) |> Repo.update!() - {elapsed_us, conn} = - :timer.tc(fn -> post(conn, "/i/#{trigger.id}", %{"foo" => "bar"}) end) + conn = post(conn, "/i/#{trigger.id}") - assert elapsed_us < 5_000_000, - "request waited on a response nothing can send" + body = assert_problem(conn, 429, :too_many_runs, []) - [work_order_id] = get_resp_header(conn, "x-meta-work-order-id") - assert %{"work_order_id" => ^work_order_id} = json_response(conn, 200) + assert [body["work_order_id"]] == + get_resp_header(conn, "x-meta-work-order-id") end - test "returns immediately when webhook_reply is before_start (default)", %{ - conn: conn - } do - %{triggers: [trigger]} = - insert(:simple_workflow) - |> Lightning.Repo.preload(:triggers) - |> with_snapshot() - - assert trigger.webhook_reply == :before_start + # The work order was saved, so this is a success, and a header that says to + # retry would make senders save it again. + test "gets the asynchronous 200 with error, byte for byte, and no Retry-After", + %{conn: conn, trigger: trigger} do + Mox.stub(MockUsageLimiter, :limit_action, fn _action, _context -> + {:error, :too_many_runs, %Lightning.Extensions.Message{text: "wait"}} + end) - conn = post(conn, "/i/#{trigger.id}", %{"foo" => "bar"}) + conn = post(conn, "/i/#{trigger.id}") [work_order_id] = get_resp_header(conn, "x-meta-work-order-id") - assert %{"work_order_id" => ^work_order_id} = json_response(conn, 200) - assert WorkOrders.get(work_order_id) - end - end - - describe "sensitive request headers" do - setup [:stub_rate_limiter_ok, :stub_usage_limiter_ok] - - test "the x-api-key consumed by webhook auth is not persisted in dataclip.request", - %{conn: conn} do - %{triggers: [trigger]} = - insert(:simple_workflow) - |> Lightning.Repo.preload(:triggers) - |> with_snapshot() - - auth_method = - insert(:webhook_auth_method, - auth_type: :api, - api_key: "sup3r-s3cret-api-key" - ) - associate_auth_methods(trigger, [auth_method]) - - conn = - conn - |> put_req_header("x-api-key", "sup3r-s3cret-api-key") - |> post("/i/#{trigger.id}", %{"foo" => "bar"}) - - assert %{"work_order_id" => work_order_id} = json_response(conn, 200) + assert conn.status == 200 - %{runs: [run]} = WorkOrders.get(work_order_id, include: [:runs]) - - request = Runs.get_dataclip_request(run) |> Jason.decode!() - - assert request["headers"]["x-api-key"] == "[REDACTED]", - "the api key Lightning already consumed to authenticate must not be retained" - - refute Runs.get_dataclip_request(run) =~ "sup3r-s3cret-api-key" - end - - test "the Basic Authorization consumed by webhook auth is redacted from dataclip.request", - %{conn: conn} do - %{triggers: [trigger]} = - insert(:simple_workflow) - |> Lightning.Repo.preload(:triggers) - |> with_snapshot() + assert conn.resp_body == + ~s({"error":"too_many_runs","work_order_id":"#{work_order_id}"}) - auth_method = - insert(:webhook_auth_method, - auth_type: :basic, - username: "caller", - password: "sup3r-s3cret-password" - ) - - associate_auth_methods(trigger, [auth_method]) - - credentials = Base.encode64("caller:sup3r-s3cret-password") - - conn = - conn - |> put_req_header("authorization", "Basic #{credentials}") - |> post("/i/#{trigger.id}", %{"foo" => "bar"}) - - assert %{"work_order_id" => work_order_id} = json_response(conn, 200) - - %{runs: [run]} = WorkOrders.get(work_order_id, include: [:runs]) - - request = Runs.get_dataclip_request(run) |> Jason.decode!() - - assert request["headers"]["authorization"] == "Basic [REDACTED]", - "the Basic credentials Lightning already consumed must not be " <> - "retained; the scheme is not a secret, so it stays legible" - - raw = Runs.get_dataclip_request(run) - refute raw =~ credentials - refute raw =~ "sup3r-s3cret-password" - end - - test "a caller's own x-api-key survives when it is not the trigger's secret", - %{conn: conn} do - %{triggers: [trigger]} = - insert(:simple_workflow) - |> Lightning.Repo.preload(:triggers) - |> with_snapshot() - - auth_method = - insert(:webhook_auth_method, - auth_type: :basic, - username: "caller", - password: "sup3r-s3cret-password" - ) - - associate_auth_methods(trigger, [auth_method]) - - credentials = Base.encode64("caller:sup3r-s3cret-password") - - conn = - conn - |> put_req_header("authorization", "Basic #{credentials}") - |> put_req_header("x-api-key", "a-token-for-the-job-to-forward") - |> post("/i/#{trigger.id}", %{"foo" => "bar"}) - - assert %{"work_order_id" => work_order_id} = json_response(conn, 200) - - %{runs: [run]} = WorkOrders.get(work_order_id, include: [:runs]) - - request = Runs.get_dataclip_request(run) |> Jason.decode!() - - assert request["headers"]["authorization"] == "Basic [REDACTED]" - - assert request["headers"]["x-api-key"] == - "a-token-for-the-job-to-forward", - "redaction is by value, not by header name: a token Lightning " <> - "never consumed belongs to the caller and their job" - end - - test "our api key is redacted from both headers webhook auth reads", - %{conn: conn} do - %{triggers: [trigger]} = - insert(:simple_workflow) - |> Lightning.Repo.preload(:triggers) - |> with_snapshot() - - auth_method = - insert(:webhook_auth_method, - auth_type: :api, - api_key: "sup3r-s3cret-api-key" - ) - - associate_auth_methods(trigger, [auth_method]) - - conn = - conn - |> put_req_header("x-api-key", "sup3r-s3cret-api-key") - |> put_req_header("authorization", "Bearer sup3r-s3cret-api-key") - |> put_req_header("x-copy-of-it", "sup3r-s3cret-api-key") - |> post("/i/#{trigger.id}", %{"foo" => "bar"}) - - assert %{"work_order_id" => work_order_id} = json_response(conn, 200) - - %{runs: [run]} = WorkOrders.get(work_order_id, include: [:runs]) - - request = Runs.get_dataclip_request(run) |> Jason.decode!() - - assert request["headers"]["x-api-key"] == "[REDACTED]" - - assert request["headers"]["authorization"] == "Bearer [REDACTED]", - "auth matched on x-api-key, but the copy in the other header we " <> - "read would otherwise be persisted intact" - - assert request["headers"]["x-copy-of-it"] == "sup3r-s3cret-api-key", - "a header webhook auth does not read is passed through verbatim, " <> - "secret-shaped or not" - end - - test "cookie and proxy-authorization are redacted from dataclip.request", - %{conn: conn} do - %{triggers: [trigger]} = - insert(:simple_workflow) - |> Lightning.Repo.preload(:triggers) - |> with_snapshot() - - conn = - conn - |> put_req_header("cookie", "session=a-third-party-session-cookie") - |> put_req_header("proxy-authorization", "Basic cHJveHk6c2VjcmV0") - |> post("/i/#{trigger.id}", %{"foo" => "bar"}) - - assert %{"work_order_id" => work_order_id} = json_response(conn, 200) - - %{runs: [run]} = WorkOrders.get(work_order_id, include: [:runs]) - - request = Runs.get_dataclip_request(run) |> Jason.decode!() - - assert request["headers"]["cookie"] == "[REDACTED]", - "caller session cookies are a load-balancer/front-end concern, not run state" - - assert request["headers"]["proxy-authorization"] == "[REDACTED]" - - raw = Runs.get_dataclip_request(run) - refute raw =~ "a-third-party-session-cookie" - refute raw =~ "cHJveHk6c2VjcmV0" - end - - test "non-sensitive caller headers are still passed through verbatim", %{ - conn: conn - } do - %{triggers: [trigger]} = - insert(:simple_workflow) - |> Lightning.Repo.preload(:triggers) - |> with_snapshot() - - conn = - conn - |> put_req_header("x-request-id", "caller-supplied-id") - |> put_req_header("user-agent", "acme-integration/1.2") - |> post("/i/#{trigger.id}", %{"foo" => "bar"}) - - assert %{"work_order_id" => work_order_id} = json_response(conn, 200) - - %{runs: [run]} = WorkOrders.get(work_order_id, include: [:runs]) + assert get_resp_header(conn, "content-type") == [ + "application/json; charset=utf-8" + ] - request = Runs.get_dataclip_request(run) |> Jason.decode!() + assert get_resp_header(conn, "retry-after") == [] - assert request["headers"]["x-request-id"] == "caller-supplied-id" - assert request["headers"]["user-agent"] == "acme-integration/1.2" + work_order = WorkOrders.get(work_order_id, include: [:runs]) + assert work_order.state == :rejected + assert work_order.runs == [] end end diff --git a/test/lightning_web/live/channel_live/form_test.exs b/test/lightning_web/live/channel_live/form_test.exs index e14c5b0da30..b0fdc67f427 100644 --- a/test/lightning_web/live/channel_live/form_test.exs +++ b/test/lightning_web/live/channel_live/form_test.exs @@ -339,6 +339,31 @@ defmodule LightningWeb.ChannelLive.FormTest do assert updated.name == "updated-name" end + @tag role: :editor + test "saving a channel another request changed flashes and returns to the list", + %{conn: conn, project: project} do + channel = insert(:channel, project: project, name: "old-name") + + {:ok, view, _html} = + live(conn, ~p"/projects/#{project.id}/channels/#{channel.id}/edit") + + Repo.query!( + "UPDATE channels SET lock_version = lock_version + 1 WHERE id = $1", + [Ecto.UUID.dump!(channel.id)] + ) + + view + |> form("#channel-form-#{channel.id}", channel: %{name: "updated-name"}) + |> render_submit() + + assert_patch(view, ~p"/projects/#{project.id}/channels") + + assert render(view) =~ + "Someone else changed or deleted this record while you were editing it. Reload and try again." + + assert Channels.get_channel!(channel.id).name == "old-name" + end + @tag role: :editor test "edit form shows available auth methods and credentials", %{ conn: conn, diff --git a/test/lightning_web/live/channel_live/index_test.exs b/test/lightning_web/live/channel_live/index_test.exs index 3947708fc0a..93e22726e1d 100644 --- a/test/lightning_web/live/channel_live/index_test.exs +++ b/test/lightning_web/live/channel_live/index_test.exs @@ -4,6 +4,7 @@ defmodule LightningWeb.ChannelLive.IndexTest do import Ecto.Query import Phoenix.LiveViewTest import Lightning.Factories + import Lightning.RaceHelpers alias Lightning.Auditing.Audit alias Lightning.Channels @@ -207,6 +208,37 @@ defmodule LightningWeb.ChannelLive.IndexTest do assert Channels.get_channel!(channel.id).enabled == true end + + @tag role: :editor + test "toggling a channel another request changed flashes and leaves it as it was", + %{conn: conn, project: project} do + channel = + insert(:channel, project: project, name: "toggle-me", enabled: true) + + {:ok, view, _html} = live(conn, ~p"/projects/#{project.id}/channels") + + after_first_query( + "channels", + channel.id, + fn id -> + Repo.query!( + "UPDATE channels SET lock_version = lock_version + 1 WHERE id = $1", + [id] + ) + end, + view.pid + ) + + html = + view + |> element("#toggle-control-#{channel.id}") + |> render_click() + + assert html =~ + "Someone else changed or deleted this record while you were editing it. Reload and try again." + + assert Channels.get_channel!(channel.id).enabled == true + end end describe "delete channel" do diff --git a/test/lightning_web/live/collection_live_test.exs b/test/lightning_web/live/collection_live_test.exs index 7a1ccbfdcb5..60cf4b762cb 100644 --- a/test/lightning_web/live/collection_live_test.exs +++ b/test/lightning_web/live/collection_live_test.exs @@ -124,6 +124,14 @@ defmodule LightningWeb.CollectionLiveTest do assert html =~ "Collection created successfully" assert html =~ "new-collection" assert html =~ project.name + + collection = + Lightning.Repo.get_by!(Lightning.Collections.Collection, + name: "new-collection" + ) + + assert %{actor_id: actor_id} = audit_for(collection, "created") + assert actor_id == user.id end test "Canceling collection creation modal closes the modal", %{conn: conn} do @@ -144,7 +152,7 @@ defmodule LightningWeb.CollectionLiveTest do refute html =~ "new-collection" end - test "Superuser can update a collection via the modal", %{ + test "Superuser can rename a collection via the modal, not move it", %{ conn: conn, user: user } do @@ -155,6 +163,11 @@ defmodule LightningWeb.CollectionLiveTest do assert has_element?(view, "#collection-form-#{collection.id}") + assert has_element?( + view, + "#collection-form-#{collection.id} select[name='collection[project_id]'][disabled]" + ) + view |> form("#collection-form-#{collection.id}", collection: %{raw_name: "Updated Collection"} @@ -166,7 +179,7 @@ defmodule LightningWeb.CollectionLiveTest do {:ok, _view, html} = view |> form("#collection-form-#{collection.id}", - collection: %{raw_name: "Updated Collection", project_id: project.id} + collection: %{raw_name: "Updated Collection"} ) |> render_submit() |> follow_redirect(conn, ~p"/settings/collections") @@ -174,6 +187,36 @@ defmodule LightningWeb.CollectionLiveTest do assert html =~ "Collection updated successfully" assert html =~ "updated-collection" assert html =~ project.name + + assert %{actor_id: actor_id} = audit_for(collection, "updated") + assert actor_id == user.id + end + + test "Saving a collection deleted since the modal opened flashes and closes it", + %{conn: conn, user: user} do + project = insert(:project, project_users: [%{user: user}]) + collection = insert(:collection, name: "old-collection", project: project) + + {:ok, view, _html} = live(conn, ~p"/settings/collections") + + view + |> form("#collection-form-#{collection.id}", + collection: %{raw_name: "Renamed"} + ) + |> render_change() + + Repo.delete!(collection) + + {:ok, _view, html} = + view + |> form("#collection-form-#{collection.id}", + collection: %{raw_name: "Renamed"} + ) + |> render_submit() + |> follow_redirect(conn, ~p"/settings/collections") + + assert html =~ + "Someone else changed or deleted this record while you were editing it. Reload and try again." end test "Creating a collection with a name that already exists fails", %{ @@ -200,6 +243,13 @@ defmodule LightningWeb.CollectionLiveTest do end end + defp audit_for(collection, event) do + Lightning.Repo.get_by!(Lightning.Collections.Audit.base_query(), + item_id: collection.id, + event: event + ) + end + defp get_sorted_collection_names(view) do html = render(view) diff --git a/test/lightning_web/live/credential_live_test.exs b/test/lightning_web/live/credential_live_test.exs index 90a5ed97253..a7e0837ba0f 100644 --- a/test/lightning_web/live/credential_live_test.exs +++ b/test/lightning_web/live/credential_live_test.exs @@ -538,6 +538,51 @@ defmodule LightningWeb.CredentialLiveTest do assert Lightning.Repo.reload(credential).scheduled_deletion end + test "the owner of a scheduled credential is offered only cancelling or deleting it now", + %{conn: conn, credential: credential} do + Repo.update!( + Changeset.change(credential, + scheduled_deletion: DateTime.utc_now() |> DateTime.truncate(:second) + ) + ) + + {:ok, view, _html} = live(conn, ~p"/credentials", on_error: :raise) + + assert Enum.map( + ~w(cancel-deletion delete-now delete edit transfer), + &has_element?(view, "#credential-actions-#{credential.id}-#{&1}") + ) == [true, true, false, false, false] + end + + test "a project member is offered no actions on someone else's scheduled credential", + %{conn: conn, user: user} do + credential_owner = insert(:user) + + project = + insert(:project, + project_users: [ + %{user: user, role: :owner}, + %{user: credential_owner, role: :admin} + ] + ) + + credential = + insert(:credential, + user: credential_owner, + scheduled_deletion: DateTime.utc_now() |> DateTime.truncate(:second), + project_credentials: [%{project: project}] + ) + + {:ok, view, html} = + live(conn, ~p"/projects/#{project}/settings#credentials", + on_error: :raise + ) + + assert html =~ credential.name + + refute has_element?(view, "#credential-actions-#{credential.id}-dropdown") + end + test "delete credentials in project settings page", %{ conn: conn, user: user @@ -585,6 +630,49 @@ defmodule LightningWeb.CredentialLiveTest do assert credential.scheduled_deletion end + # Scheduled behind the page's back, so the modal still offers scheduling. + test "a stale page cannot schedule a credential a second time", %{ + conn: conn, + user: user + } do + project = insert(:project, project_users: [%{user: user, role: :owner}]) + + credential = + insert(:credential, + user: user, + project_credentials: [%{project: project}] + ) + + {:ok, view, _html} = + live(conn, ~p"/projects/#{project}/settings#credentials", + on_error: :raise + ) + + open_delete_credential_modal(view, credential.id) + + scheduled_at = + DateTime.utc_now() |> DateTime.add(7, :day) |> DateTime.truncate(:second) + + Repo.update!( + Changeset.change(credential, scheduled_deletion: scheduled_at) + ) + + view + |> element( + "#delete-credential-#{credential.id}-modal button", + "Delete credential" + ) + |> render_click() + + flash = + assert_redirected(view, ~p"/projects/#{project}/settings#credentials") + + assert flash["error"] == + "This credential is already scheduled for deletion" + + assert Repo.reload!(credential).scheduled_deletion == scheduled_at + end + test "doesn't show delete credential to support user", %{ conn: conn, user: user @@ -1178,6 +1266,34 @@ defmodule LightningWeb.CredentialLiveTest do assert html =~ "some updated name" end + test "saving only a new body records one updated event", %{ + conn: conn, + user: user + } do + credential = + insert(:credential, user: user, name: "raw one", schema: "raw") + |> with_body(%{name: "main", body: %{"a" => "old"}}) + + {:ok, index_live, _html} = live(conn, ~p"/credentials", on_error: :raise) + + open_edit_credential_modal(index_live, credential.id) + + index_live + |> click_save( + %{credential: %{name: "raw one", body: ~s({"a":"new"})}}, + "#credential-form-#{credential.id}" + ) + + assert_redirected(index_live, ~p"/credentials") + + assert [%{event: "updated", metadata: %{"environments" => ["main"]}}] = + Repo.all( + from(a in Lightning.Credentials.Audit.base_query(), + where: a.item_id == ^credential.id + ) + ) + end + test "displays external_id in credentials table", %{ conn: conn, user: user, @@ -1196,6 +1312,51 @@ defmodule LightningWeb.CredentialLiveTest do assert html =~ "display-test-id" end + test "saving a credential deleted since the form loaded flashes and returns to the list", + %{conn: conn, credential: credential} do + {:ok, view, _html} = live(conn, ~p"/credentials", on_error: :raise) + + open_edit_credential_modal(view, credential.id) + + Repo.delete!(credential) + + view + |> form("#credential-form-#{credential.id}", + credential: %{name: "renamed"} + ) + |> render_submit() + + flash = assert_redirected(view, ~p"/credentials") + + assert flash["error"] == + "Someone else changed or deleted this record while you were editing it. Reload and try again." + end + + # Scheduled behind the page's back, so the form still offers editing. + test "a stale page cannot edit a credential scheduled for deletion", %{ + conn: conn, + credential: credential + } do + {:ok, view, _html} = live(conn, ~p"/credentials", on_error: :raise) + + open_edit_credential_modal(view, credential.id) + + Repo.update!( + Changeset.change(credential, + scheduled_deletion: DateTime.utc_now() |> DateTime.truncate(:second) + ) + ) + + view |> form("#credential-form-#{credential.id}") |> render_submit() + + flash = assert_redirected(view, ~p"/credentials") + + assert flash["error"] == + "This credential is scheduled for deletion. Cancel the deletion to edit it." + + assert Repo.reload!(credential).name == credential.name + end + test "Edit adds new project with access", %{ conn: conn, user: user @@ -3070,6 +3231,55 @@ defmodule LightningWeb.CredentialLiveTest do end) end + test "saving an oauth credential unchanged records nothing", %{ + conn: conn, + user: user + } do + oauth_client = insert(:oauth_client, user: user, userinfo_endpoint: nil) + + credential = + insert(:credential, + name: "OAuth credential", + oauth_client: oauth_client, + user: user, + schema: "oauth" + ) + |> with_body(%{ + name: "main", + body: %{ + "access_token" => "test_access_token", + "refresh_token" => "test_refresh_token", + "token_type" => "Bearer", + "expires_at" => DateTime.to_unix(DateTime.utc_now()) + 3600, + "scope" => + String.split(oauth_client.mandatory_scopes, ",") |> Enum.join(" ") + } + }) + + {:ok, view, _html} = live(conn, ~p"/credentials", on_error: :raise) + + open_edit_credential_modal(view, credential.id) + + Lightning.ApplicationHelpers.dynamically_absorb_delay(fn -> + {_, assigns} = + Lightning.LiveViewHelpers.get_component_assigns_by(view, + id: "generic-oauth-component-#{credential.id}-main" + ) + + assigns[:oauth_progress] in [:complete, :idle] + end) + + view |> form("#credential-form-#{credential.id}") |> render_submit() + + assert_redirected(view, ~p"/credentials") + + assert Repo.all( + from(a in Lightning.Credentials.Audit.base_query(), + where: a.item_id == ^credential.id + ) + ) == [] + end + test "reauthenticate banner is not rendered the first time we pick permissions", %{ conn: conn, diff --git a/test/lightning_web/live/project_live_test.exs b/test/lightning_web/live/project_live_test.exs index a2a013e921f..e9af3349780 100644 --- a/test/lightning_web/live/project_live_test.exs +++ b/test/lightning_web/live/project_live_test.exs @@ -7,6 +7,7 @@ defmodule LightningWeb.ProjectLiveTest do import Lightning.AccountsFixtures import Lightning.AdaptorTestHelpers import Lightning.Factories + import Lightning.RaceHelpers import LightningWeb.CredentialLiveHelpers import Lightning.ApplicationHelpers, @@ -153,16 +154,14 @@ defmodule LightningWeb.ProjectLiveTest do assert render(index_live) =~ "Project created successfully" project_name = String.replace(@create_attrs.raw_name, " ", "-") + subject = "You now have access to \"#{project_name}\"" - assert_email_sent( - to: [Swoosh.Email.Recipient.format(user_1)], - subject: "You now have access to \"#{project_name}\"" - ) + for user <- [user_1, user_2] do + recipient = Swoosh.Email.Recipient.format(user) - assert_email_sent( - to: [Swoosh.Email.Recipient.format(user_2)], - subject: "You now have access to \"#{project_name}\"" - ) + assert_received {:email, + %Swoosh.Email{to: [^recipient], subject: ^subject}} + end end test "project owners can delete a project from the settings page", @@ -1854,6 +1853,27 @@ defmodule LightningWeb.ProjectLiveTest do assert %{name: "dep-burundi-datafi"} = Repo.get!(Project, project.id) end + test "saving settings for a project deleted mid-save flashes and keeps the form", + %{conn: conn, user: user} do + project = + insert(:project, + name: "project-1", + project_users: [%{user_id: user.id, role: :admin}] + ) + + {:ok, view, _html} = + live(conn, ~p"/projects/#{project}/settings", on_error: :raise) + + delete_project_mid_save(project.id, view.pid) + + assert view + |> form("#project-settings-form", project: %{raw_name: "renamed"}) + |> render_submit() =~ + "Someone else changed or deleted this record while you were editing it. Reload and try again." + + assert has_element?(view, "#project-settings-form") + end + test "saving project settings with name that resolves to blank shows error", %{ conn: conn, @@ -2277,6 +2297,34 @@ defmodule LightningWeb.ProjectLiveTest do |> render_click() =~ "Project MFA requirement updated successfully" end + test "a refused MFA or support access toggle says so instead of crashing", + %{conn: conn} do + project = insert(:project) + + project_user = + insert(:project_user, role: :owner, project: project, user: build(:user)) + + stub(Lightning.Extensions.MockUsageLimiter, :limit_action, fn _, _ -> + :ok + end) + + stub(Lightning.Extensions.MockProjectHook, :handle_project_validation, fn + changeset -> Ecto.Changeset.add_error(changeset, :name, "is refused") + end) + + conn = log_in_user(conn, project_user.user) + {:ok, view, _html} = live(conn, ~p"/projects/#{project}/settings#security") + + assert render_click(view, "toggle-mfa", %{}) =~ + "Changes couldn't be saved, please try again" + + assert render_click(view, "toggle_support_access", %{}) =~ + "Changes couldn't be saved, please try again" + + assert %{requires_mfa: false, allow_support_access: false} = + Lightning.Repo.reload!(project) + end + test "MFA limited: banner displayed, button is disabled, and MFA cannot be toggled", %{conn: conn} do project = insert(:project) @@ -7570,7 +7618,7 @@ defmodule LightningWeb.ProjectLiveTest do } do project = insert(:project) - for {{conn, _user}, index} <- + for {{conn, user}, index} <- setup_project_users(conn, project, [:owner, :admin]) |> Enum.with_index() do {:ok, view, _html} = @@ -7616,9 +7664,16 @@ defmodule LightningWeb.ProjectLiveTest do assert flash["info"] == "Collection created" # collection now exists - assert Lightning.Repo.get_by(Lightning.Collections.Collection, - project_id: project.id, - name: expected_collection_name + assert collection = + Lightning.Repo.get_by(Lightning.Collections.Collection, + project_id: project.id, + name: expected_collection_name + ) + + assert Lightning.Repo.get_by(Lightning.Collections.Audit.base_query(), + item_id: collection.id, + event: "created", + actor_id: user.id ) end end @@ -7754,7 +7809,7 @@ defmodule LightningWeb.ProjectLiveTest do } do project = insert(:project) - for {conn, _user} <- + for {conn, user} <- setup_project_users(conn, project, [:owner, :admin]) do collection = insert(:collection, project: project) @@ -7807,9 +7862,47 @@ defmodule LightningWeb.ProjectLiveTest do project_id: project.id, name: new_collection_name ) + + assert Lightning.Repo.get_by(Lightning.Collections.Audit.base_query(), + item_id: collection.id, + event: "updated", + actor_id: user.id + ) end end + test "saving a collection deleted since the modal opened flashes and closes it", + %{conn: conn} do + project = insert(:project) + [{conn, _user}] = setup_project_users(conn, project, [:admin]) + collection = insert(:collection, project: project) + + {:ok, view, _html} = + live(conn, ~p"/projects/#{project.id}/settings#collections") + + view + |> element("#edit-collection-#{collection.id}-button") + |> render_click() + + view + |> form("#edit-collection-#{collection.id}-modal form", + collection: %{raw_name: "renamed"} + ) + |> render_change() + + Repo.delete!(collection) + + view + |> form("#edit-collection-#{collection.id}-modal form") + |> render_submit() + + flash = + assert_redirected(view, ~p"/projects/#{project.id}/settings#collections") + + assert flash["error"] == + "Someone else changed or deleted this record while you were editing it. Reload and try again." + end + test "unauthorized users cannot delete a collection", %{ conn: conn } do diff --git a/test/lightning_web/live/sandbox_live/form_component_test.exs b/test/lightning_web/live/sandbox_live/form_component_test.exs index 58f02304107..7d68d93772b 100644 --- a/test/lightning_web/live/sandbox_live/form_component_test.exs +++ b/test/lightning_web/live/sandbox_live/form_component_test.exs @@ -3,7 +3,9 @@ defmodule LightningWeb.SandboxLive.FormComponentTest do use Mimic import Phoenix.LiveViewTest + import Ecto.Query import Lightning.Factories + import Lightning.RaceHelpers setup_all do Mimic.copy(Lightning.Projects) @@ -431,6 +433,60 @@ defmodule LightningWeb.SandboxLive.FormComponentTest do end end + describe "edit modal when the save is refused" do + setup %{user: user} do + parent = insert(:project, project_users: [%{user: user, role: :editor}]) + + sb = + insert(:sandbox, + parent: parent, + name: "sb-1", + project_users: [%{user: user, role: :owner}] + ) + + %{parent: parent, sb: sb} + end + + test "saving a sandbox deleted since the form loaded flashes and closes it", + %{conn: conn, parent: parent, sb: sb} do + {:ok, view, _} = + live(conn, ~p"/projects/#{parent.id}/sandboxes/#{sb.id}/edit") + + delete_project_mid_save(sb.id, view.pid) + + view + |> element("#sandbox-form-#{sb.id}") + |> render_submit(%{"project" => %{"raw_name" => "sb-2"}}) + + flash = assert_redirected(view, ~p"/projects/#{parent.id}/sandboxes") + + assert flash["error"] == + "Someone else changed or deleted this record while you were editing it. Reload and try again." + end + + test "saving after losing the right to edit the sandbox flashes and closes it", + %{conn: conn, parent: parent, sb: sb, user: user} do + {:ok, view, _} = + live(conn, ~p"/projects/#{parent.id}/sandboxes/#{sb.id}/edit") + + Repo.update_all( + from(pu in Lightning.Projects.ProjectUser, + where: pu.project_id == ^sb.id and pu.user_id == ^user.id + ), + set: [role: :editor] + ) + + view + |> element("#sandbox-form-#{sb.id}") + |> render_submit(%{"project" => %{"raw_name" => "sb-2"}}) + + flash = assert_redirected(view, ~p"/projects/#{parent.id}/sandboxes") + + assert flash["error"] == + "You are not authorized to perform this action." + end + end + defp assert_redirect_or_patch(res_or_html, view, conn, to) do case res_or_html do {:error, {:redirect, _}} -> diff --git a/test/lightning_web/live/user_live_test.exs b/test/lightning_web/live/user_live_test.exs index 2a6145ba0f3..198d37c1d5b 100644 --- a/test/lightning_web/live/user_live_test.exs +++ b/test/lightning_web/live/user_live_test.exs @@ -134,6 +134,25 @@ defmodule LightningWeb.UserLiveTest do assert Repo.reload!(user) |> user_attrs_match?(@update_attrs) end + test "saving a user deleted since the form loaded flashes and returns to the list", + %{conn: conn} do + user = user_fixture() + + {:ok, form_live, _html} = + live(conn, Routes.user_edit_path(conn, :edit, user)) + + Repo.delete!(user) + + {:ok, _index_live, html} = + form_live + |> form("#user-form", user: %{first_name: "Renamed"}) + |> render_submit() + |> follow_redirect(conn, Routes.user_index_path(conn, :index)) + + assert html =~ + "Someone else changed or deleted this record while you were editing it. Reload and try again." + end + test "records the superuser behind a create, a change and a disable, but not a save that changes nothing", %{conn: conn, user: superuser} do {:ok, new_live, _html} = live(conn, Routes.user_edit_path(conn, :new)) diff --git a/test/lightning_web/plugs/access_token_auth_test.exs b/test/lightning_web/plugs/access_token_auth_test.exs index 5043976c93b..74d09ddddc3 100644 --- a/test/lightning_web/plugs/access_token_auth_test.exs +++ b/test/lightning_web/plugs/access_token_auth_test.exs @@ -1,7 +1,7 @@ defmodule LightningWeb.Plugs.AccessTokenAuthTest do use LightningWeb.ConnCase, async: true - import Lightning.Factories + import LightningWeb.API.ProblemHelpers import Lightning.ServiceAccountHelpers alias Lightning.ServiceAccount.AccessToken @@ -19,149 +19,100 @@ defmodule LightningWeb.Plugs.AccessTokenAuthTest do |> AccessTokenAuth.call(AccessTokenAuth.init([])) end - defp assert_refused(conn, status, error) do - assert conn.halted - assert json_response(conn, status) == %{"error" => error} - - assert get_resp_header(conn, "www-authenticate") == [ - ~s(Bearer error="#{error}") - ] + defp get_users(conn, token) do + conn + |> put_req_header("accept", "application/json") + |> put_req_header("authorization", "Bearer " <> token) + |> get(~p"/api/users") end - describe "call/2" do - test "lets an access token through with its claims and service account", - %{conn: conn, account: account} do - conn = authenticate(conn, AccessToken.issue(account, ["users:read"])) - - refute conn.halted - - assert %{"sub" => sub, "scope" => "users:read"} = - conn.assigns.access_token - - assert sub == account.id - assert conn.assigns.service_account == account - end - - test "refuses a token for a service account that is no longer registered", - %{conn: conn, account: account} do - token = AccessToken.issue(account, ["users:read"]) - {replacement, _private_key} = service_account_with_key() - Mox.stub(Lightning.MockConfig, :service_account, fn -> replacement end) - - conn |> authenticate(token) |> assert_refused(401, "invalid_token") + defp assert_refused(conn) do + assert_unauthenticated(conn, :invalid_token, [ + ~s(Bearer error="invalid_token") + ]) + end - Mox.stub(Lightning.MockConfig, :service_account, fn -> nil end) + defp signed_access_token(claims, typ \\ "at+jwt", key \\ nil) do + sign_assertion( + key || Lightning.Config.token_signer().jwk, + claims, + %{"alg" => "RS256", "typ" => typ} + ) + end - conn |> authenticate(token) |> assert_refused(401, "invalid_token") - end + defp claims(account, overrides \\ %{}) do + Map.merge( + %{ + "iss" => AccessToken.issuer(), + "aud" => AccessToken.issuer() <> "/api", + "sub" => account.id, + "scope" => "users:read", + "exp" => System.system_time(:second) + 300 + }, + overrides + ) + end - test "refuses a missing token", %{conn: conn} do - conn |> AccessTokenAuth.call([]) |> assert_refused(401, "invalid_token") - end + test "lets an access token through with its claims and service account", + %{conn: conn, account: account} do + conn = + authenticate( + conn, + AccessToken.issue(account, ["users:read", "projects:write"]) + ) - test "refuses a personal access token", %{conn: conn} do - token = insert(:user) |> Lightning.Accounts.generate_api_token() + refute conn.halted - conn |> authenticate(token) |> assert_refused(401, "invalid_token") - end + assert %{"sub" => sub, "scope" => "users:read projects:write"} = + conn.assigns.access_token - test "refuses a run token", %{conn: conn} do - token = - Lightning.Workers.generate_run_token(%Lightning.Run{ - id: Ecto.UUID.generate() - }) + assert sub == account.id - conn |> authenticate(token) |> assert_refused(401, "invalid_token") - end - - test "refuses an expired access token", %{conn: conn, account: account} do - signer = Lightning.Config.token_signer() - now = System.system_time(:second) - - {_, token} = - signer.jwk - |> JOSE.JWT.sign(%{"alg" => "RS256", "typ" => "at+jwt"}, %{ - "iss" => AccessToken.issuer(), - "aud" => AccessToken.issuer() <> "/api", - "sub" => account.id, - "scope" => "users:read", - "exp" => now - 1 - }) - |> JOSE.JWS.compact() - - conn |> authenticate(token) |> assert_refused(401, "invalid_token") - end + assert conn.assigns.current_resource == %{ + account + | grants: MapSet.new(["users:read", "projects:write"]) + } + end - test "refuses a token that is right in every way but its typ", %{ + describe "a token only a forger could send, on GET /api/users" do + test "with no scope claim logs in holding no grants", %{ conn: conn, account: account } do - {_, token} = - Lightning.Config.token_signer().jwk - |> JOSE.JWT.sign(%{"alg" => "RS256", "typ" => "JWT"}, %{ - "iss" => AccessToken.issuer(), - "aud" => AccessToken.issuer() <> "/api", - "sub" => account.id, - "scope" => "users:read", - "exp" => System.system_time(:second) + 300 - }) - |> JOSE.JWS.compact() - - conn |> authenticate(token) |> assert_refused(401, "invalid_token") + token = signed_access_token(Map.delete(claims(account), "scope")) + + assert_refused_for_scope(get_users(conn, token), "users:read") end - test "refuses an at+jwt signed with another key", %{ + test "for a service account that is no longer registered is refused", %{ conn: conn, account: account } do - {_, other_key} = service_account_with_key() + token = AccessToken.issue(account, ["users:read"]) + {replacement, _private_key} = service_account_with_key() + Mox.stub(Lightning.MockConfig, :service_account, fn -> replacement end) - token = - sign_assertion( - other_key, - %{ - "iss" => AccessToken.issuer(), - "aud" => AccessToken.issuer() <> "/api", - "sub" => account.id, - "scope" => "users:read", - "exp" => System.system_time(:second) + 300 - }, - %{"alg" => "RS256", "typ" => "at+jwt"} - ) - - conn |> authenticate(token) |> assert_refused(401, "invalid_token") - end - end + conn |> get_users(token) |> assert_refused() - describe "require_scope/2" do - test "lets a token with the scope through", %{conn: conn, account: account} do - conn = - conn - |> authenticate(AccessToken.issue(account, ["users:read"])) - |> AccessTokenAuth.require_scope("users:read") + Mox.stub(Lightning.MockConfig, :service_account, fn -> nil end) - refute conn.halted + conn |> get_users(token) |> assert_refused() end - test "refuses a token without the scope", %{conn: conn, account: account} do - conn - |> authenticate(AccessToken.issue(account, ["users:read"])) - |> AccessTokenAuth.require_scope("users:write") - |> assert_refused(403, "insufficient_scope") - end - end + test "that is right in every way but its typ is refused", %{ + conn: conn, + account: account + } do + token = signed_access_token(claims(account), "JWT") - test "the existing /api routes refuse an access token", %{ - conn: conn, - account: account - } do - token = AccessToken.issue(account, AccessToken.scopes()) + conn |> get_users(token) |> assert_refused() + end - conn = - conn - |> put_req_header("authorization", "Bearer " <> token) - |> get(~p"/api/projects") + test "signed with another key is refused", %{conn: conn, account: account} do + {_, other_key} = service_account_with_key() + token = signed_access_token(claims(account), "at+jwt", other_key) - assert json_response(conn, 401) + conn |> get_users(token) |> assert_refused() + end end end diff --git a/test/lightning_web/plugs/channel_proxy_auth_test.exs b/test/lightning_web/plugs/channel_proxy_auth_test.exs new file mode 100644 index 00000000000..ef4e93b85b8 --- /dev/null +++ b/test/lightning_web/plugs/channel_proxy_auth_test.exs @@ -0,0 +1,730 @@ +defmodule LightningWeb.ChannelProxyAuthTest do + use LightningWeb.ConnCase, async: true + + alias Lightning.Channels.{ChannelRequest, ChannelEvent} + + import Ecto.Query + import Lightning.Factories + import LightningWeb.API.ProblemHelpers + import LightningWeb.API.ChannelProxyHelpers + import Plug.Test, only: [conn: 2] + + setup do + bypass = Bypass.open() + project = insert(:project) + + channel = + insert(:channel, + project: project, + destination_url: "http://localhost:#{bypass.port}", + enabled: true + ) + + disabled_channel = + insert(:channel, + project: project, + destination_url: "http://localhost:#{bypass.port}", + enabled: false + ) + + {:ok, bypass: bypass, channel: channel, disabled_channel: disabled_channel} + end + + describe "client authentication" do + test "no auth methods configured — request passes through", %{ + conn: conn, + bypass: bypass, + channel: channel + } do + # Channel has no channel_auth_methods, so it's publicly accessible + Bypass.expect_once(bypass, "GET", "/open", fn conn -> + Plug.Conn.send_resp(conn, 200, "public") + end) + + resp = get(conn, "/channels/#{channel.id}/open") + + assert resp.status == 200 + assert resp.resp_body == "public" + end + + test "API key auth — correct key allows request", %{ + bypass: bypass + } do + channel = + create_client_auth_channel(bypass, [ + %{auth_type: :api, api_key: "valid-api-key"} + ]) + + Bypass.expect_once(bypass, "GET", "/protected", fn conn -> + Plug.Conn.send_resp(conn, 200, "authenticated") + end) + + resp = + conn(:get, "/channels/#{channel.id}/protected") + |> put_req_header("x-api-key", "valid-api-key") + |> send_to_endpoint() + + assert resp.status == 200 + assert resp.resp_body == "authenticated" + end + + test "API key auth — wrong key returns 401", %{bypass: bypass} do + channel = + create_client_auth_channel(bypass, [ + %{auth_type: :api, api_key: "correct-key"} + ]) + + resp = + conn(:get, "/channels/#{channel.id}/protected") + |> put_req_header("x-api-key", "wrong-key") + |> send_to_endpoint() + + assert_unauthenticated(resp, :unauthenticated, [ + ~s(ApiKey header="x-api-key") + ]) + end + + test "API key auth — no key sent returns 401", %{bypass: bypass} do + channel = + create_client_auth_channel(bypass, [ + %{auth_type: :api, api_key: "some-key"} + ]) + + resp = + conn(:get, "/channels/#{channel.id}/protected") + |> send_to_endpoint() + + assert_unauthenticated(resp, :unauthenticated, [ + ~s(ApiKey header="x-api-key") + ]) + end + + test "Basic auth — correct credentials allows request", %{bypass: bypass} do + channel = + create_client_auth_channel(bypass, [ + %{auth_type: :basic, username: "admin", password: "secret"} + ]) + + Bypass.expect_once(bypass, "GET", "/basic", fn conn -> + Plug.Conn.send_resp(conn, 200, "basic-ok") + end) + + encoded = Base.encode64("admin:secret") + + resp = + conn(:get, "/channels/#{channel.id}/basic") + |> put_req_header("authorization", "Basic #{encoded}") + |> send_to_endpoint() + + assert resp.status == 200 + assert resp.resp_body == "basic-ok" + end + + test "Basic auth — wrong credentials returns 401", %{bypass: bypass} do + channel = + create_client_auth_channel(bypass, [ + %{auth_type: :basic, username: "admin", password: "secret"} + ]) + + encoded = Base.encode64("admin:wrong") + + resp = + conn(:get, "/channels/#{channel.id}/basic") + |> put_req_header("authorization", "Basic #{encoded}") + |> send_to_endpoint() + + assert_unauthenticated(resp, :unauthenticated, []) + end + + test "Basic auth — no auth header returns 401 with no Basic challenge, so no browser prompts for a login it could send to other channels", + %{bypass: bypass} do + channel = + create_client_auth_channel(bypass, [ + %{auth_type: :basic, username: "admin", password: "secret"} + ]) + + resp = + conn(:get, "/channels/#{channel.id}") + |> send_to_endpoint() + + assert_unauthenticated(resp, :unauthenticated, []) + end + + test "multiple auth methods — either matches allows request", %{ + bypass: bypass + } do + channel = + create_client_auth_channel(bypass, [ + %{auth_type: :api, api_key: "key-one"}, + %{auth_type: :api, api_key: "key-two"} + ]) + + Bypass.expect_once(bypass, "GET", "/multi", fn conn -> + Plug.Conn.send_resp(conn, 200, "multi-ok") + end) + + # Use the second key — should still match + resp = + conn(:get, "/channels/#{channel.id}/multi") + |> put_req_header("x-api-key", "key-two") + |> send_to_endpoint() + + assert resp.status == 200 + assert resp.resp_body == "multi-ok" + end + + test "mixed types (API + Basic) — API key matches allows request", %{ + bypass: bypass + } do + channel = + create_client_auth_channel(bypass, [ + %{auth_type: :api, api_key: "mixed-key"}, + %{auth_type: :basic, username: "user", password: "pass"} + ]) + + Bypass.expect_once(bypass, "GET", "/mixed", fn conn -> + Plug.Conn.send_resp(conn, 200, "mixed-ok") + end) + + resp = + conn(:get, "/channels/#{channel.id}/mixed") + |> put_req_header("x-api-key", "mixed-key") + |> send_to_endpoint() + + assert resp.status == 200 + assert resp.resp_body == "mixed-ok" + end + + test "mixed types (API + Basic) — no credentials challenges only the API key", + %{bypass: bypass} do + channel = + create_client_auth_channel(bypass, [ + %{auth_type: :api, api_key: "mixed-key"}, + %{auth_type: :basic, username: "user", password: "pass"} + ]) + + resp = + conn(:get, "/channels/#{channel.id}/mixed") + |> send_to_endpoint() + + assert_unauthenticated(resp, :unauthenticated, [ + ~s(ApiKey header="x-api-key") + ]) + end + + test "revoked API key (scheduled for deletion) returns 401", + %{bypass: bypass} do + channel = + create_client_auth_channel(bypass, [ + %{auth_type: :api, api_key: "live-key"}, + %{auth_type: :api, api_key: "revoked-key"} + ]) + + revoke_client_auth_method(channel, "revoked-key") + + # A stub, not an expectation: if the revoked credential is accepted the + # request reaches the destination and this responds 200. + Bypass.stub(bypass, "GET", "/protected", fn conn -> + Plug.Conn.send_resp(conn, 200, "proxied-with-revoked-credential") + end) + + resp = + conn(:get, "/channels/#{channel.id}/protected") + |> put_req_header("x-api-key", "revoked-key") + |> send_to_endpoint() + + assert_unauthenticated(resp, :unauthenticated, [ + ~s(ApiKey header="x-api-key") + ]) + end + + test "revoked basic credentials (scheduled for deletion) return 401", + %{bypass: bypass} do + channel = + create_client_auth_channel(bypass, [ + %{auth_type: :api, api_key: "live-key"}, + %{auth_type: :basic, username: "partner", password: "revoked-pass"} + ]) + + revoke_client_auth_method(channel, fn wam -> wam.auth_type == :basic end) + + # A stub, not an expectation: if the revoked credential is accepted the + # request reaches the destination and this responds 200. + Bypass.stub(bypass, "GET", "/protected", fn conn -> + Plug.Conn.send_resp(conn, 200, "proxied-with-revoked-credential") + end) + + encoded = Base.encode64("partner:revoked-pass") + + resp = + conn(:get, "/channels/#{channel.id}/protected") + |> put_req_header("authorization", "Basic #{encoded}") + |> send_to_endpoint() + + assert_unauthenticated(resp, :unauthenticated, [ + ~s(ApiKey header="x-api-key") + ]) + end + + test "the remaining live key still authenticates after another is revoked", + %{bypass: bypass} do + channel = + create_client_auth_channel(bypass, [ + %{auth_type: :api, api_key: "live-key"}, + %{auth_type: :api, api_key: "revoked-key"} + ]) + + revoke_client_auth_method(channel, "revoked-key") + + Bypass.expect_once(bypass, "GET", "/protected", fn conn -> + Plug.Conn.send_resp(conn, 200, "authenticated") + end) + + resp = + conn(:get, "/channels/#{channel.id}/protected") + |> put_req_header("x-api-key", "live-key") + |> send_to_endpoint() + + assert resp.status == 200 + assert resp.resp_body == "authenticated" + end + + # Severing the join rows at delete time revokes the credential immediately, + # but a channel left with no client method authenticates nobody and keeps + # forwarding with its destination credential. That is #387, not a + # regression introduced here; this test pins the behaviour so the gap is + # visible in the suite rather than implied. Delete it when #387 lands. + test "revoking a channel's only client method opens it to anonymous callers", + %{bypass: bypass} do + project = insert(:project) + user = insert(:user) + + credential = + insert(:credential, + schema: "@openfn/language-http", + name: "destination-cred", + user: user + ) + |> with_body(%{body: %{"access_token" => "dest-token-xyz"}}) + + project_credential = + insert(:project_credential, project: project, credential: credential) + + channel = + insert(:channel, + project: project, + destination_url: "http://localhost:#{bypass.port}", + enabled: true, + channel_auth_methods: [ + build(:channel_auth_method, + role: :client, + webhook_auth_method: + build(:webhook_auth_method, + project: project, + auth_type: :api, + api_key: "only-key" + ) + ), + build(:channel_auth_method, + role: :destination, + webhook_auth_method: nil, + project_credential: project_credential + ) + ] + ) + + revoke_client_auth_method(channel, "only-key") + + Bypass.expect_once(bypass, "GET", "/protected", fn conn -> + assert Plug.Conn.get_req_header(conn, "authorization") == [ + "Bearer dest-token-xyz" + ], + "the destination credential is still spent on an unauthenticated caller" + + Plug.Conn.send_resp(conn, 200, "anonymous") + end) + + resp = + conn(:get, "/channels/#{channel.id}/protected") + |> send_to_endpoint() + + assert resp.status == 200 + assert resp.resp_body == "anonymous" + end + + defp revoke_client_auth_method(channel, api_key) when is_binary(api_key) do + revoke_client_auth_method(channel, fn wam -> wam.api_key == api_key end) + end + + defp revoke_client_auth_method(channel, matcher) when is_function(matcher) do + auth_method = + channel.channel_auth_methods + |> Enum.map(& &1.webhook_auth_method) + |> Enum.reject(&is_nil/1) + |> Enum.find(matcher) + + {:ok, _} = + Lightning.WebhookAuthMethods.schedule_for_deletion(auth_method, + actor: insert(:user) + ) + end + end + + describe "destination authentication" do + test "Bearer token sent to upstream when channel has http credential with access_token", + %{bypass: bypass} do + channel = + create_destination_auth_channel(bypass, "@openfn/language-http", %{ + "access_token" => "tok-123" + }) + + Bypass.expect_once(bypass, "GET", "/test", fn conn -> + auth = Plug.Conn.get_req_header(conn, "authorization") + assert auth == ["Bearer tok-123"] + Plug.Conn.send_resp(conn, 200, "ok") + end) + + resp = + conn(:get, "/channels/#{channel.id}/test") + |> send_to_endpoint() + + assert resp.status == 200 + end + + test "Basic auth sent when channel has http credential with username/password", + %{bypass: bypass} do + channel = + create_destination_auth_channel(bypass, "@openfn/language-http", %{ + "username" => "u", + "password" => "p" + }) + + expected = "Basic #{Base.encode64("u:p")}" + + Bypass.expect_once(bypass, "GET", "/test", fn conn -> + auth = Plug.Conn.get_req_header(conn, "authorization") + assert auth == [expected] + Plug.Conn.send_resp(conn, 200, "ok") + end) + + resp = + conn(:get, "/channels/#{channel.id}/test") + |> send_to_endpoint() + + assert resp.status == 200 + end + + test "ApiToken sent when channel has dhis2 credential with pat", + %{bypass: bypass} do + channel = + create_destination_auth_channel(bypass, "@openfn/language-dhis2", %{ + "pat" => "d2pat_abc" + }) + + Bypass.expect_once(bypass, "GET", "/test", fn conn -> + auth = Plug.Conn.get_req_header(conn, "authorization") + assert auth == ["ApiToken d2pat_abc"] + Plug.Conn.send_resp(conn, 200, "ok") + end) + + resp = + conn(:get, "/channels/#{channel.id}/test") + |> send_to_endpoint() + + assert resp.status == 200 + end + + test "no authorization header when channel has no destination auth method", + %{bypass: bypass, channel: channel} do + Bypass.expect_once(bypass, "GET", "/test", fn conn -> + auth = Plug.Conn.get_req_header(conn, "authorization") + assert auth == [] + Plug.Conn.send_resp(conn, 200, "ok") + end) + + resp = + conn(:get, "/channels/#{channel.id}/test") + |> send_to_endpoint() + + assert resp.status == 200 + end + + test "authorization header redacted in persisted ChannelEvent", + %{bypass: bypass} do + channel = + create_destination_auth_channel(bypass, "@openfn/language-http", %{ + "access_token" => "secret-token" + }) + + Bypass.expect_once(bypass, "GET", "/redact-test", fn conn -> + Plug.Conn.send_resp(conn, 200, "ok") + end) + + resp = + conn(:get, "/channels/#{channel.id}/redact-test") + |> send_to_endpoint() + + assert resp.status == 200 + + event = + Lightning.Repo.one!( + from(e in ChannelEvent, + join: r in ChannelRequest, + on: r.id == e.channel_request_id, + where: r.channel_id == ^channel.id + ) + ) + + # The handler redacts authorization headers before persisting + # Headers are native jsonb arrays, no JSON decoding needed + auth_header = + Enum.find(event.request_headers, fn [k, _v] -> k == "authorization" end) + + assert auth_header == ["authorization", "[REDACTED]"] + end + + test "credential environment_not_found returns 502 with observable error", + %{bypass: bypass} do + # Create a credential with NO credential_body for "main" environment + project = insert(:project) + user = insert(:user) + + credential = + insert(:credential, + schema: "@openfn/language-http", + name: "no-body", + user: user + ) + + # Don't call with_body — no CredentialBody exists + + project_credential = + insert(:project_credential, + project: project, + credential: credential + ) + + channel = + insert(:channel, + project: project, + destination_url: "http://localhost:#{bypass.port}", + enabled: true, + channel_auth_methods: [ + build(:channel_auth_method, + role: :destination, + webhook_auth_method: nil, + project_credential: project_credential + ) + ] + ) + + resp = + conn(:get, "/channels/#{channel.id}/test") + |> send_to_endpoint() + + assert_problem(resp, 502, :bad_gateway, []) + + request = + Lightning.Repo.one!( + from(r in ChannelRequest, where: r.channel_id == ^channel.id) + ) + + assert request.state == :error + + event = + Lightning.Repo.one!( + from(e in ChannelEvent, where: e.channel_request_id == ^request.id) + ) + + assert event.error_message == "credential_environment_not_found" + end + + test "a sandbox authenticates with its own environment, not the parent's", + %{bypass: bypass} do + # The bug this PR fixes: the proxy asked for "main" whatever project it + # was acting for, so a sandbox reached its destination holding the + # parent's production secret. Reverting the fix has to fail here. + root = insert(:project) + sandbox = insert(:project, parent: root, env: "staging") + user = insert(:user) + + credential = + insert(:credential, + schema: "@openfn/language-http", + name: "shared", + user: user + ) + |> with_body(%{ + name: "main", + body: %{"username" => "prod", "password" => "prod-secret"} + }) + |> with_body(%{ + name: "staging", + body: %{"username" => "sbx", "password" => "sbx-secret"} + }) + + project_credential = + insert(:project_credential, project: sandbox, credential: credential) + + channel = + insert(:channel, + project: sandbox, + destination_url: "http://localhost:#{bypass.port}", + enabled: true, + channel_auth_methods: [ + build(:channel_auth_method, + role: :destination, + webhook_auth_method: nil, + project_credential: project_credential + ) + ] + ) + + sandbox_auth = "Basic #{Base.encode64("sbx:sbx-secret")}" + parent_auth = "Basic #{Base.encode64("prod:prod-secret")}" + + Bypass.expect_once(bypass, "GET", "/test", fn conn -> + auth = Plug.Conn.get_req_header(conn, "authorization") + assert auth == [sandbox_auth] + refute auth == [parent_auth] + Plug.Conn.send_resp(conn, 200, "ok") + end) + + resp = + conn(:get, "/channels/#{channel.id}/test") + |> send_to_endpoint() + + assert resp.status == 200 + end + + test "a sandbox with no environment set returns 502 with observable error", + %{bypass: bypass} do + # Deriving the environment instead of assuming "main" gave this path two + # new failure reasons. They have to land as a recorded 502 like every + # other credential error, not as an unhandled crash. + root = insert(:project) + sandbox = insert(:project, parent: root, env: nil) + user = insert(:user) + + credential = + insert(:credential, + schema: "@openfn/language-http", + name: "parent", + user: user + ) + + project_credential = + insert(:project_credential, project: sandbox, credential: credential) + + channel = + insert(:channel, + project: sandbox, + destination_url: "http://localhost:#{bypass.port}", + enabled: true, + channel_auth_methods: [ + build(:channel_auth_method, + role: :destination, + webhook_auth_method: nil, + project_credential: project_credential + ) + ] + ) + + resp = + conn(:get, "/channels/#{channel.id}/test") + |> send_to_endpoint() + + assert_problem(resp, 502, :bad_gateway, []) + + request = + Lightning.Repo.one!( + from(r in ChannelRequest, where: r.channel_id == ^channel.id) + ) + + assert request.state == :error + + event = + Lightning.Repo.one!( + from(e in ChannelEvent, where: e.channel_request_id == ^request.id) + ) + + assert event.error_message == "credential_environment_not_configured" + end + + test "credential with missing auth fields returns 502 with observable error", + %{bypass: bypass} do + channel = + create_destination_auth_channel(bypass, "@openfn/language-http", %{ + "baseUrl" => "https://example.com" + }) + + resp = + conn(:get, "/channels/#{channel.id}/test") + |> send_to_endpoint() + + assert_problem(resp, 502, :bad_gateway, []) + + request = + Lightning.Repo.one!( + from(r in ChannelRequest, where: r.channel_id == ^channel.id) + ) + + assert request.state == :error + + event = + Lightning.Repo.one!( + from(e in ChannelEvent, where: e.channel_request_id == ^request.id) + ) + + assert event.error_message == "credential_missing_auth_fields" + end + + test "a credential error whose snapshot can't be saved answers 502 bad_gateway", + %{bypass: bypass} do + channel = + create_destination_auth_channel(bypass, "@openfn/language-http", %{ + "baseUrl" => "https://example.com" + }) + + unsnapshottable!(channel) + + resp = + conn(:get, "/channels/#{channel.id}/test") + |> send_to_endpoint() + + assert_problem(resp, 502, :bad_gateway, []) + + refute Lightning.Repo.exists?( + from(r in ChannelRequest, where: r.channel_id == ^channel.id) + ) + end + + test "proxy headers (x-forwarded-*) still forwarded alongside auth header", + %{bypass: bypass} do + channel = + create_destination_auth_channel(bypass, "@openfn/language-http", %{ + "access_token" => "tok-with-proxy" + }) + + Bypass.expect_once(bypass, "GET", "/test", fn conn -> + auth = Plug.Conn.get_req_header(conn, "authorization") + xff = Plug.Conn.get_req_header(conn, "x-forwarded-for") + xfh = Plug.Conn.get_req_header(conn, "x-forwarded-host") + xfp = Plug.Conn.get_req_header(conn, "x-forwarded-proto") + + assert auth == ["Bearer tok-with-proxy"] + assert xff != [] + assert xfh != [] + assert xfp != [] + + Plug.Conn.send_resp(conn, 200, "ok") + end) + + resp = + conn(:get, "/channels/#{channel.id}/test") + |> send_to_endpoint() + + assert resp.status == 200 + end + end +end diff --git a/test/lightning_web/plugs/channel_proxy_forwarding_test.exs b/test/lightning_web/plugs/channel_proxy_forwarding_test.exs new file mode 100644 index 00000000000..ccb09e64fa6 --- /dev/null +++ b/test/lightning_web/plugs/channel_proxy_forwarding_test.exs @@ -0,0 +1,691 @@ +defmodule LightningWeb.ChannelProxyForwardingTest do + use LightningWeb.ConnCase, async: true + + import Lightning.Factories + import LightningWeb.API.ChannelProxyHelpers + import Plug.Test, only: [conn: 2, conn: 3] + + setup do + bypass = Bypass.open() + project = insert(:project) + + channel = + insert(:channel, + project: project, + destination_url: "http://localhost:#{bypass.port}", + enabled: true + ) + + disabled_channel = + insert(:channel, + project: project, + destination_url: "http://localhost:#{bypass.port}", + enabled: false + ) + + {:ok, bypass: bypass, channel: channel, disabled_channel: disabled_channel} + end + + describe "proxying requests" do + test "GET proxied to destination with correct path", %{ + conn: conn, + bypass: bypass, + channel: channel + } do + Bypass.expect_once(bypass, "GET", "/test", fn conn -> + Plug.Conn.send_resp(conn, 200, "ok from destination") + end) + + resp = get(conn, "/channels/#{channel.id}/test") + + assert resp.status == 200 + assert resp.resp_body == "ok from destination" + end + + test "POST with body arrives at destination unchanged", %{ + bypass: bypass, + channel: channel + } do + body = Jason.encode!(%{"hello" => "world"}) + + Bypass.expect_once(bypass, "POST", "/api/data", fn conn -> + {:ok, received, conn} = Plug.Conn.read_body(conn) + assert received == body + Plug.Conn.send_resp(conn, 201, "created") + end) + + resp = + conn(:post, "/channels/#{channel.id}/api/data", body) + |> put_req_header("content-type", "application/json") + |> put_req_header("content-length", "#{byte_size(body)}") + |> send_to_endpoint() + + assert resp.status == 201 + assert resp.resp_body == "created" + end + + test "PUT method forwarded", %{ + conn: conn, + bypass: bypass, + channel: channel + } do + Bypass.expect_once(bypass, "PUT", "/resource", fn conn -> + Plug.Conn.send_resp(conn, 200, "updated") + end) + + resp = + conn + |> put_req_header("content-type", "application/json") + |> put("/channels/#{channel.id}/resource", "{}") + + assert resp.status == 200 + end + + test "PATCH method forwarded", %{ + conn: conn, + bypass: bypass, + channel: channel + } do + Bypass.expect_once(bypass, "PATCH", "/resource", fn conn -> + Plug.Conn.send_resp(conn, 200, "patched") + end) + + resp = + conn + |> put_req_header("content-type", "application/json") + |> patch("/channels/#{channel.id}/resource", "{}") + + assert resp.status == 200 + end + + test "DELETE method forwarded", %{ + conn: conn, + bypass: bypass, + channel: channel + } do + Bypass.expect_once(bypass, "DELETE", "/resource", fn conn -> + Plug.Conn.send_resp(conn, 204, "") + end) + + resp = delete(conn, "/channels/#{channel.id}/resource") + + assert resp.status == 204 + end + + test "query parameters preserved", %{ + conn: conn, + bypass: bypass, + channel: channel + } do + Bypass.expect_once(bypass, "GET", "/search", fn conn -> + assert conn.query_string == "q=foo&page=2" + Plug.Conn.send_resp(conn, 200, "results") + end) + + resp = get(conn, "/channels/#{channel.id}/search?q=foo&page=2") + + assert resp.status == 200 + end + + test "nested path forwarded correctly", %{ + conn: conn, + bypass: bypass, + channel: channel + } do + Bypass.expect_once(bypass, "GET", "/nested/path/here", fn conn -> + Plug.Conn.send_resp(conn, 200, "deep") + end) + + resp = get(conn, "/channels/#{channel.id}/nested/path/here") + + assert resp.status == 200 + end + + test "path traversal in channel_id position fails UUID validation", %{ + conn: conn + } do + # A path like /channels/../../secret would have ".." as channel_id, + # which fails Ecto.UUID.cast and returns 404. + resp = get(conn, "/channels/../../secret") + + assert resp.status == 404 + end + + test "path traversal segments in subpath are forwarded as-is", %{ + conn: conn, + bypass: bypass, + channel: channel + } do + # In production, HTTP clients (browsers) normalize "../" segments + # before sending the request, so they never reach the server. + # If they do arrive (e.g. from a programmatic client), they are + # forwarded to the upstream as-is — the upstream is responsible + # for its own path handling. + Bypass.expect_once(bypass, "GET", "/foo/../safe", fn conn -> + Plug.Conn.send_resp(conn, 200, "ok") + end) + + resp = get(conn, "/channels/#{channel.id}/foo/../safe") + + assert resp.status == 200 + end + + test "root path (no trailing path) proxies to /", %{ + conn: conn, + bypass: bypass, + channel: channel + } do + Bypass.expect_once(bypass, "GET", "/", fn conn -> + Plug.Conn.send_resp(conn, 200, "root") + end) + + resp = get(conn, "/channels/#{channel.id}") + + assert resp.status == 200 + assert resp.resp_body == "root" + end + + test "trailing slash on destination_url does not produce double slash", %{ + bypass: bypass + } do + project = insert(:project) + + channel = + insert(:channel, + project: project, + destination_url: "http://localhost:#{bypass.port}/", + enabled: true + ) + + Bypass.expect_once(bypass, "GET", "/", fn conn -> + Plug.Conn.send_resp(conn, 200, "root") + end) + + resp = + conn(:get, "/channels/#{channel.id}") + |> send_to_endpoint() + + assert resp.status == 200 + assert resp.resp_body == "root" + end + + test "trailing slash on destination_url with subpath", %{bypass: bypass} do + project = insert(:project) + + channel = + insert(:channel, + project: project, + destination_url: "http://localhost:#{bypass.port}/", + enabled: true + ) + + Bypass.expect_once(bypass, "GET", "/api/data", fn conn -> + Plug.Conn.send_resp(conn, 200, "ok") + end) + + resp = + conn(:get, "/channels/#{channel.id}/api/data") + |> send_to_endpoint() + + assert resp.status == 200 + end + + test "proxy headers forwarded to destination", %{ + conn: conn, + bypass: bypass, + channel: channel + } do + Bypass.expect_once(bypass, "GET", "/test", fn conn -> + xff = Plug.Conn.get_req_header(conn, "x-forwarded-for") + xfh = Plug.Conn.get_req_header(conn, "x-forwarded-host") + xfp = Plug.Conn.get_req_header(conn, "x-forwarded-proto") + + assert xff != [] + assert xfh != [] + assert xfp != [] + + Plug.Conn.send_resp(conn, 200, "ok") + end) + + resp = get(conn, "/channels/#{channel.id}/test") + + assert resp.status == 200 + end + + test "x-request-id forwarded to destination", %{ + bypass: bypass, + channel: channel + } do + test_pid = self() + + Bypass.expect_once(bypass, "GET", "/trace", fn conn -> + [received_id] = Plug.Conn.get_req_header(conn, "x-request-id") + send(test_pid, {:destination_request_id, received_id}) + Plug.Conn.send_resp(conn, 200, "traced") + end) + + resp = + conn(:get, "/channels/#{channel.id}/trace") + |> send_to_endpoint() + + assert resp.status == 200 + + [response_id] = Plug.Conn.get_resp_header(resp, "x-request-id") + assert_receive {:destination_request_id, ^response_id} + end + + test "response headers from destination forwarded to client", %{ + conn: conn, + bypass: bypass, + channel: channel + } do + Bypass.expect_once(bypass, "GET", "/test", fn conn -> + conn + |> Plug.Conn.put_resp_header("x-custom-header", "custom-value") + |> Plug.Conn.send_resp(200, "ok") + end) + + resp = get(conn, "/channels/#{channel.id}/test") + + assert resp.status == 200 + + assert Plug.Conn.get_resp_header(resp, "x-custom-header") == [ + "custom-value" + ] + end + + test "host header sent to destination matches upstream URL, not client's original host", + %{bypass: bypass, channel: channel} do + test_pid = self() + + Bypass.expect_once(bypass, "GET", "/host-check", fn conn -> + [received_host] = Plug.Conn.get_req_header(conn, "host") + send(test_pid, {:destination_host, received_host}) + Plug.Conn.send_resp(conn, 200, "ok") + end) + + # Simulate a real HTTP request with an attacker-controlled host header. + # In production, Cowboy always includes the client's Host header in + # req_headers; Plug.Test.conn does not, so we inject it manually. + base_conn = conn(:get, "/channels/#{channel.id}/host-check") + + resp = + %{ + base_conn + | host: "evil.example.com", + req_headers: [ + {"host", "evil.example.com"} | base_conn.req_headers + ] + } + |> send_to_endpoint() + + assert resp.status == 200 + + assert_receive {:destination_host, received_host} + assert received_host == "localhost:#{bypass.port}" + end + + test "strips the caller's session cookie before proxying to the destination", + %{bypass: bypass, channel: channel} do + test_pid = self() + + Bypass.expect_once(bypass, "GET", "/cookie-check", fn conn -> + send( + test_pid, + {:destination_cookie, Plug.Conn.get_req_header(conn, "cookie")} + ) + + Plug.Conn.send_resp(conn, 200, "ok") + end) + + base_conn = conn(:get, "/channels/#{channel.id}/cookie-check") + + resp = + %{ + base_conn + | req_headers: [ + {"cookie", "_lightning_key=victim-session-token"} + | base_conn.req_headers + ] + } + |> send_to_endpoint() + + assert resp.status == 200 + assert_receive {:destination_cookie, []} + end + end + + describe "client auth header stripping" do + test "strips x-api-key header when client uses API key auth", %{ + bypass: bypass + } do + channel = + create_client_auth_channel(bypass, [ + %{auth_type: :api, api_key: "valid-api-key"} + ]) + + Bypass.expect_once(bypass, "GET", "/test", fn conn -> + api_key = Plug.Conn.get_req_header(conn, "x-api-key") + + assert api_key == [], + "x-api-key should not be forwarded to the destination" + + Plug.Conn.send_resp(conn, 200, "ok") + end) + + conn(:get, "/channels/#{channel.id}/test") + |> put_req_header("x-api-key", "valid-api-key") + |> send_to_endpoint() + end + + test "strips authorization header when client uses Basic auth", %{ + bypass: bypass + } do + channel = + create_client_auth_channel(bypass, [ + %{auth_type: :basic, username: "admin", password: "secretpw"} + ]) + + encoded = Base.encode64("admin:secretpw") + + Bypass.expect_once(bypass, "GET", "/test", fn conn -> + auth = Plug.Conn.get_req_header(conn, "authorization") + + assert auth == [], + "client Basic auth should not be forwarded to the destination" + + Plug.Conn.send_resp(conn, 200, "ok") + end) + + conn(:get, "/channels/#{channel.id}/test") + |> put_req_header("authorization", "Basic #{encoded}") + |> send_to_endpoint() + end + + test "replaces client Basic auth with destination Bearer auth", %{ + bypass: bypass + } do + project = insert(:project) + user = insert(:user) + + credential = + insert(:credential, + schema: "@openfn/language-http", + name: "destination-cred", + user: user + ) + |> with_body(%{body: %{"access_token" => "dest-token-xyz"}}) + + project_credential = + insert(:project_credential, + project: project, + credential: credential + ) + + client_encoded = Base.encode64("user:password") + + channel = + insert(:channel, + project: project, + destination_url: "http://localhost:#{bypass.port}", + enabled: true, + channel_auth_methods: [ + build(:channel_auth_method, + role: :client, + webhook_auth_method: + build(:webhook_auth_method, + project: project, + auth_type: :basic, + username: "user", + password: "password" + ) + ), + build(:channel_auth_method, + role: :destination, + webhook_auth_method: nil, + project_credential: project_credential + ) + ] + ) + + Bypass.expect_once(bypass, "GET", "/test", fn conn -> + auth = Plug.Conn.get_req_header(conn, "authorization") + + assert auth == ["Bearer dest-token-xyz"], + "destination should receive the destination Bearer token, not the client Basic auth" + + Plug.Conn.send_resp(conn, 200, "ok") + end) + + conn(:get, "/channels/#{channel.id}/test") + |> put_req_header("authorization", "Basic #{client_encoded}") + |> send_to_endpoint() + end + + test "passes through client auth headers when no client auth configured", + %{bypass: bypass, channel: channel} do + Bypass.expect_once(bypass, "GET", "/test", fn conn -> + auth = Plug.Conn.get_req_header(conn, "authorization") + + assert auth == ["Bearer client-token"], + "client auth headers should pass through when no client auth is configured" + + Plug.Conn.send_resp(conn, 200, "ok") + end) + + conn(:get, "/channels/#{channel.id}/test") + |> put_req_header("authorization", "Bearer client-token") + |> send_to_endpoint() + end + + test "forwards x-api-key when no client auth configured", + %{bypass: bypass, channel: channel} do + headers = + conn(:get, "/channels/#{channel.id}/test") + |> put_req_header("x-api-key", "client-key") + |> upstream_auth_headers(bypass) + + assert headers["x-api-key"] == ["client-key"] + end + + test "forwards x-api-key and replaces authorization with the destination's when no client auth configured", + %{bypass: bypass} do + channel = create_client_auth_channel(bypass, [], "dest-token") + + headers = + conn(:get, "/channels/#{channel.id}/test") + |> put_req_header("authorization", "Bearer client-token") + |> put_req_header("x-api-key", "client-key") + |> upstream_auth_headers(bypass) + + assert headers["authorization"] == ["Bearer dest-token"] + assert headers["x-api-key"] == ["client-key"] + end + + test "forwards x-api-key when client uses Basic auth", %{bypass: bypass} do + channel = + create_client_auth_channel(bypass, [ + %{auth_type: :basic, username: "admin", password: "secretpw"} + ]) + + headers = + conn(:get, "/channels/#{channel.id}/test") + |> put_req_header( + "authorization", + "Basic #{Base.encode64("admin:secretpw")}" + ) + |> put_req_header("x-api-key", "client-key") + |> upstream_auth_headers(bypass) + + assert headers["x-api-key"] == ["client-key"] + end + + test "forwards x-api-key when client uses Basic auth and the destination has auth", + %{bypass: bypass} do + channel = + create_client_auth_channel( + bypass, + [%{auth_type: :basic, username: "admin", password: "secretpw"}], + "dest-token" + ) + + headers = + conn(:get, "/channels/#{channel.id}/test") + |> put_req_header( + "authorization", + "Basic #{Base.encode64("admin:secretpw")}" + ) + |> put_req_header("x-api-key", "client-key") + |> upstream_auth_headers(bypass) + + assert headers["x-api-key"] == ["client-key"] + end + + test "forwards authorization when client uses API key auth", + %{bypass: bypass} do + channel = + create_client_auth_channel(bypass, [ + %{auth_type: :api, api_key: "valid-api-key"} + ]) + + headers = + conn(:get, "/channels/#{channel.id}/test") + |> put_req_header("x-api-key", "valid-api-key") + |> put_req_header("authorization", "Bearer client-token") + |> upstream_auth_headers(bypass) + + assert headers["authorization"] == ["Bearer client-token"] + end + + test "strips x-api-key and replaces authorization with the destination's when client uses API key auth", + %{bypass: bypass} do + channel = + create_client_auth_channel( + bypass, + [%{auth_type: :api, api_key: "valid-api-key"}], + "dest-token" + ) + + headers = + conn(:get, "/channels/#{channel.id}/test") + |> put_req_header("x-api-key", "valid-api-key") + |> put_req_header("authorization", "Bearer client-token") + |> upstream_auth_headers(bypass) + + assert headers["authorization"] == ["Bearer dest-token"] + assert headers["x-api-key"] == [] + end + + test "strips authorization and x-api-key when client has Basic and API key auth", + %{bypass: bypass} do + channel = + create_client_auth_channel(bypass, [ + %{auth_type: :basic, username: "admin", password: "secretpw"}, + %{auth_type: :api, api_key: "valid-api-key"} + ]) + + headers = + conn(:get, "/channels/#{channel.id}/test") + |> put_req_header( + "authorization", + "Basic #{Base.encode64("admin:secretpw")}" + ) + |> put_req_header("x-api-key", "valid-api-key") + |> upstream_auth_headers(bypass) + + assert headers["authorization"] == [] + assert headers["x-api-key"] == [] + end + + test "strips x-api-key and replaces authorization with the destination's when client has Basic and API key auth", + %{bypass: bypass} do + channel = + create_client_auth_channel( + bypass, + [ + %{auth_type: :basic, username: "admin", password: "secretpw"}, + %{auth_type: :api, api_key: "valid-api-key"} + ], + "dest-token" + ) + + headers = + conn(:get, "/channels/#{channel.id}/test") + |> put_req_header( + "authorization", + "Basic #{Base.encode64("admin:secretpw")}" + ) + |> put_req_header("x-api-key", "valid-api-key") + |> upstream_auth_headers(bypass) + + assert headers["authorization"] == ["Bearer dest-token"] + assert headers["x-api-key"] == [] + end + + test "strips proxy-authorization", %{bypass: bypass, channel: channel} do + headers = + conn(:get, "/channels/#{channel.id}/test") + |> put_req_header( + "proxy-authorization", + "Basic #{Base.encode64("proxy:pw")}" + ) + |> upstream_auth_headers(bypass) + + assert headers["proxy-authorization"] == [] + end + + defp upstream_auth_headers(conn, bypass) do + test = self() + + Bypass.expect_once(bypass, "GET", "/test", fn upstream -> + headers = + Map.new( + ["authorization", "x-api-key", "proxy-authorization"], + &{&1, Plug.Conn.get_req_header(upstream, &1)} + ) + + send(test, {:upstream_auth_headers, headers}) + Plug.Conn.send_resp(upstream, 200, "ok") + end) + + assert %{status: 200} = send_to_endpoint(conn) + assert_received {:upstream_auth_headers, headers} + headers + end + end + + describe "destination answers" do + for status <- [400, 404, 422, 500, 503] do + test "a #{status} from the destination passes through untouched", %{ + conn: conn, + bypass: bypass, + channel: channel + } do + status = unquote(status) + body = ~s({"error":"Not Found","detail":"the destination's own"}) + + Bypass.expect_once(bypass, "GET", "/thing", fn conn -> + conn + |> Plug.Conn.put_resp_header("x-destination", "kept") + |> Plug.Conn.put_resp_content_type("application/vnd.dest+json") + |> Plug.Conn.send_resp(status, body) + end) + + resp = + conn + |> put_req_header("accept", "application/problem+json") + |> get("/channels/#{channel.id}/thing") + + assert resp.status == status + assert resp.resp_body == body + + assert Plug.Conn.get_resp_header(resp, "content-type") == [ + "application/vnd.dest+json; charset=utf-8" + ] + + assert Plug.Conn.get_resp_header(resp, "x-destination") == ["kept"] + assert Plug.Conn.get_resp_header(resp, "www-authenticate") == [] + end + end + end +end diff --git a/test/lightning_web/plugs/channel_proxy_persistence_test.exs b/test/lightning_web/plugs/channel_proxy_persistence_test.exs new file mode 100644 index 00000000000..2513b187ab3 --- /dev/null +++ b/test/lightning_web/plugs/channel_proxy_persistence_test.exs @@ -0,0 +1,777 @@ +defmodule LightningWeb.ChannelProxyPersistenceTest do + use LightningWeb.ConnCase, async: true + + alias Lightning.Channels.{ChannelRequest, ChannelEvent} + + import Ecto.Query + import Lightning.Factories + import LightningWeb.API.ChannelProxyHelpers + import Plug.Test, only: [conn: 2, conn: 3] + + setup do + bypass = Bypass.open() + project = insert(:project) + + channel = + insert(:channel, + project: project, + destination_url: "http://localhost:#{bypass.port}", + enabled: true + ) + + disabled_channel = + insert(:channel, + project: project, + destination_url: "http://localhost:#{bypass.port}", + enabled: false + ) + + {:ok, bypass: bypass, channel: channel, disabled_channel: disabled_channel} + end + + describe "handler persistence" do + test "creates ChannelRequest and ChannelEvent on successful proxy", %{ + conn: conn, + bypass: bypass, + channel: channel + } do + Bypass.expect_once(bypass, "GET", "/persisted", fn conn -> + Plug.Conn.send_resp(conn, 200, "persisted response") + end) + + resp = get(conn, "/channels/#{channel.id}/persisted") + assert resp.status == 200 + + request = + Lightning.Repo.one!( + from(r in ChannelRequest, where: r.channel_id == ^channel.id) + ) + + assert request.state == :success + assert request.completed_at != nil + + event = + Lightning.Repo.one!( + from(e in ChannelEvent, where: e.channel_request_id == ^request.id) + ) + + assert event.type == :destination_response + assert event.response_status == 200 + assert event.latency_us != nil + assert event.request_method == "GET" + assert event.request_path == "/persisted" + end + + test "creates error-state request on connection failure", %{ + conn: conn, + channel: channel + } do + port = Enum.random(59_000..59_999) + + channel + |> Ecto.Changeset.change(destination_url: "http://localhost:#{port}") + |> Lightning.Repo.update!() + + resp = get(conn, "/channels/#{channel.id}/fail") + assert resp.status in [502, 504] + + request = + Lightning.Repo.one!( + from(r in ChannelRequest, where: r.channel_id == ^channel.id) + ) + + assert request.state in [:error, :timeout] + assert request.completed_at != nil + end + end + + describe "query string persistence" do + test "persists query string on channel event", %{ + bypass: bypass, + channel: channel + } do + Bypass.expect_once(bypass, "GET", "/search", fn conn -> + Plug.Conn.send_resp(conn, 200, "results") + end) + + conn(:get, "/channels/#{channel.id}/search?q=foo&page=2") + |> send_to_endpoint() + + event = + Lightning.Repo.one!( + from(e in ChannelEvent, + join: r in ChannelRequest, + on: r.id == e.channel_request_id, + where: r.channel_id == ^channel.id + ) + ) + + assert event.request_query_string == "q=foo&page=2" + end + + test "empty query string when no params", %{ + bypass: bypass, + channel: channel + } do + Bypass.expect_once(bypass, "GET", "/plain", fn conn -> + Plug.Conn.send_resp(conn, 200, "ok") + end) + + conn(:get, "/channels/#{channel.id}/plain") + |> send_to_endpoint() + + event = + Lightning.Repo.one!( + from(e in ChannelEvent, + join: r in ChannelRequest, + on: r.id == e.channel_request_id, + where: r.channel_id == ^channel.id + ) + ) + + assert event.request_query_string == "" + end + end + + describe "client auth tracking" do + test "persists auth method ID and type for API key auth", %{bypass: bypass} do + channel = + create_client_auth_channel(bypass, [ + %{auth_type: :api, api_key: "track-me"} + ]) + + auth_method = + channel + |> Lightning.Repo.preload(client_webhook_auth_methods: []) + |> Map.get(:client_webhook_auth_methods) + |> hd() + + Bypass.expect_once(bypass, "GET", "/tracked", fn conn -> + Plug.Conn.send_resp(conn, 200, "ok") + end) + + conn(:get, "/channels/#{channel.id}/tracked") + |> put_req_header("x-api-key", "track-me") + |> send_to_endpoint() + + request = + Lightning.Repo.one!( + from(r in ChannelRequest, where: r.channel_id == ^channel.id) + ) + + assert request.client_webhook_auth_method_id == auth_method.id + assert request.client_auth_type == "api" + end + + test "persists auth method ID and type for Basic auth", %{bypass: bypass} do + channel = + create_client_auth_channel(bypass, [ + %{auth_type: :basic, username: "user", password: "pass"} + ]) + + auth_method = + channel + |> Lightning.Repo.preload(client_webhook_auth_methods: []) + |> Map.get(:client_webhook_auth_methods) + |> hd() + + Bypass.expect_once(bypass, "GET", "/tracked", fn conn -> + Plug.Conn.send_resp(conn, 200, "ok") + end) + + encoded = Base.encode64("user:pass") + + conn(:get, "/channels/#{channel.id}/tracked") + |> put_req_header("authorization", "Basic #{encoded}") + |> send_to_endpoint() + + request = + Lightning.Repo.one!( + from(r in ChannelRequest, where: r.channel_id == ^channel.id) + ) + + assert request.client_webhook_auth_method_id == auth_method.id + assert request.client_auth_type == "basic" + end + + test "nil auth method when no client auth configured", %{ + bypass: bypass, + channel: channel + } do + Bypass.expect_once(bypass, "GET", "/open", fn conn -> + Plug.Conn.send_resp(conn, 200, "ok") + end) + + conn(:get, "/channels/#{channel.id}/open") + |> send_to_endpoint() + + request = + Lightning.Repo.one!( + from(r in ChannelRequest, where: r.channel_id == ^channel.id) + ) + + assert request.client_webhook_auth_method_id == nil + assert request.client_auth_type == nil + end + end + + describe "destination auth tracking" do + test "persists destination_credential_id on successful proxy with destination auth", + %{bypass: bypass} do + channel = + create_destination_auth_channel(bypass, "@openfn/language-http", %{ + "access_token" => "tok-123" + }) + + project_credential_id = + channel + |> Lightning.Repo.preload(destination_auth_method: :project_credential) + |> get_in([ + Access.key(:destination_auth_method), + Access.key(:project_credential_id) + ]) + + Bypass.expect_once(bypass, "GET", "/dest-track", fn conn -> + Plug.Conn.send_resp(conn, 200, "ok") + end) + + conn(:get, "/channels/#{channel.id}/dest-track") + |> send_to_endpoint() + + request = + Lightning.Repo.one!( + from(r in ChannelRequest, where: r.channel_id == ^channel.id) + ) + + assert request.destination_credential_id == project_credential_id + refute is_nil(project_credential_id) + end + + test "persists destination_credential_id even when credential resolution fails", + %{bypass: _bypass} do + # Channel with a destination auth method but credential missing auth + # fields — destination auth resolution fails, but we still know which + # credential was configured. + project = insert(:project) + user = insert(:user) + + credential = + insert(:credential, + schema: "@openfn/language-http", + name: "bad-cred", + user: user + ) + |> with_body(%{body: %{"baseUrl" => "https://example.com"}}) + + project_credential = + insert(:project_credential, project: project, credential: credential) + + channel = + insert(:channel, + project: project, + destination_url: "http://localhost:9999", + enabled: true, + channel_auth_methods: [ + build(:channel_auth_method, + role: :destination, + webhook_auth_method: nil, + project_credential: project_credential + ) + ] + ) + + resp = + conn(:get, "/channels/#{channel.id}/test") + |> send_to_endpoint() + + assert resp.status == 502 + + request = + Lightning.Repo.one!( + from(r in ChannelRequest, where: r.channel_id == ^channel.id) + ) + + assert request.destination_credential_id == project_credential.id + assert request.state == :error + end + + test "destination_credential_id is nil when no destination auth configured", + %{bypass: bypass, channel: channel} do + Bypass.expect_once(bypass, "GET", "/no-dest-auth", fn conn -> + Plug.Conn.send_resp(conn, 200, "ok") + end) + + conn(:get, "/channels/#{channel.id}/no-dest-auth") + |> send_to_endpoint() + + request = + Lightning.Repo.one!( + from(r in ChannelRequest, where: r.channel_id == ^channel.id) + ) + + assert request.destination_credential_id == nil + end + end + + describe "collect_timing integration" do + test "persists per-direction timing after successful proxy", %{ + bypass: bypass, + channel: channel + } do + Bypass.expect_once(bypass, "GET", "/timed", fn conn -> + Plug.Conn.send_resp(conn, 200, "ok") + end) + + conn(:get, "/channels/#{channel.id}/timed") + |> send_to_endpoint() + + event = + Lightning.Repo.one!( + from(e in ChannelEvent, + join: r in ChannelRequest, + on: r.id == e.channel_request_id, + where: r.channel_id == ^channel.id + ) + ) + + # With collect_timing: true, Philter populates timing.send_us + # which the handler persists as request_send_us + assert is_integer(event.request_send_us) + assert event.request_send_us >= 0 + end + end + + describe "zero-persistence scrubbing" do + test "happy path: scrubs PII fields and sets is_wiped: true under :erase_all", + %{bypass: bypass} do + project = insert(:project, retention_policy: :erase_all) + + channel = + insert(:channel, + project: project, + destination_url: "http://localhost:#{bypass.port}", + enabled: true + ) + + body = Jason.encode!(%{"hello" => "world"}) + + Bypass.expect_once(bypass, "POST", "/erase/path", fn conn -> + conn + |> Plug.Conn.put_resp_header("x-custom-header", "value") + |> Plug.Conn.put_resp_header("content-type", "application/json") + |> Plug.Conn.send_resp(200, ~s({"ok":true})) + end) + + resp = + conn(:post, "/channels/#{channel.id}/erase/path?secret=abc", body) + |> put_req_header("content-type", "application/json") + |> put_req_header("content-length", "#{byte_size(body)}") + |> put_req_header("x-api-key", "some-key") + |> send_to_endpoint() + + assert resp.status == 200 + + request = + Lightning.Repo.one!( + from(r in ChannelRequest, where: r.channel_id == ^channel.id) + ) + + assert %ChannelRequest{ + state: :success, + client_identity: nil, + is_wiped: true + } = request + + assert request.started_at != nil + assert request.completed_at != nil + assert is_binary(request.request_id) + + event = + Lightning.Repo.one!( + from(e in ChannelEvent, where: e.channel_request_id == ^request.id) + ) + + # All eight scrubbed fields are nil on the event; is_wiped lives on the request. + assert %ChannelEvent{ + request_path: nil, + request_query_string: nil, + request_headers: nil, + request_body_preview: nil, + request_body_hash: nil, + response_headers: nil, + response_body_preview: nil, + response_body_hash: nil + } = event + + # Observability fields remain populated. + assert event.request_method == "POST" + assert event.response_status == 200 + assert is_integer(event.latency_us) and event.latency_us > 0 + assert event.request_body_size == byte_size(body) + + assert is_integer(event.response_body_size) and + event.response_body_size > 0 + end + + test "happy path: retains all PII fields and sets is_wiped: false under :retain_all", + %{bypass: bypass} do + project = insert(:project, retention_policy: :retain_all) + + channel = + insert(:channel, + project: project, + destination_url: "http://localhost:#{bypass.port}", + enabled: true + ) + + body = Jason.encode!(%{"hello" => "world"}) + + Bypass.expect_once(bypass, "POST", "/retain/path", fn conn -> + conn + |> Plug.Conn.put_resp_header("x-custom-header", "value") + |> Plug.Conn.put_resp_header("content-type", "application/json") + |> Plug.Conn.send_resp(200, ~s({"ok":true})) + end) + + resp = + conn(:post, "/channels/#{channel.id}/retain/path?secret=abc", body) + |> put_req_header("content-type", "application/json") + |> put_req_header("content-length", "#{byte_size(body)}") + |> put_req_header("x-api-key", "some-key") + |> send_to_endpoint() + + assert resp.status == 200 + + request = + Lightning.Repo.one!( + from(r in ChannelRequest, where: r.channel_id == ^channel.id) + ) + + assert request.state == :success + assert request.completed_at != nil + assert request.is_wiped == false + refute is_nil(request.client_identity) + + event = + Lightning.Repo.one!( + from(e in ChannelEvent, where: e.channel_request_id == ^request.id) + ) + + assert event.request_method == "POST" + assert event.request_path == "/retain/path" + assert event.request_query_string == "secret=abc" + assert event.response_status == 200 + assert is_list(event.request_headers) and event.request_headers != [] + assert is_list(event.response_headers) and event.response_headers != [] + assert is_binary(event.request_body_preview) + assert is_binary(event.request_body_hash) + assert is_binary(event.response_body_preview) + assert is_binary(event.response_body_hash) + assert is_integer(event.latency_us) and event.latency_us > 0 + assert event.request_body_size == byte_size(body) + end + + test "credential-error path: scrubs request_path and client_identity, sets is_wiped: true, but keeps request_method and error_message", + %{bypass: bypass} do + project = insert(:project, retention_policy: :erase_all) + user = insert(:user) + + credential = + insert(:credential, + schema: "@openfn/language-http", + name: "no-body", + user: user + ) + + # Don't call with_body — no CredentialBody exists, so credential + # resolution will fail and `record_credential_error/3` is invoked. + + project_credential = + insert(:project_credential, + project: project, + credential: credential + ) + + channel = + insert(:channel, + project: project, + destination_url: "http://localhost:#{bypass.port}", + enabled: true, + channel_auth_methods: [ + build(:channel_auth_method, + role: :destination, + webhook_auth_method: nil, + project_credential: project_credential + ) + ] + ) + + resp = + conn(:get, "/channels/#{channel.id}/secret-path") + |> send_to_endpoint() + + assert resp.status == 502 + + request = + Lightning.Repo.one!( + from(r in ChannelRequest, where: r.channel_id == ^channel.id) + ) + + assert %ChannelRequest{ + state: :error, + client_identity: nil, + is_wiped: true + } = request + + event = + Lightning.Repo.one!( + from(e in ChannelEvent, where: e.channel_request_id == ^request.id) + ) + + assert %ChannelEvent{ + type: :error, + request_path: nil, + request_method: "GET", + error_message: "credential_environment_not_found" + } = event + end + + test "credential-error path: leaves request_path populated and request.is_wiped: false under :retain_all (default)", + %{bypass: bypass} do + project = insert(:project, retention_policy: :retain_all) + user = insert(:user) + + credential = + insert(:credential, + schema: "@openfn/language-http", + name: "no-body", + user: user + ) + + project_credential = + insert(:project_credential, + project: project, + credential: credential + ) + + channel = + insert(:channel, + project: project, + destination_url: "http://localhost:#{bypass.port}", + enabled: true, + channel_auth_methods: [ + build(:channel_auth_method, + role: :destination, + webhook_auth_method: nil, + project_credential: project_credential + ) + ] + ) + + resp = + conn(:get, "/channels/#{channel.id}/keep-path") + |> send_to_endpoint() + + assert resp.status == 502 + + request = + Lightning.Repo.one!( + from(r in ChannelRequest, where: r.channel_id == ^channel.id) + ) + + assert request.state == :error + assert request.is_wiped == false + refute is_nil(request.client_identity) + + event = + Lightning.Repo.one!( + from(e in ChannelEvent, where: e.channel_request_id == ^request.id) + ) + + assert %ChannelEvent{ + type: :error, + request_path: "/channels/" <> _, + request_method: "GET", + error_message: "credential_environment_not_found" + } = event + end + end + + describe "request telemetry" do + setup do + test_pid = self() + handler_id = "channel-proxy-test-#{System.unique_integer([:positive])}" + + :ok = + :telemetry.attach_many( + handler_id, + [ + [:lightning, :channel_proxy, :inbound, :stop], + [:lightning, :channel_proxy, :request, :start], + [:lightning, :channel_proxy, :request, :stop] + ], + fn event, measurements, metadata, _config -> + if self() == test_pid do + send(test_pid, {:telemetry, event, measurements, metadata}) + end + end, + nil + ) + + on_exit(fn -> :telemetry.detach(handler_id) end) + + :ok + end + + test "resolved channel: outer :inbound :stop and inner :request :start/:stop fire with real UUIDs", + %{bypass: bypass, channel: channel} do + Bypass.expect_once(bypass, "GET", "/observed", fn conn -> + Plug.Conn.send_resp(conn, 200, "ok") + end) + + resp = + conn(:get, "/channels/#{channel.id}/observed") + |> send_to_endpoint() + + assert resp.status == 200 + + expected_channel_id = channel.id + expected_project_id = channel.project_id + + # Outer span carries outcome + resolved IDs (real UUIDs, never "unknown"). + assert_receive {:telemetry, [:lightning, :channel_proxy, :inbound, :stop], + _measurements, inbound_meta} + + assert %{ + outcome: :resolved, + status: 200, + channel_id: ^expected_channel_id, + project_id: ^expected_project_id + } = inbound_meta + + # Inner span fires only on the resolved path. :start metadata already + # carries the resolved project_id (the whole point of the split). + assert_receive {:telemetry, [:lightning, :channel_proxy, :request, :start], + _measurements, start_meta} + + assert %{ + channel_id: ^expected_channel_id, + project_id: ^expected_project_id + } = start_meta + + assert_receive {:telemetry, [:lightning, :channel_proxy, :request, :stop], + %{duration: duration}, stop_meta} + + assert is_integer(duration) and duration > 0 + + assert %{ + channel_id: ^expected_channel_id, + project_id: ^expected_project_id, + status: 200 + } = stop_meta + end + + test "unknown channel: outer :inbound :stop fires with :unknown_channel and no inner events", + %{conn: conn} do + missing_id = "00000000-0000-0000-0000-000000000000" + + resp = get(conn, "/channels/#{missing_id}/whatever") + + assert resp.status == 404 + + assert_receive {:telemetry, [:lightning, :channel_proxy, :inbound, :stop], + _measurements, inbound_meta} + + assert %{outcome: :unknown_channel, status: 404} = inbound_meta + refute Map.has_key?(inbound_meta, :channel_id) + refute Map.has_key?(inbound_meta, :project_id) + + # The inner :request span never opens for unknown channels. + refute_receive {:telemetry, [:lightning, :channel_proxy, :request, :start], + _, _} + + refute_receive {:telemetry, [:lightning, :channel_proxy, :request, :stop], + _, _} + end + + test "invalid UUID: outer :inbound :stop fires with :invalid_uuid and no inner events", + %{conn: conn} do + resp = get(conn, "/channels/not-a-uuid/whatever") + + assert resp.status == 404 + + assert_receive {:telemetry, [:lightning, :channel_proxy, :inbound, :stop], + _measurements, inbound_meta} + + assert %{outcome: :invalid_uuid, status: 404} = inbound_meta + refute Map.has_key?(inbound_meta, :channel_id) + refute Map.has_key?(inbound_meta, :project_id) + + refute_receive {:telemetry, [:lightning, :channel_proxy, :request, :start], + _, _} + + refute_receive {:telemetry, [:lightning, :channel_proxy, :request, :stop], + _, _} + end + + test "401 unauthorized: outer :inbound and inner :request both fire (auth lives inside inner span)", + %{bypass: bypass} do + project = insert(:project) + + channel = + insert(:channel, + project: project, + destination_url: "http://localhost:#{bypass.port}", + enabled: true, + channel_auth_methods: [ + build(:channel_auth_method, + role: :client, + webhook_auth_method: + build(:webhook_auth_method, + project: project, + auth_type: :api, + api_key: "correct-key" + ) + ) + ] + ) + + resp = + conn(:get, "/channels/#{channel.id}/protected") + |> put_req_header("x-api-key", "wrong-key") + |> send_to_endpoint() + + assert resp.status == 401 + + expected_channel_id = channel.id + expected_project_id = channel.project_id + + assert_receive {:telemetry, [:lightning, :channel_proxy, :inbound, :stop], + _measurements, inbound_meta} + + assert %{ + outcome: :resolved, + status: 401, + channel_id: ^expected_channel_id, + project_id: ^expected_project_id + } = inbound_meta + + assert_receive {:telemetry, [:lightning, :channel_proxy, :request, :start], + _measurements, start_meta} + + assert %{ + channel_id: ^expected_channel_id, + project_id: ^expected_project_id + } = start_meta + + assert_receive {:telemetry, [:lightning, :channel_proxy, :request, :stop], + _measurements, stop_meta} + + assert %{ + channel_id: ^expected_channel_id, + project_id: ^expected_project_id, + status: 401 + } = stop_meta + end + end +end diff --git a/test/lightning_web/plugs/channel_proxy_plug_test.exs b/test/lightning_web/plugs/channel_proxy_plug_test.exs deleted file mode 100644 index 43d081963f0..00000000000 --- a/test/lightning_web/plugs/channel_proxy_plug_test.exs +++ /dev/null @@ -1,2277 +0,0 @@ -defmodule LightningWeb.ChannelProxyPlugTest do - use LightningWeb.ConnCase, async: true - - alias Lightning.Channels.{ChannelRequest, ChannelEvent} - - import Ecto.Query - import Lightning.Factories - import Plug.Test, only: [conn: 2, conn: 3] - - setup do - bypass = Bypass.open() - project = insert(:project) - - channel = - insert(:channel, - project: project, - destination_url: "http://localhost:#{bypass.port}", - enabled: true - ) - - disabled_channel = - insert(:channel, - project: project, - destination_url: "http://localhost:#{bypass.port}", - enabled: false - ) - - {:ok, bypass: bypass, channel: channel, disabled_channel: disabled_channel} - end - - describe "proxying requests" do - test "GET proxied to destination with correct path", %{ - conn: conn, - bypass: bypass, - channel: channel - } do - Bypass.expect_once(bypass, "GET", "/test", fn conn -> - Plug.Conn.send_resp(conn, 200, "ok from destination") - end) - - resp = get(conn, "/channels/#{channel.id}/test") - - assert resp.status == 200 - assert resp.resp_body == "ok from destination" - end - - test "POST with body arrives at destination unchanged", %{ - bypass: bypass, - channel: channel - } do - body = Jason.encode!(%{"hello" => "world"}) - - Bypass.expect_once(bypass, "POST", "/api/data", fn conn -> - {:ok, received, conn} = Plug.Conn.read_body(conn) - assert received == body - Plug.Conn.send_resp(conn, 201, "created") - end) - - resp = - conn(:post, "/channels/#{channel.id}/api/data", body) - |> put_req_header("content-type", "application/json") - |> put_req_header("content-length", "#{byte_size(body)}") - |> send_to_endpoint() - - assert resp.status == 201 - assert resp.resp_body == "created" - end - - test "PUT method forwarded", %{ - conn: conn, - bypass: bypass, - channel: channel - } do - Bypass.expect_once(bypass, "PUT", "/resource", fn conn -> - Plug.Conn.send_resp(conn, 200, "updated") - end) - - resp = - conn - |> put_req_header("content-type", "application/json") - |> put("/channels/#{channel.id}/resource", "{}") - - assert resp.status == 200 - end - - test "PATCH method forwarded", %{ - conn: conn, - bypass: bypass, - channel: channel - } do - Bypass.expect_once(bypass, "PATCH", "/resource", fn conn -> - Plug.Conn.send_resp(conn, 200, "patched") - end) - - resp = - conn - |> put_req_header("content-type", "application/json") - |> patch("/channels/#{channel.id}/resource", "{}") - - assert resp.status == 200 - end - - test "DELETE method forwarded", %{ - conn: conn, - bypass: bypass, - channel: channel - } do - Bypass.expect_once(bypass, "DELETE", "/resource", fn conn -> - Plug.Conn.send_resp(conn, 204, "") - end) - - resp = delete(conn, "/channels/#{channel.id}/resource") - - assert resp.status == 204 - end - - test "query parameters preserved", %{ - conn: conn, - bypass: bypass, - channel: channel - } do - Bypass.expect_once(bypass, "GET", "/search", fn conn -> - assert conn.query_string == "q=foo&page=2" - Plug.Conn.send_resp(conn, 200, "results") - end) - - resp = get(conn, "/channels/#{channel.id}/search?q=foo&page=2") - - assert resp.status == 200 - end - - test "nested path forwarded correctly", %{ - conn: conn, - bypass: bypass, - channel: channel - } do - Bypass.expect_once(bypass, "GET", "/nested/path/here", fn conn -> - Plug.Conn.send_resp(conn, 200, "deep") - end) - - resp = get(conn, "/channels/#{channel.id}/nested/path/here") - - assert resp.status == 200 - end - - test "path traversal in channel_id position fails UUID validation", %{ - conn: conn - } do - # A path like /channels/../../secret would have ".." as channel_id, - # which fails Ecto.UUID.cast and returns 404. - resp = get(conn, "/channels/../../secret") - - assert resp.status == 404 - end - - test "path traversal segments in subpath are forwarded as-is", %{ - conn: conn, - bypass: bypass, - channel: channel - } do - # In production, HTTP clients (browsers) normalize "../" segments - # before sending the request, so they never reach the server. - # If they do arrive (e.g. from a programmatic client), they are - # forwarded to the upstream as-is — the upstream is responsible - # for its own path handling. - Bypass.expect_once(bypass, "GET", "/foo/../safe", fn conn -> - Plug.Conn.send_resp(conn, 200, "ok") - end) - - resp = get(conn, "/channels/#{channel.id}/foo/../safe") - - assert resp.status == 200 - end - - test "root path (no trailing path) proxies to /", %{ - conn: conn, - bypass: bypass, - channel: channel - } do - Bypass.expect_once(bypass, "GET", "/", fn conn -> - Plug.Conn.send_resp(conn, 200, "root") - end) - - resp = get(conn, "/channels/#{channel.id}") - - assert resp.status == 200 - assert resp.resp_body == "root" - end - - test "trailing slash on destination_url does not produce double slash", %{ - bypass: bypass - } do - project = insert(:project) - - channel = - insert(:channel, - project: project, - destination_url: "http://localhost:#{bypass.port}/", - enabled: true - ) - - Bypass.expect_once(bypass, "GET", "/", fn conn -> - Plug.Conn.send_resp(conn, 200, "root") - end) - - resp = - conn(:get, "/channels/#{channel.id}") - |> send_to_endpoint() - - assert resp.status == 200 - assert resp.resp_body == "root" - end - - test "trailing slash on destination_url with subpath", %{bypass: bypass} do - project = insert(:project) - - channel = - insert(:channel, - project: project, - destination_url: "http://localhost:#{bypass.port}/", - enabled: true - ) - - Bypass.expect_once(bypass, "GET", "/api/data", fn conn -> - Plug.Conn.send_resp(conn, 200, "ok") - end) - - resp = - conn(:get, "/channels/#{channel.id}/api/data") - |> send_to_endpoint() - - assert resp.status == 200 - end - - test "proxy headers forwarded to destination", %{ - conn: conn, - bypass: bypass, - channel: channel - } do - Bypass.expect_once(bypass, "GET", "/test", fn conn -> - xff = Plug.Conn.get_req_header(conn, "x-forwarded-for") - xfh = Plug.Conn.get_req_header(conn, "x-forwarded-host") - xfp = Plug.Conn.get_req_header(conn, "x-forwarded-proto") - - assert xff != [] - assert xfh != [] - assert xfp != [] - - Plug.Conn.send_resp(conn, 200, "ok") - end) - - resp = get(conn, "/channels/#{channel.id}/test") - - assert resp.status == 200 - end - - test "x-request-id forwarded to destination", %{ - bypass: bypass, - channel: channel - } do - test_pid = self() - - Bypass.expect_once(bypass, "GET", "/trace", fn conn -> - [received_id] = Plug.Conn.get_req_header(conn, "x-request-id") - send(test_pid, {:destination_request_id, received_id}) - Plug.Conn.send_resp(conn, 200, "traced") - end) - - resp = - conn(:get, "/channels/#{channel.id}/trace") - |> send_to_endpoint() - - assert resp.status == 200 - - [response_id] = Plug.Conn.get_resp_header(resp, "x-request-id") - assert_receive {:destination_request_id, ^response_id} - end - - test "response headers from destination forwarded to client", %{ - conn: conn, - bypass: bypass, - channel: channel - } do - Bypass.expect_once(bypass, "GET", "/test", fn conn -> - conn - |> Plug.Conn.put_resp_header("x-custom-header", "custom-value") - |> Plug.Conn.send_resp(200, "ok") - end) - - resp = get(conn, "/channels/#{channel.id}/test") - - assert resp.status == 200 - - assert Plug.Conn.get_resp_header(resp, "x-custom-header") == [ - "custom-value" - ] - end - - test "host header sent to destination matches upstream URL, not client's original host", - %{bypass: bypass, channel: channel} do - test_pid = self() - - Bypass.expect_once(bypass, "GET", "/host-check", fn conn -> - [received_host] = Plug.Conn.get_req_header(conn, "host") - send(test_pid, {:destination_host, received_host}) - Plug.Conn.send_resp(conn, 200, "ok") - end) - - # Simulate a real HTTP request with an attacker-controlled host header. - # In production, Cowboy always includes the client's Host header in - # req_headers; Plug.Test.conn does not, so we inject it manually. - base_conn = conn(:get, "/channels/#{channel.id}/host-check") - - resp = - %{ - base_conn - | host: "evil.example.com", - req_headers: [ - {"host", "evil.example.com"} | base_conn.req_headers - ] - } - |> send_to_endpoint() - - assert resp.status == 200 - - assert_receive {:destination_host, received_host} - assert received_host == "localhost:#{bypass.port}" - end - - test "strips the caller's session cookie before proxying to the destination", - %{bypass: bypass, channel: channel} do - test_pid = self() - - Bypass.expect_once(bypass, "GET", "/cookie-check", fn conn -> - send( - test_pid, - {:destination_cookie, Plug.Conn.get_req_header(conn, "cookie")} - ) - - Plug.Conn.send_resp(conn, 200, "ok") - end) - - base_conn = conn(:get, "/channels/#{channel.id}/cookie-check") - - resp = - %{ - base_conn - | req_headers: [ - {"cookie", "_lightning_key=victim-session-token"} - | base_conn.req_headers - ] - } - |> send_to_endpoint() - - assert resp.status == 200 - assert_receive {:destination_cookie, []} - end - end - - describe "response security headers" do - @security_headers [ - {"content-security-policy", - "default-src 'none'; sandbox; frame-ancestors 'none'"}, - {"x-content-type-options", "nosniff"}, - {"x-frame-options", "DENY"}, - {"referrer-policy", "no-referrer"} - ] - - test "set on a proxied response", %{ - conn: conn, - bypass: bypass, - channel: channel - } do - Bypass.expect_once(bypass, "GET", "/test", fn conn -> - Plug.Conn.send_resp(conn, 200, "ok") - end) - - resp = get(conn, "/channels/#{channel.id}/test") - - assert resp.status == 200 - assert_security_headers(resp) - end - - test "destination cannot override them", %{ - conn: conn, - bypass: bypass, - channel: channel - } do - Bypass.expect_once(bypass, "GET", "/bad", fn conn -> - conn - |> Plug.Conn.put_resp_header("content-security-policy", "default-src *") - |> Plug.Conn.put_resp_header("x-frame-options", "ALLOWALL") - |> Plug.Conn.put_resp_header("x-content-type-options", "") - |> Plug.Conn.put_resp_header("referrer-policy", "unsafe-url") - |> Plug.Conn.put_resp_content_type("text/html") - |> Plug.Conn.send_resp(200, "") - end) - - resp = get(conn, "/channels/#{channel.id}/bad") - - assert resp.status == 200 - assert_security_headers(resp) - end - - test "set-cookie from the destination is not forwarded", %{ - conn: conn, - bypass: bypass, - channel: channel - } do - Bypass.expect_once(bypass, "GET", "/cookie", fn conn -> - conn - |> Plug.Conn.put_resp_header( - "set-cookie", - "_lightning_key=forged; Path=/" - ) - |> Plug.Conn.send_resp(200, "ok") - end) - - resp = get(conn, "/channels/#{channel.id}/cookie") - - assert resp.status == 200 - assert Plug.Conn.get_resp_header(resp, "set-cookie") == [] - end - - test "set on the unknown-channel response", %{conn: conn} do - resp = get(conn, "/channels/#{Ecto.UUID.generate()}/test") - - assert resp.status == 404 - assert_security_headers(resp) - end - - test "set on the unauthorized response", %{conn: conn, bypass: bypass} do - project = insert(:project) - - channel = - insert(:channel, - project: project, - destination_url: "http://localhost:#{bypass.port}", - enabled: true, - channel_auth_methods: [ - build(:channel_auth_method, - role: :client, - webhook_auth_method: - build(:webhook_auth_method, - project: project, - auth_type: :api, - api_key: "valid-api-key" - ) - ) - ] - ) - - resp = get(conn, "/channels/#{channel.id}/test") - - assert resp.status == 401 - assert_security_headers(resp) - end - end - - describe "error cases" do - test "disabled channel returns 404", %{ - conn: conn, - disabled_channel: disabled_channel - } do - resp = get(conn, "/channels/#{disabled_channel.id}/test") - - assert resp.status == 404 - assert %{"error" => "Not Found"} = json_response(resp, 404) - end - - test "non-existent channel returns 404", %{conn: conn} do - resp = get(conn, "/channels/#{Ecto.UUID.generate()}/test") - - assert %{"error" => "Not Found"} = json_response(resp, 404) - end - - test "invalid UUID returns 404", %{conn: conn} do - resp = get(conn, "/channels/not-a-uuid/test") - - assert %{"error" => "Not Found"} = json_response(resp, 404) - end - - test "destination timeout returns 502", %{ - conn: conn, - channel: channel - } do - # Channel's destination_url points to a port with nothing running - port = Enum.random(59_000..59_999) - - channel - |> Ecto.Changeset.change(destination_url: "http://localhost:#{port}") - |> Lightning.Repo.update!() - - resp = get(conn, "/channels/#{channel.id}/test") - - assert resp.status in [502, 504] - end - end - - describe "SSRF egress guard" do - test "refuses to proxy to the cloud metadata IP (AWS IMDS / GCP metadata)", - %{conn: conn} do - project = insert(:project) - - channel = - insert(:channel, - project: project, - destination_url: "http://169.254.169.254/latest/meta-data/", - enabled: true - ) - - resp = get(conn, "/channels/#{channel.id}/latest/meta-data/") - - assert resp.status == 403 - refute resp.resp_body =~ "169.254.169.254" - end - - test "refuses to proxy to an RFC1918 private-network address", %{conn: conn} do - project = insert(:project) - - channel = - insert(:channel, - project: project, - destination_url: "http://10.0.0.1", - enabled: true - ) - - resp = get(conn, "/channels/#{channel.id}/admin") - - assert resp.status == 403 - refute resp.resp_body =~ "10.0.0.1" - end - - test "refuses to proxy to a private-range address not on the allow-list", %{ - conn: conn - } do - # A second RFC1918 address to lock in the general private-range case: - # with blocking on (default) and no allow-list entry, the block applies - # before any socket is opened. - project = insert(:project) - - channel = - insert(:channel, - project: project, - destination_url: "http://10.0.0.10", - enabled: true - ) - - resp = get(conn, "/channels/#{channel.id}/admin") - - assert resp.status == 403 - refute resp.resp_body =~ "10.0.0.10" - end - end - - describe "non-UTF-8 body handling (issue #4541)" do - # `response_body_preview` and `request_body_preview` are stored as :text - # (UTF-8 only). When an upstream returns binary content (gzip, image, PDF, - # ...) the bytes can't be persisted as text. Rather than failing the whole - # insert, we drop the offending preview to nil and persist everything else - # — headers, hash, size, timing. - - test "non-UTF-8 response body is dropped from preview; rest of event persisted", - %{bypass: bypass, channel: channel} do - # `0x8b` is the second byte of the gzip magic number — exactly the byte - # Postgres rejected in the dev reproduction. We send it raw so Finch - # cannot transparently decompress it. - raw_bytes = <<0x1F, 0x8B, 0x08, 0x00, 0xFF, 0xFE>> - - Bypass.expect_once(bypass, "GET", "/binary", fn conn -> - conn - |> Plug.Conn.put_resp_header("content-type", "application/octet-stream") - |> Plug.Conn.send_resp(200, raw_bytes) - end) - - resp = - conn(:get, "/channels/#{channel.id}/binary") - |> send_to_endpoint() - - assert resp.status == 200 - - request = - Lightning.Repo.one!( - from(r in ChannelRequest, where: r.channel_id == ^channel.id) - ) - - # The request itself succeeded — only the body preview is unstorable. - assert request.state == :success - assert request.completed_at != nil - - event = - Lightning.Repo.one!( - from(e in ChannelEvent, where: e.channel_request_id == ^request.id) - ) - - assert event.type == :destination_response - assert event.request_method == "GET" - assert event.request_path == "/binary" - assert event.response_status == 200 - - # Body preview dropped because it isn't valid UTF-8. - assert event.response_body_preview == nil - - # Hash and size are still recorded so the audit log can show that a body - # was returned, even though the bytes couldn't be persisted as text. - assert is_binary(event.response_body_hash) - assert event.response_body_size == byte_size(raw_bytes) - - # Headers and timing persist as usual. - assert is_list(event.request_headers) and event.request_headers != [] - assert is_list(event.response_headers) and event.response_headers != [] - assert is_integer(event.latency_us) and event.latency_us > 0 - end - - test "non-UTF-8 request body is dropped from preview; rest of event persisted", - %{bypass: bypass, channel: channel} do - raw_bytes = <<0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A>> - - Bypass.expect_once(bypass, "POST", "/upload", fn conn -> - Plug.Conn.send_resp(conn, 201, "ok") - end) - - resp = - conn(:post, "/channels/#{channel.id}/upload", raw_bytes) - |> Plug.Conn.put_req_header("content-type", "application/octet-stream") - |> Plug.Conn.put_req_header("content-length", "#{byte_size(raw_bytes)}") - |> send_to_endpoint() - - assert resp.status == 201 - - request = - Lightning.Repo.one!( - from(r in ChannelRequest, where: r.channel_id == ^channel.id) - ) - - assert request.state == :success - - event = - Lightning.Repo.one!( - from(e in ChannelEvent, where: e.channel_request_id == ^request.id) - ) - - assert event.request_method == "POST" - assert event.request_body_preview == nil - assert is_binary(event.request_body_hash) - assert event.request_body_size == byte_size(raw_bytes) - end - - test "valid UTF-8 body is preserved unchanged", - %{bypass: bypass, channel: channel} do - Bypass.expect_once(bypass, "GET", "/json", fn conn -> - conn - |> Plug.Conn.put_resp_header("content-type", "application/json") - |> Plug.Conn.send_resp(200, ~s({"hello":"world"})) - end) - - resp = - conn(:get, "/channels/#{channel.id}/json") - |> send_to_endpoint() - - assert resp.status == 200 - - event = - Lightning.Repo.one!( - from(e in ChannelEvent, - join: r in ChannelRequest, - on: r.id == e.channel_request_id, - where: r.channel_id == ^channel.id - ) - ) - - assert event.response_body_preview == ~s({"hello":"world"}) - end - end - - describe "handler persistence" do - test "creates ChannelRequest and ChannelEvent on successful proxy", %{ - conn: conn, - bypass: bypass, - channel: channel - } do - Bypass.expect_once(bypass, "GET", "/persisted", fn conn -> - Plug.Conn.send_resp(conn, 200, "persisted response") - end) - - resp = get(conn, "/channels/#{channel.id}/persisted") - assert resp.status == 200 - - request = - Lightning.Repo.one!( - from(r in ChannelRequest, where: r.channel_id == ^channel.id) - ) - - assert request.state == :success - assert request.completed_at != nil - - event = - Lightning.Repo.one!( - from(e in ChannelEvent, where: e.channel_request_id == ^request.id) - ) - - assert event.type == :destination_response - assert event.response_status == 200 - assert event.latency_us != nil - assert event.request_method == "GET" - assert event.request_path == "/persisted" - end - - test "creates error-state request on connection failure", %{ - conn: conn, - channel: channel - } do - port = Enum.random(59_000..59_999) - - channel - |> Ecto.Changeset.change(destination_url: "http://localhost:#{port}") - |> Lightning.Repo.update!() - - resp = get(conn, "/channels/#{channel.id}/fail") - assert resp.status in [502, 504] - - request = - Lightning.Repo.one!( - from(r in ChannelRequest, where: r.channel_id == ^channel.id) - ) - - assert request.state in [:error, :timeout] - assert request.completed_at != nil - end - end - - describe "client authentication" do - defp create_client_auth_channel(bypass, auth_method_specs) do - project = insert(:project) - - channel_auth_methods = - Enum.map(auth_method_specs, fn spec -> - build(:channel_auth_method, - role: :client, - webhook_auth_method: - build( - :webhook_auth_method, - [project: project] ++ Enum.to_list(spec) - ) - ) - end) - - insert(:channel, - project: project, - destination_url: "http://localhost:#{bypass.port}", - enabled: true, - channel_auth_methods: channel_auth_methods - ) - end - - test "no auth methods configured — request passes through", %{ - conn: conn, - bypass: bypass, - channel: channel - } do - # Channel has no channel_auth_methods, so it's publicly accessible - Bypass.expect_once(bypass, "GET", "/open", fn conn -> - Plug.Conn.send_resp(conn, 200, "public") - end) - - resp = get(conn, "/channels/#{channel.id}/open") - - assert resp.status == 200 - assert resp.resp_body == "public" - end - - test "API key auth — correct key allows request", %{ - bypass: bypass - } do - channel = - create_client_auth_channel(bypass, [ - %{auth_type: :api, api_key: "valid-api-key"} - ]) - - Bypass.expect_once(bypass, "GET", "/protected", fn conn -> - Plug.Conn.send_resp(conn, 200, "authenticated") - end) - - resp = - conn(:get, "/channels/#{channel.id}/protected") - |> put_req_header("x-api-key", "valid-api-key") - |> send_to_endpoint() - - assert resp.status == 200 - assert resp.resp_body == "authenticated" - end - - test "API key auth — wrong key returns 401", %{bypass: bypass} do - channel = - create_client_auth_channel(bypass, [ - %{auth_type: :api, api_key: "correct-key"} - ]) - - resp = - conn(:get, "/channels/#{channel.id}/protected") - |> put_req_header("x-api-key", "wrong-key") - |> send_to_endpoint() - - assert resp.status == 401 - assert %{"error" => "Unauthorized"} = json_response(resp, 401) - end - - test "API key auth — no key sent returns 401", %{bypass: bypass} do - channel = - create_client_auth_channel(bypass, [ - %{auth_type: :api, api_key: "some-key"} - ]) - - resp = - conn(:get, "/channels/#{channel.id}/protected") - |> send_to_endpoint() - - assert resp.status == 401 - assert %{"error" => "Unauthorized"} = json_response(resp, 401) - end - - test "Basic auth — correct credentials allows request", %{bypass: bypass} do - channel = - create_client_auth_channel(bypass, [ - %{auth_type: :basic, username: "admin", password: "secret"} - ]) - - Bypass.expect_once(bypass, "GET", "/basic", fn conn -> - Plug.Conn.send_resp(conn, 200, "basic-ok") - end) - - encoded = Base.encode64("admin:secret") - - resp = - conn(:get, "/channels/#{channel.id}/basic") - |> put_req_header("authorization", "Basic #{encoded}") - |> send_to_endpoint() - - assert resp.status == 200 - assert resp.resp_body == "basic-ok" - end - - test "Basic auth — wrong credentials returns 401", %{bypass: bypass} do - channel = - create_client_auth_channel(bypass, [ - %{auth_type: :basic, username: "admin", password: "secret"} - ]) - - encoded = Base.encode64("admin:wrong") - - resp = - conn(:get, "/channels/#{channel.id}/basic") - |> put_req_header("authorization", "Basic #{encoded}") - |> send_to_endpoint() - - assert resp.status == 401 - assert %{"error" => "Unauthorized"} = json_response(resp, 401) - end - - test "Basic auth — no auth header returns 401", %{bypass: bypass} do - channel = - create_client_auth_channel(bypass, [ - %{auth_type: :basic, username: "admin", password: "secret"} - ]) - - resp = - conn(:get, "/channels/#{channel.id}/basic") - |> send_to_endpoint() - - assert resp.status == 401 - assert %{"error" => "Unauthorized"} = json_response(resp, 401) - end - - test "multiple auth methods — either matches allows request", %{ - bypass: bypass - } do - channel = - create_client_auth_channel(bypass, [ - %{auth_type: :api, api_key: "key-one"}, - %{auth_type: :api, api_key: "key-two"} - ]) - - Bypass.expect_once(bypass, "GET", "/multi", fn conn -> - Plug.Conn.send_resp(conn, 200, "multi-ok") - end) - - # Use the second key — should still match - resp = - conn(:get, "/channels/#{channel.id}/multi") - |> put_req_header("x-api-key", "key-two") - |> send_to_endpoint() - - assert resp.status == 200 - assert resp.resp_body == "multi-ok" - end - - test "mixed types (API + Basic) — API key matches allows request", %{ - bypass: bypass - } do - channel = - create_client_auth_channel(bypass, [ - %{auth_type: :api, api_key: "mixed-key"}, - %{auth_type: :basic, username: "user", password: "pass"} - ]) - - Bypass.expect_once(bypass, "GET", "/mixed", fn conn -> - Plug.Conn.send_resp(conn, 200, "mixed-ok") - end) - - resp = - conn(:get, "/channels/#{channel.id}/mixed") - |> put_req_header("x-api-key", "mixed-key") - |> send_to_endpoint() - - assert resp.status == 200 - assert resp.resp_body == "mixed-ok" - end - - test "revoked API key (scheduled for deletion) returns 401", - %{bypass: bypass} do - channel = - create_client_auth_channel(bypass, [ - %{auth_type: :api, api_key: "live-key"}, - %{auth_type: :api, api_key: "revoked-key"} - ]) - - revoke_client_auth_method(channel, "revoked-key") - - # A stub, not an expectation: if the revoked credential is accepted the - # request reaches the destination and this responds 200. - Bypass.stub(bypass, "GET", "/protected", fn conn -> - Plug.Conn.send_resp(conn, 200, "proxied-with-revoked-credential") - end) - - resp = - conn(:get, "/channels/#{channel.id}/protected") - |> put_req_header("x-api-key", "revoked-key") - |> send_to_endpoint() - - assert resp.status == 401 - assert %{"error" => "Unauthorized"} = json_response(resp, 401) - end - - test "revoked basic credentials (scheduled for deletion) return 401", - %{bypass: bypass} do - channel = - create_client_auth_channel(bypass, [ - %{auth_type: :api, api_key: "live-key"}, - %{auth_type: :basic, username: "partner", password: "revoked-pass"} - ]) - - revoke_client_auth_method(channel, fn wam -> wam.auth_type == :basic end) - - # A stub, not an expectation: if the revoked credential is accepted the - # request reaches the destination and this responds 200. - Bypass.stub(bypass, "GET", "/protected", fn conn -> - Plug.Conn.send_resp(conn, 200, "proxied-with-revoked-credential") - end) - - encoded = Base.encode64("partner:revoked-pass") - - resp = - conn(:get, "/channels/#{channel.id}/protected") - |> put_req_header("authorization", "Basic #{encoded}") - |> send_to_endpoint() - - assert resp.status == 401 - assert %{"error" => "Unauthorized"} = json_response(resp, 401) - end - - test "the remaining live key still authenticates after another is revoked", - %{bypass: bypass} do - channel = - create_client_auth_channel(bypass, [ - %{auth_type: :api, api_key: "live-key"}, - %{auth_type: :api, api_key: "revoked-key"} - ]) - - revoke_client_auth_method(channel, "revoked-key") - - Bypass.expect_once(bypass, "GET", "/protected", fn conn -> - Plug.Conn.send_resp(conn, 200, "authenticated") - end) - - resp = - conn(:get, "/channels/#{channel.id}/protected") - |> put_req_header("x-api-key", "live-key") - |> send_to_endpoint() - - assert resp.status == 200 - assert resp.resp_body == "authenticated" - end - - # Severing the join rows at delete time revokes the credential immediately, - # but a channel left with no client method authenticates nobody and keeps - # forwarding with its destination credential. That is #387, not a - # regression introduced here; this test pins the behaviour so the gap is - # visible in the suite rather than implied. Delete it when #387 lands. - test "revoking a channel's only client method opens it to anonymous callers", - %{bypass: bypass} do - project = insert(:project) - user = insert(:user) - - credential = - insert(:credential, - schema: "@openfn/language-http", - name: "destination-cred", - user: user - ) - |> with_body(%{body: %{"access_token" => "dest-token-xyz"}}) - - project_credential = - insert(:project_credential, project: project, credential: credential) - - channel = - insert(:channel, - project: project, - destination_url: "http://localhost:#{bypass.port}", - enabled: true, - channel_auth_methods: [ - build(:channel_auth_method, - role: :client, - webhook_auth_method: - build(:webhook_auth_method, - project: project, - auth_type: :api, - api_key: "only-key" - ) - ), - build(:channel_auth_method, - role: :destination, - webhook_auth_method: nil, - project_credential: project_credential - ) - ] - ) - - revoke_client_auth_method(channel, "only-key") - - Bypass.expect_once(bypass, "GET", "/protected", fn conn -> - assert Plug.Conn.get_req_header(conn, "authorization") == [ - "Bearer dest-token-xyz" - ], - "the destination credential is still spent on an unauthenticated caller" - - Plug.Conn.send_resp(conn, 200, "anonymous") - end) - - resp = - conn(:get, "/channels/#{channel.id}/protected") - |> send_to_endpoint() - - assert resp.status == 200 - assert resp.resp_body == "anonymous" - end - - defp revoke_client_auth_method(channel, api_key) when is_binary(api_key) do - revoke_client_auth_method(channel, fn wam -> wam.api_key == api_key end) - end - - defp revoke_client_auth_method(channel, matcher) when is_function(matcher) do - auth_method = - channel.channel_auth_methods - |> Enum.map(& &1.webhook_auth_method) - |> Enum.reject(&is_nil/1) - |> Enum.find(matcher) - - {:ok, _} = - Lightning.WebhookAuthMethods.schedule_for_deletion(auth_method, - actor: insert(:user) - ) - end - end - - describe "client auth header stripping" do - test "strips x-api-key header when client uses API key auth", %{ - bypass: bypass - } do - channel = - create_client_auth_channel(bypass, [ - %{auth_type: :api, api_key: "valid-api-key"} - ]) - - Bypass.expect_once(bypass, "GET", "/test", fn conn -> - api_key = Plug.Conn.get_req_header(conn, "x-api-key") - - assert api_key == [], - "x-api-key should not be forwarded to the destination" - - Plug.Conn.send_resp(conn, 200, "ok") - end) - - conn(:get, "/channels/#{channel.id}/test") - |> put_req_header("x-api-key", "valid-api-key") - |> send_to_endpoint() - end - - test "strips authorization header when client uses Basic auth", %{ - bypass: bypass - } do - channel = - create_client_auth_channel(bypass, [ - %{auth_type: :basic, username: "admin", password: "secretpw"} - ]) - - encoded = Base.encode64("admin:secretpw") - - Bypass.expect_once(bypass, "GET", "/test", fn conn -> - auth = Plug.Conn.get_req_header(conn, "authorization") - - assert auth == [], - "client Basic auth should not be forwarded to the destination" - - Plug.Conn.send_resp(conn, 200, "ok") - end) - - conn(:get, "/channels/#{channel.id}/test") - |> put_req_header("authorization", "Basic #{encoded}") - |> send_to_endpoint() - end - - test "replaces client Basic auth with destination Bearer auth", %{ - bypass: bypass - } do - project = insert(:project) - user = insert(:user) - - credential = - insert(:credential, - schema: "@openfn/language-http", - name: "destination-cred", - user: user - ) - |> with_body(%{body: %{"access_token" => "dest-token-xyz"}}) - - project_credential = - insert(:project_credential, - project: project, - credential: credential - ) - - client_encoded = Base.encode64("user:password") - - channel = - insert(:channel, - project: project, - destination_url: "http://localhost:#{bypass.port}", - enabled: true, - channel_auth_methods: [ - build(:channel_auth_method, - role: :client, - webhook_auth_method: - build(:webhook_auth_method, - project: project, - auth_type: :basic, - username: "user", - password: "password" - ) - ), - build(:channel_auth_method, - role: :destination, - webhook_auth_method: nil, - project_credential: project_credential - ) - ] - ) - - Bypass.expect_once(bypass, "GET", "/test", fn conn -> - auth = Plug.Conn.get_req_header(conn, "authorization") - - assert auth == ["Bearer dest-token-xyz"], - "destination should receive the destination Bearer token, not the client Basic auth" - - Plug.Conn.send_resp(conn, 200, "ok") - end) - - conn(:get, "/channels/#{channel.id}/test") - |> put_req_header("authorization", "Basic #{client_encoded}") - |> send_to_endpoint() - end - - test "passes through client auth headers when no client auth configured", - %{bypass: bypass, channel: channel} do - Bypass.expect_once(bypass, "GET", "/test", fn conn -> - auth = Plug.Conn.get_req_header(conn, "authorization") - - assert auth == ["Bearer client-token"], - "client auth headers should pass through when no client auth is configured" - - Plug.Conn.send_resp(conn, 200, "ok") - end) - - conn(:get, "/channels/#{channel.id}/test") - |> put_req_header("authorization", "Bearer client-token") - |> send_to_endpoint() - end - end - - describe "destination authentication" do - defp create_destination_auth_channel(bypass, schema, body) do - project = insert(:project) - user = insert(:user) - - credential = - insert(:credential, schema: schema, name: "destination-cred", user: user) - |> with_body(%{body: body}) - - project_credential = - insert(:project_credential, - project: project, - credential: credential - ) - - insert(:channel, - project: project, - destination_url: "http://localhost:#{bypass.port}", - enabled: true, - channel_auth_methods: [ - build(:channel_auth_method, - role: :destination, - webhook_auth_method: nil, - project_credential: project_credential - ) - ] - ) - end - - test "Bearer token sent to upstream when channel has http credential with access_token", - %{bypass: bypass} do - channel = - create_destination_auth_channel(bypass, "@openfn/language-http", %{ - "access_token" => "tok-123" - }) - - Bypass.expect_once(bypass, "GET", "/test", fn conn -> - auth = Plug.Conn.get_req_header(conn, "authorization") - assert auth == ["Bearer tok-123"] - Plug.Conn.send_resp(conn, 200, "ok") - end) - - resp = - conn(:get, "/channels/#{channel.id}/test") - |> send_to_endpoint() - - assert resp.status == 200 - end - - test "Basic auth sent when channel has http credential with username/password", - %{bypass: bypass} do - channel = - create_destination_auth_channel(bypass, "@openfn/language-http", %{ - "username" => "u", - "password" => "p" - }) - - expected = "Basic #{Base.encode64("u:p")}" - - Bypass.expect_once(bypass, "GET", "/test", fn conn -> - auth = Plug.Conn.get_req_header(conn, "authorization") - assert auth == [expected] - Plug.Conn.send_resp(conn, 200, "ok") - end) - - resp = - conn(:get, "/channels/#{channel.id}/test") - |> send_to_endpoint() - - assert resp.status == 200 - end - - test "ApiToken sent when channel has dhis2 credential with pat", - %{bypass: bypass} do - channel = - create_destination_auth_channel(bypass, "@openfn/language-dhis2", %{ - "pat" => "d2pat_abc" - }) - - Bypass.expect_once(bypass, "GET", "/test", fn conn -> - auth = Plug.Conn.get_req_header(conn, "authorization") - assert auth == ["ApiToken d2pat_abc"] - Plug.Conn.send_resp(conn, 200, "ok") - end) - - resp = - conn(:get, "/channels/#{channel.id}/test") - |> send_to_endpoint() - - assert resp.status == 200 - end - - test "no authorization header when channel has no destination auth method", - %{bypass: bypass, channel: channel} do - Bypass.expect_once(bypass, "GET", "/test", fn conn -> - auth = Plug.Conn.get_req_header(conn, "authorization") - assert auth == [] - Plug.Conn.send_resp(conn, 200, "ok") - end) - - resp = - conn(:get, "/channels/#{channel.id}/test") - |> send_to_endpoint() - - assert resp.status == 200 - end - - test "authorization header redacted in persisted ChannelEvent", - %{bypass: bypass} do - channel = - create_destination_auth_channel(bypass, "@openfn/language-http", %{ - "access_token" => "secret-token" - }) - - Bypass.expect_once(bypass, "GET", "/redact-test", fn conn -> - Plug.Conn.send_resp(conn, 200, "ok") - end) - - resp = - conn(:get, "/channels/#{channel.id}/redact-test") - |> send_to_endpoint() - - assert resp.status == 200 - - event = - Lightning.Repo.one!( - from(e in ChannelEvent, - join: r in ChannelRequest, - on: r.id == e.channel_request_id, - where: r.channel_id == ^channel.id - ) - ) - - # The handler redacts authorization headers before persisting - # Headers are native jsonb arrays, no JSON decoding needed - auth_header = - Enum.find(event.request_headers, fn [k, _v] -> k == "authorization" end) - - assert auth_header == ["authorization", "[REDACTED]"] - end - - test "credential environment_not_found returns 502 with observable error", - %{bypass: bypass} do - # Create a credential with NO credential_body for "main" environment - project = insert(:project) - user = insert(:user) - - credential = - insert(:credential, - schema: "@openfn/language-http", - name: "no-body", - user: user - ) - - # Don't call with_body — no CredentialBody exists - - project_credential = - insert(:project_credential, - project: project, - credential: credential - ) - - channel = - insert(:channel, - project: project, - destination_url: "http://localhost:#{bypass.port}", - enabled: true, - channel_auth_methods: [ - build(:channel_auth_method, - role: :destination, - webhook_auth_method: nil, - project_credential: project_credential - ) - ] - ) - - resp = - conn(:get, "/channels/#{channel.id}/test") - |> send_to_endpoint() - - assert resp.status == 502 - assert %{"error" => "Bad Gateway"} = json_response(resp, 502) - - request = - Lightning.Repo.one!( - from(r in ChannelRequest, where: r.channel_id == ^channel.id) - ) - - assert request.state == :error - - event = - Lightning.Repo.one!( - from(e in ChannelEvent, where: e.channel_request_id == ^request.id) - ) - - assert event.error_message == "credential_environment_not_found" - end - - test "a sandbox authenticates with its own environment, not the parent's", - %{bypass: bypass} do - # The bug this PR fixes: the proxy asked for "main" whatever project it - # was acting for, so a sandbox reached its destination holding the - # parent's production secret. Reverting the fix has to fail here. - root = insert(:project) - sandbox = insert(:project, parent: root, env: "staging") - user = insert(:user) - - credential = - insert(:credential, - schema: "@openfn/language-http", - name: "shared", - user: user - ) - |> with_body(%{ - name: "main", - body: %{"username" => "prod", "password" => "prod-secret"} - }) - |> with_body(%{ - name: "staging", - body: %{"username" => "sbx", "password" => "sbx-secret"} - }) - - project_credential = - insert(:project_credential, project: sandbox, credential: credential) - - channel = - insert(:channel, - project: sandbox, - destination_url: "http://localhost:#{bypass.port}", - enabled: true, - channel_auth_methods: [ - build(:channel_auth_method, - role: :destination, - webhook_auth_method: nil, - project_credential: project_credential - ) - ] - ) - - sandbox_auth = "Basic #{Base.encode64("sbx:sbx-secret")}" - parent_auth = "Basic #{Base.encode64("prod:prod-secret")}" - - Bypass.expect_once(bypass, "GET", "/test", fn conn -> - auth = Plug.Conn.get_req_header(conn, "authorization") - assert auth == [sandbox_auth] - refute auth == [parent_auth] - Plug.Conn.send_resp(conn, 200, "ok") - end) - - resp = - conn(:get, "/channels/#{channel.id}/test") - |> send_to_endpoint() - - assert resp.status == 200 - end - - test "a sandbox with no environment set returns 502 with observable error", - %{bypass: bypass} do - # Deriving the environment instead of assuming "main" gave this path two - # new failure reasons. They have to land as a recorded 502 like every - # other credential error, not as an unhandled crash. - root = insert(:project) - sandbox = insert(:project, parent: root, env: nil) - user = insert(:user) - - credential = - insert(:credential, - schema: "@openfn/language-http", - name: "parent", - user: user - ) - - project_credential = - insert(:project_credential, project: sandbox, credential: credential) - - channel = - insert(:channel, - project: sandbox, - destination_url: "http://localhost:#{bypass.port}", - enabled: true, - channel_auth_methods: [ - build(:channel_auth_method, - role: :destination, - webhook_auth_method: nil, - project_credential: project_credential - ) - ] - ) - - resp = - conn(:get, "/channels/#{channel.id}/test") - |> send_to_endpoint() - - assert resp.status == 502 - assert %{"error" => "Bad Gateway"} = json_response(resp, 502) - - request = - Lightning.Repo.one!( - from(r in ChannelRequest, where: r.channel_id == ^channel.id) - ) - - assert request.state == :error - - event = - Lightning.Repo.one!( - from(e in ChannelEvent, where: e.channel_request_id == ^request.id) - ) - - assert event.error_message == "credential_environment_not_configured" - end - - test "credential with missing auth fields returns 502 with observable error", - %{bypass: bypass} do - channel = - create_destination_auth_channel(bypass, "@openfn/language-http", %{ - "baseUrl" => "https://example.com" - }) - - resp = - conn(:get, "/channels/#{channel.id}/test") - |> send_to_endpoint() - - assert resp.status == 502 - assert %{"error" => "Bad Gateway"} = json_response(resp, 502) - - request = - Lightning.Repo.one!( - from(r in ChannelRequest, where: r.channel_id == ^channel.id) - ) - - assert request.state == :error - - event = - Lightning.Repo.one!( - from(e in ChannelEvent, where: e.channel_request_id == ^request.id) - ) - - assert event.error_message == "credential_missing_auth_fields" - end - - test "proxy headers (x-forwarded-*) still forwarded alongside auth header", - %{bypass: bypass} do - channel = - create_destination_auth_channel(bypass, "@openfn/language-http", %{ - "access_token" => "tok-with-proxy" - }) - - Bypass.expect_once(bypass, "GET", "/test", fn conn -> - auth = Plug.Conn.get_req_header(conn, "authorization") - xff = Plug.Conn.get_req_header(conn, "x-forwarded-for") - xfh = Plug.Conn.get_req_header(conn, "x-forwarded-host") - xfp = Plug.Conn.get_req_header(conn, "x-forwarded-proto") - - assert auth == ["Bearer tok-with-proxy"] - assert xff != [] - assert xfh != [] - assert xfp != [] - - Plug.Conn.send_resp(conn, 200, "ok") - end) - - resp = - conn(:get, "/channels/#{channel.id}/test") - |> send_to_endpoint() - - assert resp.status == 200 - end - end - - # --------------------------------------------------------------- - # Phase 1a contract tests — query string + client auth tracking - # --------------------------------------------------------------- - # - # These tests define the target interface after: - # - D1: request_query_string on channel_events - # - D3: client_webhook_auth_method_id and client_auth_type on channel_requests - # - D4: Proxy plug passes query string and auth info into handler state - # - # They will not compile/pass until Phase 1b implements the changes. - - describe "query string persistence" do - test "persists query string on channel event", %{ - bypass: bypass, - channel: channel - } do - Bypass.expect_once(bypass, "GET", "/search", fn conn -> - Plug.Conn.send_resp(conn, 200, "results") - end) - - conn(:get, "/channels/#{channel.id}/search?q=foo&page=2") - |> send_to_endpoint() - - event = - Lightning.Repo.one!( - from(e in ChannelEvent, - join: r in ChannelRequest, - on: r.id == e.channel_request_id, - where: r.channel_id == ^channel.id - ) - ) - - assert event.request_query_string == "q=foo&page=2" - end - - test "empty query string when no params", %{ - bypass: bypass, - channel: channel - } do - Bypass.expect_once(bypass, "GET", "/plain", fn conn -> - Plug.Conn.send_resp(conn, 200, "ok") - end) - - conn(:get, "/channels/#{channel.id}/plain") - |> send_to_endpoint() - - event = - Lightning.Repo.one!( - from(e in ChannelEvent, - join: r in ChannelRequest, - on: r.id == e.channel_request_id, - where: r.channel_id == ^channel.id - ) - ) - - assert event.request_query_string == "" - end - end - - describe "client auth tracking" do - test "persists auth method ID and type for API key auth", %{bypass: bypass} do - channel = - create_client_auth_channel(bypass, [ - %{auth_type: :api, api_key: "track-me"} - ]) - - auth_method = - channel - |> Lightning.Repo.preload(client_webhook_auth_methods: []) - |> Map.get(:client_webhook_auth_methods) - |> hd() - - Bypass.expect_once(bypass, "GET", "/tracked", fn conn -> - Plug.Conn.send_resp(conn, 200, "ok") - end) - - conn(:get, "/channels/#{channel.id}/tracked") - |> put_req_header("x-api-key", "track-me") - |> send_to_endpoint() - - request = - Lightning.Repo.one!( - from(r in ChannelRequest, where: r.channel_id == ^channel.id) - ) - - assert request.client_webhook_auth_method_id == auth_method.id - assert request.client_auth_type == "api" - end - - test "persists auth method ID and type for Basic auth", %{bypass: bypass} do - channel = - create_client_auth_channel(bypass, [ - %{auth_type: :basic, username: "user", password: "pass"} - ]) - - auth_method = - channel - |> Lightning.Repo.preload(client_webhook_auth_methods: []) - |> Map.get(:client_webhook_auth_methods) - |> hd() - - Bypass.expect_once(bypass, "GET", "/tracked", fn conn -> - Plug.Conn.send_resp(conn, 200, "ok") - end) - - encoded = Base.encode64("user:pass") - - conn(:get, "/channels/#{channel.id}/tracked") - |> put_req_header("authorization", "Basic #{encoded}") - |> send_to_endpoint() - - request = - Lightning.Repo.one!( - from(r in ChannelRequest, where: r.channel_id == ^channel.id) - ) - - assert request.client_webhook_auth_method_id == auth_method.id - assert request.client_auth_type == "basic" - end - - test "nil auth method when no client auth configured", %{ - bypass: bypass, - channel: channel - } do - Bypass.expect_once(bypass, "GET", "/open", fn conn -> - Plug.Conn.send_resp(conn, 200, "ok") - end) - - conn(:get, "/channels/#{channel.id}/open") - |> send_to_endpoint() - - request = - Lightning.Repo.one!( - from(r in ChannelRequest, where: r.channel_id == ^channel.id) - ) - - assert request.client_webhook_auth_method_id == nil - assert request.client_auth_type == nil - end - end - - describe "destination auth tracking" do - test "persists destination_credential_id on successful proxy with destination auth", - %{bypass: bypass} do - channel = - create_destination_auth_channel(bypass, "@openfn/language-http", %{ - "access_token" => "tok-123" - }) - - project_credential_id = - channel - |> Lightning.Repo.preload(destination_auth_method: :project_credential) - |> get_in([ - Access.key(:destination_auth_method), - Access.key(:project_credential_id) - ]) - - Bypass.expect_once(bypass, "GET", "/dest-track", fn conn -> - Plug.Conn.send_resp(conn, 200, "ok") - end) - - conn(:get, "/channels/#{channel.id}/dest-track") - |> send_to_endpoint() - - request = - Lightning.Repo.one!( - from(r in ChannelRequest, where: r.channel_id == ^channel.id) - ) - - assert request.destination_credential_id == project_credential_id - refute is_nil(project_credential_id) - end - - test "persists destination_credential_id even when credential resolution fails", - %{bypass: _bypass} do - # Channel with a destination auth method but credential missing auth - # fields — destination auth resolution fails, but we still know which - # credential was configured. - project = insert(:project) - user = insert(:user) - - credential = - insert(:credential, - schema: "@openfn/language-http", - name: "bad-cred", - user: user - ) - |> with_body(%{body: %{"baseUrl" => "https://example.com"}}) - - project_credential = - insert(:project_credential, project: project, credential: credential) - - channel = - insert(:channel, - project: project, - destination_url: "http://localhost:9999", - enabled: true, - channel_auth_methods: [ - build(:channel_auth_method, - role: :destination, - webhook_auth_method: nil, - project_credential: project_credential - ) - ] - ) - - resp = - conn(:get, "/channels/#{channel.id}/test") - |> send_to_endpoint() - - assert resp.status == 502 - - request = - Lightning.Repo.one!( - from(r in ChannelRequest, where: r.channel_id == ^channel.id) - ) - - assert request.destination_credential_id == project_credential.id - assert request.state == :error - end - - test "destination_credential_id is nil when no destination auth configured", - %{bypass: bypass, channel: channel} do - Bypass.expect_once(bypass, "GET", "/no-dest-auth", fn conn -> - Plug.Conn.send_resp(conn, 200, "ok") - end) - - conn(:get, "/channels/#{channel.id}/no-dest-auth") - |> send_to_endpoint() - - request = - Lightning.Repo.one!( - from(r in ChannelRequest, where: r.channel_id == ^channel.id) - ) - - assert request.destination_credential_id == nil - end - end - - describe "collect_timing integration" do - test "persists per-direction timing after successful proxy", %{ - bypass: bypass, - channel: channel - } do - Bypass.expect_once(bypass, "GET", "/timed", fn conn -> - Plug.Conn.send_resp(conn, 200, "ok") - end) - - conn(:get, "/channels/#{channel.id}/timed") - |> send_to_endpoint() - - event = - Lightning.Repo.one!( - from(e in ChannelEvent, - join: r in ChannelRequest, - on: r.id == e.channel_request_id, - where: r.channel_id == ^channel.id - ) - ) - - # With collect_timing: true, Philter populates timing.send_us - # which the handler persists as request_send_us - assert is_integer(event.request_send_us) - assert event.request_send_us >= 0 - end - end - - describe "zero-persistence scrubbing" do - test "happy path: scrubs PII fields and sets is_wiped: true under :erase_all", - %{bypass: bypass} do - project = insert(:project, retention_policy: :erase_all) - - channel = - insert(:channel, - project: project, - destination_url: "http://localhost:#{bypass.port}", - enabled: true - ) - - body = Jason.encode!(%{"hello" => "world"}) - - Bypass.expect_once(bypass, "POST", "/erase/path", fn conn -> - conn - |> Plug.Conn.put_resp_header("x-custom-header", "value") - |> Plug.Conn.put_resp_header("content-type", "application/json") - |> Plug.Conn.send_resp(200, ~s({"ok":true})) - end) - - resp = - conn(:post, "/channels/#{channel.id}/erase/path?secret=abc", body) - |> put_req_header("content-type", "application/json") - |> put_req_header("content-length", "#{byte_size(body)}") - |> put_req_header("x-api-key", "some-key") - |> send_to_endpoint() - - assert resp.status == 200 - - request = - Lightning.Repo.one!( - from(r in ChannelRequest, where: r.channel_id == ^channel.id) - ) - - assert %ChannelRequest{ - state: :success, - client_identity: nil, - is_wiped: true - } = request - - assert request.started_at != nil - assert request.completed_at != nil - assert is_binary(request.request_id) - - event = - Lightning.Repo.one!( - from(e in ChannelEvent, where: e.channel_request_id == ^request.id) - ) - - # All eight scrubbed fields are nil on the event; is_wiped lives on the request. - assert %ChannelEvent{ - request_path: nil, - request_query_string: nil, - request_headers: nil, - request_body_preview: nil, - request_body_hash: nil, - response_headers: nil, - response_body_preview: nil, - response_body_hash: nil - } = event - - # Observability fields remain populated. - assert event.request_method == "POST" - assert event.response_status == 200 - assert is_integer(event.latency_us) and event.latency_us > 0 - assert event.request_body_size == byte_size(body) - - assert is_integer(event.response_body_size) and - event.response_body_size > 0 - end - - test "happy path: retains all PII fields and sets is_wiped: false under :retain_all", - %{bypass: bypass} do - project = insert(:project, retention_policy: :retain_all) - - channel = - insert(:channel, - project: project, - destination_url: "http://localhost:#{bypass.port}", - enabled: true - ) - - body = Jason.encode!(%{"hello" => "world"}) - - Bypass.expect_once(bypass, "POST", "/retain/path", fn conn -> - conn - |> Plug.Conn.put_resp_header("x-custom-header", "value") - |> Plug.Conn.put_resp_header("content-type", "application/json") - |> Plug.Conn.send_resp(200, ~s({"ok":true})) - end) - - resp = - conn(:post, "/channels/#{channel.id}/retain/path?secret=abc", body) - |> put_req_header("content-type", "application/json") - |> put_req_header("content-length", "#{byte_size(body)}") - |> put_req_header("x-api-key", "some-key") - |> send_to_endpoint() - - assert resp.status == 200 - - request = - Lightning.Repo.one!( - from(r in ChannelRequest, where: r.channel_id == ^channel.id) - ) - - assert request.state == :success - assert request.completed_at != nil - assert request.is_wiped == false - refute is_nil(request.client_identity) - - event = - Lightning.Repo.one!( - from(e in ChannelEvent, where: e.channel_request_id == ^request.id) - ) - - assert event.request_method == "POST" - assert event.request_path == "/retain/path" - assert event.request_query_string == "secret=abc" - assert event.response_status == 200 - assert is_list(event.request_headers) and event.request_headers != [] - assert is_list(event.response_headers) and event.response_headers != [] - assert is_binary(event.request_body_preview) - assert is_binary(event.request_body_hash) - assert is_binary(event.response_body_preview) - assert is_binary(event.response_body_hash) - assert is_integer(event.latency_us) and event.latency_us > 0 - assert event.request_body_size == byte_size(body) - end - - test "credential-error path: scrubs request_path and client_identity, sets is_wiped: true, but keeps request_method and error_message", - %{bypass: bypass} do - project = insert(:project, retention_policy: :erase_all) - user = insert(:user) - - credential = - insert(:credential, - schema: "@openfn/language-http", - name: "no-body", - user: user - ) - - # Don't call with_body — no CredentialBody exists, so credential - # resolution will fail and `record_credential_error/3` is invoked. - - project_credential = - insert(:project_credential, - project: project, - credential: credential - ) - - channel = - insert(:channel, - project: project, - destination_url: "http://localhost:#{bypass.port}", - enabled: true, - channel_auth_methods: [ - build(:channel_auth_method, - role: :destination, - webhook_auth_method: nil, - project_credential: project_credential - ) - ] - ) - - resp = - conn(:get, "/channels/#{channel.id}/secret-path") - |> send_to_endpoint() - - assert resp.status == 502 - - request = - Lightning.Repo.one!( - from(r in ChannelRequest, where: r.channel_id == ^channel.id) - ) - - assert %ChannelRequest{ - state: :error, - client_identity: nil, - is_wiped: true - } = request - - event = - Lightning.Repo.one!( - from(e in ChannelEvent, where: e.channel_request_id == ^request.id) - ) - - assert %ChannelEvent{ - type: :error, - request_path: nil, - request_method: "GET", - error_message: "credential_environment_not_found" - } = event - end - - test "credential-error path: leaves request_path populated and request.is_wiped: false under :retain_all (default)", - %{bypass: bypass} do - project = insert(:project, retention_policy: :retain_all) - user = insert(:user) - - credential = - insert(:credential, - schema: "@openfn/language-http", - name: "no-body", - user: user - ) - - project_credential = - insert(:project_credential, - project: project, - credential: credential - ) - - channel = - insert(:channel, - project: project, - destination_url: "http://localhost:#{bypass.port}", - enabled: true, - channel_auth_methods: [ - build(:channel_auth_method, - role: :destination, - webhook_auth_method: nil, - project_credential: project_credential - ) - ] - ) - - resp = - conn(:get, "/channels/#{channel.id}/keep-path") - |> send_to_endpoint() - - assert resp.status == 502 - - request = - Lightning.Repo.one!( - from(r in ChannelRequest, where: r.channel_id == ^channel.id) - ) - - assert request.state == :error - assert request.is_wiped == false - refute is_nil(request.client_identity) - - event = - Lightning.Repo.one!( - from(e in ChannelEvent, where: e.channel_request_id == ^request.id) - ) - - assert %ChannelEvent{ - type: :error, - request_path: "/channels/" <> _, - request_method: "GET", - error_message: "credential_environment_not_found" - } = event - end - end - - describe "request telemetry" do - setup do - test_pid = self() - handler_id = "channel-proxy-test-#{System.unique_integer([:positive])}" - - :ok = - :telemetry.attach_many( - handler_id, - [ - [:lightning, :channel_proxy, :inbound, :stop], - [:lightning, :channel_proxy, :request, :start], - [:lightning, :channel_proxy, :request, :stop] - ], - fn event, measurements, metadata, _config -> - send(test_pid, {:telemetry, event, measurements, metadata}) - end, - nil - ) - - on_exit(fn -> :telemetry.detach(handler_id) end) - - :ok - end - - test "resolved channel: outer :inbound :stop and inner :request :start/:stop fire with real UUIDs", - %{bypass: bypass, channel: channel} do - Bypass.expect_once(bypass, "GET", "/observed", fn conn -> - Plug.Conn.send_resp(conn, 200, "ok") - end) - - resp = - conn(:get, "/channels/#{channel.id}/observed") - |> send_to_endpoint() - - assert resp.status == 200 - - expected_channel_id = channel.id - expected_project_id = channel.project_id - - # Outer span carries outcome + resolved IDs (real UUIDs, never "unknown"). - assert_receive {:telemetry, [:lightning, :channel_proxy, :inbound, :stop], - _measurements, inbound_meta} - - assert %{ - outcome: :resolved, - status: 200, - channel_id: ^expected_channel_id, - project_id: ^expected_project_id - } = inbound_meta - - # Inner span fires only on the resolved path. :start metadata already - # carries the resolved project_id (the whole point of the split). - assert_receive {:telemetry, [:lightning, :channel_proxy, :request, :start], - _measurements, start_meta} - - assert %{ - channel_id: ^expected_channel_id, - project_id: ^expected_project_id - } = start_meta - - assert_receive {:telemetry, [:lightning, :channel_proxy, :request, :stop], - %{duration: duration}, stop_meta} - - assert is_integer(duration) and duration > 0 - - assert %{ - channel_id: ^expected_channel_id, - project_id: ^expected_project_id, - status: 200 - } = stop_meta - end - - test "unknown channel: outer :inbound :stop fires with :unknown_channel and no inner events", - %{conn: conn} do - missing_id = "00000000-0000-0000-0000-000000000000" - - resp = get(conn, "/channels/#{missing_id}/whatever") - - assert resp.status == 404 - - assert_receive {:telemetry, [:lightning, :channel_proxy, :inbound, :stop], - _measurements, inbound_meta} - - assert %{outcome: :unknown_channel, status: 404} = inbound_meta - refute Map.has_key?(inbound_meta, :channel_id) - refute Map.has_key?(inbound_meta, :project_id) - - # The inner :request span never opens for unknown channels. - refute_receive {:telemetry, [:lightning, :channel_proxy, :request, :start], - _, _} - - refute_receive {:telemetry, [:lightning, :channel_proxy, :request, :stop], - _, _} - end - - test "invalid UUID: outer :inbound :stop fires with :invalid_uuid and no inner events", - %{conn: conn} do - resp = get(conn, "/channels/not-a-uuid/whatever") - - assert resp.status == 404 - - assert_receive {:telemetry, [:lightning, :channel_proxy, :inbound, :stop], - _measurements, inbound_meta} - - assert %{outcome: :invalid_uuid, status: 404} = inbound_meta - refute Map.has_key?(inbound_meta, :channel_id) - refute Map.has_key?(inbound_meta, :project_id) - - refute_receive {:telemetry, [:lightning, :channel_proxy, :request, :start], - _, _} - - refute_receive {:telemetry, [:lightning, :channel_proxy, :request, :stop], - _, _} - end - - test "401 unauthorized: outer :inbound and inner :request both fire (auth lives inside inner span)", - %{bypass: bypass} do - project = insert(:project) - - channel = - insert(:channel, - project: project, - destination_url: "http://localhost:#{bypass.port}", - enabled: true, - channel_auth_methods: [ - build(:channel_auth_method, - role: :client, - webhook_auth_method: - build(:webhook_auth_method, - project: project, - auth_type: :api, - api_key: "correct-key" - ) - ) - ] - ) - - resp = - conn(:get, "/channels/#{channel.id}/protected") - |> put_req_header("x-api-key", "wrong-key") - |> send_to_endpoint() - - assert resp.status == 401 - - expected_channel_id = channel.id - expected_project_id = channel.project_id - - assert_receive {:telemetry, [:lightning, :channel_proxy, :inbound, :stop], - _measurements, inbound_meta} - - assert %{ - outcome: :resolved, - status: 401, - channel_id: ^expected_channel_id, - project_id: ^expected_project_id - } = inbound_meta - - assert_receive {:telemetry, [:lightning, :channel_proxy, :request, :start], - _measurements, start_meta} - - assert %{ - channel_id: ^expected_channel_id, - project_id: ^expected_project_id - } = start_meta - - assert_receive {:telemetry, [:lightning, :channel_proxy, :request, :stop], - _measurements, stop_meta} - - assert %{ - channel_id: ^expected_channel_id, - project_id: ^expected_project_id, - status: 401 - } = stop_meta - end - end - - defp assert_security_headers(resp) do - for {header, value} <- @security_headers do - assert Plug.Conn.get_resp_header(resp, header) == [value], - "expected #{header}: #{value}" - end - end - - defp send_to_endpoint(conn) do - LightningWeb.Endpoint.call(conn, LightningWeb.Endpoint.init([])) - end -end diff --git a/test/lightning_web/plugs/channel_proxy_refusals_test.exs b/test/lightning_web/plugs/channel_proxy_refusals_test.exs new file mode 100644 index 00000000000..62578273e17 --- /dev/null +++ b/test/lightning_web/plugs/channel_proxy_refusals_test.exs @@ -0,0 +1,353 @@ +defmodule LightningWeb.ChannelProxyRefusalsTest do + use LightningWeb.ConnCase, async: true + + alias Lightning.Channels.{ChannelRequest, ChannelEvent} + + import Ecto.Query + import Lightning.Factories + import LightningWeb.API.ProblemHelpers + import LightningWeb.API.ChannelProxyHelpers + import Plug.Test, only: [conn: 2, conn: 3] + + setup do + bypass = Bypass.open() + project = insert(:project) + + channel = + insert(:channel, + project: project, + destination_url: "http://localhost:#{bypass.port}", + enabled: true + ) + + disabled_channel = + insert(:channel, + project: project, + destination_url: "http://localhost:#{bypass.port}", + enabled: false + ) + + {:ok, bypass: bypass, channel: channel, disabled_channel: disabled_channel} + end + + describe "response security headers" do + test "set on a proxied response", %{ + conn: conn, + bypass: bypass, + channel: channel + } do + Bypass.expect_once(bypass, "GET", "/test", fn conn -> + Plug.Conn.send_resp(conn, 200, "ok") + end) + + resp = get(conn, "/channels/#{channel.id}/test") + + assert resp.status == 200 + assert_security_headers(resp) + end + + test "destination cannot override them", %{ + conn: conn, + bypass: bypass, + channel: channel + } do + Bypass.expect_once(bypass, "GET", "/bad", fn conn -> + conn + |> Plug.Conn.put_resp_header("content-security-policy", "default-src *") + |> Plug.Conn.put_resp_header("x-frame-options", "ALLOWALL") + |> Plug.Conn.put_resp_header("x-content-type-options", "") + |> Plug.Conn.put_resp_header("referrer-policy", "unsafe-url") + |> Plug.Conn.put_resp_content_type("text/html") + |> Plug.Conn.send_resp(200, "") + end) + + resp = get(conn, "/channels/#{channel.id}/bad") + + assert resp.status == 200 + assert_security_headers(resp) + end + + test "set-cookie from the destination is not forwarded", %{ + conn: conn, + bypass: bypass, + channel: channel + } do + Bypass.expect_once(bypass, "GET", "/cookie", fn conn -> + conn + |> Plug.Conn.put_resp_header( + "set-cookie", + "_lightning_key=forged; Path=/" + ) + |> Plug.Conn.send_resp(200, "ok") + end) + + resp = get(conn, "/channels/#{channel.id}/cookie") + + assert resp.status == 200 + assert Plug.Conn.get_resp_header(resp, "set-cookie") == [] + end + + test "set on the unknown-channel response", %{conn: conn} do + resp = get(conn, "/channels/#{Ecto.UUID.generate()}/test") + + assert resp.status == 404 + assert_security_headers(resp) + end + + test "set on the unauthorized response", %{conn: conn, bypass: bypass} do + project = insert(:project) + + channel = + insert(:channel, + project: project, + destination_url: "http://localhost:#{bypass.port}", + enabled: true, + channel_auth_methods: [ + build(:channel_auth_method, + role: :client, + webhook_auth_method: + build(:webhook_auth_method, + project: project, + auth_type: :api, + api_key: "valid-api-key" + ) + ) + ] + ) + + resp = get(conn, "/channels/#{channel.id}/test") + + assert resp.status == 401 + assert_security_headers(resp) + end + end + + describe "error cases" do + test "disabled channel returns 404", %{ + conn: conn, + disabled_channel: disabled_channel + } do + resp = get(conn, "/channels/#{disabled_channel.id}/test") + + assert_problem(resp, 404, :not_found, []) + end + + test "non-existent channel returns 404", %{conn: conn} do + resp = get(conn, "/channels/#{Ecto.UUID.generate()}/test") + + assert_problem(resp, 404, :not_found, []) + end + + test "invalid UUID returns 404", %{conn: conn} do + resp = get(conn, "/channels/not-a-uuid/test") + + assert_problem(resp, 404, :not_found, []) + end + + test "destination timeout returns 502", %{ + conn: conn, + channel: channel + } do + # Channel's destination_url points to a port with nothing running + port = Enum.random(59_000..59_999) + + channel + |> Ecto.Changeset.change(destination_url: "http://localhost:#{port}") + |> Lightning.Repo.update!() + + resp = get(conn, "/channels/#{channel.id}/test") + + assert resp.status in [502, 504] + end + end + + describe "SSRF egress guard" do + test "refuses to proxy to the cloud metadata IP (AWS IMDS / GCP metadata)", + %{conn: conn} do + project = insert(:project) + + channel = + insert(:channel, + project: project, + destination_url: "http://169.254.169.254/latest/meta-data/", + enabled: true + ) + + resp = get(conn, "/channels/#{channel.id}/latest/meta-data/") + + assert resp.status == 403 + refute resp.resp_body =~ "169.254.169.254" + end + + test "refuses to proxy to an RFC1918 private-network address", %{conn: conn} do + project = insert(:project) + + channel = + insert(:channel, + project: project, + destination_url: "http://10.0.0.1", + enabled: true + ) + + resp = get(conn, "/channels/#{channel.id}/admin") + + assert resp.status == 403 + refute resp.resp_body =~ "10.0.0.1" + end + + test "refuses to proxy to a private-range address not on the allow-list", %{ + conn: conn + } do + # A second RFC1918 address to lock in the general private-range case: + # with blocking on (default) and no allow-list entry, the block applies + # before any socket is opened. + project = insert(:project) + + channel = + insert(:channel, + project: project, + destination_url: "http://10.0.0.10", + enabled: true + ) + + resp = get(conn, "/channels/#{channel.id}/admin") + + assert resp.status == 403 + refute resp.resp_body =~ "10.0.0.10" + end + end + + describe "non-UTF-8 body handling" do + # `response_body_preview` and `request_body_preview` are stored as :text + # (UTF-8 only). When an upstream returns binary content (gzip, image, PDF, + # ...) the bytes can't be persisted as text. Rather than failing the whole + # insert, we drop the offending preview to nil and persist everything else + # — headers, hash, size, timing. + + test "non-UTF-8 response body is dropped from preview; rest of event persisted", + %{bypass: bypass, channel: channel} do + # `0x8b` is the second byte of the gzip magic number — exactly the byte + # Postgres rejected in the dev reproduction. We send it raw so Finch + # cannot transparently decompress it. + raw_bytes = <<0x1F, 0x8B, 0x08, 0x00, 0xFF, 0xFE>> + + Bypass.expect_once(bypass, "GET", "/binary", fn conn -> + conn + |> Plug.Conn.put_resp_header("content-type", "application/octet-stream") + |> Plug.Conn.send_resp(200, raw_bytes) + end) + + resp = + conn(:get, "/channels/#{channel.id}/binary") + |> send_to_endpoint() + + assert resp.status == 200 + + request = + Lightning.Repo.one!( + from(r in ChannelRequest, where: r.channel_id == ^channel.id) + ) + + # The request itself succeeded — only the body preview is unstorable. + assert request.state == :success + assert request.completed_at != nil + + event = + Lightning.Repo.one!( + from(e in ChannelEvent, where: e.channel_request_id == ^request.id) + ) + + assert event.type == :destination_response + assert event.request_method == "GET" + assert event.request_path == "/binary" + assert event.response_status == 200 + + # Body preview dropped because it isn't valid UTF-8. + assert event.response_body_preview == nil + + # Hash and size are still recorded so the audit log can show that a body + # was returned, even though the bytes couldn't be persisted as text. + assert is_binary(event.response_body_hash) + assert event.response_body_size == byte_size(raw_bytes) + + # Headers and timing persist as usual. + assert is_list(event.request_headers) and event.request_headers != [] + assert is_list(event.response_headers) and event.response_headers != [] + assert is_integer(event.latency_us) and event.latency_us > 0 + end + + test "non-UTF-8 request body is dropped from preview; rest of event persisted", + %{bypass: bypass, channel: channel} do + raw_bytes = <<0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A>> + + Bypass.expect_once(bypass, "POST", "/upload", fn conn -> + Plug.Conn.send_resp(conn, 201, "ok") + end) + + resp = + conn(:post, "/channels/#{channel.id}/upload", raw_bytes) + |> Plug.Conn.put_req_header("content-type", "application/octet-stream") + |> Plug.Conn.put_req_header("content-length", "#{byte_size(raw_bytes)}") + |> send_to_endpoint() + + assert resp.status == 201 + + request = + Lightning.Repo.one!( + from(r in ChannelRequest, where: r.channel_id == ^channel.id) + ) + + assert request.state == :success + + event = + Lightning.Repo.one!( + from(e in ChannelEvent, where: e.channel_request_id == ^request.id) + ) + + assert event.request_method == "POST" + assert event.request_body_preview == nil + assert is_binary(event.request_body_hash) + assert event.request_body_size == byte_size(raw_bytes) + end + + test "valid UTF-8 body is preserved unchanged", + %{bypass: bypass, channel: channel} do + Bypass.expect_once(bypass, "GET", "/json", fn conn -> + conn + |> Plug.Conn.put_resp_header("content-type", "application/json") + |> Plug.Conn.send_resp(200, ~s({"hello":"world"})) + end) + + resp = + conn(:get, "/channels/#{channel.id}/json") + |> send_to_endpoint() + + assert resp.status == 200 + + event = + Lightning.Repo.one!( + from(e in ChannelEvent, + join: r in ChannelRequest, + on: r.id == e.channel_request_id, + where: r.channel_id == ^channel.id + ) + ) + + assert event.response_body_preview == ~s({"hello":"world"}) + end + end + + describe "answers Lightning builds itself" do + test "a channel whose snapshot can't be saved answers 500 internal", %{ + conn: conn, + channel: channel + } do + unsnapshottable!(channel) + + resp = get(conn, "/channels/#{channel.id}/test") + + assert_problem(resp, 500, :internal, []) + assert_security_headers(resp) + end + end +end diff --git a/test/lightning_web/plugs/person_or_service_account_auth_test.exs b/test/lightning_web/plugs/person_or_service_account_auth_test.exs new file mode 100644 index 00000000000..212ade37c09 --- /dev/null +++ b/test/lightning_web/plugs/person_or_service_account_auth_test.exs @@ -0,0 +1,66 @@ +defmodule LightningWeb.Plugs.PersonOrServiceAccountAuthTest do + use LightningWeb.ConnCase, async: true + + import LightningWeb.API.ProblemHelpers + import Lightning.Factories + import Lightning.ServiceAccountHelpers + + setup %{conn: conn} do + {account, _private_key} = service_account_with_key() + Mox.stub(Lightning.MockConfig, :service_account, fn -> account end) + + %{ + conn: put_req_header(conn, "accept", "application/json"), + account: account + } + end + + defp with_token(conn, token), + do: put_req_header(conn, "authorization", "Bearer " <> token) + + describe "GET /api/projects/:id" do + test "refuses an expired service-account token with 401 invalid_token", %{ + conn: conn, + account: account + } do + conn = + conn + |> with_token(expired_access_token(account, "projects:write")) + |> get(~p"/api/projects/#{insert(:project).id}") + + assert_problem(conn, 401, :invalid_token, []) + + assert get_resp_header(conn, "www-authenticate") == [ + ~s(Bearer error="invalid_token") + ] + end + + test "shows a project to a member's personal access token", %{conn: conn} do + user = insert(:user) + project = insert(:project, project_users: [%{user: user}]) + + conn = + conn + |> with_token(Lightning.Accounts.generate_api_token(user)) + |> get(~p"/api/projects/#{project.id}") + + assert %{"id" => id} = json_response(conn, 200)["data"] + assert id == project.id + end + + test "refuses a non-member's personal access token with 404", %{ + conn: conn + } do + user = insert(:user) + project = insert(:project) + + conn = + conn + |> with_token(Lightning.Accounts.generate_api_token(user)) + |> get(~p"/api/projects/#{project.id}") + + assert_problem(conn, 404, :not_found, []) + assert get_resp_header(conn, "www-authenticate") == [] + end + end +end diff --git a/test/lightning_web/plugs/webhook_auth_test.exs b/test/lightning_web/plugs/webhook_auth_test.exs index 25e113c6854..862c215dd4b 100644 --- a/test/lightning_web/plugs/webhook_auth_test.exs +++ b/test/lightning_web/plugs/webhook_auth_test.exs @@ -3,6 +3,7 @@ defmodule LightningWeb.Plugs.WebhookAuthTest do import Plug.Test import Lightning.Factories + import LightningWeb.API.ProblemHelpers alias Lightning.Repo alias Lightning.WebhookAuthMethods @@ -40,8 +41,7 @@ defmodule LightningWeb.Plugs.WebhookAuthTest do |> WebhookAuth.call([]) assert conn.halted - assert conn.status == 404 - assert Jason.decode!(conn.resp_body) == %{"error" => "Webhook not found"} + assert_problem(conn, 404, :not_found, []) end test "assigns trigger for matching x-api-key", %{trigger: trigger} do @@ -77,20 +77,14 @@ defmodule LightningWeb.Plugs.WebhookAuthTest do conn = conn(:post, "/i/anything") |> WebhookAuth.call([]) assert conn.halted - assert conn.status == 503 + assert_problem(conn, 503, :service_unavailable, []) assert get_resp_header(conn, "retry-after") == ["1"] - - body = Jason.decode!(conn.resp_body) - assert body["error"] == "service_unavailable" - assert body["retry_after"] == 1 - assert String.contains?(body["message"], "retry in 1s") end test "responds with 404 when trigger doesn't exist", _context do conn = conn(:post, "/i/non_existent_trigger") |> WebhookAuth.call([]) - assert conn.status == 404 - assert Jason.decode!(conn.resp_body) == %{"error" => "Webhook not found"} + assert_problem(conn, 404, :not_found, []) end test "assigns the trigger when no auth method is configured", %{ @@ -131,8 +125,7 @@ defmodule LightningWeb.Plugs.WebhookAuthTest do conn = conn(:post, url) |> WebhookAuth.call([]) assert conn.halted - assert conn.status == 401 - assert Jason.decode!(conn.resp_body) == %{"error" => "Unauthorized"} + assert_problem(conn, 401, :unauthenticated, []) end test "lets a correctly authenticated request through", %{ @@ -153,16 +146,6 @@ defmodule LightningWeb.Plugs.WebhookAuthTest do end end - test "responds with 401 for an unauthenticated request to a protected trigger", - %{trigger: trigger, auth_method: auth_method} do - associate_auth_method(trigger, auth_method) - - conn = conn(:post, "/i/#{trigger.id}") |> WebhookAuth.call([]) - - assert conn.status == 401 - assert Jason.decode!(conn.resp_body) == %{"error" => "Unauthorized"} - end - test "responds with 404 for authenticated request with non-matching auth_method", %{trigger: trigger, auth_method: auth_method} do associate_auth_method(trigger, auth_method) @@ -172,8 +155,7 @@ defmodule LightningWeb.Plugs.WebhookAuthTest do |> put_req_header("authorization", "Basic wrong_encoded_auth_method") |> WebhookAuth.call([]) - assert conn.status == 404 - assert Jason.decode!(conn.resp_body) == %{"error" => "Webhook not found"} + assert_problem(conn, 404, :not_found, []) end test "assigns the trigger for authenticated request with matching auth_method", diff --git a/test/support/api_ai_assistant_helpers.ex b/test/support/api_ai_assistant_helpers.ex new file mode 100644 index 00000000000..85adf0cbcf9 --- /dev/null +++ b/test/support/api_ai_assistant_helpers.ex @@ -0,0 +1,22 @@ +defmodule LightningWeb.ApiAiAssistantHelpers do + @moduledoc """ + Builders for the answers `GET /api/ai_assistant/sessions` gives, for API tests. + """ + + @doc """ + The `updated_at` the route writes for `session`, a record as inserted. + """ + def updated_at(session), + do: NaiveDateTime.to_iso8601(session.updated_at) <> "Z" + + @doc """ + The `pagination` member of an answer. + """ + def pagination(total_count, opts \\ []) do + %{ + "total_count" => total_count, + "has_next_page" => Keyword.get(opts, :next, false), + "has_prev_page" => Keyword.get(opts, :prev, false) + } + end +end diff --git a/test/support/api_channel_proxy_helpers.ex b/test/support/api_channel_proxy_helpers.ex new file mode 100644 index 00000000000..78b677d26da --- /dev/null +++ b/test/support/api_channel_proxy_helpers.ex @@ -0,0 +1,114 @@ +defmodule LightningWeb.API.ChannelProxyHelpers do + @moduledoc """ + Builders and assertions the channel proxy test files share. + """ + import ExUnit.Assertions + import Ecto.Query + import Lightning.Factories + + @security_headers [ + {"content-security-policy", + "default-src 'none'; sandbox; frame-ancestors 'none'"}, + {"x-content-type-options", "nosniff"}, + {"x-frame-options", "DENY"}, + {"referrer-policy", "no-referrer"} + ] + + def create_client_auth_channel( + bypass, + auth_method_specs, + destination_token \\ nil + ) do + project = insert(:project) + + channel_auth_methods = + Enum.map(auth_method_specs, fn spec -> + build(:channel_auth_method, + role: :client, + webhook_auth_method: + build( + :webhook_auth_method, + [project: project] ++ Enum.to_list(spec) + ) + ) + end) + + insert(:channel, + project: project, + destination_url: "http://localhost:#{bypass.port}", + enabled: true, + channel_auth_methods: + channel_auth_methods ++ + destination_auth_methods(project, destination_token) + ) + end + + def destination_auth_methods(_project, nil), do: [] + + def destination_auth_methods(project, token) do + credential = + insert(:credential, + schema: "@openfn/language-http", + name: "destination-cred", + user: insert(:user) + ) + |> with_body(%{body: %{"access_token" => token}}) + + [ + build(:channel_auth_method, + role: :destination, + webhook_auth_method: nil, + project_credential: + insert(:project_credential, project: project, credential: credential) + ) + ] + end + + def create_destination_auth_channel(bypass, schema, body) do + project = insert(:project) + user = insert(:user) + + credential = + insert(:credential, schema: schema, name: "destination-cred", user: user) + |> with_body(%{body: body}) + + project_credential = + insert(:project_credential, + project: project, + credential: credential + ) + + insert(:channel, + project: project, + destination_url: "http://localhost:#{bypass.port}", + enabled: true, + channel_auth_methods: [ + build(:channel_auth_method, + role: :destination, + webhook_auth_method: nil, + project_credential: project_credential + ) + ] + ) + end + + # A blank name passes the channels table but not the snapshot changeset, so + # the proxy fails to snapshot without a mock. + def unsnapshottable!(channel) do + Lightning.Repo.update_all( + from(c in Lightning.Channels.Channel, where: c.id == ^channel.id), + set: [name: ""] + ) + end + + def assert_security_headers(resp) do + for {header, value} <- @security_headers do + assert Plug.Conn.get_resp_header(resp, header) == [value], + "expected #{header}: #{value}" + end + end + + def send_to_endpoint(conn) do + LightningWeb.Endpoint.call(conn, LightningWeb.Endpoint.init([])) + end +end diff --git a/test/support/api_collections_helpers.ex b/test/support/api_collections_helpers.ex new file mode 100644 index 00000000000..c895f8673e5 --- /dev/null +++ b/test/support/api_collections_helpers.ex @@ -0,0 +1,56 @@ +defmodule LightningWeb.ApiCollectionsHelpers do + @moduledoc """ + Setup shared by the `/collections` item API tests. + """ + import Lightning.Factories + + alias LightningWeb.ConnHelpers + + @doc """ + A `setup` callback giving a member of a project, the project, an empty + collection in it, and `conn` carrying the member's personal access token. + + The member is a plain project member unless the test sets the `:role` tag. + """ + def member_with_collection(%{conn: conn} = context) do + user = insert(:user) + + project_user = + case context[:role] do + nil -> %{user: user} + role -> %{user: user, role: role} + end + + project = insert(:project, project_users: [project_user]) + collection = insert(:collection, project: project) + + %{ + conn: ConnHelpers.assign_bearer(conn, user), + user: user, + project: project, + collection: collection + } + end + + @doc """ + The run token of a run in `project`, or in a new project when none is given. + """ + def run_token(project \\ nil) do + workflow = + if project, + do: insert(:simple_workflow, project: project), + else: insert(:simple_workflow) + + workorder = + insert(:workorder, workflow: workflow, dataclip: insert(:dataclip)) + + run = + insert(:run, + work_order: workorder, + dataclip: workorder.dataclip, + starting_trigger: hd(workflow.triggers) + ) + + Lightning.Workers.generate_run_token(run) + end +end diff --git a/test/support/api_credentials_helpers.ex b/test/support/api_credentials_helpers.ex new file mode 100644 index 00000000000..2011dfa40ff --- /dev/null +++ b/test/support/api_credentials_helpers.ex @@ -0,0 +1,84 @@ +defmodule LightningWeb.ApiCredentialsHelpers do + @moduledoc false + use Phoenix.VerifiedRoutes, + endpoint: LightningWeb.Endpoint, + router: LightningWeb.Router + + import Phoenix.ConnTest + import Plug.Conn, only: [put_req_header: 3] + import LightningWeb.ConnHelpers + import Lightning.Factories + import Ecto.Query + + alias Lightning.Credentials + alias Lightning.Repo + + @endpoint LightningWeb.Endpoint + @secret "s3cr3t-never-rendered" + + def secret, do: @secret + + def put_credential(conn, id, body) do + conn + |> put_req_header("content-type", "application/json") + |> put(~p"/api/credentials/#{id}", Jason.encode!(body)) + end + + def with_pat(conn, user) do + assign_bearer(conn, Lightning.Accounts.generate_api_token(user)) + end + + def credential_body(attrs) do + Map.merge( + %{ + "name" => "Acme DHIS2", + "schema" => "raw", + "credential_bodies" => [ + %{"name" => "main", "body" => %{"password" => @secret}} + ] + }, + attrs + ) + end + + def links(project_ids), + do: Enum.map(project_ids, &%{"project_id" => &1}) + + def main_body(credential_id), + do: Credentials.get_credential_body(credential_id, "main").body + + def linked_project_ids(response) do + response["credential"]["project_credentials"] + |> Enum.map(& &1["project_id"]) + |> Enum.sort() + end + + # A credential with a staging body beside its main one, linked to `project`. + # Inserted directly, so the owner needn't have access to `project`. + def existing_credential(owner, project) do + insert(:credential, + name: "Acme DHIS2", + schema: "raw", + user: owner, + project_credentials: [%{project_id: project.id}] + ) + |> with_body(%{name: "main", body: %{"password" => "old"}}) + |> with_body(%{name: "staging", body: %{"password" => "staging"}}) + end + + def scheduled_for_deletion(credential) do + credential + |> Ecto.Changeset.change( + scheduled_deletion: DateTime.utc_now() |> DateTime.truncate(:second) + ) + |> Repo.update!() + end + + def audit_ids(credential_id) do + Repo.all( + from a in Lightning.Credentials.Audit.base_query(), + where: a.item_id == ^credential_id, + select: a.id + ) + end +end diff --git a/test/support/api_problem_helpers.ex b/test/support/api_problem_helpers.ex new file mode 100644 index 00000000000..77d40d7d9ce --- /dev/null +++ b/test/support/api_problem_helpers.ex @@ -0,0 +1,114 @@ +defmodule LightningWeb.API.ProblemHelpers do + @moduledoc """ + Assertions on problem documents, for API tests. + """ + import ExUnit.Assertions + + alias LightningWeb.API.Codes + alias LightningWeb.API.Problem + + @sentinel "sentinel-5e17f4c2" + + @doc """ + Asserts `conn` answered with the problem for `code` at `status`, and returns + the decoded body. + + It checks the content type against the request's `Accept`, the members + `LightningWeb.API.Codes` fixes, and that every code, the response's and each + entry's, is in one of the two code lists. `entries` is `:any`, or the exact + list of entries in any order, as decoded JSON: + `[%{"pointer" => "#/name", "code" => "required", "detail" => "can't be blank"}]`. + """ + def assert_problem(conn, status, code, entries \\ :any) do + assert conn.status == status + + assert [content_type] = Plug.Conn.get_resp_header(conn, "content-type") + [media_type | _params] = String.split(content_type, ";") + assert media_type == Problem.content_type(conn) + + body = Jason.decode!(conn.resp_body) + + assert code in Codes.response_codes() + + assert status in Codes.statuses(code) + + assert Map.take(body, ["type", "status", "title", "code", "detail"]) == %{ + "type" => "about:blank", + "status" => status, + "title" => Plug.Conn.Status.reason_phrase(status), + "code" => Atom.to_string(code), + "detail" => Codes.detail(code) + } + + entry_codes = Enum.map(Codes.entry_codes(), &Atom.to_string/1) + + for entry <- Map.get(body, "errors", []) do + assert entry["code"] in entry_codes, + "#{inspect(entry["code"])} is in neither code list" + end + + if entries != :any do + assert Enum.sort(Map.get(body, "errors", [])) == Enum.sort(entries) + end + + body + end + + @doc """ + Asserts `conn` was refused 401 `code` with exactly the `WWW-Authenticate` + values in `challenges`, one per header line (spec §9.1, §14). + """ + def assert_unauthenticated(conn, code, challenges) do + body = assert_problem(conn, 401, code, []) + assert Plug.Conn.get_resp_header(conn, "www-authenticate") == challenges + body + end + + @doc """ + Asserts `conn` was refused the way a route that takes only the session + cookie refuses: 401 `unauthenticated` and a lone `Session` challenge on one + header line (spec §9.1). + """ + def assert_session_refused(conn) do + assert_unauthenticated(conn, :unauthenticated, [ + ~s(Session realm="Lightning") + ]) + end + + @doc """ + Asserts `conns`, one conn or a list, was refused 403 `insufficient_scope` + naming `grant`, so the answers can't tell an existing target from a missing + one (spec §4 row 2, §9.2). + """ + def assert_refused_for_scope(conns, grant) do + for conn <- List.wrap(conns) do + assert_problem(conn, 403, :insufficient_scope, []) + + assert Plug.Conn.get_resp_header(conn, "www-authenticate") == [ + ~s(Bearer error="insufficient_scope", scope="#{grant}") + ] + end + end + + @doc """ + Replaces every value in `params`, at any depth, with a marker that must + never come back in an answer. Keys are kept, since a pointer may name them. + """ + def plant_sentinel(params) when is_map(params), + do: Map.new(params, fn {key, value} -> {key, plant_sentinel(value)} end) + + def plant_sentinel(params) when is_list(params), + do: Enum.map(params, &plant_sentinel/1) + + def plant_sentinel(_value), do: @sentinel + + @doc """ + Asserts that no value `plant_sentinel/1` planted came back in the answer. + """ + def refute_sentinel(conn) do + refute conn.resp_body =~ @sentinel, + "the answer quotes a value from the request: #{conn.resp_body}" + + conn + end +end diff --git a/test/support/api_projects_helpers.ex b/test/support/api_projects_helpers.ex new file mode 100644 index 00000000000..6c0c180bebd --- /dev/null +++ b/test/support/api_projects_helpers.ex @@ -0,0 +1,50 @@ +defmodule LightningWeb.ApiProjectsHelpers do + @moduledoc false + use LightningWeb, :verified_routes + + import Ecto.Query + import Lightning.Factories + import LightningWeb.ConnHelpers, only: [assign_bearer: 2] + import Phoenix.ConnTest, only: [put: 3] + import Plug.Conn, only: [put_req_header: 3] + + @endpoint LightningWeb.Endpoint + + def put_project(conn, id, body) do + conn + |> put_req_header("content-type", "application/json") + |> put(~p"/api/projects/#{id}", Jason.encode!(body)) + end + + def with_pat(conn, user) do + assign_bearer(conn, Lightning.Accounts.generate_api_token(user)) + end + + def members_of(project_id) do + from(pu in Lightning.Projects.ProjectUser, + join: u in assoc(pu, :user), + where: pu.project_id == ^project_id, + select: {u.email, pu.role} + ) + |> Lightning.Repo.all() + |> Enum.sort() + end + + def sorted_members(response) do + response["data"]["attributes"]["members"] + |> Enum.sort_by(& &1["email"]) + end + + def four_members do + for role <- ~w(owner admin editor viewer) do + %{"email" => insert(:user).email, "role" => role} + end + end + + def scheduled_project(project_users \\ []) do + insert(:project, + scheduled_deletion: DateTime.utc_now() |> DateTime.truncate(:second), + project_users: project_users + ) + end +end diff --git a/test/support/api_provisioning_helpers.ex b/test/support/api_provisioning_helpers.ex new file mode 100644 index 00000000000..a944e6e990c --- /dev/null +++ b/test/support/api_provisioning_helpers.ex @@ -0,0 +1,114 @@ +defmodule LightningWeb.API.ProvisioningHelpers do + @moduledoc false + + def put_bearer(conn, token) do + Plug.Conn.put_req_header(conn, "authorization", "Bearer #{token}") + end + + def reenable_triggers_payload(project, workflow) do + [job] = workflow.jobs + [trigger] = workflow.triggers + [edge] = workflow.edges + + %{ + "id" => project.id, + "name" => project.name, + "workflows" => [ + %{ + "id" => workflow.id, + "name" => workflow.name, + "jobs" => [ + %{ + "id" => job.id, + "name" => job.name, + "adaptor" => job.adaptor, + "body" => job.body + } + ], + "triggers" => [ + %{"id" => trigger.id, "type" => "webhook", "enabled" => true} + ], + "edges" => [ + %{ + "id" => edge.id, + "source_trigger_id" => trigger.id, + "target_job_id" => job.id, + "condition_type" => "always", + "enabled" => true + } + ] + } + ] + } + end + + def valid_payload(project_id \\ nil) do + project_id = project_id || Ecto.UUID.generate() + first_job_id = Ecto.UUID.generate() + second_job_id = Ecto.UUID.generate() + trigger_id = Ecto.UUID.generate() + workflow_id = Ecto.UUID.generate() + job_edge_id = Ecto.UUID.generate() + + body = %{ + "id" => project_id, + "name" => "test-project", + "workflows" => [ + %{ + "id" => workflow_id, + "name" => "default", + "jobs" => [ + %{ + "id" => first_job_id, + "name" => "first-job", + "adaptor" => "@openfn/language-common@latest", + "body" => "console.log('hello world');" + }, + %{ + "id" => second_job_id, + "name" => "second-job", + "adaptor" => "@openfn/language-common@latest", + "body" => "console.log('hello world');" + } + ], + "triggers" => [ + %{ + "id" => trigger_id + } + ], + "edges" => [ + %{ + "id" => job_edge_id, + "source_job_id" => first_job_id, + "condition_type" => "on_job_success", + "target_job_id" => second_job_id + } + ] + } + ] + } + + %{ + body: body, + project_id: project_id, + workflow_id: workflow_id, + first_job_id: first_job_id, + second_job_id: second_job_id, + trigger_id: trigger_id, + job_edge_id: job_edge_id + } + end + + def add_job_to_document(document, job_params) do + update_in(document, ["workflows", Access.at(0), "jobs"], &[job_params | &1]) + end + + def remove_job_from_document(document, id) do + update_in(document, ["workflows", Access.at(0), "jobs"], fn jobs -> + Enum.map(jobs, fn + %{"id" => ^id} = job -> Map.put(job, "delete", true) + job -> job + end) + end) + end +end diff --git a/test/support/api_read_helpers.ex b/test/support/api_read_helpers.ex new file mode 100644 index 00000000000..258b6e97261 --- /dev/null +++ b/test/support/api_read_helpers.ex @@ -0,0 +1,69 @@ +defmodule LightningWeb.ApiReadHelpers do + @moduledoc """ + Builders for the API tests of the routes that list and show runs, work + orders, log lines and jobs. + """ + import Lightning.Factories + + alias Lightning.Projects.Project + + @doc """ + Inserts a work order in `target`, a workflow or a project (which gets a new + workflow). `attrs` go to the work order. + """ + def insert_work_order(target, attrs \\ []) do + workflow = workflow_for(target) + + trigger = + Keyword.get_lazy(attrs, :trigger, fn -> + insert(:trigger, workflow: workflow) + end) + + insert( + :workorder, + Keyword.merge( + [workflow: workflow, trigger: trigger, dataclip: build(:dataclip)], + attrs + ) + ) + end + + @doc """ + Inserts a work order holding one run, in `target`, and returns the run. + `run_attrs` go to the run, `work_order_attrs` to the work order, which takes + the run's `inserted_at` unless it is given its own. + """ + def insert_run(target, run_attrs \\ [], work_order_attrs \\ []) do + workflow = workflow_for(target) + trigger = insert(:trigger, workflow: workflow) + + run = + Map.merge( + %{ + state: :started, + dataclip: build(:dataclip), + starting_trigger: trigger + }, + Map.new(run_attrs) + ) + + inserted = + for {:inserted_at, at} <- Map.to_list(run), do: {:inserted_at, at} + + %{runs: [run]} = + insert_work_order( + workflow, + Keyword.merge( + [trigger: trigger, runs: [run]] ++ inserted, + work_order_attrs + ) + ) + + run + end + + defp workflow_for(%Project{} = project), + do: insert(:workflow, project: project) + + defp workflow_for(workflow), do: workflow +end diff --git a/test/support/api_users_helpers.ex b/test/support/api_users_helpers.ex new file mode 100644 index 00000000000..78ece5a8e51 --- /dev/null +++ b/test/support/api_users_helpers.ex @@ -0,0 +1,43 @@ +defmodule LightningWeb.ApiUsersHelpers do + @moduledoc """ + Setup and builders shared by the users API tests. + """ + import Lightning.ServiceAccountHelpers + import Plug.Conn, only: [put_req_header: 3] + + alias Lightning.ServiceAccount.AccessToken + alias LightningWeb.UserAuth + + @password "a long enough password" + + def password, do: @password + + @doc """ + A setup that makes a service account, configures it as the instance's one, + and asks for JSON. Adds `:account` to the context. + """ + def json_conn_with_service_account(%{conn: conn}) do + {account, _private_key} = service_account_with_key() + Mox.stub(Lightning.MockConfig, :service_account, fn -> account end) + + %{ + conn: put_req_header(conn, "accept", "application/json"), + account: account + } + end + + def with_scopes(conn, account, scopes) do + put_req_header( + conn, + "authorization", + "Bearer " <> AccessToken.issue(account, scopes) + ) + end + + def user_url(id), do: "#{LightningWeb.Endpoint.url()}/api/users/#{id}" + + def watch_sockets(user) do + LightningWeb.Endpoint.subscribe(UserAuth.user_socket_topic(user)) + LightningWeb.Endpoint.subscribe(UserAuth.live_socket_topic(user)) + end +end diff --git a/test/support/api_workflows_helpers.ex b/test/support/api_workflows_helpers.ex new file mode 100644 index 00000000000..448a6acd668 --- /dev/null +++ b/test/support/api_workflows_helpers.ex @@ -0,0 +1,96 @@ +defmodule LightningWeb.ApiWorkflowsHelpers do + @moduledoc false + import ExUnit.Assertions + + alias Lightning.Workflows + + def encode_decode(item) do + item + |> Jason.encode!() + |> Jason.decode!() + |> drop_workflow_state() + end + + defp drop_workflow_state(%{"state" => _} = workflow), + do: Map.delete(workflow, "state") + + defp drop_workflow_state(other), do: other + + # Sorted, because the database returns a workflow's jobs, edges and triggers + # in no particular order. + def remove_timestamps(list) when is_list(list) do + list + |> Enum.map(&Map.drop(&1, ["inserted_at", "updated_at"])) + |> Enum.sort_by(& &1["id"]) + end + + def remove_timestamps(workflow) do + Map.merge(workflow, %{ + "inserted_at" => nil, + "updated_at" => nil, + "edges" => remove_timestamps(workflow["edges"]), + "jobs" => remove_timestamps(workflow["jobs"]), + "triggers" => remove_timestamps(workflow["triggers"]) + }) + end + + def get_saved_workflow(%{id: workflow_id}), + do: get_saved_workflow(workflow_id) + + def get_saved_workflow(workflow_id), + do: Workflows.get_workflow(workflow_id, include: [:edges, :jobs, :triggers]) + + def assert_response(response_workflow) do + response_workflow = encode_decode(response_workflow) + + saved_workflow = + response_workflow["id"] |> get_saved_workflow() |> encode_decode() + + assert MapSet.new(response_workflow["jobs"]) == + MapSet.new(saved_workflow["jobs"]) + + assert MapSet.new(response_workflow["edges"]) == + MapSet.new(saved_workflow["edges"]) + + assert MapSet.new(response_workflow["triggers"]) == + MapSet.new(saved_workflow["triggers"]) + end + + def pluck_to_mapset(map, keys) do + map + |> Enum.into(MapSet.new(), fn e -> + Map.take(e, keys) + end) + end + + def graph_body(project_id, edges, opts \\ []) do + %{ + "name" => Keyword.get(opts, :name, "ordered"), + "project_id" => project_id, + "triggers" => [ + %{"id" => "t", "type" => "webhook", "enabled" => true} + ], + "jobs" => + for( + id <- ["a", "b", "c"], + do: %{"id" => id, "name" => "job #{id}", "body" => "fn(s => s)"} + ), + "edges" => + Enum.map(edges, fn + {"t", to} -> + %{ + "source_trigger_id" => "t", + "target_job_id" => to, + "condition_type" => "always" + } + + {from, to} -> + %{ + "source_job_id" => from, + "target_job_id" => to, + "condition_type" => "on_job_success" + } + end) + } + end +end diff --git a/test/support/fixtures/credentials_fixtures.ex b/test/support/fixtures/credentials_fixtures.ex index 4d59331e802..0ed3fd85d27 100644 --- a/test/support/fixtures/credentials_fixtures.ex +++ b/test/support/fixtures/credentials_fixtures.ex @@ -42,7 +42,9 @@ defmodule Lightning.CredentialsFixtures do %{ project_id: attrs[:project_id] || - Lightning.ProjectsFixtures.project_fixture().id + Lightning.ProjectsFixtures.project_fixture( + project_users: [%{user_id: built.user_id}] + ).id } ] }, diff --git a/test/support/projects_helpers.ex b/test/support/projects_helpers.ex index c4639ff05c6..0fff31caf2e 100644 --- a/test/support/projects_helpers.ex +++ b/test/support/projects_helpers.ex @@ -52,7 +52,7 @@ defmodule Lightning.ProjectsHelpers do @doc """ Turns off a project's support access as `actor`, returning the updated project. """ - def revoke_support_access(project, actor \\ nil) do + def revoke_support_access(project, actor) do {:ok, project} = Projects.update_project(project, %{allow_support_access: false}, actor) diff --git a/test/support/query_count_helpers.ex b/test/support/query_count_helpers.ex new file mode 100644 index 00000000000..66b4ee2d869 --- /dev/null +++ b/test/support/query_count_helpers.ex @@ -0,0 +1,44 @@ +defmodule Lightning.QueryCountHelpers do + @moduledoc """ + Counts the database queries a function costs. + """ + + @doc """ + Runs `fun` and returns `{result, query_count}`. + + Only queries from the calling process count, because other async tests query + from their own processes. Pass `source` to count only queries on that table. + """ + def count_queries(fun, source \\ nil) do + parent = self() + ref = make_ref() + handler = {__MODULE__, ref} + + :telemetry.attach( + handler, + [:lightning, :repo, :query], + fn _event, _measurements, meta, _config -> + if self() == parent and (is_nil(source) or source == meta[:source]), + do: send(parent, {ref, :query}) + end, + nil + ) + + result = + try do + fun.() + after + :telemetry.detach(handler) + end + + {result, drain(ref, 0)} + end + + defp drain(ref, count) do + receive do + {^ref, :query} -> drain(ref, count + 1) + after + 0 -> count + end + end +end diff --git a/test/support/race_helpers.ex b/test/support/race_helpers.ex new file mode 100644 index 00000000000..fd3a55cc95b --- /dev/null +++ b/test/support/race_helpers.ex @@ -0,0 +1,77 @@ +defmodule Lightning.RaceHelpers do + @moduledoc """ + Makes another request's write land in the middle of the one under test. + """ + + alias Lightning.Repo + + @doc """ + Calls `fun` with the dumped `id` the first time `pid` runs a query on the + table `source` with `id` among its parameters, and never again. + + Use it to delete or bump a row after the code under test has read it and + before it writes. Only queries from `pid` count, because other async tests + query from their own processes. The handler is detached when the test exits. + + A raise is what Sentry reports, and ConnTest hands a raise to the test, so a + conn test that reaches its assertion shows nothing was reported. + + after_first_query("users", user.id, fn id -> + Repo.query!("DELETE FROM users WHERE id = $1", [id]) + end) + """ + def after_first_query(source, id, fun, pid \\ self()) do + handler = {__MODULE__, make_ref()} + dumped = Ecto.UUID.dump!(id) + + :telemetry.attach( + handler, + [:lightning, :repo, :query], + fn _event, _measurements, meta, _config -> + if self() == pid and meta.source == source and + dumped in List.flatten(meta.params) and + not Process.get(handler, false) do + Process.put(handler, true) + fun.(dumped) + end + end, + nil + ) + + ExUnit.Callbacks.on_exit(fn -> :telemetry.detach(handler) end) + end + + @doc """ + Deletes the project `project_id` after `pid` first queries `projects` for it, + which is after the save has read the project and before it writes. + """ + def delete_project_mid_save(project_id, pid \\ self()) do + after_first_query( + "projects", + project_id, + fn id -> + Repo.query!("DELETE FROM project_users WHERE project_id = $1", [id]) + Repo.query!("DELETE FROM projects WHERE id = $1", [id]) + end, + pid + ) + end + + @doc """ + Bumps the `lock_version` of the workflow `workflow_id` after `pid` first + queries `workflows` for it, so the save under test loses the race. + """ + def bump_workflow_lock_mid_save(workflow_id, pid \\ self()) do + after_first_query( + "workflows", + workflow_id, + fn id -> + Repo.query!( + "UPDATE workflows SET lock_version = lock_version + 1 WHERE id = $1", + [id] + ) + end, + pid + ) + end +end diff --git a/test/support/service_account_helpers.ex b/test/support/service_account_helpers.ex index 736349c31fa..605f6f6c3c6 100644 --- a/test/support/service_account_helpers.ex +++ b/test/support/service_account_helpers.ex @@ -2,6 +2,7 @@ defmodule Lightning.ServiceAccountHelpers do @moduledoc false alias Lightning.ServiceAccount + alias Lightning.ServiceAccount.AccessToken @doc """ A fresh key pair and the service account its public half registers. @@ -39,6 +40,38 @@ defmodule Lightning.ServiceAccountHelpers do token end + @doc """ + Returns `{conn, account}`, with `conn` bearing a live access token carrying + `scopes` for a fresh service account that this Lightning registers. + """ + def with_service_account(conn, scopes) do + {account, _private_key} = service_account_with_key() + Mox.stub(Lightning.MockConfig, :service_account, fn -> account end) + + token = AccessToken.issue(account, scopes) + + {Plug.Conn.put_req_header(conn, "authorization", "Bearer " <> token), + account} + end + + @doc """ + An access token for `account` carrying `scope`, signed as Lightning signs + one, that expired a second ago. + """ + def expired_access_token(%ServiceAccount{id: id}, scope) do + sign_assertion( + Lightning.Config.token_signer().jwk, + %{ + "iss" => AccessToken.issuer(), + "aud" => AccessToken.issuer() <> "/api", + "sub" => id, + "scope" => scope, + "exp" => System.system_time(:second) - 1 + }, + %{"alg" => "RS256", "typ" => "at+jwt"} + ) + end + @doc """ A token that claims `alg: none` and carries no signature. """