feat(e2e): AGX1-325 — agent_api_keys authz black-box suite

[dm36]

Summary

End-to-end FGAC tests for the agent_api_keys routes. Black-box: every test hits the real scale-agentex HTTP API as one user identity, then verifies the resulting state in SpiceDB via a separate spark-authz client.

Models the KB suite from scaleapi#142983 — same conftest layout, same cleanup tracker (REST-then-SpiceDB fallback), same SparkAuthzClient (copied verbatim, it's repo-agnostic). The Agentex-specific client and four test files are new.

Linear: AGX1-325.

AGX1-325 deliverables → tests

Bullet	Test file
dual-write registers on create	test_api_key_create.py
without permission, 404 on get	test_api_key_get.py
filtered list	test_api_key_list.py
dual-write deregisters on delete	test_api_key_delete.py

Each test creates a fresh agent + api_key as user_a, then asserts wire behavior. The negative paths use a same-tenant user_b who has no grants.

Scope

• user_a (owner) + user_b (same tenant, no permission). Per the kickoff: minimize cross-user complexity for v1.
• Not in this PR: cross-tenant outsider, service identity, retention/cleanup endpoints, two-factor-via-SpiceDB schema correctness (lives in spark-authz's own suite).

Setup

cd scripts/spark-authz-e2e-tests
make install
cp config.json.example config.json
# Fill in headers / identity_ids / account_id for user_a + user_b in your target env.
make test

Out of scope

• CI integration — these run on-demand against a live cluster, not in unit CI. Wiring them into a scheduled job is a separate decision once they pass against a stable dev env.
• More resource types (events, tasks, messages, etc.) — additive tickets if/when needed.

Verified against pubsec-dev

Ran the suite against agentex.sgp-pubsec-dev.scale.com on 2026-06-04. Result: 3 passed / 11 failed / 1 skipped.

The pass set is the event suite from #277 (events delegate to parent-agent and don't depend on api_key dual-write). The 11 failures all trace to a single deployment-config blocker in pubsec-dev — not a defect in this PR's code.

The blocker: agentex-auth runs with AUTH_PROVIDER=sgp

Pubsec-dev's agentex-auth deployment has env var AUTH_PROVIDER=sgp. Every register_resource / deregister_resource call from scale-agentex's dual-write code lands on legacy SGP-authz instead of spark-authz. SGP-authz doesn't know about the api_key resource type, so:

POST /agent_api_keys            -> 200 (DB row created)
GET  /agent_api_keys/{just_id}  -> 422 wrapping 403 (agentex-auth → SGP-authz denied)

Flipping the fgac-* feature flags does not fix this — AUTH_PROVIDER is a process-level env var read at agentex-auth startup, not a per-account FF. To unblock:

• Redeploy agentex-auth in pubsec-dev with AUTH_PROVIDER=spark. spark-authz is already deployed and reachable in the same cluster (ns/spark, svc/spark-authz, ports 50052/8090 — verified via port-forward + /healthz returning SERVING). No new infra needed.

After that flip, expect ~11/12 tests to pass against pubsec-dev. The 2 create-time SpiceDB-asserting tests in test_api_key_create.py additionally require the test user to have agent.create permission on the tenant; mitigated for all other tests by the new parent_agent fixture (falls back to a pre-existing agentex.agent_id in config.json).

What works today

• The events suite from feat(e2e): AGX1-331 — event route authz black-box suite #277 passes 3/3 denied-path tests against pubsec-dev.
• The api_key client + factories + cleanup pattern + parent_agent fallback are all verified to work against the real deployed API.
• The graceful-skip wiring correctly de-risks running the suite in envs without a reachable spark-authz: no fixture errors, no interpreter crashes; everything that can't run is marked SKIPPED with a clear reason.

Run recipe used

• config.json: pubsec-dev API key + EGP account_id as x-selected-account-id (the agentex auth gateway expects the EGP shape) + agentex.agent_id set to a pre-existing agent the user has api_key.create on (because the user lacks agent.create on the tenant wildcard).
• spark_authz.host: pointed at localhost:8090 via kubectl --context <pubsec-dev-aws-eks> -n spark port-forward svc/spark-authz 8090:8090. The probe in authz_reachable confirms /healthz returns SERVING before any SpiceDB-asserting test runs.

Greptile Summary

Introduces a black-box FGAC end-to-end suite for the agent_api_keys routes, modeled after the KB suite. Tests hit the real scale-agentex HTTP API and verify authorization state directly in SpiceDB via a SparkAuthzClient, covering dual-write on create/delete, owner 200 / non-owner 404 on get, and FGAC list filtering.

• Four new test files (test_api_key_create.py, test_api_key_get.py, test_api_key_list.py, test_api_key_delete.py) plus an AgentexClient wrapper, CleanupTracker, and conftest.py with session-scoped clients and function-scoped factory fixtures.
• The healthz probe now correctly gates on resp.is_success (2xx only) and SpiceDB-asserting tests skip gracefully when spark-authz is unreachable, avoiding confusing fixture errors.
• The 404-path in test_non_owner_list_excludes_user_a_api_key now uses pytest.skip instead of a silent return, making un-exercised assertions visible in reports.

Confidence Score: 5/5

Safe to merge — all changes are test infrastructure under scripts/, with no production code touched.

The suite is well-structured test tooling. The healthz probe, graceful-skip wiring, and cleanup tracker all behave correctly. The one unresolved issue (missing agentex.agent_id in the example config) affects developer experience when running the suite in permission-constrained environments but causes no correctness problem in the tests themselves.

No files require special attention. The only nit is config.json.example missing the agentex.agent_id field that the parent_agent fallback depends on.

Important Files Changed

Filename	Overview
scripts/spark-authz-e2e-tests/conftest.py	Root conftest: session-scoped clients/credentials, function-scoped cleanup tracker, parent_agent factory with config-fallback, and healthz probe now correctly checks resp.is_success (2xx only)
scripts/spark-authz-e2e-tests/tests/test_api_key_create.py	Create dual-write tests: verifies owner read/delete in SpiceDB post-create and that same-tenant user_b has no permissions; both tests gate on authz_client so they skip gracefully when spark-authz is unreachable
scripts/spark-authz-e2e-tests/tests/test_api_key_get.py	GET tests: four cases (owner/non-owner × id/name) all clean; non-owner paths correctly assert 404 and not 403
scripts/spark-authz-e2e-tests/tests/test_api_key_list.py	List FGAC filter tests: 404-from-user_b path now uses pytest.skip (not silent return), making the skipped-assertion visible in reports
scripts/spark-authz-e2e-tests/tests/test_api_key_delete.py	Delete tests: deregister-in-SpiceDB test creates api_key directly (avoiding teardown race) but doesn't register a cleanup entry — api_key leaks if the pre-delete SpiceDB sanity assertion fails (previously flagged, still unfixed)
scripts/spark-authz-e2e-tests/clients/agentex_client.py	Thin httpx wrapper for scale-agentex routes; clean interface returning raw responses for caller-side status assertions
scripts/spark-authz-e2e-tests/clients/spark_authz_client.py	HTTP-transcoded gRPC client for spark-authz ResourceService; covers check/grant/revoke/create/delete/lookup; copied verbatim from KB suite as described in PR
scripts/spark-authz-e2e-tests/config.json.example	Example config template: missing the agentex.agent_id field used by the parent_agent fixture fallback — operators without agent.create permission have no documented path to configure it
scripts/spark-authz-e2e-tests/helpers/cleanup.py	LIFO cleanup tracker: exceptions are swallowed and logged, execute clears actions list preventing double-cleanup — solid utility
scripts/spark-authz-e2e-tests/Makefile	Well-structured Makefile with hierarchical test targets; no broken stub targets in this version

Sequence Diagram

sequenceDiagram
    participant T as Test (pytest)
    participant CA as AgentexClient (user_a)
    participant CB as AgentexClient (user_b)
    participant AX as scale-agentex
    participant AA as agentex-auth
    participant SDB as SpiceDB (via spark-authz)

    T->>CA: create_agent()
    CA->>AX: POST /agents/register-build
    AX->>AA: resolve principal
    AX->>SDB: register_resource(agent)
    AX-->>CA: "200 {id}"

    T->>CA: create_api_key(agent_id)
    CA->>AX: POST /agent_api_keys
    AX->>AA: resolve principal
    AX->>SDB: "register_resource(api_key, parent_agent=agent_id)"
    AX-->>CA: "200 {id, api_key}"

    Note over T,SDB: SpiceDB-asserting tests (skip if spark-authz unreachable)
    T->>SDB: check_permission(user_a, api_key, read) → true
    T->>SDB: check_permission(user_b, api_key, read) → false

    Note over T,AX: HTTP-only tests (always run)
    T->>CB: get_api_key(id)
    CB->>AX: "GET /agent_api_keys/{id}"
    AX->>SDB: check permission (user_b) → deny
    AX-->>CB: 404 (collapsed from 403)

    T->>CA: delete_api_key(id)
    CA->>AX: "DELETE /agent_api_keys/{id}"
    AX->>SDB: deregister_resource(api_key)
    AX-->>CA: 200

Loading

Comments Outside Diff (1)

1. scripts/spark-authz-e2e-tests/tests/test_api_key_delete.py, line 1207-1212 (link)

   Unregistered api_key leaks if pre-delete assertion fails

   The test intentionally bypasses the create_api_key factory to avoid a teardown race, but that means the freshly-created api_key has no cleanup entry. If the pre-delete check_permission_bool assertion fails, the api_key is never deleted and leaks in both the DB and SpiceDB, potentially contaminating subsequent runs. The factory teardown actually handles the 404 case gracefully (it checks not in (200, 204, 404)), so the race concern is minimal. If keeping the factory out is intentional, registering a raw cleanup.add(...) call after the create would prevent the leak without the race issue.

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: scripts/spark-authz-e2e-tests/tests/test_api_key_delete.py
   Line: 1207-1212
   
   Comment:
   **Unregistered api_key leaks if pre-delete assertion fails**
   
   The test intentionally bypasses the `create_api_key` factory to avoid a teardown race, but that means the freshly-created api_key has no cleanup entry. If the pre-delete `check_permission_bool` assertion fails, the api_key is never deleted and leaks in both the DB and SpiceDB, potentially contaminating subsequent runs. The factory teardown actually handles the 404 case gracefully (it checks `not in (200, 204, 404)`), so the race concern is minimal. If keeping the factory out is intentional, registering a raw `cleanup.add(...)` call after the create would prevent the leak without the race issue.
   
   How can I resolve this? If you propose a fix, please make it concise.

   

Prompt To Fix All With AI

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
scripts/spark-authz-e2e-tests/config.json.example:27-32
The `agentex.agent_id` fallback that `parent_agent` relies on isn't shown in the example config. In environments where the test user lacks `agent.create` permission (e.g. shared dev clusters), every test using `parent_agent` hits `Failed to create agent: 403` without any hint that setting this field would avoid the problem. Adding the commented-out section makes the fallback discoverable.

```suggestion
  "test_settings": {
    "request_timeout_seconds": 30,
    "cleanup_on_success": true,
    "cleanup_on_failure": true
  },
  "_comment_agentex": "Optional: set agent_id to a pre-existing agent if the test user lacks agent.create permission on the tenant wildcard. parent_agent uses this instead of calling create_agent().",
  "agentex": {
    "agent_id": ""
  }
}
```

_{Reviews (4): Last reviewed commit: "fix(e2e): treat non-2xx /healthz respons..." | Re-trigger Greptile}