Skip to content

Commit 123bacd

Browse files
behinddwallsbehinddwalls
committed
refactor(test): consolidate testing infrastructure with Docker Compose
Migrate from manual server setup to Docker Compose-based hermetic testing. Consolidate tests under test/ directory with consistent patterns. Key changes: - New test structure: test/e2e/, test/integration/{gateway,orchestrator,extensions}/ - Container naming: sq-test-{context}-{unique-id} for parallel execution - Shared utilities in test/testutil/ (compose, docker, mysql, schema helpers) - Removed old {service}/integration_test/ directories and speculator service - Documentation: TESTING.md → doc/howto/, STRUCTURE.md → PROJECT_STRUCTURE.md - Makefile: alphabetical targets, new integration test targets, local-start/stop naming - Consolidate CI and create separate jobs for faster run and add bazel cache
1 parent 0916f89 commit 123bacd

79 files changed

Lines changed: 3193 additions & 2607 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/actions/logs/action.yml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
name: Logs
2+
description: Display container logs on test failure
3+
4+
runs:
5+
using: composite
6+
steps:
7+
- name: Display container logs on failure
8+
if: failure()
9+
shell: bash
10+
run: |
11+
echo "=== Listing all Docker containers ==="
12+
docker ps -a --filter "name=sq-test-" || true
13+
echo ""
14+
echo "=== Dumping container logs ==="
15+
for container in $(docker ps -a --filter "name=sq-test-" --format "{{.Names}}"); do
16+
echo ">>> Logs for $container <<<"
17+
docker logs "$container" 2>&1 || true
18+
echo ""
19+
done

‎.github/actions/run-bazel-test/action.yml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
name: Run Bazel Test
2+
description: Run Bazel test and show logs on failure
3+
4+
inputs:
5+
target:
6+
description: Bazel test target pattern
7+
required: true
8+
9+
runs:
10+
using: composite
11+
steps:
12+
- name: Run Bazel test
13+
shell: bash
14+
run: ./tool/bazel test ${{ inputs.target }} --test_output=errors
15+
16+
- uses: ./.github/actions/logs
17+
if: always()

‎.github/actions/setup/action.yml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
name: Setup
2+
description: Setup Bazel cache for CI jobs
3+
4+
runs:
5+
using: composite
6+
steps:
7+
- name: Setup Bazel cache
8+
uses: actions/cache@v4
9+
with:
10+
path: |
11+
~/.cache/bazel
12+
~/.cache/bazelisk
13+
key: bazel-${{ runner.os }}-${{ hashFiles('MODULE.bazel', 'go.mod', '.bazelversion') }}
14+
restore-keys: |
15+
bazel-${{ runner.os }}-

‎.github/workflows/build_and_test.yml‎

Lines changed: 0 additions & 50 deletions
This file was deleted.

‎.github/workflows/ci.yml‎

Lines changed: 191 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,191 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
pull_request:
8+
types:
9+
- opened
10+
- reopened
11+
- synchronize
12+
13+
permissions:
14+
contents: read
15+
pull-requests: read
16+
17+
jobs:
18+
# ---------------------------------------------------------------------------
19+
# BUILD AND UNIT TESTS (special case - Gazelle + build + unit tests)
20+
# ---------------------------------------------------------------------------
21+
build-and-unit-test:
22+
name: Build and Unit Test
23+
runs-on: ubuntu-latest
24+
steps:
25+
- uses: actions/checkout@v4
26+
- uses: ./.github/actions/setup
27+
28+
- name: Check BUILD files are up to date
29+
run: |
30+
echo "Running Gazelle to check BUILD files..."
31+
make gazelle
32+
if ! git diff --quiet; then
33+
echo "❌ BUILD files are out of date!"
34+
echo ""
35+
echo "The following files were modified by Gazelle:"
36+
git diff --name-only
37+
echo ""
38+
echo "Please run 'make gazelle' locally and commit the changes."
39+
exit 1
40+
fi
41+
echo "✅ BUILD files are up to date"
42+
43+
- name: Build project
44+
run: make build
45+
46+
- name: Run unit tests
47+
run: make test || echo "No unit tests found"
48+
49+
# ---------------------------------------------------------------------------
50+
# INTEGRATION TESTS (e2e, gateway, orchestrator)
51+
# ---------------------------------------------------------------------------
52+
e2e-integration-test:
53+
name: E2E Integration Test
54+
runs-on: ubuntu-latest
55+
steps:
56+
- uses: actions/checkout@v4
57+
- uses: ./.github/actions/setup
58+
59+
- name: Run E2E tests
60+
run: make e2e-test
61+
62+
- uses: ./.github/actions/logs
63+
64+
gateway-integration-test:
65+
name: Gateway Integration Test
66+
runs-on: ubuntu-latest
67+
steps:
68+
- uses: actions/checkout@v4
69+
- uses: ./.github/actions/setup
70+
71+
- name: Run Gateway integration tests
72+
run: make integration-test-gateway
73+
74+
- uses: ./.github/actions/logs
75+
76+
orchestrator-integration-test:
77+
name: Orchestrator Integration Test
78+
runs-on: ubuntu-latest
79+
steps:
80+
- uses: actions/checkout@v4
81+
- uses: ./.github/actions/setup
82+
83+
- name: Run Orchestrator integration tests
84+
run: make integration-test-orchestrator
85+
86+
- uses: ./.github/actions/logs
87+
88+
# ---------------------------------------------------------------------------
89+
# EXTENSION TESTS (conditional on path changes)
90+
# ---------------------------------------------------------------------------
91+
detect-changes:
92+
name: Detect Extension Changes
93+
runs-on: ubuntu-latest
94+
outputs:
95+
counter: ${{ steps.filter.outputs.counter }}
96+
queue: ${{ steps.filter.outputs.queue }}
97+
storage: ${{ steps.filter.outputs.storage }}
98+
steps:
99+
- uses: actions/checkout@v4
100+
- uses: dorny/paths-filter@v3
101+
id: filter
102+
with:
103+
filters: |
104+
counter:
105+
- 'extension/counter/**'
106+
- 'test/integration/extension/counter/**'
107+
- 'entity/**'
108+
- '.github/workflows/ci.yml'
109+
queue:
110+
- 'extension/queue/**'
111+
- 'test/integration/extension/queue/**'
112+
- 'entity/**'
113+
- '.github/workflows/ci.yml'
114+
storage:
115+
- 'extension/storage/**'
116+
- 'test/integration/extension/storage/**'
117+
- 'entity/**'
118+
- '.github/workflows/ci.yml'
119+
120+
counter-integration-test:
121+
name: Counter Extension Test
122+
needs: detect-changes
123+
if: needs.detect-changes.outputs.counter == 'true'
124+
runs-on: ubuntu-latest
125+
steps:
126+
- uses: actions/checkout@v4
127+
- uses: ./.github/actions/setup
128+
- uses: ./.github/actions/run-bazel-test
129+
with:
130+
target: //test/integration/extension/counter/...
131+
132+
queue-integration-test:
133+
name: Queue Extension Test
134+
needs: detect-changes
135+
if: needs.detect-changes.outputs.queue == 'true'
136+
runs-on: ubuntu-latest
137+
steps:
138+
- uses: actions/checkout@v4
139+
- uses: ./.github/actions/setup
140+
- uses: ./.github/actions/run-bazel-test
141+
with:
142+
target: //test/integration/extension/queue/...
143+
144+
storage-integration-test:
145+
name: Storage Extension Test
146+
needs: detect-changes
147+
if: needs.detect-changes.outputs.storage == 'true'
148+
runs-on: ubuntu-latest
149+
steps:
150+
- uses: actions/checkout@v4
151+
- uses: ./.github/actions/setup
152+
- uses: ./.github/actions/run-bazel-test
153+
with:
154+
target: //test/integration/extension/storage/...
155+
156+
# ---------------------------------------------------------------------------
157+
# REQUIRED CHECKS GATE
158+
# ---------------------------------------------------------------------------
159+
required-checks:
160+
name: Required Checks
161+
runs-on: ubuntu-latest
162+
needs:
163+
- build-and-unit-test
164+
- e2e-integration-test
165+
- gateway-integration-test
166+
- orchestrator-integration-test
167+
- counter-integration-test
168+
- queue-integration-test
169+
- storage-integration-test
170+
steps:
171+
- name: All required checks passed
172+
run: |
173+
echo "✅ All required checks passed!"
174+
echo ""
175+
echo "Core checks:"
176+
echo " - build-and-unit-test"
177+
echo " - e2e-integration-test"
178+
echo " - gateway-integration-test"
179+
echo " - orchestrator-integration-test"
180+
echo ""
181+
echo "Extension checks (if paths changed):"
182+
echo " - counter-integration-test"
183+
echo " - queue-integration-test"
184+
echo " - storage-integration-test"
185+
echo ""
186+
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
187+
echo "📝 To add a new required check:"
188+
echo " 1. Add the job to this workflow"
189+
echo " 2. Add the job name to the 'needs:' list above"
190+
echo " 3. That's it! No branch protection updates needed."
191+
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,7 @@ MODULE.bazel.lock
1010

1111
# Built binaries
1212
bin/
13+
.docker-bin/
14+
15+
# Make completion cache
16+
.make_targets_cache

‎BUILD.bazel‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,4 +7,13 @@ load("@gazelle//:def.bzl", "gazelle")
77
# gazelle:resolve go github.com/uber/submitqueue/orchestrator/protopb //orchestrator/protopb
88
# gazelle:resolve go github.com/uber/submitqueue/speculator/protopb //speculator/protopb
99

10+
# Export marker files for test data dependencies (used by FindRepoRoot in tests)
11+
exports_files(
12+
[
13+
"MODULE.bazel",
14+
"go.mod",
15+
],
16+
visibility = ["//visibility:public"],
17+
)
18+
1019
gazelle(name = "gazelle")

0 commit comments

Comments
 (0)