Skip to content

1.5.x cloud benchmarks#3615

Open
kmatasfp wants to merge 55 commits into
1.5.xfrom
1.5.x-cloud-benchmarks
Open

1.5.x cloud benchmarks#3615
kmatasfp wants to merge 55 commits into
1.5.xfrom
1.5.x-cloud-benchmarks

Conversation

@kmatasfp

@kmatasfp kmatasfp commented Jun 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Also contains replacement for memory semaphore plus fixes eviction deadlock.

for memory semaphore replacement see:

  • golem-worker-executor/src/services/active_workers/admission/mod.rs
  • golem-worker-executor/src/services/active_workers/component_charge/mod.rs
  • golem-worker-executor/src/services/active_workers/memory_probe.rs

@kmatasfp kmatasfp requested a review from a team June 5, 2026 00:18
@kmatasfp
fix: make ProvidedShardManager kill/restart no-ops instead of panics
4294bdb 
kill_all() is called after cloud_preflight_warmup completes. ProvidedShardManager
wraps an already-running process we don't own, so neither kill nor restart should
crash the binary. Both are now silent no-ops, matching UnavailableShardManager.
kmatasfp added 24 commits June 5, 2026 10:44
@kmatasfp
feat(benchmark): enable all tests
5b9902b
@kmatasfp
feat: retry connectivity to shard manager
742a669
@kmatasfp
chore: fmt
18d5af6
@kmatasfp
investigation: run echo test first to see if they get stuck again
395bcd2
@kmatasfp
feat(benchmark): lower number of conccurent live apps
dac3c69
@kmatasfp
feat: more observability, make memory component coefficient configurable
2256623
@kmatasfp
feat(benchmark): run only throughput-echo test
02e527a
@kmatasfp
feat(bench): try 200 apps after tuning
faeb651
@kmatasfp
feat: try 250 again
f8dd565
@kmatasfp
feat(benchmark): run all the tests again
1bf0063
@kmatasfp
fix: metric description
2e53af6
@kmatasfp
feat: proper load for our cluster
32ef9e5
@kmatasfp
feat(benchmark): run only benchmark tests
bc11779
@kmatasfp
feat: enable all tests again
5347626
@kmatasfp
feat(benchmark): increase max number of concurrent compilations
9e582a2
@kmatasfp
feat(worker-executor): add measured-headroom memory admission gate
e7b44bf
@kmatasfp
feat(worker-executor): charge component module size once per resident…
817c672 
… component
@kmatasfp
fix(worker-executor): disable measured admission when executor does n…
35874d3 
…ot own its memory environment
@kmatasfp
feat(benchmark): add throughput-under-memory-saturation benchmarks
acb9968
@kmatasfp
test(worker-executor): exercise admission reserve under maximum concu…
bfe1b14 
…rrent overlap
@kmatasfp
feat(benchmark): longer sustained load, bumpt the number of agents
c3af739
@kmatasfp
fix: add empty workspace
7dcb2d3
@kmatasfp
fix: use snake case as method names
139aed5
@kmatasfp
chore: 300 already saturates, no need for 500
442c1c5
kmatasfp added 27 commits June 9, 2026 14:04
@kmatasfp
fix(worker-executor): avoid deadlock between memory grow and admissio…
4bbb200 
…n eviction scan
@kmatasfp
feat: change order of tests
be19cf4
@kmatasfp
feat: restore iterations count to 3
21fd401
@kmatasfp
refactor(worker-executor): make cgroup gate primary, semaphore clampe…
a9285c0 
…d second line
@kmatasfp
feat: run only initial echo test to make sure we did not make it slower
27119b2
@kmatasfp
feat: run only saturation test
b608593
@kmatasfp
feat: bigger saturation spread
1f1b77a
@kmatasfp
feat(benchmark): change the steps
1bd27ea
@kmatasfp
feat: replace estimate-semaphore completely with measured-headroom ad…
898435d 
…mission controller
@kmatasfp
fix: clippy warnings
8cecf91
@kmatasfp
fix: startup message regarding memory
8566f13
@kmatasfp
chore: run only oom test
626e4ba
@kmatasfp
feat: enable to whole perf test suite
2434047
@kmatasfp
feat: more metrics plus FixedProbe for tests
a8fcf52
@kmatasfp
fix: make admission gate reserve atomic to prevent ceiling overshoot
7eb6f08
@kmatasfp
test: gate concurrent-agent permit tests with a semaphore, not Notify
83a6b2f
@kmatasfp
feat: expose tokio metrics
24673f6
@kmatasfp
fix: prevent concurrent-agent scheduler deadlock on cancel-after-grant
a1928c5
@kmatasfp
feat: use official tokio-metrics crate to expose tokio runtime metrics
183de28
@kmatasfp
feat: use official tokio-metrics crate to expose tokio runtime metric…
78d311d 
…s vol 2
@kmatasfp
chore: cleanup comments
aec411b
@kmatasfp
feat: try mimalloc
71bee78
@kmatasfp
feat: try mimalloc vol 2
b146823
@kmatasfp
perf: enable thin LTO and codegen-units=1 for release builds
0ef2c16
@kmatasfp
perf: pin target-cpu baseline for published images (x86-64-v3, neover…
99e3633 
…se-n1)
@kmatasfp
perf: drop codegen-units=1, keep thin LTO
3218b4f 
The codegen-units=1 runtime gain is marginal once thin LTO is enabled,
while it noticeably increases release build time. Revert to the default
to keep intra-crate parallelism.
@kmatasfp
chore: lower number of cuncurrent agents to 200 in case of durability…
ac3b3ee 
… overhead test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mschuwalow mschuwalow mschuwalow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kmatasfp @mschuwalow