From 929628058543f3d8da42ff8a7f0c5671c3f9d68f Mon Sep 17 00:00:00 2001 From: Nick Fitzgerald Date: Thu, 15 Oct 2026 10:56:31 -0700 Subject: [PATCH 1/3] Discuss which benchmark suite to run in `benchmarks/README.md` --- benchmarks/README.md | 38 ++++++++++++++++++++++++++++++++------ 1 file changed, 32 insertions(+), 6 deletions(-) diff --git a/benchmarks/README.md b/benchmarks/README.md index 2fa76bfd..2a384147 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -1,12 +1,38 @@ # Candidate Benchmark Programs -This directory contains the candidate programs for the benchmark suite. They are -candidates, not officially part of the suite yet, because we [intend][rfc] to -record various metrics about the programs and then run a principal component -analysis to find a representative subset of candidates that doesn't contain -effectively duplicate workloads. +This directory contains the candidate programs for the benchmark suite. -[rfc]: https://github.com/bytecodealliance/rfcs/pull/4 +## Which Benchmark Programs Should I Run? + +**99% of the time, you want to run the `default.suite`.** + +The default suite is derived from a [principle-component analysis +(PCA)](https://github.com/bytecodealliance/sightglass/blob/main/scripts/pca.R) +to maximize the amount of benchmark variation covered while avoiding +effectively-redundant workloads (those that exercise the same kinds of code that +an existing benchmark already exercises) and conserving developer +time. `default.suite` is an alias of `pca-20-min.suite`, and completes a full +run with default options in an estimated ~20 minutes. + +For fast iteration locally, but not as a replacement for a full `default.suite` +run, you can also use one of the smaller PCA-based subsets of `default.suite`: + +* `pca-5-min.suite`: completes in an estimated ~5 minutes +* `pca-10-min.suite`: completes in an estimated ~10 minutes + +If you are specifically interested in a certain kind of program, you can also +try one of these suites: + +* `gc.suite`: the set of benchmarks that use Wasm GC +* `simd.suite`: the set of benchmarks that use SIMD + +Finally, while you *can* also run `all.suite` -- which contains every candidate +benchmark program in Sightglass -- be aware that it contains many +effectively-redundant workloads (e.g. many of the libsodium benchmarks exercise +basically the same kind of code) and therefore its geomean is lopsided, weighing +these redundant workloads heavier than workloads that appear only once in the +corpus, and also its full runs take an exceedingly long time. We do not +recommend benchmarking `all.suite`. ## Building From 3a4d78e5f7e8dbf06081332f85768c5180233914 Mon Sep 17 00:00:00 2001 From: Nick Fitzgerald Date: Thu, 15 Oct 2026 11:07:28 -0700 Subject: [PATCH 2/3] Update the GC and SIMD suites --- benchmarks/gc.suite | 16 ++++++++++++++++ benchmarks/simd.suite | 13 ++++++++----- 2 files changed, 24 insertions(+), 5 deletions(-) diff --git a/benchmarks/gc.suite b/benchmarks/gc.suite index 29692f89..cd5059b6 100644 --- a/benchmarks/gc.suite +++ b/benchmarks/gc.suite @@ -1 +1,17 @@ +# These benchmarks use the WebAssembly GC proposal (`struct`/`array` types, +# `ref.cast`, `i31`, etc.). Inclusion in this list does not guarantee that the +# benchmark is a GC-centric workload in the measured section, only that GC +# instructions are present. +# +# These files were gathered by running, from this directory: +# +# find . -name '*.wasm' | sort | while read f; do +# wasm-tools validate --features=-gc,-gc-types "$f" >/dev/null 2>&1 || echo "${f#./}" +# done + +dart-deltablue/dart-deltablue.wasm +hoot-earley-boyer/hoot-earley-boyer.wasm +java-navier-stokes/java-navier-stokes.wasm +kotlin-richards/kotlin-richards.wasm +ocaml-raytrace/ocaml-raytrace.wasm splay/splay.wasm diff --git a/benchmarks/simd.suite b/benchmarks/simd.suite index d27e8cd7..634127f9 100644 --- a/benchmarks/simd.suite +++ b/benchmarks/simd.suite @@ -2,16 +2,18 @@ # does not guarantee that the benchmark is a SIMD-centric workload in the # measured section, only that SIMD instructions are present. # -# These files were gathered by: -# - creating a `find-simd.sh` script containing: `wasm2wat --disable-simd $1 -# &>/dev/null || echo $1` -# - running the script over the benchmarks: `find . -name '*.wasm' | xargs -L 1 -# ./find-simd.sh` +# These files were gathered by running, from this directory: +# +# find . -name '*.wasm' | sort | while read f; do +# wasm-tools validate --features=-simd,-relaxed-simd "$f" >/dev/null 2>&1 || echo "${f#./}" +# done blake3-simd/benchmark.wasm hex-simd/benchmark.wasm intgemm-simd/benchmark.wasm meshoptimizer/benchmark.wasm +libsodium/libsodium-aead_aegis128l.wasm +libsodium/libsodium-aead_aegis256.wasm libsodium/libsodium-aead_chacha20poly1305.wasm libsodium/libsodium-aead_chacha20poly13052.wasm libsodium/libsodium-aead_xchacha20poly1305.wasm @@ -46,6 +48,7 @@ libsodium/libsodium-generichash3.wasm libsodium/libsodium-hash.wasm libsodium/libsodium-hash3.wasm libsodium/libsodium-kdf.wasm +libsodium/libsodium-kdf_hkdf.wasm libsodium/libsodium-keygen.wasm libsodium/libsodium-kx.wasm libsodium/libsodium-metamorphic.wasm From 419fec87912455bcd052f4e00ed2c55659693070 Mon Sep 17 00:00:00 2001 From: Nick Fitzgerald Date: Thu, 15 Oct 2026 13:22:20 -0700 Subject: [PATCH 3/3] review feedback --- benchmarks/README.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/benchmarks/README.md b/benchmarks/README.md index 2a384147..f9befe79 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -1,12 +1,13 @@ # Candidate Benchmark Programs -This directory contains the candidate programs for the benchmark suite. +This directory contains our various benchmark programs. Not all are included in +a benchmark suite. ## Which Benchmark Programs Should I Run? **99% of the time, you want to run the `default.suite`.** -The default suite is derived from a [principle-component analysis +The default suite is derived from a [principal-component analysis (PCA)](https://github.com/bytecodealliance/sightglass/blob/main/scripts/pca.R) to maximize the amount of benchmark variation covered while avoiding effectively-redundant workloads (those that exercise the same kinds of code that @@ -26,13 +27,13 @@ try one of these suites: * `gc.suite`: the set of benchmarks that use Wasm GC * `simd.suite`: the set of benchmarks that use SIMD -Finally, while you *can* also run `all.suite` -- which contains every candidate -benchmark program in Sightglass -- be aware that it contains many +Finally, while you *can* also run `all.suite` -- which contains every benchmark +program in this directory -- be aware that it contains many effectively-redundant workloads (e.g. many of the libsodium benchmarks exercise basically the same kind of code) and therefore its geomean is lopsided, weighing these redundant workloads heavier than workloads that appear only once in the -corpus, and also its full runs take an exceedingly long time. We do not -recommend benchmarking `all.suite`. +corpus, and running it takes an exceedingly long time. We do not recommend +benchmarking `all.suite`. ## Building