diff --git a/benchmarks/README.md b/benchmarks/README.md index 2fa76bfd..f9befe79 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -1,12 +1,39 @@ # Candidate Benchmark Programs -This directory contains the candidate programs for the benchmark suite. They are -candidates, not officially part of the suite yet, because we [intend][rfc] to -record various metrics about the programs and then run a principal component -analysis to find a representative subset of candidates that doesn't contain -effectively duplicate workloads. +This directory contains our various benchmark programs. Not all are included in +a benchmark suite. -[rfc]: https://github.com/bytecodealliance/rfcs/pull/4 +## Which Benchmark Programs Should I Run? + +**99% of the time, you want to run the `default.suite`.** + +The default suite is derived from a [principal-component analysis +(PCA)](https://github.com/bytecodealliance/sightglass/blob/main/scripts/pca.R) +to maximize the amount of benchmark variation covered while avoiding +effectively-redundant workloads (those that exercise the same kinds of code that +an existing benchmark already exercises) and conserving developer +time. `default.suite` is an alias of `pca-20-min.suite`, and completes a full +run with default options in an estimated ~20 minutes. + +For fast iteration locally, but not as a replacement for a full `default.suite` +run, you can also use one of the smaller PCA-based subsets of `default.suite`: + +* `pca-5-min.suite`: completes in an estimated ~5 minutes +* `pca-10-min.suite`: completes in an estimated ~10 minutes + +If you are specifically interested in a certain kind of program, you can also +try one of these suites: + +* `gc.suite`: the set of benchmarks that use Wasm GC +* `simd.suite`: the set of benchmarks that use SIMD + +Finally, while you *can* also run `all.suite` -- which contains every benchmark +program in this directory -- be aware that it contains many +effectively-redundant workloads (e.g. many of the libsodium benchmarks exercise +basically the same kind of code) and therefore its geomean is lopsided, weighing +these redundant workloads heavier than workloads that appear only once in the +corpus, and running it takes an exceedingly long time. We do not recommend +benchmarking `all.suite`. ## Building diff --git a/benchmarks/gc.suite b/benchmarks/gc.suite index 29692f89..cd5059b6 100644 --- a/benchmarks/gc.suite +++ b/benchmarks/gc.suite @@ -1 +1,17 @@ +# These benchmarks use the WebAssembly GC proposal (`struct`/`array` types, +# `ref.cast`, `i31`, etc.). Inclusion in this list does not guarantee that the +# benchmark is a GC-centric workload in the measured section, only that GC +# instructions are present. +# +# These files were gathered by running, from this directory: +# +# find . -name '*.wasm' | sort | while read f; do +# wasm-tools validate --features=-gc,-gc-types "$f" >/dev/null 2>&1 || echo "${f#./}" +# done + +dart-deltablue/dart-deltablue.wasm +hoot-earley-boyer/hoot-earley-boyer.wasm +java-navier-stokes/java-navier-stokes.wasm +kotlin-richards/kotlin-richards.wasm +ocaml-raytrace/ocaml-raytrace.wasm splay/splay.wasm diff --git a/benchmarks/simd.suite b/benchmarks/simd.suite index d27e8cd7..634127f9 100644 --- a/benchmarks/simd.suite +++ b/benchmarks/simd.suite @@ -2,16 +2,18 @@ # does not guarantee that the benchmark is a SIMD-centric workload in the # measured section, only that SIMD instructions are present. # -# These files were gathered by: -# - creating a `find-simd.sh` script containing: `wasm2wat --disable-simd $1 -# &>/dev/null || echo $1` -# - running the script over the benchmarks: `find . -name '*.wasm' | xargs -L 1 -# ./find-simd.sh` +# These files were gathered by running, from this directory: +# +# find . -name '*.wasm' | sort | while read f; do +# wasm-tools validate --features=-simd,-relaxed-simd "$f" >/dev/null 2>&1 || echo "${f#./}" +# done blake3-simd/benchmark.wasm hex-simd/benchmark.wasm intgemm-simd/benchmark.wasm meshoptimizer/benchmark.wasm +libsodium/libsodium-aead_aegis128l.wasm +libsodium/libsodium-aead_aegis256.wasm libsodium/libsodium-aead_chacha20poly1305.wasm libsodium/libsodium-aead_chacha20poly13052.wasm libsodium/libsodium-aead_xchacha20poly1305.wasm @@ -46,6 +48,7 @@ libsodium/libsodium-generichash3.wasm libsodium/libsodium-hash.wasm libsodium/libsodium-hash3.wasm libsodium/libsodium-kdf.wasm +libsodium/libsodium-kdf_hkdf.wasm libsodium/libsodium-keygen.wasm libsodium/libsodium-kx.wasm libsodium/libsodium-metamorphic.wasm