Discuss which benchmark suite to run in benchmarks/README.md

benchmarks/README.md

@@ -1,12 +1,39 @@
 # Candidate Benchmark Programs
 
-This directory contains the candidate programs for the benchmark suite. They are
-candidates, not officially part of the suite yet, because we [intend][rfc] to
-record various metrics about the programs and then run a principal component
-analysis to find a representative subset of candidates that doesn't contain
-effectively duplicate workloads.
+This directory contains our various benchmark programs. Not all are included in
+a benchmark suite.
 
-[rfc]: https://github.com/bytecodealliance/rfcs/pull/4
+## Which Benchmark Programs Should I Run?
+
+**99% of the time, you want to run the `default.suite`.**
+
+The default suite is derived from a [principal-component analysis
+(PCA)](https://github.com/bytecodealliance/sightglass/blob/main/scripts/pca.R)
+to maximize the amount of benchmark variation covered while avoiding
+effectively-redundant workloads (those that exercise the same kinds of code that
+an existing benchmark already exercises) and conserving developer
+time. `default.suite` is an alias of `pca-20-min.suite`, and completes a full
+run with default options in an estimated ~20 minutes.
+
+For fast iteration locally, but not as a replacement for a full `default.suite`
+run, you can also use one of the smaller PCA-based subsets of `default.suite`:
+
+* `pca-5-min.suite`: completes in an estimated ~5 minutes
+* `pca-10-min.suite`: completes in an estimated ~10 minutes
+
+If you are specifically interested in a certain kind of program, you can also
+try one of these suites:
+
+* `gc.suite`: the set of benchmarks that use Wasm GC
+* `simd.suite`: the set of benchmarks that use SIMD
+
+Finally, while you *can* also run `all.suite` -- which contains every benchmark
+program in this directory -- be aware that it contains many
+effectively-redundant workloads (e.g. many of the libsodium benchmarks exercise
+basically the same kind of code) and therefore its geomean is lopsided, weighing
+these redundant workloads heavier than workloads that appear only once in the
+corpus, and running it takes an exceedingly long time. We do not recommend
+benchmarking `all.suite`.
 
 ## Building
 

benchmarks/gc.suite

@@ -1 +1,17 @@
+# These benchmarks use the WebAssembly GC proposal (`struct`/`array` types,
+# `ref.cast`, `i31`, etc.). Inclusion in this list does not guarantee that the
+# benchmark is a GC-centric workload in the measured section, only that GC
+# instructions are present.
+#
+# These files were gathered by running, from this directory:
+#
+#     find . -name '*.wasm' | sort | while read f; do
+#         wasm-tools validate --features=-gc,-gc-types "$f" >/dev/null 2>&1 || echo "${f#./}"
+#     done
+
+dart-deltablue/dart-deltablue.wasm
+hoot-earley-boyer/hoot-earley-boyer.wasm
+java-navier-stokes/java-navier-stokes.wasm
+kotlin-richards/kotlin-richards.wasm
+ocaml-raytrace/ocaml-raytrace.wasm
 splay/splay.wasm

benchmarks/simd.suite

@@ -2,16 +2,18 @@
 # does not guarantee that the benchmark is a SIMD-centric workload in the
 # measured section, only that SIMD instructions are present.
 #
-# These files were gathered by:
-# - creating a `find-simd.sh` script containing: `wasm2wat --disable-simd $1
-#   &>/dev/null || echo $1`
-# - running the script over the benchmarks: `find . -name '*.wasm' | xargs -L 1
-#   ./find-simd.sh`
+# These files were gathered by running, from this directory:
+#
+#     find . -name '*.wasm' | sort | while read f; do
+#         wasm-tools validate --features=-simd,-relaxed-simd "$f" >/dev/null 2>&1 || echo "${f#./}"
+#     done
 
 blake3-simd/benchmark.wasm
 hex-simd/benchmark.wasm
 intgemm-simd/benchmark.wasm
 meshoptimizer/benchmark.wasm
+libsodium/libsodium-aead_aegis128l.wasm
+libsodium/libsodium-aead_aegis256.wasm
 libsodium/libsodium-aead_chacha20poly1305.wasm
 libsodium/libsodium-aead_chacha20poly13052.wasm
 libsodium/libsodium-aead_xchacha20poly1305.wasm
@@ -46,6 +48,7 @@ libsodium/libsodium-generichash3.wasm
 libsodium/libsodium-hash.wasm
 libsodium/libsodium-hash3.wasm
 libsodium/libsodium-kdf.wasm
+libsodium/libsodium-kdf_hkdf.wasm
 libsodium/libsodium-keygen.wasm
 libsodium/libsodium-kx.wasm
 libsodium/libsodium-metamorphic.wasm