Detect and warn if the samples are not normally distributed #142

Activity

1. fitzgen

   added this to the Statistically Sound and Rigorous Analysis milestone on Jun 4, 2021
2. fitzgen commented on Jun 14, 2021

   

   fitzgen

   on Jun 14, 2021

   MemberAuthor

   More actions

   I don't really trust myself to implement Shapiro-Wilk myself, but I whipped up this script in R that checks the normality of count for each (arch, engine, wasm, phase, event) grouping:

   #!/usr/bin/env Rscript
   
   library(dplyr)
   
   args = commandArgs(trailingOnly = TRUE)
   
   if (length(args) != 1) {
       cat("Error: must supply path to sightglass CSV data\n")
       quit(status = 1)
   }
   
   data <- read.table(args[1], header = TRUE, sep = ",", stringsAsFactors = TRUE)
   
   grouped <- data %>% group_by(arch, engine, wasm, phase, event)
   
   error_count <- 0
   
   add_error <- function (msg, key) {
       error_count <- error_count + 1
       cat(msg)
       cat(paste("    arch =", key$arch, "\n"))
       cat(paste("    engine =", key$engine, "\n"))
       cat(paste("    wasm =", key$wasm, "\n"))
       cat(paste("    phase =", key$phase, "\n"))
       cat(paste("    event =", key$event, "\n"))
       cat("\n")
   }
   
   ### Normality
   
   check_normality <- function (rows, key) {
       normality <- shapiro.test(rows$count)
       if (normality$p.value < 0.05) {
           add_error("Error: not normally distributed:\n", key)
       }
   }
   
   grouped %>% group_walk(check_normality)
   
   quit(status = error_count)

   So far, even when the benchmark results' distributions are definitely not bi-modal, they still aren't normal. Nor is the log of those results normal.

   I think this technically invalidates the results of our t-test / effect size analyses, since these things assume that the data is normally distributed...
3. fitzgen commented on Oct 15, 2026

   

   fitzgen

   on Oct 15, 2026

   MemberAuthor

   More actions

   As of #345 we no longer rely on normal distributions (Mann-Whitney U-test instead of Welch's t-test)
4. fitzgen

   closed this as completedon Oct 15, 2026