feat: load packages with compiled code, and reload earlier-loaded packages

AGENTS.md

@@ -42,6 +42,8 @@ This `AGENT.md` file is read by every agent session. !!!Keep them high-signal!!!
 
 - **`.loadfast.cache`** is a module-level environment (`parent = emptyenv()`) keyed by `normalizePath(path)`. Each entry stores the namespace env, `pkg_name`, file hashes, `renv.lock` hash baseline, the S3 methods matrix, `import_pkgs` (names of packages this package imports), `needs_full` (NULL, or a reason string set by dependency invalidation), any registered reload files, and the pending reload message to emit on the next load.
 - **Dependency invalidation**: after any load that re-sources files, `.loadfast.invalidate_dependents()` flags every cached package that (transitively) imports the reloaded package by setting `needs_full`. A flagged package's next `load_fast()` is upgraded to a full reload, and the full-load path first recursively reloads flagged *dependencies* (via `.loadfast.load()`, guarded by the `.loadfast.state$stack` path stack), so one call on the top of a chain converges the whole chain. No-change loads do not invalidate.
+- **Earlier-loaded packages**: `.loadfast.state$loaded` holds package paths in the order of their first full load. `load_fast()` (not the internal `.loadfast.load()`) first calls `.loadfast.reload_earlier()`, which runs `.loadfast.load(report_unchanged = FALSE)` on every path loaded before the target. It skips a package whose namespace was replaced since and one with the target's package name.
+- **Compiled code** (`.loadfast.load_compiled()`): a package with `src/` never goes through the sourcing path. Each change is built with `R CMD INSTALL` into a new `tempfile()` library and loaded with `library()`. A new path is required: the dynamic loader returns the already-loaded library for a path it knows, so a rebuild in place would run the old code. The old library is never unloaded, because unloading crashes R when a live external pointer's finalizer runs. Change detection hashes `DESCRIPTION`, `NAMESPACE`, `R/` and `src/`, and prunes `.Rbuildignore` paths while walking, so Cargo's `target/` is never listed.
 - **Depends**: `.loadfast.parse_depends()` reads `Depends` from DESCRIPTION (minus `R (...)`), and the full-load path attaches each with `library()` if not already attached; a missing Depends package is a clear error.
 - **Teardown** (`.loadfast.teardown()`): detach + `unloadNamespace()`. When the namespace is imported by other loaded namespaces, `unloadNamespace()` errors — the fallback runs the old namespace's `.onUnload` best-effort and force-removes it from the registry via `.loadfast.ns_unregister()`. A failed full load cleans up after itself (unregisters the half-built ns, detaches) via an `on.exit` guard.
 - **Package identity**: if the `Package:` field changes in place for a cached path, the old name is detached/unloaded and the cache entry dropped before the full load of the new name.
@@ -70,6 +72,8 @@ This `AGENT.md` file is read by every agent session. !!!Keep them high-signal!!!
   - **Stage 5**: Build ad-hoc packages (distinct class names, so nothing collides) to test cross-package fidelity: `exportPattern()` populating exports and `pkg::fn` resolving (5a), `pkg::missing` giving a proper not-exported error rather than the `lazydata` internal (5b), S3 dispatch across package boundaries for both base and package-defined generics (5c), and `exportClasses()`/`exportMethods()` + `importClassesFrom()`/`importMethodsFrom()` for S4 (5d).
   - **Stage 6**: Multi-package dependency invalidation: direct importer auto-upgraded to full reload (6a), transitive chain convergence with recursive dependency reload (6b), no-change loads do not invalidate (6c), whole-namespace `import()` invalidation (6d), S4 class redefinition propagating to importers (6e), and the forced-unregister path when `unloadNamespace()` refuses (including `.onUnload` execution) (6f).
   - **Stage 7**: Production hardening: `Depends` attachment + missing-Depends error (7a), rename-in-place cleanup (7b), failed-full-load cleanup + recovery (7c), namespace version from DESCRIPTION (7d), Collate order on incremental re-source (7e), re-entrance guard (7f), verbose timing logs (7g).
+  - **Stage 8**: A package with C code in `src/`: build, no rebuild without change, `.Rbuildignore` pruning, rebuild on C and R changes, failed build keeps the old build (8a). Earlier-loaded packages, R-only and compiled, are reloaded by a later package's `load_fast()`, but not the other way round (8b). Needs a C compiler.
+- The harness loads all its packages into one session, so every `load_fast()` call also reloads the earlier packages whose files a test changed without reloading them. A new test that leaves an edited package unloaded can change what later checks see.
 - Stage 2 always triggers a full load (different path from Stage 1 = cache miss). Stage 3 exercises the incremental path for `load_fast()` on a stable temp path while mutating files in place. The multi-package stages (3e, 3h) call `strip_s3_fixture()` on their copies so the shared `temperature` S3 fixture from the baseline does not register colliding methods across renamed packages.
 
 ## R namespace machinery gotchas

R/loadfast.R

@@ -1,6 +1,8 @@
 .loadfast.cache <- new.env(parent = emptyenv())
 .loadfast.state <- new.env(parent = emptyenv())
 .loadfast.state$stack <- character(0)
+# Package paths in the order of their first full load in this session.
+.loadfast.state$loaded <- character(0)
 
 # R's namespace registry has no public API for dev loaders: the base wrappers
 # around registerNamespace/unregisterNamespace are not exported, and rlang's
@@ -41,6 +43,17 @@
 #' against a stale snapshot. A full reload also reloads flagged dependencies
 #' first, in dependency order.
 #'
+#' Before it loads `path`, `load_fast()` reloads the packages that it loaded
+#' earlier in the session, in the order it first loaded them, if their files
+#' changed. So after you edit a dependency, one `load_fast()` call in the
+#' package you work on picks up the edit.
+#'
+#' A package with a `src/` directory is built with `R CMD INSTALL` into a new
+#' temporary library, and loaded from there. This happens on the first call
+#' and whenever a file in `DESCRIPTION`, `NAMESPACE`, `R/` or `src/` changed.
+#' Paths that `.Rbuildignore` lists do not count as changes. The library of
+#' the previous build stays loaded, so objects that point into it stay valid.
+#'
 #' Packages listed in the `Depends` field of `DESCRIPTION` are attached to
 #' the search path during a full load, mirroring `library()` semantics.
 #'
@@ -70,10 +83,31 @@ load_fast <- function(path = ".", helpers = TRUE, attach_testthat = NULL, full =
   if (length(.loadfast.state$stack) > 0L) {
     stop("load_fast() re-entrance detected -- a sourced file is calling load_fast()")
   }
+  .loadfast.reload_earlier(.loadfast.find_package_root(path), verbose)
   .loadfast.load(path, helpers = helpers, attach_testthat = attach_testthat, full = full, verbose = verbose)
 }
 
-.loadfast.load <- function(path = ".", helpers = TRUE, attach_testthat = NULL, full = FALSE, verbose = FALSE) {
+# The multi-package workflow loads dependencies first, so the packages loaded
+# before `abs_path` are the ones it can depend on. A package whose namespace was
+# replaced since, for example by library(), is left alone, and so is one with
+# the name of the package at `abs_path`, because loading that replaces it.
+.loadfast.reload_earlier <- function(abs_path, verbose) {
+  pkg_name <- read.dcf(file.path(abs_path, "DESCRIPTION"), fields = "Package")[1L, 1L]
+  earlier <- .loadfast.state$loaded
+  position <- match(abs_path, earlier)
+  if (!is.na(position)) earlier <- earlier[seq_len(position - 1L)]
+  for (key in earlier) {
+    if (!exists(key, envir = .loadfast.cache, inherits = FALSE) || !dir.exists(key)) next
+    entry <- .loadfast.cache[[key]]
+    if (identical(entry$pkg_name, unname(pkg_name))) next
+    if (!isNamespaceLoaded(entry$pkg_name) || !identical(asNamespace(entry$pkg_name), entry$ns_env)) next
+    .loadfast.load(key, helpers = FALSE, attach_testthat = FALSE, verbose = verbose, report_unchanged = FALSE)
+  }
+  invisible(NULL)
+}
+
+.loadfast.load <- function(path = ".", helpers = TRUE, attach_testthat = NULL, full = FALSE, verbose = FALSE,
+                           report_unchanged = TRUE) {
   if (verbose) {
     .t0 <- proc.time()["elapsed"]
     .t_last <- .t0
@@ -118,6 +152,13 @@ load_fast <- function(path = ".", helpers = TRUE, attach_testthat = NULL, full =
   if (!nzchar(pkg_name)) stop("No valid 'Package' field found in DESCRIPTION")
 
   pkg_env_name <- paste0("package:", pkg_name)
+
+  if (dir.exists(file.path(abs_path, "src"))) {
+    return(.loadfast.load_compiled(
+      abs_path, pkg_name, path_display, helpers, attach_testthat, full, report_unchanged
+    ))
+  }
+
   loaded_pkg_path <- .loadfast.loaded_package_path(pkg_name)
 
   if (!is.null(loaded_pkg_path) && !identical(loaded_pkg_path, abs_path)) {
@@ -182,7 +223,10 @@ load_fast <- function(path = ".", helpers = TRUE, attach_testthat = NULL, full =
     message("Full reload of '", pkg_name, "': ", needs_full_reason, ".")
   }
 
+  # A package that had src/ before was installed, and R locks an installed
+  # namespace, so its files cannot be sourced into it.
   can_incremental <- !is.null(cached) &&
+    is.null(cached$library_dir) &&
     is.null(needs_full_reason) &&
     !is.null(active_ns_env) &&
     identical(cached$ns_env, active_ns_env) &&
@@ -238,7 +282,7 @@ load_fast <- function(path = ".", helpers = TRUE, attach_testthat = NULL, full =
         registered_reload_files = character(0),
         pending_reload_message = NULL
       )
-      message("No changes in ", r_dir_display, ".")
+      if (report_unchanged) message("No changes in ", r_dir_display, ".")
       .loadfast.source_helpers(abs_path, pkg_env, helpers, attach_testthat, pkg_name)
       .timer("TOTAL (no-change)")
       return(invisible(ns_env))
@@ -594,13 +638,99 @@ load_fast <- function(path = ".", helpers = TRUE, attach_testthat = NULL, full =
     pending_reload_message = NULL
   )
   .loadfast.invalidate_dependents(pkg_name, abs_path)
+  .loadfast.state$loaded <- union(.loadfast.state$loaded, abs_path)
 
   full_load_ok <- TRUE
   message("Load ", length(r_files), " file(s) from ", r_dir_display, ".")
   .timer("TOTAL (full load)")
   invisible(ns_env)
 }
 
+# R runs compiled code only from a shared library that dyn.load() loaded, and
+# the operating system does not load a rebuilt library again from a path it has
+# already loaded. So every build goes into a new temporary library. The old
+# library stays loaded: unloading it would crash R when an object that still
+# points into it is used or garbage collected.
+.loadfast.load_compiled <- function(abs_path, pkg_name, path_display, helpers, attach_testthat, full,
+                                    report_unchanged) {
+  hashes <- tools::md5sum(.loadfast.build_inputs(abs_path))
+  cached <- if (exists(abs_path, envir = .loadfast.cache, inherits = FALSE)) .loadfast.cache[[abs_path]]
+  is_current <- !isTRUE(full) &&
+    !is.null(cached) &&
+    is.null(cached$needs_full) &&
+    identical(hashes, cached$hashes) &&
+    isNamespaceLoaded(pkg_name) &&
+    identical(asNamespace(pkg_name), cached$ns_env)
+
+  if (!is_current) {
+    library_dir <- .loadfast.install(abs_path, path_display)
+    .loadfast.teardown(pkg_name, abs_path)
+    library(pkg_name, lib.loc = library_dir, character.only = TRUE, warn.conflicts = FALSE)
+    if (!is.null(cached$library_dir)) unlink(cached$library_dir, recursive = TRUE)
+    ns_env <- asNamespace(pkg_name)
+    .loadfast.cache[[abs_path]] <- list(
+      ns_env = ns_env,
+      pkg_name = pkg_name,
+      hashes = hashes,
+      import_pkgs = setdiff(names(getNamespaceImports(ns_env)), "base"),
+      needs_full = NULL,
+      library_dir = library_dir
+    )
+    .loadfast.invalidate_dependents(pkg_name, abs_path)
+    .loadfast.state$loaded <- union(.loadfast.state$loaded, abs_path)
+    message("Load ", pkg_name, " from ", path_display, ", built with R CMD INSTALL.")
+  } else if (report_unchanged) {
+    message("No changes in ", path_display, ".")
+  }
+
+  pkg_env <- as.environment(paste0("package:", pkg_name))
+  .loadfast.source_helpers(abs_path, pkg_env, helpers, attach_testthat, pkg_name)
+  invisible(asNamespace(pkg_name))
+}
+
+# The build of a package with src/ streams its output, because the first build
+# of a large Rust or C++ package can take minutes.
+.loadfast.install <- function(abs_path, path_display) {
+  library_dir <- tempfile("loadfast-")
+  dir.create(library_dir)
+  # --use-vanilla keeps the child R from reading .Rprofile, which can switch
+  # libraries (renv does). R_LIBS gives it the libraries of this session instead.
+  old_r_libs <- Sys.getenv("R_LIBS", unset = NA)
+  Sys.setenv(R_LIBS = paste(.libPaths(), collapse = .Platform$path.sep))
+  on.exit(if (is.na(old_r_libs)) Sys.unsetenv("R_LIBS") else Sys.setenv(R_LIBS = old_r_libs), add = TRUE)
+  status <- system2(file.path(R.home("bin"), "R"), c(
+    "CMD", "INSTALL", "--use-vanilla", "--no-test-load", "--no-staged-install", "--no-byte-compile",
+    paste0("--library=", shQuote(library_dir)), shQuote(abs_path)
+  ))
+  if (status != 0L) {
+    unlink(library_dir, recursive = TRUE)
+    stop("R CMD INSTALL failed for ", path_display, ". Its output is above.", call. = FALSE)
+  }
+  library_dir
+}
+
+# A build directory, such as Cargo's src/rust/target/, can hold many thousands
+# of files. Paths that .Rbuildignore lists are not entered, and neither are the
+# object files and libraries that R CMD INSTALL writes into src/.
+.loadfast.build_inputs <- function(abs_path) {
+  ignore_file <- file.path(abs_path, ".Rbuildignore")
+  ignore <- c(
+    "^src/.*\\.(o|so|dll|dylib)$",
+    if (file.exists(ignore_file)) readLines(ignore_file, warn = FALSE)
+  )
+  ignore <- ignore[nzchar(trimws(ignore))]
+  is_ignored <- function(paths) {
+    Reduce(`|`, lapply(ignore, grepl, x = paths, ignore.case = TRUE, perl = TRUE), logical(length(paths)))
+  }
+  list_dir <- function(dir) {
+    paths <- file.path(dir, list.files(file.path(abs_path, dir), all.files = TRUE, no.. = TRUE))
+    paths <- paths[!is_ignored(paths)]
+    is_dir <- dir.exists(file.path(abs_path, paths))
+    c(paths[!is_dir], unlist(lapply(paths[is_dir], list_dir)))
+  }
+  file.path(abs_path, sort(c("DESCRIPTION", "NAMESPACE", list_dir("R"), list_dir("src"))))
+}
+
 #' Register one or more files for reload on the next `load_fast()` call
 #'
 #' @param path Package root path.

README.md

@@ -88,8 +88,11 @@ snapshot taken at its own load time. `load_fast()` tracks these relationships:
   In a chain `C -> B -> A`, editing and reloading `A` followed by a single
   `load_fast()` of `C` rebuilds `B` and `C` against the fresh namespaces.
 
-So the workflow is simply: reload the package you edited, then reload whatever
-package you are working in — everything in between converges automatically.
+You do not even need the first step. Before `load_fast()` loads a package, it
+reloads the packages that it loaded earlier in the session, in the order it
+first loaded them, if their files changed. So the workflow is: load each
+package once, dependencies first. After that, edit any of them and call
+`load_fast()` in the package you work on.
 
 Reloading a dependency that other loaded packages import cannot use
 `unloadNamespace()` (R refuses to unload an imported namespace); `load_fast()`
@@ -100,6 +103,27 @@ class is imported elsewhere.
 > Earlier versions failed here with `Error: object 'lazydata' not found` when a
 > loaded package was accessed with `::`. That is fixed.
 
+## Packages with compiled code
+
+`load_fast()` also loads a package with a `src/` directory, for example C,
+C++ or Rust code. R runs compiled code only from a shared library, and the
+operating system does not load a rebuilt library again from a path that is
+already loaded. So `load_fast()` builds the package with `R CMD INSTALL` into a
+new temporary library, and loads it from there with `library()`. It streams the
+build output.
+
+The build runs on the first call and whenever a file in `DESCRIPTION`,
+`NAMESPACE`, `R/` or `src/` changed. Paths that `.Rbuildignore` lists do not
+count. List build directories there, such as Cargo's `^src/rust/target$`, so
+that `load_fast()` does not hash their files on every call. The make and cargo
+caches in `src/` make a rebuild after a small change fast.
+
+`load_fast()` does not unload the library of the previous build. Objects that
+point into it keep working, and they keep running the old code. The package's
+functions run the new code, as long as they call native code through
+registered routines (`useDynLib(pkg, .registration = TRUE)`). A call by name,
+such as `.Call("fn", PACKAGE = "pkg")`, can keep running the old code.
+
 ## Editor setup
 
 ### RStudio
@@ -143,11 +167,12 @@ You can bind `loadfast::load_fast()` in the Zed keymap:
 
 Testing happens at two levels:
 
-1. **Behavioral harness** (`test_loadfast.R`): ~300 checks across seven stages
+1. **Behavioral harness** (`test_loadfast.R`): ~300 checks across eight stages
    covering full loads, incremental reloads, cross-file and cross-package
-   dependencies, S3/S4/R6 fidelity, multi-package dependency invalidation, and
+   dependencies, S3/S4/R6 fidelity, multi-package dependency invalidation,
    production hardening (failed-load recovery, rename-in-place, `Depends`
-   attachment, `Collate` ordering, re-entrance).
+   attachment, `Collate` ordering, re-entrance), and packages with compiled
+   code. Stage 8 needs a C compiler.
 2. **`testthat` suite** (`tests/testthat/`): self-contained smoke and error-path
    tests that run against the *installed* package during `R CMD check`.
 

TECHNICAL_DEBT.md

@@ -108,6 +108,22 @@ the class of subtle S4/S3 state this package tries not to reimplement.
 changes these conventions (unlikely; they are decades old), stages 5d/6e of the
 harness will catch it.
 
+### 7. A rebuilt compiled package leaves the old library loaded
+Each build of a package with `src/` goes into a new temporary library, and the
+previous library is never unloaded. Unloading it would crash R as soon as an
+external pointer from the old build is used or finalized. The price: every
+build stays mapped until R exits, and a native call by name
+(`.Call("fn", PACKAGE = "pkg")`) can resolve to the old library through R's
+symbol cache. Calls through registered routines, which savvy, extendr and
+current Rcpp generate, always reach the new library.
+
+A whole `R CMD INSTALL` per change is also slower than compiling `src/` and
+sourcing `R/` the way `pkgload` does. For packages whose time goes into the
+compiler, the difference is a few seconds.
+
+**Current rule**
+- Restart R if a session has rebuilt a large compiled package many times
+
 ## Notes for future reviewers
 
 - Most of the implementation complexity is justified by R namespace machinery, especially S4 behavior and imports metadata shape.