Add details on sparsity (#237)

* Add details on sparsity

* Sparsity tutorial

Author: gdalle

DifferentiationInterface/Project.toml

@@ -1,7 +1,7 @@
 name = "DifferentiationInterface"
 uuid = "a0c0ee7d-e4b9-4e03-894e-1c5f64a51d63"
 authors = ["Guillaume Dalle", "Adrian Hill"]
-version = "0.3.0"
+version = "0.3.1"
 
 [deps]
 ADTypes = "47edcb42-4c32-4615-8424-f2b9edc5f35b"

DifferentiationInterface/README.md

@@ -22,9 +22,10 @@ This package provides a backend-agnostic syntax to differentiate functions of th
 
 ## Features
 
-- First- and second-order operators (gradients, Jacobians, Hessians and [more](https://gdalle.github.io/DifferentiationInterface.jl/DifferentiationInterface/stable/overview/))
+- First- and second-order operators (gradients, Jacobians, Hessians and more)
 - In-place and out-of-place differentiation
 - Preparation mechanism (e.g. to create a config or tape)
+- Built-in sparsity handling
 - Thorough validation on standard inputs and outputs (numbers, vectors, matrices)
 - Testing and benchmarking utilities accessible to users with [DifferentiationInterfaceTest](https://github.com/gdalle/DifferentiationInterface.jl/tree/main/DifferentiationInterfaceTest)
 

DifferentiationInterface/docs/src/backends.md

@@ -26,7 +26,7 @@ const backend_examples = (
 
 checkmark(x::Bool) = x ? '✅' : '❌'
 unicode_check_available(backend) = checkmark(check_available(backend))
-unicode_check_hessian(backend)   = checkmark(check_hessian(backend))
+unicode_check_hessian(backend)   = checkmark(check_hessian(backend; verbose=false))
 unicode_check_twoarg(backend)    = checkmark(check_twoarg(backend))
 
 io = IOBuffer()

DifferentiationInterface/docs/src/overview.md

@@ -111,10 +111,9 @@ We offer two ways to perform second-order differentiation (for [`second_derivati
 
 ### Sparsity
 
-[ADTypes.jl](https://github.com/SciML/ADTypes.jl) provides `AutoSparse` to accelerate the computation of sparse Jacobians and Hessians:
-
-- for sparse Jacobians, wrap `AutoSparse` around a first-order backend.
-- for sparse Hessians, wrap `AutoSparse` around a [`SecondOrder`](@ref) backend.
+[ADTypes.jl](https://github.com/SciML/ADTypes.jl) provides [`AutoSparse`](@ref) to accelerate the computation of sparse Jacobians and Hessians.
+Just wrap it around any backend, with an appropriate choice of sparsity detector and coloring algorithm, and call `jacobian` or `hessian`: the result will be sparse.
+See the [tutorial section on sparsity](@ref sparsity-tutorial) for details.
 
 ### Split reverse mode
 

DifferentiationInterface/docs/src/tutorial.md

@@ -138,3 +138,49 @@ And you can run the same benchmarks to see what you gained (although such a smal
 In short, DifferentiationInterface.jl allows for easy testing and comparison of AD backends.
 If you want to go further, check out the [DifferentiationInterfaceTest.jl tutorial](https://gdalle.github.io/DifferentiationInterface.jl/DifferentiationInterfaceTest/dev/tutorial/).
 It provides benchmarking utilities to compare backends and help you select the one that is best suited for your problem.
+
+## [Handling sparsity](@id sparsity-tutorial)
+
+To compute sparse Jacobians or Hessians, you need three ingredients (read [this survey](https://epubs.siam.org/doi/10.1137/S0036144504444711) to understand why):
+
+1. A sparsity pattern detector
+2. A coloring algorithm
+3. An underlying AD backend
+
+ADTypes.jl v1.0 defines the [`AutoSparse`](@ref) wrapper, which brings together these three ingredients.
+At the moment, this new wrapper is not well-supported in the ecosystem, which is why DifferentiationInterface.jl provides the necessary objects to get you started:
+
+1. [`SymbolicsSparsityDetector`](@ref) (requires [Symbolics.jl](https://github.com/JuliaSymbolics/Symbolics.jl) to be loaded)
+2. [`GreedyColoringAlgorithm`](@ref)
+
+!!! warning
+    These objects are not part of the public API, so they can change unexpectedly between versions.
+
+!!! info
+    The symbolic backends have built-in sparsity handling, so `AutoSparse(AutoSymbolics())` and `AutoSparse(AutoFastDifferentiation())` do not need additional configuration.
+
+Here's an example:
+
+```@example tuto
+import Symbolics
+
+sparsity_detector = DifferentiationInterface.SymbolicsSparsityDetector()
+coloring_algorithm = DifferentiationInterface.GreedyColoringAlgorithm()
+dense_backend = AutoForwardDiff()
+
+sparse_backend = AutoSparse(dense_backend; sparsity_detector, coloring_algorithm)
+```
+
+See how the computed Hessian is sparse, whereas the underlying backend alone would give us a dense matrix:
+
+```@example tuto
+hessian(f, sparse_backend, x)
+```
+
+```@example tuto
+hessian(f, dense_backend, x)
+```
+
+The sparsity detector and coloring algorithm are called during the preparation step, which can be fairly expensive.
+If you plan to compute several Jacobians or Hessians with the same pattern but different input vectors, you should reuse the `extras` object created by `prepare_jacobian` or `prepare_hessian`.
+After preparation, the sparse computation itself will be much faster than the dense one, and require fewer calls to the function. 

DifferentiationInterface/src/utils/check.jl

@@ -26,13 +26,15 @@ Check whether `backend` supports second order differentiation by trying to compu
 !!! warning
     Might take a while due to compilation time.
 """
-function check_hessian(backend::AbstractADType)
+function check_hessian(backend::AbstractADType; verbose=true)
     try
         x = [1.0, 3.0]
         hess = hessian(sqnorm, backend, x)
         return isapprox(hess, [2.0 0.0; 0.0 2.0]; rtol=1e-3)
     catch exception
-        @warn "Backend $backend does not support hessian" exception
+        if verbose
+            @warn "Backend $backend does not support hessian" exception
+        end
         return false
     end
 end