-)</code></pre><p>At the moment, mixed mode tends to work best when the <a href="https://gdalle.github.io/SparseMatrixColorings.jl/stable/api/#SparseMatrixColorings.GreedyColoringAlgorithm"><code>GreedyColoringAlgorithm</code></a> is provided with a <a href="https://gdalle.github.io/SparseMatrixColorings.jl/stable/api/#SparseMatrixColorings.RandomOrder"><code>RandomOrder</code></a> instead of the usual <a href="https://gdalle.github.io/SparseMatrixColorings.jl/stable/api/#SparseMatrixColorings.NaturalOrder"><code>NaturalOrder</code></a>.</p><h2 id="Batch-mode"><a class="docs-heading-anchor" href="#Batch-mode">Batch mode</a><a id="Batch-mode-1"></a><a class="docs-heading-anchor-permalink" href="#Batch-mode" title="Permalink"></a></h2><h3 id="Multiple-tangents"><a class="docs-heading-anchor" href="#Multiple-tangents">Multiple tangents</a><a id="Multiple-tangents-1"></a><a class="docs-heading-anchor-permalink" href="#Multiple-tangents" title="Permalink"></a></h3><p>The <a href="../../api/#DifferentiationInterface.jacobian"><code>jacobian</code></a> and <a href="../../api/#DifferentiationInterface.hessian"><code>hessian</code></a> operators compute matrices by repeatedly applying lower-level operators (<a href="../../api/#DifferentiationInterface.pushforward"><code>pushforward</code></a>, <a href="../../api/#DifferentiationInterface.pullback"><code>pullback</code></a> or <a href="../../api/#DifferentiationInterface.hvp"><code>hvp</code></a>) to a set of tangents. The tangents usually correspond to basis elements of the appropriate vector space. We could call the lower-level operator on each tangent separately, but some packages (<a href="https://github.com/JuliaDiff/ForwardDiff.jl">ForwardDiff.jl</a> and <a href="https://github.com/EnzymeAD/Enzyme.jl">Enzyme.jl</a>) have optimized implementations to handle multiple tangents at once.</p><p>This behavior is often called "vector mode" AD, but we call it "batch mode" to avoid confusion with Julia's <code>Vector</code> type. As a matter of fact, the optimal batch size <span>$B$</span> (number of simultaneous tangents) is usually very small, so tangents are passed within an <code>NTuple</code> and not a <code>Vector</code>. When the underlying vector space has dimension <span>$N$</span>, the operators <code>jacobian</code> and <code>hessian</code> process <span>$\lceil N / B \rceil$</span> batches of size <span>$B$</span> each.</p><h3 id="Optimal-batch-size"><a class="docs-heading-anchor" href="#Optimal-batch-size">Optimal batch size</a><a id="Optimal-batch-size-1"></a><a class="docs-heading-anchor-permalink" href="#Optimal-batch-size" title="Permalink"></a></h3><p>For every backend which does not support batch mode, the batch size is set to <span>$B = 1$</span>. But for <a href="https://sciml.github.io/ADTypes.jl/stable/#ADTypes.AutoForwardDiff"><code>AutoForwardDiff</code></a> and <a href="https://sciml.github.io/ADTypes.jl/stable/#ADTypes.AutoEnzyme"><code>AutoEnzyme</code></a>, more complicated rules apply. If the backend object has a pre-determined batch size <span>$B_0$</span>, then we always set <span>$B = B_0$</span>. In particular, this will throw errors when <span>$N < B_0$</span>. On the other hand, without a pre-determined batch size, we apply backend-specific heuristics to pick <span>$B$</span> based on <span>$N$</span>.</p><script type="module">import mermaid from 'https://cdn.jsdelivr.net/npm/mermaid@11/dist/mermaid.esm.min.mjs';
0 commit comments