Using `ncolors(::SparseJacobianPrep)` / `ncolors(::SparseHessianPrep)` publicly ?

[franckgaga]

Hello!

In MPC.jl, I have a getinfo methods that return additional information about an optimum, to troubleshoot non-converging optimization or performance issues. It includes the Jacobians of the constraints and the Hessian of the Langrangian at the optimum.

I understand that the evaluated number of colors at preparation can give some information on the performances of sparse differentiation. Hence, I would like to include the number of colors for all the sparse differentiation matrices in getinfo. Is it safe to use the non-exported ncolors(::SparseJacobianPrep) and ncolors(::SparseHessianPrep) in getinfo ? If yes, why it's not in the documentation and there is no docstring ? If no, what is the safest way to extract the number of colors after preperation ?

edit: also a suggestion to simplify the user life: you could introduce a general fallback when the preperation backend is not sparse. It could return either 1 or nothing. Let me know what you think.

Thanks

[adrhill]

ncolors is part of SparseMatrixColorings.jl's stable API.
If you add a SMC compat entry to your package, I'd say it's safe to use.

If yes, why it's not in the documentation and there is no docstring ?

From my perspective, mostly due to philosophical reasons. Post-hoc sparse AD requires of 4 parts:

• (a) a sparsity detection package (e.g., SparseConnectivityTracer, Symbolics)
• (b) a matrix coloring package (currently only SparseMatrixColorings)
• (c) an AD backend (e.g., Enzyme, ForwardDiff, etc.)
• (d) DI's decompression of the sparse matrix

The original idea was to keep DI as modular and light-weight as possible for steps (a-c). DI loads SMC via a package extension, so it's not part of the "main package".
However, in practice, there is no alternative to using SMC. The cleanest and most future-proof solution would be to make ncolors part of ADTypes.jl's coloring API, which DI adheres to:
https://docs.sciml.ai/ADTypes/stable/#Coloring-algorithm

[gdalle]

Hey there!
Adrian is right, this is safe to use, but not obviously so because the question is about a method of the form SMC.ncolors(::DI.Prep). The place where I meant to document this is the sparsity tutorial, where I say

Some result analysis functions from SparseMatrixColorings.jl can help you figure out what the preparation contains.

but we could add a list of functions there. And we should probably extend the SMC docstrings to mention how they behave on DI objects. Do you want to open a PR?

[franckgaga]

Alright yes I will open a PR tommorow with new docstrings.

[gdalle]

Here are the relevant methods that we extend from SMC:

https://github.com/JuliaDiff/DifferentiationInterface.jl/blob/main/DifferentiationInterface/ext/DifferentiationInterfaceSparseMatrixColoringsExt/DifferentiationInterfaceSparseMatrixColoringsExt.jl

[adrhill]

How do you feel about upstreaming ncolors to ADTypes.jl, @gdalle ?

[gdalle]

It's a bit ambiguous due to the possible existence of neutral colors, but we could do it along with SciML/ADTypes.jl#69

[gdalle]

Actually no because ncolors is not defined on vectors, it is defined on result types from SMC which are not public. So upstreaming it to ADTypes would be meaningless