Refactor the test suite

[kellertuer]

I noticed that all things I had in mind for today I already did concerning stopping criteria, so I at least tried a new scheme for the tests

There is now a test suite and tests have their own project (they are no longer in extras and such) so that the test suite can stay a private package but serve to provide e.g. a dummy algorithm and such.

Maybe to actually do something for stopping criteria, since the package even has the ones I had in mind to print here ( StpoWhenAll, StopWhenAny, StopAfterIteration, StopAfter (a time period))

One could write

• a small factory, that would take a stopping criterion and kwargs... and would add a | StopAfterIteration(maxiter) is the maxiter keyword is present.
• one could write a stopping criterion that allows to stop on a keypress (Ctrl+S for stop? Could be confused with saving ;), but to not intervene with Ctrl+c by default)

What do you think? Besides that I was surprised that at some time I even finished what I had in mind to do with stopping criteria today, even a bit more advanced, since I forgot about SC and SCState already. Neat!

Let me know what you think @lkdvos, I hope you like the test suite idea to store common things different tests need.

[kellertuer]

Oh, that is an interesting error, locally it works to have an unregistered package with source?

[edit:] oh it seems that does not yet work on 1.10 - on 1.12 (and 1.11) the CI at least only errors since the package environment was not instantiated.

Then I have to check what best to do on 1.10 for the test suite.

[lkdvos]

I absolutely love the test suite idea, it's something I tend to implement myself most of the time as well. I usually don't deal with the Pkg issues and just include it as a local module, so simply include("test/Setup.jl"); using .TestSetup is my workflow. I prefer your approach though, it looks cleaner, if we can make that work on 1.10?

I took the liberty of rebasing the tests to better see the diff, hope you don't mind (and hopeI didn't screw anything up with the rebase)

[kellertuer]

I found a way to make it work on 1.10 after asking on slack

using Pkg;
Pkg.activate(@__DIR__)
if VERSION < v"1.11"
    Pkg.add(path = (@__DIR__) * "/../", subdir = "test/ManifoldsTestSuite.jl")
end
Pkg.resolve()
using ManifoldsTestSuite, Manifolds, LinearAlgebra, Test, Random

worked locally for the other test suite I already started. Whether that is nice or a bit over-engineered and a plain module does the job as well... I leave that to others.

The snippet above should work here (might require the package to be registered, not so sure). Will adapt soon :) though for now the test suite will mainly provide dummy types I think.

[lkdvos]

I like it, I don't think that's too over engineered at all.

The annoying thing with the local includes is that this works fine as long as the different tests are separated in their own module (ie @safetestset or something similar), but you can run into issues when running things locally and accidentally including the same module twice from different locations, in which case Julia doesnt overwrite but rather creates a new module and sometimes gives you naming clashes that way.

[kellertuer]

Well, the current solution I can not get to work on 1.10, because even with develop instead of add it either is annoyed that AlgorithmsInterface is not registered or that its test suite is not registered.

But the example above works fine (Manopt.jl is registered of course). So for 1.10 / its I am not yet sure how to trick that best here.

[kellertuer]

Oh the last commit worked for me locally with the two development mode adds but might also need one further resolve or update somewhere.

[kellertuer]

The problem on 1.10 is that this appears

JuliaLang/Pkg.jl#1585

once one does a separate Project.toml in the test folder and has a bit of specific package versions. On 1.11+ and especially with sources this no longer happens.

[kellertuer]

This can not be solved this way as long as we support 1.10, which I definitely want to continue to support. So I'll close this for now and focus on something else first.