Should the JIT compiler options be environment flags?

[schnellerhase]

Currently DOLFINx's JIT options are managed through a hierarchy respecting list of .json configuration files, which take precedence based on their location (local over global). This comes with a few key characteristics

• configuration is execution context based
• overwriting default options hierarchically is supported
• they are platform specific: an altered compiler/platform requires different options, but the program remains unchanged

This is exactly what environment variables provide.

At the same time the current approach comes with some limitations or non-consistent behaviour:

• the cache file is respecting the environment variable XDG_CACHE_HOME, however is overwritten through the config.
• Automated/CI/containerised setups require configuration through custom config files or programmatic injection in user code to provide platform specific compiler options.
• Many of the third party dependencies are fully env var configurable - most notably PETSc.

Therefore the suggestion or question: Should the JIT compiler options be environment flags?

This would simplify the present code around options management, provide configurability through the standard interface (both environmental, without I/o, and programmatically!), which simplifies interaction with other software packages, and enable platform specific execution.

The same question might be raised for form compiler options in a second instance.

[jhale]

No, we deliberately removed environmental configuration and I don't support adding it back.

Could you be more precise on what you cannot achieve with the current system? Then we could instead tweak what we have.

1. the cache file is respecting the environment variable XDG_CACHE_HOME, however is overwritten through the config.
2. Automated/CI/containerised setups require configuration through custom config files or programmatic injection in user code to provide platform specific compiler options.
3. Many of the third party dependencies are fully env var configurable - most notably PETSc.

1. Is this just a bug? Please explain further.
2. I agree with the statement but I don't follow - use echo 'options' > options.json in the CI e.g. https://github.com/FEniCS/dolfinx/blob/main/.github/workflows/ci.yml#L190. I don't see the need for programmatic injection into user code for platform specific options - this is precisely what the hierarchy of configuration files are for.
3. Indeed.

Edit; PETSc has PETSC_OPTIONS env variable at the xxx level of option priority. We could add a DOLFINX_JIT_PARAMETERS="json string" env variable at the xxx level of priority, but is it really necessary? Edit 2: I'm unsure of the priorities in PETSc, would need further investigation.

[schnellerhase]

1. Shows that we currently already need environment based control to specify the use of the correct options file, so it does not fully replace the env vars, raising the question why another system is necessary.
2. We have examples which need altered/improved options depending on compiler and platform combinations. This currently requires logical switches in user code to reasonably facilitate. Having a consistent interface with other software options that get toggled simplifies the general workflow.

We could add a DOLFINX_JIT_PARAMETERS="json string"

That would be a solution, preferably with another format than JSON which is very bad on inline readability. Since we have only a handful of options DOLFINX_XY vars might be easier without an additional formatting step, I think.

[jhale]

1. I don't agree. Reading the environment variable XDG_CONFIG_HOME or taking $HOME/.config/ is mandatory for any XDG-compliant application (like DOLFINx) and is specified in the XDG Base Directory discovery specification https://specifications.freedesktop.org/basedir/latest/#variables

2. This seems trivial to implement through a Python function which constructs the 'correct' dictionary after querying the platform and compiler information through cffi/distutils? I'm struggling to see what an environment variable system brings.