Add ensemble model entry to Matbench Discovery leaderboard

[rmabram]

Dear Matbench Discovery team,

Thank you for building and maintaining an excellent benchmarking resource for materials discovery.

I would like to propose the addition of a simple ensemble entry to the leaderboard – constructed from a subset of the currently published models.
Ensemble learning is well known to improve predictive performance when the base models exhibit some degree of diversity. This condition appears to be satisfied in Matbench Discovery, which includes models with substantially different architectures and inductive biases. Compared to previously included ensembles such as CGCNN+P – which combines ten models of the same kind – a heterogeneous ensemble could yield even greater predictive gains.

Including such an ensemble entry (e.g. through simple averaging or a weighted ensemble) could offer several benefits:

• Provide a useful reference point for the upper bound of current model performance.
• Demonstrate the value of diversity among models in a real-world materials discovery setting.
• Encourage users to think critically about combining existing models.

If this suggestion aligns with your goals, I’d be happy to assist in preparing a minimal working example or contributing a PR. Please let me know if this would be a welcome addition, or if there are any specific guidelines to follow when submitting ensemble-based results.

[janosh]

yes, that would be a useful experiment to run! 👍 i also think a diverse ensemble could show meaningful performance gains, esp. when using the uncertainty-aware energy for ranking materials by stability!

see e.g. eq. 3 in 10.1016/j.xcrp.2024.102241 [code]

$$ E_{\text{form}}^{\text{std-adj}} = E_{\text{form}} - c \cdot \sigma_{E_{\text{form}}} $$

where $\sigma_{E_{\text{form}}}^2$ is the variance of ensemble member predictions

$$ \sigma_{E_{\text{form}}} = \frac{1}{N-1}\sum_{i=1}^{N}\Bigl(E_{\text{form}}^{(i)} - \hat{E}_{\text{form}}\Bigr)^2 $$

this energy downranks materials where ensemble members disagree more, suggesting those crystals may not be as stable as some models think.

if you'd like to submit a PR consisting of combined predictions from multiple top-ranking models, that would be very welcome! i'm also very curious for which tasks the predictive performance improves most (formation energy, thermal conductivity or relaxed atomic positions). maybe the first but interesting either way