Is there a method to quickly assess a new CSV with predictions?

[rydeveraumn]

Hello all,

I have been working on new concepts for training IAP models and I am wondering if there is a way to quickly assess the WBM dataset with predicted formation energies. I saw this link that describes how to submit a new model: https://matbench-discovery.materialsproject.org/contribute but I am looking for something that can quickly score my new predictions and I can see how it compares to submitted models.

My current code is still very much in the dev phase and not ready for a full submission. Any help would be appreciated!

[janosh]

hi @rydeveraumn, have a look at https://matbench-discovery.materialsproject.org/data#--direct-download, specifically wbm_summary (wbm/2023-12-13-wbm-summary.csv.gz). the relevant column is e_form_per_atom_mp2020_corrected. if you evaluate the formation energy MAE of your model on those reference values, that'll be the same as the MAE for the per-atom convex hull distance which is what's displayed in the MAE column of the metrics table on https://matbench-discovery.materialsproject.org

see https://matbench-discovery.materialsproject.org/data#--downloading-data-files for a code snippet showing how access that data in Python.

[rydeveraumn]

Oh I am sorry I should have been more specific about my intentions - my work is geared toward enhancing the stability prediction (F1 Score) which I do not know how to get from the formation energy. Thank you for the response @janosh.

[janosh]

stable_metrics calculates the F1 score from a DFT reference values and model predictions. the energy above convex hull predictions for each model are being calculated here

matbench-discovery/matbench_discovery/preds/discovery.py

Lines 76 to 89 in 481eca6

	)
	# dataframe of all models' energy above convex hull (EACH) predictions (eV/atom)
	df_each_pred = pd.DataFrame()
	for model in Model:
	df_each_pred[model.label] = (
	df_preds[MbdKey.each_true] + df_preds[model.label] - df_preds[MbdKey.e_form_dft]
	)
	"""
	To avoid confusion for anyone reading this code, df_each_err calculates the formation
	energy MAE but reports it as the MAE for the energy above the convex hull prediction.
	The former is more easily calculated but the two quantities are the same. The formation
	energy of a material is the difference in energy between a material and its

MbdKey.each_true is the DFT hull distance and MbdKey.e_form_dft the DFT formation energies which are subtracted and the result added to the model formation energy to give the model hull distance.

[parkyjmit]

I wish to express my utmost admiration for the remarkable work accomplished by your research group.

It appears that I am at similar stages. I am currently developing an IAP as well and aim to obtain metrics including F1 and DAF for intermediate evaluation (without posting the model to the leaderboard). However, it seems that the code in matbench_discovery/preds/discovery.py evaluates already published Model objects rather than my personal model.

I have referred to several files, such as <modelname>/test_<modelname>_discovery.py, which appear to only perform relaxation. Would you be so kind as to advise me on how to compute the energy above convex hull prediction and relavent metrics?

[janosh]

the code i referenced above (#213 (reply in thread)) does show how to compute the energy above convex hull predictions. the each in df_each_pred stands for "energy above convex hull". for every model, we take the formation energy per atom predictions (df_preds[model.label]) and subtract the DFT formation energies (df_preds[MbdKey.e_form_dft]), then add the DFT above convex hull energies (df_preds[MbdKey.each_true])

[rydeveraumn]

Hey @janosh I do have one more question. First off your suggestion was very helpful! I am still trying to understand some of the fundamentals of how things are calculated and I am wondering if you can help provide some insights.

I am looking at material_id = wbm-1-1 where I have two values:

1. Energy (Uncorrected) = -42.9554
2. e_form_per_atom_mp2020_corrected = 0.544327

I used this code to reconstruct the Energy (Uncorrected) since now I am understanding how to do this with respect to the phase diagram:

from mp_api.client import MPRester
from pymatgen.core.composition import Composition
from pymatgen.entries.compatibility import MaterialsProjectCompatibility

# Your material's information
formula = 'Ac6 U2'
formation_energy_per_atom = 0.544327  # in eV/atom
composition = Composition(formula)

# Get reference energies from Materials Project
with MPRester("API_KEY") as mpr:
    # Get element reference data
    elements = [str(el) for el in composition.elements]
    
    # Get the reference energies for the elements
    reference_energies = {}
    for element in elements:
        # Get the most stable structure of the element
        entries = mpr.get_entries(element, property_data=["energy_per_atom"])
        
        compat = MaterialsProjectCompatibility()
        entries = compat.process_entries(entries)
        
        if entries:
            reference_energies[element] = entries[0].energy_per_atom
        else:
            print(f"Could not find reference energy for {element}")

# Calculate total energy from formation energy
# Formation energy = (total energy - sum(reference energies)) / num_atoms
# So: total energy = formation energy * num_atoms + sum(reference energies)

num_atoms = composition.num_atoms
sum_reference = 0
for element, amount in composition.items():
    element_str = str(element)
    sum_reference += amount * reference_energies[element_str]

total_energy = (formation_energy_per_atom * num_atoms) + sum_reference

print(f"Formation energy per atom: {formation_energy_per_atom} eV/atom")
print(f"Number of atoms: {num_atoms}")
print(f"Reference energy sum: {sum_reference} eV")
print(f"Total energy: {total_energy} eV")

and the Total Energy = -42.682396.

So I am really close and my questions are:

1. Do you happen to see where the difference could be?
2. Does this small of a difference have a significant impact?

[janosh]

1. it goes the other way: Energy (Uncorrected) = -42.9554 is the system's total energy coming from a VASP calculation. the total energy is converted into the formation energy by subtracting the elemental reference energies (the lowest-energy known polymorph for each element present in the compound) multiplied by the amount of that element in the structure. the resulting raw formation energy is then processed with MaterialsProject2020Compatibility (not MaterialsProjectCompatibility!) to get the corrected formation energy from which the convex hull distance is determined by comparing it to all the other energies of competing structures in that chemical system.
2. yes, small energy differences of 10s of meV/atom can make a big difference in the thermodynamic stability of a crystal structure

[rydeveraumn]

You are so awesome and helpful thank you!