Skip to content

README.md

Latest commit

 

History

History
116 lines (86 loc) · 3.64 KB

README.md

File metadata and controls

116 lines (86 loc) · 3.64 KB

GMTKN55 Benchmark Evaluator

This project provides a Python-based evaluation framework for computing WTMAD-2 and other statistical metrics on the GMTKN55 benchmark suite. It processes .res files, filters molecules based on chemical constraints, and parses output to compute evaluation metrics such as WTMAD-2, MAE, and more.

📦 Features

  • Automatically parses GMTKN55 benchmark subsets from a local filesystem
  • Filters molecules based on charge, number of unpaired electrons, and required/allowed elements
  • Evaluates reactions using .res or .resRC files and a user-specified method
  • Computes WTMAD-2 and per-category metrics (e.g. small reactions, NCI, barrier heights)
  • Exports results to CSV if requested
  • Includes a progress bar and detailed verbosity levels

🛠 Requirements

Install dependencies using conda:

conda env create -f environment.yaml
conda activate gmtkn55-env

The main dependencies are:

  • Python ≥ 3.12
  • numpy
  • pandas
  • tqdm

📁 Project Structure

GMTKN55/
├── eval.py                  # Main entry point for evaluating subsets
├── utils/                   # Contains all Python source code beyond the central eval.py script
│   ├── __init__.py
│   ├── statistics.py        # WTMAD-2 and statistical calculations
│   ├── constants.py         # Constant data
│   └── ...                  # Further Python source files
├── ACONF/                   # Expected location of GMTKN55 subset folders
├── ADIM6/                   # ...
├── .../                     # ...
├── environment.yaml         # Conda environment specification
└── README.md                # This file

🚀 Usage

Run the evaluation on your local GMTKN55 directory:

python eval.py --method YOUR_METHOD_NAME --verbosity 1 --write-to-csv

Further optional arguments

  • --allowed-elements '1-86'
  • --required-elements-all '6,1'
  • --required-elements-one '8,7'
  • --min-charge -1
  • --max-charge 2
  • --max-uhf 2
  • --format 13 (format of the .res files (default: 13))

Example:

python eval.py --method mydft --verbosity 2 --write-to-csv --allowed-elements '1-20'

📊 Output

With --write-to-csv, the script will generate a file: <args.format>.csv containing columns:

  • Subset
  • Reaction
  • Stochiometry
  • ReferenceValue
  • MethodValue

Statistics

The script prints:

  • Overall WTMAD-2
  • WTMAD-2 per category:
  • Small Reactions
  • Larger Reactions
  • Barrier Heights
  • Intermolecular NCI
  • Intramolecular NCI
  • Optionally: Mean Absolute Error (MAE) per subset

👥 Authors

  • Marcel Müller
  • Contributions welcome!

License

This repository contains two parts with different licensing terms:

  • Dataset (GMTKN55)
    © 2017 Lars Goerigk, Andreas Hansen, Christoph Bauer, Stephan Ehrlich, Asim Najibi, Stefan Grimme, and co-authors.
    Licensed under CC-BY-4.0.
    Use is permitted for both academic and industrial purposes, provided appropriate attribution is given.
    Please cite:
    Goerigk, L.; Hansen, A.; Bauer, C.; Ehrlich, S.; Najibi, A.; Grimme, S. "A look at the density functional theory zoo with the advanced GMTKN55 database for general main group thermochemistry, kinetics and noncovalent interactions." Phys. Chem. Chem. Phys. 2017, 19, 32184–32215. DOI:10.1039/C7CP04913G.

  • Evaluation script (Python code)
    © 2025 Marcel Müller
    Licensed under the MIT License.
    Free to use, modify, and redistribute without restriction.

For full details, see the LICENSE file.