Corpus & scope

The track runs over the elemental cubic crystals of the benchmark corpus — the bcc, fcc, simple-cubic and diamond structures whose spacegroup maps cleanly to a coincidence-site-lattice (CSL) construction (BASIS_OK = {"bcc","fcc","sc","diamond"} in benchmark/gb_compute.py). Anything non-cubic is marked unsupported and skipped, so the eligible set is recorded up front in mlip_gb_eligible (benchmark/schema_gb.sql).

The deliverable is deliberately narrow: 15 curated (KEEP) potentials × 36 industrial structures. The 36 industrial structure ids are the benchmark gate — the common, technologically-relevant elemental metals — and the 15 KEEP potentials are the curated roster used to define the consensus reference. In total 51 potentials have some GB data in the database, but only the curated 15 × 36 block is the published benchmark; everything else is exploratory and shown behind toggles on the leaderboard.

For each (potential, structure) the pipeline first relaxes the bulk crystal to that potential's own equilibrium lattice constant, then enumerates the CSL grain-boundary set at that a0. Enumeration is deterministic: the same rotation axes, the same Σ cut-off, the same de-duplication, every time. The first-pass campaign uses the anchored three-axis set — [100], [110], [111] up to Σ ≤ 30 (AXES_ANCHOR, SIGMA_LIMIT = 30) — matched to the reference GRACE study so the results are self-validating; the full 13-axis spec is env-selectable without a code change. Because every boundary is rebuilt at the potential's own a0, two models are never compared on geometrically-identical cells — they are compared on the same physical boundary, each at its own equilibrium.

What a GB study computes

Each enumerated boundary is handed to pure_gb_study (pyiron_workflow_atomistics.physics.grain_boundary, driven from benchmark/gb_compute.py::study_one_gb). One study is roughly 30 MLIP relaxations and produces a single mlip_gb row; the per-(potential, structure) bulk record — including n_gb_total, the count of boundaries enumerated — is one mlip_gb_bulk row. In prose, a study does the following:

All relaxations use ASE with fmax = 1e-2 eV/Å and a 500-step cap (FMAX_RELAX, MAX_STEPS). Each boundary is committed the instant it finishes (streaming gb_sink), so a wall-time kill loses at most one in-flight boundary; oversized post-build cells are dropped before relaxing (GB_MAX_BUILT_ATOMS filter — the huge cells are near-guaranteed timeouts that produce no usable data), and a poisoned CUDA context triggers a hard-exit-and-requeue rather than a cascade of junk rows.

Consensus & coverage — the two headline axes

There is no public DFT grain-boundary reference. Unlike the lattice or vacancy tracks, no independent ground truth exists to score against, so accuracy here means agreement with the cross-model consensus.

The consensus reference. For each (structure, boundary) the reference GB energy is the median gb_energy over the 15 KEEP potentials, kept only where at least 5 KEEP potentials converged on that boundary (a median of one or two models is not a consensus). Each of the 51 potentials is then scored by how far it sits from that median. Two numbers summarise this on the leaderboard:

Consensus ranks "most typical", not "most correct": 11 of the 15 KEEP models are GRACE-family, so the median leans toward GRACE-like behaviour. And the deviation is coverage-confounded — a model that only ran a handful of easy, low-Σ boundaries scores a deceptively small deviation. For both reasons the leaderboard default-sorts by coverage, not by the consensus column.

The coverage (clamp) metric. Coverage looks like it should be trivial — boundaries done over boundaries enumerated — but the naive ratio Σ(terminal rows) / Σ(n_gb_total) exceeds 100 %. Two things inflate the numerator: (a) legacy rows from earlier enumeration waves that used different parameters, and (b) duplicate gb_id representations of the same physical boundary — the gb_id string embeds the raw plane vectors, and different waves wrote equivalent planes under different vector conventions. Neither means "more than done". The robust fix (benchmark/gb_status.py) is a per-structure clamp: a structure cannot be more than complete, so cap its distinct-terminal count at its enumerated total before summing.

# benchmark/gb_status.py — coverage that cannot exceed 100 %
coverage = sum(min(distinct_terminal_gb_id, n_gb_total) for each structure) \
         / sum(n_gb_total for each structure)
# terminal = a gb_id with status in ('done','error','timeout')
# legacy / duplicate / convention-variant rows can only pad UP TO the cap, never past it

The published headline is clamp_pct_industrial — this same clamp restricted to the 36 industrial structure ids, i.e. the benchmark gate (benchmark/export_gb_to_parquet.py). It is an upper bound on true coverage (extras can fill a structure to its cap while a few genuine boundaries are still missing), so it is honest about incompleteness without ever reading above 100 %.

Caveats

Provenance & verification (deep, code-cited) → /methodology/deep/grain-boundary

See the leaderboard → · ← Methodology overview

Code

Elemental CSL grain-boundary study for one crystal — EOS→a0, build the GB set, then relax each GB (gb_compute.py in gb-mlip-benchmark, wrapping pure_gb_study):

from ase.build import bulk
from gb_compute import compute_gb
calc = ...   # any ASE calculator for your MLIP (e.g. mace_mp(model="medium", device="cuda"))

res = compute_gb("Fe", "bcc", bulk("Fe", "bcc", a=2.83, cubic=True), calc,
                 scratch_root="/tmp/gb")
for gb in res["gbs"]:
    print(gb["gb_id"], gb["gb_energy"],            # GB energy (J/m²)
          gb["work_of_separation_relaxed"],        # W_sep (J/m²)
          gb["excess_volume"])