Corpus & scope

The track runs over the Materials-Project grain-boundary set — the coincidence-site-lattice (CSL) boundaries of the elemental metals whose MP parents carry a DFT grain-boundary energy (the Zheng et al. 2020 GB database, matched to each MP parent id). The scored corpus is 327 grain boundaries across fcc (160), bcc (139) and hcp (28) parents, each with a DFT reference γ, run against 59 potentials — a 59 × 327 grid. Every boundary keeps its full crystallographic identity (Σ, rotation axis/angle, GB plane, tilt/twist) in mp_gb_ref, and the DFT value dft_gb_energy is the benchmark reference.

Crucially, each boundary is rebuilt at the potential's own equilibrium lattice constant, not at a fixed DFT cell: the bulk is relaxed to that model's a0 (reusing the relax-cell campaign's bulk records), and the CSL boundary is constructed at that a0. Two models are therefore never compared on geometrically-identical cells — they are compared on the same physical boundary, each at its own equilibrium, exactly as an MLIP would encounter it in practice.

The leaderboard's coverage grid is not the full 59 × 327: every metric is computed on the paired subset — the (potential × GB) cells where both registries below (shifted and zero-shift) converged and passed the γ-sanity window. A potential's leaderboard n is this paired count for the active protocol, not the raw 327; read it alongside accuracy the same way you would read n on any finite sample.

What the benchmark computes

For each (potential, grain boundary) the pipeline builds the boundary with pymatgen and relaxes it under two independent starting registries, then derives a third, hybrid protocol from the two. In prose:

• Bulk equilibrium. The parent cell is relaxed to (a0, E0/atom, V0/atom) for that potential; the boundary and its reference surface are built and referenced against this single equilibrium.

• pymatgen CSL build. The minimal coincidence cell is expanded to ≈ 15 Å per grain along the boundary normal (REQ_LENGTH_GRAIN = 15, the smallest integer expand_times that clears the threshold), and for the 60°/120° coincidence cells the in-plane cell is orthogonalised (orthogonalize_inplane) so the γ-surface shift grid is rectangular. This build is what inflates the σ7 [111] cells — a ×3 grain expansion followed by a ×2 in-plane orthogonalisation makes them the largest, hardest cells in the set (they dominate the RMSE tail and the timeout/convergence losses).

• Two relaxation registries, one boundary. The two grains can be rigidly translated relative to each other before relaxing — the in-plane γ-surface shift. The pipeline relaxes each boundary from two starting registries and keeps both:

  • shifted (g_shift) — a static in-plane shift grid is scanned first, the minimum-energy registry (ab_shift_a, ab_shift_b) selected, and that cell relaxed (ASE, fmax = 1e-2 eV/Å). This is the original γ-surface-optimised build.
  • zero (g_zero) — the same cell is relaxed directly from the naive, unshifted [0,0] registry, with no scan step at all.

  Both are genuine, independently-relaxed builds, not one build read at two points — they can and do land in different local minima. Excess volume (ev_shift, ev_zero, Å) is recorded alongside γ for each registry from the same relaxed cells.

• hybrid — the recommended default. A row-wise combination of the two: shifted for twist boundaries, zero for tilt boundaries. This mirrors how the Materials-Project reference calculations themselves treat the two GB types — MP only performs an in-plane registry scan for twist boundaries; tilt boundaries are relaxed from the unshifted registry. Matching that asymmetry, rather than applying one scan policy uniformly, is what makes hybrid the most accurate protocol overall: fleet γ MAE 0.1105 J/m², against 0.1258 (shifted) and 0.1263 (zero), and it wins for 51 of the 59 potentials individually (the single most accurate model is equiformer_v3_oam at 0.0447 J/m² under hybrid). The static scan's own benefit over the naive build (≈ 14 % at the unrelaxed γ-surface stage) largely survives the subsequent relaxation, which is why hybrid — keeping the scan only where MP's own protocol says it should help — edges out either pure protocol rather than splitting the difference.

Each result is committed the instant it finishes, so a wall-time kill loses at most one in-flight boundary. A short-lived minimal-grain rebuild (expand_times = 1) was trialled to shrink the σ7 tail but under-converged and is quarantined (build_variant = 'small_cell_underconverged'), excluded from every rollup; only the correct 15 Å-grain build feeds the exports.

Accuracy against DFT

Because a real DFT reference exists, accuracy is absolute, and every potential is directly rankable — there is no consensus caveat and no coverage-based default sort. Two metric modes are available, selected on the leaderboard, and both are scored per protocol (shifted / zero / hybrid):

γ vs DFT (the default metric mode):

• MAE vs DFT — mean |γ_MLIP − γ_DFT| over a potential's paired boundaries (J/m²) [HEADLINE].
• RMSE vs DFT — the tail-sensitive companion; the σ7 [111] over-build cells drive most of the gap between MAE and RMSE.
• Pearson r — correlation of γ_MLIP against γ_DFT; rank fidelity across the boundary set.
• MAE cubic / hex — the fcc/bcc and hcp families are built by different geometry paths and carry different error scales, so the split is reported separately and the leaderboard offers a family toggle.

Excess volume vs DFT (the second metric mode): each relaxed boundary's excess volume (Å) is scored against the DFT excess volume from the same reference set. The comparison is restricted to boundaries the DFT side flags as trustworthy — ev_quality == 'reference' — and to a sanity window |ev| ≤ 5 Å on the MLIP value, dropping the rare relaxation blow-ups (every reference DFT excess volume already sits well inside ±5 Å, so no boundary is lost on the DFT side). The DFT excess volume itself is always the legacy GGA Ω₀ reference geometry (never an r2SCAN recomputation, and never a fitted Ω₀ substituted for a boundary flagged unreliable_residual_strain). Under this window the headline is EV MAE (Å), alongside a signed EV bias and an EV-vs-DFT correlation — the excess-volume analogue of MAE/r for γ.

Caveats

• Unstable potentials crash, they don't mis-rank. A handful of potentials (orb_v2, eqV2, matris, several MatBench checkpoints) throw genuine relaxer failures on many cells and therefore carry low convergence and poor MAE together. They are not recoverable by retry (same input, same crash). Read MAE alongside the converged/n column — a small n on a hard subset flatters or punishes unfairly.

• The σ7 [111] tail is a build artefact, not a physics signal. The pymatgen 15 Å-grain + in-plane-orthogonalisation build makes the σ7 [111] twist/tilt cells very large; they are the slowest to converge and the biggest single contributor to RMSE and to timeout-driven convergence loss. Their difficulty is geometric, not element-specific.

• Cubic vs hcp are different build paths. fcc/bcc CSL boundaries come from the cubic gb_code enumeration; hcp boundaries are built from the MP structure directly. Their error scales are not interchangeable — compare within a family, and use the crystal-family toggle.

• The paired subset is the coverage basis, not the raw 327. A potential's leaderboard n counts boundaries where the relevant registry (or, for hybrid, the relevant registry per boundary type) actually converged and passed the γ-sanity window — it is not out of a fixed 327 denominator. Compare n alongside MAE, and expect it to shift when you switch protocol or metric mode.

• DFT reference provenance. The reference γ is the MP/Zheng-2020 DFT grain-boundary energy matched to each MP parent; it is a fixed external ground truth, independent of any MLIP in the benchmark. The excess-volume reference is that same DFT relaxation's legacy GGA Ω₀ geometry. This is what distinguishes this track from the consensus-only GB energetics track.

See the leaderboard → · ← Methodology overview

Code

Pymatgen grain-boundary energy for one boundary under both registries — build the CSL GB at the model's own a₀ (≈15 Å per grain), relax from the naive [0,0] registry (zero), separately scan the in-plane rigid shift and relax the minimum-energy registry (shifted), then score γ (and excess volume) against MP DFT for each (mp_gb/gamma_worker.py in the campaign, wrapping pure_gb_study via gb_compute.study_one_gb; the hybrid protocol is a row-wise pick between the two, not a separate calculation):

from pymatgen.core.interface import GrainBoundaryGenerator
from mp_gb.gamma_builder import orthogonalize_inplane          # squares 60°/120° in-plane cells
from benchmark.gb_compute import study_one_gb
calc = ...   # any ASE calculator for your MLIP (e.g. mace_mp(model="medium", device="cuda"))

def build(ab):   # pymatgen CSL GB at this in-plane shift, ~15 Å grains, at the model's own a₀
    gb = GrainBoundaryGenerator(bulk).gb_from_parameters(
        rotation_axis=[1, 1, 1], rotation_angle=38.21, plane=[1, 1, 1],
        expand_times=3, vacuum_thickness=0.0, ab_shift=ab, rm_ratio=0.7)
    return orthogonalize_inplane(gb)

# zero — relax the naive [0,0] registry directly, no scan
zero = study_one_gb(build((0, 0)), calc, relax=True)
g_zero, ev_zero = zero["gb_energy"], zero["excess_volume"]

# shifted — static scan of a 20x20 γ-surface, then relax the minimum-energy registry
grid = [(a / 20, b / 20) for a in range(20) for b in range(20)]
ab_min = min(grid, key=lambda ab: study_one_gb(build(ab), calc)["gb_energy"])
shifted = study_one_gb(build(ab_min), calc, relax=True)
g_shift, ev_shift = shifted["gb_energy"], shifted["excess_volume"]

# hybrid — shifted for twist boundaries, zero for tilt (mirrors MP's own DFT protocol)
g_hybrid = g_shift if gb_type == "twist" else g_zero

print(g_hybrid, g_hybrid - dft_gb_energy, ev_zero - dft_excess_volume)