Overview

This page collects analysis, observations, and discussion. Per-potential commentary lives on individual potential pages.

DFT+U inconsistencies (O, F)

The unusually high RMSE for oxygen and fluorine across all potentials reflects a DFT+U convention mismatch in the ASSYST training set vs. the MLIPs' typical training. Not a model failure per se.

Element-wise outliers

The Gd raw RMSE values are large for most potentials; after offset correction they drop by ~20–60×. This is consistent with element-dependent reference shifts not being captured by the MLIP's intrinsic scale.

UMA-1.1 magnetic-metal collapse (and the 1.2 fix)

Three FAIR-chem checkpoints — UMA-S-1.1, UMA-M-1.1, and EquiformerV2-OMat24 — record anomalously large energy errors (corrected RMSE ≈1.3, 1.4 and 0.9 eV/atom) that are driven almost entirely by spin-polarized magnetic transition metals (Fe, Ni, Mn, Co, Cr, V). UMA-S-1.2, run through the identical pipeline, is clean (0.137 eV/atom). Mean offset-corrected energy error on the magnetic 3d metals (unary, eV/atom):

• Whole-PES corruption, not just an energy reference. On the same magnetic structures, forces blow up to ~36 eV/Å RMSE on Fe (vs ~0.7 for UMA-S-1.2; the true DFT force scale is ~2.3 eV/Å) and stress likewise; energy/force/stress errors are correlated and collapse together.
• Heavy-tailed and one-sided. The median structure is fine, but ~12–32% of magnetic-metal structures are over-predicted by up to tens of eV/atom (a 2-atom Fe cell at DFT −6.55 eV is predicted at +153.9 eV by UMA-S-1.1, +93.2 by UMA-M-1.1, −6.95 by UMA-S-1.2). This is why corrected RMSE exceeds raw RMSE for these models — a single per-element offset cannot absorb a structure-dependent corruption, and over-correcting the well-predicted bulk makes the headline worse (a transferability signal, not a fitting bug).
• Not a benchmark bug. The harness builds every UMA checkpoint identically (task_name="omat", same env/cluster), runs completed cleanly, and re-running the checkpoints reproduces the stored predictions to <0.004 eV/atom. The defect is intrinsic to the 1.1 weights: the default omat spin = 0 embedding row is a near-null vector in 1.1 (norm ≈0.02–0.08) but a learned vector in 1.2 (≈1.9). UMA's omat head also targets a DFT+U / Materials-Project-style level of theory whose magnetic-metal energetics differ from the spin-polarized PBE reference used here.

Takeaway: cite UMA-S-1.2 for magnetic alloy chemistry; treat the UMA-1.1 magnetic-metal metrics as a checkpoint defect rather than a benchmark artifact. Per-potential detail lives on the UMA-S-1.1, UMA-M-1.1, and UMA-S-1.2 pages.

TODO — fill in

• Cross-potential clustering / ranking under different metrics
• Stress error vs energy error correlation
• Per-partition strengths and weaknesses