Deep, code-cited reference. See the readable summary for the public version. The As-run & site implementation section below documents what actually ran and how the leaderboard is computed; the Design Spec that follows is the original (2026-06-06) plan, preserved for provenance.

As-run & site implementation (2026-06-19; data refreshed 2026-06-21)

The campaign first ran and was exported as phonon_qha_benchmark_export_2026-06-19; the site now serves the 2026-06-21 re-export (phonon_qha_benchmark_export_2026-06-21 — parquet bundle + 66 GB zstd FC2 archive, 47,213 per-(structure, potential) force-constant files, per-file sha256 in the manifest), which extended the set by running the GRACE potentials to completion on dais.

The campaign compute pipeline (how the phonons were run)

The export the site consumes is the output of a Postgres-backed claim-queue campaign on the dais GPU cluster (github.com/ligerzero-ai/mlip-elemental-benchmark, branch dais-port @ 095e4bf8cf12). The unit of work is one structure = one claim → one mlip_phonon row + N mlip_phonon_volume rows + one mlip_qha row. Full step-by-step in the repo's benchmark/PHONON_CAMPAIGN_STEPS.md; the single-structure recipe (relax → 0.01 Å displacements → forces → FC2 → mesh/DOS/thermo → imaginary flag → QHA) is on the phonon page → "Reproduce these phonons".

The export itself comes from export/export_phonon_to_parquet.py (DB → 9 parquet tables) + compress_fc2.sh (FC2 .phonopy.yaml → zstd-16 archive, ~12.4×, status-tagged manifest) + recompute_fmax.py (the achieved_fmax.parquet sidecar — the DB stores only the target fmax).

Provenance of the site assets:

Source → vendored CSV → parquet

Metric → source → aggregator (the leaderboard engine)

src/components/elemental-leaderboard-agg.js (buildRows groups by potential, aggregates each metric's source with its agg over the per-structure rows). All robust by construction:

Metric (key) source agg dir
phonon_omega_consensus_mae (headline) omega_dev_thz medianAbsSkipNull lower
phonon_imag_frac has_imaginary meanBool lower
phonon_qha_bulk_mae b_err_gpa medianAbsSkipNull lower
phonon_qha_bulk_mbe b_err_gpa medianSkipNull lower
phonon_mp_omega_mae mp_omega_err_thz medianAbsSkipNull lower

Verified by node-running the real engine on the real rollup (test/phonon-track.test.js + a rollup replay): the done-only imaginary split reproduces 9.0 % GS / 34.3 % excited / 31.3 % all exactly.

Materials-Project DFT-phonon overlay

tools/fetch_mp_phonon_reference.py (mp-api ≥ 0.46.4 — 0.46.1 cannot resolve the PHEASY S3 objects) determines coverage from summary.has_props['phonon'] (381/834, 48/88 GS) and pulls ω_max + ω_min per mp_id via materials.phonon.get_bandstructure_from_material_id(mp_id, method) over {pheasy, dfpt}src/assets/elemental/phonon/mp_phonon_reference.csv (379 refs: all PHEASY/PBE bar a few DFPT/PBEsol; 2 listed-but-S3-absent misses). regen left-joins it onto the rollup (mp_omega_err_thz). The imaginary flag uses the campaign's min ω < −0.1 THz threshold on both sides.

Caveats (tagged, never primary): MP phonons are at MP's relaxed cell vs each MLIP's own V₀ (both at-equilibrium ω_max); PHEASY = PBE (fair), DFPT = PBEsol (softer). The phonon collection keys its S3 objects by mp_id, not the doc's opaque identifier, and exposes no material_id on the doc — hence the per-mp_id accessor route.

Phonon DOS drill-in (expand-to-DOS)

tools/precompute_phonon_dos.py streams the ~66 GB FC2 tar once (whole-node interactive job, 72 workers, ~45–75 min; phonopy 4.1 in the pwa_elastic venv): per (structure, potential) → zstd-decompress → phonopy.loadrun_mesh(40.0) (length-based L=40 → phonopy keeps full symmetry and diagonalises only the irreducible q-points; an explicit [na,nb,nc] mesh would disable symmetry and diagonalise the whole zone, ~10-50x slower — that plus a too-dense L=80 was the OOM/slowness; L=40 length keeps it ~0.5 GB/curve, fast, and visually identical for a display DOS) + run_total_dos() (tetrahedron, faithful — resolves Van Hove peaks, vs the σ-broadening sacrifice of Gaussian smearing) → resample onto the fixed grid linspace(-2, 50, 1041) THz (0.05 THz pitch, via run_total_dos(freq_pitch=0.05)) → normalize /n_atoms_primitive (∫ ≈ 3) → per-element parquet shards (BLAS pinned to 1 thread so forked workers don't oversubscribe). Coverage: all 58 potentials — the 2026-06-21 FC2 archive (47,213 files) added the GRACE force constants after those potentials were unblocked on dais and run to completion (+1,946 files over the 06-20 repack). DOS is computed for done + unconverged cells (both have FC2); regen joins achieved_fmax.parquet (residual force per unconverged cell, recomputed via the same FrechetCellFilter) onto the rollup, the structures table shows a ⚠ chip + fmax, and the DOS panel warns on unconverged. src/assets/elemental/phonon/dos/{element}.parquet (rows mp_id, potential, dos: list<float32>). tools/fetch_mp_phonon_dos.py (mp-api ≥ 0.46.4) appends the MP-DFT DOS as a potential="__MP-DFT__" row per covered structure (get_dos_from_material_id, same grid, normalized ∫=3). FC-blow-up outliers (e.g. mace_MP-0's 42,139 THz) fall off-grid → empty curve (correct). Pure shaping + the grid constant live in src/components/phonon-dos.js (GRID_* MUST match the tool); test/phonon-dos.test.js covers it. Delivery: the page holds one FileAttachment literal per element and awaits .parquet() only for the clicked element (lazy, same-origin, no CORS / 25 MiB / file-count issue). The expand panel (structures-table.js phonon variant, now caret-expandable) renders a Plot DOS overlay — MLIP vs MP-DFT, imaginary region shaded. Why not client-side: a DOS is a q-mesh dynamical-matrix diagonalization needing the full FC2 (~17 MB raw each) + phonopy; reimplementing that in JS/WASM (or phonopy-in-Pyodide, infeasible) buys nothing — the output curve is ~1–3 KB, so precompute-once + ship curves beats shipping ~66 GB of force constants for browser recompute. (The FC2 archive itself is downloadable from the Data page for offline recompute; it is just not used for in-browser rendering.)

Band structure (as-run)

Band structures are derived from the same FC2 archive as the DOS (no new compute). Three tools handle the full pipeline:

tools/gen_band_paths.py (login node, runs once per structure, fast): generates the q-point path for each of the two conventions.

Both path dicts (fractional q-coords + segment labels/bounds) are written to src/assets/elemental/phonon/band_paths.parquet (one row per (mp_id, convention)).

tools/precompute_phonon_bands.py (compute node, streams the FC2 tar): per (structure, potential), for both conventions:

tools/fetch_mp_phonon_bands.py (login node): appends potential="__MP-DFT__", convention="sc" rows to each structure's shard from the Materials-Project phonon band-structure collection (materials.phonon.get_bandstructure_from_material_id(mp_id, method="pheasy")). MP uses the Setyawan–Curtarolo convention (verified), so these rows are always tagged "sc". Coverage: 379 structures (the same 379 as the DOS overlay; 2 listed but S3-absent misses).

Verified facts (adversarially checked):

R2 shard schema (phonon/bands/{mp_id}.parquet): one row per (potential, convention).

Column Type Notes
potential string e.g. mace_MP-0 or __MP-DFT__
convention string "hinuma" or "sc"
q_dist list<float32> cumulative path distance (one value per q-point)
freqs list<list<float32>> shape [n_q][n_bands]; THz
seg_bounds list<float32> q_dist values at segment boundaries
seg_labels list<string> high-symmetry point labels at each boundary
n_atoms_primitive int32 primitive cell atom count
n_bands int32 number of phonon branches (= 3 × n_atoms_primitive)
has_imaginary bool any mode < −0.1 THz on this path

Shards are not bundled in the site build — they are fetched at runtime per-structure (the per-atom-loader pattern), so they scale with the 834-structure corpus without contributing to Cloudflare's deployment file count or size limit.

Robustness corrections vs the export's derived tables

Flags

orb_v2, orb_v2-MPtrj, FM_MatBench_orb, eqV2_DeNS-31M-MP, alphanet_v1-OAM carry a phonon flag (completion ≤ ~52 % with anomalous imaginary fractions); the cliff above is clean (≥ 59 % with ~30 % imaginary). See src/data/elemental-flags.json.


PHONON + QHA Benchmark — Design Spec

Status (original): design, 2026-06-06 (now superseded by the as-run section above). Sibling track to vacancy/elastic in dft_surrogate_mlip_elemental (db-host:db-port). Author: assembled for Han. Prior art: Loew, Sun, Wang, Botti, Marques, "Universal Machine-Learning Interatomic Potentials are Ready for Phonons," npj Comput. Mater. 11, 178 (2025) (arXiv:2412.16551) — an MLIP harmonic phonon benchmark against a PBE re-computation of Togo's MDR/PhononDB. This track extends it to (a) the metallic / magnetic elementals that DB excludes, (b) QHA / thermal expansion (Loew is harmonic-only), and (c) the full ~60-potential ASSYST set.


0. TL;DR

For each of ~60 MLIP potentials, over all 834 MP elemental crystals, compute:

Reference model = MLIP-vs-MLIP spread (cross-potential median/MAD + imaginary-mode consensus) on all 834 — the vacancy-track model. A small public-DFT validation overlay (~7 non-metal ground states; see §10) is tagged, never primary. A self-computed DFT reference (same pipeline, VASP engine) is a scoped follow-on; the schema reserves the hook.

Unit of work = 1 structure = 1 claim → 1 mlip_phonon row + N mlip_phonon_volume rows + 1 mlip_qha row. Plumbing is cloned from the vacancy track; the only genuinely new code is phonon_compute.py (composes the pyiron_workflow_atomistics phonon primitives).

precompute supercell_matrix per structure (once, potential-independent)
  └─ seed per-potential claim queue ─ sbatch one job per potential
       └─ SLURM job = 1 potential = 1 env, gpu:a100:4, 24h, auto-resubmit until drained
            └─ flux driver fans 4 workers (1/GPU), shared DELETE-on-claim queue
                 └─ worker: claim a structure → phonon_compute (harmonic @V0 + QHA over volumes)
                              → write 1 mlip_phonon + N mlip_phonon_volume + 1 mlip_qha row
  └─ rollup: cross-potential median/MAD + imaginary consensus → CSV ; overlay public DFT

1. Scope & decisions (locked with Han)

Decision Choice
Corpus All 834 structures (matches vacancy/elastic). Stability headline reported on the 90 ground states (where imaginary modes are suspicious; metastable-polymorph instabilities recorded, not penalized).
Methods Harmonic + QHA (isotropic-volume lattice expansion). No phono3py κ(T) (separate costly "hard mode", deferred).
Reference MLIP-vs-MLIP spread primary (all 834) + tagged public-DFT validation overlay + schema hook for a future self-computed DFT reference. Score on PBE (our MLIPs are PBE-trained); tag any PBEsol/vdW ref as functional-mismatch.
QHA volume grid Fixed fractional (linear) strain grid applied around each MLIP's own V₀ (potential-independent grid, per-potential equilibrium → determinism).
QHA intra-volume relaxation (c) Full: relax cell shape (c/a) + ions at fixed volume per point (constant-volume variable-shape) — captures anisotropic thermal expansion.
QHA observables All; headline V(T), B(T), C_p, C_v, G(P,T) (+ α_V(T), Grüneisen γ).
Unstable QHA volumes Drop from the EOS fit + flag (don't abort the structure).
Persistence FC2 as a portable non-pickle sidecar (force_constants.hdf5 + phonopy.yaml) at the equilibrium volume; scalars + thermo arrays in-DB; band/DOS derived on demand. (Per-volume FC2 not persisted — see §8.)
Energy convention smeared VASP energy (not energy_zero) for any DFT reference.

2. Architecture — reuse vs new

Reused unchanged Cloned w/ edits New
db.py, worker._resolve_factory (+ gpu_mem_mb), ASEEngine/engine.calculate, env_map_elemental.sh, the structure corpus, flux-4GPU pattern, auto-resubmit launcher logic, bigcell_probe.py claim queue → mlip_phonon_claim_queue; vacancy_driver_4gpu.pyphonon_driver_4gpu.py (WORKER path + log names); slurm_elem_vacancy.shslurm_elem_phonon.sh; seed_track.py += "phonon": "mlip_phonon_claim_queue"; precompute_sites* pattern → precompute_phonon_cells.py (writes per-structure supercell_matrix, not SOAP sites) phonon_compute.py, phonon_worker.py, schema_phonon.sql, eval/build_phonon_rollup.py + eval/phonon_metrics.py, ingest_phonon_reference.py, smoke_phonon_emt.py, PHONON.md (this file)

Run order: (0) psql -f schema_phonon.sql; (1) sbatch precompute_phonon_cells_node.shphonon_cells.json; (2) bash launch_phonon_all.sh (seed 834×60 + sbatch 60 jobs); (3) monitor; (4) build_phonon_rollup.py; (4b) ingest_phonon_reference.py for the overlay.


3. Method — harmonic (@ equilibrium volume)

Composed from pyiron_workflow_atomistics.physics.phonons.harmonic primitives — not the harmonic_free_energy macro, because the macro's run_thermal_properties throws on a dynamically unstable spectrum, and "has imaginary modes" is exactly our headline metric.

  1. Pre-relax bulk cell+ions to the MLIP's V₀ (FrechetCellFilter, fmax=1e-4, §8).
  2. _generate_fc2_supercells(structure, supercell_matrix, displacement_distance=0.01, is_plusminus="auto") → list of displaced ASE supercells (PM displacements).
  3. Loop them through the same engine.calculate static-force primitive → stack (n_disp, n_at, 3) forces.
  4. phonopy.forces = forces; phonopy.produce_force_constants(); symmetrize FC + enforce the acoustic sum rule (FC_SYMMETRY). (Handles the phonopy v3→v4 FC2-shape gotcha internally.)
  5. Imaginary-mode detection on the supercell-commensurate q-points (exact, non-interpolated), Γ-acoustic exempt (§6).
  6. If stable (or always, storing NaN thermo when not): run_mesh(length=100) + run_total_dos() (tetrahedron) + run_thermal_properties(T=0..1000 K) → F/S/C_v; ZPE = F(0 K); ω_max from the mesh.
  7. Persist FC2 sidecar (§8); return scalars.

status = done if the bulk pre-relax converged and FC2/mesh succeeded (imaginary modes are a result, not a failure); unconverged if pre-relax didn't hit fmax; one error row on exception.


4. Method — QHA (lattice expansion)

Composed from physics.free_energy.quasiharmonic primitives (generate_structures, _static_energies_per_volume, _harmonic_grid_over_volumes, _fit_qha, _pack_qha_output) — not the quasiharmonic_free_energy macro, because the macro does not skip unstable volumes (it lets phonopy emit garbage, then _check_qha_volume_range raises on non-finite V*(T)).

  1. Pre-relax bulk cell+ions → V₀ (the ε=0 grid point = the harmonic headline of §3).
  2. Build the linear-strain grid cell × (1+ε), ε ∈ linspace(−0.04, +0.06, 11) → target volumes (≈ V −11.5 %…+19 %, positive-biased so V*(T) stays in range at high T).
  3. Per volume: set the target volume, then constant-volume shape+ion relax (FrechetCellFilter(constant_volume=True), fmax=1e-3, §8) → static E(V); build FD supercells (disp 0.01) → forces → FC2 → harmonic F(T,V) at mesh length L=60–80 (coarser; F(V,T) is far less mesh-sensitive than DOS); record per-volume ω_max + imaginary flag.
  4. Drop unstable volumes (any with imaginary modes) from the fit; flag in mlip_qha.
  5. _fit_qha(..., eos_type="vinet", pressure_GPa=P) over the pressure grid {0,2,5,10} GPaphonopy.qha.QHAV(T), α_V(T), B(T), C_v(T), G(P,T); C_p(T) from the qha handle (get_heat_capacity_P_polyfit).
  6. Return arrays + 300 K / 0-GPa headline scalars; persist per-volume F(T,V) (not FC2; §8).

The integer supercell_matrix is fixed per structure (from the MP cell, §5) and applied to each volume's relaxed cell — so all potentials and all volumes evaluate commensurate cells.


5. Determinism (the cross-potential guarantee)


6. Imaginary-mode detection (the stability discriminator)


7. Parameters (cited)

Param Value Source / rationale
Supercell min ⊥ spacing 12 Å (conv-check vs 15 on ~5 cells) FC2 real-space range; cost-aware
Displacement (FC2) 0.01 Å phonopy harmonic default; matches PhononDB/Alexandria. (0.03 Å is the phono3py FC3 default — not used here.) Per-potential bump to 0.015–0.02 Å allowed + documented if a MLIP is force-noisy.
q-mesh — production DOS/thermo length-based L=100 (N_i=max(1,nint(L·|b_i*|))) phonopy density mesh — constant q-density across the heterogeneous corpus; Togo JPCM 35 353001 (2023).
q-mesh — QHA per-volume L=60–80 F(V,T) far less mesh-sensitive; ~halves QHA q-cost.
q-mesh — overlay control fixed 20×20×20 reference-matched subset only (Loew/MDR).
DOS/thermo integration linear tetrahedron (phonopy default; no SIGMA) Togo.
Temperatures 0–1000 K, 10 K step QHA breaks near melting anyway.
Pressure grid (G(P,T)) 0, 2, 5, 10 GPa for Gibbs(P,T).
Strain grid linear ε ∈ [−0.04, +0.06], 11 pts positive-biased for high-T V*(T).
EOS Vinet (phonopy.qha) also accepts birchmurnaghan.
fmax — bulk FC2 pre-relax 1e-4 eV/Å (max cell+ion force; done iff reached within the step cap, else unconverged) DFT refs use 1e-8…5e-5; above MLIP noise floor yet keeps Φ clean.
fmax — QHA constant-V relax 1e-3 eV/Å E(V) forgiving; robust over 11 volumes.
Cell filter / optimizer FrechetCellFilter + FIRE/LBFGS, max_steps≈500 as Loew et al.; cap steps so noisy MLIPs can't loop.
fc_calculator phonopy FD (symmetry-reduced); symfc optional for nasty low-sym cells
Acoustic sum rule / PM ON (FC_SYMMETRY, is_plusminus=auto) residual-force / ASR mitigation.

8. Pre-relaxation & storage

Pre-relax: bulk cell+ions to each MLIP's equilibrium (relax_cell, fmax=1e-4); displaced supercells are never relaxed (single-point forces only); QHA per-volume does a constant-volume shape+ion relax (fmax=1e-3). Tightness rationale: §7 + the pre-relax consult (DFT refs go 1e-8…5e-5 eV/Å because residual forces bias the FD-derived FC2 and break the acoustic sum rule → spurious Γ imaginaries).

FC2 storage policy (non-pickle):


9. Database schema (schema_phonon.sql)

All keyed/indexed like the vacancy tables; FK structure_id → structure(id) ON DELETE CASCADE; aggressive autovacuum on the queue.

mlip_phonon — UNIQUE (structure_id, potential); the equilibrium-volume harmonic headline: id, structure_id, potential, status · supercell supercell_nx/ny/nz, n_atoms_supercell, min_perp_angstrom, supercell_matrix(int[9]) · method displacement_distance, n_displacements, q_mesh_length, symprec, fmax · freqs omega_max_thz, has_imaginary_modes, n_imaginary_qpoints, most_negative_freq_thz, frac_imaginary · thermo zpe_ev_per_atom, free_energy_300k, entropy_300k, cv_300k (+ temperature_array, free_energy_array, entropy_array, cv_array as float8[] ) · fc2 fc2_path, fc2_sha256 · convergence bulk_converged · provenance runtime_ms, gpu_mem_mb, device, worker_host, slurm_job_id, error_msg, started_at, finished_at.

mlip_phonon_volume — UNIQUE (structure_id, potential, volume_index): strain, volume_per_atom, static_energy_ev, omega_max_thz, has_imaginary_modes, used_in_qha(bool), free_energy_tv(float8[] # F(T) at this V), relax_converged, fc2_path(NULL by policy).

mlip_qha — UNIQUE (structure_id, potential): eos_type, n_volumes_total, n_volumes_used, n_volumes_dropped, temperature_array, pressure_grid · arrays volume_T(float8[]), thermal_expansion_T, bulk_modulus_T, cv_T, cp_T and gibbs_PT(float8[] per P) · headlines v_300k, alpha_300k, b_300k, cp_300k, cv_300k, gibbs_0gpa_300k, gruneisen_300k · qha_status (done/partial/failed), status, provenance.

mlip_phonon_claim_queue(potential, structure_id) PK (byte-identical clone of the vacancy queue).

mlip_phonon_reference — the DFT overlay and the self-DFT hook: UNIQUE (mp_id, source, ref_method); mp_id, element, is_ground_state, source(text), ref_functional(text), ref_method(text # dfpt/phonopy_fd/self_vasp), omega_max_thz, stable(bool), free_energy_300k, entropy_300k, cv_300k, raw(jsonb).


10. Reference handling


11. Cost & scale


12. Validation (before the fleet)

  1. EMT smoke (smoke_phonon_emt.py): fcc-Al QHA → positive α, V*(300 K) > V*(0) (mirrors the pwa test_quasiharmonic_free_energy_emt_al); fcc-Cu harmonic → no spurious imaginaries.
  2. Cu + Si across all 60 potentials on gpudev (single-structure load+run sanity; correct status logic).
  3. Pipeline-vs-public-DFT on the 7 overlap (B, Si, P, …): our phonopy F/S/C_v/ω_max within tolerance of the Loew/MDR + MP-DFPT published values (on the 20³ control mesh).
  4. bigcell_probe OOM screen on the largest supercell × memory-heavy potentials.

13. Install / environments


14. Gotchas (carried + new)

  1. Compose primitives, not the macros — both harmonic_free_energy and quasiharmonic_free_energy throw/mishandle unstable spectra; we need the imaginary flag and "drop unstable volumes."
  2. Min-image / supercell sizing uses perpendicular spacing, never cell.lengths() (vacancy bug).
  3. Pre-relax tightness matters — under-converged forces → spurious near-Γ imaginaries (mistaken for real instability). 1e-4 FC2 / 1e-3 QHA + ASR ON.
  4. displacement = 0.01 Å (harmonic) ≠ 0.03 Å (phono3py FC3). Easy to conflate.
  5. q-mesh = length-based L, not a fixed grid — fixed grids mis-sample a heterogeneous corpus.
  6. Strain is linear (cell×(1+ε), V∝(1+ε)³); positive-bias the grid or V*(T) walks off it (phonopy raises).
  7. 40 GB A100 OOM — accept graceful errors; screen with bigcell_probe.
  8. Atomic sidecar + json writes (temp+os.replace) so running workers never read half-written files.
  9. slurm_elem_phonon.sh env fallback via env_map_elemental.sh for the ~25 potentials lacking an own env dir — don't hardcode envs_elemental/<POT>.
  10. C_p vs C_v — the package returns C_v; C_p comes from the phonopy.qha handle (or C_p = C_v + V·T·α²·B_T). Han wants both.

15. Open items for Han


16. Key references


Verification addendum (run wf_7fbcd118-848, 2026-06-08)

Track is designed but not yet run; verified the code against this design doc: