Base URL

https://api.mlip-bench.dev/v1

All responses are static JSON files served with Content-Type: application/json.

Authentication

The API is protected by Cloudflare Access using a service token. Include both headers in every request:

Contact the maintainers to obtain a service token.

Index & discovery

curl examples

1 — Fetch the catalog

curl -s \
  -H "Cf-Access-Client-Id: $CF_CLIENT_ID" \
  -H "Cf-Access-Client-Secret: $CF_CLIENT_SECRET" \
  https://api.mlip-bench.dev/v1/index.json | jq .

2 — Fetch a potential record

curl -s \
  -H "Cf-Access-Client-Id: $CF_CLIENT_ID" \
  -H "Cf-Access-Client-Secret: $CF_CLIENT_SECRET" \
  https://api.mlip-bench.dev/v1/potentials/gracefm_GRACE-2L-OAM.json | jq .

3 — Fetch a track leaderboard

curl -s \
  -H "Cf-Access-Client-Id: $CF_CLIENT_ID" \
  -H "Cf-Access-Client-Secret: $CF_CLIENT_SECRET" \
  https://api.mlip-bench.dev/v1/tracks/elastic/leaderboard.json | jq '.entries[:5]'

Python client (mlip_bench)

A typed Python client wraps every endpoint on this page — typed structures and workflow instances, ase.Atoms geometry, pandas leaderboard/parity DataFrames, bulk dataset download, and client-side point lookups into the hash-sorted geometry/corpus stores over plain HTTP Range reads. It lives at clients/python/ in this repo; the walkthrough below is the full tour, and clients/python/README.md + examples/quickstart.py mirror it.

Install

Not yet on PyPI — install from a checkout of this repo:

pip install -e clients/python[pandas,datasets]

Credentials

Client() takes no required arguments; it resolves the same Cloudflare Access service token documented above, from the first fully-specified source of:

1. explicit args — Client(client_id="...", client_secret="...")
2. env vars — MLIP_BENCH_CLIENT_ID / MLIP_BENCH_CLIENT_SECRET
3. ~/.config/mlip-bench.toml — [auth] client_id / client_secret
4. ~/.config/mlip-bench.env — shell-style CF_CLIENT_ID= / CF_CLIENT_SECRET=

With none found it proceeds unauthenticated (emitting a UserWarning; pass anonymous=True to silence). The secret is never logged, and is redacted from repr().

Pulling data — a full tour

from mlip_bench import Client

client = Client()                        # credentials resolved automatically (see above)

Structures — by atoms_hash, mp_id, or a {namespace}:{structure_id} alias (aliases 302-redirect transparently wherever a hex hash is accepted):

s = client.structure("mp-132")           # -> Structure
s.record.element                         # 'Fe'
s.record.formula                         # 'Fe'
s.record.workflow_ids                    # ['elastic:allegro_MP-L', 'relax-full:...', ...]
s.record.geometry_available              # True

Geometry — as an ase.Atoms (the input / DFT-reference cell):

atoms = s.geometry()                     # ase.Atoms
atoms.get_chemical_formula()             # 'Fe'
atoms.cell, atoms.positions, atoms.numbers

Workflow instances — one typed record per (workflow, potential):

inst = s.workflow("elastic:allegro_MP-L")    # -> WorkflowInstance
inst.status                                  # 'done'
unit = inst.units[0]
unit.outcomes["k_vrh"]                       # e.g. bulk modulus (GPa)
unit.geometry["mlip_relaxed"].availability   # 'served' | 'not_computed' | 'bulk_only' | 'sp_only' | 'not_persisted'

Each unit's geometry is a set of honestly labelled references. Follow a served one to its ase.Atoms; a non-served one raises rather than hand you a dangling link:

from mlip_bench import GeometryUnavailable

ref = unit.geometry["mlip_relaxed"]
try:
    relaxed = client.fetch_geometry(ref)     # ase.Atoms, only if availability == 'served'
except GeometryUnavailable:
    print(ref.availability, "—", ref.note)   # e.g. 'not_computed — no geometry persisted for this unit'

Most workflows expose a single relaxed cell (ref.href is one URL). surface-slab and gb-energetics expose several — the relaxed slab and its reference bulk, several GB cells — so ref.artifacts is a {name: url} dict and you pass artifact=:

ref = unit.geometry["mlip_relaxed"]          # surface-slab unit (one Miller/termination)
ref.artifacts                                # {'slab_relaxed': ..., 'bulk_relaxed': ...}
slab = client.fetch_geometry(ref, artifact="slab_relaxed")   # ase.Atoms — the relaxed slab
bulk = client.fetch_geometry(ref, artifact="bulk_relaxed")   # the relaxed reference bulk

The input / initial cell is always unit.geometry["input"] (equivalently s.geometry()); note that for surfaces this is the bulk seed cell — the unrelaxed slab is not persisted. Calling fetch_geometry without an artifact on a multi-artifact ref (or with one on a single-artifact ref) raises ValueError telling you the available names.

Leaderboards & parity — as pandas DataFrames (needs the pandas extra):

lb = client.track("elastic").leaderboard()   # DataFrame indexed by `potential`
lb.head()
client.track("ordering").leaderboard(scheme="sp")   # the ordering track is published per-scheme

ASSYST corpus (~7.29M single-point structures) — client.structure(hash) transparently falls back to the corpus for a corpus-only hash (s.corpus is then populated, and s.geometry() still works, served from the corpus-geometry store); or fetch the raw record directly:

rec = client.corpus_record("a3f…64-hex")     # dict: {structure, potentials{}, geometry}

Grain-boundary segregation (Worker-served from a hash-sorted store, ~10M instances):

gb = client.gbseg_instance("aa…hash", "GRACE-2L-OAM-L")   # -> WorkflowInstance

Bulk datasets — list the catalog, download a parquet (Range-resumable), or stream it without downloading:

cat  = client.datasets.list()          # {"bulk": [{name, url, size}, ...], "subsets": [...]}
name = cat["bulk"][0]["name"]
path = client.datasets.download(name, dest_dir="/tmp")    # Range-resumable
fs   = client.datasets.filesystem()                       # fsspec 'http' fs (needs [datasets])

Direct store point-lookups — resolve one row of any hash-sorted store client-side (manifest → prefix-shard index → one row group → filter by key), over plain HTTP Range reads, with no server-side query route:

atoms = client.query_geometry("geom-elemental-relax", "…::relax-full::…::-::relaxed")
row   = client.stores.lookup("wf-gbseg-instances", "…::gbseg-relax::…::bfgs")

Errors

from mlip_bench import (
    NotFoundError, ApiError, GeometryUnavailable,
    ParityNotPublished, DataVersionMismatch,
)

Property reference — every field the site serves

Everything the website plots is reachable here. The data comes in four families, each with a fixed set of named properties:

1. Structure → workflow outcomes — the physical result of running one potential on one structure (elastic Cᵢⱼ, vacancy formation energy, surface γ, …). Keyed (atoms_hash, workflow, potential).
2. Track leaderboards — the aggregate score of one potential on one benchmark track, ranked. Keyed (track, potential).
3. Potential records — metadata about one potential (architecture, training set, params, links).
4. Element aggregates — coverage counts per chemical element.

Scalars only. Every property below is a scalar (or a small fixed vector like the 21 elastic constants). Curve/array data — phonon DOS spectra, band structures, QHA F(T) curves — is not served by this API; it lives as bulk assets behind the /data page. See Curve data below.

Family 1 — Structure → workflow outcomes

Record path (static, one file per structure×potential):

/v1/structures/{atoms_hash}/workflows/{workflow}:{potential}.json

Each record has status, a method block, provenance, and a units[] list. Every unit carries an outcomes{} dict (the properties) and a geometry{} block of honestly-labelled cell references. Multi-unit workflows (surfaces, vacancies, GB) split one record into several units, discriminated by a key{}.

The complete outcome catalog, workflow by workflow:

Stability ordering — ordering-sp · 1 unit · live

Cell relaxation — relax-hydro (hydrostatic) & relax-full (full cell+ions) · 1 unit · live

(relax-fixedcell — legacy fixed-cell variant — carries the same fields minus volume_ang3/pressure_gpa; superseded, not currently emitted.)

Elastic tensor — elastic · 1 unit · live

Vacancy formation — vacancy-site · 1 unit per inequivalent site (key.site_index) · live

Surface energy — surface-slab · 1 unit per (Miller, termination) (key.miller_index, key.termination_index) · live

(surface-bulk — the reference-bulk relaxation feeding γ — is registered (bulk_energy_ev_atom, bulk_converged, bulk_n_atoms) but not separately emitted; its bulk cell is reachable via any slab unit's bulk_relaxed geometry.)

GB energetics — gb-energetics · 1 unit per GB (key.gb_id) · live

GB segregation — gbseg-relax · 1 unit per optimizer (key.algo = BFGS/FIRE) · Worker-served

Reached via the Worker route (not a static file) — see the query recipes below.

(The relaxation runtime — runtime_ms, gpu_mem_mb, device — rides in the record's separate runtime{} block. The segregation energy itself, E_seg, is a leaderboard-level scored quantity — see Family 2, track segregation.)

ASSYST corpus single-point — sp · ~7.29M structures · Worker-served

Reached via /v1/query/corpus/{atoms_hash}.

Not yet emitted (registered, gated until their instance tables land): phonon-fc2 (omega_max_thz, has_imaginary, frac_imaginary, most_negative_freq_thz), qha (alpha_300k_per_k, b_300k_gpa, cv_300k_j_per_mol_k, qha_converged), and per-structure wsep (wsep_rigid_j_per_m2, wsep_relaxed_j_per_m2) whose scalar rows have no store export today (the cleavage leaderboard below is served, the per-structure records are not). The scalar phonon/QHA summaries are downloadable in bulk from /data.

Geometry references. Each unit's geometry{} gives input, dft_relaxed, and mlip_relaxed cells, each tagged with an honest availability label:

How to query one outcome

# curl — the elastic tensor of mp-20 under uma_M-1p1
curl -s -H "Cf-Access-Client-Id: $CF_CLIENT_ID" -H "Cf-Access-Client-Secret: $CF_CLIENT_SECRET" \
  "https://api.mlip-bench.dev/v1/structures/$(
    curl -s -H "Cf-Access-Client-Id: $CF_CLIENT_ID" -H "Cf-Access-Client-Secret: $CF_CLIENT_SECRET" \
    -o /dev/null -w '%{redirect_url}' https://api.mlip-bench.dev/v1/structures/mp-20 | sed -E 's#.*/structures/##')/workflows/elastic:uma_M-1p1.json" \
  | jq '.units[0].outcomes | {k_vrh, g_vrh, mechanically_stable}'

from mlip_bench import Client
c = Client()

# scalar outcomes, any workflow, by mp_id / atoms_hash / alias:
inst = c.structure("mp-20").workflow("elastic:uma_M-1p1")
o = inst.units[0].outcomes
o["k_vrh"], o["g_vrh"], o["mechanically_stable"]      # 122.4, 73.5, True

# per-site / per-facet workflows expose one unit each — read the key:
vac = c.structure("mp-20").workflow("vacancy-site:GRACE-2L-OAM-L")
for u in vac.units:
    print(u.key["site_index"], u.outcomes["site_element"], u.outcomes["formation_energy"])

# Worker-served corpus & gbseg:
c.corpus_record("a3f…64hex")                          # sp outcomes for a corpus structure
c.gbseg_instance("…hash", "GRACE-2L-OAM-L")           # gbseg-relax instance

Discover which (workflow:potential) pairs a structure has before fetching:

s = c.structure("mp-20")
s.record.tracks_present          # ['elastic', 'lattice', 'phonon', ...]
s.record.workflow_ids            # ['elastic:uma_M-1p1', 'relax-full:...', ...]

Family 2 — Track leaderboard metrics

/v1/tracks/{track}/leaderboard.json → {track, scheme, rows[]}; each row is one potential. The metric columns differ per track — the full set:

The ordering track publishes per-scheme sub-leaderboards instead of a single leaderboard.json: /v1/tracks/ordering/leaderboard-{sp,hydro,full}.json.

curl -s -H "Cf-Access-Client-Id: $CF_CLIENT_ID" -H "Cf-Access-Client-Secret: $CF_CLIENT_SECRET" \
  https://api.mlip-bench.dev/v1/tracks/phonon/leaderboard.json | jq '.rows[:3]'

lb = c.track("elastic").leaderboard()          # DataFrame indexed by `potential`
lb.sort_values("median_abs_k_pct").head()
c.track("ordering").leaderboard(scheme="sp")   # ordering is per-scheme

Family 3 — Potential records

/v1/potentials.json (all) and /v1/potentials/{id}.json (one). Fields:

c.potentials()                       # list of all records
c.potential("gracefm_GRACE-2L-OAM")  # one record

Family 4 — Element aggregates

/v1/elements.json → {elements: [{element, n_potentials}, …]} — how many potentials have data for each chemical element.

curl -s -H "Cf-Access-Client-Id: $CF_CLIENT_ID" -H "Cf-Access-Client-Secret: $CF_CLIENT_SECRET" \
  https://api.mlip-bench.dev/v1/elements.json | jq '.elements[] | select(.element=="Fe")'

Curve data (not in the API)

Phonon DOS spectra, band structures, and QHA F(T)/α(T)/Cv(T) curves are the one thing the website renders that this API does not serve per-id — only the scalar summaries above (omega_max_thz, alpha_300k_per_k, …) are exposed. The full curves are downloadable in bulk from the /data page (phonon parquet bundles, per-structure band shards, the raw FC2 archive). A per-material curve endpoint (client.phonon_dos(id, potential) → (freq, dos)) is designed but not yet built.

Endpoint reference

Terse index of every route (properties for each are catalogued above).

Potentials

Structures → workflows

A non-hex id (mp-132, elemental:59, …) 302-redirects to its atoms_hash.

Tracks

Available track ids: pes, ordering, elastic, lattice, surface, vacancy, phonon, segregation, cleavage, technical-performance.

Elements

Datasets

OPTIMADE

An additive OPTIMADE v1.2 Tier-1 facade lives alongside the native API above, for tooling that speaks the OPTIMADE standard (e.g. optimade-client, aggregators, the official validator). It is honest about scope: fully conformant where we can be, and a proper OPTIMADE 501 error object — never a silent guess — everywhere our serverless architecture can't yet do more. The native API above stays the primary product; OPTIMADE _mlipbench_-prefixed fields link back into it for anything OPTIMADE has no room to express (workflow outcomes — elastic/vacancy/phonon/etc. results).

Base URL

https://api.mlip-bench.dev/optimade/v1

(versions is the one endpoint the OPTIMADE spec pins at the unversioned base: https://api.mlip-bench.dev/optimade/versions.)

Endpoints

Note on page 1 shape: The static unfiltered page at /optimade/v1/structures (no query string) returns full OPTIMADE entries. Dynamic pages (any query parameters: filter=, page_offset>0, page_limit, etc.) return the reduced 4-field index subset; clients walking links.next will observe this shape change. Full OPTIMADE attributes are always available via the per-id route /optimade/v1/structures/{id}.

Single-entry coverage is total: every corpus/elemental structure the native API knows by atoms_hash is reachable here too, one entry at a time. The collection route (structures?filter=...), by contrast, only ever indexes the 834-structure elemental set — filtering the full corpus is out of scope for Tier-1 (see Non-goals in the design doc).

A non-hex id (an mp_id like mp-132, or a namespaced alias) 302-redirects to the canonical atoms_hash path, same alias resolution the native API uses. An uppercase-hex id canonicalizes (302) to lowercase.

Supported filter subset (collection route only)

Tier-1 supports exactly these atoms, combined with AND only (no OR, NOT, parentheses, or range comparisons):

response_fields= on the collection route is similarly limited to the four attributes the listing actually indexes (elements, nelements, chemical_formula_reduced, chemical_formula_anonymous) — fetch /structures/{id} for the full mandatory attribute set (positions, cell, species, ...). An empty response_fields= is treated as absent (default four-field set), not "zero fields."

The 501 policy (honest Tier-1)

Anything recognized as valid OPTIMADE filter grammar but outside the subset above — OR, NOT, comparison operators (!=, <, <=, >, >=), CONTAINS/STARTS/ENDS/LENGTH, IS KNOWN/IS UNKNOWN, HAS ONLY, parenthesized/grouped expressions, or a real OPTIMADE property we simply don't index for filtering (lattice_vectors, species, ...) — returns 501 with an OPTIMADE error object (title: "UnsupportedFilterFeature"), not a silent partial match or a fabricated result:

{
  "errors": [{ "status": "501", "title": "UnsupportedFilterFeature", "detail": "..." }],
  "meta": { "api_version": "1.2.0", "provider": { "prefix": "mlipbench", ... } }
}

A filter that doesn't parse as valid OPTIMADE grammar at all (unterminated quote, stray character, right property/wrong shape) returns 400 (title: "FilterSyntaxError") instead — 501 is reserved for constructs we recognize but don't (yet) implement, 400 for ones that are simply malformed. Filtering the ASSYST corpus collection (as opposed to a single-entry lookup by id) is always 501 — collection filtering is elemental-only in Tier-1. The references/files OPTIMADE entry types are not implemented at all — any request under /optimade/v1/references or /optimade/v1/files gets the same 404 OPTIMADE error object every unknown route gets.

Auth

Same gate as the native API: Cloudflare Access via a service token, identical Cf-Access-Client-Id/Cf-Access-Client-Secret headers (see Authentication above). There is no separate, more-open auth tier for OPTIMADE — public-read exposure and registration with providers.optimade.org are recorded Tier-2 decisions, not taken here.

Workflow outcomes: use the native API

OPTIMADE has no vocabulary for benchmark results (elastic constants, vacancy formation energies, phonon spectra, ...). Every entry carries _mlipbench_workflows_url, a link back into the native structure→workflow API (/v1/structures/{id}/workflows.json for the 834 elemental structures, /v1/query/corpus/{atoms_hash} for ASSYST-corpus structures) — follow it for anything beyond geometry and chemistry. The Python client above wraps that structure→workflow API (typed WorkflowInstances, honest geometry labels); see /v1/openapi.json for the full schema of every workflow/query route.

Regenerating the API

The static files under api_dist/api/v1/ are generated locally and pushed to R2:

# Regenerate all JSON endpoints (routes heavy Python builders via srun)
npm run build:api

# Push to R2 (requires rclone configured with r2:benchmark-assets)
PUSH_API=1 bash tools/push_to_r2.sh

The light-weight node-only step (skips Python builders) is also available:

npm run build:api:node