Published emulators¶

Trained bundles, and the datasets they were trained on, can be downloaded by name instead of computed locally:

class-e list                               # what is published, what is downloaded
class-e download emu lcdm --backend jax    # -> ~/.cache/class-e/blobs/<sha256>.cemu
class-e info lcdm@jax                      # a name works wherever a .cemu path does

from class_e.theory import EmulatedCosmology

cosmo = EmulatedCosmology.load("lcdm@jax")   # downloaded on first use, then cached

theory:
  class_e.cobaya.ClassE:
    bundle: lcdm@jax       # or a path; `offline: true` on nodes without network

Names¶

[registry/]name[@variant][:version] for a bundle, [registry/]name#split[:version] for a dataset (lcdm#train, lcdm#test).

part	meaning	default
`registry`	which registry to look in	every configured one (the name must be in only one)
`name`	the parameter space (`lcdm`, `w0wacdm-mnu`, ...)	required
`variant`	the backend of the bundle: `jax`, `sklearn`, `vae`	`jax` if JAX is installed, else `sklearn`
`version`	the bundle's version	newest one this class-e can read (`latest`)

Each published file is one bundle per parameter space and backend: every observable of the space (CMB spectra, $P(k)$, distances, scalars) is in it.

A string is a name only if it is not an existing file and has no .cemu suffix, so paths keep working unchanged.

Registries¶

A registry is a registry.json listing, per parameter space, the published files with their sha256 and download sources (mirrors, tried in order). Every download is checked against the sha256 before it is used.

class-e registry list                                   # in order of precedence
class-e registry add fzu /scratch/cosmo/emus/registry.json   # or an https:// url
class-e registry remove fzu
class-e registry refresh

The list is kept in ~/.config/class-e/registries.yaml ($CLASS_E_REGISTRIES to use another file); official comes first unless the file sets it (official: null turns it off). $CLASS_E_REGISTRY=<url or path> replaces the whole list, e.g. in CI.

Relative path sources in a registry file are read next to it, so a group can share emulators from a directory on a cluster:

{"schema": "class-e-registry/1", "name": "fzu",
 "spaces": {"ede": {"summary": "EDE (AxiCLASS)", "license": "CC-BY-4.0", "cite": ["arXiv:..."],
   "artifacts": [{"kind": "emu", "variant": "jax", "version": "1", "schema": "1.0",
                  "sha256": "...", "size": 5320616, "min_class_e": "0.0.6",
                  "sources": [{"host": "file", "path": "ede-jax-1.cemu"}]}]}}}

A space may record its fingerprint (see Parameter space and CLASS build) and classy build, which class-e list shows. A downloaded bundle whose own fingerprint differs from the registry's is refused.

Source hosts:

host	source fields
`http`, `https`, `github`, `zenodo`	`url` (and `meta_url` for a dataset's sidecar, if not `url` + `.meta.json`)	interrupted downloads resume
`hf`	`repo`, `filename`, `revision` (a commit), `repo_type` (`model` or `dataset`)	plain HTTPS; `$HF_TOKEN` for private repositories
`file`	`path` (relative: next to the registry file)

Sources of other hosts are skipped.

Datasets¶

class-e download data lcdm                 # every split (train, test, ...) of the newest version
class-e download data lcdm --split test
class-e download data --for lcdm@jax       # exactly the sets this bundle was trained and tested on
class-e train --dataset lcdm#train --test-dataset lcdm#test ...

A dataset is downloaded with its .meta.json sidecar, and both are checked against the registry. Downloads over 100 MB are confirmed first (--yes skips this). --for finds the datasets by the sha256 the bundle records, whatever their names. In a registry, a dataset is {"kind": "data", "split": "train", "meta_sha256": "...", ...}, and every source serves the sidecar next to the file.

Lineage and reproducing a bundle¶

class-e lineage lcdm@jax      # bundle <- training and test sets <- settings and CLASS build
class-e reproduce lcdm@jax -o again.cemu

lineage shows a bundle's space, CLASS build and backend. For its training and test sets it shows the size, the generation settings (design, seed, YAML), the CLASS build and where to get them: the recorded path if the file there is unchanged, the cache, or a registry entry.

reproduce gets those datasets (downloading them if needed) and retrains with the bundle's recorded training configuration. Per observable, it prints the original and new test errors, and the largest relative difference between the two bundles' predictions at the original's reference points. GP fits start from random restarts, so expect close numbers, not identical ones.

Publishing (registry maintainers)¶

# a group registry on a shared disk: copy next to it
class-e publish lcdm-jax.cemu --space lcdm --registry-file /scratch/emus/registry.json

# the official registry (a clone of class-e-registry): emulators as GitHub release assets,
# datasets on a Hugging Face dataset repository
class-e publish lcdm-jax.cemu --space lcdm --registry-file ~/class-e-registry/registry.json \
    --to github --gh-repo rcalderonb6/class-e-registry --gh-tag emulators \
    --summary "Flat LCDM, 6 parameters" --license CC-BY-4.0
class-e publish train.npz --space lcdm --split train --registry-file ~/class-e-registry/registry.json \
    --to hf --hf-repo rcalderonb6/class-e-data --hf-type dataset

publish checks the file before uploading anything (--dry-run stops there):

it records its parameter space (--allow-no-space for older files);
its CLASS build had no uncommitted changes (--allow-dirty): nobody could rebuild it;
it emulates the space the registry already lists under that name;
the version (default: the next one) and the file are not in the registry yet.

Then it asks for confirmation, uploads to each destination and adds the entry, with one source per destination, to the registry file. The entry includes what search needs: the space's parameter box and CLASS build, and the file's observables.

`--to`	needs	source
`file` (default)	`--dir` (default: next to the registry file)	`file`
`github`	the `gh` command, logged in; `--gh-repo`, `--gh-tag` (the release is created if missing)	release asset
`hf`	`class-e[hf]`, `huggingface-cli login`; `--hf-repo` (must exist), `--hf-type`	the upload's commit

Uploads to GitHub and Hugging Face are public. Commit the registry file afterwards; for the official registry, open a pull request.

Releases and a DOI (Zenodo)¶

class-e release --registry-file ~/class-e-registry/registry.json --dry-run   # what would be archived
class-e release --registry-file ~/class-e-registry/registry.json --author "Calderon, Rodrigo" \
    --license CC-BY-4.0                       # $ZENODO_TOKEN; --sandbox to try on sandbox.zenodo.org

A release puts every file of the registry in one Zenodo record: the files of earlier releases and those published since. The first release creates the record; later ones publish a new version of it. All versions share one concept DOI, which is added to every space's cite: cite that one. New files get a zenodo source as a further mirror. Files not on this machine are downloaded from their sources first. The registry's zenodo section records each release (version, DOI, record, files).

A published Zenodo record is permanent, so release asks first. The token needs the deposit:write and deposit:actions scopes.

Finding emulators¶

class-e search -p w0 -p wa -o Hz                # spaces with w0 and wa, files with Hz
class-e search -p h=0.6:0.75 --class AxiCLASS   # a prior box covering h in [0.6, 0.75]
class-e search --kind data --json               # JSON, for scripts (also `class-e list --json`)

-p NAME asks for a parameter, and -p NAME=LO:HI for a box that covers that range (either end may be empty). -o asks for an observable, --class for a CLASS build's name and --kind for emu or data. Entries published before class-e recorded this metadata match only queries that don't ask for it.

Running a registry¶

class-e registry init class-e-registry --name official   # registry.json, README, CI workflow
class-e registry check registry.json                     # entries and sources (no network)
class-e registry check registry.json --download emu      # + download, verify and load every bundle
class-e registry check registry.json --reachable         # + every remote source still serves its file

check reports these as errors: unparsable files, versions or files listed twice, bad sha256s, missing sources or sidecar checksums, and file sources that don't exist. It warns about spaces without a fingerprint, parameter box or license, Hugging Face sources not pinned to a commit, and hosts this class-e can't download from (--strict makes warnings fail too). It exits 1 on errors.

The workflow that init writes (.github/workflows/check.yml) runs --download emu on every pull request and push, and --reachable weekly.

Checking a bundle¶

class-e verify lcdm@jax      # or a path

For a name, this checks that the cached file still has the registry's sha256 and that the bundle records the registry's space. For any bundle, it checks that every array matches the sha256 in the manifest and that the emulators reproduce the reference predictions stored at training time, on this machine and with the installed libraries. It exits 1 if a check fails. Then it prints the bundle's summary.

Offline use¶

--offline (or $CLASS_E_OFFLINE=1, or offline: true in Cobaya) uses only the cache: registries from their last fetched copy, files already downloaded. Names already downloaded resolve even when no registry can be read. Download on a login node, run on compute nodes:

class-e download emu lcdm --backend jax     # login node
export CLASS_E_OFFLINE=1                    # job script

Cache¶

class-e cache path            # $CLASS_E_CACHE, else $XDG_CACHE_HOME/class-e, else ~/.cache/class-e
class-e cache remove lcdm@jax # forget a download
class-e cache prune           # delete files nothing uses, interrupted downloads

Files are stored by sha256, so two names for the same file share one copy.

API¶

class_e.hub.resolve.resolve ¶

resolve(
    target: str | PathLike,
    *,
    kind: Optional[str] = None,
    offline: Optional[bool] = None,
    download: bool = True,
    cache: Optional[CacheDir] = None,
    registries: Optional[Sequence[Registry]] = None,
    progress: Optional[ProgressCallback] = None,
) -> Path

A local file for target: an existing path, or a published bundle's or dataset's name.

Parameters:

Name	Type	Description	Default
`target`	`str or path`	A path, `[registry/]name[@variant][:version]` (a bundle, e.g. `lcdm@jax`) or `[registry/]name#split[:version]` (a dataset, e.g. `lcdm#train`).	required
`kind`	`('emu', 'data')`	What `target` names when it does not say (`name#split` is always a dataset).	`"emu"`
`offline`	`bool`	Use only the cache and the cached registries (default: `$CLASS_E_OFFLINE`).	`None`
`download`	`bool`	Download a name that is not in the cache (else `HubError`).	`True`

Examples:

>>> from class_e.hub import resolve
>>> resolve("lcdm@jax")
PosixPath('/home/me/.cache/class-e/blobs/9f3c....cemu')

class_e.hub.resolve.lookup ¶

lookup(
    spec: Spec | str,
    *,
    kind: Optional[str] = None,
    offline: Optional[bool] = None,
    registries: Optional[Sequence[Registry]] = None,
    cache: Optional[CacheDir] = None,
    progress: Optional[ProgressCallback] = None,
) -> tuple[Space, Artifact]

The registry entry spec names (see class_e.hub.registry.find); kind defaults to the name's (data for name#split).

class_e.hub.resolve.install ¶

install(
    artifact: Artifact,
    *,
    cache: Optional[CacheDir] = None,
    offline: Optional[bool] = None,
    force: bool = False,
    space: Optional[Space] = None,
    progress: Optional[ProgressCallback] = None,
) -> Path

The cached file of artifact, downloaded first if needed (or if force).

A dataset comes with its sidecar (from the same source). With the registry's space, a download whose space fingerprint differs from the registry's is refused.

class_e.hub.verify.verify ¶

verify(
    target: str,
    *,
    offline: Optional[bool] = None,
    progress: Optional[ProgressCallback] = None,
) -> VerifyReport

Check the bundle target (path or name); see the module docstring.

class_e.hub.lineage.lineage ¶

lineage(
    path: Path,
    *,
    registries: Optional[Sequence[Registry]] = None,
    cache: Optional[CacheDir] = None,
    progress: Optional[ProgressCallback] = None,
) -> list[Node]

The bundle at path and the datasets it records, each with what is known and where it is.

class_e.hub.publish.plan ¶

plan(
    file: str | PathLike,
    registry_file: str | PathLike,
    space: str,
    *,
    variant: Optional[str] = None,
    split: Optional[str] = None,
    version: Optional[str] = None,
    summary: Optional[str] = None,
    license: Optional[str] = None,
    cite: Sequence[str] = (),
    min_class_e: Optional[str] = None,
    allow_dirty: bool = False,
    allow_no_space: bool = False,
) -> Plan

Check file (.cemu or .npz) for publication under space in registry_file.

class_e.hub.publish.publish ¶

publish(
    plan: Plan,
    uploaders: Sequence[Uploader],
    *,
    progress: Optional[ProgressCallback] = None,
) -> dict

Upload with every uploader, then add the entry to the registry file; the entry written.

A failed upload stops before the registry is changed (files already uploaded stay).

class_e.hub.release.release ¶

release(
    plan: ReleasePlan,
    client: ZenodoClient,
    creators: Sequence[str],
    *,
    title: str,
    description: str = "",
    license: Optional[str] = None,
    progress: Optional[ProgressCallback] = None,
) -> dict

Publish the release on Zenodo and update the registry file; its zenodo section.

class_e.hub.search.search ¶

search(
    registries: Sequence[Registry],
    *,
    params: Sequence[ParamQuery] = (),
    observables: Sequence[str] = (),
    classy: Optional[str] = None,
    kind: Optional[str] = None,
    compatible_only: bool = False,
) -> list[tuple[Space, Artifact]]

The (space, artifact) pairs matching the query (see the module docstring).

class_e.hub.audit.audit ¶

audit(
    registry_file: str | Path,
    *,
    reachable: bool = False,
    download: Literal["none", "emu", "all"] = "none",
    progress: Optional[ProgressCallback] = None,
) -> list[Problem]

Problems of the registry at registry_file (empty: none); see the module docstring.