Cluster (SLURM)

class-e generate can split a design into chunks and write a SLURM array job (one task per chunk) plus a merge job. No MPI is involved: each task is an independent process.

class-e generate cfg.yaml --slurm            # write the scripts only
class-e generate cfg.yaml --slurm --submit   # write and submit both jobs
class-e status data/toy/parts                # progress of the chunks
class-e merge data/toy/parts/toy_train_N256_s0 -o data/toy/data_train.npz

Chunks checkpoint every run.checkpoint_every points and resume after a crash or a requeue.

The cluster: section

cluster:
  profile: my-cluster          # ~/.config/class-e/clusters/my-cluster.yaml (or $CLASS_E_CLUSTERS)
  partition: long
  cpus_per_task: 4
  mem: 14G
  time: "06:00:00"
  max_running: 20              # array throttle
  setup:                       # shell lines run before class-e
    - source ~/venvs/class-e/bin/activate
  sbatch: ["--qos=long"]       # extra #SBATCH options

Keys given in the YAML override the profile.

Note

A profile that exports OPENBLAS_NUM_THREADS=1 in setup (against OpenBLAS crashes while sampling) also holds class-e train with the JAX backends to one thread for the Cholesky factorizations. Train with it unset, or on a GPU: see GPU training.

Running one chunk by hand

class-e generate cfg.yaml --design train --chunk 3 --n-chunks 16 --out data/toy/parts