Evaluation and metrics¶

Metrics¶

class_e.stats.metrics ¶

Accuracy metrics for emulated observables.

CMB spectra are compared in units of their cosmic-variance error, the standard yardstick for emulators used in inference (a fraction of \(\sigma_{\rm CV}\) per multipole is invisible to any experiment). Relative errors are meaningless for spectra that cross zero (\(C_\ell^{TE}\)), so they are used only for strictly positive observables without a cosmic-variance model (\(P(k)\), transfer functions, ...).

mse ¶

mse(y_true, y_pred)

Mean squared error between true and predicted values.

Parameters:

Name	Type	Description	Default
`y_true`	`array_like`	True values.	required
`y_pred`	`array_like`	Predicted values, same shape as `y_true`.	required

Returns:

Type	Description
`float`	The mean over all elements, \(\frac{1}{n}\sum_i (y_i - \hat y_i)^2\), for input of any shape.

Raises:

Type	Description
`ValueError`	If the shapes of `y_true` and `y_pred` differ.

cv_sigma ¶

cv_sigma(
    obs: str, ell: ndarray, spectra: Mapping[str, ndarray]
) -> Optional[np.ndarray]

Full-sky cosmic-variance error of spectrum obs, same shape as spectra[obs].

For \(XX \in \{TT, EE, PP\}\) and for \(TE\) respectively,

\[ \sigma_{C_\ell^{XX}} = \sqrt{\frac{2}{2\ell+1}}\,|C_\ell^{XX}|, \qquad \sigma_{C_\ell^{TE}} = \sqrt{\frac{C_\ell^{TT} C_\ell^{EE} + (C_\ell^{TE})^2}{2\ell+1}}. \]

All spectra must use the same convention (\(C_\ell\) or \(D_\ell\)); the ratio of an error to \(\sigma\) does not depend on it. Returns None when obs is not a CMB spectrum or TE lacks TT and EE.

normalized_errors ¶

normalized_errors(
    obs: str,
    y_true: ndarray,
    y_pred: ndarray,
    ell: Optional[ndarray] = None,
    spectra: Optional[Mapping[str, ndarray]] = None,
    signed: bool = False,
) -> tuple[np.ndarray, str]

Absolute (or, with signed, signed: emulated - true) emulator errors in the most meaningful unit available.

Parameters:

Name	Type	Description	Default
`obs`	`str`	Name of the observable (TT, TE, EE, PP, mPk, ...).	required
`y_true`	`ndarray`	True and emulated values, shape (N, L).	required
`y_pred`	`ndarray`	True and emulated values, shape (N, L).	required
`ell`	`ndarray`	Multipoles of the columns, shape (L,), for CMB spectra.	`None`
`spectra`	`mapping of str to numpy.ndarray`	True spectra of the same N samples (keys TT, EE, ...), needed for the cosmic variance; `spectra[obs]` defaults to `y_true`.	`None`
`signed`	`bool`	Return emulated - true instead of its absolute value.	`False`

Returns:

Name	Type	Description
`errors`	`ndarray`	Errors, shape (N, L), in the unit named by `metric`.
`metric`	`str`	The unit used: `'cosmic_variance'`: \(\|\Delta\| / \sigma_{\rm CV}\) (CMB spectra, see `cv_sigma`), `'relative'`: \(\|\Delta\| / \|y_{\rm true}\|\) (observables that never vanish), `'peak_relative'`: \(\|\Delta\| / \max_\ell \|y_{\rm true}\|\) per sample (anything else, e.g. TE without TT and EE).

error_summary ¶

error_summary(errors: ndarray) -> dict

p50, p95, p99 and maximum of errors over all samples and outputs.

relative_percent ¶

relative_percent(
    obs: str,
    y_true: ndarray,
    y_pred: ndarray,
    spectra: Optional[Mapping[str, ndarray]] = None,
) -> tuple[np.ndarray, str]

Signed relative error in percent, and an axis label (mathtext) for it.

TE (it crosses zero): \(100\,(\mathrm{pred}-\mathrm{true})/\sqrt{C_\ell^{TT}C_\ell^{EE}}\) when spectra holds TT and EE, the standard normalisation of a cross-spectrum;
observables that never vanish: \(100\,(\mathrm{pred}-\mathrm{true})/|\mathrm{true}|\);
anything else: relative to each sample's peak \(\max|\mathrm{true}|\).

Accuracy reports¶

class_e.commands.evaluate ¶

Emulator accuracy reports: evaluate.

Global percentiles hide where an emulator fails (e.g. near a prior edge), so a report breaks the errors down by multipole range and by bins of any expression of the parameters (--by "w0 + wa:-3,-0.5,-0.3,-0.15,0"), gives the compression (PCA) floor and lists the worst test points with their parameters.

Errors are in units of the cosmic variance for CMB spectra (\(|\Delta C_\ell| / \sigma_{\rm CV}\)), relative otherwise (class_e.stats.metrics.normalized_errors); percentiles are taken over test points and outputs together, as in error_summary.

The functions are importable (error_report, evaluate_bundle, load_test_set).

error_report ¶

error_report(
    obs: str,
    y_true: ndarray,
    y_pred: ndarray,
    *,
    ell: Optional[ndarray] = None,
    spectra: Optional[Mapping[str, ndarray]] = None,
    theta: Optional[ndarray] = None,
    names: Optional[Sequence[str]] = None,
    ell_ranges: Optional[Sequence[float]] = None,
    by: Optional[tuple[str, Sequence[float]]] = None,
    worst: int = 5,
    floor: Optional[ndarray] = None,
) -> dict

Accuracy of y_pred against y_true (both (N, L)).

Parameters:

Name	Type	Description	Default
`obs`	`str`	Name of the observable (TT, TE, EE, PP, mPk, ...).	required
`y_true`	`ndarray`	True and emulated values, shape (N, L).	required
`y_pred`	`ndarray`	True and emulated values, shape (N, L).	required
`ell`	`ndarray`	Multipoles of the columns (CMB spectra: errors in units of \(\sigma_{\rm CV}\)).	`None`
`spectra`	`mapping of str to numpy.ndarray`	True spectra of the same points (TT and EE are needed for TE's \(\sigma_{\rm CV}\)).	`None`
`theta`	`ndarray`	Parameters of the points, shape (N, D), for `by` and the worst-point list.	`None`
`names`	`sequence of str`	Names of the D parameters.	`None`
`ell_ranges`	`sequence of float`	Edges of the multipole ranges reported separately (default 2, 30, 500, 2500, 4000).	`None`
`by`	`tuple of (str, sequence of float)`	`(expression, bin edges)`: the errors per bin of an expression of the parameters.	`None`
`worst`	`int`	Number of worst points (by their largest error) to list.	`5`
`floor`	`ndarray`	The compression floor's reconstruction of `y_true` (e.g. `emulator.reconstruct`).	`None`

Returns:

Type	Description
`dict`	With `metric`, `overall`, `ell_ranges`, `by`, `floor` and `worst`.

evaluate_bundle ¶

evaluate_bundle(
    bundle: Any,
    test: str | Path | Mapping[str, Any],
    *,
    observables: Optional[Sequence[str]] = None,
    ell_ranges: Optional[Sequence[float]] = None,
    by: Optional[tuple[str, Sequence[float]]] = None,
    worst: int = 5,
    data: Optional[Mapping[str, AccuracyData]] = None,
) -> dict

error_report of every (or each requested) spectrum of a bundle on a test set.

The test set must have the bundle's parameters (any column order) and contain the bundle's multipoles. Returns {observable: report}; observables the test set does not contain (e.g. a scalar h) are skipped. Pass data (from accuracy_data) to reuse its predictions.

load_test_set ¶

load_test_set(path: str | Path) -> dict

A dataset written by class-e generate: {"theta", "names", "ell", "spectra"}.

Figures¶

class_e.plotting ¶

Figures of emulator accuracy (class-e evaluate --plots/--report), drawn with cplots.

Needs the plot extra (pip install 'class-e[plot]'). The arrays come from class_e.commands.evaluate.accuracy_data; the figures to draw from a class_e.config.PlotsSpec (the YAML's evaluate.plots).

make_figures ¶

make_figures(
    data: Mapping[str, Any],
    spec,
    *,
    reference: Optional[ndarray] = None,
    labels: Optional[list] = None,
) -> Dict[str, Any]

Every figure spec (a class_e.config.PlotsSpec) asks for: {name: cplots plot}.

write_report ¶

write_report(
    path: str | Path,
    *,
    reports: Mapping[str, Any],
    figures: Mapping[str, Any],
    bundle: Any = None,
    bundle_path: Optional[str] = None,
    test_path: Optional[str] = None,
    dpi: int = 160,
) -> Path

Write the report: a header (bundle, data), the figures, then one table card per observable.

figures are cplots plots/figures (rendered here as PNG); reports are error_reports.