scoring

scoring

Scoring rules for natural gradient boosting.

A scoring rule maps a predicted distribution and observed target to a scalar loss. The engine uses the scoring rule's score, d_score, metric, and natural_gradient methods during the boosting loop.

Two scoring rules are provided:

• :class:LogScore — negative log-likelihood (default, equivalent to MLE).
• :class:CRPScore — Continuous Ranked Probability Score, a proper scoring rule that rewards well-calibrated predictive CDFs.

CRPS is only defined for continuous (and discrete-count) distributions, not for categorical classification.

ScoringRule

Bases: Protocol

Protocol for pluggable scoring rules.

score

score(
    dist: Distribution, y: NDArray[floating]
) -> NDArray[floating]

Per-sample score.

PARAMETER	DESCRIPTION
dist	Predicted distribution instance. TYPE: Distribution
y	Observed target values, shape [n_samples]. TYPE: NDArray[floating]

RETURNS	DESCRIPTION
NDArray[floating]	Score values, shape [n_samples]. Lower is better.

Source code in ngboost_lightning/scoring.py

def score(
    self,
    dist: Distribution,
    y: NDArray[np.floating],
) -> NDArray[np.floating]:
    """Per-sample score.

    Args:
        dist: Predicted distribution instance.
        y: Observed target values, shape ``[n_samples]``.

    Returns:
        Score values, shape ``[n_samples]``. Lower is better.
    """
    ...

d_score

d_score(
    dist: Distribution, y: NDArray[floating]
) -> NDArray[floating]

Gradient of score w.r.t. distribution parameters.

PARAMETER	DESCRIPTION
dist	Predicted distribution instance. TYPE: Distribution
y	Observed target values, shape [n_samples]. TYPE: NDArray[floating]

RETURNS	DESCRIPTION
NDArray[floating]	Gradient array, shape [n_samples, n_params].

Source code in ngboost_lightning/scoring.py

def d_score(
    self,
    dist: Distribution,
    y: NDArray[np.floating],
) -> NDArray[np.floating]:
    """Gradient of score w.r.t. distribution parameters.

    Args:
        dist: Predicted distribution instance.
        y: Observed target values, shape ``[n_samples]``.

    Returns:
        Gradient array, shape ``[n_samples, n_params]``.
    """
    ...

metric

metric(dist: Distribution) -> NDArray[floating]

Riemannian metric tensor for the natural gradient.

PARAMETER	DESCRIPTION
dist	Predicted distribution instance. TYPE: Distribution

RETURNS	DESCRIPTION
NDArray[floating]	Metric tensor, shape [n_samples, n_params, n_params].

Source code in ngboost_lightning/scoring.py

def metric(
    self,
    dist: Distribution,
) -> NDArray[np.floating]:
    """Riemannian metric tensor for the natural gradient.

    Args:
        dist: Predicted distribution instance.

    Returns:
        Metric tensor, shape ``[n_samples, n_params, n_params]``.
    """
    ...

natural_gradient

natural_gradient(
    dist: Distribution, y: NDArray[floating]
) -> NDArray[floating]

Natural gradient: metric^{-1} @ d_score.

PARAMETER	DESCRIPTION
dist	Predicted distribution instance. TYPE: Distribution
y	Observed target values, shape [n_samples]. TYPE: NDArray[floating]

RETURNS	DESCRIPTION
NDArray[floating]	Natural gradient, shape [n_samples, n_params].

Source code in ngboost_lightning/scoring.py

def natural_gradient(
    self,
    dist: Distribution,
    y: NDArray[np.floating],
) -> NDArray[np.floating]:
    """Natural gradient: metric^{-1} @ d_score.

    Args:
        dist: Predicted distribution instance.
        y: Observed target values, shape ``[n_samples]``.

    Returns:
        Natural gradient, shape ``[n_samples, n_params]``.
    """
    ...

total_score

total_score(
    dist: Distribution,
    y: NDArray[floating],
    sample_weight: NDArray[floating] | None = None,
) -> float

Weighted mean score across all samples.

PARAMETER	DESCRIPTION
dist	Predicted distribution instance. TYPE: Distribution
y	Observed target values, shape [n_samples]. TYPE: NDArray[floating]
sample_weight	Per-sample weights, shape [n_samples]. If None, all samples are weighted equally. TYPE: NDArray[floating] | None DEFAULT: None

RETURNS	DESCRIPTION
float	Scalar (weighted) mean score.

Source code in ngboost_lightning/scoring.py

def total_score(
    self,
    dist: Distribution,
    y: NDArray[np.floating],
    sample_weight: NDArray[np.floating] | None = None,
) -> float:
    """Weighted mean score across all samples.

    Args:
        dist: Predicted distribution instance.
        y: Observed target values, shape ``[n_samples]``.
        sample_weight: Per-sample weights, shape ``[n_samples]``.
            If ``None``, all samples are weighted equally.

    Returns:
        Scalar (weighted) mean score.
    """
    ...

LogScore

Negative log-likelihood scoring rule (default).

Delegates to the distribution's existing score, d_score, metric, and natural_gradient methods. This is a zero-cost wrapper that preserves backward compatibility.

score

score(
    dist: Distribution, y: NDArray[floating]
) -> NDArray[floating]

Per-sample NLL.

Source code in ngboost_lightning/scoring.py

def score(
    self,
    dist: Distribution,
    y: NDArray[np.floating],
) -> NDArray[np.floating]:
    """Per-sample NLL."""
    return dist.score(y)

d_score

d_score(
    dist: Distribution, y: NDArray[floating]
) -> NDArray[floating]

Gradient of NLL w.r.t. parameters.

Source code in ngboost_lightning/scoring.py

def d_score(
    self,
    dist: Distribution,
    y: NDArray[np.floating],
) -> NDArray[np.floating]:
    """Gradient of NLL w.r.t. parameters."""
    return dist.d_score(y)

metric

metric(dist: Distribution) -> NDArray[floating]

Fisher Information matrix.

Source code in ngboost_lightning/scoring.py

def metric(
    self,
    dist: Distribution,
) -> NDArray[np.floating]:
    """Fisher Information matrix."""
    return dist.metric()

natural_gradient

natural_gradient(
    dist: Distribution, y: NDArray[floating]
) -> NDArray[floating]

Natural gradient using Fisher Information.

Source code in ngboost_lightning/scoring.py

def natural_gradient(
    self,
    dist: Distribution,
    y: NDArray[np.floating],
) -> NDArray[np.floating]:
    """Natural gradient using Fisher Information."""
    return dist.natural_gradient(y)

total_score

total_score(
    dist: Distribution,
    y: NDArray[floating],
    sample_weight: NDArray[floating] | None = None,
) -> float

Weighted mean NLL.

Source code in ngboost_lightning/scoring.py

def total_score(
    self,
    dist: Distribution,
    y: NDArray[np.floating],
    sample_weight: NDArray[np.floating] | None = None,
) -> float:
    """Weighted mean NLL."""
    return float(np.average(dist.score(y), weights=sample_weight))

CRPScore

Continuous Ranked Probability Score.

Dispatches to crps_score, crps_d_score, and crps_metric methods on each distribution. Raises NotImplementedError if the distribution does not implement CRPS (e.g. Categorical).

The CRPS for a distribution with CDF F and observation y is:

.. math::

CRPS(F, y) = \int_{-\infty}^{\infty} (F(x) - \mathbf{1}(x \geq y))^2 dx

Each distribution provides a closed-form implementation.

score

score(
    dist: Distribution, y: NDArray[floating]
) -> NDArray[floating]

Per-sample CRPS.

Source code in ngboost_lightning/scoring.py

def score(
    self,
    dist: Distribution,
    y: NDArray[np.floating],
) -> NDArray[np.floating]:
    """Per-sample CRPS."""
    return dist.crps_score(y)

d_score

d_score(
    dist: Distribution, y: NDArray[floating]
) -> NDArray[floating]

Gradient of CRPS w.r.t. parameters.

Source code in ngboost_lightning/scoring.py

def d_score(
    self,
    dist: Distribution,
    y: NDArray[np.floating],
) -> NDArray[np.floating]:
    """Gradient of CRPS w.r.t. parameters."""
    return dist.crps_d_score(y)

metric

metric(dist: Distribution) -> NDArray[floating]

Riemannian metric for CRPS natural gradient.

Source code in ngboost_lightning/scoring.py

def metric(
    self,
    dist: Distribution,
) -> NDArray[np.floating]:
    """Riemannian metric for CRPS natural gradient."""
    return dist.crps_metric()

natural_gradient

natural_gradient(
    dist: Distribution, y: NDArray[floating]
) -> NDArray[floating]

Natural gradient under the CRPS metric.

Default: solve metric @ nat_grad = d_score per sample. Distributions may override crps_natural_gradient for a fast path.

Source code in ngboost_lightning/scoring.py

def natural_gradient(
    self,
    dist: Distribution,
    y: NDArray[np.floating],
) -> NDArray[np.floating]:
    """Natural gradient under the CRPS metric.

    Default: solve ``metric @ nat_grad = d_score`` per sample.
    Distributions may override ``crps_natural_gradient`` for a
    fast path.
    """
    if hasattr(dist, "crps_natural_gradient"):
        ng: NDArray[np.floating] = dist.crps_natural_gradient(y)
        return ng
    grad = dist.crps_d_score(y)
    met = dist.crps_metric()
    result: NDArray[np.floating] = np.linalg.solve(met, grad[..., np.newaxis])[
        ..., 0
    ]
    return result

total_score

total_score(
    dist: Distribution,
    y: NDArray[floating],
    sample_weight: NDArray[floating] | None = None,
) -> float

Weighted mean CRPS.

Source code in ngboost_lightning/scoring.py

def total_score(
    self,
    dist: Distribution,
    y: NDArray[np.floating],
    sample_weight: NDArray[np.floating] | None = None,
) -> float:
    """Weighted mean CRPS."""
    return float(np.average(dist.crps_score(y), weights=sample_weight))