fusionlab.metrics.theils_u_score¶

fusionlab.metrics.theils_u_score(y_true, y_pred, sample_weight=None, nan_policy='propagate', multioutput='uniform_average', eps=1e-08, verbose=0)[source]¶

Compute Theil’s U Statistic.

Measures the relative accuracy of a forecast compared to a naive persistence (random walk) forecast. The last dimension of the inputs is treated as the time/horizon dimension.

Theil’s U is defined as: .. math:

U = \sqrt{
\frac{\sum_{i,o,t}(y_{i,o,t} - \hat y_{i,o,t})^2}
     {\sum_{i,o,t}(y_{i,o,t} - y_{i,o,t-1})^2}
},

where sums are over valid samples \(i\), outputs \(o\) (if applicable), and time steps \(t\) (from the second time step onwards). \(y_{i,o,t}\) is the true value, \(\hat y_{i,o,t}\) is the forecast, and \(y_{i,o,t-1}\) is the true value at the previous time step (naive forecast).

U < 1: Forecast is better than the naive model.
U = 1: Forecast is as good as the naive model.
U > 1: Forecast is worse than the naive model.

Parameters:

y_true (array-like) – True target values. Expected shapes: - (n_timesteps,) for a single trajectory. - (n_samples, n_timesteps) for single output, multiple samples. - (n_samples, n_outputs, n_timesteps) for multi-output.
y_pred (array-like) – Predicted values, matching y_true in shape.
sample_weight (array-like of shape (n_samples,), optional) – Sample weights. If None, samples are equally weighted. Sum of weights must be > eps.
nan_policy ({'omit', 'propagate', 'raise'}, default 'propagate') –
How to handle NaNs in y_true or y_pred:
- 'raise': Raise an error on any NaN.
- 'omit': Drop samples (rows) containing NaNs that would affect the error calculation.
- 'propagate': If NaNs are involved in calculating the sum of squared errors for an output, that output’s U score will be NaN.
multioutput ({'raw_values', 'uniform_average'}, default 'uniform_average') –
Defines aggregation if inputs are multi-output.
- 'raw_values': Returns a U score for each output.
- 'uniform_average': U scores of all outputs are averaged.
eps (float, default 1e-8) – Small epsilon value to add to the denominator (sum of squared errors of the naive forecast) to prevent division by zero. If the denominator sum is less than eps, U might be NaN or a large value depending on the numerator.
verbose (int, default 0) – Verbosity level: 0 (silent), 1 (summary), >=2 (debug details).

Returns:

score – Theil’s U statistic. Scalar if multioutput=’uniform_average’ or if inputs represent a single output. Array of shape (n_outputs,) if multioutput=’raw_values’ and inputs are multi-output.

Return type:

float or ndarray of floats

Examples

>>> import numpy as np
>>> # from fusionlab.metrics import theils_u_score

>>> # Single output (2 samples, 4 timesteps)
>>> y_t = np.array([[1,2,3,4],[2,2,2,2]])
>>> y_p = np.array([[1,2,3,5],[2,1,2,3]])
>>> # SSE_model = ((2-2)^2+(3-3)^2+(4-5)^2) + ((2-1)^2+(2-2)^2+(2-3)^2)
>>> #           = (0+0+1) + (1+0+1) = 1 + 2 = 3
>>> # SSE_base  = ((2-1)^2+(3-2)^2+(4-3)^2) + ((2-2)^2+(2-2)^2+(2-2)^2)
>>> #           = (1+1+1) + (0+0+0) = 3 + 0 = 3
>>> # U = sqrt(3/3) = 1.0
>>> u = theils_u_score(y_t, y_p)
>>> print(f"Theil's U: {u:.4f}")
Theil's U: 1.0000

>>> # Example with NaN
>>> y_t_nan = np.array([[1,2,np.nan,4],[2,2,2,2]])
>>> y_p_nan = np.array([[1,2,3,5],[2,1,2,3]])
>>> u_prop = theils_u_score(y_t_nan, y_p_nan, nan_policy='propagate')
>>> print(f"Theil's U (propagate): {u_prop}") # Will be NaN
Theil's U (propagate): nan
>>> u_omit = theils_u_score(y_t_nan, y_p_nan, nan_policy='omit')
>>> # Sample 0 omitted. Only sample 1 used.
>>> # SSE_model_s1 = 2, SSE_base_s1 = 0. U = sqrt(2/eps) -> large or NaN
>>> # If SSE_base is near zero, result is sensitive.
>>> # For y_s1: SSE_model=2, SSE_base=0. Result depends on eps.
>>> # If sse_base < eps, np.divide returns nan.
>>> print(f"Theil's U (omit, sse_base=0): {u_omit}")
Theil's U (omit, sse_base=0): nan