Uncertainty Calibration

class softsensor.calibration.TemperatureScaling(model, temperature)

Wrapper class for uncertainty models that applies temperature scaling after prediction

Parameters:

• model (uncertainty model) – model that is used for prediction

• temperature (tensor[float]) – factors for std scaling (squared for)

Return type:

None.

Examples

Define Model

>>> import softsensor.autoreg_models
>>> from softsensor.calibration import TemperatureScaling
>>> import torch
>>> m = softsensor.autoreg_models.DensityEstimationARNN(2, 1, 10, 10, [16, 8])
>>> temps = torch.tensor([.5])
>>> scaled_model = TemperatureScaling(m, temps)
>>> input = torch.randn(32, 2, 10)
>>> rec_input = torch.randn(32, 1, 10)
>>> output = scaled_model(input, rec_input)
>>> print(output[0].shape) # mean
torch.Size([32, 1, 1])
>>> print(output[1].shape) # new var
torch.Size([32, 1, 1])

Define Data

>>> import softsensor.meas_handling as ms
>>> import numpy as np
>>> import pandas as pd
>>> t = np.linspace(0, 1.0, 101)
>>> d = {'inp1': np.random.randn(101),
         'inp2': np.random.randn(101),
         'out': np.random.randn(101)}
>>> handler = ms.Meas_handling([pd.DataFrame(d, index=t)], ['train'],
                               ['inp1', 'inp2'], ['out'], fs=100)

Check model prediction scaled and unscaled

>>> loader = handler.give_list(window_size=10, keyword='training',
                               rnn_window=10, batch_size=1)

>>> mean, var = m.prediction(loader[0])
>>> print(var[0][:5]*0.25)
tensor([0.1769, 0.1831, 0.1769, 0.1770, 0.1798])
>>> mean, var_scaled = scaled_model.prediction(loader[0])
>>> print(var[0][:5])
tensor([0.1769, 0.1831, 0.1769, 0.1770, 0.1798])

estimate_uncertainty_mean_std(inp, x_rec)

forward(*args)

Forward function to propagate through the network

Parameters:

*args (input arguments) – (need to be the same as the Point model in __init__() needs)

Returns:

• mean (torch.tensor dtype=torch.float()) – shape=[batch size, pred_size]

• std (torch.tensor dtype=torch.float() in [0,1]) – shape=[batch size, pred_size]

forward_sens(*args)

Forward function to propagate through the network

Parameters:

*args (input arguments) – (need to be the same as the Point model in __init__() needs)

Returns:

• mean (torch.tensor dtype=torch.float()) – shape=[batch size, pred_size]

• std (torch.tensor dtype=torch.float() in [0,1]) – shape=[batch size, pred_size]

get_recurrent_weights()

Function that returns the weight that effect the Recurrent input of the underlying Network (mean network)

Returns:

recurrent_weights – List of the Weights that effect the Recurrent input of the Network.

Return type:

list of weight Tensors

prediction(dataloader, device='cpu', sens_params=None)

Prediction of a whole Time Series

Parameters:

• dataloader (Dataloader) – Dataloader to predict output

• device (str, optional) – device to compute on. The default is ‘cpu’.

• sens_params (dict, optional) – Dictionary that contains the parameters for the sensitivity analysis. Key ‘method’ defines the method for sensitivity analysis: ‘gradient’ or ‘perturbation’. Key ‘comp’ defines whether gradients are computed for sensitivity analysis. Key ‘plot’ defines whether the results of the sensitivity analysis are visualized. Key ‘sens_length’ defines the number of randomly sampled subset of timesteps for the analysis. (If not a multiple of the model’s forecast, the number will be rounded up to the next multiple.) The default is None, i.e. no sensitivity analysis is computed.

Returns:

• if loss_ft=None –

  (torch.Tensor, list[torch.Tensor])

  tuple of Torch Tensor of same length as input and var

• if loss_ft=torch loss funciton –

  (torch.Tensor, list[torch.Tensor], loss)

  tuple of Torch Tensor of same length as input, var and loss

softsensor.calibration.optimize_temperature(model, data_handle, val_tracks, criterion=<function ece>, device='cpu', worst_cal=0.5, optimizer_bounds=(0.01, 5))

Optimizes the calibration temperature for each output on the calibration set

Parameters:

• model (uncertainty model) – model to calibrate

• calibration_loader (List[Dataloader]) – dataset to fit the scaling factor

• criterion (function with inputs (mean, targets, var), optional) – criterion to evaluate for optimization. The default is ECE

• worst_cal (float, optional) – worst score for criterion. The default is 0.5 (worst ECE)

Returns:

temperature

Return type:

torch.tensor[float], shape=[model.pred_size]