Autoregressive Models

Created on Fri Mar 11 17:26:16 2022

@author: WET2RNG

class softsensor.autoreg_models.ARNN(input_channels, pred_size, window_size, rnn_window, hidden_size=None, activation='relu', bias=True, dropout=None, forecast=1, concrete_dropout=False, bn=False)

Autoregressive Neural Network with linear layers

\[window_size = rnn_window = tau\]

\[forecast = 1\]

Parameters:

• input_channels (int) – Number of input channels

• pred_size (int) – Number of predicted values

• window_size (int) – Size of the sliding window applied to the time series

• rnn_window (int) – Window Size of the Recurrent Connection before the DNN.

• hidden_size (list of int or None, optional) – List gives the size of hidden units. The default is None.

• activation (str, optional) – Activation function to activate the feature space. The default is ‘relu’.

• bias (bool, optional) – If True, bias weights are used. The default is True.

• dropout (float [0,1], optional) – Adds dropout Layers after each Linear Layer. The default is None.

• forecast (int, optional) – Size of the forecast. The default is 1

• concrete_dropout (bool, optional) – Whether to use normal or concrete dropout Layers if dropout is not None. The default is False

Return type:

None.

Examples

>>> import softsensor.autoreg_models
>>> import torch
>>> m = softsensor.autoreg_models.ARNN(2, 1, 10, 10, [16, 8])
>>> input = torch.randn(32, 2, 10)
>>> rec_input = torch.randn(32, 1, 10)
>>> output = m(input, rec_input)
>>> print(output.shape)
torch.Size([32, 1, 1])

>>> import softsensor.meas_handling as ms
>>> import numpy as np
>>> import pandas as pd
>>> t = np.linspace(0, 1.0, 101)
>>> d = {'inp1': np.random.randn(101),
         'inp2': np.random.randn(101),
         'out': np.random.randn(101)}
>>> handler = ms.Meas_handling([pd.DataFrame(d, index=t)], ['train'],
                               ['inp1', 'inp2'], ['out'], fs=100)
>>> loader = handler.give_list(window_size=10, keyword='training',
                               rnn_window=10, batch_size=1)
>>> pred = m.prediction(loader[0])
>>> print(pred.shape)
torch.Size([1, 101])

forward(inp, x_rec)

Forward function to propagate through the network

Parameters:

• inp (torch.tensor dtype=torch.float) – Input tensor for forward propagation, shape=[batch size, external channels, window_size]

• x_rec (torch.tensor, dtype=torch.float) – Recurrent Input for forward Propagation. shape=[batch size, pred_size, rnn_window]

Returns:

output – shape=[batch size, pred_size, forecast]

Return type:

torch.tensor dtype=torch.float()

forward_sens(inp)

Forward function to propagate through the network, but only with one input tensor that is already concatenated to allow for gradient-based sensitivity analysis

Parameters:

inp (torch.tensor dtype=torch.float) – Input tensor for forward propagation that is already concatenated, shape=[batch size, external channels*window_size + pred_size*rnn_window]

Returns:

output – shape=[batch size, pred_size, forecast]

Return type:

torch.tensor dtype=torch.float()

get_recurrent_weights()

Function that returns the weight that effect the Recurrent input of the Network

Returns:

recurrent_weights – List of the Weights that effect the Recurren input of the Network.

Return type:

list of weight Tensors

Example

Based on the example in the introduction

>>> rec_w = m.get_recurrent_weights()
>>> print(rec_w[0].shape)
torch.Size([16, 10])
>>> print(rec_w[1].shape)
torch.Size([8, 16])
>>> print(rec_w[2].shape)
torch.Size([1, 8])

class softsensor.autoreg_models.DensityEstimationARNN(input_channels, pred_size, window_size, rnn_window, hidden_size=None, activation='relu', bias=True, dropout=None, forecast=1, concrete_dropout=False, bn=False)

ARNN with two outputs to predict mean and variance (aleatoric uncertainty)

Parameters:

• input_channels (int) – Number of input channels

• pred_size (int) – Number of predicted values

• window_size (int) – Size of the sliding window applied to the time series

• rnn_window (int) – Window Size of the Recurrent Connection before the DNN.

• hidden_size (list of int or None, optional) – List gives the size of hidden units. The default is None.

• activation (str, optional) – Activation function to activate the feature space. The default is ‘relu’.

• bias (bool, optional) – If True, bias weights are used. The default is True.

• dropout (float [0,1], optional) – Adds dropout Layers after each Linear Layer. The default is None

• forecast (int, optional) – Size of the forecast. The default is 1

• concrete_dropout (bool, optional) – Whether to use normal or concrete dropout Layers if dropout is not None. The default is False

Return type:

None.

Examples

>>> import softsensor.autoreg_models
>>> import torch
>>> params = {'input_channels': 2,
              'pred_size': 1,
              'window_size': 10,
              'rnn_window': 10}
>>> m = softsensor.autoreg_models.DensityEstimationARNN(**params, hidden_size=[16, 8])
>>> input = torch.randn(32, 2, 10)
>>> rec_input = torch.randn(32, 1, 10)
>>> output = m(input, rec_input)
>>> print(output[0].shape) #Mean Prediction
torch.Size([32, 1, 1])
>>> print(output[1].shape) #Var Prediction
torch.Size([32, 1, 1])

estimate_uncertainty(inp, x_rec)

Wrapper of forward pass

Parameters:

• inp (torch.tensor dtype=torch.float) – Input tensor for forward propagation, shape=[batch size, external channels, window_size]

• x_rec (torch.tensor, dtype=torch.float) – Recurrent Input for forward Propagation. shape=[batch size, pred_size, rnn_window]

Returns:

mean: torch.tensor dtype=torch.float()

shape=[batch size, pred_size]

var: torch.tensor dtype=torch.float() in [0,1]

shape=[batch size, pred_size]

Return type:

(mean, var)

estimate_uncertainty_mean_std(inp, x_rec)

forward(inp, x_rec)

Forward function to propagate through the network

Parameters:

• inp (torch.tensor dtype=torch.float) – Input tensor for forward propagation, shape=[batch size, external channels, window_size]

• x_rec (torch.tensor, dtype=torch.float) – Recurrent Input for forward Propagation. shape=[batch size, pred_size, rnn_window]

Returns:

• mean (torch.tensor dtype=torch.float()) – shape=[batch size, pred_size, forecast]

• var torch.tensor dtype=torch.float() in [0,1] – shape=[batch size, pred_size, forecast]

forward_sens(inp)

Forward function to propagate through the network, but only with one input tensor that is already concatenated to allow for gradient-based sensitivity analysis

Parameters:

inp (torch.tensor dtype=torch.float) – Input tensor for forward propagation, shape=[batch size, external channels*window_size + pred_size*rnn_window]

Returns:

• mean (torch.tensor dtype=torch.float()) – shape=[batch size, pred_size, forecast]

• var torch.tensor dtype=torch.float() in [0,1] – shape=[batch size, pred_size, forecast]

get_recurrent_weights()

Function that returns the weight that effect the Recurrent input of the Network (mean network)

Returns:

recurrent_weights – List of the Weights that effect the Recurrent input of the Network.

Return type:

list of weight Tensors

prediction(dataloader, device='cpu', sens_params=None)

Prediction of a whole Time Series

Parameters:

• dataloader (Dataloader) – Dataloader to predict output

• device (str, optional) – device to compute on. The default is ‘cpu’.

• sens_params (dict, optional) – Dictionary that contains the parameters for the sensitivity analysis. Key ‘method’ defines the method for sensitivity analysis: ‘gradient’ or ‘perturbation’. Key ‘comp’ defines whether gradients are computed for sensitivity analysis. Key ‘plot’ defines whether the results of the sensitivity analysis are visualized. Key ‘verbose’ defines whether the information about the sensitivity analysis is printed. Key ‘sens_length’ defines the number of randomly sampled subset of timesteps for the analysis. The default is None, i.e. no sensitivity analysis is computed.

Returns:

• if loss_ft=None –

  (torch.Tensor, list[torch.Tensor])

  tuple of Torch Tensor of same length as input and var

• if loss_ft=torch loss funciton –

  (torch.Tensor, list[torch.Tensor], loss)

  tuple of Torch Tensor of same length as input, var and loss

class softsensor.autoreg_models.QuantileARNN(input_channels, pred_size, window_size, rnn_window, hidden_size=None, activation='relu', bias=True, dropout=None, forecast=1, concrete_dropout=False, n_quantiles=39, mean_model=None, bn=False)

ARNN with multiple outputs to predict quantiles

Parameters:

• input_channels (int) – Number of input channels

• pred_size (int) – Number of predicted values

• window_size (int) – Size of the sliding window applied to the time series

• rnn_window (int) – Window Size of the Recurent Connection before the DNN.

• hidden_sizes (list of three lists of int or None, optional) – [hidden_mean_size, hidden_var_size, hidden_shared_size] List gives the size of hidden mean, variance and shared network units. The default is None.

• activation (str, optional) – Activation function to activate the feature space. The default is ‘relu’.

• bias (bool, optional) – If True, bias weights are used. The default is True.

• dropout (float [0,1], optional) – Adds dropout Layers after each Linear Layer. The default is None

• forecast (int, optional) – Size of the forecast. The default is 1

• concrete_dropout (bool, optional) – Whether to use normal or concrete dropout Layers if dropout is not None. The default is False

• n_quantiles (int, optional) – Number of quantiles to predict. The default is 39 (median and 19 PIs between 0 and 1)

• mean_model (torch.Module, optional) – Model for point prediction. The default is None

Return type:

None.

forward(inp, x_rec)

Forward function to propagate through the quantile network

If mean_model is not None but a point prediction model, the mean_model is used for point prediction This is useful to keep the point prediction frozen during training without teacher forcing

Parameters:

• inp (torch.tensor dtype=torch.float) – Input tensor for forward propagation, shape=[batch size, external channels, window_size]

• x_rec (torch.tensor, dtype=torch.float) – Recurrent Input for forward Propagation. shape=[batch size, pred_size, rnn_window]

Returns:

pred – shape=[batch size, pred_size, n_quantiles]

Return type:

torch.tensor dtype=torch.float()

get_recurrent_weights()

Function that returns the weight that effect the Recurrent input of the Network (mean network)

Returns:

recurrent_weights – List of the Weights that effect the Recurrent input of the Network.

Return type:

list of weight Tensors

prediction(dataloader, device='cpu')

Prediction of a whole Time Series

Parameters:

• dataloader (Dataloader) – Dataloader to predict output

• device (str, optional) – device to compute on. The default is ‘cpu’.

Returns:

• if loss_ft=None –

  quantiles: list[torch.Tensor]

  list of n_quantile tensors of same length as input

• if loss_ft=torch loss funciton –

  (list[torch.Tensor], loss)

  list of n_quantile tensors of same length as input and loss

class softsensor.autoreg_models.SeparateMVEARNN(input_channels, pred_size, window_size, rnn_window, mean_model, var_hidden_size=None, activation='relu', bias=True, dropout=None, forecast=1, concrete_dropout=False, bn=False)

ARNN with two independent subnetworks to predict mean and variance (aleatoric uncertainty)

Parameters:

• input_channels (int) – Number of input channels

• pred_size (int) – Number of predicted values

• window_size (int) – Size of the sliding window applied to the time series

• rnn_window (int) – Window Size of the Recurrent Connection before the DNN.

• mean_model (torch.Module) – Model for point prediction

• var_hidden_size (list[int] or None, optional) – List gives the size of hidden variance network units. The default is None.

• activation (str, optional) – Activation function to activate the feature space. The default is ‘relu’.

• bias (bool, optional) – If True, bias weights are used. The default is True.

• dropout (float [0,1], optional) – Adds dropout Layers after each Linear Layer. The default is None

• forecast (int, optional) – Size of the forecast. The default is 1

• concrete_dropout (bool, optional) – Whether to use normal or concrete dropout Layers if dropout is not None. The default is False

Return type:

None.

Note

See “Optimal Training of Mean Variance Estimation Neural Networks” [Sluijterman et al. 2023 https://arxiv.org/abs/2302.08875]

Examples

>>> import softsensor.autoreg_models
>>> import torch
>>> params = {'input_channels': 2,
              'pred_size': 1,
              'window_size': 10,
              'rnn_window': 10}
>>> mean_model = softsensor.autoreg_models.ARNN(**params, hidden_size=[16, 8])
>>> m = softsensor.autoreg_models.SeparateMVEARNN(**params,mean_model=mean_model,
                                                  var_hidden_size=[16, 8])
>>> input = torch.randn(32, 2, 10)
>>> rec_input = torch.randn(32, 1, 10)
>>> output = m(input, rec_input)
>>> print(output[0].shape) #Mean Prediction
torch.Size([32, 1, 1])
>>> print(output[1].shape) #VarPrediction
torch.Size([32, 1, 1])

estimate_uncertainty(inp, x_rec)

Wrapper of forward pass

Parameters:

• inp (torch.tensor dtype=torch.float) – Input tensor for forward propagation, shape=[batch size, external channels, window_size]

• x_rec (torch.tensor, dtype=torch.float) – Recurrent Input for forward Propagation. shape=[batch size, pred_size, rnn_window]

Returns:

mean: torch.tensor dtype=torch.float()

shape=[batch size, pred_size]

var: torch.tensor dtype=torch.float() in [0,1]

shape=[batch size, pred_size]

Return type:

(mean, var)

estimate_uncertainty_mean_std(inp, x_rec)

Wrapper of forward pass

Parameters:

• inp (torch.tensor dtype=torch.float) – Input tensor for forward propagation, shape=[batch size, external channels, window_size]

• x_rec (torch.tensor, dtype=torch.float) – Recurrent Input for forward Propagation. shape=[batch size, pred_size, rnn_window]

Returns:

mean: torch.tensor dtype=torch.float()

shape=[batch size, pred_size]

var: torch.tensor dtype=torch.float() in [0,1]

shape=[batch size, pred_size]

Return type:

(mean, var)

forward(inp, x_rec)

Forward function to propagate through the MVE network

Parameters:

• inp (torch.tensor dtype=torch.float) – Input tensor for forward propagation, shape=[batch size, external channels, window_size]

• x_rec (torch.tensor, dtype=torch.float) – Recurrent Input for forward Propagation. shape=[batch size, pred_size, rnn_window]

Returns:

• mean (torch.tensor dtype=torch.float()) – shape=[batch size, pred_size, forecast]

• var (torch.tensor dtype=torch.float() in [0,1]) – shape=[batch size, pred_size, forecast]

forward_sens(inp)

Forward function to propagate through the network, but only with one input tensor that is already concatenated to allow for gradient-based sensitivity analysis

Parameters:

inp (torch.tensor dtype=torch.float) – Input tensor for forward propagation, shape=[batch size, external channels*window_size + pred_size*rnn_window]

Returns:

• mean (torch.tensor dtype=torch.float()) – shape=[batch size, pred_size, forecast]

• var torch.tensor dtype=torch.float() in [0,1] – shape=[batch size, pred_size, forecast]

get_recurrent_weights()

Function that returns the weight that effect the Recurrent input of the Network (mean network)

Returns:

recurrent_weights – List of the Weights that effect the Recurrent input of the Network.

Return type:

list of weight Tensors

prediction(dataloader, device='cpu', sens_params=None)

Prediction of a whole Time Series

Parameters:

• dataloader (Dataloader) – Dataloader to predict output

• device (str, optional) – device to compute on. The default is ‘cpu’.

• sens_params (dict, optional) – Dictionary that contains the parameters for the sensitivity analysis. Key ‘method’ defines the method for sensitivity analysis: ‘gradient’ or ‘perturbation’. Key ‘comp’ defines whether gradients are computed for sensitivity analysis. Key ‘plot’ defines whether the results of the sensitivity analysis are visualized. Key ‘verbose’ defines whether the information about the sensitivity analysis is printed. Key ‘sens_length’ defines the number of randomly sampled subset of timesteps for the analysis. The default is None, i.e. no sensitivity analysis is computed.

Returns:

• if loss_ft=None –

  (torch.Tensor, list[torch.Tensor])

  tuple of Torch Tensor of same length as input and var

• if loss_ft=torch loss function –

  (torch.Tensor, list[torch.Tensor], loss)

  tuple of Torch Tensor of same length as input, var and loss