GPz 2.0

Heteroscedastic Gaussian processes for uncertain and incomplete data

Introduction

This is a matlab implementation of the sparse heteroscedastic Gaussian process described here. A dataset is considered which consists of input and the target output , where n is the number of samples in the dataset and d is the dimensionality of the input. The target is generated by a function of input plus additive noise :

.

The objective of this work is to find the simplest function which maximize the probability of observing the target output given the input.

Radial Basis Functions (RBF)

The model optimizes the probability of the data generated by a linear combination of m Radial Basis Functions. In the proposed solution, the RBF has different methods,see figure 1, as follows:

• Global length-scale (GL): All the basis functions share the same length scale.
• Variable length-scale (VL): Each basis has a specific length-scale.
• Global Diagonal (GD): All the basis functions share the same diagonal covariance.
• Variable Diagonal (VD): Each basis has a specific diagonal covariance.
• Global Covariance (GC): All the basis functions share the same full covariance.
• Variable Covariance (VC): Each basis has a specific full covariance.

Figure 1: The results of training GPVL, GPVD and GPVC with different numbers of basis functions (m) on the same data. The ellipses represent the learned covariances of the RBFs. The log marginal likelihoods are shown above each plot (Almosallam ,2017)

Heteroscedastic Noise

The model predicts variance of two components. These include the model variance and the term which is the noise uncertainty and assumed to be white Gaussian noise. The predictive variance enhanced by modelling the noise variance as a function, linear combination of basis functions, of input to account for variable and input-dependent noise such as heteroscedastic noise. In the proposed code, when the heteroscedastic noise option is disabled, the model assumed a global Gaussian noise variance. However, If the point estimates was only the interest, this feature can be disabled to minimize the required number of basis functions and speedup the optimization process. Figure 2 shows the effect of modelling with heteroscedastic noise.

Figure 2: Presenting the effect of modelling the noise variance as a function of the input(x) compared to the full GP all using an RBF function. Note that the BFM with heteroscedastic noise prediction fits the data best with the largest log marginal likelihood (Almosallam ,2017)

Splitting the dataset

The data is split into three sets including training, validation and testing. The training set is used for optimizing the models while the validation set for model selection. Finally, the testing set to predict and report the results.

Input noise

The model deals with the magnitude errors in the dataset in two different ways. If the input noise option is assigned to false, the errors will be treated as input features. However, they will be used as an additional input noise if the option was set to true value.

Cost-sensitive Learning

Cost-sensitive learning is used to train the model with balanced data distribution. For this reason, the model uses input weight vector in order to weigh the samples differently during the optimization time. There are three weighting options:

• Balanced: Rare samples are weighted more heavily during the training stage.
• Normalized: This is specific to photometric redshift estimation, where each sample is weighted inversely proportional to its redshift (i..e. w=1/(1+z))
• Normal: The samples are equally important so no weights are assigned.

References

Almosallam, I. (2017). Heteroscedastic Gaussian processes for uncertain and incomplete data (Doctoral dissertation).