Function Autoencoders: A Neural Network Approach to Gaussian Processes

Gaussian processes (GPs) are data-efficient and flexible probabilistic methods that learn distributions of functions based on given priors. However, GPs suffer from unscalability as they become very computationally expensive on large datasets, and choosing the appropriate priors for GPs can be nontrivial. In this project, I investigated a neural network (NN) alternative to GPs, and introduced the function autoencoders that preserve GPs’ own advantages and avoid their weaknesses with NNs’ benefits. I tested the performance of the various function autoencoders on a 1-dimensional function regression task, using random functions generated by a GP with varying kernel parameters, and compared and analysed their results. The trained function autoencoder models indeed have the ability to learn distributions over random functions, and performed decently on the selected task. Moreover, the function autoencoders demonstrate a great potential for further improvements.