🃏Engineering Probability Unit 16 – Gaussian Processes & Brownian Motion

Key Concepts and Definitions

• Gaussian processes generalize the Gaussian probability distribution to infinite dimensionality
• Consist of a collection of random variables, any finite number of which have a joint Gaussian distribution
• Completely specified by a mean function $m(x)$ and a covariance function $k(x, x')$
• Brownian motion is a continuous-time stochastic process with independent, stationary increments
• Covariance functions, also known as kernels, measure the similarity between two points in the input space
• Gaussian processes can be used for both regression and classification tasks
• Real-world applications include financial modeling (stock prices), physics (particle diffusion), and machine learning (Bayesian optimization)

Probability Theory Foundations

• Probability theory provides the mathematical framework for quantifying uncertainty and making predictions
• Key concepts include random variables, probability distributions, expectation, and variance
• Gaussian distribution, also known as the normal distribution, is a continuous probability distribution characterized by its bell-shaped curve
  • Defined by two parameters: mean $\mu$ and variance $\sigma^2$
  • Probability density function: $f(x) = \frac{1}{\sqrt{2\pi\sigma^2}} e^{-\frac{(x-\mu)^2}{2\sigma^2}}$
• Joint probability distributions describe the probability of multiple random variables occurring together
• Conditional probability measures the probability of an event occurring given that another event has already occurred
• Bayes' theorem relates conditional probabilities and is fundamental to Bayesian inference

Introduction to Gaussian Processes

• Gaussian processes extend multivariate Gaussian distributions to infinite dimensionality
• Can be thought of as a distribution over functions, where each point in the input space is associated with a random variable
• Fully characterized by a mean function $m(x)$ and a covariance function $k(x, x')$
  • Mean function represents the expected value of the function at each point
  • Covariance function captures the correlation between function values at different points
• Gaussian processes are non-parametric models, meaning they can adapt to the complexity of the data without specifying a fixed number of parameters
• Provide a principled way to incorporate prior knowledge and uncertainty in function estimation and prediction tasks

Properties of Gaussian Processes

• Marginalization property: the marginal distribution of any subset of random variables in a Gaussian process is also Gaussian
• Conditioning property: the conditional distribution of a Gaussian process given observations is also a Gaussian process
  • Allows for efficient computation of posterior distributions and updates
• Consistency: a Gaussian process is consistent if its finite-dimensional distributions are consistent with the specified mean and covariance functions
• Stationarity: a Gaussian process is stationary if its statistical properties are invariant to translations in the input space
  • Implies that the covariance function depends only on the relative positions of the points, not their absolute locations
• Smoothness: the smoothness of a Gaussian process is determined by the choice of covariance function
  • Smoother covariance functions lead to smoother sample functions

Brownian Motion: Basics and Applications

• Brownian motion, also known as the Wiener process, is a continuous-time stochastic process with independent, stationary increments
• Named after Robert Brown, who observed the random motion of particles suspended in a fluid
• Mathematical properties:
  • $B_0 = 0$ (starts at the origin)
  • $B_t - B_s \sim \mathcal{N}(0, t-s)$ for $0 \leq s < t$ (increments are normally distributed with variance proportional to time)
  • Increments are independent for non-overlapping time intervals
• Brownian motion is a fundamental building block for more complex stochastic processes (geometric Brownian motion, fractional Brownian motion)
• Applications include modeling stock prices (Black-Scholes model), particle diffusion (Einstein's theory), and random walks in various domains

Covariance Functions and Kernels

• Covariance functions, or kernels, are a crucial component of Gaussian processes that measure the similarity between two points in the input space
• Must be symmetric and positive semi-definite to ensure a valid Gaussian process
• Common covariance functions include:
  • Squared exponential (RBF) kernel: $k(x, x') = \sigma^2 \exp\left(-\frac{||x-x'||^2}{2l^2}\right)$
  • Matérn kernel: $k(x, x') = \frac{2^{1-\nu}}{\Gamma(\nu)} \left(\frac{\sqrt{2\nu}||x-x'||}{l}\right)^\nu K_\nu\left(\frac{\sqrt{2\nu}||x-x'||}{l}\right)$
  • Periodic kernel: $k(x, x') = \sigma^2 \exp\left(-\frac{2\sin^2(\pi||x-x'||/p)}{l^2}\right)$
• Choice of covariance function encodes assumptions about the smoothness, periodicity, and other properties of the underlying function
• Kernels can be combined and modified to create more expressive and problem-specific covariance functions

Modeling and Prediction with Gaussian Processes

• Gaussian processes can be used for both regression and classification tasks
• In regression, the goal is to estimate a continuous-valued function $f(x)$ given noisy observations $y = f(x) + \epsilon$
  • Gaussian process regression assumes a Gaussian likelihood and computes the posterior distribution over functions given the data
  • Predictive distribution at a new point $x_*$ is Gaussian with mean and variance determined by the posterior
• In classification, the goal is to assign discrete class labels to input points
  • Gaussian process classification models the latent function $f(x)$ as a Gaussian process and maps it to class probabilities using a link function (probit or logistic)
  • Inference is more challenging due to the non-Gaussian likelihood, but approximations (Laplace, expectation propagation) can be used
• Hyperparameter optimization is important for selecting the best covariance function and its parameters
  • Can be done using maximum likelihood estimation or Bayesian methods (marginal likelihood, MCMC)

Real-world Applications and Examples

• Gaussian processes have been successfully applied to a wide range of real-world problems
• In finance, Gaussian processes can be used to model stock prices, estimate volatility, and price options (Black-Scholes model)
• In geostatistics, Gaussian processes (kriging) are used for spatial interpolation and prediction of environmental variables (soil properties, air pollution)
• In robotics, Gaussian processes are used for motion planning, control, and learning from demonstrations
• In Bayesian optimization, Gaussian processes guide the search for the optimal hyperparameters or design parameters by balancing exploration and exploitation
• In time series analysis, Gaussian processes can be used for forecasting, anomaly detection, and modeling temporal dependencies
• In computer vision, Gaussian processes have been applied to image denoising, inpainting, and depth estimation
• In natural language processing, Gaussian processes can be used for text classification, sentiment analysis, and language modeling