2.3 Maximum likelihood and Bayesian estimation methods

Maximum Likelihood Estimation (MLE) is a powerful statistical method for estimating model parameters. It maximizes the likelihood function, assuming data is drawn from a known probability distribution, to find parameters that make observed data most probable.

Bayesian estimation takes MLE a step further by incorporating prior knowledge about parameters. It treats parameters as random variables with distributions, updating beliefs as new data is observed. This approach handles small sample sizes better and provides uncertainty quantification.

Maximum Likelihood Estimation

Principles of maximum likelihood estimation

Top images from around the web for Principles of maximum likelihood estimation

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

• 

  Maximum likelihood estimation - Wikipedia View original

  Is this image relevant?

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

1 of 3

Top images from around the web for Principles of maximum likelihood estimation

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

• 

  Maximum likelihood estimation - Wikipedia View original

  Is this image relevant?

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

1 of 3

• Statistical method estimates model parameters maximizing likelihood function
• Assumes data drawn from known probability distribution seeks parameters making observed data most probable
• Estimates parameters of dynamic systems identifies transfer functions or state-space models
• Consistency achieves estimates converging to true values as sample size increases
• Efficiency achieves minimum variance among unbiased estimators
• Requires large sample sizes for accuracy sensitive to initial parameter guesses (gradient descent, Newton-Raphson method)

Formulation of likelihood functions

• Joint probability density function of observed data treated as function of unknown parameters
• Define probability distribution of data given parameters
• Express joint probability as product of individual probabilities
• Take logarithm to simplify calculations (log-likelihood)
• Set partial derivatives of log-likelihood to zero
• Solve resulting equations for parameter estimates
• Applies to parametric models in control systems (ARX, ARMAX)

Bayesian Estimation

Concept of Bayesian estimation

• Incorporates prior knowledge about parameters treats them as random variables with distributions
• Prior distribution represents initial belief about parameters
• Likelihood function same as in MLE
• Posterior distribution updated belief after observing data
• Handles small sample sizes better provides uncertainty quantification
• Allows incorporation of expert knowledge
• Updates prior distribution with observed data uses Bayes' theorem to compute posterior
• Choosing appropriate prior distributions presents challenge
• Computational complexity increases in high-dimensional problems

Application of Bayesian techniques

• Maximum a posteriori (MAP) estimation maximizes posterior distribution balances prior knowledge with observed data
• Posterior $p(\theta|y) \propto p(y|\theta)p(\theta)$
• Maximize $\log p(y|\theta) + \log p(\theta)$
• MAP reduces to MLE with uniform prior provides regularization through prior
• Markov Chain Monte Carlo (MCMC) methods and Variational inference offer alternative techniques
• Online parameter estimation enables adaptive control
• Robust control design accounts for uncertainty
• Choosing conjugate priors improves computational efficiency
• Non-informative priors require careful consideration in practical applications