3.2 Stationarity and ergodicity

Stationarity and ergodicity are key concepts in stochastic processes. They help us understand how random systems behave over time and across different realizations. These properties are crucial for modeling and analyzing time series data in various fields.

Stationarity means a process's statistical properties don't change over time. Ergodicity allows us to infer a process's overall behavior from a single, long realization. Together, these concepts form the foundation for many statistical techniques used in time series analysis and signal processing.

Stationarity

• Stationarity is a fundamental concept in stochastic processes that describes the statistical properties of a process over time
• A stationary process has constant statistical properties, such as mean, variance, and autocorrelation, that do not change with time
• Understanding stationarity is crucial for modeling and analyzing time series data in various applications, such as finance, economics, and signal processing

Strict stationarity

Top images from around the web for Strict stationarity

• 

  Stationair proces - Wikipedia View original

  Is this image relevant?

• 

  Stochastic process - Wikipedia View original

  Is this image relevant?

• 

  Dynamic Stochastic General Equilibrium models made (relatively) easy with R View original

  Is this image relevant?

• 

  Stationair proces - Wikipedia View original

  Is this image relevant?

• 

  Stochastic process - Wikipedia View original

  Is this image relevant?

1 of 3

Top images from around the web for Strict stationarity

• 

  Stationair proces - Wikipedia View original

  Is this image relevant?

• 

  Stochastic process - Wikipedia View original

  Is this image relevant?

• 

  Dynamic Stochastic General Equilibrium models made (relatively) easy with R View original

  Is this image relevant?

• 

  Stationair proces - Wikipedia View original

  Is this image relevant?

• 

  Stochastic process - Wikipedia View original

  Is this image relevant?

1 of 3

• Strict stationarity requires that the joint probability distribution of any subset of random variables in a stochastic process remains the same under time shifts
• Implies that all moments of the process, including mean, variance, and higher-order moments, are constant over time
• Strict stationarity is a strong condition that is often difficult to verify in practice, especially for processes with unknown distributions

Weak stationarity

• Weak stationarity, also known as second-order stationarity, is a less restrictive form of stationarity that focuses on the first and second moments of a stochastic process
• Requires that the mean of the process is constant over time and the autocovariance function depends only on the time lag between observations, not on the absolute time
• Weak stationarity is more commonly used in practice than strict stationarity, as it is easier to test and verify using sample statistics

Wide-sense stationarity

• Wide-sense stationarity is another term for weak stationarity, emphasizing that only the first and second moments of the process are considered
• Assumes that the mean and autocovariance function of the process are finite and do not change with time
• Wide-sense stationary processes have a constant mean and a covariance function that depends only on the time lag (autocovariance function)

Covariance stationarity

• Covariance stationarity is synonymous with weak stationarity and wide-sense stationarity
• Requires that the mean and autocovariance function of the process are invariant to time shifts
• Covariance stationary processes have a constant mean and a covariance structure that depends only on the time difference between observations (autocovariance function)

Stationarity vs non-stationarity

• Non-stationary processes have statistical properties that change over time, such as a time-varying mean, variance, or autocorrelation
• Examples of non-stationary processes include trends (linear or non-linear), seasonality, and processes with time-varying volatility (heteroscedasticity)
• Distinguishing between stationary and non-stationary processes is crucial for selecting appropriate modeling techniques and avoiding spurious regression results

Ergodicity

• Ergodicity is a property of stochastic processes that relates the time average of a single realization to the ensemble average across multiple realizations
• In an ergodic process, the time average of a sufficiently long realization converges to the ensemble average as the length of the realization increases
• Ergodicity is a stronger property than stationarity, as all ergodic processes are stationary, but not all stationary processes are ergodic

Ergodic processes

• An ergodic process is one in which the time average of any measurable function of the process converges to the ensemble average as the length of the realization tends to infinity
• Ergodicity implies that the statistical properties of the process can be inferred from a single, sufficiently long realization
• Examples of ergodic processes include stationary Gaussian processes, stationary Markov chains, and certain types of stationary point processes

Ergodic theorem

• The ergodic theorem states that, for an ergodic process, the time average of a measurable function of the process converges almost surely to the ensemble average as the length of the realization tends to infinity
• The ergodic theorem provides a theoretical foundation for estimating the statistical properties of an ergodic process from a single realization
• The ergodic theorem has important implications for parameter estimation, as it justifies the use of time averages to estimate ensemble averages in ergodic processes

Relationship between stationarity and ergodicity

• Ergodicity is a stronger property than stationarity, as all ergodic processes are stationary, but not all stationary processes are ergodic
• A process can be stationary without being ergodic if it consists of multiple distinct subpopulations with different statistical properties (e.g., a mixture of stationary processes)
• For a stationary process to be ergodic, it must also satisfy a mixing condition, which ensures that the process "forgets" its initial conditions over time

Ergodicity in parameter estimation

• Ergodicity plays a crucial role in parameter estimation for stochastic processes, as it allows the use of time averages to estimate ensemble averages
• In ergodic processes, the sample mean and sample autocovariance function converge to their true values as the sample size increases, enabling consistent parameter estimation
• Ergodicity is a key assumption in many statistical inference methods, such as maximum likelihood estimation and method of moments, applied to time series data

Time averages vs ensemble averages

• Time averages and ensemble averages are two fundamental concepts in the study of stochastic processes, related to the notions of stationarity and ergodicity
• Understanding the relationship between time averages and ensemble averages is crucial for interpreting and analyzing the statistical properties of stochastic processes

Time average

• The time average is a measure of the average behavior of a stochastic process over time, computed from a single realization of the process

• For a stochastic process $X(t)$, the time average of a function $f(X(t))$ over an interval $[0, T]$ is defined as: $\bar{f}_T = \frac{1}{T} \int_0^T f(X(t)) dt$

• In practice, time averages are often estimated using sample means or other summary statistics computed from a finite realization of the process

Ensemble average

• The ensemble average is a measure of the average behavior of a stochastic process across multiple realizations, computed at a fixed point in time

• For a stochastic process $X(t)$, the ensemble average of a function $f(X(t))$ at time $t$ is defined as: $\langle f(X(t)) \rangle = \mathbb{E}[f(X(t))]$

• Ensemble averages are often used to characterize the statistical properties of a process, such as its mean, variance, and autocorrelation function

Equality of averages for ergodic processes

• For ergodic processes, the time average of a measurable function converges to the ensemble average as the length of the realization tends to infinity

• The equality of time and ensemble averages in ergodic processes is a consequence of the ergodic theorem, which states that: $\lim_{T \to \infty} \frac{1}{T} \int_0^T f(X(t)) dt = \mathbb{E}[f(X(t))]$

• The equality of averages in ergodic processes allows the estimation of ensemble properties from a single, sufficiently long realization of the process

Autocorrelation and autocovariance

• Autocorrelation and autocovariance are key concepts in the analysis of stochastic processes, particularly in the context of stationarity and ergodicity
• These functions measure the degree of similarity between observations of a process at different time lags, providing insight into the temporal dependence structure of the process

Autocorrelation function

• The autocorrelation function (ACF) measures the correlation between observations of a stochastic process at different time lags

• For a stationary process $X(t)$ with mean $\mu$ and variance $\sigma^2$, the ACF at lag $\tau$ is defined as: $\rho(\tau) = \frac{\mathbb{E}[(X(t) - \mu)(X(t+\tau) - \mu)]}{\sigma^2}$

• The ACF takes values between -1 and 1, with $\rho(0) = 1$ and $\rho(\tau) = \rho(-\tau)$ for stationary processes

• The ACF provides information about the persistence of the process and can be used to identify patterns, such as trends, seasonality, and cyclical behavior

Autocovariance function

• The autocovariance function (ACVF) measures the covariance between observations of a stochastic process at different time lags

• For a stationary process $X(t)$ with mean $\mu$, the ACVF at lag $\tau$ is defined as: $\gamma(\tau) = \mathbb{E}[(X(t) - \mu)(X(t+\tau) - \mu)]$

• The ACVF is related to the ACF by $\gamma(\tau) = \sigma^2 \rho(\tau)$, where $\sigma^2$ is the variance of the process

• The ACVF provides information about the magnitude of the temporal dependence in the process and is used in the estimation of model parameters

Stationarity and autocorrelation

• For a stationary process, the ACF and ACVF depend only on the time lag $\tau$ and not on the absolute time $t$
• The ACF and ACVF of a stationary process are well-defined and do not change over time, reflecting the constant statistical properties of the process
• The sample ACF and ACVF, computed from a finite realization of a stationary process, can be used to estimate the true ACF and ACVF of the process

Ergodicity and autocorrelation

• For an ergodic process, the sample ACF and ACVF, computed from a single realization, converge to their true values as the length of the realization tends to infinity
• The ergodicity property ensures that the temporal dependence structure of the process can be accurately estimated from a sufficiently long realization
• The convergence of the sample ACF and ACVF to their true values is a consequence of the ergodic theorem and is crucial for consistent parameter estimation in time series models

Stationarity tests

• Stationarity tests are statistical methods used to determine whether a given time series exhibits stationary behavior
• These tests are crucial for selecting appropriate modeling techniques and avoiding spurious regression results in time series analysis
• Several stationarity tests are commonly used in practice, each with its own assumptions and limitations

Visual inspection of time series

• Visual inspection of a time series plot can provide initial insights into the stationarity of the process
• Non-stationary behavior, such as trends, seasonality, and time-varying volatility, can often be identified through visual examination
• However, visual inspection is subjective and may not always provide conclusive evidence of stationarity or non-stationarity

Augmented Dickey-Fuller test

• The Augmented Dickey-Fuller (ADF) test is a widely used statistical test for assessing the presence of a unit root in a time series, which indicates non-stationarity

• The ADF test estimates a regression model of the form: $\Delta y_t = \alpha + \beta t + \gamma y_{t-1} + \sum_{i=1}^p \delta_i \Delta y_{t-i} + \varepsilon_t$

• The null hypothesis of the ADF test is that the series has a unit root (i.e., is non-stationary), while the alternative hypothesis is that the series is stationary

• The ADF test is sensitive to the choice of lag length $p$ and the inclusion of deterministic terms (constant and trend)

Kwiatkowski-Phillips-Schmidt-Shin test

• The Kwiatkowski-Phillips-Schmidt-Shin (KPSS) test is another widely used stationarity test, which assesses the null hypothesis of stationarity against the alternative of a unit root
• The KPSS test is based on the residuals from a regression of the time series on deterministic terms (constant and trend)
• The test statistic is computed as the sum of squared partial sums of the residuals, normalized by an estimate of the long-run variance
• The KPSS test is often used in conjunction with the ADF test to provide a more comprehensive assessment of stationarity

Phillips-Perron test

• The Phillips-Perron (PP) test is a non-parametric alternative to the ADF test, which allows for weakly dependent and heterogeneously distributed innovations
• The PP test estimates the same regression model as the ADF test but uses a modified test statistic that accounts for serial correlation and heteroscedasticity in the errors
• The null and alternative hypotheses of the PP test are the same as those of the ADF test (unit root vs. stationarity)
• The PP test is less sensitive to the choice of lag length than the ADF test but may have lower power in some cases

Applications of stationarity and ergodicity

• The concepts of stationarity and ergodicity have wide-ranging applications in various fields, including time series analysis, signal processing, and stochastic modeling
• Understanding the stationarity and ergodicity properties of a process is crucial for selecting appropriate modeling techniques and ensuring the validity of statistical inference

Time series analysis

• Stationarity is a fundamental assumption in many time series models, such as autoregressive (AR), moving average (MA), and autoregressive integrated moving average (ARIMA) models
• Stationarity tests are used to determine whether a time series needs to be differenced or detrended before fitting a stationary model
• Ergodicity is important for the consistency of parameter estimates and the validity of forecasts in time series analysis

Signal processing

• Stationarity and ergodicity are important concepts in the analysis and processing of random signals, such as audio, video, and communication signals
• Stationary signal processing techniques, such as Fourier analysis and spectral estimation, rely on the assumption of stationarity to provide meaningful results
• Ergodicity is crucial for the estimation of signal properties, such as power spectral density and autocorrelation, from a single realization of the signal

Markov chains

• Stationarity and ergodicity are important properties of Markov chains, which are widely used to model stochastic processes with a discrete state space
• A stationary Markov chain has a time-invariant transition probability matrix and a unique stationary distribution
• An ergodic Markov chain converges to its stationary distribution regardless of the initial state, enabling the estimation of long-run properties from a single simulation

Queueing theory

• Stationarity and ergodicity are fundamental concepts in queueing theory, which studies the behavior of waiting lines in service systems
• Stationary queueing models, such as the M/M/1 and M/M/c queues, assume that the arrival and service processes are stationary and independent
• Ergodicity is important for the existence and uniqueness of steady-state performance measures, such as average waiting time and system occupancy, in stable queueing systems