5.1 Autoregressive (AR) Models

ARIMA models are the big guns of time series forecasting. They combine three powerful techniques: differencing, autoregression, and moving averages. This combo lets us tackle complex patterns in data that change over time.

Autoregressive (AR) models are a key part of ARIMA. They predict future values based on past ones, like using yesterday's weather to guess today's. AR models are great for data with strong self-correlation, making them super useful in many fields.

Autoregressive Models: Concept and Structure

Definition and Basic Structure

Top images from around the web for Definition and Basic Structure

• 

  Frontiers | Autoregressive Times Series Methods for Time Domain Astronomy | Physics View original

  Is this image relevant?

• 

  Autoregressive processes : learning with an interactive tool – pierreh.eu View original

  Is this image relevant?

• 

  Autoregressive model - Wikipedia View original

  Is this image relevant?

• 

  Frontiers | Autoregressive Times Series Methods for Time Domain Astronomy | Physics View original

  Is this image relevant?

• 

  Autoregressive processes : learning with an interactive tool – pierreh.eu View original

  Is this image relevant?

1 of 3

Top images from around the web for Definition and Basic Structure

• 

  Frontiers | Autoregressive Times Series Methods for Time Domain Astronomy | Physics View original

  Is this image relevant?

• 

  Autoregressive processes : learning with an interactive tool – pierreh.eu View original

  Is this image relevant?

• 

  Autoregressive model - Wikipedia View original

  Is this image relevant?

• 

  Frontiers | Autoregressive Times Series Methods for Time Domain Astronomy | Physics View original

  Is this image relevant?

• 

  Autoregressive processes : learning with an interactive tool – pierreh.eu View original

  Is this image relevant?

1 of 3

• Autoregressive (AR) models are a class of linear models used for modeling and forecasting univariate time series data where the current value of the series depends on its own past values
• The basic structure of an AR model of order p, denoted as AR(p), is a linear combination of the past p values of the series, with coefficients representing the influence of each past value on the current value
• The general form of an AR(p) model is given by: $X_t = c + \phi_1 X_{t-1} + \phi_2 X_{t-2} + \ldots + \phi_p X_{t-p} + \varepsilon_t$

where: - $X_t$ is the value of the series at time t - $c$ is a constant term - $\phi_1, \phi_2, \ldots, \phi_p$ are the autoregressive coefficients - $\varepsilon_t$ is the white noise term representing random innovations or shocks

Order and Complexity

• The order p of an AR model determines the number of lagged values included in the model, with higher orders capturing more complex dependence structures but also increasing the risk of overfitting
• For example, an AR(1) model only includes the immediate past value ($X_{t-1}$), while an AR(3) model includes the past three values ($X_{t-1}, X_{t-2}, X_{t-3}$)
• Choosing the appropriate order is crucial for balancing the model's ability to capture the underlying patterns and avoid overfitting
• Information criteria such as Akaike Information Criterion (AIC) or Bayesian Information Criterion (BIC) can be used to select the optimal order by comparing the goodness of fit and complexity of different models

Stationarity Assumption

• AR models assume that the time series is stationary, meaning that its statistical properties (mean, variance, and autocorrelation) do not change over time
• Stationarity ensures that the patterns and relationships captured by the model remain consistent and reliable for forecasting
• Non-stationary series can lead to spurious regression results and unreliable forecasts, requiring transformations such as differencing or detrending to achieve stationarity

Characteristics of AR Models

Components and Coefficients

• The main components of an AR model include the order p, the autoregressive coefficients ($\phi_1, \phi_2, \ldots, \phi_p$), and the white noise term ($\varepsilon_t$) representing the random innovations or shocks to the system
• The autoregressive coefficients quantify the relationship between the current value of the series and its past values, with positive coefficients indicating a direct relationship and negative coefficients indicating an inverse relationship
• For instance, if $\phi_1 = 0.8$ in an AR(1) model, it means that a 1-unit increase in $X_{t-1}$ is associated with a 0.8-unit increase in $X_t$, assuming other factors remain constant

White Noise Term

• The white noise term ($\varepsilon_t$) is assumed to be independently and identically distributed (i.i.d.) with zero mean and constant variance, representing the unpredictable component of the series
• The white noise term captures the random fluctuations or shocks that are not explained by the autoregressive components
• The independence assumption implies that the white noise terms are uncorrelated across time, while the identical distribution assumption ensures that their statistical properties remain constant

Stability Condition

• The sum of the autoregressive coefficients in a stationary AR model must be less than 1 in absolute value to ensure stability and avoid explosive behavior

• Mathematically, for an AR(p) model to be stationary, the following condition must hold: $|\phi_1 + \phi_2 + \ldots + \phi_p| < 1$

• If the stability condition is violated, the series will exhibit explosive or non-stationary behavior, making it unsuitable for AR modeling and requiring alternative approaches such as differencing or transformations

Time Series Forecasting with AR Models

Model Identification and Order Selection

• The first step in developing an AR model is to determine the appropriate order p by examining the partial autocorrelation function (PACF) of the time series, which measures the correlation between the series and its lagged values after removing the effect of intervening lags
• The PACF plot can help identify the significant lags and suggest the order p of the AR model, with the last significant spike indicating the maximum lag to include
• For example, if the PACF plot shows significant spikes at lags 1 and 3, an AR(3) model might be appropriate to capture the dependence structure

Coefficient Estimation

• Once the order is determined, the autoregressive coefficients can be estimated using methods such as ordinary least squares (OLS) or maximum likelihood estimation (MLE)
• OLS minimizes the sum of squared residuals between the observed values and the model's predictions, while MLE finds the parameter values that maximize the likelihood of observing the given data
• The estimated coefficients provide insights into the strength and direction of the relationship between the current value and its lagged values

Forecasting and Interpretation

• The fitted AR model can be used to generate point forecasts and prediction intervals for future values of the series, with the forecast horizon depending on the order of the model and the available data
• Point forecasts provide a single estimate of the future value, while prediction intervals quantify the uncertainty associated with the forecast by providing a range of plausible values
• The interpretation of AR model coefficients involves assessing the magnitude and sign of each coefficient, with larger absolute values indicating stronger influence of the corresponding lagged value on the current value
• Positive coefficients suggest a direct relationship (an increase in the lagged value leads to an increase in the current value), while negative coefficients indicate an inverse relationship (an increase in the lagged value leads to a decrease in the current value)

Stationarity Requirements for AR Models

Importance of Stationarity

• Stationarity is a crucial assumption for AR models, as non-stationary series can lead to spurious regression results and unreliable forecasts
• A stationary time series has constant mean, variance, and autocorrelation structure over time, allowing the model to capture the underlying patterns and make accurate predictions
• Non-stationary series often exhibit trends, seasonality, or changing variance, which can distort the relationships and lead to misleading conclusions

Visual Inspection and Statistical Tests

• Visual inspection of the time series plot can provide initial insights into the presence of trends, seasonality, or changing variance, which are indicators of non-stationarity
• Trends can be identified by a consistent upward or downward movement in the series over time, while seasonality is characterized by regular patterns or cycles
• Statistical tests such as the Augmented Dickey-Fuller (ADF) test or the Kwiatkowski-Phillips-Schmidt-Shin (KPSS) test can be used to formally assess the stationarity of the series
• The ADF test has the null hypothesis of non-stationarity (presence of a unit root), while the KPSS test has the null hypothesis of stationarity
• Rejecting the null hypothesis in the ADF test or failing to reject it in the KPSS test provides evidence of stationarity

Transformations for Non-Stationary Series

• If the series is found to be non-stationary, transformations such as differencing or detrending can be applied to remove the non-stationary components and make the series suitable for AR modeling
• Differencing involves taking the difference between consecutive observations to remove trends or seasonality, with the order of differencing determined by the degree of non-stationarity
• Detrending involves subtracting the estimated trend component from the series to obtain a stationary residual series
• Log transformations can be used to stabilize the variance if the series exhibits increasing or decreasing variability over time
• After applying the necessary transformations, the resulting stationary series can be modeled using AR techniques for accurate forecasting and inference