6.4 Evaluating Forecast Accuracy

Evaluating forecast accuracy is crucial in time series analysis and forecasting. It helps determine how well predictions match actual values, guiding model selection and improvement. Various metrics like MAE, MSE, and RMSE quantify errors, each offering unique insights into forecast performance.

Accuracy metrics have limitations, though. They don't account for business context or economic impact of errors. Cross-validation techniques help assess model robustness across different data subsets. Combining quantitative metrics with qualitative factors ensures a comprehensive evaluation of forecasting models.

Forecast Accuracy Metrics

Quantifying Prediction Errors

Top images from around the web for Quantifying Prediction Errors

• 

  Lab 10 - Machine Learning [CS Open CourseWare] View original

  Is this image relevant?

• 

  Mean absolute error - Wikipedia View original

  Is this image relevant?

• 

  预测问题评价指标：MAE、MSE、R-Square、MAPE和RMSE - 代码先锋网 View original

  Is this image relevant?

• 

  Lab 10 - Machine Learning [CS Open CourseWare] View original

  Is this image relevant?

• 

  Mean absolute error - Wikipedia View original

  Is this image relevant?

1 of 3

Top images from around the web for Quantifying Prediction Errors

• 

  Lab 10 - Machine Learning [CS Open CourseWare] View original

  Is this image relevant?

• 

  Mean absolute error - Wikipedia View original

  Is this image relevant?

• 

  预测问题评价指标：MAE、MSE、R-Square、MAPE和RMSE - 代码先锋网 View original

  Is this image relevant?

• 

  Lab 10 - Machine Learning [CS Open CourseWare] View original

  Is this image relevant?

• 

  Mean absolute error - Wikipedia View original

  Is this image relevant?

1 of 3

• Forecast accuracy metrics measure the difference between predicted and actual values
• Mean Absolute Error (MAE) calculates the average of absolute differences between forecasted and actual values
  • Gives equal weight to all errors
  • Formula: $MAE = \frac{1}{n} \sum_{i=1}^n |y_i - \hat{y}_i|$
  • Where $y_i$ represents actual values and $\hat{y}_i$ represents forecasted values
• Mean Squared Error (MSE) computes the average of squared differences between forecasted and actual values
  • Penalizes larger errors more heavily
  • Formula: $MSE = \frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2$
• Root Mean Squared Error (RMSE) takes the square root of MSE
  • Provides a metric in the same unit as the original data
  • Emphasizes larger errors
  • Formula: $RMSE = \sqrt{\frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2}$

Interpreting Accuracy Metrics

• Lower values of MAE, MSE, and RMSE indicate better forecast accuracy
• Perfect forecast represented by a value of 0
• Scale-dependent metrics influenced by the scale of the data being forecasted
• Interpretation requires understanding units and scale in relation to business context
• Example interpretations
  • MAE of 5 units in retail sales forecasting (5 items off on average)
  • RMSE of 2.5 degrees Celsius in temperature forecasting

Model Performance Comparison

Quantitative Evaluation Methods

• Accuracy metrics provide objective basis for comparing forecasting models
• Use consistent metrics across different models for fair comparisons
• Align choice of accuracy metric with specific forecasting task goals
• Rank models based on accuracy metric scores (lower scores generally better)
• Apply statistical significance tests to determine meaningful differences
  • Diebold-Mariano test assesses statistical significance of forecast accuracy differences
• Consider multiple accuracy metrics for comprehensive performance view
  • Captures different aspects of forecast quality (accuracy, bias, consistency)

Visual and Analytical Comparison Tools

• Employ visual tools to complement numerical metrics
  • Error distribution plots reveal patterns in forecast errors
  • Residual analysis graphs highlight model fit quality
• Create comparison tables summarizing multiple metrics for each model
• Utilize time series cross-validation to assess performance across different time periods
• Conduct sensitivity analysis to evaluate model robustness to input changes
• Example comparison scenario
  • Model A: MAE = 10, RMSE = 15
  • Model B: MAE = 12, RMSE = 13
  • Analysis considers trade-off between average error (MAE) and large error impact (RMSE)

Limitations of Forecast Accuracy

Metric Limitations

• Forecast accuracy metrics provide simplified view of model performance
• Scale-dependent metrics mislead when comparing forecasts across different scales
  • MAE of 100 units significant for small-scale data, negligible for large-scale data
• Metrics typically ignore economic or operational impact of forecast errors
  • Underforecasting inventory may lead to stockouts and lost sales
  • Overforecasting may result in excess inventory and storage costs
• Over-reliance on single metric leads to suboptimal model selection
  • Different metrics favor different types of forecasting models
• Evaluation period choice significantly impacts accuracy metrics
  • Short evaluation periods may not capture seasonal patterns
  • Long periods may include outdated data not representative of current trends

Business Context Considerations

• Incorporate business context alongside numerical metrics
  • Consider asymmetric costs of over-forecasting versus under-forecasting
  • Example (retail inventory)
    • Under-forecasting cost high due to lost sales and customer dissatisfaction
    • Over-forecasting cost lower but includes storage and potential markdowns
• Evaluate qualitative factors alongside quantitative accuracy
  • Model interpretability crucial for stakeholder buy-in and decision-making
  • Ease of implementation impacts model adoption and maintenance costs
• Assess forecast horizon relevance to business decision-making
  • Short-term accuracy may be prioritized for operational decisions
  • Long-term accuracy critical for strategic planning

Cross-Validation for Robustness

Time Series Cross-Validation Techniques

• Implement cross-validation to assess forecasting model robustness
• Systematically partition data into training and testing sets
• Time series cross-validation methods account for temporal nature of data
  • Rolling-origin evaluation simulates real-world forecasting process
    • Incrementally increase training set size and forecast next time period
  • K-fold cross-validation adapted for time series maintains temporal order
    • Divide time series into k contiguous folds, use earlier folds for training
• Choose cross-validation method based on specific forecasting task
  • Consider data characteristics (stationarity, seasonality)
  • Align with intended forecast horizon

Cross-Validation Benefits and Considerations

• Provides insights into model stability and sensitivity to different data subsets
• Helps identify potential overfitting or underfitting
  • Overfitting detected if model performs well on training data but poorly on test data
  • Underfitting indicated by consistently poor performance across all folds
• Performance metrics across multiple folds offer robust estimate of predictive capability
• Guide model selection and hyperparameter tuning
  • Compare average performance across folds for different model configurations
• Assess model suitability for real-world deployment
• Consider computational intensity, especially for complex models or large datasets
  • Balance between thorough validation and practical time constraints
• Example cross-validation scenario
  • 5-fold time series cross-validation on monthly sales data
  • Evaluate model performance across different seasonal patterns and trend changes