Daniel Borup, Philippe Goulet Coulombe, David E. Rapach, Erik Christian Montes Schütte, and Sander Schwenk-Nebbe
Working Paper 2022-16b
November 2022 (Revised October 2023 and February 2024)
Abstract:
:: We introduce the performance-based Shapley value (PBSV) to measure the contributions of individual predictors to the out-of-sample loss for time-series forecasting models. Our new metric allows a researcher to anatomize out-of-sample forecasting accuracy, thereby providing valuable information for interpreting time-series forecasting models. The PBSV is model agnostic—so it can be applied to any forecasting model, including "black box" models in machine learning, and it can be used for any loss function. We also develop the TS-Shapley-VI, a version of the conventional Shapley value that gauges the importance of predictors for explaining the in-sample predictions in the entire sequence of fitted models that generates the time series of out-of-sample forecasts. We then propose the model accordance score to compare predictor ranks based on the TS-Shapley-VI and PBSV, thereby linking the predictors' in-sample importance to their contributions to out-of-sample forecasting accuracy. We illustrate our metrics in an application forecasting US inflation.
JEL classification: C22, C45, C52, C53, E31, E37
Key words: model interpretation, Shapley value, predictor importance, loss function, machine learning, inflation
https://doi.org/10.29338/wp2022-16b
The authors thank seminar and conference participants at the European Commission Joint Research Center: Online Seminar, 2022 International Symposium on Forecasting, the Workshop on Advances in Alternative Data and Machine Learning for Macroeconomics and Finance, the Federal Reserve Bank of Atlanta, the 12th European Central Bank Conference on Forecasting Techniques, and the International Association for Applied Econometrics 2023 Conference, as well as Daniele Bianchi, Giulio Caperna, Todd Clark, Marco Colagrossi, Jonas N. Eriksen, Claudia Foroni (Workshop on Advances in Alternative Data and Machine Learning discussant), Nikolay Gospodinov, Andreas Joseph, Juri Marcucci, Michael McCracken, Marcelo Medeiros, Stig Møller, Mirco Rubin, and Michel van der Wel (ECB Conference on Forecasting Techniques discussant), for insightful comments. The views expressed here are those of the authors and not necessarily those of the Federal Reserve Bank of Atlanta or the Federal Reserve System. Any remaining errors are the authors' responsibility.
Daniel Borup is with Aarhus University. Philippe Goulet Coulombe is with the Université du Québec à Montrééal. David E. Rapach is with the Federal Reserve Bank of Atlanta. Erik Christian Montes Schütte is with Aarhus University and DFI. Sander Schwenk-Nebbe is with Aarhus University. Please address questions regarding content to David Rapach (corresponding author), Research Department, Federal Reserve Bank of Atlanta, 1000 Peachtree Street NE, Atlanta, GA 30309.
To receive e-mail notifications about new papers, subscribe. Under "Publications" select "Working Papers."