IDEAS home Printed from https://ideas.repec.org/p/cdf/wpaper/2018-24.html
   My bibliography  Save this paper

A New Approach for Detecting Shifts in Forecast Accuracy

Author

Listed:

Abstract

Forecasts play a critical role at inflation targeting central banks, such as the Bank of England. Breaks in the forecast performance of a model can potentially incur important policy costs. Commonly used statistical procedures, however, implicitly put a lot of weight on type I errors (or false positives), which result in a relatively low power of tests to identify forecast breakdowns in small samples. We develop a procedure which aims at capturing the policy cost of missing a break. We use data-based rules to find the test size that optimally trades of the costs associated with false positives with those that can result from a break going undetected for too long. In so doing, we also explicitly study forecast errors as a multivariate system. The covariance between forecast errors for different series, though often overlooked in the forecasting literature, not only enables us to consider testing in a multivariate setting but also increases the test power. As a result, we can tailor the choice of the critical values for each series not only to the in-sample properties of each series but also to how the series for forecast errors covary.

Suggested Citation

  • Chiu,Ching-Wai & Hayes, Simon & Kapetanios, George & Theodoridis, Konstantinos, 2018. "A New Approach for Detecting Shifts in Forecast Accuracy," Cardiff Economics Working Papers E2018/24, Cardiff University, Cardiff Business School, Economics Section.
    • Handle: RePEc:cdf:wpaper:2018/24
    as

    Download full text from publisher

    File URL: https://carbsecon.com/wp/E2018_24.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Granger, Clive W.J. & Machina, Mark J., 2006. "Forecasting and Decision Theory," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 2, pages 81-98, Elsevier.
    2. Jushan Bai & Pierre Perron, 1998. "Estimating and Testing Linear Models with Multiple Structural Changes," Econometrica, Econometric Society, vol. 66(1), pages 47-78, January.
    3. Raffaella Giacomini & Barbara Rossi, 2009. "Detecting and Predicting Forecast Breakdowns," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 76(2), pages 669-705.
    4. Frank Smets & Rafael Wouters, 2007. "Shocks and Frictions in US Business Cycles: A Bayesian DSGE Approach," American Economic Review, American Economic Association, vol. 97(3), pages 586-606, June.
    5. Hansen, Lars Peter & Sargent, Thomas J., 1980. "Formulating and estimating dynamic linear rational expectations models," Journal of Economic Dynamics and Control, Elsevier, vol. 2(1), pages 7-46, May.
    6. Tetenov, Aleksey, 2012. "Statistical treatment choice based on asymmetric minimax regret criteria," Journal of Econometrics, Elsevier, vol. 166(1), pages 157-165.
    7. Jan J. J. Groen & George Kapetanios & Simon Price, 2013. "Multivariate Methods For Monitoring Structural Change," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 28(2), pages 250-274, March.
    8. Aleksey Tetenov, 2016. "An economic theory of statistical testing," CeMMAP working papers CWP50/16, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    9. Chu, Chia-Shang James & Stinchcombe, Maxwell & White, Halbert, 1996. "Monitoring Structural Change," Econometrica, Econometric Society, vol. 64(5), pages 1045-1065, September.
    10. Ploberger, Werner & Kramer, Walter & Kontrus, Karl, 1989. "A new test for structural stability in the linear regression model," Journal of Econometrics, Elsevier, vol. 40(2), pages 307-318, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jeronymo Marcondes Pinto & Jennifer L. Castle, 2022. "Machine Learning Dynamic Switching Approach to Forecasting in the Presence of Structural Breaks," Journal of Business Cycle Research, Springer;Centre for International Research on Economic Tendency Surveys (CIRET), vol. 18(2), pages 129-157, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Raffaella Giacomini & Barbara Rossi, 2015. "Forecasting in Nonstationary Environments: What Works and What Doesn't in Reduced-Form and Structural Models," Annual Review of Economics, Annual Reviews, vol. 7(1), pages 207-229, August.
    2. Jana Eklund & George Kapetanios & Simon Price, 2013. "Robust Forecast Methods and Monitoring during Structural Change," Manchester School, University of Manchester, vol. 81, pages 3-27, October.
    3. Eklund, Jana & Kapetanios, George & Price, Simon, 2010. "Forecasting in the presence of recent structural change," Bank of England working papers 406, Bank of England.
    4. KUROZUMI, Eiji & 黒住, 英司, 2016. "Monitoring Parameter Constancy with Endogenous Regressors," Discussion Papers 2016-01, Graduate School of Economics, Hitotsubashi University.
    5. Abhijit Sharma & Kelvin G Balcombe & Iain M Fraser, 2009. "Non-renewable resource prices: Structural breaks and long term trends," Economics Bulletin, AccessEcon, vol. 29(2), pages 805-819.
    6. Raffaella Giacomini, 2015. "Economic theory and forecasting: lessons from the literature," Econometrics Journal, Royal Economic Society, vol. 18(2), pages 22-41, June.
    7. Pesaran, M. Hashem & Timmermann, Allan, 2004. "How costly is it to ignore breaks when forecasting the direction of a time series?," International Journal of Forecasting, Elsevier, vol. 20(3), pages 411-425.
    8. Anatolyev Stanislav & Kosenok Grigory, 2018. "Sequential Testing with Uniformly Distributed Size," Journal of Time Series Econometrics, De Gruyter, vol. 10(2), pages 1-22, July.
    9. Pesaran, M. Hashem & Timmermann, Allan, 2005. "Small sample properties of forecasts from autoregressive models under structural breaks," Journal of Econometrics, Elsevier, vol. 129(1-2), pages 183-217.
    10. Boot, Tom & Pick, Andreas, 2020. "Does modeling a structural break improve forecast accuracy?," Journal of Econometrics, Elsevier, vol. 215(1), pages 35-59.
    11. Yazgan, M. Ege & Özkan, Harun, 2015. "Detecting structural changes using wavelets," Finance Research Letters, Elsevier, vol. 12(C), pages 23-37.
    12. Carsten J. Crede, 2019. "A Structural Break Cartel Screen for Dating and Detecting Collusion," Review of Industrial Organization, Springer;The Industrial Organization Society, vol. 54(3), pages 543-574, May.
    13. Rapach, David & Zhou, Guofu, 2013. "Forecasting Stock Returns," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 328-383, Elsevier.
    14. Tom Boot & Andreas Pick, 2017. "A near optimal test for structural breaks when forecasting under square error loss," Tinbergen Institute Discussion Papers 17-039/III, Tinbergen Institute.
    15. Raffaella Giacomini, 2013. "The relationship between DSGE and VAR models," CeMMAP working papers CWP21/13, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    16. Terasvirta, Timo, 2006. "Forecasting economic variables with nonlinear models," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 8, pages 413-457, Elsevier.
    17. Giovanni Angelini & Luca Fanelli, 2016. "Misspecification and Expectations Correction in New Keynesian DSGE Models," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 78(5), pages 623-649, October.
    18. John Geweke & Joel Horowitz & M. Hashem Pesaran, 2006. "Econometrics: A Bird’s Eye View," CESifo Working Paper Series 1870, CESifo.
    19. Oscar Jorda & Alan Taylor & Sanjay Singh, 2019. "The Long-Run Effects of Monetary Policy," 2019 Meeting Papers 1307, Society for Economic Dynamics.
    20. Barigozzi, Matteo & Trapani, Lorenzo, 2020. "Sequential testing for structural stability in approximate factor models," Stochastic Processes and their Applications, Elsevier, vol. 130(8), pages 5149-5187.

    More about this item

    Keywords

    Forecast Breaks; Statistical Decision Making; Central Banking;
    All these keywords.

    JEL classification:

    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods
    • E47 - Macroeconomics and Monetary Economics - - Money and Interest Rates - - - Forecasting and Simulation: Models and Applications
    • E58 - Macroeconomics and Monetary Economics - - Monetary Policy, Central Banking, and the Supply of Money and Credit - - - Central Banks and Their Policies

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cdf:wpaper:2018/24. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Yongdeng Xu (email available below). General contact details of provider: https://edirc.repec.org/data/ecscfuk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.