IDEAS home Printed from https://ideas.repec.org/p/pra/mprapa/117706.html
   My bibliography  Save this paper

Forecast Evaluation in Macroeconomics and International Finance. Ph.D. thesis, George Washington University, Washington, DC, USA

Author

Listed:
  • Bespalova, Olga

Abstract

This dissertation focuses on forecasting rare macroeconomic events, such as GDP declines and currency crises, using non-parametric methods, highlighting the advantages of the Receiver Operating Characteristic (ROC) curves analysis and the value of qualitative information from expert surveys and textual analysis in macroeconomic forecasting. Chapter 1 shows that the qualitative WES survey can produce accurate directional macroeconomic forecasts for the USA by combining the ROC curves analysis with contingency tables, and that the ROC-optimal thresholds yield more accurate predictions than the alternatives in Hutson et al. (2014). Chapter 2 re-examines indicators of currency crises from Kaminsky and Reinhart (1999) and subsequent studies using the ROC curves analysis and shows that the ROC-optimal thresholds issue more accurate signals than the minimum noise-to-signal ratio previously used in the literature. Modified ROC curves display the relationship between the precision of sent signals and the recall of crisis episodes. Forecast combinations using several ad-hoc rules help to improve forecast accuracy. Chapter 3 highlights asymmetric information about the U.S. economy between the Federal Reserve System (FRS) and the Survey of Professional Forecasters (SPF) via textual analysis of the Federal Open Market Committee (FOMC) minutes. It shows that the SPF forecasters pay close attention to the FOMC minutes available to them at the time of the forecast deadline and efficiently use its information in their set. Yet, they could improve their forecasts should the FOMC minutes from the first quarterly meetings become available without a three-week publication lag. However, during their second quarterly meetings, the FOMC policy-makers accounted only for their own earlier assessment of the U.S. macroeconomy – they did not put weight on the SPF forecasts released a few weeks earlier in the same quarter.

Suggested Citation

  • Bespalova, Olga, 2018. "Forecast Evaluation in Macroeconomics and International Finance. Ph.D. thesis, George Washington University, Washington, DC, USA," MPRA Paper 117706, University Library of Munich, Germany.
    • Handle: RePEc:pra:mprapa:117706
    as

    Download full text from publisher

    File URL: https://mpra.ub.uni-muenchen.de/117706/1/Bespalova_CCAS_PhDThesis_2018_ETD.pdf
    File Function: original version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Ottaviani, Marco & Sorensen, Peter Norman, 2006. "The strategy of professional forecasting," Journal of Financial Economics, Elsevier, vol. 81(2), pages 441-466, August.
    2. Gorr, Wilpen L. & Schneider, Matthew J., 2013. "Large-change forecast accuracy: Reanalysis of M3-Competition data using receiver operating characteristic analysis," International Journal of Forecasting, Elsevier, vol. 29(2), pages 274-281.
    3. Balke, Nathan S & Petersen, D'Ann, 2002. "How Well Does the Beige Book Reflect Economic Activity? Evaluating Qualitative Information Quantitatively," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 34(1), pages 114-136, February.
    4. Henriksson, Roy D & Merton, Robert C, 1981. "On Market Timing and Investment Performance. II. Statistical Procedures for Evaluating Forecasting Skills," The Journal of Business, University of Chicago Press, vol. 54(4), pages 513-533, October.
    5. Lahiri, Kajal & Monokroussos, George, 2013. "Nowcasting US GDP: The role of ISM business surveys," International Journal of Forecasting, Elsevier, vol. 29(4), pages 644-658.
    6. Drehmann, Mathias & Juselius, Mikael, 2014. "Evaluating early warning indicators of banking crises: Satisfying policy requirements," International Journal of Forecasting, Elsevier, vol. 30(3), pages 759-780.
    7. Matthieu Bussière, 2013. "Balance of payment crises in emerging markets: how early were the ‘early’ warning signals?," Applied Economics, Taylor & Francis Journals, vol. 45(12), pages 1601-1623, April.
    8. Yock Y. Chong & David F. Hendry, 1986. "Econometric Evaluation of Linear Macro-Economic Models," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 53(4), pages 671-690.
    9. Carmen M. Reinhart & Graciela L. Kaminsky, 1999. "The Twin Crises: The Causes of Banking and Balance-of-Payments Problems," American Economic Review, American Economic Association, vol. 89(3), pages 473-500, June.
    10. Graciela L. Kaminsky, 2003. "Varieties of Currency Crises," NBER Working Papers 10193, National Bureau of Economic Research, Inc.
    11. Boumans Dorine & Garnitz Johanna, 2017. "Ifo World Economic Survey Database – An International Economic Expert Survey," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 237(1), pages 71-80, February.
    12. Òscar Jordà & Moritz Schularick & Alan M Taylor, 2011. "Financial Crises, Credit Booms, and External Imbalances: 140 Years of Lessons," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 59(2), pages 340-378, June.
    13. Claveria, Oscar & Pons, Ernest & Ramos, Raul, 2007. "Business and consumer expectations and macroeconomic forecasts," International Journal of Forecasting, Elsevier, vol. 23(1), pages 47-69.
    14. Hali J. Edison, 2003. "Do indicators of financial crises work? An evaluation of an early warning system," International Journal of Finance & Economics, John Wiley & Sons, Ltd., vol. 8(1), pages 11-53.
    15. Ang, Andrew & Bekaert, Geert & Wei, Min, 2007. "Do macro variables, asset markets, or surveys forecast inflation better?," Journal of Monetary Economics, Elsevier, vol. 54(4), pages 1163-1212, May.
    16. Lamont, Owen A., 2002. "Macroeconomic forecasts and microeconomic forecasters," Journal of Economic Behavior & Organization, Elsevier, vol. 48(3), pages 265-280, July.
    17. Graciela Kaminsky & Saul Lizondo & Carmen M. Reinhart, 1998. "Leading Indicators of Currency Crises," IMF Staff Papers, Palgrave Macmillan, vol. 45(1), pages 1-48, March.
    18. Faust, Jon & Wright, Jonathan H., 2009. "Comparing Greenbook and Reduced Form Forecasts Using a Large Realtime Dataset," Journal of Business & Economic Statistics, American Statistical Association, vol. 27(4), pages 468-479.
    19. Sydney C. Ludvigson, 2004. "Consumer Confidence and Consumer Spending," Journal of Economic Perspectives, American Economic Association, vol. 18(2), pages 29-50, Spring.
    20. Trueman, Brett, 1994. "Analyst Forecasts and Herding Behavior," The Review of Financial Studies, Society for Financial Studies, vol. 7(1), pages 97-124.
    21. Henzel, Steffen & Wollmershäuser, Timo, 2008. "The New Keynesian Phillips curve and the role of expectations: Evidence from the CESifo World Economic Survey," Economic Modelling, Elsevier, vol. 25(5), pages 811-832, September.
    22. Christian Seiler, 2015. "On the robustness of balance statistics with respect to nonresponse," OECD Journal: Journal of Business Cycle Measurement and Analysis, OECD Publishing, Centre for International Research on Economic Tendency Surveys, vol. 2014(2), pages 45-62.
    23. Nicolas Carnot & Vincent Koen & Bruno Tissot, 2005. "Economic Forecasting," Palgrave Macmillan Books, Palgrave Macmillan, number 978-0-230-00581-5, March.
    24. Schnader, M H & Stekler, H O, 1990. "Evaluating Predictions of Change," The Journal of Business, University of Chicago Press, vol. 63(1), pages 99-107, January.
    25. Hutson, Mark & Joutz, Fred & Stekler, Herman, 2014. "Interpreting and evaluating CESIfo's World Economic Survey directional forecasts," Economic Modelling, Elsevier, vol. 38(C), pages 6-11.
    26. Newey, Whitney & West, Kenneth, 2014. "A simple, positive semi-definite, heteroscedasticity and autocorrelation consistent covariance matrix," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 33(1), pages 125-132.
    27. Moritz Schularick & Alan M. Taylor, 2012. "Credit Booms Gone Bust: Monetary Policy, Leverage Cycles, and Financial Crises, 1870-2008," American Economic Review, American Economic Association, vol. 102(2), pages 1029-1061, April.
    28. Joutz, Fred & Stekler, H. O., 2000. "An evaluation of the predictions of the Federal Reserve," International Journal of Forecasting, Elsevier, vol. 16(1), pages 17-38.
    29. Margaret S. Pepe & Gary Longton & Holly Janes, 2009. "Estimation and comparison of receiver operating characteristic curves," Stata Journal, StataCorp LP, vol. 9(1), pages 1-16, March.
    30. Michela Nardo, 2003. "The Quantification of Qualitative Survey Data: A Critical Assessment," Journal of Economic Surveys, Wiley Blackwell, vol. 17(5), pages 645-668, December.
    31. Stekler, Herman & Symington, Hilary, 2016. "Evaluating qualitative forecasts: The FOMC minutes, 2006–2010," International Journal of Forecasting, Elsevier, vol. 32(2), pages 559-570.
    32. Ericsson, Neil R., 2016. "Eliciting GDP forecasts from the FOMC’s minutes around the financial crisis," International Journal of Forecasting, Elsevier, vol. 32(2), pages 571-583.
    33. Croushore, Dean, 2005. "Do consumer-confidence indexes help forecast consumer spending in real time?," The North American Journal of Economics and Finance, Elsevier, vol. 16(3), pages 435-450, December.
    34. Robert Goldfarb & H. O. Stekler & Joel David, 2005. "Methodological issues in forecasting: Insights from the egregious business forecast errors of late 1930," Journal of Economic Methodology, Taylor & Francis Journals, vol. 12(4), pages 517-542.
    35. Guangqin Ma & W.J. Hall, 1993. "Confidence Bands for Receiver Operating Characteristic Curves," Medical Decision Making, , vol. 13(3), pages 191-197, August.
    36. Jacob A. Mincer & Victor Zarnowitz, 1969. "The Evaluation of Economic Forecasts," NBER Chapters, in: Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance, pages 3-46, National Bureau of Economic Research, Inc.
    37. Stekler, H. O., 1991. "Macroeconomic forecast evaluation techniques," International Journal of Forecasting, Elsevier, vol. 7(3), pages 375-384, November.
    38. Christopher A. Sims, 2002. "The Role of Models and Probabilities in the Monetary Policy Process," Brookings Papers on Economic Activity, Economic Studies Program, The Brookings Institution, vol. 33(2), pages 1-62.
    39. Lahiri, Kajal & Zhao, Yongchen, 2015. "Quantifying survey expectations: A critical review and generalization of the Carlson–Parkin method," International Journal of Forecasting, Elsevier, vol. 31(1), pages 51-62.
    40. A. Berg & C. Pattillo, 1999. "What Caused the Asian Crises: An Early Warning System Approach," Economic Notes, Banca Monte dei Paschi di Siena SpA, vol. 28(3), pages 285-334, November.
    41. Bertrand Candelon & Elena-Ivona Dumitrescu & Christophe Hurlin, 2012. "How to Evaluate an Early-Warning System: Toward a Unified Statistical Framework for Assessing Financial Crises Forecasting Methods," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 60(1), pages 75-113, April.
    42. Johanna Garnitz & Timo Wollmershäuser, 2017. "Innovations to the ifo World Economic Survey," CESifo Forum, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 18(01), pages 33-36, April.
    43. Johanna Garnitz & Timo Wollmershäuser, 2017. "Neuerungen im Ifo World Economic Survey," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 70(01), pages 24-28, January.
    44. Jacob A. Mincer, 1969. "Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance," NBER Books, National Bureau of Economic Research, Inc, number minc69-1.
    45. Kaminsky, Graciela L., 2006. "Currency crises: Are they all the same?," Journal of International Money and Finance, Elsevier, vol. 25(3), pages 503-527, April.
    46. Travis J. Berge & Òscar Jordà, 2011. "Evaluating the Classification of Economic Activity into Recessions and Expansions," American Economic Journal: Macroeconomics, American Economic Association, vol. 3(2), pages 246-277, April.
    47. David H. Romer & Christina D. Romer, 2000. "Federal Reserve Information and the Behavior of Interest Rates," American Economic Review, American Economic Association, vol. 90(3), pages 429-457, June.
    48. Berg, Andrew & Pattillo, Catherine, 1999. "Predicting currency crises:: The indicators approach and an alternative," Journal of International Money and Finance, Elsevier, vol. 18(4), pages 561-586, August.
    49. Lahiri, Kajal & Wang, J. George, 2013. "Evaluating probability forecasts for GDP declines using alternative methodologies," International Journal of Forecasting, Elsevier, vol. 29(1), pages 175-190.
    50. Johanna Garnitz & Timo Wollmershäuser, 2017. "Neuerungen im Ifo World Economic Survey," Ifo Schnelldienst, Ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 70(1), pages 24-28, 01.
    51. Merton, Robert C, 1981. "On Market Timing and Investment Performance. I. An Equilibrium Theory of Value for Market Forecasts," The Journal of Business, University of Chicago Press, vol. 54(3), pages 363-406, July.
    52. repec:bla:econom:v:42:y:1975:i:166:p:123-38 is not listed on IDEAS
    53. Graciela Laura Kaminsky, 1999. "Currency and Banking Crises: The Early Warnings of Distress," IMF Working Papers 1999/178, International Monetary Fund.
    54. Michael P. Clements, 2015. "Are Professional Macroeconomic Forecasters Able To Do Better Than Forecasting Trends?," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 47(2-3), pages 349-382, March.
    55. Leitch, Gordon & Tanner, J Ernest, 1991. "Economic Forecast Evaluation: Profits versus the Conventional Error Measures," American Economic Review, American Economic Association, vol. 81(3), pages 580-590, June.
    56. Burkart, Oliver & Coudert, Virginie, 2002. "Leading indicators of currency crises for emerging countries," Emerging Markets Review, Elsevier, vol. 3(2), pages 107-133, June.
    57. Jon Frost & Ayako Saiki, 2014. "Early Warning for Currency Crises: What Is the Role of Financial Openness?," Review of International Economics, Wiley Blackwell, vol. 22(4), pages 722-743, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. S. Yanki Kalfa & Jaime Marquez, 2021. "Forecasting FOMC Forecasts," Econometrics, MDPI, vol. 9(3), pages 1-21, September.
      • S. Yanki Kalfa & Jaime Marquez, 2018. "Forecasting FOMC Forecasts," Working Papers 2018-007, The George Washington University, Department of Economics, H. O. Stekler Research Program on Forecasting.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bespalova, Olga, 2020. "GDP forecasts: Informational asymmetry of the SPF and FOMC minutes," International Journal of Forecasting, Elsevier, vol. 36(4), pages 1531-1540.
    2. Bespalova, Olga, 2015. "The Good, the Bad, and the Ugly…signals of currency crises: Does signal approach work in ex-ante forecasting of currency crises?," MPRA Paper 117863, University Library of Munich, Germany, revised 15 Jul 2017.
    3. Baghestani, Hamid, 2011. "Federal Reserve and private forecasts of growth in investment," Journal of Economics and Business, Elsevier, vol. 63(4), pages 290-305, July.
    4. Ullrich Heilemann & Herman Stekler, 2010. "Perspectives on Evaluating Macroeconomic Forecasts," Working Papers 2010-002, The George Washington University, Department of Economics, H. O. Stekler Research Program on Forecasting.
    5. Jianping Shi & Yu Gao, 2010. "A study on KLR financial crisis early-warning model," Frontiers of Economics in China, Springer;Higher Education Press, vol. 5(2), pages 254-275, June.
    6. Capistrán, Carlos, 2008. "Bias in Federal Reserve inflation forecasts: Is the Federal Reserve irrational or just cautious?," Journal of Monetary Economics, Elsevier, vol. 55(8), pages 1415-1427, November.
    7. Jianping Shi & Yu Gao, 2010. "A Study on KLR Financial Crisis Early-Warning Model," Frontiers of Economics in China-Selected Publications from Chinese Universities, Higher Education Press, vol. 5(2), pages 254-275, June.
    8. Baghestani, Hamid & AbuAl-Foul, Bassam M., 2017. "Comparing Federal Reserve, Blue Chip, and time series forecasts of US output growth," Journal of Economics and Business, Elsevier, vol. 89(C), pages 47-56.
    9. Kajal Lahiri & Liu Yang, 2018. "Confidence Bands for ROC Curves With Serially Dependent Data," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 36(1), pages 115-130, January.
    10. Carsten Detken & Olaf Weeken & Lucia Alessi & Diana Bonfim & Miguel M. Boucinha & Christian Castro & Sebastian Frontczak & Gaston Giordana & Julia Giese & Nadya Jahn & Jan Kakes & Benjamin Klaus & Jan, 2014. "Operationalising the countercyclical capital buffer: indicator selection, threshold identification and calibration options," ESRB Occasional Paper Series 05, European Systemic Risk Board.
    11. Chung‐Hua Shen & Hsing‐Hua Hsu, 2022. "The determinants of Asian banking crises—Application of the panel threshold logit model," International Review of Finance, International Review of Finance Ltd., vol. 22(1), pages 248-277, March.
    12. Jan Babecky & Tomas Havranek & Jakub Mateju & Marek Rusnak & Katerina Smidkova & Borek Vasicek, 2011. "Early Warning Indicators of Economic Crises: Evidence from a Panel of 40 Developed Countries," Working Papers 2011/08, Czech National Bank.
    13. Julien Champagne & Guillaume Poulin‐Bellisle & Rodrigo Sekkel, 2020. "Introducing the Bank of Canada staff economic projections database," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(1), pages 114-129, January.
    14. Peltonen, Tuomas A., 2006. "Are emerging market currency crises predictable? A test," Working Paper Series 571, European Central Bank.
    15. Betz, Frank & Oprică, Silviu & Peltonen, Tuomas A. & Sarlin, Peter, 2014. "Predicting distress in European banks," Journal of Banking & Finance, Elsevier, vol. 45(C), pages 225-241.
    16. Ryota Nakatani, 2017. "The Effects of Productivity Shocks, Financial Shocks, and Monetary Policy on Exchange Rates: An Application of the Currency Crisis Model and Implications for Emerging Market Crises," Emerging Markets Finance and Trade, Taylor & Francis Journals, vol. 53(11), pages 2545-2561, November.
    17. Ari, Ali, 2008. "An Early Warning Signals Approach for Currency Crises: The Turkish Case," MPRA Paper 25858, University Library of Munich, Germany, revised 2009.
    18. Henning Fischer & Marta García-Bárzana & Peter Tillmann & Peter Winker, 2014. "Evaluating FOMC forecast ranges: an interval data approach," Empirical Economics, Springer, vol. 47(1), pages 365-388, August.
    19. Hamid Baghestani, 2017. "Do US consumer survey data help beat the random walk in forecasting mortgage rates?," Cogent Economics & Finance, Taylor & Francis Journals, vol. 5(1), pages 1343017-134, January.
    20. Messina, Jeffrey D. & Sinclair, Tara M. & Stekler, Herman, 2015. "What can we learn from revisions to the Greenbook forecasts?," Journal of Macroeconomics, Elsevier, vol. 45(C), pages 54-62.

    More about this item

    Keywords

    Forecasting; macroeconomics; international finance; rare events; forecasts accuracy; forecasts evaluation; GDP; currency crises; real exchange rate (RER); trend; foreign reserves; broad money (M2); the ratio of broad money to reserves; exports; World Economic Survey (WES); Federal Reserve System (FRS); Survey of Professional Forecasters (SPF); SPF forecasters; Federal Open Market Committee (FOMC); FOMC minutes; Greenbook (GB); GB forecasts; elicit casts; textual analysis; non-parametric methods; leading indicators; binary indicators; Receiver Operating Characteristic (ROC) curves; ROC-optimal thresholds; ROC curves analysis; in-sample predictive value; out-of-sample predictive value; signal approach; precision; recall; true positive rate; false positive rate; true negative rate; false negative rate; economic surveys; consumer surveys; business tendency surveys; qualitative surveys; quantitative surveys; contingency table; present economic conditions; future economic conditions; regression approach; balance statistics; probability approach; forecast error; consensus scores; forecasting rule; directional accuracy; accuracy ratio; probability of detection; classifier; informative classifier; random guess; J-index; threshold; area under the curve (AUC); future expectations; Early Warning Indicators (EWIs); encompassing tests; rationality tests; orthogonality tests; forecasts unbiasedness; forecasts efficiency.;
    All these keywords.

    JEL classification:

    • C14 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Semiparametric and Nonparametric Methods: General
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • C23 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Models with Panel Data; Spatio-temporal Models
    • C44 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - Operations Research; Statistical Decision Theory
    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods
    • E17 - Macroeconomics and Monetary Economics - - General Aggregative Models - - - Forecasting and Simulation: Models and Applications
    • E52 - Macroeconomics and Monetary Economics - - Monetary Policy, Central Banking, and the Supply of Money and Credit - - - Monetary Policy
    • E58 - Macroeconomics and Monetary Economics - - Monetary Policy, Central Banking, and the Supply of Money and Credit - - - Central Banks and Their Policies
    • E65 - Macroeconomics and Monetary Economics - - Macroeconomic Policy, Macroeconomic Aspects of Public Finance, and General Outlook - - - Studies of Particular Policy Episodes
    • E66 - Macroeconomics and Monetary Economics - - Macroeconomic Policy, Macroeconomic Aspects of Public Finance, and General Outlook - - - General Outlook and Conditions
    • F31 - International Economics - - International Finance - - - Foreign Exchange
    • F37 - International Economics - - International Finance - - - International Finance Forecasting and Simulation: Models and Applications
    • F43 - International Economics - - Macroeconomic Aspects of International Trade and Finance - - - Economic Growth of Open Economies
    • F47 - International Economics - - Macroeconomic Aspects of International Trade and Finance - - - Forecasting and Simulation: Models and Applications

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pra:mprapa:117706. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Joachim Winter (email available below). General contact details of provider: https://edirc.repec.org/data/vfmunde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.