IDEAS home Printed from https://ideas.repec.org/p/yor/hectdg/11-25.html
   My bibliography  Save this paper

Measuring overfitting and mispecification in nonlinear models

Author

Listed:
  • Bilger M.
  • Manning W.G

Abstract

We start by proposing a new measure of overfitting expressed on the untransformed scale of the dependent variable, which is generally the scale of interest to the analyst.We then show that with nonlinear models shrinkage due to overfitting gets confounded by shrinkage—or expansion— arising from model misspecification. Out-of-sample predictive calibration can in fact be expressed as in-sample calibration times 1 minus this new measure of overfitting. We finally argue that re-calibration should be performed on the scale of interest and provide both a simulation study and a real-data illustration based on health care expenditure data.

Suggested Citation

  • Bilger M. & Manning W.G, 2011. "Measuring overfitting and mispecification in nonlinear models," Health, Econometrics and Data Group (HEDG) Working Papers 11/25, HEDG, c/o Department of Economics, University of York.
    • Handle: RePEc:yor:hectdg:11/25
    as

    Download full text from publisher

    File URL: https://www.york.ac.uk/media/economics/documents/herc/wp/11_25.pdf
    File Function: Main text
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Manning, Willard G. & Basu, Anirban & Mullahy, John, 2005. "Generalized modeling approaches to risk adjustment of skewed outcomes data," Journal of Health Economics, Elsevier, vol. 24(3), pages 465-488, May.
    2. Anirban Basu & Bhakti V. Arondekar & Paul J. Rathouz, 2006. "Scale of interest versus scale of estimation: comparing alternative estimators for the incremental costs of a comorbidity," Health Economics, John Wiley & Sons, Ltd., vol. 15(10), pages 1091-1107, October.
    3. Blough, David K. & Madden, Carolyn W. & Hornbrook, Mark C., 1999. "Modeling risk using generalized linear models," Journal of Health Economics, Elsevier, vol. 18(2), pages 153-171, April.
    4. Manning, Willard G. & Mullahy, John, 2001. "Estimating log models: to transform or not to transform?," Journal of Health Economics, Elsevier, vol. 20(4), pages 461-494, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. repec:spo:wpmain:info:hdl:2441/64devegb4f8l7a342ageb19ehc is not listed on IDEAS
    2. Anna Gdakowicz & Ewa Putek-Szelag, 2020. "Is It Possible to Overfit the Algorithm? Case Study of Mass Valuation of Land Properties in Szczecin," European Research Studies Journal, European Research Studies Journal, vol. 0(Special 2), pages 110-122.
    3. repec:hal:spmain:info:hdl:2441/64devegb4f8l7a342ageb19ehc is not listed on IDEAS

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Marcel Bilger & Willard G. Manning, 2015. "Measuring Overfitting In Nonlinear Models: A New Method And An Application To Health Expenditures," Health Economics, John Wiley & Sons, Ltd., vol. 24(1), pages 75-85, January.
    2. Jones, A. & Lomas, J. & Rice, N., 2014. "Going Beyond the Mean in Healthcare Cost Regressions: a Comparison of Methods for Estimating the Full Conditional Distribution," Health, Econometrics and Data Group (HEDG) Working Papers 14/26, HEDG, c/o Department of Economics, University of York.
    3. Jones, A.M, 2010. "Models For Health Care," Health, Econometrics and Data Group (HEDG) Working Papers 10/01, HEDG, c/o Department of Economics, University of York.
    4. Steven C. Hill & G. Edward Miller, 2010. "Health expenditure estimation and functional form: applications of the generalized gamma and extended estimating equations models," Health Economics, John Wiley & Sons, Ltd., vol. 19(5), pages 608-627, May.
    5. Borislava Mihaylova & Andrew Briggs & Anthony O'Hagan & Simon G. Thompson, 2011. "Review of statistical methods for analysing healthcare resources and costs," Health Economics, John Wiley & Sons, Ltd., vol. 20(8), pages 897-916, August.
    6. Linnea Polgreen & John Brooks, 2012. "Estimating Incremental Costs with Skew," Applied Health Economics and Health Policy, Springer, vol. 10(5), pages 319-329, September.
    7. Andrew M. Jones & James Lomas & Nigel Rice, 2015. "Healthcare Cost Regressions: Going Beyond the Mean to Estimate the Full Distribution," Health Economics, John Wiley & Sons, Ltd., vol. 24(9), pages 1192-1212, September.
    8. Onur Başer & Joseph C. Gardiner & Cathy J. Bradley & Hüseyin Yüce & Charles Given, 2006. "Longitudinal analysis of censored medical cost data," Health Economics, John Wiley & Sons, Ltd., vol. 15(5), pages 513-525, May.
    9. Keane, Michael & Stavrunova, Olena, 2016. "Adverse selection, moral hazard and the demand for Medigap insurance," Journal of Econometrics, Elsevier, vol. 190(1), pages 62-78.
    10. Manning, Willard G. & Basu, Anirban & Mullahy, John, 2005. "Generalized modeling approaches to risk adjustment of skewed outcomes data," Journal of Health Economics, Elsevier, vol. 24(3), pages 465-488, May.
    11. Liu, Lei & Conaway, Mark R. & Knaus, William A. & Bergin, James D., 2008. "A random effects four-part model, with application to correlated medical costs," Computational Statistics & Data Analysis, Elsevier, vol. 52(9), pages 4458-4473, May.
    12. Manos Matsaganis & Theodore Mitrakos & Panos Tsakloglou, 2008. "Modelling Household Expenditure on Health Care in Greece," Working Papers 68, Bank of Greece.
    13. Kathleen Carey & Theodore Stefos, 2011. "Measuring the cost of hospital adverse patient safety events," Health Economics, John Wiley & Sons, Ltd., vol. 20(12), pages 1417-1430, December.
    14. Michael Keane & Olena Stavrunova, 2011. "A smooth mixture of Tobits model for healthcare expenditure," Health Economics, John Wiley & Sons, Ltd., vol. 20(9), pages 1126-1153, September.
    15. Jean‐Paul Chaze, 2005. "Assessing household health expenditure with Box–Cox censoring models," Health Economics, John Wiley & Sons, Ltd., vol. 14(9), pages 893-907, September.
    16. Brilleman, Samuel L. & Gravelle, Hugh & Hollinghurst, Sandra & Purdy, Sarah & Salisbury, Chris & Windmeijer, Frank, 2014. "Keep it simple? Predicting primary health care costs with clinical morbidity measures," Journal of Health Economics, Elsevier, vol. 35(C), pages 109-122.
    17. Andrew M. Jones & James Lomas & Peter T. Moore & Nigel Rice, 2016. "A quasi-Monte-Carlo comparison of parametric and semiparametric regression methods for heavy-tailed and non-normal data: an application to healthcare costs," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 179(4), pages 951-974, October.
    18. Samuel L Brilleman & Hugh Gravelle & Sandra Hollinghurst & Sarah Purdy & Chris Salisbury & Frank Windmeijer, 2011. "Keep it Simple? Predicting Primary Health Care Costs with Measures of Morbidity and Multimorbidity," Working Papers 072cherp, Centre for Health Economics, University of York.
    19. Karine Moschetti & Katia Iglesias & Stéphanie Baggio & Venetia Velonaki & Olivier Hugli & Bernard Burnand & Jean-Bernard Daeppen & Jean-Blaise Wasserfallen & Patrick Bodenmann, 2018. "Health care costs of case management for frequent users of the emergency department: Hospital and insurance perspectives," PLOS ONE, Public Library of Science, vol. 13(9), pages 1-15, September.
    20. Keane, Michael & Stavrunova, Olena, 2016. "Adverse selection, moral hazard and the demand for Medigap insurance," Journal of Econometrics, Elsevier, vol. 190(1), pages 62-78.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:yor:hectdg:11/25. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Jane Rawlings (email available below). General contact details of provider: https://edirc.repec.org/data/deyoruk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.