IDEAS home Printed from https://ideas.repec.org/p/cch/wpaper/190001.html
   My bibliography  Save this paper

Publication Bias and Editorial Statement on Negative Findings

Author

Listed:
  • Cristina Blanco-Perez
  • Abel Brodeur

Abstract

In February 2015, the editors of eight health economics journals sent out an editorial statement which aims to reduce the extent of specification searching and reminds referees to accept studies that: "have potential scientific and publication merit regardless of whether such studies' empirical findings do or do not reject null hypotheses". Guided by a pre-analysis, we test whether the editorial statement decreased the extent of publication bias. Our differences-in-differences estimates suggest that the statement decreased the proportion of tests rejecting the null hypothesis by 18 percentage points. Our findings suggest that incentives may be aligned to promote more transparent research.

Suggested Citation

  • Cristina Blanco-Perez & Abel Brodeur, 2019. "Publication Bias and Editorial Statement on Negative Findings," Working Papers 190001, Canadian Centre for Health Economics.
    • Handle: RePEc:cch:wpaper:190001
    as

    Download full text from publisher

    File URL: https://www.canadiancentreforhealtheconomics.ca/wp-content/uploads/2019/08/Blanco-Perez-Brodeur-2019.pdf
    File Function: First version, 2019
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. repec:hal:spmain:info:hdl:2441/eu4vqp9ompqllr09iatr74eao is not listed on IDEAS
    2. Alberto Abadie, 2020. "Statistical Nonsignificance in Empirical Economics," American Economic Review: Insights, American Economic Association, vol. 2(2), pages 193-208, June.
    3. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    4. Emeric Henry, 2009. "Strategic Disclosure of Research Results: The Cost of Proving Your Honesty," Economic Journal, Royal Economic Society, vol. 119(539), pages 1036-1064, July.
    5. Dewald, William G & Thursby, Jerry G & Anderson, Richard G, 1986. "Replication in Empirical Economics: The Journal of Money, Credit and Banking Project," American Economic Review, American Economic Association, vol. 76(4), pages 587-603, September.
    6. Tomas Havranek & Anna Sokolova, 2016. "Do Consumers Really Follow a Rule of Thumb? Three Thousand Estimates from 130 Studies Say "Probably Not"," Working Papers 2016/08, Czech National Bank.
    7. Cristina Blanco-Perez & Abel Brodeur, 2020. "Publication Bias and Editorial Statement on Negative Findings," The Economic Journal, Royal Economic Society, vol. 130(629), pages 1226-1247.
    8. Brodeur, Abel & Cook, Nikolai & Heyes, Anthony, 2018. "Methods Matter: P-Hacking and Causal Inference in Economics," IZA Discussion Papers 11796, Institute of Labor Economics (IZA).
    9. Orley Ashenfelter & Michael Greenstone, 2004. "Estimating the Value of a Statistical Life: The Importance of Omitted Variables and Publication Bias," American Economic Review, American Economic Association, vol. 94(2), pages 454-460, May.
    10. Katherine Casey & Rachel Glennerster & Edward Miguel, 2012. "Reshaping Institutions: Evidence on Aid Impacts Using a Preanalysis Plan," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 127(4), pages 1755-1812.
    11. Glenn Ellison, 2002. "The Slowdown of the Economics Publishing Process," Journal of Political Economy, University of Chicago Press, vol. 110(5), pages 947-993, October.
    12. De Long, J Bradford & Lang, Kevin, 1992. "Are All Economic Hypotheses False?," Journal of Political Economy, University of Chicago Press, vol. 100(6), pages 1257-1272, December.
    13. David Card & Stefano DellaVigna, 2013. "Nine Facts about Top Journals in Economics," Journal of Economic Literature, American Economic Association, vol. 51(1), pages 144-161, March.
    14. Zacharias Maniadis & Fabio Tufano & John A. List, 2017. "To Replicate or Not To Replicate? Exploring Reproducibility in Economics through the Lens of a Model and a Pilot Study," Economic Journal, Royal Economic Society, vol. 127(605), pages 209-235, October.
    15. Leamer, Edward E & Leonard, Herman B, 1983. "Reporting the Fragility of Regression Estimates," The Review of Economics and Statistics, MIT Press, vol. 65(2), pages 306-317, May.
    16. Eva Vivalt, 2019. "Specification Searching and Significance Inflation Across Time, Methods and Disciplines," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 81(4), pages 797-816, August.
    17. Tomáš Havránek, 2015. "Measuring Intertemporal Substitution: The Importance Of Method Choices And Selective Reporting," Journal of the European Economic Association, European Economic Association, vol. 13(6), pages 1180-1204, December.
    18. B.D. McCullough & Kerry Anne McGeary & Teresa D. Harrison, 2008. "Do economics journal archives promote replicable research?," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 41(4), pages 1406-1420, November.
    19. Isaiah Andrews & Maximilian Kasy, 2019. "Identification of and Correction for Publication Bias," American Economic Review, American Economic Association, vol. 109(8), pages 2766-2794, August.
    20. T. D. Stanley, 2008. "Meta‐Regression Methods for Detecting and Estimating Empirical Effects in the Presence of Publication Selection," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 70(1), pages 103-127, February.
    21. Benjamin A. Olken, 2015. "Promises and Perils of Pre-analysis Plans," Journal of Economic Perspectives, American Economic Association, vol. 29(3), pages 61-80, Summer.
    22. McCloskey, Donald N, 1985. "The Loss Function Has Been Mislaid: The Rhetoric of Significance Tests," American Economic Review, American Economic Association, vol. 75(2), pages 201-205, May.
    23. Orley Ashenfelter & Michael Greenstone, 2004. "Estimating the Value of a Statistical Life: The Importance of Omitted Variables and Publication Bias," American Economic Review, American Economic Association, vol. 94(2), pages 454-460, May.
    24. Ashenfelter, Orley & Harmon, Colm & Oosterbeek, Hessel, 1999. "A review of estimates of the schooling/earnings relationship, with tests for publication bias," Labour Economics, Elsevier, vol. 6(4), pages 453-470, November.
    25. Orley Ashenfelter & Michael Greenstone, 2004. "Estimating the Value of a Statistical Life: The Importance of Omitted Variables and Publication Bias," Working Papers 858, Princeton University, Department of Economics, Industrial Relations Section..
    26. Emeric Henry, 2009. "Disclosure of research results: the cost of proving your honesty," Post-Print hal-01023670, HAL.
    27. Marianne Bertrand & Esther Duflo & Sendhil Mullainathan, 2004. "How Much Should We Trust Differences-In-Differences Estimates?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 119(1), pages 249-275.
    28. Leamer, Edward E, 1985. "Sensitivity Analyses Would Help," American Economic Review, American Economic Association, vol. 75(3), pages 308-313, June.
    29. T. D. Stanley, 2005. "Beyond Publication Bias," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 309-345, July.
    30. Martin Dufwenberg & Peter Martinsson, 2014. "Keeping Researchers Honest: The Case for Sealed-Envelope-Submissions," Working Papers 533, IGIER (Innocenzo Gasparini Institute for Economic Research), Bocconi University.
    31. Cristina Blanco-Perez & Abel Brodeur, 2019. "Transparency in empirical economic research," IZA World of Labor, Institute of Labor Economics (IZA), pages 467-467, November.
    32. Chris Doucouliagos & T.D. Stanley, 2013. "Are All Economic Facts Greatly Exaggerated? Theory Competition And Selectivity," Journal of Economic Surveys, Wiley Blackwell, vol. 27(2), pages 316-339, April.
    33. Daniele Fanelli, 2009. "How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data," PLOS ONE, Public Library of Science, vol. 4(5), pages 1-11, May.
    34. repec:hal:wpspec:info:hdl:2441/eu4vqp9ompqllr09iatr74eao is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    2. Abel Brodeur & Nikolai Cook & Anthony Heyes, 2020. "Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics," American Economic Review, American Economic Association, vol. 110(11), pages 3634-3660, November.
    3. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    4. Abel Brodeur & Scott Carrell & David Figlio & Lester Lusher, 2023. "Unpacking P-hacking and Publication Bias," American Economic Review, American Economic Association, vol. 113(11), pages 2974-3002, November.
    5. Brodeur, Abel & Cook, Nikolai & Heyes, Anthony, 2018. "Methods Matter: P-Hacking and Causal Inference in Economics," IZA Discussion Papers 11796, Institute of Labor Economics (IZA).
    6. Furukawa, Chishio, 2019. "Publication Bias under Aggregation Frictions: Theory, Evidence, and a New Correction Method," EconStor Preprints 194798, ZBW - Leibniz Information Centre for Economics.
    7. Abel Brodeur & Nikolai Cook & Carina Neisser, 2024. "p-Hacking, Data type and Data-Sharing Policy," The Economic Journal, Royal Economic Society, vol. 134(659), pages 985-1018.
    8. Roman Horvath & Ali Elminejad & Tomas Havranek, 2020. "Publication and Identification Biases in Measuring the Intertemporal Substitution of Labor Supply," Working Papers IES 2020/32, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Sep 2020.
    9. Dominika Ehrenbergerova & Josef Bajzik & Tomas Havranek, 2023. "When Does Monetary Policy Sway House Prices? A Meta-Analysis," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 71(2), pages 538-573, June.
    10. Sebastian Gechert & Tomas Havranek & Zuzana Irsova & Dominika Kolcunova, 2022. "Measuring Capital-Labor Substitution: The Importance of Method Choices and Publication Bias," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 45, pages 55-82, July.
    11. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.
    12. Tomas Havranek & Anna Sokolova, 2020. "Do Consumers Really Follow a Rule of Thumb? Three Thousand Estimates from 144 Studies Say 'Probably Not'," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 35, pages 97-122, January.
    13. Jindrich Matousek & Tomas Havranek & Zuzana Irsova, 2022. "Individual discount rates: a meta-analysis of experimental evidence," Experimental Economics, Springer;Economic Science Association, vol. 25(1), pages 318-358, February.
    14. Tomas Havranek, 2013. "Publication Bias in Measuring Intertemporal Substitution," Working Papers IES 2013/15, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Oct 2013.
    15. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    16. Anton Astakhov & Tomas Havranek & Jiri Novak, 2019. "Firm Size And Stock Returns: A Quantitative Survey," Journal of Economic Surveys, Wiley Blackwell, vol. 33(5), pages 1463-1492, December.
    17. Brodeur, Abel & Cook, Nikolai & Hartley, Jonathan & Heyes, Anthony, 2022. "Do Pre-Registration and Pre-analysis Plans Reduce p-Hacking and Publication Bias?," MetaArXiv uxf39, Center for Open Science.
    18. Havranek, Tomas & Horvath, Roman & Irsova, Zuzana & Rusnak, Marek, 2015. "Cross-country heterogeneity in intertemporal substitution," Journal of International Economics, Elsevier, vol. 96(1), pages 100-118.
    19. Tomas Havranek & Anna Sokolova, 2016. "Do Consumers Really Follow a Rule of Thumb? Three Thousand Estimates from 130 Studies Say "Probably Not"," Working Papers 2016/08, Czech National Bank.
    20. Ali Elminejad & Tomas Havranek & Roman Horvath & Zuzana Irsova, 2023. "Intertemporal Substitution in Labor Supply: A Meta-Analysis," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 51, pages 1095-1113, December.

    More about this item

    Keywords

    publication bias; specification searching; pre-analysis plan; research in economics; incentives to publish;
    All these keywords.

    JEL classification:

    • A11 - General Economics and Teaching - - General Economics - - - Role of Economics; Role of Economists
    • C13 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Estimation: General
    • C44 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - Operations Research; Statistical Decision Theory
    • I10 - Health, Education, and Welfare - - Health - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cch:wpaper:190001. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Adrian Rohit Dass (email available below). General contact details of provider: https://edirc.repec.org/data/cchetca.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.