IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp11863.html
   My bibliography  Save this paper

Distributional Impact Analysis: Toolkit and Illustrations of Impacts beyond the Average Treatment Effect

Author

Listed:
  • Bedoya, Guadalupe

    (World Bank)

  • Bittarello, Luca

    (Northwestern University)

  • Davis, Jonathan

    (University of Chicago)

  • Mittag, Nikolas

    (CERGE-EI)

Abstract

Program evaluations often focus on average treatment effects. However, average treatment effects miss important aspects of policy evaluation, such as the impact on inequality and whether treatment harms some individuals. A growing literature develops methods to evaluate such issues by examining the distributional impacts of programs and policies. This toolkit reviews methods to do so, focusing on their application to randomized control trials. The paper emphasizes two strands of the literature: estimation of impacts on outcome distributions and estimation of the distribution of treatment impacts. The article then discusses extensions to conditional treatment effect heterogeneity, that is, to analyses of how treatment impacts vary with observed characteristics. The paper offers advice on inference, testing, and power calculations, which are important when implementing distributional analyses in practice. Finally, the paper illustrates select methods using data from two randomized evaluations.

Suggested Citation

  • Bedoya, Guadalupe & Bittarello, Luca & Davis, Jonathan & Mittag, Nikolas, 2018. "Distributional Impact Analysis: Toolkit and Illustrations of Impacts beyond the Average Treatment Effect," IZA Discussion Papers 11863, Institute of Labor Economics (IZA).
    • Handle: RePEc:iza:izadps:dp11863
    as

    Download full text from publisher

    File URL: https://docs.iza.org/dp11863.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. John A. List & Azeem M. Shaikh & Yang Xu, 2019. "Multiple hypothesis testing in experimental economics," Experimental Economics, Springer;Economic Science Association, vol. 22(4), pages 773-793, December.
    2. Marianne P. Bitler & Jonah B. Gelbach & Hilary W. Hoynes, 2006. "What Mean Impacts Miss: Distributional Effects of Welfare Reform Experiments," American Economic Review, American Economic Association, vol. 96(4), pages 988-1012, September.
    3. Victor Chernozhukov & Iván Fernández‐Val & Blaise Melly, 2013. "Inference on Counterfactual Distributions," Econometrica, Econometric Society, vol. 81(6), pages 2205-2268, November.
    4. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2008. "Nonparametric Tests for Treatment Effect Heterogeneity," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 389-405, August.
    5. Cornelissen, Thomas & Dustmann, Christian & Raute, Anna & Schönberg, Uta, 2016. "From LATE to MTE: Alternative methods for the evaluation of policy interventions," Labour Economics, Elsevier, vol. 41(C), pages 47-60.
    6. Marcel Fafchamps & Julien Labonne, 2016. "Using Split Samples to Improve Inference about Causal Effects," NBER Working Papers 21842, National Bureau of Economic Research, Inc.
    7. Stéphane Bonhomme & Jean-Marc Robin, 2010. "Generalized Non-Parametric Deconvolution with an Application to Earnings Dynamics," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 77(2), pages 491-533.
    8. Amanda Kowalski, 2016. "Doing more when you're running LATE: Applying marginal treatment effect methods to examine treatment effect heterogeneity in experiments," Artefactual Field Experiments 00560, The Field Experiments Website.
    9. Jacobson, Louis S & LaLonde, Robert J & Sullivan, Daniel G, 1993. "Earnings Losses of Displaced Workers," American Economic Review, American Economic Association, vol. 83(4), pages 685-709, September.
    10. Stephen G. Donald & Yu-Chin Hsu & Robert P. Lieli, 2014. "Testing the Unconfoundedness Assumption via Inverse Probability Weighted Estimators of (L)ATT," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 32(3), pages 395-415, July.
    11. Miriam Bruhn & Luciana de Souza Leão & Arianna Legovini & Rogelio Marchetti & Bilal Zia, 2016. "The Impact of High School Financial Education: Evidence from a Large-Scale Evaluation in Brazil," American Economic Journal: Applied Economics, American Economic Association, vol. 8(4), pages 256-295, October.
    12. Pedro Carneiro & Karsten T. Hansen & James J. Heckman, 2002. "Removing the Veil of Ignorance in Assessing the Distributional Impacts of Social Policies," NBER Working Papers 8840, National Bureau of Economic Research, Inc.
    13. Louis S. Jacobson & Robert J. LaLonde & Daniel G. Sullivan, 1993. "Long-term earnings losses of high-seniority displaced workers," Economic Perspectives, Federal Reserve Bank of Chicago, vol. 17(Nov), pages 2-20.
    14. Christian N. Brinch & Magne Mogstad & Matthew Wiswall, 2017. "Beyond LATE with a Discrete Instrument," Journal of Political Economy, University of Chicago Press, vol. 125(4), pages 985-1039.
    15. James J. Heckman & Jeffrey Smith & Nancy Clements, 1997. "Making The Most Out Of Programme Evaluations and Social Experiments: Accounting For Heterogeneity in Programme Impacts," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 487-535.
    16. Brendon McConnell & Marcos Vera-Hernandez, 2015. "Going beyond simple sample size calculations: a practitioner's guide," IFS Working Papers W15/17, Institute for Fiscal Studies.
    17. Soohyung Lee & Azeem M. Shaikh, 2014. "Multiple Testing And Heterogeneous Treatment Effects: Re‐Evaluating The Effect Of Progresa On School Enrollment," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(4), pages 612-626, June.
    18. Heckman, James J & Honore, Bo E, 1990. "The Empirical Content of the Roy Model," Econometrica, Econometric Society, vol. 58(5), pages 1121-1149, September.
    19. Joshua D. Angrist & Jörn-Steffen Pischke, 2009. "Mostly Harmless Econometrics: An Empiricist's Companion," Economics Books, Princeton University Press, edition 1, number 8769.
    20. Djebbari, Habiba & Smith, Jeffrey, 2008. "Heterogeneous impacts in PROGRESA," Journal of Econometrics, Elsevier, vol. 145(1-2), pages 64-80, July.
    21. Alexandre Belloni & Victor Chernozhukov & Christian Hansen, 2014. "Inference on Treatment Effects after Selection among High-Dimensional Controlsâ€," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 81(2), pages 608-650.
    22. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    23. Fan, Yanqin & Park, Sang Soo, 2010. "Sharp Bounds On The Distribution Of Treatment Effects And Their Statistical Inference," Econometric Theory, Cambridge University Press, vol. 26(3), pages 931-951, June.
    24. Mark M. Pitt & Mark R. Rosenzweig & Mohammad Nazmul Hassan, 2012. "Human Capital Investment and the Gender Division of Labor in a Brawn-Based Economy," American Economic Review, American Economic Association, vol. 102(7), pages 3531-3560, December.
    25. Joel L. Horowitz & N. E. Savin, 2001. "Binary Response Models: Logits, Probits and Semiparametrics," Journal of Economic Perspectives, American Economic Association, vol. 15(4), pages 43-56, Fall.
    26. Abbring, Jaap H. & Heckman, James J., 2007. "Econometric Evaluation of Social Programs, Part III: Distributional Treatment Effects, Dynamic Treatment Effects, Dynamic Discrete Choice, and General Equilibrium Policy Evaluation," Handbook of Econometrics, in: J.J. Heckman & E.E. Leamer (ed.), Handbook of Econometrics, edition 1, volume 6, chapter 72, Elsevier.
    27. Colm O'Muircheartaigh & Larry V. Hedges, 2014. "Generalizing from unrepresentative experiments: a stratified propensity score approach," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 63(2), pages 195-210, February.
    28. Blimpo,Moussa P. & Evans,David & Lahire,Nathalie, 2015. "Parental human capital and effective school management : evidence from The Gambia," Policy Research Working Paper Series 7238, The World Bank.
    29. Jorg Stoye, 2009. "More on Confidence Intervals for Partially Identified Parameters," Econometrica, Econometric Society, vol. 77(4), pages 1299-1315, July.
    30. Donald W. K. Andrews & Moshe Buchinsky, 2000. "A Three-Step Method for Choosing the Number of Bootstrap Repetitions," Econometrica, Econometric Society, vol. 68(1), pages 23-52, January.
    31. Aakvik, Arild & Heckman, James J. & Vytlacil, Edward J., 2005. "Estimating treatment effects for discrete outcomes when responses to treatment vary: an application to Norwegian vocational rehabilitation programs," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 15-51.
    32. Fafchamps, Marcel & Labonne, Julien, 2017. "Using Split Samples to Improve Inference on Causal Effects," Political Analysis, Cambridge University Press, vol. 25(4), pages 465-482, October.
    33. Dan A. Black & Jeffrey A. Smith & Mark C. Berger & Brett J. Noel, 2003. "Is the Threat of Reemployment Services More Effective Than the Services Themselves? Evidence from Random Assignment in the UI System," American Economic Review, American Economic Association, vol. 93(4), pages 1313-1327, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Shukla, Sumedha & Arora, Gaurav, 2022. "A rank similarity test for quantile treatment effects in conjunction with propensity score matching: An application to crop yield impacts of agricultural credit," 2022 Annual Meeting, July 31-August 2, Anaheim, California 322569, Agricultural and Applied Economics Association.
    2. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    3. Guadalupe Bedoya & Aidan Coville & Johannes Haushofer & Mohammad Isaqzadeh & Jeremy P. Shapiro, 2019. "No Household Left Behind: Afghanistan Targeting the Ultra Poor Impact Evaluation," NBER Working Papers 25981, National Bureau of Economic Research, Inc.
    4. De Frahan, B. Henry & Bali, J. & Tuyishime, C., 2018. "Income and welfare effects of input subsidies across representative agricultural households of rural Rwanda," 2018 Conference, July 28-August 2, 2018, Vancouver, British Columbia 277469, International Association of Agricultural Economists.
    5. Francesca Caselli & Mr. Philippe Wingender, 2018. "Bunching at 3 Percent: The Maastricht Fiscal Criterion and Government Deficits," IMF Working Papers 2018/182, International Monetary Fund.
    6. Robson, Matthew & Vollmer, Frank & Doğan, Basak Berçin & Grede, Nils, 2024. "Distributional impacts of cash transfers on the multidimensional poverty of refugees: The Emergency Social Safety Net in Turkey," World Development, Elsevier, vol. 179(C).
    7. Pushkar Maitra & Sandip Mitra & Dilip Mookherjee & Sujata Visaria, 2021. "Evaluating the Distributive Effects of a Development Intervention," HKUST CEP Working Papers Series 202106, HKUST Center for Economic Policy.
    8. Maitra, Pushkar & Mitra, Sandip & Mookherjee, Dilip & Visaria, Sujata, 2022. "Evaluating the distributive effects of a micro-credit intervention," Journal of Development Economics, Elsevier, vol. 158(C).
    9. Caselli, Francesca & Wingender, Philippe, 2021. "Heterogeneous effects of fiscal rules: The Maastricht fiscal criterion and the counterfactual distribution of government deficits✰," European Economic Review, Elsevier, vol. 136(C).
    10. Jonathan M.V. Davis, 2017. "The Short and Long Run Impacts of Centralized Clearinghouses: Evidence from Matching Teach For America Teachers to Schools," 2017 Papers pda791, Job Market Papers.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Callaway, Brantly, 2021. "Bounds on distributional treatment effect parameters using panel data with an application on job displacement," Journal of Econometrics, Elsevier, vol. 222(2), pages 861-881.
    2. Yanqin Fan & Sang Soo Park, 2009. "Partial identification of the distribution of treatment effects and its confidence sets," Advances in Econometrics, in: Nonparametric Econometric Methods, pages 3-70, Emerald Group Publishing Limited.
    3. Słoczyński, Tymon, 2012. "New Evidence on Linear Regression and Treatment Effect Heterogeneity," MPRA Paper 39524, University Library of Munich, Germany.
    4. Jeffrey Smith & Arthur Sweetman, 2016. "Viewpoint: Estimating the causal effects of policies and programs," Canadian Journal of Economics, Canadian Economics Association, vol. 49(3), pages 871-905, August.
    5. Steven F. Lehrer & R. Vincent Pohl & Kyungchul Song, 2022. "Multiple Testing and the Distributional Effects of Accountability Incentives in Education," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 40(4), pages 1552-1568, October.
    6. John Mullahy, 2017. "Individual Results May Vary: Elementary Analytics of Inequality-Probability Bounds, with Applications to Health-Outcome Treatment Effects," NBER Working Papers 23603, National Bureau of Economic Research, Inc.
    7. Mullahy, John, 2018. "Individual results may vary: Inequality-probability bounds for some health-outcome treatment effects," Journal of Health Economics, Elsevier, vol. 61(C), pages 151-162.
    8. Sung Jae Jun & Yoonseok Lee & Youngki Shin, 2016. "Treatment Effects With Unobserved Heterogeneity: A Set Identification Approach," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 34(2), pages 302-311, April.
    9. Sloczynski, Tymon, 2018. "A General Weighted Average Representation of the Ordinary and Two-Stage Least Squares Estimands," IZA Discussion Papers 11866, Institute of Labor Economics (IZA).
    10. Pereda-Fernández, Santiago, 2023. "Identification and estimation of triangular models with a binary treatment," Journal of Econometrics, Elsevier, vol. 234(2), pages 585-623.
    11. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    12. Tymon S{l}oczy'nski, 2018. "Interpreting OLS Estimands When Treatment Effects Are Heterogeneous: Smaller Groups Get Larger Weights," Papers 1810.01576, arXiv.org, revised May 2020.
    13. Hernæs, Øystein M., 2020. "Distributional effects of welfare reform for young adults: An unconditional quantile regression approach," Labour Economics, Elsevier, vol. 65(C).
    14. Sokbae Lee & Yoon-Jae Whang, 2009. "Nonparametric Tests of Conditional Treatment Effects," Cowles Foundation Discussion Papers 1740, Cowles Foundation for Research in Economics, Yale University.
    15. Ismaël Mourifié & Marc Henry & Romuald Méango, 2020. "Sharp Bounds and Testability of a Roy Model of STEM Major Choices," Journal of Political Economy, University of Chicago Press, vol. 128(8), pages 3220-3283.
    16. Jaap Abbring & James Heckman, 2008. "Dynamic policy analysis," CeMMAP working papers CWP05/08, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    17. Sergio Firpo & Cristine Pinto, 2016. "Identification and Estimation of Distributional Impacts of Interventions Using Changes in Inequality Measures," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 31(3), pages 457-486, April.
    18. Fan, Yanqin & Park, Sang Soo, 2012. "Confidence intervals for the quantile of treatment effects in randomized experiments," Journal of Econometrics, Elsevier, vol. 167(2), pages 330-344.
    19. Steven F. Lehrer & R. Vincent Pohl & Kyungchul Song, 2016. "Targeting Policies: Multiple Testing and Distributional Treatment Effects," NBER Working Papers 22950, National Bureau of Economic Research, Inc.
    20. Brigham R. Frandsen & Lars J. Lefgren, 2021. "Partial identification of the distribution of treatment effects with an application to the Knowledge is Power Program (KIPP)," Quantitative Economics, Econometric Society, vol. 12(1), pages 143-171, January.

    More about this item

    Keywords

    policy evaluation; distributional impact analysis; heterogeneous treatment effects; impacts on outcome distributions; distribution of treatment effects; random control trials;
    All these keywords.

    JEL classification:

    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • C54 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Quantitative Policy Modeling
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments
    • D39 - Microeconomics - - Distribution - - - Other

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp11863. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Holger Hinte (email available below). General contact details of provider: https://edirc.repec.org/data/izaaade.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.