IDEAS home Printed from https://ideas.repec.org/p/osf/osfxxx/nwp8k.html
   My bibliography  Save this paper

Lost in the Design Space? Construct Validity in the Microfinance Literature

Author

Listed:
  • Masselus, Lise
  • Petrik, Christina
  • Ankel-Peters, Jörg

Abstract

While results from individual Randomized Controlled Trials (RCTs) often do not hold beyond their setting, the accumulation of many RCTs can be used to guide policy. But how many studies are required to confidently generalize? Our paper examines construct validity, an often neglected yet important element affecting generalizability. Construct validity deals with how the operationalization of a treatment corresponds to the broader construct it intends to speak to. The universe of potential operationalizations is referred to as the design space. As an empirical example, we review 45 microfinance RCTs to estimate the size of the design space and to underpin that variations in the treatment operationalization matter for the observed effects. We also show that most papers nevertheless generalize from the operationalized treatment to a broad construct, mostly without acknowledging underlying assumptions, and many lack a transparent reporting of construct validity-relevant features.

Suggested Citation

  • Masselus, Lise & Petrik, Christina & Ankel-Peters, Jörg, 2024. "Lost in the Design Space? Construct Validity in the Microfinance Literature," OSF Preprints nwp8k, Center for Open Science.
    • Handle: RePEc:osf:osfxxx:nwp8k
    DOI: 10.31219/osf.io/nwp8k
    as

    Download full text from publisher

    File URL: https://osf.io/download/66b9f1f36240b8f9585d9baa/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/nwp8k?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    2. A Stefano Caria & Grant Gordon & Maximilian Kasy & Simon Quinn & Soha Osman Shami & Alexander Teytelboym, 2024. "An Adaptive Targeted Field Experiment: Job Search Assistance for Refugees in Jordan," Journal of the European Economic Association, European Economic Association, vol. 22(2), pages 781-836.
    3. Rajeev Dehejia & Cristian Pop-Eleches & Cyrus Samii, 2021. "From Local to Global: External Validity in a Fertility Natural Experiment," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 39(1), pages 217-243, January.
    4. Kim, J.C. & Watts, C.H. & Hargreaves, J.R. & Ndhlovu, L.X. & Phetla, G. & Morison, L.A. & Busza, J. & Porter, J.D.H. & Pronyk, P., 2007. "Understanding the impact of a microfinance-based intervention on women's empowerment and the reduction of intimate partner violence in South Africa," American Journal of Public Health, American Public Health Association, vol. 97(10), pages 1794-1802.
    5. Maitra, Pushkar & Mitra, Sandip & Mookherjee, Dilip & Motta, Alberto & Visaria, Sujata, 2017. "Financing smallholder agriculture: An experiment with agent-intermediated microloans in India," Journal of Development Economics, Elsevier, vol. 127(C), pages 306-337.
    6. Alessandro Tarozzi & Jaikishan Desai & Kristin Johnson, 2015. "The Impacts of Microcredit: Evidence from Ethiopia," American Economic Journal: Applied Economics, American Economic Association, vol. 7(1), pages 54-89, January.
    7. Abhijit Banerjee & Rukmini Banerji & James Berry & Esther Duflo & Harini Kannan & Shobhini Mukerji & Marc Shotland & Michael Walton, 2017. "From Proof of Concept to Scalable Policies: Challenges and Solutions, with an Application," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 73-102, Fall.
    8. Marshall Burke & Lauren Falcao Bergquist & Edward Miguel, 2019. "Sell Low and Buy High: Arbitrage and Local Price Effects in Kenyan Markets," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 134(2), pages 785-842.
    9. Jason T. Kerwin & Rebecca L. Thornton, 2021. "Making the Grade: The Sensitivity of Education Program Effectiveness to Input Choices and Outcome Measures," The Review of Economics and Statistics, MIT Press, vol. 103(2), pages 251-264, May.
    10. Benjamin Feigenberg & Erica Field & Rohini Pande, 2013. "The Economic Returns to Social Interaction: Experimental Evidence from Microfinance," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 80(4), pages 1459-1483.
    11. Dean Karlan & Jonathan Zinman, 2010. "Expanding Credit Access: Using Randomized Supply Decisions to Estimate the Impacts," The Review of Financial Studies, Society for Financial Studies, vol. 23(1), pages 433-464, January.
    12. Abhijit Banerjee & Esther Duflo & Richard Hornbeck, 2018. "How Much do Existing Borrowers Value Microfinance? Evidence from an Experiment on Bundling Microcredit and Insurance," Economica, London School of Economics and Political Science, vol. 85(340), pages 671-700, October.
    13. Hunt Allcott, 2015. "Site Selection Bias in Program Evaluation," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 130(3), pages 1117-1165.
    14. Gharad T. Bryan & Dean Karlan & Adam Osman, 2021. "Big Loans to Small Businesses: Predicting Winners and Losers in an Entrepreneurial Lending Experiment," NBER Working Papers 29311, National Bureau of Economic Research, Inc.
    15. Britta Augsburg & Ralph De Haas & Heike Harmgart & Costas Meghir, 2015. "The Impacts of Microcredit: Evidence from Bosnia and Herzegovina," American Economic Journal: Applied Economics, American Economic Association, vol. 7(1), pages 183-203, January.
    16. Erica Field & Rohini Pande & John Papp & Y Jeanette Park, 2012. "Repayment Flexibility Can Reduce Financial Stress: A Randomized Control Trial with Microfinance Clients in India," PLOS ONE, Public Library of Science, vol. 7(9), pages 1-7, September.
    17. Nusrat Abedin Jimi & Plamen V. Nikolov & Mohammad Abdul Malek & Subal Kumbhakar, 2019. "The effects of access to credit on productivity: separating technological changes from changes in technical efficiency," Journal of Productivity Analysis, Springer, vol. 52(1), pages 37-55, December.
    18. Manuela Angelucci & Dean Karlan & Jonathan Zinman, 2015. "Microcredit Impacts: Evidence from a Randomized Microcredit Program Placement Experiment by Compartamos Banco," American Economic Journal: Applied Economics, American Economic Association, vol. 7(1), pages 151-182, January.
    19. Pronyk, Paul M. & Harpham, Trudy & Busza, Joanna & Phetla, Godfrey & Morison, Linda A. & Hargreaves, James R. & Kim, Julia C. & Watts, Charlotte H. & Porter, John D., 2008. "Can social capital be intentionally generated? A randomized trial from rural South Africa," Social Science & Medicine, Elsevier, vol. 67(10), pages 1559-1570, November.
    20. Joshua D. Angrist & Jörn-Steffen Pischke, 2009. "Mostly Harmless Econometrics: An Empiricist's Companion," Economics Books, Princeton University Press, edition 1, number 8769.
    21. Bruno Crépon & Florencia Devoto & Esther Duflo & William Parienté, 2015. "Estimating the Impact of Microcredit on Those Who Take It Up: Evidence from a Randomized Experiment in Morocco," American Economic Journal: Applied Economics, American Economic Association, vol. 7(1), pages 123-150, January.
    22. Faisal Bari & Kashif Malik & Muhammad Meki & Simon Quinn, 2024. "Asset-Based Microfinance for Microenterprises: Evidence from Pakistan," American Economic Review, American Economic Association, vol. 114(2), pages 534-574, February.
    23. Cameron, Lisa & Olivia, Susan & Shah, Manisha, 2019. "Scaling up sanitation: Evidence from an RCT in Indonesia," Journal of Development Economics, Elsevier, vol. 138(C), pages 1-16.
    24. Esther Duflo, 2020. "Field Experiments and the Practice of Policy," American Economic Review, American Economic Association, vol. 110(7), pages 1952-1973, July.
    25. Davis, C. Austin & Mobarak, Ahmed Mushfiq, 2020. "The challenges of scaling effective interventions: A path forward for research and policy," World Development, Elsevier, vol. 127(C).
    26. Channa, Hira & Ricker-Gilbert, Jacob & Feleke, Shiferaw & Abdoulaye, Tahirou, 2022. "Overcoming smallholder farmers’ post-harvest constraints through harvest loans and storage technology: Insights from a randomized controlled trial in Tanzania," Journal of Development Economics, Elsevier, vol. 157(C).
    27. Giné, Xavier & Karlan, Dean S., 2014. "Group versus individual liability: Short and long term evidence from Philippine microcredit lending groups," Journal of Development Economics, Elsevier, vol. 107(C), pages 65-83.
    28. Erica Field & Rohini Pande & John Papp & Natalia Rigol, 2013. "Does the Classic Microfinance Model Discourage Entrepreneurship among the Poor? Experimental Evidence from India," American Economic Review, American Economic Association, vol. 103(6), pages 2196-2226, October.
    29. Rachael Meager, 2019. "Understanding the Average Impact of Microcredit Expansions: A Bayesian Hierarchical Analysis of Seven Randomized Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 11(1), pages 57-91, January.
    30. Orazio Attanasio & Britta Augsburg & Ralph De Haas & Emla Fitzsimons & Heike Harmgart, 2015. "The Impacts of Microfinance: Evidence from Joint-Liability Lending in Mongolia," American Economic Journal: Applied Economics, American Economic Association, vol. 7(1), pages 90-122, January.
    31. Michael Gechter, 2024. "Generalizing the Results from Social Experiments: Theory and Evidence from India," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 42(2), pages 801-811, April.
    32. Nadel, Sara & Pritchett, Lant, 2016. "Searching for the Devil in the Details: Learning about Development Program Design," Working Paper Series rwp16-041, Harvard University, John F. Kennedy School of Government.
    33. Arielle Bernhardt & Erica Field & Rohini Pande & Natalia Rigol, 2019. "Household Matters: Revisiting the Returns to Capital among Female Microentrepreneurs," American Economic Review: Insights, American Economic Association, vol. 1(2), pages 141-160, September.
    34. Lars Ivar Oppedal Berge & Kjetil Bjorvatn & Kartika Sari Juniwaty & Bertil Tungodden, 2012. "Business Training in Tanzania: From Research-driven Experiment to Local Implementation-super- †," Journal of African Economies, Centre for the Study of African Economies, vol. 21(5), pages -827, November.
    35. Eva Vivalt, 2020. "How Much Can We Generalize From Impact Evaluations?," Journal of the European Economic Association, European Economic Association, vol. 18(6), pages 3045-3089.
    36. Bold, Tessa & Kimenyi, Mwangi & Mwabu, Germano & Ng’ang’a, Alice & Sandefur, Justin, 2018. "Experimental evidence on scaling up education reforms in Kenya," Journal of Public Economics, Elsevier, vol. 168(C), pages 1-20.
    37. Benjamin Feigenberg & Erica Field & Rohini Pande & Natalia Rigol & Shayak Sarkar, 2014. "Do Group Dynamics Influence Social Capital Gains Among Microfinance Clients? Evidence From A Randomized Experiment In Urban India," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 33(4), pages 932-949, September.
    38. Reidpath, Daniel D. & Allotey, Pascale & Barker, S. Fiona & Clasen, Thomas & French, Matthew & Leder, Karin & Ramirez-Lovering, Diego & Rhule, Emma L.M. & Siri, José, 2022. "Implementing “from here to there”: A case study of conceptual and practical challenges in implementation science," Social Science & Medicine, Elsevier, vol. 301(C).
    39. Duflo, Esther, 2017. "The Economist as Plumber," CEPR Discussion Papers 11881, C.E.P.R. Discussion Papers.
    40. Popova,Anna & Evans,David & Arancibia,Violeta, 2016. "Training teachers on the job : what works and how to measure it," Policy Research Working Paper Series 7834, The World Bank.
    41. Aragón, Fernando M. & Karaivanov, Alexander & Krishnaswamy, Karuna, 2020. "Credit lines in microcredit: Short-term evidence from a randomized controlled trial in India," Journal of Development Economics, Elsevier, vol. 146(C).
    42. Abhijit Banerjee & Esther Duflo & Rachel Glennerster & Cynthia Kinnan, 2015. "The Miracle of Microfinance? Evidence from a Randomized Evaluation," American Economic Journal: Applied Economics, American Economic Association, vol. 7(1), pages 22-53, January.
    43. Esther Duflo, 2017. "Richard T. Ely Lecture: The Economist as Plumber," American Economic Review, American Economic Association, vol. 107(5), pages 1-26, May.
    44. Kaushik Basu, 2014. "Randomisation, Causality and the Role of Reasoned Intuition," Oxford Development Studies, Taylor & Francis Journals, vol. 42(4), pages 455-472, December.
    45. Abhijit Banerjee & Dean Karlan & Jonathan Zinman, 2015. "Six Randomized Evaluations of Microcredit: Introduction and Further Steps," American Economic Journal: Applied Economics, American Economic Association, vol. 7(1), pages 1-21, January.
    46. Basu, Karna & Wong, Maisy, 2015. "Evaluating seasonal food storage and credit programs in east Indonesia," Journal of Development Economics, Elsevier, vol. 115(C), pages 200-216.
    47. Günther Fink & B. Kelsey Jack & Felix Masiye, 2020. "Seasonal Liquidity, Rural Labor Markets, and Agricultural Production," American Economic Review, American Economic Association, vol. 110(11), pages 3351-3392, November.
    48. Jaikishan Desai & Alessandro Tarozzi, 2011. "Microcredit, Family Planning Programs, and Contraceptive Behavior: Evidence From a Field Experiment in Ethiopia," Demography, Springer;Population Association of America (PAA), vol. 48(2), pages 749-782, May.
    49. Shareen A Iqbal & Joshua D Wallach & Muin J Khoury & Sheri D Schully & John P A Ioannidis, 2016. "Reproducible Research Practices and Transparency across the Biomedical Literature," PLOS Biology, Public Library of Science, vol. 14(1), pages 1-13, January.
    50. Esterling, Kevin M. & Brady, David & Schwitzgebel, Eric, 2023. "The Necessity of Construct and External Validity for Generalized Causal Claims," I4R Discussion Paper Series 18, The Institute for Replication (I4R).
    51. Lant Pritchett & Justin Sandefur, 2015. "Learning from Experiments When Context Matters," American Economic Review, American Economic Association, vol. 105(5), pages 471-475, May.
    52. C. S. C. Sekhar & Namrata Thapa, 2024. "Experimental approach to development economics: a review of issues and options," Journal of Economic Methodology, Taylor & Francis Journals, vol. 31(2), pages 63-77, April.
    53. Sara Nadel and Lant Pritchett, 2016. "Searching for the Devil in the Details: Learning about Development Program Design," Working Papers 434, Center for Global Development.
    54. Marup Hossain & Mohammad Abdul Malek & Md Amzad Hossain & Md Hasib Reza & Md Shakil Ahmed, 2019. "Agricultural Microcredit for Tenant Farmers: Evidence from a Field Experiment in Bangladesh," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 101(3), pages 692-709.
    55. Mo, Di & Bai, Yu & Shi, Yaojiang & Abbey, Cody & Zhang, Linxiu & Rozelle, Scott & Loyalka, Prashant, 2020. "Institutions, implementation, and program effectiveness: Evidence from a randomized evaluation of computer-assisted learning in rural China," Journal of Development Economics, Elsevier, vol. 146(C).
    56. Morduch, Jonathan, 2020. "Why RCTs failed to answer the biggest questions about microcredit impact," World Development, Elsevier, vol. 127(C).
    57. Esther Duflo, 2017. "The Economist as Plumber," NBER Working Papers 23213, National Bureau of Economic Research, Inc.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Rose, Julian & Neubauer, Florian & Ankel-Peters, Jörg, 2024. "Long-Term Effects of the Targeting the Ultra-Poor Program - A Reproducibility and Replicability Assessment of Banerjee et al. (2021)," I4R Discussion Paper Series 142, The Institute for Replication (I4R).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Oriana Bandiera & Robin Burgess & Erika Deserranno & Ricardo Morel & Imran Rasul & Munshi Sulaiman & Jack Thiemel, 2022. "Microfinance and Diversification," Economica, London School of Economics and Political Science, vol. 89(S1), pages 239-275, June.
    2. Jonathan Fu & Annette Krauss, 2024. "Preparing fertile ground: how does the quality of business environments affect MSE growth?," Small Business Economics, Springer, vol. 63(1), pages 51-103, June.
    3. Meager, Rachael, 2022. "Aggregating distributional treatment effects: a Bayesian hierarchical analysis of the microcredit literature," LSE Research Online Documents on Economics 115559, London School of Economics and Political Science, LSE Library.
    4. Ankel-Peters, Jörg & Schmidt, Christoph M., 2023. "Rural electrification, the credibility revolution, and the limits of evidence-based policy," Ruhr Economic Papers 1051, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    5. João Paulo Coelho Ribeiro & Fábio Duarte & Ana Paula Matias Gama, 2022. "Does microfinance foster the development of its clients? A bibliometric analysis and systematic literature review," Financial Innovation, Springer;Southwestern University of Finance and Economics, vol. 8(1), pages 1-35, December.
    6. Meager, Rachael & Sturdy, Jennifer, 2017. "Aggregating Distributional Treatment Effects: A Bayesian Hierarchical Analysis of the Microcredit Literature," MetaArXiv 7tkvm, Center for Open Science.
    7. Emily Breza & Cynthia Kinnan, 2021. "Measuring the Equilibrium Impacts of Credit: Evidence from the Indian Microfinance Crisis," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 136(3), pages 1447-1497.
    8. Daniel Bjorkegren & Joshua Blumenstock & Omowunmi Folajimi-Senjobi & Jacqueline Mauro & Suraj R. Nair, 2022. "Instant Loans Can Lift Subjective Well-Being: A Randomized Evaluation of Digital Credit in Nigeria," Papers 2202.13540, arXiv.org.
    9. Dahal, Mahesh & Fiala, Nathan, 2020. "What do we know about the impact of microfinance? The problems of statistical power and precision," World Development, Elsevier, vol. 128(C).
    10. Ernest Liu & Benjamin N. Roth, 2020. "Contractual Restrictions and Debt Traps," Working Papers 2020-30, Princeton University. Economics Department..
    11. Ahlin, Christian & Gulesci, Selim & Madestam, Andreas & Stryjan, Miri, 2020. "Loan contract structure and adverse selection: Survey evidence from Uganda," Journal of Economic Behavior & Organization, Elsevier, vol. 172(C), pages 180-195.
    12. Nakano, Yuko & Magezi, Eustadius F., 2020. "The impact of microcredit on agricultural technology adoption and productivity: Evidence from randomized control trial in Tanzania," World Development, Elsevier, vol. 133(C).
    13. Chowdhury, Shyamal & Smits, Joeri & Sun, Qigang, 2020. "Contract structure, time preference, and technology adoption," GLO Discussion Paper Series 633, Global Labor Organization (GLO).
    14. Rachael Meager, 2015. "Understanding the Impact of Microcredit Expansions: A Bayesian Hierarchical Analysis of 7 Randomised Experiments," Papers 1506.06669, arXiv.org, revised Jul 2016.
    15. Chowdhury, Shyamal & Smits, Joeri & Sun, Qigang, 2020. "Contract Structure, Time Preference, and Technology Adoption," IZA Discussion Papers 13590, Institute of Labor Economics (IZA).
    16. Abhijit Banerjee & Emily Breza & Esther Duflo & Cynthia Kinnan, 2019. "Can Microfinance Unlock a Poverty Trap for Some Entrepreneurs?," NBER Working Papers 26346, National Bureau of Economic Research, Inc.
    17. Czura, Kristina, 2015. "Do flexible repayment schedules improve the impact of microcredit?," Discussion Papers in Economics 26608, University of Munich, Department of Economics.
    18. Meager, Rachael, 2019. "Understanding the average impact of microcredit expansions: a Bayesian hierarchical analysis of seven randomized experiments," LSE Research Online Documents on Economics 88190, London School of Economics and Political Science, LSE Library.
    19. Baulia, Susmita, 2019. "Take-up of joint and individual liability loans: An analysis with laboratory experiment," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 82(C).
    20. Bernardus Van Doornik & Armando Gomes & David Schoenherr & Janis Skrastins, 2024. "Financial Access and Labor Market Outcomes: Evidence from Credit Lotteries," American Economic Review, American Economic Association, vol. 114(6), pages 1854-1881, June.

    More about this item

    JEL classification:

    • A11 - General Economics and Teaching - - General Economics - - - Role of Economics; Role of Economists
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments
    • D04 - Microeconomics - - General - - - Microeconomic Policy: Formulation; Implementation; Evaluation
    • O12 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development - - - Microeconomic Analyses of Economic Development
    • O16 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development - - - Financial Markets; Saving and Capital Investment; Corporate Finance and Governance

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:osfxxx:nwp8k. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.