Measuring the measurement error: A method to qualitatively validate survey data(Article)(Open Access)
- aColumbia University SIPA, 420 W 118th St., New York, NY, United States
- bGlobal Insights Initiative, The World Bank, 1818 H St NW, Washington, DC, United States
- cThe World Bank, 1818 H St NW, Washington, DC, United States
- dInternational Rescue Committee, Research Department, 122 East 42nd St., New York, NY, United States
- eUniversity of North Carolina at Chapel Hill, Clinical Psychology, United States
Abstract
Empirical social science relies heavily on self-reported data, but subjects may misreport behaviors, especially sensitive ones such as crime or drug abuse. If a treatment influences survey misreporting, it biases causal estimates. We develop a validation technique that uses intensive qualitative work to assess survey misreporting and pilot it in a field experiment where subjects were assigned to receive cash, therapy, both, or neither. According to survey responses, both treatments reduced crime and other sensitive behaviors. Local researchers spent several days with a random subsample of subjects after surveys, building trust and obtaining verbal confirmation of four sensitive behaviors and two expenditures. In this instance, validation showed survey underreporting of most sensitive behaviors was low and uncorrelated with treatment, while expenditures were under reported in the survey across all arms, but especially in the control group. We use these data to develop measurement error bounds on treatment effects estimated from surveys. © 2016 .
Indexed keywords
| GEOBASE Subject Index: | crimedrugerror analysismeasurement methodqualitative analysissurvey |
|---|---|
| Regional Index: | Liberia [West Africa] |
Funding details
| Funding sponsor | Funding number | Acronym |
|---|---|---|
| National Science Foundation See opportunities by NSF | SES-1317506 | NSF |
| Directorate for Social, Behavioral and Economic Sciences See opportunities by SBE | 1317506 | SBE |
| United States Agency for International Development See opportunities by USAID | AID-OAA-A-12-00066 | USAID |
| World Bank Group See opportunities by WBG | WBG | |
| Harvard University | ||
| Department for International Development, UK Government See opportunities by DFID | GA-C1-RA2-114 | DFID |
-
1
Acknowledgements: For comments we thank Neal Beck, Alex Coppock, Dan Corstange, Macartan Humphreys, Don Green, Cyrus Samii, Chris Udry, several anonymous referees, and participants at the NYU 2014 CESS conference. This study was funded by the National Science Foundation ( SES-1317506 ), the World Bank's Learning on Gender and Conflict in Africa (LOGiCA) trust fund, the World Bank's Italian Children and Youth (CHYAO) trust fund, the Department of International Development, UK (DFID, GA-C1-RA2-114 ) via the Institute for the Study of Labor (IZA), a Vanguard Charitable Trust, the American People through the United States Agency for International Development (USAID, AID-OAA-A-12-00066 ) DCHA/CMM office, and the Robert Wood Johnson Health and Society Scholars Program at Harvard University (Cohort 5). The contents of this study are the sole responsibility of authors and do not necessarily reflect the views of their employers or any of these funding agencies or governments. Finally, for research assistance we thank Foday Bayoh Jr., Natalie Carlson, Camelia Dureng, Mathilde Emeriau, Yuequan Guo, Rufus Kapwolo, James Kollie, Rebecca Littman, Richard Peck, Patryk Perkowski, Colombine Peze-Heidsieck, Joe St. Clair, Joseph Sango Jr., Helen Smith, Abel Welwean, Prince Williams, and John Zayzay through Innovations for Poverty Action (IPA).
- ISSN: 03043878
- Source Type: Journal
- Original language: English
- DOI: 10.1016/j.jdeveco.2016.01.005
- Document Type: Article
- Publisher: Elsevier B.V.
Blattman, C.; Columbia University SIPA, 420 W 118th St., New York, NY, United States;
© Copyright 2017 Elsevier B.V., All rights reserved.
Cited by 32 documents
(2022) Journal of Statistical Theory and Practice
(2022) Social Psychiatry and Psychiatric Epidemiology
(2022) Sustainability (Switzerland)