Response Rates - An Overview
Response Rates - An Overview
Summary:Calculating response rates - the number of eligible sample units thatcooperate in a survey -- has historically been central to surveyresearch in the United States because of the assumption that thelarger the proportion of participating sample units, the more accuratethe survey estimates. Formulas for calculating rates are nowstandardized, but the relationship between response rates and surveyquality has become much less clear.
Measuring Response Rates
Until recently, there were almost as many ways to calculate responserates as there were researchers. Response rates, cooperation rates, andcompletion rates were often treated as interchangeable in theliterature. In the early 1980s, the Council of American SurveyResearch Organizations (CASRO) made the first attempt to standardizethe definition of a response rate, an effort completed in the late1990s by AAPOR with the publication of Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys.
The 2008 edition of Standard Definitionsclearly distinguishes between the response rate and the cooperationrate, covers household, telephone, mail, and Internet modes ofadministration, discusses the criteria for ineligibility, and specifiesmethods for calculating refusal and noncontact rates. As a result,response and nonresponse rates can now be successfully compared acrosssurveys of different topics and organizations. In addition, thesedefinitions and their widespread acceptance have resulted in a greaterwillingness of researchers to report low response rates.
Response Rates and Survey Quality
However, two factors have now undermined the role of the responserate as the primary arbiter of survey quality. Largely due toincreasing refusals, response rates across all modes of surveyadministration have declined, in some cases precipitously. As a result,organizations have had to put additional effort into administration,thus making all types of surveys more costly. At the same time, studiesthat have compared survey estimates to benchmark data from the U.S.Census or very large governmental sample surveys have also questionedthe positive association between response rates and quality.Furthermore, a growing emphasis on total survey error has causedmethodologists to examine surveys—even those with acceptably highresponse rates--for evidence of nonresponse bias.
Results that show the least bias have turned out, in some cases, tocome from surveys with less than optimal response rates. Experimentalcomparisons have also revealed few significant differences betweenestimates from surveys with low response rates and short field periodsand surveys with high response rates and long field periods. (Thedifficulty of determining bias by comparing survey estimates to outsidemeasurements, however, has led to ingenious strategies. One recentstudy developed an internal benchmark by using the 50/50 gender splitof heterosexual, married couples to gauge the accuracy of surveyestimates by gender among the respondents in six different surveys. )
There is currently no consensus about the factors that produce thedisjuncture between response rates and survey quality. But the evidencedoes suggest several rules of thumb for consumers of survey reports andfor researchers.
Researchers should always include in their survey reports the response rate, computed according to the appropriate AAPOR formula (see AAPOR Response Rate Calculator here - Excel) oranother similar formula fully described. Furthermore, several othermeasures of quality should become part of reports, especially when aresponse rate is low. On their side, consumers of survey results shouldtreat all response rates with skepticism, since these rates do notnecessarily differentiate reliably between accurate and inaccuratedata. Instead consumers should pay attention to other indicators ofquality that are included in reports and on websites, such asinsignificant levels of bias, low levels of missing data, andconformity with other research findings.
More Information
Back to top
###
Read MoreNonresponse Bias| Response Rates