Author Archive

Jon Krosnick and his students fulfill a very important role in survey research that more academics should be concerned about – establishing scientifically what the best way to conduct research is, but also what tools are effective in improving the less scientific methods that most of us are forced to rely on instead.  In the latest volume of POQ, David Yaeger, Dr. Krosnick and others argue in “Comparing the Accuracy of RDD Telephone Surveys and Internet Surveys…” that despite declining response rates, telephone surveys remained reliably more accurate on a number of non-demographic benchmarks than were seven non-probabilistic online samples.

We applaud an approach driven by behaviors, but as the authors admit, the selection of benchmarks is a weak one, reflecting variables that are by and large likely to be independent of the selection biases caused by an RDD design, particularly after post-stratification weighting (the paper uses the non-demographic benchmarks of drivers’ license and passport ownership, cigarette smoking and alcohol consumption).   This is through no fault of their own, but rather a very high standard for reliable benchmarks, drawing only what could be found via large-sample government population surveys.   The need for reliable and relevant benchmarks is becoming overwhelming as online research seeks its coming of age.

One issue that bothers us, however, is the insistence on treating telephone and online sources as being mutually exclusive modes.  It is our belief that the true future of representative sample lies in using references that consist of some blend of the two, with each compensating for the others’ selection biases.

A secondary point the article draws attention to, and that we feel is very important, is that even rigorous post-stratification weighting on non-probabilistic samples will not necessarily ameliorate non-representativeness.  In fact, the results suggest that weighting only inconsistently shows even a positive effect on the accuracy against other benchmarks.  This confirms what we have claimed for years.  As many industry-leaders now realize, controlling demographics is simply not enough to achieve a behaviorally representative sample frame.