Posts Tagged ‘Grand Mean’

The very nature of survey research requires that online sample sources have robust quality standards. It is impossible to interpret market research results with confidence without a thorough understanding of the sample sources from which respondents are drawn. This is particularly important in the case of both tracking studies and multinational studies, where both validity and consistency are critical.

Since late 2007, our firm has been compiling and analyzing data for just such an assessment. The Grand Mean Project is an extensive study of global online panels. For the study, at least 400 respondents are collected from each participating panel. A standard online questionnaire, translated into local languages, is utilized, including a focus on buying behavior and a broad spectrum of other subjects. This program has collected data on more than 150 panels across 35 countries. In eight countries, at least five companies have participated, allowing the creation of a grand mean. In 17 countries, multiple panel data has been collected and in 10 countries one panel has participated. To our knowledge, it is the largest and most comprehensive online sample assessment to date.

You can read the full article published in Quirks / November 2009:
Finding the strength in numbers by Steve Gittelman and Elaine Trimarchi

Many of us remember the good old days…when a decent telephone study was six figures; when Americans were more then willing to suffer through our incessant questionnaires; when there were thousands of interviewers and RDD samples were all the rage.  If phone today is not the same as phone of yesterday how can we expect online to be the same as phone?  The answer is…we can’t.  The online world is one that is based upon a non-probabilistic framework.  There is no census like the good old days of phone to hang our hat on.  It is the body of knowledge that becomes the new census.  The creation of the Grand Mean Project™ is the new body of knowledge needed to move Online Research forward. “The more we know, the better off the online data collection universe will be, the healthier our profession will be”…and isn’t that what we all want.

You can read the full article published in MRA’s Alert! Magazine September 2009: The Virtues of Consistent Bias: Online Research Must Move On, by Steve Gittelman and Elaine Trimarchi

When Charles Darwin hauled himself onto the volcanic shores of the Galapagos Islands he took samples from as many islands as he could reach. For the most part, these isolated little islands were different from one another. Even birds that could fly from one island to the next were different. He didn’t have a census to draw conclusions: He was the census!

Darwin took samples and wrote a pretty good book. The samples were not grounded in probability theory nor could he generalize from island to island. Vive la difference! It was the differences in the samples that gave him clues. Each island was an ecosystem unto itself and the differences that species on the islands had to endure shaped them into the specialists that they would become.

Charles was a smart guy. He knew that the differences in the islands and the changes in the animals that lived there had taken time. Lots of time; he called it evolution!

Our use of online panel data has much to learn from island biogeography. Think of each online panel as an island. They have similarities but are drawn from different sources. We should not expect them to be identical; we should expect them to be different. Our research regarding the US data sources has shown them to be quite inconsistent (Gittelman and Trimarchi, Feb 2009 CASRO) and the ARF supports this point. The panels are not interchangeable. The online panels are drawn from different sources, are subject to differing management practices and for a host of reasons yield different results.

Darwin spent quite a bit of time explaining how different birds adapted to eating nuts of different sizes and textures: their beaks changed in time. We explain to our clients how buying behavior is affected by events; we must separate event related change from underlying sample shifts.

We need to know the differences between panels at any given moment so that we can understand how the panel we use changes through time and events; we need to know its consistency.

In the pursuit of analyzing consistency Sample Source Auditors, a division of Mktg, Inc., has moved onward from its initial study of the American markets and has expanded its research to include 150 panels in 35 nations. For each panel a standard instrument is used in a tracking study that includes a diversity of measures, but mostly focuses on buying behavior segmentations. By conducting repeat waves of this consistency study, a local Grand Mean is calculated for each market. In addition, using standard quality control techniques an analysis of the consistency of each panel is conducted.

CONSISTENCY TEST SUMMARY

How consistent is the performance of the data source?

Respondent data quality affects survey results. Shifts in quality create inconsistencies. Here failures to follow instructions and logic errors in responses are tracked from wave to wave in a consistency study.  Chart 2 shows the relative performance of the series of datasets for one source along with references. The lines on the bars represent the error bounds.

Buying behavior, our most important measure.

Changes in the panel population are measured by the distribution of structural segments including buyer behavior segments.  This segmentation scheme captures the overall effective changes in over 30 variables and reflects the population.  Variation in this distribution across countries is fairly large but only marginal within countries.

How different are these segment distributions?

Distance measures and statistical tests (chi-square) are used to test the difference between distributions.  Chart 4 shows (blue line) the results of the distance tests of various datasets collected over time compared to the references.  Note that the most recent dataset shows larger differences than the previous sets.