Archive for September, 2009

Many of us remember the good old days…when a decent telephone study was six figures; when Americans were more then willing to suffer through our incessant questionnaires; when there were thousands of interviewers and RDD samples were all the rage.  If phone today is not the same as phone of yesterday how can we expect online to be the same as phone?  The answer is…we can’t.  The online world is one that is based upon a non-probabilistic framework.  There is no census like the good old days of phone to hang our hat on.  It is the body of knowledge that becomes the new census.  The creation of the Grand Mean Project™ is the new body of knowledge needed to move Online Research forward. “The more we know, the better off the online data collection universe will be, the healthier our profession will be”…and isn’t that what we all want.

You can read the full article published in MRA’s Alert! Magazine September 2009: The Virtues of Consistent Bias: Online Research Must Move On, by Steve Gittelman and Elaine Trimarchi

At lunch we met Mr. Happy; a young man from Thailand who came all the way to Switzerland to talk about happiness.     His research was the investigation of the Gross Domestic Happiness (GDH) of Thailand.  He seemed so proud of his work and told us about his challenge in defining happiness. We were struck by the twinkle in this man’s eye and the sweetness in his personality as he explained how the GDH aims to measure the happiness of Thai citizens as a whole, not individual happiness.  I found myself hanging on his every word.  He also told us he was traveling with his wife and 5 year old son.  I could clearly see that this was one happy fellow and the key to his happiness was his work and family.  His talk would be right after lunch and it was one I was looking forward to.

Prior to the start of his presentation he was having difficulty finding his Power Point presentation. He searched and searched and finally stood up and with a smile said “it is not here”.    How many of us would become frustrated and thrown off our game if we were unable to find our presentation…one million eyes are watching you…But not Mr. Happy.  Someone came over to help him find his presentation and he cheerfully began.  In the middle of his discussion about the independent variables such as sense of community, workplace life etc…   he asked the man sitting in the front row if he wouldn’t mind taking a picture of him so he can show his friends back home.  You could see that he was a proud man who truly was very happy.  His smile went from ear to ear.

The awards dinner was held that evening.  Awards were presented for various topics and as usually is the case the most honorable award was left for last.  This award was recognition for the best paper.  The person announcing the awards stated how hard the selection for this award was but how one stood out from the rest.  We were excited and “happy” when they announced the award to our friend from Thailand… Noppadon Kannika!  To our great disappointment, he was not in attendance.  We were unable to share with him this great moment.  We assumed that he was with his wife and son enjoying Switzerland.   But we knew wherever he was, he was contributing to the Gross Domestic Happiness of his family and his country.  Through all the conferences I have attended and the miles of smiles I have seen I was touched by Noppadon Kannika and will remember him and remember to be Happy!

When Charles Darwin hauled himself onto the volcanic shores of the Galapagos Islands he took samples from as many islands as he could reach. For the most part, these isolated little islands were different from one another. Even birds that could fly from one island to the next were different. He didn’t have a census to draw conclusions: He was the census!

Darwin took samples and wrote a pretty good book. The samples were not grounded in probability theory nor could he generalize from island to island. Vive la difference! It was the differences in the samples that gave him clues. Each island was an ecosystem unto itself and the differences that species on the islands had to endure shaped them into the specialists that they would become.

Charles was a smart guy. He knew that the differences in the islands and the changes in the animals that lived there had taken time. Lots of time; he called it evolution!

Our use of online panel data has much to learn from island biogeography. Think of each online panel as an island. They have similarities but are drawn from different sources. We should not expect them to be identical; we should expect them to be different. Our research regarding the US data sources has shown them to be quite inconsistent (Gittelman and Trimarchi, Feb 2009 CASRO) and the ARF supports this point. The panels are not interchangeable. The online panels are drawn from different sources, are subject to differing management practices and for a host of reasons yield different results.

Darwin spent quite a bit of time explaining how different birds adapted to eating nuts of different sizes and textures: their beaks changed in time. We explain to our clients how buying behavior is affected by events; we must separate event related change from underlying sample shifts.

We need to know the differences between panels at any given moment so that we can understand how the panel we use changes through time and events; we need to know its consistency.

In the pursuit of analyzing consistency Sample Source Auditors, a division of Mktg, Inc., has moved onward from its initial study of the American markets and has expanded its research to include 150 panels in 35 nations. For each panel a standard instrument is used in a tracking study that includes a diversity of measures, but mostly focuses on buying behavior segmentations. By conducting repeat waves of this consistency study, a local Grand Mean is calculated for each market. In addition, using standard quality control techniques an analysis of the consistency of each panel is conducted.

CONSISTENCY TEST SUMMARY

How consistent is the performance of the data source?

Respondent data quality affects survey results. Shifts in quality create inconsistencies. Here failures to follow instructions and logic errors in responses are tracked from wave to wave in a consistency study.  Chart 2 shows the relative performance of the series of datasets for one source along with references. The lines on the bars represent the error bounds.

Buying behavior, our most important measure.

Changes in the panel population are measured by the distribution of structural segments including buyer behavior segments.  This segmentation scheme captures the overall effective changes in over 30 variables and reflects the population.  Variation in this distribution across countries is fairly large but only marginal within countries.

How different are these segment distributions?

Distance measures and statistical tests (chi-square) are used to test the difference between distributions.  Chart 4 shows (blue line) the results of the distance tests of various datasets collected over time compared to the references.  Note that the most recent dataset shows larger differences than the previous sets.