Archive for March, 2010

Before you read on we ask that you consider this: the ability of this industry to prosper depends upon its credibility. If the road to the future is paved by a gag order that shuts down free dialogue, then we deserve what that brings us.
Samplemetrics is a forum. In this forum we have continuously advocated for online quality. We believe that this argument is an important turning point in the history of the market research industry.
Those who depend upon our research need to believe in the conclusions we draw from it. The most important element of that belief is the credibility that we bring to the table. They will only respect us if we address issues of concern squarely.
When Kim Dedeker spoke years ago about the need for quality, she had the bully pulpit of P & G. Those of us who lack the clout of one of the largest research budgets in the world actually play a more critical role than Kim. After all, we outnumber her by an untold ratio. We are the research industry. Why shouldn’t we have a voice?
Why shouldn’t you have a voice?
What does that voice wish to say?

We estimate that there are 20 panels in Canada. Fifteen participated in the Grand Mean Project™ data collection effort that was presented at the Net Gain 4.0 conference in Toronto. There are about thirty million people in Canada. The ratio of panels to people is about 1.5 million per panel. Compare this to Australia where there are about twenty million people and some sixty panels. The Canadians have not yet reached saturation but they rank second behind Australia in having panelists who belong to five or more panels. In Canada, 46% belong to greater than 4 panels, in the United States 45% and in Australia, 55%.

These are important numbers. The rest of the world is watching. Will the Canadians step up to the plate or let it pass?

Why them? These data imply that they need to understand their panels now. The impact of panel overuse is not well documented but they have seen the first indications. They have one association that appears to have the courage to move forward. In the United States the associations compete.

The Canadians also have a research community that acts like a community. Yes there are disparate voices but they seem to understand the need for cooperative action. We believe that the Canadians are the ones.

From where we sit it’s obvious. Either with us or on their own they should be moving forward to establish online metrics to measure the status of the online community.
Whether they agree with us or not that they represent a community in transition is not relevant. They have a fabulous research community that does not appear embroiled in the politics so profound here in the States. This is their time to lead in a global framework!
They now have baseline data to build from. Our offer to assist is on the table. They have the power to establish a template for the world that could resonate around the globe. We hope that they are listening.
We ask them to look at where they are and what they have done and assess the possibilities. How should the baseline be maintained? Should it be maintained? Is there a better way to do it? Where do they go from here?

As the online communities heat up around the world, the panel method of completing market research interviews grows in importance. We must know how the respondents are changing due to all this attention.
At MRIA Net Gain 4.0 we presented Canadian data which demonstrates a likely conditioning/attrition effect that should have all of us take notice. We are not suggesting anywhere that hyperactive respondents are better or worse than new respondents. We are suggesting that those who conduct 30 or more interviews a month, belong to five panels or more, and have been opting in to do surveys, are different from those of us who choose not to do the same.
We will pass on proving that the two populations are different demographically. The MRIA did that for us in their analysis. We were bold enough to say that if you quota control the respondents and then compare along different levels of hyperactivity, tenure and panel membership, the impact on buying behavior is striking in Canada.
No we are not saying anything about the rest of the World. The rest of the world will have to make a choice: do they want to know?
If the hyperactive respondents are different from the less active or inactive respondents, do you want to know? Knowledge is a funny thing: If we are ignorant than we can always profess that we would have done differently had we known.
Online research has its ills, but we will never learn how to fix them unless we look them squarely in the face.
We say we all need to know.

What a dream.   Split samples.  Almost universal cooperation.  No politics that we became aware of.  A lead off keynote by Jeff Miller of Burke.  An independent analysis by some great researchers (Don Ambrose and Peter Chan) and then a chance to present our own slide deck to a standing room only audience.

I had to admit— we wondered if we would be upstaged by the two talks before us.  Maybe we were but we were amazed by the different perspective taken by the two analytic teams.  The takeaway from Jeff’s talk was the pluses and minuses, kind of this is where we were and this is where we are and this is where we need to go.  The MRIA presentation by Don Ambrose was along the lines of the ARF, the panels are not interchangeable, the sourcing matters and the professional respondents are at play.

What a setup for us.  The Canadians focused on inter-panel differences and we focused on factors that drove that same variability.  Our favorite these days is the combination of hyperactives/panel tenure/multi-panel membership.  The Canadian data supports the contention that all three matter having considerable influence on the buying behaviors represented by the panels themselves.  It should come as no surprise that those who are loyal to the panel process and stick with it for years join five or more panels and complete thirty or more surveys a month are different from the balance of the population that will not consider such dedication.

Just for a taste, look at the figure above from our MRIA presentation.

In the finest tradition of market research, the Canadians, Don Ambrose and Peter Chan, split sample with us.   Each team was allowed to collect its own data and present its own conclusions.  Although slight differences had been injected into the questionnaire, the Mktg, Inc. and MRIA data was put up side by side for comparative purposes.

As a point in case, the MRIA version quota controlled geography and captured more French speaking Canadians as a result.  See above.

The Mktg, Inc. sample was quota controlled for income and captured a greater percentage of those with higher incomes (see below).

No surprise.  The data compared nicely.  Conclusions presented by Ambrose included some that we had heard before and a few that came as a surprise.  Just as the ARF found in the United States the panels are different.   One should move with considerable caution before switching from one panel to another.   They source differently, have different ageing patterns, different percentages of hyperactive respondents.  They are different.

Of course everyone focused on a few panels that were extremely different.  It’s natural to look at the outliers and ask “Ugly duckling or swan?”  No matter what the conclusion drawn, Ambrose captured his audience.

Don had free rein over the data and presented a very nice overview of all fifteen panels (one panel participated twice on the Mktg and MRIA sides).  The panels were blinded so that no one got ruffled.  The room was filled to overflowing with admission tickets sold for standing room only.

Our hats are off to David and Peter!  Nice job!

Now, it was our turn to present our data.