Archive for February, 2010

Here’s a tip. If you want to know what is really going on let people speak their minds. Set up rules for good conduct, but let the data speak. That’s what happened at Net Gain 4.0. We applaud the Canadians.
They did it right. The data sets were split. Separate analytical teams had open season on data sets. The questionnaire was agreed upon in advance. All had one objective: learn.
Don Ambrose and Peter Chan were given the data file to do as they saw best. We did not exchange slide decks weeks in advance. There was no agreement as to who would cover what. Intellectual freedom reigned. If there was an intellectual filter in the form of a programming committee, we certainly never felt its wrath nor did we hear from them.
A day or two before the event we exchanged slide decks. The MRIA was given complete right to eliminate anything they chose and we, as guests, would have complied. They told us that the names of the panels should not be associated with any data points and both sides kept to that understanding. Our analysis was on aggregate data looking at driving variables not inter-panel differences while theirs tread closer to inflaming a panel or two by showing the variability between panels, but not labeling who was who. In fact they don’t know who was who since the data was sent blinded to them; they also never asked.
We are grateful to the MRIA and their leadership for intellectual freedom. We are moved.

We have long admired Jeff Miller’s studious approach toward the online panels. No one that we know of in the full service community has spent more time understanding the panels themselves then Jeff. Well maybe, we have, but Jeff did it before us and because he pioneered the regimen, he deserves the credit of being the father of the science.

He has ushered Burke into competitive superiority by understanding the panels and their underlying sampling frame. It made good sense that he was chosen as the lead keynote at Net Gain 4.0 in Toronto.

We are so involved in daily struggles with online quality that it is good for us to lean back and watch a master at play. Jeff outlined the panels, provided a cogent overview of their problems and virtues so well that for a moment not much could be said: But only for a moment.

Apparently Jeff had been keynote up in Toronto before. So his was a repeat performance. It was a very good one.

Now don’t get carried away. We didn’t agree with everything that Jeff said but it was close…for those who keep score of such things.

As senior statesman, Jeff’s overview made countless points. We’ve stolen a few figures here to carry his message. He spoke of online research spending as having reached a mature stage. In this stage, he suggested, we can take a relative breather and address the problems that the “new” methodology has brought with it. Among them he provided the following cogent overview:

We believe that he has provided an accurate summary. We were particularly thankful that he mentioned professional respondent conditioning as a threat. After all, that was to be the subject of our talk an hour later.

Elaine and I had been looking forward to the Net Gain 4.0 conference in Toronto for quite a while. The build up for us has been fantastic. We were approached by the MRIA in the Spring of 2009 to conduct a baseline study of the Canadian online panels in cooperation with the Market Research Intelligence Society.

The proposal was simple: we would complete a split sample of as many panel companies as could be brought to the table. Peter Chan of TNS and Don Ambrose of Consumer Contact/Research by Net, representing the society, were given nearly free reign over the questionnaire. They made changes eliminating the income quota and adding geographic quotas. As we had already studied five panels before they initiated their effort it was understood that we would split the sample, some using the Mktg, Inc.’s version of the Grand Mean questionnaire and the rest using the slightly changed version offered by the Canadians.

This was our dream model. The Grand Mean Project™ has always sought cooperation with the market research societies in the 35 countries where we have been collecting data. In the United States, the climate among the associations is quite competitive. In Canada the research community appears to have unified behind one association and to their credit it works.

To understand Net Gain you had to be there. Standing room only, lot’s of energy and a great crowd. The talks inter-related nicely and maintained their focus. It was quite a show. Our hats are off to our neighbors up North.

When I was a teen I worked as a carpenter’s helper in a cabinet shop.  Good experience overall, ended up keeping all fingers but nearly lost a thumb.

I remember Joe Balino (pronounced Baleeeeeeeeeeeeeeeeeeno).  Picking me up like a rag doll and carrying me to the hospital where eight stitches closed the wound but not the pain of the memory.

Joe had warned me of how quickly a table saw could make a small mistake into a big one.

He had also told me to measure twice and cut once.  As I had slid that piece of walnut into the saw I wasn’t focused on the possibilities of injury.  Instead my head was someplace else.  Walnut is an expensive wood and Joe trusted me not to waste any.  I wasn’t sure of my measurements and I was focused questioning the cut I was making.  Would I be wasting precious material?  Losing Joe’s confidence?

I had measured once carefully.  I had skipped the second step.  I knew the rules and I was breaking them.  I was relying on my skill of measurement and assuming that nothing had gone astray.

Joe never let me into the shop again.  Not even to sweep the floors.  He wasn’t a cruel man but I had lost his confidence and represented too big of a risk.

So it goes.

None of us are confident in our online sampling frame.   Yet we have confidence that our questionnaires will turn the trick for us.  In the thirty years of helping people collect data I have seen them all.   Most don’t belong in the shop.

Do you?  If you aren’t distracted by the weakness in the online sampling frame than you are prone to disaster and if you are, then you will watch your own undoing.

We must begin to measure the quality of our sampling frame.  Then, and then only, can we entrust it as a vehicle for the measurement of the questions that our clients put to us.  If you don’t want to adhere to that simple concept than maybe you ought to re-consider if you belong in the shop in the first place.  After all, you are a danger to yourself and, I’m sorry, a hazard to all the rest of us.

Thank you Eric Bell and the MRGA for hosting Market Research Radio and having us as your guests…

Whew, glad we got that over. Almost forgot to thank Eric. While most of the questions were staged, we couldn’t get our screen to see the online questions that kept popping up. So, in case you were wondering there were a few that took us off guard. We were told by Eric that some 300 people were listening. So maybe we left some question unanswered questions. We are going to answer one with a bit more depth.

The most uncomfortable question came when someone stated: That it’s a small industry, the buzz was out, most companies who have participated in the Grand Mean Project™ are going to do the first wave, which is the free wave, and then stop.

Eric asked what our reaction was and we gave the following answer: It could be because companies are afraid of being transparent, or, they feel that they can’t pass the consistency test, or everyone is going to hide behind proprietary solutions. We don’t believe it is the money. Our rates are too low. The consistency test isn’t for everyone; companies who can’t be consistent should avoid it like the plague, unless they want to find out what it takes.

We forgot one other possibility: They just wish that we would go away and they don’t want transparency. They want to go back to the old ways. They are uncomfortable with what we are doing and just don’t want to hear about it anymore.

We think it’s a great idea. Let’s boycott anyone who pushes for quality and professionalism in our industry. Find some way to doubt their intentions. Undermine their effort. Keep them from moving forward.

So here’s our response. Thanks for the inspiration.