Posts Tagged ‘Sample’

Well, we are Yankee fans and we write this Blog so we must be right.   But this has to tie into sampling or it makes no sense here.   Baseball teams have to fill out a roster according to a fundamental model.   They will be playing a game that has offense and defense.  During those moments that they play defense, they require nine players, each with a particular position to fill.  Think of those nine positions as the model that drives sample selection in the baseball world.

The baseball diamond, with its nine field players, structures the kinds of people that will be on a baseball team.

So when George S. puts together a team, he has to keep in mind that his census of nine must be considered as part of his criteria.

When we chose sample we need to fill out some form of model that we seek to represent.   It could be the census of the United States or that of an industry.  Whatever it is, the sample frame must represent something.  Keep that in mind next time you order sample from your friendly provider.

What does your sample frame represent?

We believe that when you buy sample from a panel house you are accepting their blending model by default.   Therefore, even if they do not do any blending you are assuming that they do.

Baseball is a game of stats and blending is the practice of combining sources with the use of statistics.  Baseball is ubiquitous in the US as are soccer and rugby and cricket wherever you call home.

Blending and the rules of the game had best become second nature to all of us.

When we say that a blend is a statement of intent on the part of a researcher we mean that in every possible way.  Our conclusion is that we have the professional responsibility to know what the blends we receive are best used for.

Many panel companies mix sample sources together for good business purpose.  At best they might pursue some element of consistency.  Think of the last time you used a sample from an online provider, did you really enquire as to its sample design.

The samples we receive are the product of their constituent parts.  To assume that these parts are the same from sample to sample and time to time is almost ludicrous.  What we get is what we ask for.  When we fail to ask the right questions, we get an unknown amalgam of whatever is best assembled by the sample provider.

Yes.  They intend well.  But they are not the sampling professional.  We are the ones who will take data that comes from those respondents and make something out of it.  What can you make when you have no understanding of its component parts?

When I was a teen I worked as a carpenter’s helper in a cabinet shop.  Good experience overall, ended up keeping all fingers but nearly lost a thumb.

I remember Joe Balino (pronounced Baleeeeeeeeeeeeeeeeeeno).  Picking me up like a rag doll and carrying me to the hospital where eight stitches closed the wound but not the pain of the memory.

Joe had warned me of how quickly a table saw could make a small mistake into a big one.

He had also told me to measure twice and cut once.  As I had slid that piece of walnut into the saw I wasn’t focused on the possibilities of injury.  Instead my head was someplace else.  Walnut is an expensive wood and Joe trusted me not to waste any.  I wasn’t sure of my measurements and I was focused questioning the cut I was making.  Would I be wasting precious material?  Losing Joe’s confidence?

I had measured once carefully.  I had skipped the second step.  I knew the rules and I was breaking them.  I was relying on my skill of measurement and assuming that nothing had gone astray.

Joe never let me into the shop again.  Not even to sweep the floors.  He wasn’t a cruel man but I had lost his confidence and represented too big of a risk.

So it goes.

None of us are confident in our online sampling frame.   Yet we have confidence that our questionnaires will turn the trick for us.  In the thirty years of helping people collect data I have seen them all.   Most don’t belong in the shop.

Do you?  If you aren’t distracted by the weakness in the online sampling frame than you are prone to disaster and if you are, then you will watch your own undoing.

We must begin to measure the quality of our sampling frame.  Then, and then only, can we entrust it as a vehicle for the measurement of the questions that our clients put to us.  If you don’t want to adhere to that simple concept than maybe you ought to re-consider if you belong in the shop in the first place.  After all, you are a danger to yourself and, I’m sorry, a hazard to all the rest of us.

The very nature of survey research requires that online sample sources have robust quality standards. It is impossible to interpret market research results with confidence without a thorough understanding of the sample sources from which respondents are drawn. This is particularly important in the case of both tracking studies and multinational studies, where both validity and consistency are critical.

Since late 2007, our firm has been compiling and analyzing data for just such an assessment. The Grand Mean Project is an extensive study of global online panels. For the study, at least 400 respondents are collected from each participating panel. A standard online questionnaire, translated into local languages, is utilized, including a focus on buying behavior and a broad spectrum of other subjects. This program has collected data on more than 150 panels across 35 countries. In eight countries, at least five companies have participated, allowing the creation of a grand mean. In 17 countries, multiple panel data has been collected and in 10 countries one panel has participated. To our knowledge, it is the largest and most comprehensive online sample assessment to date.

You can read the full article published in Quirks / November 2009:
Finding the strength in numbers by Steve Gittelman and Elaine Trimarchi

When Charles Darwin hauled himself onto the volcanic shores of the Galapagos Islands he took samples from as many islands as he could reach. For the most part, these isolated little islands were different from one another. Even birds that could fly from one island to the next were different. He didn’t have a census to draw conclusions: He was the census!

Darwin took samples and wrote a pretty good book. The samples were not grounded in probability theory nor could he generalize from island to island. Vive la difference! It was the differences in the samples that gave him clues. Each island was an ecosystem unto itself and the differences that species on the islands had to endure shaped them into the specialists that they would become.

Charles was a smart guy. He knew that the differences in the islands and the changes in the animals that lived there had taken time. Lots of time; he called it evolution!

Our use of online panel data has much to learn from island biogeography. Think of each online panel as an island. They have similarities but are drawn from different sources. We should not expect them to be identical; we should expect them to be different. Our research regarding the US data sources has shown them to be quite inconsistent (Gittelman and Trimarchi, Feb 2009 CASRO) and the ARF supports this point. The panels are not interchangeable. The online panels are drawn from different sources, are subject to differing management practices and for a host of reasons yield different results.

Darwin spent quite a bit of time explaining how different birds adapted to eating nuts of different sizes and textures: their beaks changed in time. We explain to our clients how buying behavior is affected by events; we must separate event related change from underlying sample shifts.

We need to know the differences between panels at any given moment so that we can understand how the panel we use changes through time and events; we need to know its consistency.

In the pursuit of analyzing consistency Sample Source Auditors, a division of Mktg, Inc., has moved onward from its initial study of the American markets and has expanded its research to include 150 panels in 35 nations. For each panel a standard instrument is used in a tracking study that includes a diversity of measures, but mostly focuses on buying behavior segmentations. By conducting repeat waves of this consistency study, a local Grand Mean is calculated for each market. In addition, using standard quality control techniques an analysis of the consistency of each panel is conducted.

CONSISTENCY TEST SUMMARY

How consistent is the performance of the data source?

Respondent data quality affects survey results. Shifts in quality create inconsistencies. Here failures to follow instructions and logic errors in responses are tracked from wave to wave in a consistency study.  Chart 2 shows the relative performance of the series of datasets for one source along with references. The lines on the bars represent the error bounds.

Buying behavior, our most important measure.

Changes in the panel population are measured by the distribution of structural segments including buyer behavior segments.  This segmentation scheme captures the overall effective changes in over 30 variables and reflects the population.  Variation in this distribution across countries is fairly large but only marginal within countries.

How different are these segment distributions?

Distance measures and statistical tests (chi-square) are used to test the difference between distributions.  Chart 4 shows (blue line) the results of the distance tests of various datasets collected over time compared to the references.  Note that the most recent dataset shows larger differences than the previous sets.

She’s dead.

By now she would have been too old anyway. So I guess the fact that she’s dead isn’t a tragedy.

I loved dinosaurs when I was a kid. I dreamt of discovering one of my own. Secretly, I still hold them in high esteem.

The subject of their demise has fascinated generations. They say an asteroid did it.

Probably did.

I guess we would all have to agree that an asteroid crashing into earth and kicking up a cloud of dust so thick that the sun disappears from sight is the consummate unpredictable event.

Yet there are scientists who have loud, deep voices and speak in planetariums that will tell you the probability of one coming soon.

How soon?

It’s the kind of mystery that kept me up at night. After all if the asteroid was coming there probably was a lot I wanted to experience before I was twelve.

I guess it would be good to know.

Planning for the unexpected is what insurance companies tell you they are all about. But no insurance company could survive the payout.

Maybe that’s why we need a space shuttle.

I guess there’s not much we can do about a catastrophic global disaster – the ultimate unpredictable event.

Market research depends on the reliability of our sample frame. And if the sampling frame is going to change, we must be able to predict that change. Sample must be reliable and predictable.

Consistency includes the concept of measuring change in our sample frame. If we are really good at it, the more data we compile, the better we will be at predicting that change.

No, asteroids do not fit into our marketing plan, but changes in seasons and cyclic shifts in the economy could find a place.

Throughout the history of market research, we have grounded ourselves to Census Bureau statistics. They changed through time, even though the big count was every ten years. If it didn’t change, the founding fathers could have done it back in the early 1800’s and called it a day.

No the census changed, albeit at a crawl, but the changes provided us with trends. Slow change sounds more like a trend. An asteroid making a big bang does not.

We need metrics in our research that measure change in our sampling frame on a more microscopic basis than the old macroscopic shifts we have counted on all these years. No, we will not capture the asteroid stuff, but seasons make sense.

Now that we are in the age of online research, where fast is the guiding philosophy, it is past time to come up with a family of progressive metrics that give us anchors for our research.

If we don’t we will go the way of my favorite dinosaur. It was discovered by Elizabeth Gomani from Southern Methodist University, who calls Madagascar home. That’s where the big guy laid his head down to rest and that is where Elizabeth found him and wrote her doctorate about him.

My favorite dinosaur is in the picture below. The big one on the right. Sweet as a lamb, couldn’t hurt a fly, unless she stepped on one. She was about the size of a bus.