Posts Tagged ‘Quality’

With the economic recovery already peeking, we went in hope of seeing what changed.  Of course, we are thinking of only one issue.  Is there a switch from price to quality? 

We sell quality.  We are quite pleased to inform you that sales were good in Asia Pacific.   With pricing pressure starting to lift, there seems to be a move toward quality.  And yes, the end users drive the change.

Have faith.

When Charles Darwin hauled himself onto the volcanic shores of the Galapagos Islands he took samples from as many islands as he could reach. For the most part, these isolated little islands were different from one another. Even birds that could fly from one island to the next were different. He didn’t have a census to draw conclusions: He was the census!

Darwin took samples and wrote a pretty good book. The samples were not grounded in probability theory nor could he generalize from island to island. Vive la difference! It was the differences in the samples that gave him clues. Each island was an ecosystem unto itself and the differences that species on the islands had to endure shaped them into the specialists that they would become.

Charles was a smart guy. He knew that the differences in the islands and the changes in the animals that lived there had taken time. Lots of time; he called it evolution!

Our use of online panel data has much to learn from island biogeography. Think of each online panel as an island. They have similarities but are drawn from different sources. We should not expect them to be identical; we should expect them to be different. Our research regarding the US data sources has shown them to be quite inconsistent (Gittelman and Trimarchi, Feb 2009 CASRO) and the ARF supports this point. The panels are not interchangeable. The online panels are drawn from different sources, are subject to differing management practices and for a host of reasons yield different results.

Darwin spent quite a bit of time explaining how different birds adapted to eating nuts of different sizes and textures: their beaks changed in time. We explain to our clients how buying behavior is affected by events; we must separate event related change from underlying sample shifts.

We need to know the differences between panels at any given moment so that we can understand how the panel we use changes through time and events; we need to know its consistency.

In the pursuit of analyzing consistency Sample Source Auditors, a division of Mktg, Inc., has moved onward from its initial study of the American markets and has expanded its research to include 150 panels in 35 nations. For each panel a standard instrument is used in a tracking study that includes a diversity of measures, but mostly focuses on buying behavior segmentations. By conducting repeat waves of this consistency study, a local Grand Mean is calculated for each market. In addition, using standard quality control techniques an analysis of the consistency of each panel is conducted.

CONSISTENCY TEST SUMMARY

How consistent is the performance of the data source?

Respondent data quality affects survey results. Shifts in quality create inconsistencies. Here failures to follow instructions and logic errors in responses are tracked from wave to wave in a consistency study.  Chart 2 shows the relative performance of the series of datasets for one source along with references. The lines on the bars represent the error bounds.

Buying behavior, our most important measure.

Changes in the panel population are measured by the distribution of structural segments including buyer behavior segments.  This segmentation scheme captures the overall effective changes in over 30 variables and reflects the population.  Variation in this distribution across countries is fairly large but only marginal within countries.

How different are these segment distributions?

Distance measures and statistical tests (chi-square) are used to test the difference between distributions.  Chart 4 shows (blue line) the results of the distance tests of various datasets collected over time compared to the references.  Note that the most recent dataset shows larger differences than the previous sets.

So when are we dumber than a guppy?

The answer is simple:  when we neglect to collect the information we require for our own survival.

When we fail to gather information about our sample sources.

On a typical project we’re operating at a deficit of information.  We are too willing to accept sample sources’ black box mentality, their unwillingness to share information critical to our ability to assess them.

When a sample source dries up as its respondents get burned out, we either allow those sources to backfill with unnamed partner sources, or we switch to another source entirely.  We are focused on reaching our desired number of completes and keeping costs down rather than underlying sample consistency.

The real risk with sample inconsistency is shouldered by the end users of research, not the designers and executors.  It is they who should be the most alarmed.

Ron Gailey, formerly with Washington Mutual and now with Coca Cola, released data in 2008 that illustrates the pain that can be imposed on client companies.  A superb researcher, Ron collected 40,000 interviews for 29 studies, only to find out that demand for WaMu’s financial products had suffered a meltdown in his surveys.  It was not the kind of data that a market researcher wants to bring to management. Especially when it is unsupported by reality.

Ron tore into the data to find out what was going on.  With the help of a high quality panel source, he discovered the panel tenure of respondents driving the survey results was increasing.  Longer term respondents were
more conservative about purchasing than were new ones.

After all that research, Ron had no choice but to pray that business decisions that had relied on the data would not hurt WaMu.  A total of 29 studies were affected.  The scope of the damage was unknowable.

In order to optimize, we must know more about our sample sources.

At Mktg, Inc. our Sample Source Auditors division collects data on panels through various optimization techniques.  We also audit the consistency of panels over time and/or the compatibility of panels around the world.

We grew up measuring all samples against the census and adhering to a strict standard of probabilistic sampling.  Those days are over.

Now our samples are increasingly different from the census, and we’re operating with a lack of information.  If the data we produce is the product of inconsistent sample, then we are advising our clients to take unmeasured risks.

We are out there without a probabilistic framework.  If we don’t adopt a new model for sampling, we’re going to be someone’s lunch.

The guppy makes its decisions based on potential risks.  If we are exposed to the risk of having our research invalidated due to inconsistent online samples, then we had best begin to track that consistency.

The risks of not knowing are too great.  Ask your suppliers if they have passed the guppy test.  It is better to eat than be eaten.

Like many marketing researchers, I didn’t plan this career path from a young age.  I brought transferrable skills from another field.

As far as I know, I’m the only person in our industry with a doctorate in Ecology.

My background in the “hard” sciences gives me a unique perspective on the research industry.

Ecology is not for the faint of heart.  Nor is business.  Things are born, grow, and get eaten by predators.  All in a day’s work.

The marketing research industry is a pond, and you are a guppy.

Guppies have buddies who never make it to the pet store.  These little guys mean business.   And it’s their approach to business that interests us most here.

Guppies do quite a bit of research in their purchasing decisions.  They forage around looking for a good meal in all the likely places.

Their decisions are much like ours.  In these days of busy schedules we shop where we can optimize our time and energy.

Guppies do the same – they optimize.  While food is abundant and efficiently obtained they limit their travels.  There is a risk for guppies in swimming around:  they could become someone else’s lunch.

When the aquatic supermarket is well stocked guppies will eat as they keep an eye out for their own safety.  In the millions of guppy generations that have preceded this one, this process of information gathering has
become essential to guppy survival.  There is no need to take an excessive risk by wantonly traveling all over the pond when staying put will do.

Guppies are good mathematicians.  They constantly calculate the risk/return of their decisions.  There is no hope in the pond for a guppy that doesn’t have a well designed optimization formula in its head.

Optimization is a fairly complex thought process, but it has been shown that even insects can do it.

Once the formula is in place, then the hard work is gathering the data to keep it up to date.  Our little fish friend constantly monitors food abundance, how much time it takes to gather lunch, the energy it must
expend and the amount of attention it attracts in its foraging.  All this goes into its computer-like mind, and a constant flow of decisions follow.

There wouldn’t be a guppy left on the planet if they couldn’t do it.  They are born knowing it’s a fish eat fish world out there.  Information is king, and they gather and monitor it constantly.

Guppies know optimization isn’t a luxury, it’s a necessity.

When food becomes more scarce, the guppy must forage further and further.  Perhaps swimming through the weeds to the bottom of the pond will serve best.  Our little friends take the information they have gathered about
risk from the cues they glean from the environment.

When it seems prudent down they go, through the water to the bottom where they hope to find their next meal.  Such “switching” behavior isn’t taken lightly.  Who knows what lurks in the darkness beyond the lily pad?

The essence of market research is information gathering and interpretation.  We help our clients optimize decisionmaking in their own ponds.

So why am I telling you this tale?

It’s because our guppy friends have something to teach us.  About survival.

In the marketing research industry, we’re ignoring the most basic, instinctual survival system, a system so simple and basic that even a guppy is doing a better job than we are at it.

We’re ironically overriding our instincts and skills as information gatherers and denying ourselves critical information we need for survival.