Jon Krosnick and his students fulfill a very important role in survey research that more academics should be concerned about – establishing scientifically what the best way to conduct research is, but also what tools are effective in improving the less scientific methods that most of us are forced to rely on instead.  In the latest volume of POQ, David Yaeger, Dr. Krosnick and others argue in “Comparing the Accuracy of RDD Telephone Surveys and Internet Surveys…” that despite declining response rates, telephone surveys remained reliably more accurate on a number of non-demographic benchmarks than were seven non-probabilistic online samples.

We applaud an approach driven by behaviors, but as the authors admit, the selection of benchmarks is a weak one, reflecting variables that are by and large likely to be independent of the selection biases caused by an RDD design, particularly after post-stratification weighting (the paper uses the non-demographic benchmarks of drivers’ license and passport ownership, cigarette smoking and alcohol consumption).   This is through no fault of their own, but rather a very high standard for reliable benchmarks, drawing only what could be found via large-sample government population surveys.   The need for reliable and relevant benchmarks is becoming overwhelming as online research seeks its coming of age.

One issue that bothers us, however, is the insistence on treating telephone and online sources as being mutually exclusive modes.  It is our belief that the true future of representative sample lies in using references that consist of some blend of the two, with each compensating for the others’ selection biases.

A secondary point the article draws attention to, and that we feel is very important, is that even rigorous post-stratification weighting on non-probabilistic samples will not necessarily ameliorate non-representativeness.  In fact, the results suggest that weighting only inconsistently shows even a positive effect on the accuracy against other benchmarks.  This confirms what we have claimed for years.  As many industry-leaders now realize, controlling demographics is simply not enough to achieve a behaviorally representative sample frame.

We find that Kristen is a great source of entertainment.   She is an endless stream of quotables.   Let us provide another example.

Early in the conference Kristen could be heard saying, “I am not presenting and I don’t care.”

Right.

That will be the day.

But Kristen made a quick recovery:    She reported that she will be presenting at the Best of ESOMAR in San Francisco, ESOMAR 3D, and on and on.   We are quite impressed and proud of our friend.   But when the day comes that she doesn’t care about presenting that will be a sad day indeed.   Let’s face it, she is a presentation junky.

ESOMAR picks the best venues, has the best dinner party, and has the most fanfare of any conference we attend.  The sessions can be memorable as well.   One in particular resonates with us.   The question from the audience related to a comment from a panel that stated that we had not worked from representative samples for a long time—thirty years.   Thus the audience member asked if being consistent didn’t hide possible changes in the underlying sample frame that might represent valuable changes in sentiment.   If in fact the predilection to purchase dropped but was stabilized by researchers seeking to have consistent data, when is it that the dangerous trend would be discovered?  The response from the panel was nothing less than, “In the sales data.”

A memorable comment indeed.   We suggest that it should be a rally cry for all researchers.  This came from a powerful end user who simply made it clear that what we do was of diminishing value as they would rather purchase data that might potentially mask a dangerous trend rather than pursue the collection of accurate data that would protect us all.

Our view is that the failure of the sample frame to be reflective of real changes in the data is our worst enemy.   Everytime an important trend is missed our credibility goes out with the numbers.

No analyst looking back on such an important oversight would blame the purchasing agent who wouldn’t pay for something short of excellence.

According to one of the travel guides the only thing in Amsterdam that is worse than the accumulated dog feces is the silent attack of thousands of bicycles.   Okay not thousands, tens of thousands, if not hundreds of thousands.   Everyone rides a bike there.   This doesn’t take your smartest statistician to figure out.   One bike parking garage alone, filled to capacity, seemingly boasted that it was limited to 3000 bikes.

The streets are clogged with them all chained together in bicycle chaos.   Some hang over the many bridges and others simply fall into the canals.   One lofty tour guide reported to us that some 100,000 bicycles are stolen and then dumped into the 2500 miles of canals every year.

Bicycles don’t have motors.   Silently they come upon you at break neck speeds.   Often they carry children in baskets.   The whole family is out there to get about town.    Seemingly, no pedestrian is safe as the bicycles come at you from every direction.   Yours truly was nearly part of three accidents on the first day or two.   Then you become wise.   You either rent a bike yourself and become part of the assailants themselves or you simply walk along clinging to protective devices like trees.  It’s a tough job but I’d rather cross the street at 42^nd and Broadway in Manhattan than brave a side street in Amsterdam.

…at least when we arrived at 5:00 in the morning on Saturday.   We had no idea where to go so we just walked.   The place looked as if a giant party had left the place in a glass, litter and foul mess.   It makes sense:  the place had not quite cleaned up after the happenings of the night before.   The garbage trucks had barely ventured out, probably discouraged by the slow rain and darkness.   Nothing was open for us to hide in and we had severe cases of jet lag.   There is nothing like a red eye from JFK to burn you out.   The buildings seemed to be swaying in the grim grey light.  Some leaned in to the street others side to side.   The canals were clogged with refuse.   We wanted to take the next plane out of there.

We struggled for hours trying to convince ourselves that we hadn’t made a huge mistake.   There was no place for us to have a warm breakfast.   There were lots of coffee shops (please don’t think Starbucks) but none were open.

The first breakfast appeared available around 9:30.  It all seemed like a nightmare. But after the sun rose and people started to appear and we did more exploring around the city, Amsterdam showed us what it was really all about.  It was a wonderful city!

We found inspiration in the museums, canals, conference sessions and coffee shops. Perhaps a tremendous opportunity was missed because had we held the session in the coffee shops we might have achieved a new and vibrant interpretation of our beloved discipline.  Oh well, next time, Atlanta.

We are thrilled to have the next ESOMAR in that beautiful southern bell Atlanta. But we cannot help but think that there should be a special gathering of everyone who wished to attend Amsterdam and missed it or just those who plain old miss Amsterdam. We suggest a special cigar party—Kind of a spinoff from the CASRO party held at the annual each year. You figure it out.

We love Kristen Luck.  But Kristen, you out did yourself this time.  Now if Kristen wasn’t smarter than every other banana in the bunch we would assume that she meant what she said and said what she meant.   However, that’s what she said.   We’ll try to leave out any talk of lofty statistics in your presence Kristen.

Panel Study,  L. Ren, L. Li,     Q. Xu, J. Yan, Z. Qiu,  Peking University China.

Proper occupation assignment is often critical in interpreting social science and market research data.   It is a dilemma that haunts researchers around the world, one not particularly limited to China.   The challenges inherent in different modes of data entry, complicated by differences in the nuances of meaning that can be found in fine grain interpretation of occupations are a nightmare.

Here two methods of entering/interpreting the data, drawn from a Computer-Aided Personal Interviewing (CAPI) survey, were employed:  entering the data live by the interviewer and entering the data after the fact by experienced coders.   The results differed between the two methods significantly and meaningfully.   Imagine how different such results must be between respondent entered data in an online survey and interviewer entered data in a telephone interview.   It’s a frightening prospect.

Academicians are very sensitive to the sample frame.   Market researchers should be as well but we have adopted the non-probabilistic access panels as our main online sample source.   It was refreshing to see how many talks were devoted to probabilistically created panels.   For us it was a sampling heaven.

Based on the talks we heard, the Europeans are still pioneering on their use of probabilistic panels.  No large commercial panels of the type have made their appearance and those that exist are academic or government based.   They are still testing the waters but are clearly sensitive to the issues.

A battery of studies came out of work performed by the CentERdata Tilburg University, the Netherlands.   In 2007 CentERdata started the LISS panel (Longitudinal Internet Studies for the Social Sciences), which is “an online panel which is based on a true probability sample of households.”  It is representative of the Dutch population and consists of 5000 members.  This may be small by commercial standards but apparently they are producing quite a bit of research as we were privileged to listen to three good presentations.

The LISS panel began with a random sample of some 8000 respondents in 2007.  Those who did not have internet connection were provided with a computer and a connection.    By 2010 approximately 5000 were still active (see www.lissdata.nl).  The “attriters” were found to be different from the “stayers” in that the latter were more conscientious, open and female.   Those that tend to leave the panel were found to be Christian Democrats less often.

“Lurkers”, those who come and go in terms of completing some or many of the 25 monthly waves of the subject study, tended to have a lower level of education.

Those who abandoned the panel entirely, as opposed to the lurkers who came and went, were labeled “monotonic attriters”.

This is the first paper that we know of, besides our own, that drew a behavioral distinction between attriters and stayers.   We consider this to be an important finding;   One that we do not feel that we can do justice to here.   But the concept of a changing panel population created by differential attrition is one that we must begin to address.

The big conferences have a pattern of their own.   Soon you find your groove and recognize those who share the same interest that you do.  Time and again they take seats in the same crowded room or are found waiting outside of the same room where the talk has gone over time.

Often the colleagues that we make are fellow speakers who are advancing their craft.   Like us they are there to soak up content.   It is with them that the side discussions can become lively.   You exchange business cards or ideas.   Whatever it is, like trout going upstream, you find yourselves climbing the same staircases.

The academic buildings are rather drab.   We are not speaking of fancy conference centers here.   There are no bars to gather in, no plush carpets, or chandeliers.   Instead, un-air-conditioned, formed concrete towers with lecture halls instead of banquet halls.   Don’t fool yourselves, while the trappings may not be fancy, the intellectual stuff really is.

Oh, don’t get me wrong, some of the talks can be deadly.  One professor droned on in a monotone voice heavily laden with equations.  A doctoral student sat listlessly reading from her notes.   One government type got up there and winged it without slides but handed out a photocopied list of variables for every other person.   When you get pinned into a room by a crowd and someone comes unprepared or presents poorly it is frustrating.    Most of the talks are vibrant and stimulating.   Some of the graduate students are so wired into what they do that they speak their subject matter with a natural fluency that can only be admired.