In one talk, a lively discussion took off regarding attrition bias in a probabilistic panel.   I guess we were pressing the speakers on the issue of how they sustain the representative nature of a panel through time.   Perhaps they were surprised at our keen interest.

The speaker was a doctoral student who was quickly defended by her major professor.   We meant no offense but my passion got the better of me.   They defensively said that they had seen only one paper on the subject after an exhaustive literature search.   I offered one of our papers in support of our thesis.

Afterwards, the professor approached us stating that she would like to see the offered paper as they had failed to find more than the one paper now in their files.   She seemed disturbed that she would have missed anything out there.   So I offered her my business card and said that she could pull it down from our website.

A broad grin appeared on her face.   She seemed relieved.   Next thing I knew a warm handshake was offered.   Apparently they hadn’t missed a paper, the one reference on the subject was still the only reference—ours.

We had colleagues.

This was the fourth time the ESRA held its bi-annual conference.   For those of you who have attended AAPOR in the US you already know the look and feel of the big academic conferences.   Switzerland is a very expensive place where a plain pizza will set you back $28.  But the academic conferences are rather reasonable.   So once you get there and you can work the public transport (free for visitors with a visitor card) it’s not as bad.   But a bottle of water will cost you $5 on the street so buy your food at the university.

The conference itself is a bruiser with 170 pages of small print in the conference program.   Elaine and I were both fortunate to present.  The audiences were highly engaged and the attendance was somewhere around 1000.

The breadth of subject matter was clearly different from the other academic conferences we had attended in past years—the internet had arrived.   The difference in Lausanne was the focus on probability panels.

The airport in Geneva is conveniently connected to the rail system.   Given that our luggage was delayed in route (oh how we love those connections) taking the train was a breeze.  However, given that neither Elaine nor I speak French figuring out which train was the right one required a bit of maneuvering.   So, being the experienced travelers that we are, we stood in the middle of the platform looking perplexed until a conductor came along.   His French was perfect.   His English was as good as our French.  But he understood the word “Lausanne” and pointed to the only train at the platform.

The car was rather attractive.  Nice seats; Empty except for a man and a lady with plenty of room to stretch out.   Apparently we had cut it pretty close because the conductor followed right behind.  He immediately asked for a ticket from the lady behind us and we were privileged to watch a bit of drama.   The lady declared that the ride was “free”.  The conductor suggested that it was not.  The conductor held the train and soon escorted the lady onto the platform in what appeared to be a rather efficient maneuver.  Despite the little altercation the train moved along like a Swiss watch (had to get that in there).  The conductor looked at Elaine and me to see if there was any trouble forthcoming from us.   We looked sheepishly back.  The conductor whipped out his pad and wrote us two tickets.   He said in English, “First Class”.   Then, realizing that the language barrier was insurmountable, wrote us one coach and one First Class ticket as a Swiss compromise.   No negotiating required.

The summer is over, there are tales to tell, and some stories too.   Elaine and I attended the ESRA in Lausanne, Switzerland.  It was a week of talks running four sessions simultaneously (sometimes six) and rain every day.   Lake Geneva provided a beautiful backdrop for the University of Lausanne, which reminds me of the larger colleges in the State University of New York system.  Not a particularly pretty campus, with big sterile cast concrete structures.

I would be remiss if I failed to mention the field of sunflowers in the middle of campus, with their heads bowed as the rain dripped off them.   They don’t grow much grass; the open fields were filled with low growing wildflowers.   The flat roofs of the buildings were strange; the wildflower meadows grew there as well.  I wonder how they keep the whole thing from leaking.

If you thought that no one would pay for sampling science then think again.

We saw this one coming, only we didn’t recognize who the players would be.   We thought another big player in the space was going to make the move.  It’s not the first time that a company was bought for its intellectual property as opposed to its assets. Ever since they turned Mitch Eggers from chief scientist to a road warrior demonstrating his Pinnacle platform, GMI has gained traction.   Many of us thought that GMI was a dying breed until Mitch went on the road with his novel approach to obtaining a representative online sample.   We could sense that something was afoot when Mitch stopped testing and began promoting in earnest. We tip our hat to Mitch.

If you get to London you have to go to the Parliament. But we doubt you will be as lucky as we were. Not only did we get to hear both houses in session but we heard them debate polling data.

One MP in the Commons quoted some data. We could barely catch who it was from up there in the bleacher seats. The quote went something like this, “the majority of students feel such and such”. Then he went on to another comment.

A few minutes later a member of the opposition challenged the opinion of his esteemed colleague and demanded to know his source. Immediately, as if he was waiting for the mouse to stick its head out of the hole the MP stated, “such and such a poll showed that 46% said this and 45% said that.” There was a sound of aaaaaaaaaaaaaaaaahhhhhhhhhhhhhhhhh from the members assembled, the challenging member sat down, and the truth was accepted.

Such is the power of a public opinion poll. Fill in the blanks yourselves.

It didn’t take Phil Garland (Doctor Phil to you) more than nine minutes to convey his message. And it won’t take you more than that amount of time to read the three page paper he submitted to the MRS in support of his thesis. This reminds me of an undergraduate friend of mine who wrote a fourteen page doctoral dissertation. He wrote it beautifully but completed it before his new advisors were ready for it. Of course the bureaucracy of the University he attended resisted granting a PhD in what amounted to a matter of months. He had to hang around taking courses for two years to fill out the residency requirement. Eventually his dissertation was published in a top journal and he received his doctorate. I believe he just retired as a tenured professor from a top university. Phil already has a doctorate, obtained at Stanford under the aegis of Krosnick. But his current thesis may be his most important work to date.

Phil correctly captured a few salient concepts in his nine minute presentation and three page paper. “Sampling is, of course, faster and cheaper than interviewing an entire population of any size, regardless of its possible pitfalls. However, the frames have to a) have nearly 100% coverage of the population of interest and b) ensure an equal chance of selection for each member, and c) feature a reasonable response rate for the laws of sampling to work.” George Gallup would smile.

The probability samples that we relied upon to fulfill these fundamentals have suffered dearly in recent decades and the likelihood of them being resurrected is slim. But how does SurveyMonkey stack up? “SurveyMonkey processes more than one million survey responses each day—opinions given to surveyors that, collectively, seek to indicate hundreds of thousands of objectives. The sheer diversity of those research problems all but guarantees that the targeted respondents—gathered solely by the survey researcher through their own proprietary means—are also heterogeneous.”

In other words the volume and diversity of what is going on in SurveyMonkey is so comprehensive (imagine seven million researchers putting their questions through a single platform and that is the power of SurveyMonkey—you read it right, not 7MM respondents, but 7MM survey writers) that it almost can’t help but replicate a considerable portion of the population. To test his point Garland compared the responses from 10,000 survey takers per day from June 10, 2010 to July 29, 2010 to Gallup’s United States Presidential approval rating question.

The results tracked within the margin of error almost every day. Gallup weights its data, Garland did not.
The trick is two-fold coverage and response rate. The average daily response rate of the randomly selected sample in Phil’s data was 46% and those who responded to the question represented 8,300 of 19,000 American cities (43%).

Garland summarizes as follows: “These results highlight the need for discourse about the ways coverage and response rates converge to produce high quality data.” “…for virtually no cost.”

A rather demure conclusion indeed.

Source—Garland, Philip, 2011. Sampling v. scale in the internet age: An investigation of the tension between convenience sampling, response rates, probability, and coverage. MRS research. The Annual Conference. Paper 18.

Phil Garland, Ph.D., VP Methodology for SurveyMonkey, blew the lid off of at the MRS Annual Conference in late March, in a nine minute blockbuster talk during the online quality session. His talk gave an overview of SurveyMonkey’s coverage as well as the DIY house’s enormous capability to complete hundreds of thousands of surveys in a brief period of time. With 1,000,000 respondents a day the power of SurveyMonkey to transform itself into the Gorilla of sample sources is obvious. But it was the incredible coverage that Garland demonstrated that drove the concept home.

To say that Phil spoke to an audience that perceived his employer as an enemy was obvious. Probably few had kind words to say about a company that has taken online survey’s from the hands of professionals and into the realm of just about anyone who is in the mood. But like it or not, Garland is no dummy and he speaks authoritatively about his company’s capabilities. By the way, his talk was nominated as best presentation at the conference. Congratulations Phil. Watch out industry.

We’d love to make a habit of this (we received best presentation award at ESOMAR Berlin) but we have to tip our hat to our co-presenter Martin Filz, Managing Director, UK and Senior Vice President, New Proposition Development, of Research Now for his dynamic and engaging presentation style. Given that this is a nomination and not an award, we will not find out for some months to come. If we receive the honor it will be at the MRS award banquet later in the year. Our co-presenters in the session included Philip Garland of SurveyMonkey who also received the nomination for best presentation and Joel Williams, Head of Methods, TNS-BMRB.

Research Now purchased Peanut Labs at the end of last year. The important strategic move put the sampling giant in a powerful strategic position to access social network respondents. The importance is as obvious as the reach that social network respondents provide. Clearly, they are the largest potential source of respondents that our industry has seen in a decade. Peanut Labs, the company that brought us digital fingerprinting a few years ago, has always been at the forefront of innovation. Their approach to harnessing social network respondents has to be admired. The question that remained was how to blend social network respondents with Research Now panels around the world (in the US, Valued Opinion Panel) without disturbing the consistency of those panels. Given that we are speaking of one of the most valuable sampling assets in the industry, it is no wonder that great care was taken to arrive at a conservative means to introduce this new resource to an established panel. We are quite grateful to have been hired by Research Now to create their blending models.

It is time for us to put up or shut up. We are building standards in the United States, United Kingdom and Canada. That means that we are ready to compare our method of sampling to anyone out there who has a sample…..anyone.