Thu 17 Jan 2008
The pollsters and the pundits were doing a lot of back peddling following the results of the New Hampshire Primary on Tuesday. At least eight polls released in the days before the vote showed Democratic candidate Barack Obama leading Hilary Clinton by anywhere between seven to thirteen points – all well above the published margin of error.
The cognoscenti not running for cover immediately began trotting out their theories to explain away these numbers. Principal among these was “the Bradley effect†– a notion first broached in 1982 when Tom Bradley, a former black Los Angeles Mayor, was given a nine to 22 point over lead in the run-up to the California Governor’s race, only to end up losing to (white) Republican George Deukmejian. The theory held that voters, responding to telephone surveys, offered a “politically correct†answer that they were going to vote for an African American candidate when in fact, that was not their intention at all.
Twenty-five years ago this theory was simply that – an unknowable and improvable supposition. Today, as pollsters begin to explore the differences between results obtained by telephone surveys, compared to those administered over the internet, we have empirical evidence to suggest that telephone research may systematically overstate “socially desirable†outcomes. Whether the question explores church attendance, extra-marital affairs, smoking pot or virtually any other behaviour or belief that carries a social stigma, these comparison have shown that voters contacted by telephone consistently offer a more socially desirable response than those answering the same questions (without the intermediary of another person being party to their views), via the Internet.
The confluence of American attitudes towards race and the social desirability bias of telephone interviewing therefore could account for some of the difference between the published polls and the actual results. But the very fact that this social interaction bias is inherent in telephone interviewing, means that these same polls cannot give us a precise measure of this effect, before election day.
Less frequently offered as an explanation for these “bad polls†is the very nature of small-state, US primary contests themselves. Notwithstanding headlines declaring record-breaking turnout, only about 250,000 voters cast their ballot in the Democratic race in New Hampshire – equal to what we might expect of a mayoralty contest in a mid-size Canadian City. Unlike the race for mayor — say in Ottawa — however, voters in New Hampshire are registered and their political affiliation is known, in advance, by the respective campaign organizations. Also, unlike municipal politics in Canada, Democratic candidates in the Granite State spent tens of millions – on mere, hundreds of thousands of voters – to identify and make direct contact with those individuals known to be most likely to support their candidates.
While we may try, polling cannot predict who will actually turnout and vote and who will not. In other words, voters might tell us they are going to go to the polls but if their babysitter doesn’t show up or if there is a massive snow storm or even if their favourite television program is on, the pollster has no ability to factor these circumstances into their findings.
Normally we don’t worry much about this and tend to ignore our own admonishments about the discipline’s limits when it comes to predicting turnout rates. And normally we can get away with this because, in general elections, no one Party’s supporters are more or less likely to turn out than supporters of the other Parties. In small population, big money campaigns however, where the spending and organization of one candidate can be superior to the other’s, differences – undetectable by the polls — in turnout can effect the actual vote result and make the polls look like they are in error.
Race and the idiosyncrasies of small state US primaries aside, the additional fact remains that, while rooted in scientific random probability theory, polling is, by definition, imprecise (ergo, the margin of error) and unable to replicate the very thing it purports to measure. Typically, pollsters will ask “how would you vote for if the election was held tomorrow†or “which candidate are you most likely to support in the election to be held on Date X�
As if it needs to be said, these are hypothetical questions that are not asked to voters as they stand over their ballot, preparing to cast their vote. Yet we know (from post-election research conducted in Canada, the US and virtually every other jurisdiction in the Western world) that between 8 and 13 percent of voters claim this is exactly where and when they made up their minds. In fact, exit polling conducted by the New York Times when New Hampshire voters leaving the polling stations showed that among those who claimed to have made up their minds “in the last weekâ€, Obama had a 15 point lead, but among those who decided “todayâ€, Clinton was ahead by three percent. Clearly there was a significant shift towards the New York Senator at the last moment that went undetected by the polls conducted in the days – and even the day – before the actual vote.
What went on in those voter’s minds as they stood in the ballot box? Were they thinking about Hillary’s emotional outburst in a restaurant the day before? Were they balancing the merits and demerits of race versus gender politics? Or were they thinking about their favourite television program that was going to be over if they didn’t hurry up and make up their mind?
Over my career, I have conducted surveys in over 50 election campaigns in three continents and I have never – and as far as I know, nor has anyone else – conducted surveys while voters were in the polling booth. So the honest answer to the question of what goes through the minds of voters in this private moment is that we do not have a clue….and in this regard, I am 100% confident of the pollster’s inability to absolutely predict election outcomes, 20 times out of 20, with zero margin of error.
New Hampshire should remind us all of us in the prognostication business that we are not god-like clairvoyants, but in the words of The Godfather of modern-day polling, George Gallup, merely taking a “snaphot in a particular point in timeâ€.
