Political Insider

Actually, as a statistics student, it is exactly the reasons above that you point to as being reasons not to average results that make averaging results in this matter useful.

So rather than disparaging them, they should be looked at as a general idea of where the race is. Don't believe me? Take a look at the results vs. polling averages of the senate races in 2006:

http://www.realclearpolitics.com/epolls/writeup/2006_senate_realclearpolitics_poll_averages-63.html

Although some were off by a few points, in no instance did RCP's average incorrectly predict the winner.

In general, I quite agree with Mr. Goddard on the dangers of averaging polls without regard to any of the internal factors governing each survey. However, if one imposes certain guidelines, such exercises can be very useful.

In my own case, I've been using a system which incorporates each poll's margin of error. While I NEVER make predictions until the last survey is released (usually the evening before or the morning of a given election), after that last survey is public, I usually go with a prediction based on my system. In general, my predictions have averaged around 80% over the past decade or so. (In case you're wondering, no, I do not issue predictions professionally; I do this strictly for myself, though I'm happy to share my thoughts on these matters with anyone interested (can you tell?).)

There was a dramatic exception to that in the early contests this year through the Potomac Primary. My correct prediction rate was a miserable 63%. And no, it wasn't just New Hampshire where the pollsters screwed up this year. From South Carolina through several of the Super Tuesday states and beyond, they called state after state incorrectly.

Curiously enough, starting with Wisconsin, the polls have righted themselves. What they're doing correctly now, and what they were doing wrong earlier, will almost certainly be a subject of intense study after this election is over.

In any event, my system uses each poll's margin of error as a gateway to permit entrance and triangulation. What the hell does that mean? I'll explain.

To begin with, I ALWAYS check the poll's survey dates. I never reject or accept any survey out-of-hand without first establishing its chronology relative to other surveys. And here's an example of how my system works.

Suppose I'm looking at three surveys, Poll A, B, and C, all conducted on the same dates. Suppose A has a margin of error of plus or minus 2 and shows Obama at 45% in State X. B has a margin of error of plus or minus 5 points and shows Obama at 40%. C has a margin of error of plus or minus 3 points and shows Obama at 51%. I then translate these numbers into PROBABLE SUPPORT RANGES. I change A's number to 43-47, B's number to 35-45, and C's number to 48-54.

Looking at these numbers, one can see that A's and B's support ranges overlap each other. C's, on the other hand, do not. Therefore I conclude that C was looking at the electorate using a different selection of screens than were A and B. Consequently I conclude not only that one cannot compare C directly with A and B. I also conclude that it is more likely that C is in error than that A or B is in error. I therefore ELIMINATE it from my averaging.

I then superimpose A's support range over B's support range, and eliminate the spillover. The result? I conclude that the polls are telling us that Obama's support range for that period of time when A, B, and C took their surveys was 43-45%.

At that point, I just keep collecting and collecting surveys on an ongoing and continuous basis. At some point, polls will be released which AGREE with each other in their reporting on candidates' support ranges but DISAGREE with a previous collection of polls. So I then ELIMINATE FROM MY SUPPORT RANGE TRIANGULATION all those earlier surveys, since they obviously reflect a state of play no longer in effect. This is my other major elimination mechanism to ensure that the support range averages in my current polling picture are up-to-date.

Now contrast this with Real Clear Politics' "dumb" averaging mechanism. They operate with no such elimination procedures. Outlier polls (like the hypothetical C above), some polls not taken as recently as others, etc. etc., -- all of these rogue categories are indiscriminately included in such averaging, rendering them highly suspect, just as Mr. Goddard has pointed out.

However, to conclude, I think it is also incumbent on us to realize that averaging systems can still be very useful provided one implements very strict precautions and criteria before admitting surveys into your calculations and before forming conclusions with regard to the information they purport to provide.