The Good & Bad of Weighting the Data
One of the reasons polls may differ is the way the results are weighted. It is common practice to weight to demographic characteristics. Most times this reduces the sampling error. That is the point of weighting. Sometimes weighting makes things worse. Here's how it works.
Good Weighting: If people in the sample are selected with different probabilities sampling theory requires that those probabilities be used in the weights. If they are not used the results will be biased. Unequal probabilities occur in telephone polls when a person can be reached at more than one phone number. We also have unequal probabilities of selection if one person is selected from households with different numbers of adults. If you live in a one-adult household your chance of selection is 1 in 1. If you live in a three-adult household your chance of selection is 1 in 3. In both these circumstances weights must be applied to keep from distorting the results.
There is another type of weighting that may be useful. It requires two things to be true. First we must know some characteristic for all people in the population and all people in the sample. Second, that characteristic must be correlated in an election poll with the vote. For example, men and women vote differently. Gender is correlated with vote. If we weight the sample to reflect the correct proportions of men and women in the population we will improve the results. In non-election polls there must be a correlation between the weighting characteristics and the most important items being estimated.
Bad Weighting: The most common bad weighting in political polls is weighting just the likely voters for the number of Democrats, Republicans and others. Party identification is correlated with voting, but what is missing are meaningful numbers of party members for the whole population. Without that, the weighting is a guessing game rather than good theory. Some pollsters use the numbers from an exit poll from a past election, but the numbers of people who consider themselves members of a party changes from month to month and year to year.
Consider the change in party identification from the Pew Research Center polls throughout 1996. In the beginning of the year the Republican-Democrat split was 30%-30%. On Election Day it was 26%-36%. The number who considered themselves Republicans went down steadily the closer the survey was to the election. Using party identification to weight just the likely voters in a political poll is little better than a guessing game where the pollster is substituting his or her judgement for scientific method.
Another type of bad weighting for elections polls occurs when the weighting characteristic has a low correlation with key variables such as voting. For example, in some elections there may be no relationship between age and voting. If the age-voting relationship is weak then weighting for age will make the sampling error larger. Sometimes there is a strong relationship with vote but a weak relationship with some other characteristic. That other characteristic will have a larger sampling error. One also should avoid weighting that creates a large discrepancy in the weights applied to different sample cases.
The rules of weighting are simple. Always weight for the different probabilities of selecting the sample and be cautious about weighting for other things.
For more information about this and other polling issues, contact the NCPP Polling Review Board Members.