Learn & Understand

The Margin of Error, and the Errors It Doesn't Cover

The companion calculator determines the sample size needed for a target margin of error, the familiar "plus or minus 3 points" quoted with polls. That margin of error is genuinely useful, but it is also widely misunderstood, because it captures only one source of uncertainty, random sampling error, while ignoring other errors that are often larger and more damaging. Understanding what the margin of error really measures, the non-sampling errors it does not cover, and why the quoted margin often understates a survey's true uncertainty turns a sample-size calculation into an appreciation of the full picture of survey error. This is general educational information.

What the Margin of Error Measures

The margin of error quantifies the uncertainty in a survey estimate that comes purely from having sampled a subset of the population rather than everyone, the random sampling error. Because a survey questions only a sample, the estimate (say, a proportion favoring something) would differ somewhat if a different random sample had been drawn, and the margin of error expresses how much the estimate could vary due to this sampling randomness, at a given confidence level. A margin of plus or minus three points, for instance, means that if the sampling were repeated, the estimate would usually fall within three points of the true value, purely accounting for the luck of which people happened to be sampled. The margin of error shrinks as the sample grows (with diminishing returns), which is why the calculator relates sample size to margin of error. This is a real and important measure of one kind of uncertainty. But understanding what the margin of error measures also reveals its crucial limitation: it captures only the uncertainty from random sampling, assuming the sample is a perfect random draw and everything else about the survey is flawless. It says nothing about the many other ways a survey can be wrong, which is where the misunderstanding begins. The calculator computes sample size for a target sampling margin; understanding that this margin covers only sampling error is the first step to reading polls honestly.

The Errors It Ignores

Beyond random sampling error, surveys are subject to non-sampling errors that the margin of error does not capture and that are often larger and more consequential.

Coverage error arises when the method of reaching people systematically misses part of the population, so the sample cannot represent everyone. Nonresponse bias arises when the people who respond differ systematically from those who do not, so even a large sample is skewed if certain groups are less likely to participate. Measurement error arises from how questions are worded, ordered, or asked, which can push answers in a direction unrelated to true opinion. There are others, including errors in processing and analysis. These non-sampling errors are not random noise that averages out with a bigger sample; they are systematic biases that can shift the estimate away from the truth regardless of sample size, and crucially, the margin of error does not account for any of them. A poll can have a small margin of error yet be badly wrong because of nonresponse or coverage bias. Understanding the errors the margin of error ignores is essential: the biggest problems in surveys are often these non-sampling biases, which the quoted margin entirely omits, so a small margin of error is no guarantee of accuracy. The calculator addresses only the sampling margin; understanding non-sampling error is what reveals the many ways a survey can be wrong despite a tight margin.

Why the Quoted Margin Understates Uncertainty

Because the margin of error covers only sampling error and ignores the non-sampling biases, the quoted "plus or minus" figure systematically understates a survey's true uncertainty, sometimes badly. When a poll reports a margin of error of a few points, it is describing only the sampling uncertainty under ideal conditions, as if the sample were a perfect random draw with no coverage gaps, no nonresponse bias, and no measurement error. In reality, these other errors add uncertainty and potential bias that the quoted margin does not include, so the true range of plausible values for the estimate is wider, and possibly shifted, compared to what the margin suggests. This is why polls sometimes miss the actual outcome by more than their margin of error would imply: the miss came from non-sampling errors the margin never captured. The reported margin thus conveys a false precision, presenting the survey as more accurate than it is by accounting for only the one error source that is easy to quantify while omitting the harder-to-measure biases that often dominate. Understanding why the quoted margin understates uncertainty is a vital corrective: the familiar plus-or-minus figure is a lower bound on the uncertainty, not the whole of it, so a poll's true reliability depends on the quality of its sampling frame, response rate, and questionnaire, none of which the margin reflects. The calculator computes the sampling margin; understanding that it understates total uncertainty is what keeps you from treating the quoted margin as the full measure of a survey's accuracy.

Judging a Survey's Real Accuracy

The practical lesson is to judge a survey's accuracy by more than its margin of error, considering the quality of its whole design, since the total survey error, not just the sampling margin, determines how much to trust it. The concept of total survey error encompasses all the sources, sampling and non-sampling, so evaluating a survey means asking not only about the sample size and margin of error but about how the sample was drawn (coverage), how many responded and whether they differ from non-respondents (nonresponse), and how the questions were asked (measurement). A well-designed survey with a modest margin of error can be more accurate than a poorly designed one with a smaller margin, because good design controls the non-sampling errors the margin ignores. This is why sophisticated survey evaluation looks past the headline margin to the methodology, and why the margin of error, though a useful and honest measure of sampling uncertainty, should never be mistaken for the survey's overall accuracy. Understanding how to judge a survey's real accuracy completes the picture: the margin of error is one component of uncertainty, valuable but partial, and the true reliability of a survey depends on controlling the non-sampling errors that the margin does not capture. The calculator determines the sample size for a target sampling margin; understanding the margin of error and the non-sampling errors it ignores is what lets you read polls and surveys with appropriate skepticism, recognizing that a tight margin is necessary but far from sufficient for accuracy.

Understanding the Margin of Error

Use the calculator to size a sample for a target margin of error, and understand what that margin covers: it measures only the random sampling error, not the non-sampling biases, coverage, nonresponse, and measurement error, that are often larger and more damaging, so the quoted "plus or minus" figure understates a survey's true uncertainty. Real accuracy depends on the whole survey design, not just the margin. The calculation relates sample size to the sampling margin; understanding non-sampling error is what reveals why a small margin is no guarantee of an accurate survey.