A margin of error that isn't quite true

Every published poll carries a margin of error. It's the number that lets a headline writer say a candidate is "within the margin" or "clearly ahead." But that number is almost always calculated as if the sample were a simple random sample — as if every respondent had an equal, independent chance of being selected and answering.

In practice, almost no modern poll works that way. Panels are stratified. Quotas are set by region, age, and gender. Non-response is corrected with weights. Some respondents are deliberately oversampled to get enough cases in a hard-to-reach subgroup. All of this is good, necessary methodology — but it has a side effect, literally: it changes how much statistical information each respondent actually contributes. The textbook margin of error ignores this. The design effect does not.

What design effect actually measures

Design effect (often written Deff) is the ratio between the variance of an estimate under the actual, complex sample design and the variance you would get from a simple random sample of the same size.

A simplified version, commonly attributed to Leslie Kish, applies specifically to the variance inflation caused by weighting:

Deff ≈ 1 + CV²(w)

where CV(w) is the coefficient of variation of the survey weights — essentially, how unequal the weights are across respondents.

From Deff you get the effective sample size:

n_eff = n / Deff

This is the number that actually matters. A survey of 1,000 respondents with a Deff of 1.5 behaves, statistically, like a simple random sample of about 667. The real margin of error is wider than the one printed at the bottom of the topline — sometimes meaningfully so.

Why this happens in almost every real poll

Design effect isn't a flaw in a particular study; it's a near-unavoidable consequence of doing rigorous survey research:

  • Weighting for representativeness. Correcting a sample to match census or voter-file targets on age, gender, region, education, or past vote almost always produces unequal weights, and unequal weights inflate variance.
  • Oversampling subgroups. Boosting a small population — a province, an age cohort, an ethnic group — so it can be analyzed on its own increases precision for that subgroup but adds variance to the overall estimate once weights are rebalanced.
  • Clustering. Multi-stage designs (sampling points, then households, then individuals) mean respondents within a cluster are more similar to each other than to the population at large, which also inflates variance relative to a simple random sample.
  • Panel and online non-probability designs. These frequently require heavier weighting to correct for who opts in, which tends to push Deff higher than in well-executed probability samples.

None of this means the poll is bad. It means the poll's true precision is different from — usually lower than — what a naive ±3% suggests.

Why most firms don't tell you

Reporting design effect, effective sample size, or a weighting-adjusted margin of error takes extra work: it has to be calculated, checked, and explained in plain language to a non-technical audience. Professional standards bodies, including AAPOR, recommend disclosing it. In practice, most public polling — and a good share of private, client-commissioned research — simply reports the unweighted N and a textbook margin of error, because it's simpler to produce and easier to sell. The effect is a margin of error that looks tighter, and more authoritative, than the data actually support.

Why McAllister reports it

We treat design effect as a standard line item, not an optional footnote. On every study where weighting is applied, we report:

  • the unweighted and weighted sample sizes,
  • the design effect and effective sample size,
  • and a margin of error adjusted for design effect, alongside the conventional one — so clients can see both and understand the difference.

This isn't about undermining confidence in survey research. It's the opposite: methodologically transparent numbers hold up under scrutiny, age well when compared against election results or later benchmarks, and give clients an honest basis for risk assessment before they act on a finding. A result that's "significant" under a naive margin of error but not under the design-effect-adjusted one is exactly the kind of distinction that should inform a strategic decision — and it's exactly the kind of distinction that gets lost when firms only report the simpler number.

Good survey research doesn't just produce a topline. It tells you how much to trust it, and why.