The paradox of big data spoils vaccination surveys

 

https://www.eurekalert.org/news-releases/937034

 

 News Release 8-ec-2021
Researchers analyze where COVID vaccination surveys went wrong
Peer-Reviewed Publication

Harvard University

When Delphi-Facebook and the U.S. Census Bureau provided near-real time estimates of COVID-19 vaccine uptake last spring, their weekly surveys drew on responses from as many as 250,000 people.

The large data sets provided statistically tiny margins of error, a key measure of a poll’s accuracy, and raised confidence that the numbers were correct. But when the Centers for Disease Control and Prevention later provided figures of actual reported vaccination rates, the two polls were off — by a lot. By the end of May, the Delphi-Facebook study overestimated vaccine uptake by 17 percentage points — 70 percent versus 53 percent, according to the CDC — and the Census Bureau’s Household Pulse Survey did the same by 14 percentage points.

A comparative analysis by statisticians and political scientists from Harvard, Oxford, and Stanford universities concludes that the surveys fell victim to the “Big Data Paradox,” the mathematical tendency of big data sets to minimize one type of error -- that due to small sample size – but to magnify another that tends to get lesser attention: errors due to systematic biases that make the surveyed sample a poor representation of the larger population.

-----

“This is the Big Data Paradox: the larger the data size, the surer we fool ourselves when we fail to account for bias in data collection,” the paper’s authors wrote in their analysis, published Dec. 8 in the journal Nature.

Those misleading results can be particularly harmful when actions are taken based on them, the authors point out. The governor of a state where a survey shows that 70 percent are vaccinated against COVID, for example, might relax public health measures. If actual vaccination rates are closer to 55 percent, instead of fostering a return to normal life, the step could result in a spike in cases and a rise in COVID deaths.

-----