What readers’ comments tell us about the Stanford study estimating the seroprevalence of SARS-COV-2 in a California population
What readers’ comments tell us
A data-driven rejection of herd immunity
A study by Stanford researchers was released on April 14, 2020 on MedRxiv, a preprint website for the health sciences. The study¹ was designed to estimate the number of persons in Santa Clara County, California who were previously infected with SARS-COV-2, the coronavirus that causes the disease COVID-19. The county had imposed strict mitigation measures early in the COVID-19 epidemic. Knowing the number of infections in the population might reveal how successful the interventions were or, conversely, how close to achieving “herd immunity” might the County residents be. The number of infections would also provide a base from which to estimate the chance of dying after becoming infected. That chance, defined as the number of deaths divided by the number of infections, is called the “infection fatality rate” and is important for public health planning and hospital readiness.
The subjects of the study were 3,330 persons recruited through Facebook ads and tested over a two-day period, April 3–4, for evidence of past infection. Blood samples were drawn at a drive-through testing site and then analyzed for the presence of antibodies to SARS-COV-2.²
As soon as the study was published in its preprint form, comments began to flow in questioning the specificity of the antibody test. According to the manufacturers,³ the test has a small false positive rate of 1 in 186. While this rate seems unimportant upon first inspection, low rates of false positives can lead to substantial overestimation if there is also a low rate of the disease in the population. With a 1 in 186 chance of a false positive identification, 18 false positives would be expected among 3,330 samples of blood (~ 186/3,330=18), or, as it turns out, as much as one third of the 50 seropositive cases identified in the study. False positive identifications can occur when other coronaviruses, like the one that causes about 15% of common colds, are accidentally picked up, or for other reasons.
In addition to questioning the specificity of the antibody test, many readers questioned the appropriateness of the self-selected group of study participants. Who would come out during a lockdown in the middle of an epidemic, based on an ad in Facebook? one reader posed. The answer: persons with prior COVID-19 symptoms who want confirmation that they had the disease. This potential selection bias would lead to over representation of the very thing under investigation, persons with antibodies to the SARS-COV-2 virus.
In reading through the comments, I found the majority to be well-thought-out and likely written by persons with statistical sophistication and medical knowledge of the caliber of the researchers themselves. It occurred to me that the preprint article was undergoing peer review right in the public sphere.
The authors estimated that the number of people with prior SARS-COV-2 infections in Santa Clara County at the beginning of April was at least 48,000 individuals. According to many readers’ comments, even this lower estimate was likely to be too large.
I went ahead and plugged in 48,000 as the denominator into the equation for the infection fatality rate. What would be the numerator? The numerator would be the number of deaths occurring in that cohort of 48,000 infected people.
The authors estimated that the number of people with prior SARS-COV-2 infections in Santa Clara County at the beginning of April was at least 48,000 individuals. According to many readers’ comments, even this lower estimate was likely to be too large.
The authors estimated the numerator to be 100 (by April 23), based on death certificate reporting up until the time of the study’s publication, adjusted for two assumptions (because some deaths belonging to the cohort would not yet have occurred). The assumptions were: 1) the average time from infection to death is under 3 weeks, and 2) the growth in the number of deaths stays at 6% per day.
After calculating the infection fatality rate as somewhere between 0.12% and 0.20%, the authors say,
“ … if the average duration from case identification to death is less than 3 weeks, or if the epidemic wave has peaked and growth in deaths is less than 6% daily, then the infection fatality rate would be lower.”
But, what if the opposite were true? Suppose, for example, the average duration to death (or more likely, to reporting of death) is more than 3 weeks. If this opposite assumption were true, then the infection fatality rate would be higher, based on the numerator alone.
With a possibly incomplete numerator and a likely exaggerated denominator, the study probably underestimates the infection fatality rate by a substantial, if unknown, degree.
The authors say,
“The most important implication of these findings is that the number of infections is much greater than the reported number of cases.”
This strikes me as somewhat anticlimactic because underreporting of COVID-19 cases is well-known, due to its often mild presentation and a lack of testing for the disease. How much greater the true number is relative to the reported number has yet to be confirmed.
I venture that the more important implication of this study is that we are nowhere near herd immunity in Santa Clara County, even accepting the authors’ estimate of 2–4% seropositivity to SARS-COV-2 in the population. If the true prevalence is a fraction of this amount, or even multiples of it, we are very far from herd immunity. A strong, data-driven rejection of herd immunity is what makes this study important and timely. We can conclude that mitigation efforts in Santa Clara County are working — but also that the vast majority of us are still at risk.
A strong, data-driven rejection of herd immunity is what makes this study important and timely. We can conclude that mitigation efforts in Santa Clara County are working — but also that the vast majority of us are still at risk.
Regarding the infection fatality rate, I’m trying to square this study’s findings with those of closed populations, where everyone was tested and followed up. Studies from closed populations show much higher infection fatality rates.⁴ This includes the well-known case of the Diamond Princess cruise ship (mentioned in several comments) whose infection fatality rate (age-adjusted to the US population) is on the order of 6–8 per thousand, as opposed to the current study’s rate on the order of one-to-two per thousand.
Perhaps the initial inoculum of the virus on the ship was large, due to multiple exposures in close quarters coupled with the physical and emotional stress of being confined on a boat in the middle of an epidemic. Perhaps there were more smokers on the boat compared to other populations. These might account for a poorer outcome. But 3–8 times poorer relative to Santa Clara County is surprising to me.
The saga of the Diamond Princess is not over. According to Wikipedia,⁵ as recently as April 14th, the day of the release of the Stanford study, and three months after the epidemic started on the boat, the 14th person from the Diamond Princess succumbed to the disease. Infections on the boat are presumed to have occurred mostly during the latter part of January, 2020. During the first month of follow-up, 7 persons died (out of 3,711 onboard with 712 testing positive for SARS-COV-2). The first four persons were in their 80s and the later three in their 70s. Over the second month of follow-up (and until April 14) another 7 persons died. These persons were in their 60s and 70s (with 2 ages unreported).
It is interesting to note that half the passengers were citizens of Japan, a country with one of the healthiest elders in the world. One would expect a lower disease and fatality response (not higher) in this population relative to that of Santa Clara County, which includes elders too compromised to consider taking a two-week vacation on a boat in the middle of winter.
I would like to know what the authors think about observations from closed systems where everyone was tested. Surely the researchers have puzzled over these observations compared to their findings in the northern California County of Santa Clara.
(Many of the points I make here in this article are also discussed in a Nature article⁶ published contemporaneously.)
* * * * * * * *
- COVID-19 Antibody Seroprevalence in Santa Clara County, California Eran Bendavid, Bianca Mulaney, Neeraj Sood, Soleil Shah, Emilia Ling, Rebecca Bromley-Dulfano, Cara Lai, Zoe Weissberg, Rodrigo Saavedra, James Tedrow, Dona Tversky, Andrew Bogan, Thomas Kupiec, Daniel Eichner, Ribhav Gupta, John Ioannidis, Jay Bhattacharya medRxiv 2020.04.14.20062463; doi: https://doi.org/10.1101/2020.04.14.20062463
- Premier Biotech https://premierbiotech.com/innovation/rapid-testing/
- Neutralizing antibody responses to SARS-CoV-2 in a COVID-19 recovered patient cohort and their implication Fan Wu, Aojie Wang, Mei Liu, Qimin Wang, Jun Chen, Shuai Xia, Yun Ling, Yuling Zhang, Jingna Xun, Lu Lu, Shibo Jiang, Hongzhou Lu, Yumei Wen, Jinghe Huang medRxiv 2020.03.30.20047365; doi: https://doi.org/10.1101/2020.03.30.20047365
- Public Health Responses to COVID-19 Outbreaks on Cruise Ships — Worldwide, February–March 2020 https://tools.cdc.gov/medialibrary/index.aspx#/media/id/405651 2020 coronavirus pandemic on cruise ships
- Wikipedia: (Diamond Princess) 2020 coronavirus pandemic on cruise ships https://en.wikipedia.org/wiki/2020_coronavirus_pandemic_on_cruise_ships
- Antibody tests suggest that coronavirus infections vastly exceed official counts: Study estimates a more than 50-fold increase in coronavirus infections compared to official cases, but experts have raised concerns about the reliability of antibody kits Smriti Mallapaty doi: 10.1038/d41586–020–01095–0 https://www.nature.com/articles/d41586-020-01095-0
