Pfizer CEO Albert Bourla’s recent open letter on the potential timing of his company’s COVID-19 vaccine candidate provides clarity where earlier unscripted comments sowed confusion. His previous statements to the media led some to believe he would push for approval of his company’s vaccine in October. New guidance on the safety data required by regulators has ruled out that timeline. In other words, there isn’t going to be a COVID-19 vaccine available to the public before the November election, as President Trump had hoped.
But that does not mean October won’t bring a vaccine surprise. As Bourla explains in his letter, event data from the vaccine’s Phase III trial might accumulate to a sufficient level to allow for an early reading on whether or not it works. Even without regulatory approval, an announcement this month by Pfizer and its German partner, BioNTech, that their vaccine candidate works well enough in preventing COVID-19 to satisfy efficacy requirements would be major international news (Bourla says he feels obligated to announce the results of early looks at efficacy data soon after receiving them). It would then be a matter of weeks before sufficient time passed before regulators could reach some conclusions on the vaccine’s safety profile too.
Phase III trials for COVID-19 vaccines involve giving the vaccine, or a placebo, to tens of thousands of participants. The Pfizer-BioNTech trial is enrolling 44,000 volunteers in testing sites around the world. A large sample is needed to increase the chances that serious but likely rare adverse reactions are detected before giving the vaccines to millions of people.
Large trial enrollment also is needed to ascertain vaccine efficacy. Even in this pandemic, the probability of becoming sick with COVID-19 is relatively low. To see if a vaccine works, a test has to turn up sufficient numbers of people who develop the targeted disease. When the chances of getting sick are 1 percent or less over a given time period (such as two to three months), the trial needs the participation of tens of thousands of volunteers to ensure statistically valid results will emerge within a reasonable timeframe.
Both the U.S. Food and Drug Administration (FDA) and the World Health Organization (WHO) have established a minimum efficacy standard of 50 percent for COVID-19 vaccines, along with a 97.5 percent probability that they will work to prevent disease at least 30 percent of the time. Vaccine sponsors must prove, through the data collected in their trials, that their candidates meet both of these requirements.
The Pfizer-BioNTech Phase III trial protocol calls for looking at the number of cases that occur in the vaccine and placebo arms when the total case count reaches 32, 62, 92, and 120. The final required case count is 164. The sponsors of the trials do not know in advance who among the volunteers received the vaccine or the placebo. After the 32nd case of COVID-19 emerges among the trial’s participants, independent monitors will look at the data to see how many of those who got sick received the vaccine versus those who received the placebo. If there is a sufficiently wide difference (the “efficacy rate”) between disease emergence in the vaccine and placebo groups, then the vaccine can be deemed efficacious.
Importantly, statistical tests published in the Pfizer-BioNTech Phase III trial protocol are more stringent than the WHO’s recommended criteria. For example, in Pfizer-BioNTech’s final analysis, which occurs after 164 positive cases are observed, the observed efficacy rate required for the trial to be a success is 52.3 percent, not 50 percent. While this may sound like a small difference, if the vaccine’s true efficacy in the entire population were 55 percent, the probability of a successful trial under Pfizer-BioNTech’s protocol is 9 percentage points lower than the probability of success under the WHO’s recommended criteria.
At just 32 total cases, the Pfizer-BioNTech vaccine would satisfy their trial protocols if at least 26 of the volunteers who get sick with COVID-19 received the placebo and 6 (or fewer) received the vaccine. The observed efficacy in that scenario would be at least 76.9 percent, and, though the case count would be very small, the probability that the actual efficacy was below 30 percent would be far less than 2.5 percent.
At 62 cases, the observed efficacy of the vaccine would need to be at least 68.l percent to be deemed effective under the trial protocol, which means that, of the 62 volunteers who become sick, 15 or fewer would need to be among those who received the vaccine. At each of the pre-determined review intervals, the required observed efficacy of the vaccine would fall until it reached just 52.3 percent at 164 cases.
Bourla’s repeated references to a potential October signal on efficacy is, in part, an indication in his belief that the trial sites are located in areas where the virus is prevalent. With 44,000 trial participants (when fully enrolled), 32 cases of COVID-19 disease would almost certainly emerge before two months, if Pfizer-BioNTech’s assumptions about true vaccine efficacy and the rate at which nonvaccinated trial participants contract the disease are correct.
It also shows he has high confidence in the actual efficacy of his company’s vaccine candidate. The odds of a vaccine meeting the Pfizer-BioNTech’s criteria at 32 cases if the actual efficacy of the vaccine is 70 percent is less than 2 in 5. If the actual efficacy is only 60 percent, the odds drop to 15 in 100. With more case counts, the odds improve for meeting the efficacy criteria. If the actual efficacy of a vaccine is 70 percent, then the odds of meeting the trial protocol's efficacy target at 62 cases is around 2 in 3, and it rises to 9 in 10 at 92 cases.
While it is possible that a signal of the Pfizer-BioNTech vaccine’s efficacy may come this month, Bourla also states in his letter that the companies will continue the trial through to its full 164-case end point, regardless of what the reviews indicate at the earlier case count intervals. In the current environment, with so much concern about the government cutting corners to approve a vaccine, that is wise. Because even though it may be possible to reach valid statistical conclusions when just 32 cases of COVID-19 disease emerge, the public is likely to want more convincing evidence, with greater numbers of cases to observe. Given the prevalence of the virus throughout the U.S., it seems likely that another few weeks of observations will provide even more conclusive evidence on whether the vaccine works, or not.
James C. Capretta is a resident fellow and holds the Milton Friedman chair at the American Enterprise Institute. Scott Ganz is a research fellow in economic policy studies at AEI and an assistant professor at Georgia Tech’s School of Public Policy. They are the authors of “Awaiting the Signal: Assessing the Efficacy of COVID-19 Vaccines,” published this month by AEI.