Anthony Fauci insists that the only valid test of efficacy for a pharmacological treatment is a randomized control trial (RCT). Other kinds of evidence, he claims, are merely “anecdotal”. Well-designed, large sample RCTs are highly desirable, of course, but both of Dr. Fauci’s statements are balderdash. Real world RCTs often have design flaws and drawbacks, and they often produce biased results. We certainly shouldn’t invest such confidence in their universal superiority over other clinical evidence, which for years has been relied upon in the FDA’s reviews of drugs and other interventions for safety and efficacy.

An RCT is a prospective study in which subjects are randomly assigned to one or more groups who receive different treatments, one of which is a control group receiving “standard care” or a placebo. The so-called “gold standard” of trials is the double blind RCT, which means that neither the subject nor the researchers know the treatment to which the subject is assigned.

On multiple occasions, Fauci has erroneously claimed that positive findings from anything short an RCT are “anecdotal”, which, if meaningful in any way, implies that only RCTs have samples of adequate size. That’s false: traditional clinical trials (TCTs) are not at any systematic disadvantage to RCTs in terms of sample size. The difference is that individuals are not randomly assigned to different treatment groups, but rather are assigned with the researcher’s intent, by dint of opportunity, or happenstance. These groups may include a pure control, and they may be balanced according to medical history, condition, or other potentially confounding influences. TCTs might be prospective (subjects are observed over time), or retrospective (which exploit previous case files).

The idea of double-blind, random assignment to treatment groups is appealing because it prevents researchers from exerting any bias in the selection of groups that might influence the results. That’s good, but random assignment can still lead to unbalanced comparisons, and RCTs can be flawed in many other ways. This paper discusses a number of fine points of RCTs that can lead to bias, but here are a few important ones, not all of which are covered at the link:

• The most glaring difficulty is that random assignment can result in very unbalanced characteristics across groups. The findings can be so sample-specific as to lack external validity. This is especially problematic when group sub-samples are small, as is often the case in medical research, but it is often true in samples of moderate size or even large samples. This contrasts with selecting groups with deliberate balance across key characteristics.

“Contrary to frequent claims in the applied literature, randomization does not equalize everything other than the treatment in the treatment and control groups, it does not automatically deliver a precise estimate of the average treatment effect (ATE), and it does not relieve us of the need to think about (observed or unobserved) covariates. Finding out whether an estimate was generated by chance is more difficult than commonly believed.”

• An implication of the heterogeneity across participants and random assignment of confounding attributes is that even with large treatment groups, the tests reveal differences in central tendencies, but they might not apply well to large subsets of patients. Some researchers go so far as to say all RCTs are biased in one way or another. TCT’s are also subject to bias, of course, but the point is that RCT’s are subject to significant risks of bias for reasons that TCTs often avoid.
• Comparisons of small treatment group samples results in low-powered tests that are often statistically insignificant. This weakness is shared by all RCTs and TCTs having inadequate samples to divide between the desired number of treatment groups.
• “Blindness” is often violated because treatment can involve a large number of  personnel and roles. This may influence outcomes, for example, if some caregivers alter standard treatment in an effort to compensate for its perceived deficiencies.
• Recruiting for RCTs is often difficult. This leads to the small sample problems discussed above. Sometimes participation in RCTs is heavily qualified. Sometimes patients are reluctant to participate because they don’t want to be assigned to a treatment randomly. Sometimes delays are caused by the fact that RCTs require approval by an independent review board, whereas assignment in a TCT might require only treatment decisions by different physicians.
• An RCT can be highly misleading if treatments are poorly targeted. This might take several forms: Failure to screen for conditions that might lead to treatment complications can be dangerous and counter-productive, since the general safety of the treatment might be falsely implicated. Likewise, a treatment might be effective only under certain conditions or at a certain stage of a disease, but the selection of participants might not meet those conditions. Or a treatment might be most effective in combination with other interventions, but failing to combine them will overlook the effect. Misapplications of this kind are likely to lead to erroneous conclusions.

The last bullet point has been a major bone of contention in the debate over the efficacy of hydroxychloroquine (HCQ) in the treatment of the novel coronavirus. Proponents of the drug contend it is most effective in early treatment, but a number of negative tests have studied only late treatment. Also, proponents contend that HCQ works best in combination with zinc and a Z-pak (antibiotic), but many studies have failed to use or control for those combinations.

Here are a few examples of the kinds of difficulties encountered by RCTs, as well as issues creating doubts about the results. All involve trials of HCQ .

• NIH cancels three trials: the first trial involved only hospitalized patients, though that might not have qualified as early treatment in all subjects. The other two trials were cancelled because of recruitment problems!
• A study of HCQ without zinc or Z-Pac antibiotic on hospitalized patients found that HCQ was associated with a greater likelihood of death and longer hospital stays, but in addition to the use of HCQ only, the study appears to have been mis-targeted at advanced cases of C19 infection.
• This study also endeavored to investigate HCQ as a treatment, but not only did it fail to combine HCQ with zinc and a Z-pac; over 40% of the participants never tested positive for COVID-19! It’s also not clear that participants were adequately screened for complications. The following results were statistically insignificant, indicating a possible lack of statistical power, though they favored HCQ (which is not noted by the authors):

“At 14 days, 24% (49 of 201) of participants receiving hydroxychloroquine had ongoing symptoms compared with 30% (59 of 194) receiving placebo (P = 0.21).  … With placebo, 10 hospitalizations occurred (2 non–COVID-19–related), including 1 hospitalized death. With hydroxychloroquine, 4 hospitalizations occurred plus 1 nonhospitalized death (P = 0.29).”

• This study was on a relatively small sample of non-hospitalized patients. It found only a small difference favoring HCQ in terms of viral load at day 7, as well as the following statistically insignificant results otherwise favoring HCQ:

“This treatment regimen did not reduce risk of hospitalization (7.1%, control vs. 5.9%, intervention; RR 0.75 [0.32;1.77]) nor shortened the time to complete resolution of symptoms (12 days, control vs. 10 days, intervention; p = 0.38).”

For a more comprehensive view of the evidence, this link contains a compendium of studies on HCQ 1) as a treatment at various stages of C19 infection, 2) as pre-exposure prophylaxis (PrEP) against infection; or 3) a post-exposure prophylaxis (PEP). It includes high-level details on many of the studies as well as links to most of the studies. A few of the studies are RCTs, but most are either prospective or retrospective TCTs; some are in vitro (lab) studies, and some are meta-analyses covering multiple prior studies; some address the safety of HCQ only.

The site includes a kind of “scorecard” at the top categorizing 66 of the studies as either positive (HCQ is effective) or negative within four categories: PrEP, PEP, early-stage infection, and late-stage infection. Studies were excluded from the scorecard for various reasons, including meta-analyses, in vitro studies, safety studies, those terminated due to inadequate recruitment, and studies that were deemed inconclusive due to data inadequacies and questions of interpretation awaiting feedback from authors.

The results for HCQ as a prophylactic were uniformly positive, as were the studies involving early-stage treatment. Results were mixed for late-stage treatment. Of special interest is the meta-analysis of 12 studies of high-risk outpatients by Harvey A. Risch, the seventh listed in the compendium referenced above. The 12 studies analyzed by Risch all showed that HCQ is highly effective. He calls out those who would insist that those studies be disregarded because they were not RCTs, including one critic who, like Dr. Fauci, abuses the term “anecdotal”:

“… to distinguish from the ‘magic’ of randomized controlled trials, when government medical and scientific regulatory agencies of western countries around the world routinely use epidemiologic evidence to establish facts of causation, benefit and harm. This disingenuous argument has been discussed at length elsewhere…. Finally, in pandemic times when months and years of delay cannot be tolerated before large randomized controlled trials are completed, it is possible to quibble with apparent imperfections in almost any study. That misses the forest for the trees.”

The “elsewhere” link in the quote above includes an excellent summary of the battle waged over the efficacy of HCQ. It became a media war, which relied in part on the false assertion that only RCTs are acceptable. That was abetted by certain public health experts and researchers who might have had financial or political interests in promoting new drugs, rather than the safe, cheap alternative that had been used safely for many decades. The article notes that few media sources carried the following, which was released only days after the FDA revoked its Emergency Use Authorization for HCQ (based on faulty evidence):

“TUCSON, Ariz., June 22, 2020 /PRNewswire/ — Today the Association of American Physicians & Surgeons files its motion for a preliminary injunction to compel release to the public of hydroxychloroquine by the Food & Drug Administration (FDA) and the Department of Health & Human Services (HHS), in AAPS v. HHS, No. 1:20-cv-00493-RJJ-SJB (W.D. Mich.). Nearly 100 million doses of hydroxychloroquine (HCQ) were donated to these agencies, and yet they have not released virtually any of it to the public…

‘Why does the government continue to withhold more than 60 million doses of HCQ from the public?’ asks Jane Orient, M.D., the Executive Director of AAPS. ‘This potentially life-saving medication is wasting away in government warehouses while Americans are dying from COVID-19.'”

Positive COVID-19 tests continue to mount, which is scary, but the more I learn about the processes generating the data, the more skeptically I regard the numbers. And whether the data is junk or otherwise, it’s often misinterpreted or misused by the media. Here I’ll focus mainly on issues related to testing and cause-of-death. What’s striking is the likelihood of upward bias in the reported case counts based on one set of tests, even while the incidence of antibodies to the virus appear to be more widespread.

Testing

Almost all of the C19 test results included in the case counts we’ve seen are from polymerase chain reaction (PCR) tests, the kind involving samples collected with “brain scrambling” nasal swabs. These tests detect whether the subject is shedding any viral particles. The other kind of test is for antibodies, called a serological test, which focuses on whether a subject has HAD the virus, not on whether they have it currently. The latter test, however, might catch some active cases in addition to resolved infections.

The first problem is that some states have combined results from these two kinds of tests. That’s likely to inflate the case count because they would capture those who are infected, and those who were infected but aren’t any longer.

A second problem is the faulty reporting of test results we’ve seen in states like Florida, where some labs have been reporting an implausible 100% positivity rate over certain periods. That might or might not imply an exaggerated count of positives, but it certainly inflates the positivity rate. There are other practices that systematically inflate the positive test count, however, such as counting all members of a household as “probable positives”, and counting multiple positive tests on the same patient as multiple cases.

Test reliability

This is the third problem and it’s really a biggie. It’s also more complicated because there is more than one kind of accuracy on which tests are evaluated.

1) The PCR tests are said to have a sensitivity of anywhere from 66%-80%, depending on testing and lab conditions. That means about one of every three or four tests on infected people will miss the actual infection: that’s a false negative and a horrible mistake. An article in The New England Journal of Medicine puts sensitivity at 70%. These levels of sensitivity are poor, so there is good reason for repeat testing, or to develop and implement more sensitive tests!

2) The other kind of accuracy is called specificity. It indicates the percentage of uninfected subjects who actually test negative. If it’s 90%, then one out of every ten tests identifies an infection that really isn’t there. That’s a false positive. It’s extremely hard to find estimates of specificity for PCR tests outside of perfect lab conditions. We know there are false positives in the real world, however, and I’ll get to that evidence below. But we know, for example, that individuals will continue to shed virus for a short time even after the virus is dead, and that reduces specificity. False positives can also result from poor testing or lab conditions.

So here’s an example: let’s be generous and assume that test sensitivity is 80%, and we’ll give the benefit of the doubt to test specificity and say it’s 95%. Further suppose that 2% of the population is currently infected. Out of 1,000 tests, 20 involve infected subjects. The sensitivity implies that we’ll correctly identify 16 of them (80%) and we’ll miss four. The other 980 tests subjects are virus-free, but 95% specificity implies that about 49 of those tests will come back positive (49/980 = 5%). All together, that yields a whopping positivity rate of:

(49 + 16)/1000, or 6.5%, well above the true infection rate of 2%.

So it’s very easy for a test having inadequate specificity to inflate the number of positives. That’s less problematic when prevalence is high, since fewer virus-free subjects are available to misidentify. Unfortunately, it becomes a larger concern when testing is broad and less focused on symptoms, since that implies lower prevalence in the tested population. The U.S. has increased testing over the past two months, roughly quintupling the number of daily tests over a span of three months. The tested population has therefore broadened to include many more subjects who are either asymptomatic, freaked out about their allergy symptoms, or have been routinely tested on admittance to hospitals for other illnesses or procedures.

Discussion

It’s absolutely necessary for society to have testing capacity for those with symptoms and those likely to be exposed to the virus, such as first responders. But rolling out the test to the broader population means the case data are much less accurate unless positive diagnoses are based on repeated tests. Unfortunately, the bulk of the testing we’ve seen thus far has been so lacking in specificity as to inflate the number of cases as testing became more widespread.

Evidence for this claim is offered by a paper just published by a Connecticut epidemiologist. He used a more robust technique to re-examine ten positive and ten negative tests provided by the CT Department of Public Health. He found that nine of the 20 cases were true infections, but two of those came from the ten negative tests! So, in fact, there were three false positives and two false negatives among the 20 tests. Therefore, the tests overestimated the number of actual cases by around 11% in the sample, net of both kinds of errors. Granted, this was a small sample, and we don’t know the true prevalence of the virus in the full population of test subjects, but if we assume the positive tests and negative tests were representative, a prevalence of 5% would imply, after weighting, a rather drastic inflation of the positivity rate to 32.5%!

0.95(3/10) + 0.05(8/10)

That’s just outrageous!

The U.S. positivity rate by the end of April was about 12%, when testing was still limited; it’s been running at about 8% recently. The decline almost certainly reflects both a broadening of the test population and a decline in prevalence among the tested population. That, in turn, implies that a positive test has less predictive value, for even though the test captures the same percentage of true positives, a larger percentage of all positive tests will be false negatives. It might seem paradoxical, but it’s likely that the 12% positivity rate early in the pandemic had a smaller upward bias than the 8% we see currently, but that is due to the composition of the population tested. Under current testing, the specificity percentage is applied to a larger proportion of uninfected subjects, so the number of false positives overwhelms the test’s ability to identify true positives.

The first priority of testing is to reliably identify true cases. The current PCR tests fail in that objective due to low sensitivity. But inspecific test are costly too. First, they waste medical resources on uninflected subjects. Second, a major set of worries and inconveniences are imposed on false positives, which have a real cost. Third, inspecific tests can be costly because of the even higher likelihood of false positives over several rounds of tests. For example, it will be extremely difficult for sports teams to establish continuity, or even to maintain a full roster, because so many players are likely to become victims of false-positivity under repeated testing.

Death tolls

Anything that inflates the C19 case count tends to inflate C19-attributed deaths. For example, almost all hospital admissions are now tested. A high number of false positives leads to more deaths being wrongly attributed to C19. Other issues related to counting deaths go beyond the vagaries of test accuracy. Hospitals have a perverse incentive to boost their C19 cases and deaths via more generous Medicare reimbursements. Deaths are also attributed to C19 in a variety of other circumstances, some quite suspicious, but we are constantly told without evidence that C19 deaths are undercounted and so these additions must be reasonable.

The argument that deaths of C19 patients with comorbidities are rightfully attributed to C19 is likewise flawed for some of the reasons discussed above. False positives are all too common. Furthermore, patients might be admitted to a hospital with advanced or terminal conditions and die having caught C19 coincidentally at the hospital. And one can certainly quibble with the notion that the deaths of otherwise terminal patients should be attributed to C19. There is a significant grey area.

Finally, as I discussed in a previous post, the deaths reported each week are at odds with the actual timing of those deaths. There are occasionally large additions to the CDC’s provisional deaths counts many weeks in the past. It’s bad enough that those deaths are reported so late and treated by the media as if they just occurred. Possibly worse is the potential for manipulating death counts for political purposes, which is enabled by the large backlog of deaths lacking attributed causes over the course of weeks and months.

Serological tests and false positives

The first serological tests for C19 antibodies, back in April, yielded surprisingly high estimates of individuals with acquired immunity to the virus, often 10 or more times the number of infections based on case counts (also see here and here). The earliest antibody test results were criticized because their specificity and the prevalence of antibodies in the general population were thought to be low. That made it relatively easy for critics to rationalize the high estimates as a consequence of false positives. We now know, however, that serological tests have higher specificity than the PCR tests for active infections, and those tests have consistently shown a larger than expected share of individuals having acquired immunity. But how does that square with the argument that case counts based on PCR tests are inflated? How can so many have developed antibodies if the case counts are so exaggerated?

To rephrase: how can the population with antibodies, those who have HAD the virus, accumulate to a level several times the case count? Keep in mind that a high proportion of the serological tests have been conducted in relative hot spots, where there are likely to be many undetected cases. There is also some question about the real timing of the pandemic in the U.S. Some believe it was spreading prior to March, so the true number of cases, diagnosed and undiagnosed, might have mounted more quickly early in the pandemic than later case diagnoses suggested. Moreover, serological testing has not been conducted on a random sample of the population. In fact, those tests are more often administered when patients go to labs for other blood work, so there is reason to believe that prevalence in this group might exceed that of the general population. It’s also possible that the serological tests are picking up antibodies developed in response to other forms of the coronavirus, which might in fact be protective. Finally, the serological tests are still subject to a level of false positives. So the antibody findings from serological tests are not necessarily inconsistent with the notion that case counts and death counts are inflated now.

Summary

We truly need better, quicker tests, and many talented people are now working to improve them. My point is not to degrade the effort to conduct testing, but to note that our current testing regime has many flaws, one of which is to raise alarm about extremely high case and death counts. I do not doubt that the number of actual infections has grown in June and July. However, the positivity rate remains lower than early in the pandemic with a much larger, less focused selection of test subjects. Many of the cases identified by PCR tests are false positives. As disappointing as it is to someone who loves to work with data, C19 case counts and mortality look unreliable.