Anthony Fauci insists that the only valid test of efficacy for a pharmacological treatment is a randomized control trial (RCT). Other kinds of evidence, he claims, are merely “anecdotal”. Well-designed, large sample RCTs are highly desirable, of course, but both of Dr. Fauci’s statements are balderdash. Real world RCTs often have design flaws and drawbacks, and they often produce biased results. We certainly shouldn’t invest such confidence in their universal superiority over other clinical evidence, which for years has been relied upon in the FDA’s reviews of drugs and other interventions for safety and efficacy.

An RCT is a prospective study in which subjects are randomly assigned to one or more groups who receive different treatments, one of which is a control group receiving “standard care” or a placebo. The so-called “gold standard” of trials is the double blind RCT, which means that neither the subject nor the researchers know the treatment to which the subject is assigned.

On multiple occasions, Fauci has erroneously claimed that positive findings from anything short an RCT are “anecdotal”, which, if meaningful in any way, implies that only RCTs have samples of adequate size. That’s false: traditional clinical trials (TCTs) are not at any systematic disadvantage to RCTs in terms of sample size. The difference is that individuals are not randomly assigned to different treatment groups, but rather are assigned with the researcher’s intent, by dint of opportunity, or happenstance. These groups may include a pure control, and they may be balanced according to medical history, condition, or other potentially confounding influences. TCTs might be prospective (subjects are observed over time), or retrospective (which exploit previous case files).

The idea of double-blind, random assignment to treatment groups is appealing because it prevents researchers from exerting any bias in the selection of groups that might influence the results. That’s good, but random assignment can still lead to unbalanced comparisons, and RCTs can be flawed in many other ways. This paper discusses a number of fine points of RCTs that can lead to bias, but here are a few important ones, not all of which are covered at the link:

• The most glaring difficulty is that random assignment can result in very unbalanced characteristics across groups. The findings can be so sample-specific as to lack external validity. This is especially problematic when group sub-samples are small, as is often the case in medical research, but it is often true in samples of moderate size or even large samples. This contrasts with selecting groups with deliberate balance across key characteristics.

“Contrary to frequent claims in the applied literature, randomization does not equalize everything other than the treatment in the treatment and control groups, it does not automatically deliver a precise estimate of the average treatment effect (ATE), and it does not relieve us of the need to think about (observed or unobserved) covariates. Finding out whether an estimate was generated by chance is more difficult than commonly believed.”

• An implication of the heterogeneity across participants and random assignment of confounding attributes is that even with large treatment groups, the tests reveal differences in central tendencies, but they might not apply well to large subsets of patients. Some researchers go so far as to say all RCTs are biased in one way or another. TCT’s are also subject to bias, of course, but the point is that RCT’s are subject to significant risks of bias for reasons that TCTs often avoid.
• Comparisons of small treatment group samples results in low-powered tests that are often statistically insignificant. This weakness is shared by all RCTs and TCTs having inadequate samples to divide between the desired number of treatment groups.
• “Blindness” is often violated because treatment can involve a large number of  personnel and roles. This may influence outcomes, for example, if some caregivers alter standard treatment in an effort to compensate for its perceived deficiencies.
• Recruiting for RCTs is often difficult. This leads to the small sample problems discussed above. Sometimes participation in RCTs is heavily qualified. Sometimes patients are reluctant to participate because they don’t want to be assigned to a treatment randomly. Sometimes delays are caused by the fact that RCTs require approval by an independent review board, whereas assignment in a TCT might require only treatment decisions by different physicians.
• An RCT can be highly misleading if treatments are poorly targeted. This might take several forms: Failure to screen for conditions that might lead to treatment complications can be dangerous and counter-productive, since the general safety of the treatment might be falsely implicated. Likewise, a treatment might be effective only under certain conditions or at a certain stage of a disease, but the selection of participants might not meet those conditions. Or a treatment might be most effective in combination with other interventions, but failing to combine them will overlook the effect. Misapplications of this kind are likely to lead to erroneous conclusions.

The last bullet point has been a major bone of contention in the debate over the efficacy of hydroxychloroquine (HCQ) in the treatment of the novel coronavirus. Proponents of the drug contend it is most effective in early treatment, but a number of negative tests have studied only late treatment. Also, proponents contend that HCQ works best in combination with zinc and a Z-pak (antibiotic), but many studies have failed to use or control for those combinations.

Here are a few examples of the kinds of difficulties encountered by RCTs, as well as issues creating doubts about the results. All involve trials of HCQ .

• NIH cancels three trials: the first trial involved only hospitalized patients, though that might not have qualified as early treatment in all subjects. The other two trials were cancelled because of recruitment problems!
• A study of HCQ without zinc or Z-Pac antibiotic on hospitalized patients found that HCQ was associated with a greater likelihood of death and longer hospital stays, but in addition to the use of HCQ only, the study appears to have been mis-targeted at advanced cases of C19 infection.
• This study also endeavored to investigate HCQ as a treatment, but not only did it fail to combine HCQ with zinc and a Z-pac; over 40% of the participants never tested positive for COVID-19! It’s also not clear that participants were adequately screened for complications. The following results were statistically insignificant, indicating a possible lack of statistical power, though they favored HCQ (which is not noted by the authors):

“At 14 days, 24% (49 of 201) of participants receiving hydroxychloroquine had ongoing symptoms compared with 30% (59 of 194) receiving placebo (P = 0.21).  … With placebo, 10 hospitalizations occurred (2 non–COVID-19–related), including 1 hospitalized death. With hydroxychloroquine, 4 hospitalizations occurred plus 1 nonhospitalized death (P = 0.29).”

• This study was on a relatively small sample of non-hospitalized patients. It found only a small difference favoring HCQ in terms of viral load at day 7, as well as the following statistically insignificant results otherwise favoring HCQ:

“This treatment regimen did not reduce risk of hospitalization (7.1%, control vs. 5.9%, intervention; RR 0.75 [0.32;1.77]) nor shortened the time to complete resolution of symptoms (12 days, control vs. 10 days, intervention; p = 0.38).”

For a more comprehensive view of the evidence, this link contains a compendium of studies on HCQ 1) as a treatment at various stages of C19 infection, 2) as pre-exposure prophylaxis (PrEP) against infection; or 3) a post-exposure prophylaxis (PEP). It includes high-level details on many of the studies as well as links to most of the studies. A few of the studies are RCTs, but most are either prospective or retrospective TCTs; some are in vitro (lab) studies, and some are meta-analyses covering multiple prior studies; some address the safety of HCQ only.

The site includes a kind of “scorecard” at the top categorizing 66 of the studies as either positive (HCQ is effective) or negative within four categories: PrEP, PEP, early-stage infection, and late-stage infection. Studies were excluded from the scorecard for various reasons, including meta-analyses, in vitro studies, safety studies, those terminated due to inadequate recruitment, and studies that were deemed inconclusive due to data inadequacies and questions of interpretation awaiting feedback from authors.

The results for HCQ as a prophylactic were uniformly positive, as were the studies involving early-stage treatment. Results were mixed for late-stage treatment. Of special interest is the meta-analysis of 12 studies of high-risk outpatients by Harvey A. Risch, the seventh listed in the compendium referenced above. The 12 studies analyzed by Risch all showed that HCQ is highly effective. He calls out those who would insist that those studies be disregarded because they were not RCTs, including one critic who, like Dr. Fauci, abuses the term “anecdotal”:

“… to distinguish from the ‘magic’ of randomized controlled trials, when government medical and scientific regulatory agencies of western countries around the world routinely use epidemiologic evidence to establish facts of causation, benefit and harm. This disingenuous argument has been discussed at length elsewhere…. Finally, in pandemic times when months and years of delay cannot be tolerated before large randomized controlled trials are completed, it is possible to quibble with apparent imperfections in almost any study. That misses the forest for the trees.”

The “elsewhere” link in the quote above includes an excellent summary of the battle waged over the efficacy of HCQ. It became a media war, which relied in part on the false assertion that only RCTs are acceptable. That was abetted by certain public health experts and researchers who might have had financial or political interests in promoting new drugs, rather than the safe, cheap alternative that had been used safely for many decades. The article notes that few media sources carried the following, which was released only days after the FDA revoked its Emergency Use Authorization for HCQ (based on faulty evidence):

“TUCSON, Ariz., June 22, 2020 /PRNewswire/ — Today the Association of American Physicians & Surgeons files its motion for a preliminary injunction to compel release to the public of hydroxychloroquine by the Food & Drug Administration (FDA) and the Department of Health & Human Services (HHS), in AAPS v. HHS, No. 1:20-cv-00493-RJJ-SJB (W.D. Mich.). Nearly 100 million doses of hydroxychloroquine (HCQ) were donated to these agencies, and yet they have not released virtually any of it to the public…

‘Why does the government continue to withhold more than 60 million doses of HCQ from the public?’ asks Jane Orient, M.D., the Executive Director of AAPS. ‘This potentially life-saving medication is wasting away in government warehouses while Americans are dying from COVID-19.'”