Introductory considerations

Determining when there is a reasonable ratio between the risks and benefits of an early trial is a task that typically falls only to RECs.⁴ This may catch some readers by surprise, given the vetting done by regulatory agencies such as the US Food and Drug Administration (FDA) and the European Union (EU) European Medicines Agency. Phase 1 trials in either jurisdiction cannot launch unless the relevant regulatory agency issues permissions. These agencies vet the safety data produced by preclinical studies pertaining to new treatment modalities and sanction early trials only when they are satisfied that the preclinical data indicate that something is safe enough to try in humans. However, these agencies do not condition their permissions on review of efficacy data.10 11 They typically assess efficacy when they review licensure applications that contain supporting phase 3 data. RECs, however, in order to satisfy the ethical requirement of proportionality between risks and benefits, must consider the evidence on both sides of the risk/benefit ledger. Hence, preclinical efficacy data are just as critical as preclinical safety data in any and all REC attestations that benefits and risks ratios in early trials are reasonable.

These attestations stand or fall on the reliability of the scientific evidence that informs inferences REC members make about the potential for risks and benefits. The evidence comes from preclinical studies, the vast majority of which consists in animal studies. An advisory committee for the development of certain new therapeutics has recently reminded us how central these studies are to clinical translation efforts. They are used to both develop and test ‘novel therapeutics, including small molecules, biologics, gene modifiers and cell therapies’, as well as inform ‘optimal dosing regime[s] and route[s].12’ They cross a broad spectrum of investigations, ranging from ‘identify[ing] new druggable targets and [helping with the] design or test[ing of] specific therapeutic modalities, as well as detect[ing] diagnostic or therapeutic biomarkers.’ [Finally, they] allow the comparison of pathological phenotypes with the human disease, and [provide] insight into the underlying reasons for differences at varying levels of complexity, [thereby] allowing the identification of protective pathways in animals that could be enhanced in humans.’12

It should be noted that many are deeply sceptical about the value of animal studies to clinical translation, given how vast the differences can be between species.13 Whether one shares this scepticism or not, these are the studies RECs must rely on to support inferences committee members make regarding the nature and potential of both benefits and risks in early trials. The strength of those inferences is determined by the scientific quality of the studies that generate them, since ‘a well-designed experiment is a fundamental criterion for reliable information and for generating any benefit at all.’14 Let us now turn to the evidence about their quality.

Relevant evidence about preclinical research

The strength of inferences about safety and efficacy can turn on several factors of preclinical studies. To begin with, problems with the construct, internal and/or external validity of individual studies can undermine their reproducibility or generalisability. Construct validity refers to whether an experiment is capable of studying the outcome it is meant to measure.15 For example, studying a chronic human disease in an acutely injured animal can cause investigators to ‘mischaracterize the relationship between a preclinical study and an ensuing [clinical] trial.’16 Internal validity refers to multiple study factors that influence what can be gleaned from the experimental population being studied.15 Factors include matters like a study’s design and analysis. Problems such as inappropriate selection of control groups, lack of blinding and randomisation procedures and incorrect statistical analysis plans15 can produce ‘spurious causal inferences’16 that result in erroneous research reports and irreproducible findings. External validity refers to ‘experimental design features that [support] reproducibility and generalisability of the expected results’ outside of the experimental setting that produced the results.15 For example, studies conducted on animals may discover things about a phenotype that get wrongly construed to a genotype instead.17 It is concerns about external validity that fuel much of the aforementioned scepticism about the value of animal studies.

Should RECs worry that there is more than the occasional preclinical study that lacks construct, internal and/or external validity? Ample evidence shows they should. It reveals substantial as opposed to occasional numbers of preclinical studies reporting unreliable findings. For instance, consider how mislabelled cell lines can pollute preclinical research findings.18 This results in preclinical studies meant, for example, to investigate one type of cancer getting conducted with cell lines from a different type of cancer instead, which can lead to an early phase trial that lacks a sound scientific basis and thus ethical risk/benefit ratio.18 Though this problem has been known about since 1950,19 it is estimated that fully a quarter of NIH sponsored research projects on cell lines may still be using misidentified or contaminated cell lines.20 NIH at long last initiated efforts to prevent researchers from using mislabelled cell lines,21 but its efforts have limited reach, meaning that the problem still persists.

Perhaps more concerning is a finding based on meta-analyses and simulation of published preclinical studies as a whole. It looked at various features of the studies, such as their design and analyses features like sample sizes and data analysis plans, and reported that ‘most research findings are false for most research designs and for most research fields.’22 Meta-analyses of preclinical studies for individual disease areas report similarly discouraging results. For example, a 2013 assessment of 4445 animal studies for 160 candidate treatments for a range of neurological disorders that were subsequently tested in humans found that, while over 1700 of them reported positive findings, ‘only 919 studies would a priori be expected to have such a result.’ This finding led the authors to conclude that ‘only eight of the 160 evaluated treatments should have been … tested in humans.’23 Yet all 160 studies received REC approval. Finally, a 2020 review of preclinical studies related to neuromuscular disorders reports that almost 30% had deficient use of control groups, blinding or randomisation.12 Such studies help to explain the findings of a recent survey of biomedical researchers showing that the vast majority of respondents believe there is a ‘reproducibility crisis’ in biomedical research.24

One might question the direct relevance of these aggregate findings to RECs since they have to review individual trials, not aggregate challenges of preclinical research generally. While true, these aggregate findings suggest there are genuine reasons to worry that any given preclinical study, including those used to support REC inferences about potential risks and benefits of early trials, may well report a false-positive finding. A recent analysis of failure rates in clinical trials of acute stroke bears this out. Its authors estimate that a majority of the individual reports of positive findings from animal studies meant to inform clinical studies of acute stroke are actually false positive results.25

A likely explanation for this prevalence of false-positive findings is because, as Kimmelman et al have explained, the vast majority of preclinical studies are best viewed as hypothesis generating, as opposed to hypothesis confirming, studies.26 This means that, while those studies may have produced potentially promising findings, it is not known whether those findings are false, which would predict likely failure in human studies. This helps explain why researchers are increasingly realising that ‘most preclinical experiments do not represent a true preclinical efficacy study’12 capable of supporting reliable REC inferences about efficacy that need to be factored into REC assessments of risk/benefit ratios in early trials. However, rather than requesting additional data from follow-up preclinical studies that would be needed to rule out the possibility that the initial positive findings were false-positives, RECs too often permit researchers to proceed straight to clinical trials.26

To appreciate the ethical import of this, consider remarks made at an annual meeting of the American Society of Clinical Oncology by Glenn Begley, coauthor of a landmark study about the irreproducibility of preclinical studies.27 He stated that preclinical studies should be treated like single case reports of clinical findings. Just as one ought not alter the standard of patient care on the basis of a single case report, RECs should never approve a clinical trial on a similarly slim basis. ‘[These hypothesis generating trials] should not, under any circumstances, trigger a clinical trial. I think there are thousands of patients [who] have been treated inappropriately based on publications [about such studies] in the top journals.’28

One might grant the ubiquitous nature of such problems in individual studies as well but counter that what is really important to REC risk/benefit assessments are the specific preclinical efficacy studies reported in the investigator brochures (IB) that RECs receive when reviewing individual trials. These are the actual studies that an early trial will be based on and thus the only studies that an REC should concern itself with. Unfortunately, a recent study of IBs supporting the early phase trials approved at three research sites between 2010 and 2016 revealed severe problems that suggest the need for concern regarding the reliability of REC approvals based on the specific preclinical studies referenced in the IBs.29

The study examined 109 IBs that reported on 708 preclinical efficacy studies. Less than 5% of them contained any information about such basic study characteristics as randomisation, sample size calculation and blinded outcome assessment that would help RECs make reliable inferences from study findings. Most alarming is the fact that no publications were referenced for almost 90% of the studies. Thus, even if an REC was truly curious about the reliability of a study, this examination of IBs suggests the REC would likely be unable to ascertain it because it would be unable to examine design details about the individual preclinical efficacy study. And when a REC does not ascertain the quality of any given preclinical efficacy study, then the reliability of assessments regarding whether risk/benefit ratios are reasonable and thus whether it is ethical to conduct a given early phase trial remain unknown. All this combined evidence about preclinical studies raises alarms about whether RECs are adequately preventing early trials with unacceptable risk/benefit ratios from passing through their gate.

Introductory considerations

Let us now move further along the clinical translation pathway to phase 3 and 4 trials to see how REC review fares at this end of the spectrum. It is here where we can, for example, meaningfully assess whether a new drug actually works, see if a drug proven to work for one health condition might work in others, or see if one drug works better than other approved ones. All of these types of investigations can advance biomedical research’s goal of improved health and healthcare. However, here too this outcome is contingent on the extent of fealty to the demands of good science, fealty on display in the design, conduct, analysis and reporting of the clinical trials.

Knowing the extent of fealty about any particular trial can prove far from straightforward. Returning to early phase trials, the objectives of them are more narrowly constrained than they are for many later phase trials: can we learn enough about safety and potential efficacy to see if a new modality should progress to later phase studies? Later phase trials pose different questions and can have different challenges. Ideally, we would like to know such things as whether a new drug can extend life or reduce suffering and life-limiting symptoms. However, this societal objective usually gets interwoven with trial sponsor interests, especially since private companies fund the majority of later phase trials. Their interests can include such matters as, can we produce a dataset that will sway a regulatory agency to approve a new drug application or, if a drug is already approved, can we produce a dataset that will result in peer-reviewed articles or impactful educational presentations that help persuade physicians to prescribe our drug instead of another?

It is worth noting this distinction between early and later phase trials because the clinical trials that get done are the clinical trials that someone is willing to pay for. Organisations like the NIH can rarely afford to do more than try to shape research agendas that later phase trials can investigate while regulators like the FDA review the applications that sponsors choose to bring them. This places sponsors, the majority of whom are profit-oriented companies, in the driver’s seat of most of the clinical trials related to licensing and postlicensure use of drugs.

There is, however, a critical similarity between early and later phase trials to note: just as it is difficult to study human disease in animal models, it is often difficult to adequately study drugs in humans. When chronic diseases like diabetes can take years to shorten a life or may or may not cause grave damage to organs, conducting trials that interrogate a drug’s effects on these endpoints may prove impossible, due to such matters as feasibility, expense or drug adherence. So, we frequently are left with trials that study secondary endpoints that, while they may or may not be able to tell us if a drug can truly improve healthcare, are well suited for creating datasets suitable for licensure and postlicensure marketing purposes.

These considerations are mentioned in order to highlight both how the safety and efficacy endpoints in early phase trials can be less fungible than the endpoints for later phase trials and why, in light of the high financial stakes following a drug’s approval, it is unwise for RECs to be incurious about the ways that sponsor pecuniary interests can get woven into the planning, design, conduct and reporting of later phase trials. While drug companies can earn immense profits from drugs approved on the basis of datasets about surrogate endpoints that may not really improve health or lessen disability, society has an interest in avoiding approving ineffective new drugs or having physicians and consumers preferring more expensive ones that may in reality be no better than less expensive ones. Either outcome provides little social value or may cause social disvalue. RECs are our last and, at times, only line of defence for preventing clinical trials that lead to these unwelcome outcomes.

Are such unwelcome outcomes frequent enough, though, that RECs need to concern themselves with them? There is certainly ample scholarship that casts many pharmaceutical companies’ clinical trial practices in a negative light. There is space here only to note this scholarship. Some work looks more at the industry as a whole and finds it lacking30 31 while some looks more at troubling episodes when sponsors withheld critical safety data about drugs from licensure application datasets.32 33

More germane to the task at hand is a body of work34–37 from Science and Technology Studies scholar Sergio Sismondo describing what he calls ‘ghost managed medicine,’ which refers to how drug companies socially construct the knowledge that informs physician prescribing behaviours. This social construction occurs in the planning, design and conduct of clinical trials, as well as the reporting of their findings through scientific journals and other dissemination strategies which influence physician prescribing practices and thus drive corporate profits. Sismondo artfully describes how drug companies use the leeway they have to decide what they will study, how they will study it, and what will get said about it by whom, all within the scientific strictures of clinical research and the legitimacy those strictures confer.

A brief illustration is in order. The gold standard of clinical trials is a randomised double-blinded trial where neither the physician nor the patient/research participant knows which of two or more treatments being compared is given to whom. Findings from these kinds of trials are the ones that physicians typically prize the most because they can possess the requisite rigour needed to confer high reliability on the results such trials produce. This no doubt was the kind of trial that Anthony Fauci had in mind during his remarks quoted above in the Introduction.

Even with the strictures of a randomised double-blinded trial in place, sponsors can nevertheless select, for example, not only what drugs but at what doses, get compared. They also decide who is eligible for inclusion in the studies. These and other choices about a trial’s design can bias a trial’s outcome in a particular direction. But the sponsor’s influence does not end here. They also own the data, control data analysis and determine what data to include in the datasets that journal readers see. This extent of leeway also permits them to run multiple analyses on data subsets or entire datasets when preparing drug applications. These choices can mean the difference between approval or denial of licensure. In other words, ghost management can effectively sabotage the regulatory review process.37 38

Post regulatory approvals, companies also control what journal articles contain and who the authors are. Many drug companies contract with publication planning companies to do the actual writing. These companies are quite adept at getting studies through the peer-review process of leading medical journals.37 They can also allow sponsor approval of authorship, which typically comes in the form of lead authors who are well-regarded physicians who may have had no or little say over a trial’s design or data review and analysis. Further, they can withhold authorship from those most responsible for the design, analysis and reporting of the trial and its outcome, while still complying with the authorship policies of most all major medical journals.39 Such concerns are borne out by a 2018 study documenting the outsized, and often hidden, role of industry employees in the design, analysis and reporting of industry sponsored clinical trials. Among the more alarming findings of the study were those showing that only 40% of the academic authors were involved in data analysis of trials and that only a third of academic authors reported having final say on trial design.40

Relevant evidence about the social value of later trials

If Sismondo’s contention that corporate ‘ghost management’ of clinical trials can be at odds with good science is correct, there should be evidence substantiating it. Again, due to space constraints, we can only note some of it. First is work ongoing since 198041 by numerous scholars about what has been dubbed ‘sponsorship bias,’ wherein sponsors ‘distor[the] design and reporting of clinical experiments to favour the sponsor’s aims.’42 More recent examples of the scholarship document sponsors both selectively reporting data to regulatory agencies and over-reporting positive as opposed to negative results in the published literature.43–46 While much of this body of work looks at aggregate findings that do not necessarily prove that there is a preponderance of industry-sponsored trials that produce reports used to elevate company interests over the social value that clinical trials are meant to serve, they certainly lend credence to Sismondo’s claim that too many clinical trials are ‘sophisticated marketing disguised as disinterested science.’36

A look at additional evidence found in two recent studies looking at the impact of company marketing strategies on clinical trials strengthens these suspicions. The first study examined a group of head-to-head trials comparing the effectiveness of competing drugs.47 In theory, such trials are among the most important trials to be conducted. They can tell us, for example, whether one of those drugs is truly superior to the other or whether a cheaper one is comparable to a more expensive one. When such knowledge is achieved, future patients and society as a whole benefit. Unfortunately, the study revealed that few of the trials reviewed in it rose to this level of social value.47 Its authors conducted a random review of 319 randomised trials that were published in 2011 that compared drugs or biologics. The vast majority (82%) was sponsored by industry. Only 3 among them involved ‘truly antagonistic comparisons’ that would be capable of producing accurate comparisons of the drugs or biologics. Most alarming for our purposes is that the study also found that all but 2 of the industry-sponsored trials reported favourable results. These results, we will see shortly, can boost marketing efforts and future prescribing practices for products that, for all we know, may in fact be inferior, that is, neither non-inferior or superior, ones.48

The other study looked at 194 drug trials, also published in 2011, in 5 leading medical journals.49 The study’s authors wanted to determine whether trials might have been conducted largely, if not exclusively, for marketing purposes in order to get physicians comfortable with prescribing a new drug. To test whether this was the case, the authors looked at studies whose results were published in the New England Journal of Medicine, The Lancet, Annals of Internal Medicine, PLOS Medicine, and BMJ. The authors concluded that 41 of the 194 trials were found to be ‘suspected marketing’ and that all the suspicious ones were industry funded. They reached the conclusion about suspected marketing in large part due to the fact that those studies tested drugs on patients from an average of 171 different geographical areas, compared with just 13 different geographical areas for the trials deemed not to be marketing trials. Such additional geographic dispersion assured that there were vastly higher numbers of prescribing physicians giving the study drug to their patients in the suspected marketing trials compared with the other trials. Physicians who recruited their patients into the trials would be more likely to prescribe the study medication to their future patients after completion of the trial and many other physicians who read about the trial results will be persuaded to prescribe these new medicines since the studies appear in such prestigious medical journals and, as the authors of the study report, mask not only the marketing nature of the studies but the role of sponsors in designing and analysing them.49

Readers may think that the fact that the articles passed the high peer-review bar set by such prestigious medical journals with very low acceptance rates allays concerns about the quality of the science behind the trials in question. However, it is at the very least equally plausible that the ‘ghost’ or company-affiliated authors with little or no actual involvement in the studies who take the lead in writing the text and coaching the first and/or senior authors are quite talented in navigating the peer-review process, as their considerably higher than average acceptance rates attest.36 This interpretation is supported by a recent analysis of the latest version of the ’Good Publication Practices 3’ (GPP3), a set of voluntary guidelines established by the International Society for Medical Publication Professionals to strengthen medical publications.50 The analysis contends that GPP3, despite its lofty aims, actually ‘provides a de facto manual for how marketing through academic journal content can be conducted in compliance with contemporary editorial standards.’51

All of this evidence suggests that sponsors of ghost-managed clinical trials derive financial benefit by cloaking themselves in both the scientific and ethical trappings of clinical trials. Their reports of the trials they sponsor enjoy both the scientific imprimatur regulatory approvals and peer-reviewed journals confer and the ethical imprimatur REC approvals confer on the trials themselves.

This evidence, like the evidence about preclinical research, raises concerns about REC practices. At a minimum, we see how REC approvals facilitated recruitment of hundreds of thousands of people47 into ethically suspect trials because the REC reviews failed to flag the potentially flawed findings to be produced by the sponsorship bias that can result in deficient designs, analyses and reports. REC approvals also facilitated the subsequent disproportionate influence of those findings on future clinical guidelines and prescribing practices and the social disvalue as opposed to value that influence can cause.