Let's Not Compromise on the Need to Strengthen Preclinical Research

Maurie Markman, MD

Clinical oncologists may not be fully aware of shortcomings in the laboratory research paradigm that have the potential to distort the scientific underpinnings of vital cancer studies. Yet there are problems surfacing in existing protocols—and the disconcerting reaction of some academics to questions about these procedures— that are critical to the future development of innovative cancer therapeutics that clinicians will be employing in their practices.

Any objective observer would recognize that being able to replicate a scientific claim, including observations related to laboratory-based cancer investigative efforts, is a core component of the process of science.

Of course, there are different forms of replication within the scientific domain. One form is the requirement that an individual scientist or research laboratory repeat experiments several or even multiple times to be certain any observation is reproducible and not simply a laboratory error. Certainly, this process should be undertaken before any results, with their corresponding interpretation, are presented to an external audience or submitted for peer-reviewed publication.

However, a second essential and separate component of the replication process occurs when a completely independent research group subsequently examines any initial claims and confirms the findings, even if not necessarily the interpretation of the observations, made by the first group.

In the clinical investigative domain, this often translates to a request or requirement for a confirmatory study by a second collection of researchers, especially in the setting of unexpected trial results, where the sample size might have led to the outcome due solely to small patient numbers or when the study is being reported by a research unit without an established track record in clinical investigation.

Lack of Independent Oversight

Importantly, trials involving human subjects are—or at least absolutely should be—conducted under well-established and widely accepted national/international guidelines that include the development of a formal research protocol, scientific and ethical review, and regular reporting to an oversight body (an independent review board) on the progress of the investigative effort. In addition, prospective studies should include a formal statistical plan based on the study question being addressed.But although laboratory based-research may be conducted under quite rigorous internal controls, there is no such established external oversight. Further, there is no mandate for how such efforts should be designed or conducted, for blinding or randomization of experiments, for the number of times a given experiment should be repeated to confirm reproducibility, or even for how outliers in any given test should be handled.

For example, it is understood in clinical research that all subjects who entered a trial must be reported even if they never received therapy and that outcomes must include patients who discontinued therapy for any reason.

By contrast, how does one know if a laboratory investigator simply elected to discard the results of several mice that died unexpectedly in the control or study arm of an individual experiment? Or if a particular experiment was repeated 10 times by an investigator and only once gave “favorable results”? What is to prevent an individual investigator or research unit from solely reporting that single “positive” experiment?

With this background in mind, is it really any wonder that when an independent attempt was made by a pharmaceutical company to replicate 53 landmark laboratory-based research results, conformation was only possible in six studies (11%)?1 A similar disquieting outcome was reported by a second pharmaceutical company attempting to replicate previously reported laboratory findings.¹

These analyses have appropriately concerned laboratory-based academic leaders including top officials at the National Institutes of Health.² Unfortunately, as a remarkable indication of the rather profound arrogance of some in the academic community, a number of investigators have questioned the need to be required to replicate their findings, apparently feeling they alone have the ability to conduct high-level quality research.^3,4

Such hubris might be amusing if these self-righteous beliefs were not potentially translated into practices that have led to a profound waste of time, effort, and financial resources of many individuals and organizations.

Further, it is critical to acknowledge that it is the American taxpayers, through the awarding of federal grants and contracts, who often pay these bills. Finally, one can only imagine the negative impact associated with a third-party (eg, industry or subsequent government-funded academic research) wasting many millions of dollars due to the publication of severely inadequate or even bogus laboratory-based results that claimed tremendous promise to be of major clinical relevance in the management of a life-threatening illness such as cancer.

Focus on Replicating Results

Of course, it is essential to note the failure of a single investigator to be able to reproduce the results of another laboratory does not by itself indicate the first laboratory was wrong, but if these results cannot be reproduced through several or multiple efforts by individuals/groups with acceptable expertise/experience, it is most appropriate to raise serious concerns.Fortunately, a number of strategies have been initiated or planned to help correct this rather alarming deficiency in the process of verifying the validity of laboratory-based clinical research, including in the cancer domain.²

A new online, open-access channel, Preclinical Reproducibility and Robustness, has been launched specifically to publish laboratory research results that relate to an attempt to reproduce the work of original investigative efforts.⁵

The need for such a resource is clear when one recognizes the difficulty of publishing and receiving appropriate recognition within the domain of replication research versus the potential splash made by an original project that is far more likely to be published in a high-impact scientific journal.

Maurie Markman, MD, editor-in-chief, is president of Medicine & Science at Cancer Treatment Centers of America, and clinical professor of Medicine, Drexel University College of Medicine. maurie.markman@ctca-hope.com.

References

1. Begley CG, Ellis LM. Drug development: raise standards for preclinical cancer research. Nature. 2012;483(7391):531-533.
2. Collins FS, Tabak LA. NIH plans to enhance reproducibility. Nature. 2014;(7485)505:612-614.
3. Bissell M. Reproducibility: the risks of the replication drive. Nature. 2013;503(7476):333-334.
4. Kaiser J. The cancer test. Science. 2015;348(6242)1411-1413.
5. Kaiser J. Calling all failed replication experiments. Science. 2016;351(6273):548.