Preclinical Workshop: Reproducibility and Reliability of preclinical research: All in the design?

I attended the preclinical workshop on Sunday afternoon, which included very interesting talks relevant to reproducibility and reliability of preclinical research.

The session started with Professor Marcus Munafo from University of Bristol, who covered hot topics on reproducibility of psychological research. Existing evidence shows that only around 40% of psychology-based experiments can be replicated and in most cases the effect sizes vary widely from the reported ones. Questionable practises such as the determination of cut-off points to define outliers were discussed, as well as the impact of these practises on the quality of research. Reproducibility problems seem also to be relevant in fMRI experiments. Professor Munafo discussed that most of the published fMRI studies do not provide adequate information in terms of analysis and/or methods procedures, leading to difficulties in reproducibility of findings. Another common problem that may explain why psychology-based experiments show reduced reproducibility is relevant to power issues; <80% of experiments have been reported to have the required power. The reporting of only positive findings (citation bias) was also discussed, as well as the impact of the unpublished negative findings in terms of ‘biasing’ the evidence. Professor Munafo closed his presentation proposing that existing problems can be viewed as opportunities to adopt new strategies in research, including but not limited to quality control at every stage of the research process, which will enhance overall quality of research and efficiency at every stage.

Following Professor Munafo, Dr Sarah McCann a post-doctoral research fellow at the University of Edinburgh talked about some of the challenges in animal preclinical studies which may lead in poor translation of findings from animals to humans. Existing problems with internal validity, external validity and construct validity were discussed, as well as the impact of publication bias in the estimation of  actual effect sizes. For example, Dr McCann, referring to animal stroke studies, presented evidence supporting that approximately 16% of studies in the field remain unpublished, resulting in an overstatement of efficacy. Overall, this is a common problem across various animal models and the current estimation is that around 20% of animal studies remain unpublished, especially those with negative or neutral outcomes, resulting in biased outcomes and overestimated effect sizes. Dr McCann finished her talk outlining the importance of following reported guidelines when it comes to publishing animal research studies and discussing the need for tools that allow for rapid living systematic reviews and meta-analysis. In this way, translation of animal preclinical research to humans can be enhanced.

The following talk was by Dr Elliot Lilley, senior scientific officer at the RSPCA and editor at the BJP, who discussed issues relevant to reporting of animal research and why welfare matters. Dr Lilley started his talk reporting that reproducibility is a big problem in animal research, and as an editor himself discussed how reporting of animal research could be enhanced to reduce biased outcomes. Towards this aim the ARRIVE (Animal Research: Reporting of In Vivo Experiments) guidelines and the importance of following these guidelines when reporting animal research were discussed. Dr Lilley outlined the importance of welfare in animal research through the suggestion that welfare problems produce pathology characterised by physiological and behavioural response. What is more, scientific method assumes the absence of confounding factors or uncontrolled variables and that is also another reason why welfare matters in animal research. Small changes for animals, such as improvement of housing, care and welfare assessment can augment animal health, leading to increased quality research outcomes. The take home message  from Dr Lilley’s talk was clear, when it comes to reporting of animal research: ‘Write as you would like to read’

BAP-Brighton-Centre-Day-1-2_SimonCallaghanPhotography116The final talk was delivered by Professor Jo Neill from University of Manchester, who discussed sex issues in animal preclinical research and why this may lead to biased outcomes. Professor Neill started her talk presenting data relevant to PCP pharmacokinetics in animal research, showing than male and female animals differ both in the required dose and the circulating levels of PCP in brain and blood. Sex differences have also been reported in behavioural tasks, such as the object recognition task, during which females seem to perform better when they are presented with an ‘unknown’ object, and males to perform better when they are presented with the same object but in a different location. Therefore, sex is a very important factor in animal research. However, sex, among other variables such as age, number and/or strain of animals, are rarely reported in published studies. Most studies also tend to use male animals, and this is an important limitation that may explain why preclinical outcomes fail to be replicated in clinical studies. However, as Professor Neill presented, more and more papers are published outlying the importance of including female animals in neuroscience and biomedical research. What is more, accumulating evidence suggests that the estrous-cycle should not be considered as a problem for variability when researching female animals, while researchers need to consider other sources of variability when conducting preclinical research such as differences in circadian rhythms.

Overall, all talks discussed various critical aspects when it comes to reproducibility and reliability of preclinical research, with suggestions of standard approaches to be followed that will enhance the quality of preclinical research and ultimately the translation from animals to humans.

Leave a Reply

Your email address will not be published. Required fields are marked *