7th May: Lecture on Scientific research: The Bad and the preregistered by Stefan Wiens. Venue: Scenen, Frescatibiblioteket. No registration is needed.

What do you see as the greatest advantages of open science?

Stefan Wiens

– The greatest advantage is that there are no disadvantages for science. Science that is not open and transparent is not science. To count as a scientific fact, there has to be convincing evidence in support of it. To allow others to evaluate the strength of evidence, scientists must share their data and material. Because science is complicated and errors happen, all steps of the research need to be transparent and open so that they can be checked. Further, this openness facilitates future research because scientists can reuse the data (for example meta-analysis) and build on the shared material. Because most scientists are paid by public money, they owe to the public to share their research openly by providing open data and access.

What kinds of practical applications within open science do you see as the most meaningful for good research practice?

– Psychology and other disciplines have had difficulties in reproducing and replicating previous findings. This is serious because it undermines the trust in science. However, the movement towards open science has helped tremendously in identifying and resolving the underlying problems.

– First, there are many resources that encourage open science and that provide simple but powerful tools to share data and material openly (for example COS.io, figshare.com). In principle, anybody can go through the material and check if all steps are correct and reasonable. For example, if there are errors or odd analysis decision, it comes as no surprise that others cannot obtain similar results.

– Second, a good theory should explain previous findings but also generate hypotheses that predict new findings. Unfortunately, it has been common practice to come up with hypotheses after exploring the data (HARKing - Hypothesizing After Results are Known). Of course, if the hypotheses are generated after the data, they match the data perfectly (because hindsight is 20/20). This perfect fit is misleading because it suggests that the theory is better than it actually may be. In response, scientists need to preregister their hypotheses. Preregistration allows scientists to define their actual predictions and prove that their hypotheses came before the data. Preregistration also encourages scientists to define methods and analyses to reduce issues with analytical flexibility.

– Third, Registered Reports is an excellent publication format that should maximize trust in the results. Currently, many journals practice a tabloid culture. For a manuscript to get accepted, findings have to be novel, and results clean and unambiguous. Unfortunately, this focus on clean results is misguided because scientific research should be evaluated primarily on whether the ideas are worth pursuing and if the planned methods and analyses are reasonable. If so, any results should be informative and published. In a Registered Report, scientists submit a manuscript with their ideas for a study and how they are planning to conduct it. The manuscript is reviewed and in-principle accepted before any data are collected. Apparent problems with the idea and design can be addressed early on. Critically, this procedure reduces any criticisms after results are known (CARKing - Critiquing After the Results are Known). It also reduces any motivation for the scientists to provide overtly clean results that may be distorting the truth.

What do you see as the greatest challenges in practicing open science?

– If scientists are rewarded mainly for publications that contain novel, spectacular ideas with overtly clean results, then they will continue to deliver as requested. However, everybody needs to realize that this way of conducting science has serious issues and needs to act accordingly. Although scientists acknowledge that they have made mistakes, many are slow to practice open science. Similarly, journals and funders need to realign their guidelines to facilitate open science. That is, everybody should push towards transparency, openness, and reproducibility. In this process, a major challenge will be to provide the necessary training and tools to implement these changes. For example, few scientists are trained in how to manage and document data properly. Also, many open-access journals have fancy titles but publish any manuscript without peer review (predatory journals). The public as well as scientists need expert help to identify and avoid these journals.

What changes within research politics and the research community could counter these challenges?

– Because open science reduces biases, incentives need to change so that scientists are rewarded for open science. Accordingly, research politics should reward scientists for publishing open data and using tools that emphasize open and transparent science. Research politics should provide scientists with opportunities to learn open science (for example data management and documentation). Politics should also support open science journals and tools because many of these are developed and maintained by non-profit organizations. Further, when evaluating each other’s work, scientists should view open science practices as important professional contributions.

Do you have any concrete examples – positive or negative – of where open science applications have made a difference in practice?

– An eye opener for me was a study by Eyding et al. (2010, BMJ, https://doi.org/10.1136/bmj.c4737) on reboxetine as an acute treatment of major depression. The authors integrated findings across published studies (meta-analysis) and found that reboxetine was effective with few side effects. However, during their search for studies, they discovered that many unpublished studies existed. They had to struggle to get access to these unpublished studies. When they integrated the unpublished studies with the published studies, the combined evidence showed that reboxetine was ineffective and potentially harmful.

In this context, it may not be surprising that pharmaceutical companies were not interested in publishing findings that were not positive. However, this is true even for many scientists at universities because many journals reject articles if the results are not positive or clear cut. For example, one of my manuscripts was rejected with the argument that although the study was well designed and executed, we must have done something wrong because we did not find a statistically significant, positive effect. Studies should be evaluated on the basis of the soundness of the idea and the method, not on the actual results. If somebody conducts a sound study, it is important that the results are published no matter the results. This is critical to obtain an unbiased estimate of the true effect. Otherwise, results will be biased, as shown by Eyding et al. (2010). Open science is necessary to permit quality control and to ensure that all material, method, and data are preserved for the future.