Can We Trust the Science? The Challenge of Irreproducible Results
In the peer review process, articles submitted to scientific journals are sent to experts in the field who then assess the methodology, results and conclusions. Based on their feedback, authors often revise and re-submit, publishing an improved article as a result. Peer reviewers rarely attempt the actual experiments described in the paper. Irreproducible results are always a potential problem. Indeed, there is even a satirical journal dedicated to the issue (including a Graph That Proves All Theories).
The general assumption has been that irreproducible results (whether through fraud or error) will eventually be exposed by others working in the field. High-profile cases of fraud are often reported in the press, such as in cancer research or stem cell research. Despite climate deniers’ claims of shoddy research or worse behind research on climate change, it has widely been assumed that irreproducible results were low frequency occurrences. A recent study, reported in the Washington Post, however, suggests this may be a much bigger problem than recognized.
Psychologists sought to reproduce the experimental results of 100 experimental studies. They could reproduce the data only 39 times. Put another way, almost two-thirds of the published studies in their sample could not be independently verified. The senior editor of Science reacted with the defense that “This somewhat disappointing outcome does not speak directly to the validity or the falsity of the theories. What it does say is that we should be less confident about many of the experimental results that were provided as empirical evidence in support of those theories.” Somewhat disappointing outcome? Sounds like putting lipstick on a pig.
Scientific data are where the rubber meets the road in environmental law. Should pollutant standards be strengthened? Should a particular species be listed under the ESA? Should we regulate Dimethyl Terrible? All of these questions turn on the underlying research data. If the data themselves are suspect, the policy consequences will be, as well.
A single study in psychology research surely does not undermine the state of modern science. One hundred experiments may be too small a sample size. But this article reinforces fundamental concerns that have been voiced by well-respected researchers in recent years. One hopes that other researchers will assess the reproducibility of results in their fields, as well. Whether the major peer review journals will respond adequately remains to be seen (the scientific journal Nature dedicated a website to this problem). One could imagine EPA’s Science Advisory Boards having to address this concern in the near future, not to mention Congressional hearings about “sound science.” However it develops, this is an issue that will not go away soon.
Reader Comments
One Reply to “Can We Trust the Science? The Challenge of Irreproducible Results”
Comments are closed.
About Jim
James Salzman is the Donald Bren Distinguished Professor of Environmental Law with joint appointments at the UCLA School of Law and at the Bren School of the Environment …
READ more
This result should not be surprising, that a majority of experimental results could not be replicated. Researchers want results and that often biases their observations and their reporting.
Anyone who has ever conducted experimental research has undoubtedly felt the pull towards preferred outcomes. When rewards are offered for desired results, the pull is even stronger.
When “peers” applaud your results, without validating them, there is no punishment for fabrication. The researcher who gets away with faking their results, learns from and repeats that behavior – and tends to extend the same “benefit of the doubt” to others who work in their field.
Imagine how much worse the “science” must be in fields where theories can’t be tested by experiment, and where there are no experimental results, fields such as “climate science.”
Fortunately for “researchers” in many [allegedly scientific] fields, including “climate science,” research often consists, not of experiments, but of correlation of model results with very limited and dubious data. It’s hard to invalidate a model based on assumptions, when there is no way to test the assumptions.
Confident predictions of the outcomes of complex systems are presumptuous and pretentious. Attempts to control complex systems with simple control schemes – like stop burning fossil fuels – are foolish and dangerous.
Econometricians pretend to mathematically model the economy, and to be able to predict the outcome after perturbations to the economic system. No matter how often their predictions fail, their enthusiasm never wanes. Similarly, the Federal Reserve System keeps tinkering with interest rates, margin requirements, money creation, etc., pretending that they know what they are doing. The reality is that the market does what it does because of the complex interaction of all of its participants, then the Fed and the econometricians tweak their models, take credit for good outcomes, and rationalize the unexpected and the unfortunate outcomes.
Correlation of spurious data with the outcomes of mathematical models, no matter how elegant, proves nothing of any real significance. Models are not reality, correlation does not equal causation.
Climate change can be studied but not accurately predicted. Climate science cannot be predictive because climate processes are too complex. As time has proven, and will continue to prove.