Evaluating Scientific Claims

On Twitter today, Joe asked me whether two Italian scientists’ claim to have demonstrated working cold fusion was real or not. I took two minutes, did some Googling, and told him that their cold fusion device may turn out to be really producing cold fusion, but that there are too many red flags for me to accept it now. Those red flags aren’t necessarily obvious if you’re not familiar with how science works, but knowing them can help you decide how much trust to put into a scientist’s claims and make you a better-informed person. You don’t even have to know any actual science to see them. Here’s your field guide to some of these red flags, as demonstrated by this cold fusion claim.

According to the write-up on PhysOrg, Italian scientists Andrea Rossi and Sergio Focardi of the University of Bologna held a press conference to announce that they had created a low-energy nuclear reactor (similar to a cold fusion reactor) that could turn 400 W of input power into 12,400 W of heat. Their paper on the reactor’s process was rejected by peer-reviewed journals, instead appearing on a blog, Journal of Nuclear Physics, that the two researchers created. The reactor is generating heat through some mechanism that the two researchers say they don’t have a theory for, and if it’s doing so, it runs counter to earlier theoretical work on these low-energy nuclear reactions.

You don’t have to know anything about the science to be skeptical about their results: their paper didn’t appear in a peer-reviewed journal, they announced their scientific discovery at a press conference, and their discovery contradicts current theory. None of these are slam-dunk reasons for why their claims can’t be true, but taken together they make it far more unlikely that their results are real.

No peer-reviewed paper. Peer review as a system has its flaws, sometimes rejecting good papers or accepting bad ones, but it gives other researchers a chance to evaluate your work. Rossi and Focardi claim it wasn’t published because they don’t have a theory about how the reaction works. That doesn’t ring true, since experimental papers are often published ahead of the theory to explain them. I’d guess that they didn’t describe in detail how they performed their experiments or what is really inside their reactor, preventing other researchers from duplicating their efforts. As corroborating evidence, their patent was rejected for a lack of detail.

Announcing scientific results at a press conference. If they’ve truly discovered new science that they want evaluated, scientists will need to take a look at it, not the media. It’s possible that they’ve had to go straight to the media because they aren’t getting attention inside the scientific community, but it’s much more likely that they’re having to go this route because their work won’t withstand scientific scrutiny.

Overturning currently-established theories. Theories can, and are, overturned with regularity in science. Scientists eat their own young, devouring old theories as new evidence comes to light. But the stronger your claim to have disproved a previous and tested theory, the stronger your evidence has to be. In this case, Rossi and Focardi have demonstrated a device but provided no evidence about what’s inside that box, nor have they given enough information to let others reproduce their results. What they’ve done isn’t falsifiable at this point. Their scientific evidence is weak at best.

If you look at scientists pushing theories and experiments outside the boundaries of accepted science, you’ll see these warning signs repeated time and time again. Any one of them by themselves isn’t a death knell to a scientist’s claims, but as they add up I become extremely skeptical, and so should you. It’s possible that Rossi and Focardi have discovered cold fusion, and I’d love to be proven wrong. But with this many red flags, I doubt it.

6 Comments