This blog post is what you might call "metascience." I address a scientific issue but do not use experiments or calculations, just educated common sense. Common sense is not always right or fair, but it usually is.
You may have seen news articles about a device called the EM-drive that is supposed to produce thrust without releasing any exhaust or other momentum-carrying particles like ions. That would violate conservation of momentum, which if true would start the biggest scientific revolution in a century, if not ever. (In an earlier post, I discussed conservation of momentum and the tricky way it applies to thrust-generating devices like rockets.) Conservation of momentum is connected to the fact that the laws of physics are independent of location, so you can see what trouble would arise if it turned out not to be true.
The EM-drive is a cavity in which electromagnetic (hence the EM) waves are generated and confined. NASA tested this device and reported to AIAA Journal that the EM-drive produced 1.2 millinewtons per kilowatt of power consumed. A millinewton is only equal to the weight of about two grains of rice, but it is comparable to the thrust of other types of electric rockets that do release momentum. It is certainly a measurable force. But, is it real?
The AIAA Journal paper is very careful in its technical approach. First, it documents an independent test run by someone other than the guy who invented the thing. Second, it lists nine separate possible sources of error that the researchers considered and rejected. But the results have a lot of scatter. One red flag is that the thrust per watt is reported to two decimal places, but just eyeballing the data shows that there is probably not even one significant digit, just a statistically significant difference from zero thrust.
Now, a cautionary tale from science history. In 1989 some chemists observed a sporadic heating effect in a certain kind of electrochemical cell. They could not explain the heating chemically and leaped to the conclusion that cold nuclear fusion was taking place. Cold fusion was then "replicated" by several other labs around the world. Like the EM-drive thrust, the heating was a small effect that was subject to many possible measurement errors. Although many labs replicated the effect, the reported temperature rises were inconsistent. Also, some labs could not replicate the effect. After a few months most scientists concluded that the heating was a measurement error, and cold fusion is now a fringe theory.
There are some striking parallels between cold fusion and the EM-drive:
1. Cold fusion heating and EM-drive propulsion are "small" effects. By small, I mean they require extraordinary measurement techniques to rule out error. We're not talking about mercury thermometers or postage scales here. But the experience physicists had with relativistic and quantum effects - both completely invisible at the human scale - prevents them from dismissing small effects out of hand.
2. Both the cold fusion and the EM-drive devices are claimed to have implications for fundamental physics, but neither was intended to demonstrate a new physical law. The required changes to physics came as explanations after the fact.
3. Both devices stand in contradiction to every other of the thousands or millions of attempts to invalidate their claims. In the EM-drive case, there have been myriad experiments in which very accurate momentum balances had to be measured, including ones that involve a resonant RF cavity.
This third point bears some discussion. Many people think of math as a science, but the standard of truth in math is logic, not experiment. For this reason, math is the least controversial of the sciences. No mathematical theorem was accepted for years before being proven wrong. But in physics, Newton's theory of gravity was thought to be final for centuries before being overturned by general relativity. So such things have happened. Generally, the less mathematical a science is, the more controversies it has. Biologists used to believe all kinds of crazy things, like spontaneous generation and the quinarian system. Don't even get me started about psychiatry or economics.
But then, there's the biggest red flag of all.
4. Both devices appeared in the media, with claims of new physics, before hardly any independent science at all had been done. The involvement of the media enticed the inventors to make some very bold, very public claims that they could not walk back without losing serious face. (My impression is that the cold fusion guys were in no way conscious frauds. They were scientists who stepped into media quicksand and never escaped.)
There are conflicting hypotheses about the measured EM-drive thrust. The most popular and likely is that it's measurement error. The AIAA Journal article claims that hidden quantum-mechanical variables could explain the effect. (Even Einstein was a fan of hidden-variable theories, but he was far outside the mainstream in quantum mechanics.) The builder of the EM-drive actually contends that the device does not violate any known laws of physics, but his explanation doesn't make a lot of sense to me.
Another red flag is the authors' use of the commercial physics simulation software COMSOL to support the hidden-variable claims. There's nothing wrong with COMSOL in its conventional uses, but I don't need a Ph.D. in physics to say that software that admits hidden-variable quantum mechanics could be used to prove just about anything you want. One author's self-identification as a "COMSOL Multiphysics Analyst" is just weird. That's like a mechanic calling himself a "wrench operator."
I am glad I was not asked to referee the AIAA Journal article. But if I had, I'd have recommended that it be published as a purely experimental paper, without all the theoretical stuff, just a frank admission that the result appears to violate known physics, and with a statistically valid analysis of the experimental scatter.
A good approach for a new study (hey, NASA!) would be to statistically evaluate the measurement errors that would be necessary to preserve momentum conservation. If that paper were markedly less convincing than the AIAA Journal paper, it would suggest that further investigation might not be a total waste of time.