There’s a saying in negotiation that the most powerful asset is the ability to walk away from the deal.
Similarly, in science (or engineering, business decision making, etc.), you have to be willing to give up your favorite ideas. When I look at various embarrassing examples in science during the past decade, a common thread is researchers and their supporters not willing to throw in the towel.
Don’t get me wrong: failure is not the goal of a research program. Of course you want your idea to succeed, to flourish, and only to be replaced eventually with some improved version of itself. Similarly, you don’t go into a negotiation intending to walk away. But if you do research without the willingness to walk away if necessary, you’re playing science with one hand behind your back.
As with negotiation, your efforts to keep your theory alive, to defend and improve it, will be stronger if you are ultimately willing to walk away from your theory if necessary. If you know ahead of time that you’ll never give up, then you’ve painted yourself in a corner.
Look what happened with Marc Hauser, or Brian Wansink. They staked their reputations on their theories and their data. That was a mistake. When their data didn’t support their theories, they had to scramble, and it wasn’t pretty.
Look at what happened with Daryl Bem. He collected some data and fooled himself into thinking it represented strong evidence for his theory. When that didn’t work out—when careful examination revealed problems with his methods—he doubled down. Too bad. He could’ve preserved his theory to some extent by just recognizing that his experimental methods were too noisy to measure the ESP he was looking for. If he really wanted to make progress on the science of ESP, he’d have to think more about measurement. Bem’s defense of his scientific theory was too brittle. The front-line defense—defending not just the concept of ESP but the particular methods he used to study it and his particular method of data analyses—left him with no ability to go forward. We can learn from our mistakes, but only if we are willing to learn.
Look at what happened with Satoshi Kanazawa. He found some data and fooled himself into thinking it represented strong evidence for his theory. When that didn’t work out—when careful examination revealed problems with his methods—he doubled down. Too bad. His general theory of evolutionary psychology has to have some truth to it, but he won’t be able to learn anything about it with such noisy methods, any more than I could learn anything useful about special relativity by studying the trajectories of billiard balls. If Kanawawa were willing to walk away from his claims, he could have a chance of doing some useful research here. But instead he’s painted himself into a corner.
To continue with the analogy: Walking away should be an option, but it’s not the only option. A negotiator who walks away too soon can miss out on opportunity. And, similarly, as a scientist you should not abandon your pet theory too quickly. It’s your pet theory—nobody else is as motivated as you are to nurture it, so try your best. So, please, defend your theory—but, at the same time, be willing to walk away. It is only by recognizing that you might be wrong that you can make progress. Otherwise you’re tying yourself into knots, trying to give a deep explanation of every random number that crosses your path.
Tomorrow’s post: Stan saves Australians $20 billion