One of the great weaknesses of Science is this mistaken idea that if an experiment contradicts the dominant theory, we should throw out the theory instead of the experiment.

Experiments can go awry. They can contain design flaws. They can be deliberately corrupted. They can be unconsciously corrupted. They can be selectively reported. Most of all, 1 time in 20 they can be "statistically significant" by sheer coincidence, and there are a lot of experiments out there.

Unfortunately, Science has this notion that you can never go against an honestly obtained experimental result. So, when someone obtains an experimental result that contradicts the standard model, researchers are faced with a dilemma for resolving their cognitive dissonance: they either have to immediately throw away the standard model, or else attack the experiment - accuse the researchers of dishonesty, or flawed design, or conflict of interest...

Someone once presented me with a new study on the effects of intercessory prayer (that is, people praying for patients who are not told about the prayer), which showed 50% of the prayed-for patients achieving success at in-vitro fertilization, versus 25% of the control group. I liked this claim. It had a nice large effect size. Claims of blatant impossible effects are much more pleasant to deal with than claims of small impossible effects that are "statistically significant".

So I cheerfully said: "I defy the data."

My original phrasing was actually "I deny the data". Nonetheless I said it outright, without apology, and with deliberate insolence. I am keeping my theory; your experiment is wrong.

If an experimental result contradicts the Standard Model, this is an important fact. It needs to be openly acknowledged. An experiment that makes traditionalists want to discard the data - or even an experiment that makes traditionalists very skeptical of the data - should be a high priority for replication. An experiment worth defying should command attention!

But it is not socially acceptable to say, "The hell with your experimental falsification, I'm keeping my theory." So the data has to be defied covertly - by character assassination of the researchers, by sly innuendos, by dire hints of controversy. The data has to be dismissed, excused away, swept under a rug, silently into the dark, because you can't admit you're defying the data. This is not a good way of focusing attention on an anomalous result. This is not a good way to ensure funding for replication attempts.

It would be much better if science had a standard procedure for saying, "I defy the data!" It would be clearly understood that this was a bold act, and someone else in the audience might stand up and say, "Wait a minute, is that data really worth defying?" If a major figure in the field said "I defy the data!", this would be sufficient justification on grant proposals for why the result urgently needed replication. Scientists could say, "I'm holding my breath, waiting for replication," rather than having to take sides immediately in the character-assassination controversy.

Maybe you could even get the media to report that the experiment has been "published but defied". Then the replication, or failure to replicate, would be news. The replicators could get their names in the newspaper, and the negative result could be published in a major journal. If you want replications done, you'll have to offer some incentive.

I would also suggest that when an experiment is defied, the replication must pre-declare a minimum effect size, and attain significance of p<0.01. In extreme cases where claims have been made and shot down before, p<0.001.

Oh, and the prayer study? Soon enough we heard that it had been retracted and was probably fraudulent. But I didn't say fraud. I didn't speculate on how the results might have been obtained. That would have been dismissive. I just stuck my neck out, and nakedly, boldly, without excuses, defied the data.

Addendum: I should have spelled this out explicitly: You can defy the data on one experiment. You can't defy the data on multiple experiments. At that point you either have to relinquish the theory or dismiss the data - point to a design flaw, or refer to an even larger body of experiments that failed to replicate the result, or accuse the researchers of a deliberate hoax, et cetera. But you should not turn around and argue that the theory and the experiment are actually compatible. Why didn't you think of that before you defied the data? Defying the data admits that the data is not compatible with your theory; it sticks your neck way out, so your head can be easily chopped off.