Please pardon the acronyms in the title (expansions forthcoming in the post), I’m attempting to keep my post titles reasonably concise. Which is exactly what the final sentence in the quote below does in relation to the rest of the paragraph preceding it: it restates the whole argument of the paragraph pithily in just five words (bolded in the following quote).
“In the vastness of the cosmos, there must be other civilizations far older and more advanced than ours — so shouldn’t we have been visited? Shouldn’t there be every now and then alien ships in the skies of Earth? There’s nothing impossible in this idea, and no one would be happier than me if we were being visited, but has it happened in fact? What counts is not what sounds plausible, not what we’d like to believe, not what one or two witnesses claim, but only what is supported by hard evidence, rigorously and skeptically examined. Extraordinary claims require extraordinary evidence.”
~Carl Sagan; Cosmos, Episode IV
Those five words have now been further abbreviated to the acronym ECREE due to the discussion I’m about to outline below.
A bit of background: I have been working through another bout of this recurrent bronchitis, and when I am ill I get cranky and prone to engaging in nitpicking of the most extraordinarily pedantic kind. In order to avoid raising hostility levels within the family to Critical, at such times I prefer to go and play around online in various sceptical forums so that I keep my extreme nitpicking on the internet where it belongs. Skeptical forums specialising in the paranormal are particularly good value for the nitpicking because there are so many folks who think they’re sceptics because they have rejected their Christian upbringing but who nonetheless buy into a mindboggling array of New Age woo claims which cannot be verified by hard evidence rigorously and skeptically examined.
So I was pleased to hit absolute semantic quibbling gold on the JREF forums with a glorious example of a Thread O’ Doom (TOD) that has been continuing for months now: 80+ pages of a contrarian arguing that, unless one can give hir a perfect definition of “extraordinary claims” and “extraordinary evidence”, Sagan’s popular maxim should be denounced as nonsense and abandoned in skeptical arguments. Many forum members have given hir what would seem to be perfectly acceptable answers to anyone genuinely seeking to understand the phrase, yet hir response is always to reject these explanations, usually by finding a way to recast anything at all that is claimed to be “extraaordinary” as actually “ordinary” (this has led to hir appearing to tie hirself up in some astonishingly Gordian knots, but whenever sie gets too entangled sie just goes back to an earlier post of hirs as if the last pages of discussion have not occurred).
Annnnyyyywwwaaaayyyy: this TOD has been my procrastination tool of choice for the last few days of also being busy with a very many tedious tasks, and being at my most nitpickety, I discovered one angle that has not yet been tried. I have no real expectation that my effort will be accepted by the contrarian crew, but this particular TOD has largely become SIWOTI as performance art for the sake of the lurkers, so I wanted my turn to exhibit. Because the JREF forums take a while to approve one’s application to join, I had plenty of time to work on my nitpickery. If anyone is at all interested in seeing just how excruciatingly pedantic I can get, you can read my contribution here.
A quick summary of my more detailed argument on the thread (from the post linked above and my supplementary clarifying post) below especially for the benefit of those who may have come across other contrarians challenging ECREE with similar arguments:
- Maxims are used as summary-phrases that encapsulate more complex concepts in a memorable soundbite.
- So long as the background concept is not nonsense, the summary phrase cannot be nonsense either.
- ECREE summarises the broader concept of applying the scientific method as the necessary test standard for scientifically contentious claims.
- The scientific method of critical analysis of contentious claims is not nonsense, therefore ECREE cannot be nonsense.
One thing the discussion did reveal was that for many skeptics, already immersed in a worldview of holding evidence to the standards required to withstand scrutiny by the scientific method before they judge it to be credible evidence, their understanding of ECREE is holistic and heuristic rather than having a neat explication ready for anybody who challenges ECREE. Frankly, there was quite a bit of floundering around in circular arguments, which was probably the contrarian’s intent when sie attempted to frame ECREE as saying something about any claims provided by anybody in the world about anything instead of sticking to what is absolutely clear from the context: that Sagan was talking specifically about analysing scientifically contentious claims.
For anyone still hung up on the exact definitions of “extraordinary claims (EC)” and “extraordinary evidence (EE)” (the contrarian has demanded simple and objective definitions) I’ve taken a go at that as well.
- Let EC be “scientifically contentious claims”.
- Let EE be “evidence able to settle these contentions when critically scrutinised via the scientific method”
- Let “scientifically contentious claims” = “claims whose proponents seek scientific credibility but whose supporting evidence has not yet been able to withstand critical scrutiny by the scientific method”
- Let “supporting evidence able to withstand critical scrutiny by the scientific method” = “evidence sufficient to reject the relevant null hypotheses at a statistical probability equal to or greater than the 95% confidence level”.
This leads to a more precise rephrasing of ECREE as
Scientifically contentious claims require evidence able to settle these contentions when critically scrutinised via the scientific method.
I’m fairly happy with this as an intermediate-level explication of ECREE when ECREE is challenged as confusing. It places ECREE firmly within the context of scientific credibility, which was always Sagan’s point, and is not so overly-precise that it is totally unwieldy. My points (3) and (4) above are included to demonstrate that increased precision exists within nested explications of terms used in my intermediate explication should they be required.
I still prefer Sagan’s elegant and poetic five word summation though. It’s much easier to remember for a start.
I’m sure you can see why I try not to burden my family with dealing with me when I go fully into Extreme Nitpicker mode 🙂
Categories: culture wars, education, skepticism
Hey Dad – this time it’s all YOUR fault 
Hoyden About Town 
Phew. I enjoyed that.
Heh – and that was my abbreviated version! I haven’t yet regaled you with the detailed analysis I have done on that paragraph (awaiting the inevitable challenge for me to demonstrate exactly how we can be so sure that Sagan’s point was all about the necessity for applying the scientific method to contentious claims) and my demonstration of how ECREE leads naturally to a Socratic dialogue on the nature of how science works!
As predicted, the inevitable challenge from the contrarian over at JREF forums, to demonstrate how ECREE encapsulates the requirements for scientific credibility, arrived. Here’s what I prepared earlier:
I anticipate sie will remain unconvinced.
I think so. I suspect that sie is enjoying the hell out of themselves and hasn’t had this much fun for ages. I hope that you are enjoying your extreme nit pickiness as well. That is one thing that I really enjoy about TOD – you can come back the next day once you’ve thought up a really great comeback and stir it all up again. Unless some other bugger got there before you, but sometimes even then.
Quite possibly, although it seems that this particular contrarian has also been involved in another TOD for even longer, so I don’t think sie’s enjoying hirself any more than usual. Sie posts example after example of dodgy photos and personal testimonials which sie wants to accepted as credible evidence for AFCs (Alien Flying Craft) . Sie also keeps on wanting to equivocate between AFCs and UFOs, but as we all know there are plenty of Unidentified Flying Objects sightings, such sightings occur most days of any given week, there’s also plenty of mundane explanations, and no supporting evidence has yet been shown to support ETs as the most plausible explanation.)
Interesting, of course, that attempting to define all types of evidence one could possible present as always “ordinary evidence” and never “extraordinary evidence” would, in hir mind, make it justifiable to argue that any evidence of AFCs, including dodgy photos and personal testimonials, should be persuasive evidence.
This indicates an inability or dogged unwillingness to acknowledge that aphorisms used in informal discussions have no influence at all the standards of formal analysis anyway, so even if sie shoots down ECREE as a popular phrase in skeptical circles the background concept will remain intact: hir claims can only garner scientific credibility if the supporting evidence meets scientific standards.
I wouldn’t call ”evidence sufficient to reject the relevant null hypotheses at a statistical probability equal to or greater than the 95% confidence level” extrordinary evidence. This could still have you wrong one time in twenty. Considering how many extraordinary claims there are around you could be believing many strange things.
I would consider the 95% confidence level useful evidence for ordinary claims but not convincing evidence of extraordinary claims.. Ordinary claims would be consistent with other scientific theories for which there is good evidence such as “eating oranges reduces the chances of getting colds” or “stretching before exercise prevents injury”. If you say you’ve built a machine that breaks conservation of energy it would need much more evidence than gave a 95% confidence interval to convince me.
The likelihood of the initial “extraordinary claim” effects the assesment of the results of the statistical test. This is similar to the statistics behind positive predictive value of medical testing. Say you have a test for a rare disease that only affects one in a thousand people. Suppose the test gives you a false positive one time in 20 (equivalent to a 95% confidence interval). If someone tests positive then their chance of having the disease (equivalent to the chance that the claim is correct) is still only one in 50. The difficulty with this argument is in quantifying the likelihood of the initial claim.
Sally, I agree that meeting a 95% confidence level is very much a minimum standard for scientific evidence, but that’s exactly Sagan’s point. If one’s evidence cannot even meet the 95% confidence level, then one’s claim has no scientific credibility.
Since it’s a statement made as part of a public outreach science education program, aimed at people who have mostly never studied statistics at all, I don’t at all wonder that it fails to go into the nitty-gritty of experimental design and the ranking of alternative hypotheses. I don’t think it should, either, because when the goal is science education, one has to go in steps that take the audience’s level of understanding into account.
Scientists who already apply the viewpoint of the rigorously peer-reviewed scientific method do, I think, often forget just how extraordinary this form of critical analysis is in comparison to other methods of evaluating claims. Insisting on applying a rigorous scientific standard is what is extraordinary about both the claim (is it consistent with science we already understand or not?) and the evidence (does it support the proposed explanation for the observed phenomenon to a significant confidence level?) in ECREE.
Once the audience shows that they’ve grokked ECREE thus far, then it’s time to explain how, even if one’s evidence manages to meet the 95% confidence level, that should a competing hypothesis have supporting evidence that meets the 98% confidence level, or perhaps even the 99.99% confidence level, then that competing hypothesis is even more scientifically credible. Then you can start getting into margins of error and false positives, and then you can drill even deeper. But starting with those details is just a wall o’ text. The explanation has to build the foundation first before going further.
What I do find problematic about ECREE is that when it comes to social sciences the choice of null hypothesis is essentially arbitrary, and has in practice been a default/privileged perspective. So ECREE can be used as an excuse to completely ignore lived experience (correctly so, within its own terms, even as you expand and restate them) – and so you get experiments like the recent sexual proposition one to prove beyond all reasonable doubt something that was – from a non-privileged or even vaguely empathic standpoint – completely obvious.
I’d argue that “fat and health” has ended up caught in the same sort of ECREE trap, in the medical sciences, as some bad but not caught-at-the-time research ends up setting an unjustified null hypothesis. The mass of the electron, I believe, ended up in much the same sort of trap for the physical sciences, but I can’t find the reference for that right now – I think I originally read it as a Feynman story, but I can’t remember which.
(shorter: ECREE is great as long as you pick the right null hypothesis, but how do you tell when you’ve picked the wrong one?)
cim, while I think you are making some good points about experimental design there, I’m not sure that they apply directly to ECREE itself. Especially with regard to the recent sexual proposition study, I actually view that one as a triumph for the scientific method, because a well-designed experiment has been able to show that a prior badly-designed experiment drew invalid conclusions from its experimental results.
P.S. to get extra nitpicky here, I don’t think there was anything wrong with the null hypothesis in the first badly-designed study, either. Their hypothesis was that men and women would agree to the proposition at different rates, so the null hypothesis would have been that both men and women would agree at similar rates. Their experimental data led them to reject the null hypothesis, but this wasn’t a problem with the choice of null hypothesis.
What was problematic about that study was the drawing of unwarranted conclusions from data that strongly supported their hypothesis. The second study challenging their
interpretationconclusions showed why those conclusions were indeed unwarranted by taking into account confounding variables which the original study had not even considered – let alone controlled for. (that’s the bad experimental design).tigtog: Oh, absolutely, that experiment was definitely a good demonstration of the scientific method, and a well-designed experiment. But – take fat and health as the example – the null hypothesis used right now is essentially that fat strongly causes bad health. There’s seen no need to question this hypothesis in a lot of papers, just as one wouldn’t generally start a physics experiment with a preliminary test that gravity was still working as expected. It’s “established science”.
So experiments confirming the null hypothesis get subjected to generally less scrutiny over possible flaws in their methodology than ones that fail to confirm it. The ones confirming the null hypothesis are only making an “ordinary” claim (just as no-one would expect me to do a full scientific analysis if I claimed I wasn’t psychic, but would if I claimed that gravitational attraction is independent of mass)
Ordinarily, this works quite well, but it breaks down when previous experiments have created a new null hypothesis that isn’t a real null (gravitational attraction does not exist) but is nevertheless less strongly supported by the evidence than a significantly different hypothesis. (It works fine for incremental – Newton to Einstein, for instance – improvements on hypotheses, of course, but not for complete changes)
@cim, you seem to be using “null hypothesis” contrary to my understanding of the term, and that’s making it difficult for me to follow your arguments. I see “fat strongly causes bad health” as an underlying hypothesis, not as a null hypothesis for just about any experiment I read about.
ETA: if “fat strongly causes bad health” was being used as the null hypothesis for an experiment, that would make the hypothesis being tested “fat is neutral with respect to health”. That’s actually exactly the sort of experiments we do want to see more of, isn’t it?
tigtog,
I may have taken the ECREE in a different context. I tend to view the 95% confidence interval published study as “ordinary evidence”, and less as not really evidence at all. Although some types of less formal evidence may still be stronger depending on the situation.
I see ECREE being used to explain why I demand much higher levels of evidence for “your” bizarre claims than I would for “my” completely plausible ones. Which does seem rather arrogant unless there is a good objective definition of what an extraordinary claim is.
Also cim I would have considered many of Einstein’s theories to be extraordinary claims rather than incremental changes, even though the changes to measured quantities in most cases were small. The confidence level on the accumulated evidence (with Newton as the null hypothesis) would be huge by now though.
Definitely ECREE is used as a garbage filter, but especially in the way that Sagan actually used it (versus how it’s often used now), the “you must be this high to ride” standard he was laying out for what would make evidence for ET visitations extraordinary would be that it was “hard evidence, rigorously and skeptically examined”. That’s just laying out a minimum standard of scientific credibility for evidence of ET visitations to be considered even examining seriously. ECREE is definitely extensible to other scientifically contentious claims than ET visitations, but the dividing line is so very clear in that case because ET claims so far have pretty much nothing that meets the basic evidence standard (verifiable, repeatable) that would even allow it to be tested further by the scientific method.
What has been demonstrated in the thread over at JREF is that when arguing with a ET creduloid about how ECREE encapsulates why hir blurry photos and anecdotal evidence have no scientific credibility, is that arrogance is exactly what skeptics are being accused of by using ECREE as a mantra when arguing with ET creduloids. When we stick to arguing that Sagan, due to the sentences surrounding ECREE in his writings, was clearly using “extraordinary” to compare the scientific method as a whole to compare to other standards for evaluating claims, that’s when the UT creduloid starts to paint hirself into a corner and tries to jump back to simple semantic quibbling again.
Sagan coined ECREE to sum up the importance of applying the scientific method to contentious claims for an audience that didn’t know much about how science works. He never meant it to be used the way that some folks stretch it now. If ECREE is stretched too far, then maybe it does become nonsense. But when using it exactly as Sagan did, as a garbage filter for claims about ET visitation, it’s clearly not nonsense.
tigtog: Yes, you’re right – “underlying hypothesis” probably is much closer to what I mean. So I think there can be a problem with ECREE if the underlying hypotheses aren’t properly acknowledged, since it’s easy to treat experiments as requiring different levels of quality – and flaws in experimental methods as having different importance – depending on whether the experiment appears to confirm or deny current scientific thinking.
Here’s the Feynman essay I was thinking of – about half way down he talks about Millikan and electron charge (not mass, as I misremembered), and how the assumption that the previous experiments were largely correct caused problems in actually getting to the correct answer
http://www.lhup.edu/~DSIMANEK/cargocul.htm
Unlike Feynman, I don’t believe for a second that scientists have stopped making that sort of mistake.
And yes, more experiments of that sort would be good, definitely – but I think there’s a definite weakness of the current scientific process – no idea how to solve it – in that it would give those experiments far more scrutiny than one that reinforced the status quo thinking, despite the status quo not being one that you would get to nowadays if you started from scratch with the experiments.
@cim – I too don’t believe that scientists are immune from confirmation bias, but at least the scientific method of provisional acceptance of data/hypotheses only until there is better data/hypotheses is self-correcting for confirmation bias to a better extent than other systems. Ssomebody with a different bias than one’s own can clearly demonstrate with a better designed experiment that one’s own conclusions have been confounded in one way or another. I can’t really think of a another system of fact-finding which allows for instances of confirmation bias in forming conclusions to be so thoroughly challenged and rebutted when they does occur.