Article written by :: (RSS)

tigtog (aka Viv) is the founder of this blog. She lives in Sydney, Australia: husband, 2 kids, cat, house, garden, just enough wine-racks and (sigh) far too few bookshelves.

This author has written 3457 posts for Hoyden About Town. Read more about tigtog »

16 responses to “SIWOTI: Sagan’s ECREE maxim is NOT nonsense”

  1. orlando

    Phew. I enjoyed that.

  2. tigtog

    Heh – and that was my abbreviated version! I haven’t yet regaled you with the detailed analysis I have done on that paragraph (awaiting the inevitable challenge for me to demonstrate exactly how we can be so sure that Sagan’s point was all about the necessity for applying the scientific method to contentious claims) and my demonstration of how ECREE leads naturally to a Socratic dialogue on the nature of how science works!

  3. tigtog

    As predicted, the inevitable challenge from the contrarian over at JREF forums, to demonstrate how ECREE encapsulates the requirements for scientific credibility, arrived. Here’s what I prepared earlier:

    *********begin cut and paste*********
    +++++++++++++++++++++++++++++++++++++++++
    SAVE FOR INEVITABLE CHALLENGE TO DEMONSTRATE IT
    +++++++++++++++++++++++++++++++++++++++++

    As blobru’s original contribution to this thread first demonstrated, ECREE is a summary-phrase encapsulating a full paragraph. The pedagogical purpose of this particular paragraph, in the context of a scientist explaining how science comes to know what it knows about the cosmos, is to explain what makes the scientific method different from and superior to other ways of evaluating claims:

    “In the vastness of the cosmos, there must be other civilizations far older and more advanced than ours — so shouldn’t we have been visited? Shouldn’t there be every now and then alien ships in the skies of Earth? There’s nothing impossible in this idea, and no one would be happier than me if we were being visited, but has it happened in fact? What counts is not what sounds plausible, not what we’d like to believe, not what one or two witnesses claim, but only what is supported by hard evidence, rigorously and skeptically examined. Extraordinary claims require extraordinary evidence.” Carl Sagan; Cosmos

    So, in the context of that full paragraph, we see that ECREE is a summary of the paragraph of which it forms the final sentence. We see this clearly because the structure of the paragraph is the classic presentation technique of Tell ‘Em Thrice, which you may be more familiar with in this formulation: tell ‘em what you’re gonna tell ‘em, tell ‘em, then tell ‘em what you told ‘em.

    So, applying the Tell ‘Em Thrice guideline to Sagan’s paragraph:

    1. tell ‘em what you’re gonna tell ‘em
    “In the vastness of the cosmos, there must be other civilizations far older and more advanced than ours — so shouldn’t we have been visited? Shouldn’t there be every now and then alien ships in the skies of Earth? There’s nothing impossible in this idea, and no one would be happier than me if we were being visited, but has it happened in fact?”
    i.e. There’s lots and lots of claims out there about extraterrestrial visitations to our planet. Can we test these claims to see whether they are true?
    2. tell ‘em:
    “What counts is not what sounds plausible, not what we’d like to believe, not what one or two witnesses claim, but only what is supported by hard evidence, rigorously and skeptically examined.”
    i.e. Yes, there is a way to test these claims scientifically, and when supporting evidence is able to withstand the scrutiny of the scientific method, then the claim gains scientific credibility.
    3. tell ‘em what you told ‘em.
    “Extraordinary claims require extraordinary evidence.”
    i.e. Those claims of ET visitations I just told you about? Those are extraordinary claims because evidence provided in support has not yet withstood scrutiny via the scientific method (which is a shorter name for that hard evidence, rigorously and skeptically examined that I just told you about) . Evidence that can withstand the extraordinary scrutiny of the scientific method is, therefore, extraordinary evidence.

    So, ECREE is just the summation of the second part of the paragraph that reworks the earlier phrases of Hume and La Place, regarding the necessity for using the scientific method to test whether a claim provides the best explanation for an observation, into a pithier and more memorably poetic phrase. We can also be sure that this was Sagan’s meaning for the phrase because when he used it again in his later work he used it in exactly the same way – to summarise a longer description of the necessity for applying the scientific method.

    ECREE works as an aide memoire because it evokes the larger, more detailed explanation. It works especially well as an aide memoire about the scientific method because it can be applied to other extraordinary claims, not just claims about ET visitations. ECREE itself doesn’t have to carry the burden of explaining every last detail of the scientific method, since ECREE implicitly uses the scientific method as its standard because it is summarising what has already been told to the audience explicitly.

    Since the scientific method is not nonsense, and ECREE is simply a summary-phrase regarding the the necessity for claims of unusual events to be scrutinised using the scientific method, then ECREE cannot be nonsensical.

    *********end cut and paste*********

    I anticipate sie will remain unconvinced.

  4. Mindy

    I anticipate sie will remain unconvinced.

    I think so. I suspect that sie is enjoying the hell out of themselves and hasn’t had this much fun for ages. I hope that you are enjoying your extreme nit pickiness as well. That is one thing that I really enjoy about TOD – you can come back the next day once you’ve thought up a really great comeback and stir it all up again. Unless some other bugger got there before you, but sometimes even then.

  5. tigtog

    I suspect that sie is enjoying the hell out of themselves and hasn’t had this much fun for ages.

    Quite possibly, although it seems that this particular contrarian has also been involved in another TOD for even longer, so I don’t think sie’s enjoying hirself any more than usual. Sie posts example after example of dodgy photos and personal testimonials which sie wants to accepted as credible evidence for AFCs (Alien Flying Craft) . Sie also keeps on wanting to equivocate between AFCs and UFOs, but as we all know there are plenty of Unidentified Flying Objects sightings, such sightings occur most days of any given week, there’s also plenty of mundane explanations, and no supporting evidence has yet been shown to support ETs as the most plausible explanation.)

    Interesting, of course, that attempting to define all types of evidence one could possible present as always “ordinary evidence” and never “extraordinary evidence” would, in hir mind, make it justifiable to argue that any evidence of AFCs, including dodgy photos and personal testimonials, should be persuasive evidence.

    This indicates an inability or dogged unwillingness to acknowledge that aphorisms used in informal discussions have no influence at all the standards of formal analysis anyway, so even if sie shoots down ECREE as a popular phrase in skeptical circles the background concept will remain intact: hir claims can only garner scientific credibility if the supporting evidence meets scientific standards.

  6. Sally

    I wouldn’t call “evidence sufficient to reject the relevant null hypotheses at a statistical probability equal to or greater than the 95% confidence level” extrordinary evidence. This could still have you wrong one time in twenty. Considering how many extraordinary claims there are around you could be believing many strange things.

    I would consider the 95% confidence level useful evidence for ordinary claims but not convincing evidence of extraordinary claims.. Ordinary claims would be consistent with other scientific theories for which there is good evidence such as “eating oranges reduces the chances of getting colds” or “stretching before exercise prevents injury”. If you say you’ve built a machine that breaks conservation of energy it would need much more evidence than gave a 95% confidence interval to convince me.

    The likelihood of the initial “extraordinary claim” effects the assesment of the results of the statistical test. This is similar to the statistics behind positive predictive value of medical testing. Say you have a test for a rare disease that only affects one in a thousand people. Suppose the test gives you a false positive one time in 20 (equivalent to a 95% confidence interval). If someone tests positive then their chance of having the disease (equivalent to the chance that the claim is correct) is still only one in 50. The difficulty with this argument is in quantifying the likelihood of the initial claim.

  7. cim

    What I do find problematic about ECREE is that when it comes to social sciences the choice of null hypothesis is essentially arbitrary, and has in practice been a default/privileged perspective. So ECREE can be used as an excuse to completely ignore lived experience (correctly so, within its own terms, even as you expand and restate them) – and so you get experiments like the recent sexual proposition one to prove beyond all reasonable doubt something that was – from a non-privileged or even vaguely empathic standpoint – completely obvious.

    I’d argue that “fat and health” has ended up caught in the same sort of ECREE trap, in the medical sciences, as some bad but not caught-at-the-time research ends up setting an unjustified null hypothesis. The mass of the electron, I believe, ended up in much the same sort of trap for the physical sciences, but I can’t find the reference for that right now – I think I originally read it as a Feynman story, but I can’t remember which.

    (shorter: ECREE is great as long as you pick the right null hypothesis, but how do you tell when you’ve picked the wrong one?)

  8. tigtog

    Sally, I agree that meeting a 95% confidence level is very much a minimum standard for scientific evidence, but that’s exactly Sagan’s point. If one’s evidence cannot even meet the 95% confidence level, then one’s claim has no scientific credibility.

    Since it’s a statement made as part of a public outreach science education program, aimed at people who have mostly never studied statistics at all, I don’t at all wonder that it fails to go into the nitty-gritty of experimental design and the ranking of alternative hypotheses. I don’t think it should, either, because when the goal is science education, one has to go in steps that take the audience’s level of understanding into account.

    Scientists who already apply the viewpoint of the rigorously peer-reviewed scientific method do, I think, often forget just how extraordinary this form of critical analysis is in comparison to other methods of evaluating claims. Insisting on applying a rigorous scientific standard is what is extraordinary about both the claim (is it consistent with science we already understand or not?) and the evidence (does it support the proposed explanation for the observed phenomenon to a significant confidence level?) in ECREE.

    Once the audience shows that they’ve grokked ECREE thus far, then it’s time to explain how, even if one’s evidence manages to meet the 95% confidence level, that should a competing hypothesis have supporting evidence that meets the 98% confidence level, or perhaps even the 99.99% confidence level, then that competing hypothesis is even more scientifically credible. Then you can start getting into margins of error and false positives, and then you can drill even deeper. But starting with those details is just a wall o’ text. The explanation has to build the foundation first before going further.

  9. tigtog

    cim, while I think you are making some good points about experimental design there, I’m not sure that they apply directly to ECREE itself. Especially with regard to the recent sexual proposition study, I actually view that one as a triumph for the scientific method, because a well-designed experiment has been able to show that a prior badly-designed experiment drew invalid conclusions from its experimental results.

  10. tigtog

    P.S. to get extra nitpicky here, I don’t think there was anything wrong with the null hypothesis in the first badly-designed study, either. Their hypothesis was that men and women would agree to the proposition at different rates, so the null hypothesis would have been that both men and women would agree at similar rates. Their experimental data led them to reject the null hypothesis, but this wasn’t a problem with the choice of null hypothesis.

    What was problematic about that study was the drawing of unwarranted conclusions from data that strongly supported their hypothesis. The second study challenging their interpretation conclusions showed why those conclusions were indeed unwarranted by taking into account confounding variables which the original study had not even considered – let alone controlled for. (that’s the bad experimental design).

  11. cim

    tigtog: Oh, absolutely, that experiment was definitely a good demonstration of the scientific method, and a well-designed experiment. But – take fat and health as the example – the null hypothesis used right now is essentially that fat strongly causes bad health. There’s seen no need to question this hypothesis in a lot of papers, just as one wouldn’t generally start a physics experiment with a preliminary test that gravity was still working as expected. It’s “established science”.

    So experiments confirming the null hypothesis get subjected to generally less scrutiny over possible flaws in their methodology than ones that fail to confirm it. The ones confirming the null hypothesis are only making an “ordinary” claim (just as no-one would expect me to do a full scientific analysis if I claimed I wasn’t psychic, but would if I claimed that gravitational attraction is independent of mass)

    Ordinarily, this works quite well, but it breaks down when previous experiments have created a new null hypothesis that isn’t a real null (gravitational attraction does not exist) but is nevertheless less strongly supported by the evidence than a significantly different hypothesis. (It works fine for incremental – Newton to Einstein, for instance – improvements on hypotheses, of course, but not for complete changes)

  12. Sally

    tigtog,
    I may have taken the ECREE in a different context. I tend to view the 95% confidence interval published study as “ordinary evidence”, and less as not really evidence at all. Although some types of less formal evidence may still be stronger depending on the situation.

    I see ECREE being used to explain why I demand much higher levels of evidence for “your” bizarre claims than I would for “my” completely plausible ones. Which does seem rather arrogant unless there is a good objective definition of what an extraordinary claim is.

    Also cim I would have considered many of Einstein’s theories to be extraordinary claims rather than incremental changes, even though the changes to measured quantities in most cases were small. The confidence level on the accumulated evidence (with Newton as the null hypothesis) would be huge by now though.

  13. tigtog

    I see ECREE being used to explain why I demand much higher levels of evidence for “your” bizarre claims than I would for “my” completely plausible ones. Which does seem rather arrogant unless there is a good objective definition of what an extraordinary claim is.

    Definitely ECREE is used as a garbage filter, but especially in the way that Sagan actually used it (versus how it’s often used now), the “you must be this high to ride” standard he was laying out for what would make evidence for ET visitations extraordinary would be that it was “hard evidence, rigorously and skeptically examined”. That’s just laying out a minimum standard of scientific credibility for evidence of ET visitations to be considered even examining seriously. ECREE is definitely extensible to other scientifically contentious claims than ET visitations, but the dividing line is so very clear in that case because ET claims so far have pretty much nothing that meets the basic evidence standard (verifiable, repeatable) that would even allow it to be tested further by the scientific method.

    What has been demonstrated in the thread over at JREF is that when arguing with a ET creduloid about how ECREE encapsulates why hir blurry photos and anecdotal evidence have no scientific credibility, is that arrogance is exactly what skeptics are being accused of by using ECREE as a mantra when arguing with ET creduloids. When we stick to arguing that Sagan, due to the sentences surrounding ECREE in his writings, was clearly using “extraordinary” to compare the scientific method as a whole to compare to other standards for evaluating claims, that’s when the UT creduloid starts to paint hirself into a corner and tries to jump back to simple semantic quibbling again.

    Sagan coined ECREE to sum up the importance of applying the scientific method to contentious claims for an audience that didn’t know much about how science works. He never meant it to be used the way that some folks stretch it now. If ECREE is stretched too far, then maybe it does become nonsense. But when using it exactly as Sagan did, as a garbage filter for claims about ET visitation, it’s clearly not nonsense.

  14. tigtog

    @cim, you seem to be using “null hypothesis” contrary to my understanding of the term, and that’s making it difficult for me to follow your arguments. I see “fat strongly causes bad health” as an underlying hypothesis, not as a null hypothesis for just about any experiment I read about.

    ETA: if “fat strongly causes bad health” was being used as the null hypothesis for an experiment, that would make the hypothesis being tested “fat is neutral with respect to health”. That’s actually exactly the sort of experiments we do want to see more of, isn’t it?

  15. cim

    tigtog: Yes, you’re right – “underlying hypothesis” probably is much closer to what I mean. So I think there can be a problem with ECREE if the underlying hypotheses aren’t properly acknowledged, since it’s easy to treat experiments as requiring different levels of quality – and flaws in experimental methods as having different importance – depending on whether the experiment appears to confirm or deny current scientific thinking.

    Here’s the Feynman essay I was thinking of – about half way down he talks about Millikan and electron charge (not mass, as I misremembered), and how the assumption that the previous experiments were largely correct caused problems in actually getting to the correct answer
    http://www.lhup.edu/~DSIMANEK/cargocul.htm
    Unlike Feynman, I don’t believe for a second that scientists have stopped making that sort of mistake.

    And yes, more experiments of that sort would be good, definitely – but I think there’s a definite weakness of the current scientific process – no idea how to solve it – in that it would give those experiments far more scrutiny than one that reinforced the status quo thinking, despite the status quo not being one that you would get to nowadays if you started from scratch with the experiments.

  16. tigtog

    @cim – I too don’t believe that scientists are immune from confirmation bias, but at least the scientific method of provisional acceptance of data/hypotheses only until there is better data/hypotheses is self-correcting for confirmation bias to a better extent than other systems. Ssomebody with a different bias than one’s own can clearly demonstrate with a better designed experiment that one’s own conclusions have been confounded in one way or another. I can’t really think of a another system of fact-finding which allows for instances of confirmation bias in forming conclusions to be so thoroughly challenged and rebutted when they does occur.

The commenting period has expired for this post. If you wish to re-open the discussion, please do so in the latest Open Thread.