The trouble with Statistics
February 23rd, 2010It is my belief that the recession has given the Irish media their best story in decades. They have explored the horror from every angle. The story is so powerful: A poor country that rose high and like the Greek character Icarus, believed in their own hype, got too close to the sun and tumbled down to earth. That’s the story; all you need now are a few statistics to “prove” that it’s true.
Every day new statistics are released, whether it’s the CSO, IBEC, the ESRI, innumerable stockbrokers, economists, company accounts, stock markets, international markets… there’s always a statistic to be found that “proves” your story. The media are the storytellers of Ireland and they have unashamedly told their story and still do. The trouble is it’s just not true. Ireland is a well respected country that has continually outperformed its competitors. Even through the recession, Irish Government bonds were snapped-up by international investors who recognised the underlying strength of the Irish economy.
Every week I read the International Herald Tribune, The Wall Street Journal and The Financial Times and there’s barely a mention about Ireland, except in praise. So let’s be clear, the trouble with statistics is that they mean nothing. It is the story that gives the statistics meaning and its past time the media found a new story, lest they lose the last shred of credibility they still hold with the people concerned with Ireland Inc. What’s that saying? “You should never let the facts get in the way of a good story”?! Below is a tale by William Bastland that illustrates my points exactly:
“Once upon a time, Orkney had one of the highest teenage pregnancy rates in Scotland. Scotland itself is high in the international league. So health workers in Orkney tried something new. They began talking to young people about sex in terms of relationships, not only mechanics. They also made condoms easily available because in a small community the shopkeeper might just be your auntie. Then came data showing that Orkney’s teenage pregnancy rate had dramatically halved.

All this was widely reported last summer. Convinced of the happy ending?
Let me introduce you to a radical and highly complex, story-wrecking mathematical insight. Ready? Numbers go up and down. All right, I lied about the complexity. I also lied about the maths. This is not really mathematics, it is everyday life. In life, things do not happen with perfect regularity. Some days, or years, there are more, sometimes fewer. And it’s not radical either. Everyone knows it. Until then they tell themselves stories. The truth in the Orkney case is that the number of teenage pregnancies goes up and down, and ups and downs may have nothing to do with the stories told to explain them. When new data came out since last summer, it was not reported. Or at least I can’t find any reference to it. Perhaps it didn’t fit the story. Here is the data in a little more detail.

The figure reported last summer, the most recent then available, really was lower than in 1994. But it’s clear that the numbers go sharply up and down, much more so in a small community than the larger one of Scotland as a whole. What happened when more recent data came out? This:

The same as usual. What had gone down, briefly, went up, just as what sometimes jumps up often tends to come down. You can talk about life, relationships, morality – and so we should. But there’s another knowledge of life that statistical thinking sees and other mind-sets often miss. This second kind of thinking relies not particularly on maths, but imagination, imagination for what can go wrong with narratives that describe the way one thing leads to another. This is the best kind of story-telling; story-telling wise to the ways that stories might mislead. I’m quite sure that no-one lied in telling the story in Orkney. I think they were just wrong. I suspect that all concerned, including journalists, found the story of relationship advice and condoms plausible enough to convince them that the numbers they looked at in those two salient years – a beginning and an end – told them something important. And the data does not prove that their new approach to sex education is wrong. It might have benefits that these statistics don’t capture.
Nevertheless, a plausible explanation for change helps convince people that the change really occurred. A plausible story for why it happened persuades us that it did. The explanation becomes the story. But “it happened because…” can disguise the fact that it didn’t really happen at all, or at least not the way we think it did. The truth is that we still don’t really know if there is an underlying change in the pattern of teenage pregnancies in Orkney. There might be. But it’s not evident yet. Another recent example appeared in an editorial in a serious national newspaper – which had better remain nameless – about the revival of marriage, with a reflective account of why this had happened deduced from a short run of recent data. Five weeks later the same newspaper found itself reporting that marriage had in fact, according to new data, fallen to its lowest level in one hundred and eleven years.
Horizontal tennis
Another is the way that both sides in the climate change argument have seized on single-year fluctuations to the extent of arctic sea ice as “proof” that trends are going further, faster, or in the other direction, or whatever. Anyone might guess that such numbers go up and down. But even serious people ignore or forget this in the haste to tell a story. The real difficulty in almost all these cases is to work out how long you have to look at the data before being confident that change is sustained.
Observing the excitability at life’s natural yo-yo, you might wonder if parts of politics, journalism, even sometimes science, resemble nothing so much as an insane commentary on a game of (horizontal) tennis, in which it is assumed that whatever just happened tells us all we need to know, as if whoever just hit the ball must be winning. The underlying trends and often slow nature of real change are lost in a frantic effort by all sides to grab at any short-term snippet of data and claim support for their beliefs or policies, to tell stories with an instant moral”.
I call on the Irish media to start telling us a new story (you’ll find plenty of statistics to back you up), because I’m really tired of the Icarus story!

