Lies! Damn Lies! And Statistics! :D

I have long been fascinated with social statistics, and for some time, business statistics or economic statistics have been a major preoccupation -- although to be fair, I spend plenty of time with census and health statistics too!

It's not just because I like to derive some feeling of control from them, you know, are things really that bad, are things really that good. For some stupid reason, "things" are always reduced to economic activity in my mind.

Social statistics are not just a form of clairvoyance and anxiety management. I am also fascinated by them as a cultural product. The people that report them and consume them use them to gauge reality, there is a whole social apparatus that is dedicated to their production and distribution. All too often, I am surprised by the way people use them not to discern reality, but as talismans to ward off fears of social ills, often with little or no understanding of what these number mean.

Today for example, The Institute for Supply Management released its monthly "Report on Business" Index for non-manufacturing industries. The basic measure attempts to divine whether business activity is expanding or contracting relative to the previous month. A score below 50 represent overall contraction from the previous month, a score above 50 represents overall business expansion. The report for may is summarized as, "

The NMI (Non-Manufacturing Index) registered 44 percent in May, 0.3 percentage point higher than the 43.7 percent registered in April, indicating contraction in the non-manufacturing sector for the eighth consecutive month, but at a slightly slower rate.

Now, generally everyone on the various sites I read interprets this to mean what the ISM implies it means, that overall business activity is contracting but not as rapidly as it had been the previous month. The take home message, the recession is losing steam, or, to paraphrase one blogger, it turns out that the aircraft carrier that is the US economy cannot turn on a dime [note the implied optimism, that the aircraft carrier is turning!]

But can we have any confidence in the validity or the reality behind such an interpretation of this number? Actually, no. The ISM index is compromised of a number of measures taken from a questionnaire sent to a panel of 300 purchasing and supply managers from around the country. As near as I can tell from the website, the sample is not representative of all, but even if it is a representative sample, which requires a precise sampling strategy, it is a relatively small group of a much, much larger population of purchasing and supply managers around the country.

So what you ask? Well, the statistics taken from the ISM represent business activity for these 300 members of the panel, but it is only an approximation of the real value in the real world (assuming the sample is representative of the whole). The real world number lies either somewhat higher than the survey statistic or somewhat lower -- and, if the sample is representative of the whole, this range of error is knowable with a fairly high probability of certainty (say 95 or 99 percent). So, the ISM misleads, or better, the ISM ritually reproduces an economic talisman that can give a sense of where we might be going in terms of economic activity in the past month. But we cannot know from this statistic where we are going. In point of fact there is an equal chance that the contraction of business activity is actually slightly worse than last month, as much as it is to be the case that it is slightly better! We'd be given this range and more information about the sample if this statistic was, in fact, intended to be used as a indicator or reality. That we are not given this information suggests to me that its purpose is for clairvoyance and other anxiety management purposes.

Of course, if the sample of 300 executives is a non-random, convenience sample ... well all bet are off. The statistic in that case is not very helpful at all.