During out Wire roundtable way back when, Kinukitty had some depressing thoughts on the use of statistical methods in government. Thought I’d reprint it since we’re doing our democracy thing today.
I was delighted when the Wire opened with that CompStat meeting. I don’t know if many people understand the tyranny of the stat programs. Many governments and government agencies wrestle with some kind of performance measurement system, and they tend to work pretty much as you described – there are lies, damned lies, and statistics.
Performance measurement isn’t hopeless, exactly. There are some (probably not a lot) of governments using it right and getting good results. It takes someone special, though, to turn an organization around and create true accountability (which does not include firing people because you don’t like their stats). Especially in an enormous bureaucracy like a government. And then there are the elected officials. But as long as some organizations are doing something good with stats, it seems best not to throw out the baby with the bath water.
Because I don’t know if there are a lot of alternatives. I don’t see NGOs as helping very much. Too many obstacles, including the fact that stats can be altered just by the choices of what is measured, and how. And news coverage? I don’t think the problem there is that news outlets are only interested in sensationalizing stories to sell copies, advertising, etc. Well, it’s not the only problem. A lot of reporters and editors just don’t understand what they’re publishing, and the more sophisticated or complicated the issue is, the less likely they are to really get it. The current hysteria about state and local government pensions is a good example. Yes, they have an incentive to report that the sky is falling, since people are more likely to be interested in that than the sky not falling, but they also don’t understand the issue well enough to challenge any lies, misrepresentations, or mistakes their sources feed them. I’m not actually completely down on journalism — I more or less believe in the fourth estate thing. We’d be screwed without it. But there are problems.
Which leaves me in a Wire frame of mind, too. I appreciate it, though. I think it’s kind of important to make people understand that the problems are complicated.
