Around the end of the 19th century, Progressives celebrated the dawn of rational government. Public policy was to become a social science, and like all sciences it was to be based on quantitative facts. Every aspect of humanity and all possible outcomes would be assigned a numerical value, and then these numbers could be crunched into a thousand equations that would give us perfect governance. No more trial-and-error, no more dishonesty, no more messy emotions getting in the way of sound policy. It was salvation through statistics.
The faith in scientific government may seem absurd today, but much of our public policy is still built around the statistics game. Partly, this is because quantitative measures are generally easy to analyze and explain. If the police report that crime rates rose by 5%, we all understand what that means, and we can draw general conclusions on how to respond. Plus, statistics just seem so damn rational and authoritative, not like those touchy-feely qualitative observations. Decision-makers, bureaucrats, and the public all crave certainty, or at least a close facsimile.
But numbers can be wrong. Or to be more precise, people are lying liars who make up numbers to “prove” whatever they want to prove. In “The Wire,” the public institutions of Baltimore are constantly “juking the stats,” a term referring to various methods by which they falsify statistical data to show progress where there is none, all while the city continues to crumble around them.
While the series creators have plenty of bile for every aspect of Baltimore’s government, none of the institutions are criticized as thoroughly as the Police Department. The Department is portrayed as the apotheosis of public sector dysfunction, and juking the stats is hardwired into its institutional DNA. On several occasions, Commissioner Burrell alters statistical data so as to gloss over the embarrassingly high rates of violent crime. The simplest way that police departments juke the stats is by re-classifying reported crimes as less serious offenses (as an example, aggravated assaults become assaults). The only crime that can’t be downgraded in this manner is homicide, for obvious reasons. This was why Burrell and Rawls were obsessed with the “clearance rate” (the number of murders solved), and homicide detectives concerned about their clearance numbers looked for ways to avoid investigating a (potentially unsolvable) murder, such as by dumping the bodies on another jurisdiction (as with the bodies of the dead prostitutes in season 2).
Baltimore’s school system does not escape scrutiny either, especially during season 4. The public schools are dependent on federal funding, and juking the stats is necessary to show improvement on the standardized tests administered under the federal No Child Left Behind Act (there are unpleasant consequences for a school and its faculty should their students repeatedly under-perform on the tests). As shown on “The Wire,” one common method of improving the scores is to teach to the test. Students are given rote lessons on how to answer specific questions. This can lead to modest improvements in aggregate test scores, but the students are not really learning anything except how to take a test, and so the scores do not accurately measure the students’ mastery of basic academic skills.
For both the police and the schools, much of the problem arises from a conflict of interest: the institutions responsible for reporting the stats are the same institutions that will be judged on them. Few people are willing to admit failure, especially when the consequences include losing your job. Thus, Police Commissioners have every incentive to paint a rosy picture of the city’s crime rates. Nor does auditing by an outside agency solve the problem. An agency tasked with analyzing statistics from the police department would still be dependent on the department for crime data, which means they would likely get the same juked stats. The only alternative would be an independent means of detecting crimes and collating the data, which would be prohibitively expensive, especially for a cash-strapped city such as Baltimore.
Statistics were supposed to give us scientific government, one where the ideal public policy would be crafted in accordance with hard data. This was implausible even in the best case scenario, but when data is falsified, statistics actually grant further authority to the lies of public officials. And without trustworthy stats, how is the public supposed to the judge the performance of public institutions? How are voters supposed to hold their elected officials accountable when we can’t be sure if their policies succeeded or failed miserably?
“The Wire” doesn’t offer any solutions to this dilemma. Rather, it suggests that the nature of our political system, particularly the never-ending electoral cycle, creates irresistible incentives to lie. Mayor Carcetti, for example, initially forces Burrell to resign when he lies one too many times about the crime rates. But when Carcetti begins his gubernatorial campaign, his staff pressures incoming Commissioner Daniels to juke the stats so that Carcetti can claim that crime rates fell during his term as mayor. When Daniels refuses to compromise his morals any further, he’s forced out of office and replaced with a more compliant lackey. The public, by and large, is ignorant of the mayor’s deceit, because the city government controls most of the data collection and analysis.
So how should the public deal with juked stats? One alternative would be to provide more resources to non-governmental organizations that compile and analyze their own data, but NGOs have their own agendas and are equally capable of lying. Or we could abandon our stat-based approach to public policy entirely, and rely more on qualitative observations, such as in-depth news articles. For many reasons, this is highly unlikely to happen, but even if it did, qualitative accounts can also be fraudulent, and they are more often anecdotal rather than reflective of larger social trends.
I was hoping to end my post on an upbeat note. But this is a roundtable on “The Wire,” so maybe it’s appropriate that I throw my hands up in frustration.
_____
Update by Noah: The entire Wire Roundtable is here.
I was delighted when the Wire opened with that CompStat meeting. I don’t know if many people understand the tyranny of the stat programs. Many governments and government agencies wrestle with some kind of performance measurement system, and they tend to work pretty much as you described – there are lies, damned lies, and statistics.
Performance measurement isn’t hopeless, exactly. There are some (probably not a lot) of governments using it right and getting good results. It takes someone special, though, to turn an organization around and create true accountability (which does not include firing people because you don’t like their stats). Especially in an enormous bureaucracy like a government. And then there are the elected officials. But as long as some organizations are doing something good with stats, it seems best not to throw out the baby with the bath water.
Because I don’t know if there are a lot of alternatives. I don’t see NGOs as helping very much. Too many obstacles, including the fact that stats can be altered just by the choices of what is measured, and how. And news coverage? I don’t think the problem there is that news outlets are only interested in sensationalizing stories to sell copies, advertising, etc. Well, it’s not the only problem. A lot of reporters and editors just don’t understand what they’re publishing, and the more sophisticated or complicated the issue is, the less likely they are to really get it. The current hysteria about state and local government pensions is a good example. Yes, they have an incentive to report that the sky is falling, since people are more likely to be interested in that than the sky not falling, but they also don’t understand the issue well enough to challenge any lies, misrepresentations, or mistakes their sources feed them. I’m not actually completely down on journalism — I more or less believe in the fourth estate thing. We’d be screwed without it. But there are problems.
Which leaves me in a Wire frame of mind, too. I appreciate it, though. I think it’s kind of important to make people understand that the problems are complicated.
I appreciated in the Wire too that Carcetti, the idealistic young reformer, is about the least sympathetic character on the show (Maurice Levy, the Jew lawyer, maybe less so — but perhaps we should pass over that….) Carcetti is much less sympathetic I think than Burrell or Rawls, both of whom are openly cynical, and therefore significantly less oleaginous than Carcetti, who’s basically powered by self-deception and hypocrisy.
Well, statistics are abused– but to a great extent, it’s because we allow this abuse. We really aren’t taught how to interpret statistics.
Back in the ’80s two Islamicist terror attacks (bombs) hit the Paris Metro subway system. Immediately, one of my colleagues gave up the Metro for a 30-minute commute by car for work.
It was an irrational decision, on the statistical face of it: she had a far, far greater chance of dying from a car accident than from a bomb on a subway train.
Is this really an interpretive failure? I doubt it. Most people are well aware that planes (even after 9/11) are safer than cars. Likewise with cars vs. trains. Fears of this kind are rarely rational. You can “know” what the statistics say, even believe it, but still make decisions based on irrational fears
Actually, the entire plot of James Cain’s “Double Indemnity” hinges on this irrational fear.
I definitely like this show more from a distance. “Performance management” is pretty obviously a nice way of saying “hand gently yet firmly grasping the short and curlies.”
I actually took some advanced stats classes in college, which really illuminated nothing so much as the a) complexity of factors determining predictions, and b) the difficulty of monitoring or controlling variables, even for trained analysts, and c) there are not that many trained analysts.
It’s math bullying. One of the most intriguing things is that math bullying is simultaneously both a hallmark of privatizing services, and a weapon used to discredit public service providers.
Near and dear to me– tomorrow morning I find out if my school is being “turned around,” “consolidated,” etc., and what my future will be, based on factors having almost nothing to do with rigorous teaching practices.
Hey Kinu: I’m skeptical of performance measurements for most public sector jobs. Public service isn’t the same thing as selling cars. Though I should add that I’m not completely anti-statistics. In my own experience in the federal government, I’ve worked with economists with a great deal of integrity. They published data that is honest and professional, so it should come as no surprise that it’s completely ignored by the higher-ups.
Noah- I have to give the actor and writers props on Carcetti: they really nailed down what makes politicians so insufferable.
Bert- I’m sorry to hear about your school.
Noah- thanks for adding the picture. A perfect choice with Burrell and Daniels.
There are loads of public sector jobs and services for which performance measures are desirable. We should want to know literacy and numeracy levels in primary school; recidivism rates for different criminal justice programs; re-addiction rates for methadone treatments; immunisation rates and disease incidence for vaccination programs; etc. etc.
In general: the government employs people to achieve certain desired outcomes. We want to know whether those outcomes are achieved; which ways of achieving them are more effective; and, given that resources are limited, whether we’re achieving them cost-effectively. Quantitative and statistical performance measures are the worst way of assessing government spending and holding public officials accountable, except for all the other ways.
I don’t think that’s true. Qualitative assessments (portfolios, personal discussions with teachers) are used in many settings (the socialist utopia of scandinavia, for example) and work much better than our idiotic testing regime. Many schools (my son’s for example) don’t use testing, and function much better than the public school system.
In police work, as presented in the wire, I think there’s a case to be made that the quantitative system produces worse outcomes as well. Simon certainly seems to think that manipulating the stats corrupts the department and destroys effective policing. They’d be better with honest qualitative assessments…and barring that, they’d be better off with dishonest qualitative assessments, probably, because everyone knows not to put scientific faith in qualitative assessments.
You can’t measure something without changing what you’re measuring. It’s entirely possible to change it in a way that makes you much, much worse off, not better. At least in the world the wire presents, I think they’d all be better off without the crazed pursuit of numbers.
“Fears of this kind are rarely rational. You can “know” what the statistics say, even believe it, but still make decisions based on irrational fears”
On behalf of the irrational, what people are probably considering (implicitly) is survival rate. That if you can choose between being in a plane crash or a car crash, it’s not so irrational to pick the latter. Also, I wonder if everyone were in planes as much as cars, would the former really result in fewer deaths? I haven’t really looked at the stats involved, but something seems a bit fishy about the claims.
What about the statistics concerning travel on airplanes in the immediate aftermath of a terrorist attack? Does it become safer to travel on commercial airliners in the 3-4 weeks after a hijacking/bomb blast for example?
Charles: as I’m sure you know, there’s a huge body of research in the “heuristics and biases” tradition which is generally taken to show that decisions and reasoning deviate significantly from traditional norms of rationality, particularly when thinking about probability. So I’d be surprised to learn that people were reasoning along the lines you suggest.
I don’t have my Kahneman, Slovic and Tversky handy, but isn’t the standard explanation of overreaction to extremely unlikely risks that rare catastrophes are more salient in reasoning?
It could just be what I was reading when I watched it, but the wire strikes me relentlessly focussed on the micro-dynamics of power… the way that we all end up governing ourselves on behalf of institutions of which we’re scarcely aware. And while the show never goes full Foucault (power isn’t quite diffuse enough), it does a good job of showing that the interests of the public and private sector are thoroughly wound together, and that they largely effect their goals by exploiting the perceived self-interest of individuals.
This gets back to that whole issue of realism. I never really saw the show as aiming for realism. It’s an argument about power delivered in the key of tragedy (complete with speeches, archetypes, fatal flaws, and so on).
There’s probably a connection to Dickens here, too, something about social reform movements, but that falls outside my period.
The point about public-private overlap is a good one. It got me thinking about the cycle of money working its way through drug dealers, real estate developers, and the city government.
And it’s funny that you mention Dickens, given the content of Wednesday’s post.
The devil is in the details — as applies all over in The Wire.
What, precisely, are you measuring? For basic schooling, literacy and numeracy can be tested in ways that are more or less susceptible to gaming (and when I see the registers at Burger King have pictographs of burger/fries/soda, I don’t need to see statistics to know how the wind’s blowing).
For crime, I would suggest neighborhood-based polling on “how safe do you feel?” — but at this point too many people are already way too far gone in their fear instilling TV programming — feature a few weeks about some kidnapped girl and no mother feels safe, etc.