Modest but meaningful: Sara Heller on the benefits and costs of a Chicago youth program – Part 1
Sara Heller is the lead author of Preventing Youth Violence and Dropout: A Randomized Field Experiment, published in May as part of the National Bureau of Economic Research’s working paper series. She recently joined the faculty at the University of Pennsylvania as an assistant professor of criminology. She earned her doctorate at the Harris School of Public Policy at the University of Chicago, where she worked as a researcher for the University of Chicago Crime Lab. We spoke with her about the study—which includes a cost-benefit analysis (CBA)—and its findings and implications. This is the first of two parts of our interview with her.
Your study was about an intervention on the south and west sides of Chicago called Becoming a Man (BAM) that works with young men in grades 7 to 10. Can you summarize the study, the methods you used, and what you found?
Our study was a large randomized controlled trial of an intervention for disadvantaged male youth from high-crime Chicago neighborhoods. The intervention was delivered by two local nonprofits and included after-school programming, regular interactions with a pro-social adult, and, perhaps the most novel ingredient, in-school programming designed to reduce common problems related to judgment and decision making, or what psychologists call cognitive behavioral therapy (CBT).
The motivating idea behind CBT is that quick, automatic decision-making often drives youth behavior. (Anyone who has ever interacted with an adolescent knows what these fast and seemingly disproportionate reactions look like!) It’s easy to see why relying on such initial, unthinking reactions may lead youth to conflicts with teachers or peers. So CBT teaches youth how to slow down and make reflective rather than reflexive decisions, at least in situations when that’s appropriate. Although meta-analyses suggest the promise of this approach, convincing causal evidence has been quite scarce.
Our study generates that evidence. We randomly assigned individual youth to treatment and control conditions within 18 Chicago Public Schools, then tracked them in administrative records on schooling (from the school district) and crime (arrest records from the Illinois State Police). What we found was pretty striking: a 44 percent decrease in violent-crime arrests among participants during the program year and an increase in schooling engagement—measured by days present, grade point average, and school persistence (enrollment status at the end of the academic year)—that lasts through the follow-up year. Although the study youth aren’t old enough to have graduated yet, we extrapolate from other observational work to predict that the improvement in schooling outcomes we observe might raise future graduation rates between 3 and 10 percentage points (7 and 22 percent).
The working paper says that the program generated benefit-cost ratios ranging from 5:1 to 30:1, when you take into account the monetary value of intangible benefits. Tell us more about what went into calculating the costs and benefits.
Calculating the program costs is pretty straightforward; the budgets from the nonprofit organizations that ran the program show they spent about $1,100 per participant. Calculating the benefits, however, is a little more involved.
The range of 5:1 to 30:1 comes from comparing the social benefits of the one-year drop in violent crime, calculated in different ways, to the program costs. We don’t yet include the benefits of increased schooling engagement, since the clearest way to monetize that benefit would be through the value of increased graduation, which hasn’t happened yet. But if we use our estimate of what might happen to future graduation rates, combined with the present-value benefits of graduation from Levin et al (2007), the benefit-cost ratio could be as high as 77:1.
To calculate the crime benefits, we start with the direct administrative savings to the government from fewer arrests. These savings estimates include arrest, detention, and placement/incarceration costs from Cook County (Chicago) and Illinois data whenever possible. That makes them pretty conservative, since they don’t include related expenditures like policing and court costs.
Although these administrative costs are important, especially to policymakers, it is clear that the total benefits to society are far broader than just the tangible costs of crime. Yet monetizing the value to society from reduced crime is challenging on a number of levels, in part because there is no perfect way to monetize the intangible costs of crime, such as physical and psychological harm to victims and lost productivity.
Past research has tried to estimate intangible costs in two main ways. One approach is to try to capture the public’s willingness to pay for reduced crime through contingent valuation surveys. Conceptually, this approach measures the right thing: the aggregate sum of the public’s willingness to pay for reducing the risk of crime victimization in the future. But in practice, many people are understandably nervous about relying on survey responses to hypothetical questions about what people would be willing to pay to reduce crime.
A second approach has been to use jury award data. This is a little more concrete than hypothetical valuations, but jury awards adopt an “ex post” perspective, after a victim is identified, so they are problematic from a conceptual perspective. That’s because a jury’s response to a particular person and crime could be quite different from the group’s response to crime in general. Jury awards also rely on the very small and unusual subset of criminal events that result in civil litigation for damages.
We decided to remain agnostic on the “right” method, using both approaches to see how robust our conclusions were to different ways of doing things. We started by following the basic approach from Kling, Ludwig, and Katz (2005), assigning each type of crime the social costs estimated by Miller, Cohen, and Wiersema (1996) that rely on jury award data (inflated to 2010 dollars), and examining the sensitivity of our estimates to how we handle the social costs of homicides (for example, trimming the cost of homicide by half). We then repeated the process using the willingness-to-pay estimates from Cohen et al. (2004)—amounts that are much higher—to generate the upper-bound estimates.
Since our estimates are based on arrests, and not all criminal offenses result in arrest, we may have understated the total social benefits from averted crimes among treatment youth (although it is also true that not all arrested youth committed the crime for which they were arrested). But no matter how we estimated program benefits, the range of 5:1 to 30:1 sends a pretty compelling message about BAM’s approach to reducing an outcome that is as socially costly as violent crime.