The Charter School Lottery Studies and What They Show About Cities

Economists would give a great deal for a randomized experiment in education. They rarely get one, because nobody will flip a coin to decide which children get the better school. Then, by accident, they found one in the hallway of a charter school on lottery day.

When a charter school has more applicants than seats, the law in most places requires that the seats be assigned by lottery. That is a fair rule for families. For researchers it is a gift, because it sorts children by chance. This post explains what those lotteries have taught us, in a case where the evidence is better than it is in most education debates. I will show the method, the best-known studies, the places where charters failed to beat the district schools, and the lesson that sits underneath the numbers.

The claim

Admission lotteries let researchers compare charter winners and losers almost as in a randomized trial, the best of those studies find strong gains for urban charter students, and the results also show that freedom from rules works only when it comes with accountability for results. I will build it in steps and separate what is well established from what is still argued.

Why a lottery makes good evidence

Here is the central problem in comparing charter schools to district schools. The families who choose a charter are different from the families who do not. They may be more motivated, more informed or more determined, and they might have had better results anywhere. If you simply compare charter students to everyone else, you cannot tell whether the school did the work or the families did.

A lottery solves the problem. Suppose a charter school has 200 applicants for 100 seats and draws names by chance. The 100 who win and the 100 who lose are, on average, the same kind of family: they all wanted the school enough to apply. The only systematic difference between the groups is that one got in. If the winners later score higher, the school, and not the families, gets the credit. That is the logic of a randomized controlled trial, the standard method for testing a medicine.

Honest researchers state the limits. The method only works at schools that have more applicants than seats, so it tells us about oversubscribed schools, the ones in demand, and cannot tell us about charters that nobody wants to attend. It also needs good records of who applied and who won, and it needs to follow the losers, who scatter to other schools. None of these limits is fatal, but they matter when we draw conclusions about charters in general.

The Boston studies

The most influential lottery studies were done in Boston. In 2011, five economists, Atila Abdulkadiroglu, Joshua Angrist, Susan Dynarski, Thomas Kane and Parag Pathak, published a study in the Quarterly Journal of Economics under the title Accountability and Flexibility in Public Schools: Evidence from Boston's Charters and Pilots. Using lottery records from Boston's oversubscribed charter schools, they compared students who won seats with students who lost.

What they found was large. Students who won seats at Boston's charter schools gained substantially more in math and in English than the students who lost the lottery. The researchers also looked at the city's "pilot" schools, which are public schools that were given charter-like autonomy over budgets and staffing inside the district, and found that the estimates for pilot schools were smaller and more variable than those for charters. I will not quote effect sizes from memory, because the details are in the paper and you should read them there. The direction was clear and the magnitude was large by the standards of education research.

This is a conclusion that does not come from an advocacy group. Angrist, one of the authors, later shared the Nobel Prize in economics for methods of just this kind. The credibility of the finding comes in part from the method itself.

What explained the Boston gains

A follow-up study, by Angrist, Pathak and Christopher Walters in 2013, asked why some charters do better than others. They looked at Massachusetts charters and found a split. Urban charter schools, the ones in Boston and similar cities, produced large gains in the lottery comparisons. Non-urban charters did not, and in some cases their results were no better than or worse than the traditional schools the students would have attended.

The researchers connected the urban results to a particular set of practices that they called, following the language of the sector, "No Excuses": a long school day and year, strict behavioral expectations, intensive tutoring, frequent assessment and data-driven instruction, and a strong emphasis on basic skills. The non-urban charters tended to follow a different model.

Notice what this implies. The charter label is not what produced the gains. A set of practices did, and the practices happened to be more common in the urban charters that were studied. That is a finding about how a school operates, and it is the thing that policymakers should want to copy.

The national picture, CREDO

A different line of research comes from CREDO, the Center for Research on Education Outcomes at Stanford. CREDO studies work differently. They do not rely on lotteries. Instead, they build a "virtual control record" for each charter student, by matching him to students in nearby district schools who share demographic characteristics and a similar prior test score, and then they compare progress over time. It is a reasonable method, though it cannot fully control for the families' unmeasured differences the way a lottery can.

In 2009, CREDO published a national study of charters in several states. Its headline finding was sobering for charter advocates: about 17 percent of charter schools produced math gains that were significantly better than their district comparison, about 37 percent were significantly worse and the rest, about 46 percent, were not significantly different. That is a mixed result, and a fair reader should say so.

The picture changed in later studies. CREDO's later national studies found improvement in the sector over time, over time. And in 2015, CREDO released its Urban Charter School Study, which looked at charters in 41 cities. It found that, on average, students in urban charter schools gained the equivalent of about 40 additional days of learning in math and 28 additional days in reading per year, compared to similar students in the district schools.

Put together: the lottery studies found large gains in the cities where they were done, and the larger matched-comparison study found positive average effects in cities. Two different methods pointed in the same direction. That consistency is what makes the urban result credible.

Where charters did not outperform

An honest account does not stop there. Charters are not magic, and the evidence on where they fail is part of the lesson.

• The 2009 CREDO national study found that a large share of charters did no better than, or worse than, nearby district schools. The average hid a wide spread.
• A federal evaluation of charter middle schools, conducted by Mathematica Policy Research and published in 2010, used lotteries across a sample of schools in several states. It found that on average the charter middle schools did not produce significant gains in test scores, though it found positive effects for lower-income and lower-achieving students and negative effects for higher-income students in the sample.
• Online charter schools have performed poorly in CREDO's work, with results that were markedly worse than the district comparisons. A virtual school is a very different thing from a classroom with a teacher, and the finding should make any legislator cautious about expanding that model without evidence.
• Non-urban charters in the Massachusetts study showed no gains, as I described.

Taken together, the evidence says charters are a vehicle, not a guarantee. What the vehicle carries matters most.

The steelman against

Here is the best case against drawing broad conclusions, in its strongest form.

Lottery studies can only study the popular schools. If the best charters are the ones with waiting lists, then a study of those schools will show the best charters, not the typical one. The families who apply, even if the lottery sorts them randomly, are self-selected, so results for them may not hold for the whole population. Charter schools may also have effects on the district schools around them, drawing away motivated students or resources, and a study of winners and losers cannot capture that. And some charters achieve results by methods, such as strict discipline or attrition of struggling students, that cannot be scaled or may carry costs of their own.

These are fair points, and they should temper any claim about charters as a whole. Now the answer.

1. The claim is about urban charters, not charters in general. I have said so repeatedly. The lottery evidence is strongest for urban, oversubscribed schools. That is a legitimate result for the setting where the achievement gap is largest and the need greatest.
2. Two methods agree. The lottery results and the larger matched studies, which are not limited to oversubscribed schools, both point to gains in cities. When different methods with different weaknesses agree, the finding is stronger.
3. Attrition and peer effects can be tested. Lottery studies follow the winners whether or not they stay in the charter school, so pushing out weaker students cannot by itself produce the measured gains. Peer-effect and competitive-effect studies exist as well, and their findings are debated.
4. A mixed average is the argument for accountability. If some charters are excellent and some are not, the solution is to close the ones that fail, not to abandon the sector. That is the next point.

The lesson, autonomy plus accountability

What do the studies together teach? I think it is this: freedom from rules produces results only when it is paired with accountability for those results.

A charter school is a bargain. It is freed from many of the rules that bind a district school, such as hiring and firing rules, a fixed schedule or a prescribed curriculum, and in return it agrees to a contract, a charter, that the authorizer can decline to renew if the school fails. The school gets autonomy. The public gets accountability. Take away either half and the bargain fails.

• Autonomy without accountability is how you get the weak charters, the ones that never close and the online schools that keep enrolling students with poor results. The freedom is used and the discipline of consequences is missing.
• Accountability without autonomy is how you get a district school that is judged on results but handed a rulebook, a contract and a budget it cannot change. Leaders in such a school cannot be held to account for what they have no power to change.

The Boston comparison between charters and pilot schools illustrates the point. The pilot schools were given some of the same freedoms, but inside the district, with the same ultimate accountability structure, and they did not match the charter results. I would not over-read a single comparison. But the pattern fits what Americans have always believed: power and responsibility belong together. The Founders built the Constitution on that insight. Madison's Federalist 51 describes the need for a government that can control the governed and also be obliged to control itself, with each branch given the means and the motive to check the others. A school system needs the same architecture: power to act, and a check that bites.

What this means for policy

• Authorizers matter. The body that approves and renews charters has to be willing to close failing schools. States with strong authorizers have better sectors.
• Let the successful grow. Caps on the number of charters, or on enrollment, keep the best operators from serving the families on their waiting lists. Where there are lines of children outside a school, the law should let it open more seats.
• Copy the practices. The features associated with the urban results, such as more time, tutoring and consistent expectations, are available to any public school. A district can borrow them.
• Be skeptical of untested models. Evidence against online charters is a reason for caution about expansion.
• Pair it with other choice. Charters are one piece of a larger picture, which includes Your ZIP Code Picks Your Child's School, the Case for Open Enrollment, Arizona Made Every Child Eligible for an ESA, How the Idea Spread So Fast and Tax-Credit Scholarships, the Choice Program Built on Private Donations.

For the data discipline behind all of this, it helps to read What the Nation's Report Card Measures and What Proficient Really Means, and for the broader argument about money and results, Real Spending Per Pupil More Than Doubled Since 1970, So Where Are the Gains. The man who first proposed paying for the child rather than the building, Milton Friedman, is the subject of Milton Friedman's 1955 Voucher Essay and Its Seventy-Year Road.

A word about children, not charts

It is easy to lose the children in the statistics, so let me end with one of them, a hypothetical.

Picture a mother in a large city whose daughter's assigned school is among the lowest-performing in the district. She cannot move. She enters the lottery for a charter two miles away, with a waiting list several times its size. If the name is called, her daughter gets a seat and, by the best evidence we have, a real chance at stronger skills. If it is not, she goes back to the assigned school. The mother does not experience this as a research design. She experiences it as a drawing in which her child's future is at stake, and a system that has made a lottery necessary has already admitted that it has too few good seats. The answer to that is more good seats.

The case in brief

1. A charter admission lottery sorts applicants by chance, so comparing winners and losers works almost like a randomized trial.
2. The Boston lottery studies of 2011, by Abdulkadiroglu, Angrist, Dynarski, Kane and Pathak, found large gains for charter students, and a 2013 follow-up found those gains concentrated in urban charters with specific practices.
3. CREDO's 2009 national study found a wide spread in charter quality, and its 2015 urban study found gains in cities of roughly 40 extra days in math and 28 in reading per year.
4. Charters did not outperform in some settings, including non-urban Massachusetts charters, online charters and the middle schools in a 2010 federal study, which shows that the label is not the cause.
5. The lesson is autonomy plus accountability: give schools freedom, hold them to results and close the ones that fail.

What you can do: look up your state's charter school law and authorizer, find out how many failing charters it has closed, and ask a legislator whether caps are keeping successful schools from opening more seats.