Mobility: High Schools
[ Part of a sequence of posts on income mobility ]
Chetty et al.
Chetty et al explore how well high schools predict census-tract level mobility The opportunity atlas: Mapping the childhood roots of social mobility:
One natural hypothesis for the variation across tracts within counties is that children in different parts of a county attend different schools, which attract different types of families and may differ in their value-added (Black 1999, Bayer et al. 2007). As a simple method of assessing the potential explanatory power of schools, we examine the fraction of variance that is across tracts within high school catchment areas vs. between high school catchment areas.33 Figure III shows that 28% of the total variance in outcomes – and about half of the local tract-within-county variation – can be explained by school catchment area fixed effects 34 Hence, although a significant share of the tract-level variation in outcomes could potentially be due to school effects, there is clearly substantial variation in outcomes even across neighborhoods among children who attend the same high school.
Look, I love Chetty et al's work, but this paragraph is kind of silly since it only looks at high schools: it's entirely possible all the remaining variance is due to elementary and middle schools. Equally obviously, this is a purely correlational observation, so we can't inferring causation. So, ultimately, we can't really infer anything about the causal effects of school quality on earnings using this approach.
I took my own stab at answering this question by looking at the association between Great Schools ratings and Chetty et al's income mobility estimates. I found positive associations: a 1 point increase in school rating was associated with a ~2% increase in adulthood income mobility). However, this disappeared after controlling for basic demographic confounders, and the power was good enough to rule out slopes greater than 2%.
This type of correlational analysis is obviously not conclusive regarding causal effects, but its lack of evidence is consistent with the lack of effects I found in the broader literature (see below). I mention it mainly because it is the only attempt I know of to link high school attendance with adulthood earnings.
Literature Limitations
There are a number of potentially relevant academic literatures here. For instance, there is an enormous literature on the importance of class size on test scores. This part of a wider literature on the impact of various interventions on test scores. There's a literature on the impact of teachers on test scores, especially Measuring the impacts of teachers I: Evaluating bias in teacher value-added estimates Measuring the impacts of teachers II: Teacher value-added and student outcomes in adulthood.
Why the focus on test scores?
Well, they're "objective", easy to measure, and presumably correlate with the knowledge and skills we expect schools to impart onto students. Studies like Measuring the impacts of teachers I: Evaluating bias in teacher value-added estimates suggest that changes in test scores predict changes in other variables of interest such as income, at least in some situations.
This is all interesting, but there are reasons to dislike these measures:
- Effects on test scores tend to fade out significantly over time.
- Individual parents can't impart an intervention like shrinking class sizes onto their children, which makes such findings relatively useless for individual people.
- Despite the fascinating literature linking teacher quality to test score changes to income changes, I know of no literature that generalizes these findings to non-teacher interventions.
For these reasons, I will focus on the thing parents can choose: which school their children attend. As such, I focus on the literature that tries to tie a change in school attendance with a change in relevant outcomes.
Unfortunately, I couldn't find any such studies that used adulthood income as an outcome, so we'll instead have to focus on proxy outcomes such as test scores, high school graduation rates, college attendance rates, college graduation rates, and measures of college "quality".
It is, obviously, unethical to force someone to attend a particular school: among other factors, parents can always choose to move to a different state. For this reason, we have no studies that are pure experiments.
We do, however, have a number of experimental studies where the treatment is being assigned/accepted to a school. These studies impose a methodological problem: if you define the experimental group as those who are both assigned to treatment and who actually undergo it, then you introduce bias since deciding to actually undergo treatment is probably correlated with variables of interest; most likely this is an upwards biasd.
If you define the experimental group as all those assigned to treatment, whether or not they undergo it, then you are biased downwards. For instance, if only 40% of those assigned to a good school actually go, then the inferred effect size will probably be about 2.5x too small.
Reviews
TODO What Works Clearinghouse
Bloggers
Freddie deBoer, a self-described Marxist and author of The Cult of Smart The Cult of Smart: How Our Broken Education System Perpetuates Social Injustice, concludes that which school you go to doesn't affect educational outcomes Education Doesn't Work 2.0. He further argues most evidence to the contrary are due to poorly run lotteries Charter School Lotteries Are a Black Box. Why Don't Charter School Advocates Care?. Meanwhile, Chad Aldeman from Slow Boring points out that the effect of high schools on college attendance tend to be larger than the effects on test scores - an concentrated among low-income students Aldeman.
Rcafdm documents that school funding is actually progressive, which suggests that any effects "good" schools have on their students isn't driven by funding rcafdm.
College Studies
I collected all quasi-experimental studies in the US that evaluated the effect of attending a different high school on college attendance. Due to the concerns raised by DeBoer about lottery studies of charter schools, I excluded such studies (but included both non-charter lottery studies and also charter non-lottery studies). I found five such studies, which I will summarize below.
Note: an important part of reviewing these kinds of studies is verify that the schools in question are actually significantly different: obviously attending a school that is 0.1% better isn't going to cause any noticeable changes for a student's achievements. Keep this in mind for later.
The first study Allensworth examines schools in Chicago and employs a regression-discontinuity design. Their analysis separates schools into four tiers (Table 1):
| Tier | Avg ACT Score | 4-Year Graduation Rate |
| Selective | 24.1 | 90.7% |
| Top | 18.1 | 79.0% |
| Middle | 15.6 | 68.0% |
| Bottom | 14.8 | 48.3% |
They estimate (Table 7) no causal effect on 4-year college enrollment. They find a small negative effect from being in a bottom-tier school on college selectivity and whether a student was still in college 2 years later, but no difference between the middle, top, and selective tiers.
The second study Deming uses a lottery to examine attendance in the Charlotte-Mecklenburg area in North Carolina. The authors construct a "peer index" based on test scores, days suspended or absent from school, and prior coursework. Students at the better schools were ~0.5 z-scores higher on average (Table 4).
They estimate (Table 3) no causal effect on 4-year college attendance or degree completion. They find a ~4% boost in attending and graduating from a selective college. They find the positive effects are almost entirely due to effects on females.
The third study Dobbie examined three schools in New York City and employed a regression-discontinuity design. The schools (Table 1):
| School | Avg SAT Score | Graduation Rate |
| Stuyvesant | 2059 | 93.0% |
| Bronx Science | 1935 | 91.1% |
| Brooklyn Tech | 1795 | 78.1% |
They find no effects on college enrollment, college graduation, or college quality.
Study four Abdulkadiroğlu (see here for their supplementary material) examined schools in Boston and New York City and employed a regression-discontinuity design. Students just under the cut-off attended schools with very low SAT scores, while students just over attended schools with medium SAT scores. The authors find little effect on college enrollment or quality (Table VI).
Finally, I ignored the fifth study, because the authors didn't report enough data to work into my meta-analysis Chingos. Speaking of which...
I made a best-effort attempt to synthesize these studies into a mini meta-analysis, using a simple model: if you attending a school where x% more kids go to college, your probability of going to college increases by k*x%. My meta-estimate for the value of k is 0.027, with a standard error of 0.026. In English, we have little reason to believe attending a school where students are 10pp more likely to attend has a large causal effect on whether you attend college. The most likely effect would be 0.3pp, and we can rule out an effect larger than 0.8pp.
I repeated this analysis using attendance of a selective college as the outcome variable. I meta-estimated k=-0.008 (SE=0.047). In other words, no evidence of any effect here.
What all these ~zero effects suggest to me, is that college admissions officers do a reasonably unbiased job at giving you kudos for attending a more selective school, where your class rank will be lower.
Some studies find evidence of heterogeneity where school quality matters more for girls and more if you're escaping particularly low-achieving schools. This has also been found by more conventional regression studies Berkowitz.
I found no evidence that trying to attend the crème de la crème had any noticeable effect on collegial achievement.
In short
There is some good evidence that high school quality affects test scores. Test scores gains typically fade out several-fold after an intervention ends, so, realistically, we're talking a couple hundredths of a standard deviation increase in adulthood test scores. Given this, it seems unlikely that attending a better high school would significantly affect your income by making you smarter and more knowledgeable.
There is also some evidence that high school quality affects propensity to attend college, but the effects tend to be small and not statistically significant. Suppose we take the highest possible estimate: that a 10pp increase in college attending at a high school has a causal effect of 0.8pp. The causal effect of attending college is probably about 35% more income, so the causal effect that change in high school quality has on adult income via attending college is about a 0.3% boost (0.008*0.35). This is plausibly higher for girls and students from low-achieving areas, but such claims smell of p-hacking and, in any case, imply the effect is even smaller for boys and students from high-achieving areas.
Likewise, even if we accept the maximum plausible effect on attending a selective college (1pp), when we combine this with our analysis of the effect of college quality on adulthood income, we find the result is minuscule (~0.1%).
One last note before my conclusion: if you end up attending college, it seems unlikely to me that your high school signals much to employers. If this intuition is true, it removes the most direct possible connection between high school attendance and earnings.
Moreover, what little evidence I've been able to scrounge found no association between Great Schools rating and income mobility. This is consistent with the quite small association I've found between traditional measures of college quality (e.g. average SAT score) and measures of college income mobility (their fixed effects).
Ultimately, then, I don't see any good empirical evidence that high schools affect adult income in any measurable way.
The only way I can see to salvage the idea that high school quality affects adulthood income significantly is to (1) focus on girls from low-achieving areas while ignoring concerns of p-hacking or (2) posit that high school quality affects human capital in a non-academic way, such as by grounding students upper class culture, teaching executive function skills, allowing networking, etc.
These hypotheses are all possible, but I know of no rigorous attempt to test them, and, given the non-results we've seen so far, I suspect the overall effects on income are fairly small. To quote Bryan Caplan:
When someone insists their product has big, hard-to-see benefits, you should be dubious by default - especially when the easy-to-see benefits are small.