Education Reads to Start Your Week, April 20th

This continuity matters for the empirical question of whether accountability “worked.” The basic architecture - annual census-level testing, achievement-gap reporting, consequences for low performers, evaluation regimes tied to score - has been running continuously for almost twenty-five years. So, can flat long-term standardized test trend lines, persistent (and in some subjects widening) racial achievement gaps, and continued middling U.S. performance on PISA honestly be blamed on a premature retreat from “accountability”? What does that even really mean? The ed reformers claim that the accountability project was abandoned before it could deliver what it was intended to, but that’s a claim the statutes, state rating systems, and evaluation policies do not support. A more candid reading of the record is that accountability was sustained long enough and consistently enough to be judged on its own terms - and on those terms, the theory of action has not been vindicated. Because while federal policy can have big impacts on fairness and quality of life for teachers and schools, there’s simply no reason to think that it has consistent or meaningful effects on test scores.

NAEP scores in math and reading were already rising before NCLB was signed in January 2002. The 1990s had seen consistent gains, particularly among Black and Hispanic students. Why those gains were occurring is subject to as much debate as all the rest of this stuff; it will not surprise you to hear that I suspect that the remarkable decline in concentrated poverty during that decade played a large role. Whatever the causes, to credit NCLB with gains that began a decade prior doesn’t make much sense. More fundamentally, there is no counterfactual: every American public school was subject to NCLB simultaneously, so there is no control group against which to measure its effect. Without one, the claim that NCLB caused gains is simply not falsifiable in any rigorous sense, which means it’s a story imposed on a trend line rather than a scientific claim…

The PISA declines visible in American math and reading scores over the 2003–2022 period aren’t remotely anomalous; they’re part of a near-universal pattern among wealthy, developed democracies. In particular, the Netherlands, Finland, Belgium, Canada, and Australia - that is, countries with many economic and social similarities but radically different curriculum philosophies, funding structures, pedagogical traditions, etc - all show trajectories strikingly similar to that of the United States. (In fact Finland, long held up as the gold standard of education reform and frequently invoked as a rebuke to American approaches, has seen some of the steepest reading declines in the developed world.) If policy and pedagogy were the primary drivers of American underperformance, one would expect American trends to diverge from those of peer nations, to look distinctively bad in ways that track distinctively American choices. Instead, what the data show is convergence: a broad, shared downward drift across the developed world that almost certainly reflects forces operating above the level of any individual nation’s classroom policy. Pinning these trends on American policy choices, without accounting for why virtually identical trends appear in countries that made very different choices, is not serious analysis.

Were there some bad educational ideas batted around in the 2010s? Sure. Ending algebra in 8th grade out of specious equity concerns, for example, was misguided in profoundly predictable ways. (The affluent kids just get algebra instruction outside of public schools, whether in private schools or with private tutors, rendering the whole thing a farce.) But there’s a reason that San Francisco is so often singled out for this bad policy: the number of districts who enacted it was tiny. I can’t get straight numbers, but I believe that the number of districts that eliminated the option for 8th graders to take algebra is in the single digits; there are more than 13,000 public school districts in the United States. Even San Francisco has rolled this policy back. Yes, participation rates are down, but that’s largely due to changes in standards, particularly Common Core sequencing - yes, the de facto national curricular standards beloved by the accountability people. This is one of the weird things about this whole debate, the way that the rhetoric of a loud fringe and the actions of a tiny number of outlier schools and districts are mistaken for actual meaningful pedagogical and policy change.

On a recent webinar, Kristen DiCerbo indicated that student usage of Khanmigo was not what Khan Academy wanted. “I will tell you, we see more ‘IDK IDK,’” she said, “more passive kinds of interaction than we would like.” Critics suggested that the difference between Khanmigo and human tutors were vast, with the chatbot unable to draw on a relationship with students, unable to initiate or end conversations with the sensitivity possessed by even quite average human tutors.

At this point, with Khanmigo already struggling to meet Sal Khan’s expectations, critics began to scrutinize Khan Academy’s efficacy research. Laurence Holt dubbed its deficiencies “The 5 Percent Problem,” so called because Khan Academy’s strongest effect sizes (0.26 standard deviations above the mean in a 2022 study, for example) were achieved only after excluding 95% of the study population. In a more recent study, Khan Academy lowered their inclusion threshold significantly, avoiding the 5 Percent Problem, but also watching their effect size boil away to nearly nothing.

Sal Khan has blamed teachers for low student Khanmigo usage, saying that teachers “need to figure out ways to engage them more” with Khanmigo. In Chalkbeat, DiCerbo blamed students, saying, “Students aren’t great at asking questions well,” which will come as a surprise to anyone who has ever known a small child. Yes, it’s possible that some of the most inquisitive beings on earth aren’t all that great at asking questions, but it seems more likely that chatbots like Khanmigo aren’t all that great at inviting, understanding, or answering those questions.

The applicability of these studies today is an open question. Combined, the studies focus on just three schools and a few hundred students. And since this was done more than 30 years ago, things like what traditional instruction looks like may have substantially changed.

The papers include little information about those final tests, but it appears they were designed by the researchers, unlike a traditional standardized test. Researcher-created assessments on subjects that are totally new to students — like cartography and probability, in this case — tend to see students make the largest gains.

Bloom’s work also doesn’t focus on technology-based tutoring, a point personalized learning advocates usually acknowledge. “If it supports anything, it supports one-on-one human tutoring,” Riley said.

Second, subbing helps keep “Ivory Tower Syndrome” at bay. Ivory Tower Syndrome is the kitschy term for when faculty members become disconnected from the realty of life beyond campus. Serving as a substitute teacher keeps me grounded in a way that cannot be achieved in any other way. I face the same challenges with disruptions and off-task behavior. I face the same pressures of time management, engagement, and differentiation. Though I only get a small taste of what it is like to face these issues day after day, I become more connected to the challenges teachers and students face in schools every day.