Metrics and Values

We're in a bind because we only value the metrics, not what they represent

James Shanahan
June 12, 2026

Hi! This is Scholastic Alchemy, a twice-weekly blog where I write about education and related topics. Wednesday posts are typically a deep dive into an education topic of my choosing and Mondays usually see me posting a selection of education links and some commentary about each. If Scholastic Alchemy had a thesis, I suppose it would go a little like this: We keep trying to induce educational gold from lead and it keeps not working but we keep on trying. My goal here is to talk about curriculum, instruction, policy, public opinion, and other topics in order to explain why I think we keep failing to produce this magical educational gold. If you find that at all interesting, please consider a paid subscription here, or at the parallel publishing spot on Beehiiv. (Some folks hate the ‘stack, I get it.) That said, all posts are going to remain free for the foreseeable future. Thanks for reading!

Subscribe now

Value Capture, Metrics, and Agency

I’ve been thinking about agency a lot recently. Agency featured heavily in part 3 of my series on gamification and I brought it up again in Monday’s links in relation to some problems with AI writing instruction. Agency, our ability to act on the world around us, turns out to be a very important component of learning and development. We use our agency to accomplish goals and that’s where the trouble begins because we often assign value to goals that aren’t our own with values that aren’t our own.

If this sounds like that section I included from Michael Perhshan at the end of last week’s post on gamification, you’re not wrong. I picked up a copy of N. Thi Nguyen’s The Score: How to stop playing somebody else’s game that Perhshan was discussing and I’ve been working through the first few chapters this week. The incisive point that Nguyen is playing around with in the book is value capture. He’s really worried that people are unhappy because they’ve oriented their lives around playing, well, somebody else’s game. Instead of developing our own values, setting goals based on those values, and then exercising agency to achieve those goals, we’re adopting other peoples’ values and end up feeling dissatisfied when we use agency to achieve goals based on other’s values. We are, in effect, captured by their values. For Nguyen these values most often come from institutions (e.g. workplaces, schools, hospitals) and are made attractive to people through gamification.

Nguyen offers an example of how one aspect of gamification, transparency and scoring systems, can create bad incentives that hinder the supposed goals they’re trying to achieve. He says that Charity Navigator, by creating a transparent scoring system for charities based on their overhead ratio, a measure of how little a charity spent on employees and facilities.

I even used it in my introductory ethics classes. I had my students read arguments about whether there was some obligation to give money to charity. As a grand finale for the unit, I’d take donations from the class, I’d match their donations, and then we would discuss the most effective use for that money. I always had us look at Charity Navigator’s efficiency ratings, and then we’d vote. The students inevitably picked a charity from near the top of the rankings. And then I’d give all out money away, right there in class. It was a satisfyingly dramatic moment.

But it turns out that the overhead ratio is a terrible metric. It targets internal costs—the amount a nonprofit spends on administrative costs, like paying its employees. It presumes nonprofits do more good when they spend money externally, on things like digging wells, building houses, or buying food for tsunami victims. It presumes that internal costs are a form of waste. The metric paints a picture for us: Nonprofits are a pipeline for money. Good nonprofits pass money through as efficiently as possible; anything that gets stuck in the pipeline is a waste. The metric paints this picture by taking a neutral, simple, mechanical fact—the overhead ratio—and then using it as the measure of success.

This has all kinds of bad effects but one of the big ones is that the rankings made it hard for charities to hire qualified experts in the relevant fields. They couldn’t pay well enough to compete with the private sector or with government because that would increase their overhead ratio and tank their rankings on charity navigator which, in turn, would decrease their donations. The impact of these rankings was to shift the focus of charities on to small, cheap problems that they could generally pay others to solve. Sometimes these still had a huge positive impact (bed nets being a prime example) but it also meant that problems which required time and staff and a place for that staff to work were sidelined. Nguyen reminds us, too, that evaluating a charity requires more than simple knowledge of their finances.

The problem is that nonprofits are trying to do good in complicated ways, like protecting ecosystems, building sustainable housing, or improving healthcare. Whether they have succeeded is actually a difficult matter. To evaluate it properly, you need to be buried in the particular domain of the nonprofit. Proper evaluation requires trusting other actual experts in the relevant field—housing, sanitation, education—along with experts in the particular geographical area.

I’m sure teachers in the audience are recognizing a lot of this and we’ll get back to education shortly but let’s put together this process that Nguyen is talking about. Instead of charitable donors making decisions about where to send their money based on their values and the goals they hope to achieve, they are making donations based on Charity Navigator’s metrics. Those metrics, while seemingly a neutral fact, change the idea of what a nonprofit does to being primarily about the efficiency of spending money. This is value capture. By accepting the metric, you are agreeing to its premises. Your agency as a giver is now circumscribed by those same premises. Yet, giving in this way undermines accurate judgement of successful nonprofits while also incentivizing nonprofits to work only on the kinds of problems that minimize their overhead. In the end, value capture leaves donors dissatisfied and important problems unaddressed.

In fact, Nguyen notes that this is exactly what happened.

Several major donor organizations eventually saw the problem with overhead ratio. After significant study, they concluded that pervasive use of this metric was forcing nonprofits to cut costs so dramatically that they couldn’t function effectively. Charity Navigator has listened to pushback from experts in the nonprofit sector and moved to decrease the centrality of the overhead metric in its rating systems, though it’s still controversial whether the company has done enough.

Schools have plenty of metrics but few values

NAEP scores are in again and these are what’s called the “long-term NAPE”. These would be sections of the test that are specifically designed to be very similar year after year across the decades to better represent long term differences between students. Along with the long term NAEP we also have a survey section that looks at a variety of things like time spent on homework or reading for pleasure. This has, of course, generated lots of commentary. See Chad Adleman here and Tim Daily here (and a somewhat amusing rebuttal of Tim Daily here). But what if we channeled a bit of Nguyen here?

I think we probably have more data about student performance than ever before. The NAEP has been around for a long time, but we also have ACT and SAT scores. We have state exams in all 50 states. There’s attendance data and discipline data and accessibility data. We have MAP assessments. We have i-Ready’s assessment suite. We have learning management systems that also collect data on student performance in real time. We have systems that track children’s eye movements and ambient noise levels as they work. If there were a metric for having metrics, a meta-metric, we would definitely be at an all-time high.

Anyone who spends time in schools recognizes that essentially all decisions are made with an eye toward optimizing those metrics, often focusing on short-term solutions and quick fixes rather than considering less proximate causes. Nguyen argues this is because the schools’ have substituted rich, deep values for superficial measures that, while transparent, serve students poorly. We optimize our curriculum to fit state tests and find out that giving kids small reading passages instead of whole books is contributing to problems with attention and reading stamina. We prioritize improving engagement by adopting edtech but end up with kids watching YouTube all day on school Chromebooks.

We’re not spending time considering our values and setting goals based on those values. We’re devaluing local knowledge and educational expertise in favor of chasing metrics but haven’t stopped to consider whether those metrics will deliver us anything we’re actually happy with. It’s one reason I think we’re seeing parents object to standardization, to edtech, and even to public schools. These systems simply aren’t legible to them anymore and they don’t have a trustworthy person to whom they can turn to explain what’s happening and why.

Now, I’m not done with The Score so I don’t know exactly what Nguyen’s remedies are but I think he’s on the right track with identifying some of our problems. We want to make better choices as individuals but also want our institutions to reflect those better choices. Instead, what we have are institutions trying to get us to substitute our values for theirs so that we make the choices they prefer. It’s unfulfilling and, to Nguyen’s philosophical point, it damages our agency, suborning it to something that is not ours.

Looking forward, I think what I want to do is connect all of this back to our frameworks, namely technocratically induced populist backlash and the “revolt of the public.” I think there are some similarities here that I need more time to think about and write up.

Until then, thanks for reading.