More on Engagement and Student Outcomes

Welcome to Scholastic Alchemy! I’m James and I write mostly about education. I find it fascinating and at the same time maddening. Scholastic Alchemy is my attempt to make sense of and explain the perpetual oddities around education, as well as to share my thoughts on related topics. On Wednesdays I post a long-ish dive into a topic of my choosing. On Fridays I post some links I’ve encountered that week and some commentary about what I’m sharing. Scholastic Alchemy will remain free for the foreseeable future but if you like my work and want to support me, please consider a paid subscription. If you have objections to Substack as a platform, I maintain a parallel version using BeeHiiv and you can subscribe there.

Skepticism is recursive

Being skeptical requires the skeptic to regularly go back and re-analyze his thinking. Often times I will learn something new in the process of working out my skeptical tendencies and that may mean something I’d previously thought may be wrong or, at the very least, in need of some revision and clarification. First and foremost, you must be skeptical of yourself because as a skeptic, your mind is the tool whereby your skepticism is operationalized. It has to keep its edge.

Anyway, what came up this week was some discussion of my thinking around engagement and motivation and whether or not that is at odds with something I’d written previously about learning having important prerequisites that, at first blush, sound an awful lot like engagement. I’d be a bad skeptic if I didn’t take a look back at what I’d written and see how that compares with my more recent skeptical take on engagement and motivation.

Disasters, Disruption, and What’s Needed Before Learning Resumes

Back in March I proposed thinking about school closures in the same way that we think about chronically absent students. As a society we seem to be in agreement that it’s really bad for a kid to miss 10% of the school year, or more. This is a crisis that has garnered national attention and proposed solutions to the attendance crisis are often broadly aware that attendance is a complex problem with lots of uniqueness in each community and even for each kid. For some reason, though, we do not think of schools in the same way. When a school is closed for 10% of the school year, such as in the aftermath of a natural disaster, we do not think of that school as “chronically closed” and we often focus our policy response on returning the school to normal functioning as fast as possible with most of the attention going to supporting students’ academic needs. What if, like with chronically absent students, we started to see chronically closed schools as in need of more holistic support for students, staff, and the community that relies on those schools? I wrapped up that post by arguing that students need more than academic interventions and that their academic performance is also downstream from other more pressing considerations such as their mental health and material needs.

Whence Engagement?

What, you may ask, does this have to do with my skepticism of engagement and motivation interventions in student performance? Well, the causal flow I advocate for in my disruptions post is kinda the same as the causal flow I am skeptical of when it comes to engagement and motivation. Interventions often assume that student performance is downstream from student engagement. That is, a student needs to be engaged and motivated in order to be academically successful. I am skeptical because, while there is absolutely a strong correlation between student engagement and academic performance, few studies attempt to show the causal connection between them. What if, I asked, academic performance was actually a prerequisite to students being engaged? I even found some evidence that this causal flow, from performance to engagement, may be the true one.

Moreover, there is some overlap between what engagement interventions look at and what I highlighted in my disaster disruption post. Namely, mental health. Engagement research often looks at students’ sense of belonging, safety, and wellbeing in the classroom or school. They assess students’ perceptions of their own competency and self-image. Sometimes they incorporate social or emotional learning into the interventions. It’s not a perfect overlap but clearly some of the same mental and material content is being covered in both.

And that gets us to the question I have for myself: if the causal connections between student engagement and academic performance are bi-directional or ambiguous, is there reason to suspect that the causal connections between post-disaster supports and student academic performance are also bi-directional or ambiguous? Therefore, I would ask whether I am overstating my case when I wrote:

Perhaps centering academics is an important part of helping students recover, just as academic performance may be part of building engagement?

Scale, Type, Differences, Similarities

I think, actually, the answer is no. I am not off base and I’ll give a few bulleted reasons why in no particular order.

• There is a difference in scale between these two topics. Disasters that lead to chronically closed schools are community-wide events that include non-scholastic disruptions and real threats to students physical and mental safety and wellbeing. Engagement interventions typically occur within the classroom in the context of a lesson or course activity, but the expectation is that they lead to school-wide improvements.

• There is a difference in type that also matters here. Although I do see some overlap between the mental health needs of students after a disaster and the mental health conditions that foster engagement, I also think they’re addressing a different typology. I don’t love Maslow’s hierarchy but one way to think about this is that kids’ psychological needs after a disaster are those at the base of the pyramid whereas kids’ psychological needs for becoming engaged learners are somewhere in the middle.

• Causal flows from disasters are not the same as causal flows from classroom interventions. We wouldn’t ever say that students’ poor academic performance causes school-closing disasters because that defies common logical sense. So, despite talking about supporting students’ mental, emotional, and material needs in both cases, the origin of the lack of those needs is fundamentally different and the root cause of the lack in the case of disasters is the disaster itself.

• My disaster disruption post is really about tradeoffs. Schools are often choosing to center academics and returning to normal academic functions at the cost of centering other functions, such as students mental and material needs. Interventions in student engagement are centering academics by way of engaging students and those other functions are only considered important because of the expected academic outcome. That is, nobody operating under this model truly cares if students are engaged unless that engagement maintains a tight correlation with academic outcomes. In a sense, they are guilty of the same de-prioritization of students’ non-academic needs. Whatever positives that may come with engagement (flow, perhaps?) are not important enough to be ends, only means.

• Epistemological differences and methodological differences are important here, too. The research I cite following disasters is largely observational because you cannot experimentally create the conditions of a hurricane and then subject some communities to the hurricane and others act as a control. Instead, researchers look at schools where the hurricane (or other disaster) has caused a disruption and follow various measurable outcomes in the students or schools. At most they may issue a survey of staff or students. Engagement studies, meanwhile, can occur under experimental conditions (although these are hardly laboratories so don’t expect physical-science level experiments). They can create a representative sample of students and test an intervention on some while others act as a control. Because experimental studies are usually considered to be more revealing of causal relationships, they are often considered a gold standard and to be more accurate representations of reality than other kinds of research. BUT one of the biggest caveats here is that the more rigorously controlled the study is, the further it gets from real-world classroom conditions. This means what we learn from experimental studies in education or psychology or economics sometimes cannot be replicated in actual schools or classrooms. Practitioners have to operate within an environment where their own situated knowledge may be more useful than what comes from experimental studies. Obeservational studies often find themselves a bit closer to that situated knowledge and therefore more applicable and relevant, even if it’s true that we can’t say with a high degree of certainty that what is known is the truest and closest to reality.

Warranted Self Skepticism?

Ultimately, I don’t think my writing about engagement and academic outcomes contradicts my writing about supporting students post-disaster. Although there are some connections between the mental supports I’d like to see students receive after disasters and the mental supports that are a part of student engagement, the contexts in which these discussions are happening are totally different and not comparable. That said, it’s good to go through thought exercises like these because it’s important to understand why engagement interventions and holistic disaster relief shouldn’t be compared.

Thanks for reading!