Ep 23. Red flags in education research with Ben Solomon

​

 

[00:31:15] Ben Solomon: Sure. A randomized control trial, which I'll call an RCT so I don't trip over my own words, is a discipline-independent design. It's used across different disciplines and people are probably familiar with those words from other areas in which we ensure groups are equivalent at the beginning of a study.

 

And we do this by identifying our sample first and then randomly assigning them to the condition so we ensure their equivalent at the beginning of the study. And what that does is it eliminates things we call threats. When we talk about threats, specifically threats to validity of a study, we're saying there's things in the study that might be occurring aside from the intervention, or more generally, the independent variable that explain the results.

 

If we don't randomize at the beginning of a study and maybe one group is substantially different than another. This can happen often in education because, let's say you're going in to introduce a pilot intervention, and all the teachers get really excited, so they say, “Oh, can you take student A, can you take student B, can you take student C, they really need help,” and so all these students end up in your intervention group.

 

But you're trying to compare that intervention group to some other group, usually a business-as-usual group. And now one group is loaded, right, with the students that are all at risk. The other group isn't. When you apply then the intervention to that group, but not the other group, the results can look really wonky and funny.

 

Because in this situation, what we have is called a selection threat. One group at outset looks very different than another. One way, probably the most efficient way in all of science, that you can eliminate these threats is by randomly assigning participants at the outside of a study. That doesn't eliminate all the threats, there's other things that can interfere that explain results that isn't the intervention. 

 

So if you say you're doing an RCT, that's not the final step. There are other things you need to show, but that eliminates a lot of those threats. And when we see a randomized control trial, that gives me as a reader a lot more confidence, not perfect confidence, but a lot more confidence that the conclusions the authors draw are sound and valid.

 

Now, there are some studies that are pretty good that don't use randomized control trials. There's a field of research methodology called quasi-experimental design. Those designs are far more complicated to do right. They tend to require much larger sample sizes. They tend to be rarer. That's a whole other thing.

 

And to do a high-quality quasi-experimental study is very, very effortful and, again, very complicated in terms of setting up the study and controlling for threats. Usually, engaging in more complicated statistics as well to wrap all that up. So I don't want to go there, you know, in education, we have numerous examples of really high-quality randomized control trials, which again, just like they do in the medical field, And that leads to conclusions that are more believable and more amenable to being considered within like a policy realm.

 

[00:34:30] Anna Stokke: The study I have in front of me is called “Testing the Immediate and Long-Term Efficacy of a Tier 2 Kindergarten Mathematics Intervention.” It's by Ben Clarke et al. I won't list the other authors. It's in the Journal of Research on Educational Effectiveness, 2016. 

 

And the study examined the efficacy of a kindergarten math intervention program called Roots. It's focused on developing whole number understanding in the areas of counting and cardinality and operations and algebraic thinking for students at risk in mathematics. So we're using this as an example of a rigorous, good education study because we're trying to learn about this today. 

 

So, can you take me through some of the elements that make this a good study?

 

[00:35:19] Ben Solomon: Sure. And I also think it's important to mention I've actually never met Ben Clarke, you know, I respect his research, but I, it's a completely independent review. In reading this study, I think there's a couple sort of features that identify it as a high-quality study that again, in isolation, wouldn't be used to make policy changes, but as part of a sequence of studies that show a pattern that might be used to inform policy for thousands and millions of students.

 

And so a couple things. One, if you look in the introduction, they set up the problem really well. It's a well-defined problem. They look at the 2013 NAEP scores. They show that students are, severely underachieving. They briefly discuss, means that, that might be, resolved. And this all leads to reasonable research questions and then hypotheses. That setup in of itself, usually lens confidence because it shows you that the researcher’s sort of understand the problem and there's a logical flow to how they develop their intervention. 

 

Another thing to notice is it's a randomized control trial, which we just discussed, you know, that's again considered a gold standard and, you know, dollar for dollar is the best way to, you know, have a high-quality study. And you might also notice that it's funded by the Institute for Educational Sciences, or what's often known as IES.  

 

IES funds a lot of more sort of large scale studies nationally in the US. That doesn't guarantee that those studies are good, but they only fund studies that have made really good proposals, in which they plan to attend to a lot of features about their studies in terms of the reliability and validity and the sample and student dropout and the kind of statistics they'll use and where it will do the study.

 

If a study is funded by IES that means they've at least planned for that study to be very rigorous. So when I see IES-funded research, again, that doesn't guarantee it's a good study. That's a point, if you will. Another thing I appreciate about the study is it tests a realistic intervention sort of, you know, framework.

 

It was 20 minutes long. It was done daily. It was done in ratios of 5 to 1 or 2 to 1. And so immediately, I think, oh, this is something a teacher could do. It's not super intensive. Sometimes you see, for example, studies that test like these hour-long interventions that are done one-on-one, and it's wholly unrealistic in a school.

 

So another feature I appreciated was that it tested an intervention that you could see they had an eye towards implementation at scale, and that's really important. They identify a business-as-usual condition. And they have a large sample. I think they had 37 classrooms and 290 students in the study.

 

That's a pretty healthy number. That's on the larger side. As far as these things go, some get much bigger, but 290 is a, is a good sample. You know, we just talked about the Kamii study, they had you know, I believe it was 57 students. You know, that's on the smaller end. So the larger the sample, the more confidence we can have that the conclusions would replicate again. They painfully discuss their procedures, and I mean that in the best way. 

 

They talk about what the intervention is going to focus on. They talk about how they focus on whole number knowledge. They talk about conceptual and procedural knowledge. They talk about count counting and cardinality and operations. So they really flesh out what this intervention is about.

 

And they do that so we as a, as a reader or as a, as a reviewer can evaluate that study and really think how that content jives with other good studies that are sort of competitors to this one. And it's done in a way that it's, you know, that the intervention is scripted and that the study could be replicated.

 

And that's not easy to do. It's a lot of effort to write out a study in this way with this level of detail. They discuss what's called statistical power which is basically, it's related to sample size, what's their chance that they'll find an effect if there is an effect in the study, and they talk about dropout of students too. They talk about their measurement system and they talk about what are known as proximal and distal measures. 

 

Proximal being measures that are sort of really closely aligned with the study and should change drastically because it totally reflects the content of the study and they talk about distal measures, measures that are sort of unrelated to the study that they didn't create. You know, that should reflect sort of broader achievement and it can be harder to find effects on those distal measures. You know, they're sort of challenging themselves. They still put those measures in even though they may not favour them as much as the proximal measures.

 

And that's really important because those are the big ticket items. They discuss fidelity of implementation of the study. They made sure that they implemented it the same way every time. And they had a lot of checks and balances for that. There are statistics I'm not going to get into, but I'll just generally say they were very rigorous.

 

And they talk a lot about what analysis they were going to use. And then they try other analyses afterwards to show that, you know, that the first patterns were, were correct. So the fact they put all that effort into that was, you know, reassuring. They discuss how their results fit within a pattern of results found across studies, and last, and this is really important, they have a huge limitation section. 

 

You know, the limitation section of a paper is where the researcher talks about all the things that could have gone wrong or reasons maybe we should doubt their conclusions. And you may think, “Well, why is that good that they had such a long section?” Well, because they were honest. They really thought out, “Here's all the ways that what we just concluded could be limited. Here are some of the ways it could be wrong, even though it was still a very rigorous study.”

 

So when you put all that together, you know, for a researcher to consider all those features of dropout, of statistics, of design, it's not easy. It requires enormous effort and years-long planning. So when you see all that mentioned in that kind of way, that gives me a lot of reassurance that the conclusions from that study were good and sound.

 

And again, probably the most important feature of the study was that it was a randomized control trial. That's the bedrock, but there's so much else in there that boosts the confidence a reader can have in the conclusions the authors drew.

 

[00:41:43] Anna Stokke: And I'm going to post a link to that paper on the resource page and maybe it'll have your notes too. What I'm sort of hearing is this paper is quite scientific. They've been very careful in making sure that they're following maybe a scientific method.

 

So that brings me to my next question. Is education science? Like is learning a science, is there a science of learning?

 

[00:42:07] Ben Solomon: Yes. It's interesting that we're still having this debate because in, in very few other fields are people having this debate, right? No one says, well, is physics a science, Or is medicine a science? Those fields have evolved , well, well, well past that point where now the scientific method guides their entire epistemology, their entire framework for knowing.

 

In learning, unfortunately, we're still having that debate. We're still having a discussion as to whether learning should be construed as a science. My strong opinion is yes, it should. The scientific framework accelerated the growth of those other fields. Think of the discoveries that were made in biology and physics and astronomy and medicine once they adopted the scientific framework. 

 

Civilization as a whole, went off onto the highway. That's where we need to be with education. That framework worked for all other fields that adopted it. We should be using it in education to promote the best outcomes for our students. And so far, you know, when it has been applied, it's been shown to work. When we look at the data on schools that have adopted interventions and curricula that have gone through rigorous testing using that framework, those students have been shown to do much better. And there's no reason at this point that can't be elevated at a national level.

 

That doesn't mean that all we should be looking at is hard data. I really respect that in education, there are stories to tell, that there are perspectives that need to be shared that sometimes as scientists, we need what's called qualitative data to really get a good sense of how certain practices are affecting people.

 

That's important. I don't want to dismiss that. But at the core, using evidence-based practices and using data to inform our decision-making has been shown to result in the best outcomes for our students. So, yes, there is a science of learning. We just need to more broadly adopt it and, you know, use those findings at a much larger level to accelerate our students' learning.

 

It can happen. It has been shown to work at local levels. We can do this.

 

[00:44:35] Anna Stokke: But one issue is the following. We talked about the Kamii paper and then we talked about this Clarke paper and they're both published in journals, They're both, you know, sold as research. One is really rigorous research; the other isn't. How can a teacher or parent tell, though? To someone just looking at this and hearing that, “Well, there's this research that says that algorithms are harmful,” and they look and, you know, it's published in a journal and, the people who published it, one of them was an education researcher, so why should we think that there are problems with it?

 

This is a big issue. So here's the thing, if you're a teacher or a parent and you're given a research paper as evidence for a claim, what do you do? Like, what are some red flags that we can look for that might indicate that there are problems with the methodology?

 

[00:45:30] Ben Solomon: That's a really good question. And it's really hard; I've been in situations where I go into schools, and an administrator will put their hands up and say, “Well, peer-reviewed research can say all kinds of things, so I don't believe any of it. Anyone can say anything in peer review.”

 

And so they dismiss that entire framework outright. And it is true that lots of things are said in peer-reviewed research and different journals emerged out of different interest groups and sometimes they reflect good high-quality science, but other times those journals sort of elevate just out of sort of shared interest amongst the people within it.

 

So, unfortunately, just saying a paper is peer-reviewed is not enough. A peer-reviewed paper increases the chances that those papers' claims are true, but it doesn't, by no means, ensure it. Any person that is handed a paper as you just described, they need to apply what I call an analytical tool belt. They need to have some sort of rules of thumb, some perspectives sort of ready at hand so they can critically evaluate the paper. I thought of the best way to make a shortlist and I shared it with you. And I think there's five big ones that anyone can use to sort of critically evaluate a paper.

 

The first I would say is exaggerated claims and hyperbole. If you look in the introduction of a paper, what kind of claims are they making? Are they claiming they can change night into day?  Water into wine? Are they claiming they can change a student's literacy or math proficiency overnight with little effort?

 

If they sort of set up those kinds of propositions within the introduction, or they describe the study they did in those kinds of terms in the discussion, you know, immediately we should be highly skeptical. It takes a lot of work to teach a child and there are no magic bullets in education, that has been shown.

 

When you look at high-quality research like that Clarke et al. piece, they're skeptical of their own findings. You know, you can see that they're really giving it a good critical evaluation or the responding to peer reviewers that have given it a good critical evaluation. So if interventionists or authors are overly optimistic in their claims, that for me is one of the first red flags and you often do see it.

 

For example, I've seen it in the brain training research. Sometimes you see researchers who claim with brief video game like activities, they can permanently change an adult or child's working memory. That's a huge claim. If I see a claim like that, I'm going to be not closed minded. I'm not going to be cynical, but I'm going to get very skeptical.

 

I'm going to be looking for really high levels of evidence. So those big claims sometimes are the first, what we'll say, red flag. The second that I think anyone can look for. Are meaningful and measurable criteria used to index the effect of the study. So if you look in that Clarke et al. paper, they had five or six measures, I believe and they explained the reliability, they explained why they're relevant for students. They explained how they predict bigger outcomes. 

 

You know, they really spent some time defending their measures and showing how they're socially meaningful. You compare that to the Kamii paper, right, where they did a brief interview with students based on a single problem. And again, that may be the start of a good line of inquiry, but that in of itself is not sufficient. And I think anyone would be able to see that. 

 

You know, getting interviewed on a single problem is not predictive of how you might do math five years down the line. So we want to look for meaningful, relevant measures embedded into these studies to really objectively evaluate, you know, what that intervention or practice did for that student. Authors will often go at great length to discuss their measures and prove to you that they're meaningful.

 

Another red flag I would point out is what we might call unusual experimental conditions. Sometimes you see in these studies that they engaged in an intervention and maybe it did work, but the conditions under which they did it were sort of funny. Maybe it's again, as I said before, one-on-one in a lab setting. Or maybe it, they did it for one or two hours at a time.

 

You know, there's just not that much time to engage in intervention in a, typical school schedule. So if the conditions don't look like how your child is educated in the school, it doesn't look like something that could be implemented within a school environment, you know, that immediately leads me to be skeptical because they're almost stacking the deck.

 

Another red flag is the research design, which we just discussed. We want to look for really high-quality research designs and looking for, you know, a randomized control trial is a great way to efficiently root out a lot of the more questionable research. Again, not all great researches are randomized control trials, but a lot of it is. And when we see other words to describe a study's design: correlational, case study interviews, descriptive study, again, it doesn't shut us down, it doesn't immediately mean we stop reading and the study isn't good, but it does raise some red flags in terms of really, trusting the conclusions from that study. So we do want to look for high-quality designs. 

 

And the last red flag I would mention are questionable sort of citations referenced within a study. And I see this a lot. I do a lot of peer-reviewed work. I'm on the editorial board for three major journals. And so I am peer reviewing manuscripts left, right, and center. And oftentimes, when I'm peer-reviewing these manuscripts, I can see a study sort of heading off the rails, if you will.

 

They start citing a lot of conceptual papers. They start citing books. posters given at a conference, newspaper articles. And you can tell they're avoiding citing peer-reviewed research because there may not be a lot of it in the area that they're describing, or they know a lot of that peer-reviewed research says something completely different.

 

And so when I review a study, I want to see that study cite contemporary, high-quality, peer-reviewed work. That doesn't mean they can't cite, you know, a book chapter here and there, or someone's argument from a, you know, from a newspaper. But the preponderance of citations of research shown to set up the study should be high-quality, peer-reviewed work.

 

And oftentimes when you're in schools or you're having these debates with different people and, you know, whether it be online or in person I find in some of the weaker arguments, they immediately throw up someone's blog. And again, that blog may be good or bad. I don't know. But again, we always want to go back to the peer-reviewed research, and then even within that peer-reviewed research, we want to look for certain indicators of a high-quality study. 

 

So, you know, again, the five I think are good, efficient tools, you know, to keep in your toolbox when you're reviewing a study, even if you haven't done a thousand manuscript reviews: look for exaggerated claims, look for meaningful and measurable criteria, you know, to measure the outcomes of the study, look for unusual experimental conditions, you know, look to see whether that study test their interventions in a situation that makes sense, look to the research design and then look to the kinds of authors and other research that study sites to set up their own study.

 

[00:52:57] Anna Stokke: These are great and this is so helpful and I'm going to put these red flags up on the resource page and we can all carry those around in our pockets for the next time. We hear an exaggerated claim. One thing I wanted to mention, in post-secondary teaching there's also always sort of this debate about active learning and then traditional instruction. 

 

And my colleague mentioned to me that he'd looked at quite a few studies comparing these different methods of teaching, and often, in fact, almost always, the traditional instruction takes the following form: Instructor lectures at the front of the room, and there's absolutely no participation in the class. No questions, the instructor doesn't ask the class any questions, basically just lectures the entire time. But realistically, that is a bad way of teaching. 

 

Like, I consider myself to be on the traditional side when it comes to teaching, but there's a lot of engagement in my class, lots of participation and lots of questions. So I think one thing that happens in these studies is the method of teaching that people don't want to be the one that the study favours, they mischaracterize it so that it won't come out to be the best method of teaching in the study. Have you observed that?

 

[00:54:22] Ben Solomon: Oh, absolutely. So it's really hard to do what we'll call comparative research where you test one thing against another because inevitably the author probably has a bias towards one condition or the other, and there's always the opportunity to mischaracterize, you know, the other condition to set yourself up for success.

 

So we, we see that often. And it's really important for the authors to go to great pains to show that whatever they're comparing their intervention to that it is done correctly and it is done sort of in the spirit of how those authors, you know, developed it. That can be really hard. And again, that should be something that's extensively discussed within the study because you really bear the burden of proof there. 

 

And to give another example, we've been discussing the Kapur article, “Designing for Productive Failure.” And in that situation, they described direct instruction (DI) as sort of their comparative condition, their business as usual, but it's really unclear what that DI condition was, how it was set up and whether it was being implemented in a true DI spirit. And again, If you don't describe it very well, that opens the room for bias again, not necessarily intentional bias, but nevertheless, you know, everyone sort of implicitly is going to try to work to favour their products, you know, and their inventions.

 

So I see it a lot. And the authors have the burden of proof to show that they weren't biased and implementing sort of their counter condition.

 

[00:55:46] Anna Stokke: Yeah. And you mentioned the Kapur article and that is an article on productive failure. And we've sort of been discussing that in an email chain about some issues with that paper that we'll leave for another day or another place, because we had to do the Kamii paper on standard algorithms today.

 

Another thing I wanted to mention, I think people often assume that something's true when they're told it's true by someone who is considered an authority on the matter. So an example in reading would be Lucy Calkins. And certainly we have examples of math education celebrities that will be cited, their claims cited as evidence for sketchy things sometimes. But we should know now that This is not necessarily a good indication that something is evidence-based, and we should know this because of what happened with reading.

 

And again, if people haven't listened, I do recommend that they listen to my episode with Matt Burns. As an example, we heard about Matt Burns’ research on the Fountas and Pinnell Benchmark Assessment System that he basically found it was ineffective. Yet, Fountas and Pinnell haven't acknowledged that there are problems with the program, despite the research evidence.

 

So, how can this happen? Shouldn't researchers or educators actually change their mind to be in line with evidence?

 

[00:57:14] Ben Solomon: You know, so this is a problem that's sort of endemic to a lot of different places. It's not just education, right? When Matt did that wonderful research, looking at the Fountas and Pinnell system, that vendor had a financial interest to ignore that research.

 

So in that situation, what finally created some level of concern was the grassroots movement, when finally parents and teachers absorb that research and then elevated it from researcher to vendor. That rarely is a situation sometimes, but rarely it's a situation where you see a lot of change.

 

It's complicated, at the university level, in schools of education, you have researchers doing all kinds of things and they tend to be isolated. We tend not to have great collaborations amongst each other. And so people can get into these tracks where they sort of reinforce their own beliefs and develop studies that, favour their ideologies or philosophical opinions.

 

And then that all finds its way out of the woodwork into schools, which I think was one of the major problems of Lucy Calkins. You know, within Columbia, you know, Lucy Calkins developed this bubble. She was very profitable in what she was doing both for herself and the university.

 

Although there are fantastic methodologists at Columbia and there are actually, fantastic direct instruction, precision instruction people there, she operated in her bubble. And that, unfortunately, happens in universities and something, to be frank, we really need to work on because then, in the eyes of the consumer, it looks like every one of these researchers has an equal opinion or has equal credibility or equal weight when that's not the case.

 

From a more consumer side, there's an older researcher named Raven who developed what are called power bases and what these power bases are is basically the sources of power a given coach, or specifically, consultant has to change people's minds.

 

It's sort of a cute heuristic. I don't know how overall accurate the framework is, but they talk about the expert power. They talk about how when you, as a consumer, when you see someone who quote-unquote has “expert power”, you believe them because they have these credentials. Who are we to disagree with someone who identifies themselves as a learning scientist? But then at the top, as I just discussed, we have these people sort of entrenched in their own beliefs. 

 

So it can create a very confusing system. The other thing I think that is worth mentioning, sort of, is quite frankly, sunken cost. A social phenomenon where people who invest a lot of time or energy into a certain initiative or belief, you know, it's going to be harder to nudge them out of that because they've invested so much time into it And they don't want to look like a fool. They don't want to look like they've been going down the wrong path for 10, or 15 or 20 years whether it be a teacher in a classroom, whether it be a principal or whether it be a researcher at a university.

 

For example, with the citing science of reading movement, as it really evolves into sort of everyday conversation some researchers have been producing papers on the debated methods for 30 years to nudge them out of that's going to be very difficult. And it's naive to think that everyone's going to sort of evolve on their own. 

 

I don't know if I have a great solution to that, but it is definitely a concern. We have both serve at the school level and then at the sort of the academic level. And again, I think it's why it's really important that whether you be a teacher or whether you be a parent or a principal, that you be a skeptical consumer of research,

 

Be open-minded, listen to people's arguments, but ask for evidence. And when they produce evidence, evaluate it for yourself. Because a lot of these problems in terms of, you know, evolving and adjusting to the current research, it's not just a problem in schools and in practice. It's a problem at the university level too.

 

[01:00:56] Anna Stokke: And you mentioned sunken costs, and that's a really good point. I don't think the people that have invested years in these, and they're philosophies, really, a lot of these things are philosophies or ideologies. Even though you do see these things getting published in journals, they're still philosophies and ideologies. 

 

I don't think they're likely to take a step back. However, I do think that public funds shouldn't be spent on programs that are likely to be ineffective. And I think that something needs to happen there and I'm not sure what. I mean, Matt Burns mentioned something like an FDA for education. What do you think about that? 

 

[01:01:41] Ben Solomon: I completely see where he's coming from with that and it makes sense, right? You want some level of quality control at, the federal level. We have it in other areas and at this point we certainly know enough about instructional learning that we can make some confident recommendations and what should and shouldn't be part of that.

 

What I would say, however, is that, unfortunately, that would be very difficult to implement, at least in the states, just by the way our educational systems are structured, By design, there's a lot of power at the local level individual schools, individual districts, individual states get enormous decision making power on what occurs at the student level. 

 

And so if we were to set up some kind of FDA-like system at the federal level, and I'm sure I'm sure Matt understands this is that's probably more idealistic than practical, but the point absolutely still stands, to set something up at the federal level would be very difficult and there'd probably be a lot of kickback because, you know, at least in the states, there's such a belief that the local community should have such a heavy say and choice in what occurs in the school.

 

So with that said, I think the more efficient solution for the time being would be to inform parents and teachers to the best of our ability and have them be the people demanding high-quality practices for their students. We saw it in the science of reading; that was a grassroots movement where finally, journalists and teachers and educators got together and organized; they pulled the science together. 

 

So they were in a very defensible position. They, they brought in researchers, they didn't do it on their own. They brought in, researchers and professors and academics and they got together and they demanded better practice for their students. And that resulted in an enormous change that is still unfolding in exciting ways.

 

As we think about math now, which is unfortunately a few steps behind, I think that's the better route is to promote high-quality education, high-quality science and research literacy, and really try to, you know, bring this to the school boards, bring this to your local politicians, and say, this is what I want, this is what the research states, and this is what my child deserves.

 

[01:03:50] Anna Stokke: Yeah, absolutely. We have to do this for math. And reading is important, but math is also really important. And when children do not get a good math education, they're shut out of a lot of careers. And I would argue that also for our economy, we need to make sure that we have students that are able to perform well in mathematics. We need them for those quantitative careers,

 

[01:04:14] Ben Solomon: Absolutely. You know, I think there's this assumption that, technology is at such a state that it will just carry us through. We're having discussions at the university about how we integrate ChatGPT, and do we teach foundational skills now that ChatGPT can do anything the answer is yes. Yes, you still need to teach the foundational skills because you still need people to understand the underpinnings of what's going on. And there are everyday math skills that are still really important. So we need to, I think, double down on the foundations. At the elementary, secondary and post-secondary levels. you need high-quality math skills in everyday life regardless of the field you're in. You open yourself up to a naivete and a certain level of ignorance if you can't do basic computations. 

 

There's a huge risk in being illiterate that has been well documented. It closes all kinds of doors. And the same is absolutely true in math. And that's the case even if, you're not aspiring to the highest level of STEM education.

 

[01:05:10] Anna Stokke: So, is there anything else you want to add today?

 

[01:05:12] Ben Solomon: I think what I'd like to conclude with is to remind everyone it's interesting that science never proceeds linearly. You know, it's always in punctuated steps. It's not incremental improvement. We tend to have these fierce debates, they tend to come to a head, hopefully some level of truth is revealed, and then that field accelerates based on that discussion. And maybe sometimes it's a little static afterwards. And then it comes to a head again later on.

 

I'm hoping we're at a point right now where we're having one of those fierce debates. Where all of a sudden things are coming together based on this great research that has been done over 30 or 40 years. And it's finally really coming to a head and people are discussing it and we're talking about it on a podcast.

 

And I'm hoping out of this comes that acceleration where nationally it reaches a policy level and good practices come out of this and they're implemented with far greater distribution than they currently are. I think we can get there. You sort of see all the ingredients in place for people to really absorb this stuff and then carry it forward. 

 

So I'm very optimistic. I think these kinds of discussions are really valuable. There's a lot greater visibility with all the research. So let's accelerate out of this, to really improve outcomes for students.

 

[01:06:25] Anna Stokke: Well, thank you for ending on that positive note. And I want to thank you so much for coming on my podcast today. I really appreciate all the work you put into helping. prepare for this and coming up with the red flags document and I'm going to post all that and thank you so much for sharing your expertise with me and my listeners today.

 

[01:06:44] Ben Solomon: Oh, it's been an absolute pleasure. Thank you so much.

 

[01:06:47] Anna Stokke: More in just a moment. If you are interested in the science of learning and would like to learn more, consider attending researchED Canada, which will take place in Toronto, May 3rd through 4th of 2024. I will give a presentation there, as will many other guests I've had on the podcast. I will include a link to the website in the show notes. 

 

As always, we've included a resource page for this episode that has links to the two articles discussed in the episode, as well as the list of red flags we talked about. And I'll include a link to that in the show notes as well. Follow me on X for updates on new episode dates.

 

If you enjoy this podcast, please consider showing your support by leaving a five-star review on Spotify or Apple Podcasts. Chalk and Talk is produced by me, Anna Stokke, transcript and resource page by Jazmin Boisclair, social media images by Nicole Maylem Gutierrez.

 

Subscribe on your favourite podcast app to get new episodes delivered as they become available. You can follow me on X for notifications or check out my website, annastokke.com, for more information. This podcast received funding through a University of Winnipeg Knowledge Mobilization and Community Impact grant funded through the Anthony Swaity Knowledge Impact Fund.