Ep 24. Raising student achievement with Dylan Wiliam

[00:30:24] Dylan Wiliam: Absolutely, but your example of diagnostic tests were also formative. The problem, of course, is that there's a really weird problem with diagnostic testing, which is in most of the domains that we actually use them.people would much prefer a diagnostic set of subscores. Rather than giving me a score for math, I'd much rather have a score for numbers and algebra and probability and geometry, for example.

Because that feels more useful. Turns out that the errors of measurement of those subscores are pretty large because they're based on far fewer questions. And so, in math, there is usually zero diagnostic information in those sub-scores. You'd be better off relying on the total test score, even if you want to make an inference about a particular sub-score. Everybody likes diagnostic assessment because we think that tests should do double duty. 

But in fact, in many domains, particularly in science and math, those sub scores have zero diagnostic value.

[00:31:26] Anna Stokke: So let's actually talk about some specific techniques that could help teachers, and I think the book that you recommend that would be really great is Embedding Formative Assessment. That's your book that gives specific techniques for formative assessment. So maybe teachers could use that book, but I want to say that I’ve been quite guilty of, you know, I teach and I think I’m doing a great job and I ask the whole class questions, and I teach at the university level, but it's the same idea, I should be thinking of ways to have formative assessment in my class too. 

And I ask a question and generally, you know, I might have 50 students in my class and I have maybe around 8 to 10 students who generally answer all the questions. And then there's the rest of the students. And you don't really know what's going on there.

So it could be that sometimes the students are too quiet to speak up, or it could be that the students are confused and I don't know and I move on because I think that I've gotten the right answer so everybody understands. So I’ve been trying to change that as well and one of the things that I've been doing, and I got this idea from Patrice Bain, who came on the podcast earlier is cue cards.

Every student in my class has these index cards. They have A, B, C, D,  I make up questions as I go along, multiple choice questions, and everybody has to hold up the card. One of the things you also mentioned was cold-calling. I'll admit, I'm very hesitant on cold calling. So you're going to have to convince me. I think, I have students that would literally drop the class if I started cold calling.  I don't know what to think about it, so maybe you can talk a bit about cold calling.

[00:33:12] Dylan Wiliam: Well first of all, I’m not sure that you know that your students would drop the class if you cold-called, I’m certain you believe that. There's a nice series of experiments by Dallimore and her colleagues where they looked at how, this is a college level, BS in accounting and management sciences. 

And they looked at the teachers who used cold-calling, and they actually found that students, originally, beginning of the study were really apprehensive. But the more they use cold calling, the more students participated, the more relaxed they were about cold calling. And in fact, the females and answered more questions than the males.

So this, this is a quite interesting research here. But yeah, absolutely, people are worried about cold-calling. So what we realize there's two reasons to do cold-calling. So Doug Lemov doesn't like the idea of random calling, because for him, cold-calling is a way, particularly in English or history, to bring in a student you haven't heard of. I think you might have an interesting perspective on this.

So you bring in a student who hasn't raised their hand because you think they might have something interesting or different to offer. That's a good reason. But I also value the increase in engagement. So, I actually haven't done this in a while, but when I used to lecture at the Institute of Education, I would lecture all our teacher training students, and we had literally a thousand of them, in a huge lecture theater.

And I would ask them about a study about the effects of feedback, for example, and I'd say, “What do you think happened in this study?” And there's five options, and I would ask them to use finger voting. And I would then pick one of them at random and say, “Why did you choose A?”, “You chose C, why was that?” I then interviewed the students afterwards.

I do this two or three times in a lecture, and I said, “Did the fact that there was a one in a thousand chance that I would pick on you for a response increase your engagement?” And every student I interviewed said yes. It's that dual role of getting information that you wouldn't get from the confident students, but it's also that idea that at any point you could ask a student to answer a question or to comment on another student's response. 

So that's one technique, is this idea of random questioning. But before we go into the others, I think there's another point I'd like to make, which is that our starting point here, in terms of professional development, is that each teacher should choose which techniques to work on.

So our process model is choice, asking teachers to work on the things that they want to work on, rather than trying to have every teacher do the same thing. Encouraging them to adapt them to fit their own classrooms and context. Allowing them to work on one or two ideas. For months, if necessary, and then making them accountable for making changes and providing support.

So the choice is really important, and so some teachers choose wait time, for example. Now, wait time is really interesting because every teacher I know knows they don't wait long enough after asking a question to give a student a chance to respond. And yet reminding teachers of that research has zero impact on practice.

It has less of an impact as reminding smokers of the harmful effects of smoking because it's not a knowledge problem, it's a habit change problem. And so one of the things I've been trying to do in my own teaching, now with, teachers, is I've been trying to move away from questions to statements. It turns out that if you make statements rather than asking questions, people tend to give longer and more thoughtful responses.

I'm not sure why this is, but my hunch is you can be wrong answering a question, you can't be wrong responding to a statement. So it's about having a mindset where I'm just thinking, “How can I get more out of my students? How can I get my students to share with me what's going on in their heads?” Because if I can't get that information, I can't teach effectively.

And so, for the last, basically, four years now, I've been trying to move away from questions and towards statements. “You chose A.” Rather than saying, “Why did you choose A?”, “You chose A. I'm interested in hearing a little more about that.” It seems a gentler approach. Making statements rather than asking questions. Focusing on wait time. Using multiple choice questions. One of the things about multiple-choice questions is that it's really hard to find good responses. So one of the things I like to do is to give students little 3in x5in little index cards, and you ask a question at the end of the lesson, so-called exit ticket or exit pass, and then you look at the most interesting incorrect answers, and you use those as distractors, as incorrect options, in your multiple choice questions.

You know, makes sure that the responses you're offering are things that students actually thought. Which makes the questions more revealing, and recent research has suggested that asking the question before you show the options increases long-term retention.

And very much more recently, asking students to rate their confidence that this response is correct when compared to the other options they've been offered, and interestingly, asking students to rate their confidence in their answer had zero impact on long-term retention. But asking them to rate it versus the other options seemed to increase the amount of engagement that the students had in answering the question, which, of course, then causes long-term retention. 

So, you know, there's lots of textbooks like this. There's, there's 50 of them in the book that you mentioned. People want the 50. Give me the list. And then they just choose one and try it once and go on to the next one. What I'm saying is, we need to be helping teachers accumulate. I've lost track of the number of times I've described a formative assessment technique to a group of teachers. And somebody says to me, “Oh yeah, I used to do that, it was good.”

“I used to do that, it was good, but something else came along, and I don't do it anymore.” So if we're interested in making this change sustainable, we have to give teachers time to make these techniques part of their classroom habits. So I'm very happy if a teacher works on wait time for six months.

Because once you get to that point, you never go back. And once you understand the value of wait time, you are generally horrified when you visit other people's classrooms and you see how little time students are given to think. So it's about changing that orientation towards, you know, “How can I get more out of my students? How can I get more coming back so that I can make better decisions about what to do next?”

And of course, them articulating their thoughts is also good for their own cognition. So it's that two-way process of me getting better evidence to make better decisions that results in better learning.

[00:39:54] Anna Stokke: And those are really great tips. And so you mentioned that the book has 50 ideas in there, try some of them and try to stick with them. I actually want to pick up on that point because. There's a lot of switching around in education from what I've noticed, so if people don't maybe stick with things long enough or they stick with the wrong things, and I've heard you say actually that it's self-indulgent for teachers to spend time getting better at things that actually don't help children learn when there's evidence about things that do help students learn. 

And I think that is actually a really important point because, and I don't always think it's the teachers who are doing this. I think it's sometimes district administrators bringing in professional development from people that are promoting methods that are actually unhelpful for learning. I mean, I see this all the time in Canada and so I'm wondering if you can elaborate a little bit on that.

[00:40:56] Dylan Wiliam: I think it's difficult because educational research is very messy. We don't have the clarity of evidence we'd like. So, the work of Robert Pianta at the University of Virginia, and Bridget Hamre, has shown that roughly 25% of students get reading in elementary school. Their progress is independent of the quality of teaching they receive.

So these kids just take off, they're called fast readers. For the other 75%, their progress in reading depends on the quality of instruction they receive. And so people who want to say, “Well, this approach really works,” can always point to a lot of students for whom it did work. And so, the stuff about inquiry-based versus explicit instruction.

You know, people will say, well, “inquiry-based is better”, or “explicit instruction is better.” We're now understanding, with a bit more nuance, that it depends the level of expertise the student has. So, in this situation, it works; in this situation, it doesn't work. And so, we have to be quite careful about what is our evidence base.

We get these things that are called aptitude treatment interactions. It works for some students and not others. We also have to be careful what kind of thing we're focusing on. So the work of David Geary contrasts biologically primary and biologically secondary knowledge.

So biologically primary knowledge is the knowledge of things that we are basically evolved to do. So, recognizing faces, speaking, and listening. You don't need to teach these things. In fact, they can't be taught. They're not innate, but they are learnable if students are exposed to a rich environment.

Other things, like writing and math, he describes as biologically secondary. And those things were not evolved to do, we need explicit instruction. So often, I see people arguing for play, but they're arguing for play in things that would probably be more beneficial with direct instruction, because they're biologically secondary.

On the other hand, I see people using direct instruction for things that really are biologically primary and therefore, you can't do much about it. So John Sweller suggests that transfer and generalization are biologically primary things. We are evolved to do those things. So what you need is not to teach people how to do this, but to expose them to an environment where they explore those ideas themselves.

So it's getting those things right. That's why the research is so inconclusive because everybody can find evidence to support just about any point of view in educational research.

I personally think that in most situations explicit instruction, direct instruction as Siegfried Engelmann called it, is highly effective. And in fact there's now research that shows that achievement causes motivation at least as much as motivation causes achievement.

The fact is that many teachers are faced with students who are bored and who'd rather be doing something else. And so, I think that our focus shouldn't necessarily be on achievement right now. The thing I'm convinced of is that 50 years ago, we could probably claim that students could learn everything they needed for the rest of their lives by the time they were 18.

Now we know that's not true. So, for me, it's much more important that our students leave us at the age of 18 with a desire to carry on learning. If you teach your students a lot of stuff but extinguish that passion for learning that every five-year-old has when they arrive in school, you have failed that student.

So I can see a rationale for inquiry-based instruction. First of all, in developing things like curiosity, which is biologically primary. And also, in just breaking up the pattern of study. So, for me, it's much more complicated because the aim is not just raw achievement right now. We also have to be making sure that our students are able to carry on learning and then also understand how to carry on learning themselves so they can learn how to learn. 

[00:44:59] Anna Stokke: You talk a lot about formative assessment, but what about summative assessment? Is summative assessment also important?

[00:45:06] Dylan Wiliam: It is. It's the backdrop. the nice thing about summative assessments, especially when they're externally set and graded, is they make the teacher into the student's ally. So let's take an extreme case. In the U. S., the grades that students get from their teachers determine which universities they get.

And therefore, students see their teacher as the enemy. The teacher gives you a B when you need an A to get into your college of choice. In the European system, also in Japan, there's an exam set by external people, scored by external people, and the teacher is your coach.

Like trying to get the athlete to clear the bar, a high jump bar at two meters. I've never seen a high jumper say to the coach when they fail to clear the bar and the coach says, “You failed.” I've never seen an athlete say, “Well, that's just your opinion.” It's an objective standard. And so the teacher, in that examination-based system can become the student's ally.

I think increasingly we're going to see AI making the traditional coursework grade-based approach increasingly impossible to implement fairly. It's now clear to me that AI can't detect AI. So I've tried doing things with AI checkers. I asked an AI package to write a an assessment policy for a high school, it did a pretty good job. 

I submitted that to an AI checker. It said there was a 75% chance it was produced by AI. I deleted the first sentence and then ran it again, and this time it was only a 4% chance. So basically, students can now submit their work to an AI checker, and it says, yes, this is going to get tagged as being AI produced, so what they can then do is put it through a paraphraser and students can keep on checking with an AI checker until this actually comes back as something that is not likely to get detected. 

So I think that whole coursework-based model is going to be impossible to maintain. And so we can have examinations, we can have controlled conditions, so students don't take work away from the classroom, but we actually have them do a piece of writing in a class where we can control access to AI, and the other approach we might be able to adopt is to have some way of judging the provenance of a piece of work. 

So we might ask students to submit drafts and see how they've taken the feedback they've been given and used that to improve. So that might also help. But I think exams are going to become more and more important because of the way that AI basically makes it impossible to authenticate work that students take out of the classroom.

[00:48:03] Anna Stokke: You mentioned an exam in, in the UK. Is it written at age 16?

[00:48:09] Dylan Wiliam: Yeah, so basically there's a, students will typically take 10 exams at the age of 16. English, math, science, history, geography, music, whatever, and then they will focus down. So in the last two years of high school, they will choose typically three or four subjects.

[00:48:27] Anna Stokke: And that's written by an external body and every student writes the same exam.

[00:48:33] Dylan Wiliam: There are competing boards, so groups of universities have their own examination agencies, but I think there's now, I think it's down to three now in England and , the rigour of their examinations is examined to make sure that they're not making the exams too easy to attract business.

​

[00:48:53] Anna Stokke: Okay. And what's the purpose of the exam? What do they use it for? What do they use the results for?

​

[00:48:59] Dylan Wiliam: Well, the exams taken at 18 were basically for university entrance. So, you get certain grades, so, you know, you apply to a university, and the university will say, Okay, yes, “we'd like you, and we will offer you a place if you get two grade As and one grade B in your three subjects at the age of 18.” These exams were originally written by the universities for university entrance.

The exams at 16 were originally a school leaving examination because many students have the right to leave school at the age of 15, then 16. But they are becoming a bit irrelevant now because de facto, education is extending to 18, even if kids get into work, they have a right to get some kind of work-based training.

So, there's a debate about whether these exams are useful because they're becoming an increasingly irrelevant punctuation mark in a 5 to 18 education system. 

[00:49:58] Anna Stokke: So, I mean, it's interesting because in some Canadian provinces, we really don't even have standardized, what I'd call standardized testing or very much of it. So, for instance, okay. BC, I don't think they have much at all anymore. There's not a lot in my province. I mean, for a university entrance it would be better if there was a more uniform exam that students took or just in general, I personally think it's a good idea because it checks how things are going, whether things need to improve and it's an accountability measure in some sense.

[00:50:35] Dylan Wiliam: Absolutely. And I think that the trap to avoid is what's happened in the United States, where they've gone down a kind of aptitude testing program with these, what used to be called the Scholastic Aptitude Test, which is now just the SAT. The problem with that is that I don't really want to measure aptitude, I want to measure math achievement. 

So I think that if we are going to go towards examinations, they have to be achievement-based examinations. Can students do the following kinds of things? Can they integrate 1 over 1 plus x squared or whatever, you know? That, to me, would be much more useful for the university to say, Yeah, this student is ready to go straight into our honours mathematics course, and this student probably doesn't actually have the background, and they will struggle if we take them. 

So that would seem to me to be much more useful to the universities, you know, in making these choices about which students to accept.

[00:51:26] Anna Stokke: You talk about teacher effectiveness and things we can do to improve teacher practice. And, I do want to ask you about one thing, because you've taught math education, you've directed a math education program, and I’m not sure what it was like in the UK, but here in Canada, it's actually quite common that a lot of primary school teachers actually struggle a lot with math.

And in Canada, primary school teachers teach all subjects. That's often the case up to even grade eight sometimes. And it has been suggested, some people will suggest, that it might be worth having math specialists teaching math in elementary schools. Do you have any thoughts on that?

[00:52:14] Dylan Wiliam: I used to believe that was true. I used to believe quite strongly that, say, for fifth and sixth grade, we should actually have math being taught by subject specialists. But I was really brought up short by a study from Houston where they actually discovered that the students who were taught math in 5th and 6th grade by their general teacher did better than the students who were taught by a specialist 5th and 6th grade teacher.

Now, it could be that those specialist teachers weren't that specialist. So I think that, again, these issues are very complex. Everybody decries the math capacity of elementary school teachers in most countries that I've visited. The interesting thing is, it's really hard to show a strong link between teachers math capability and the progress their kids make.

So probably the best work in this area is done by Deborah Ball and her colleagues at the University of Michigan, and they developed this assessment of mathematical knowledge for teaching. So it wasn't abstract content knowledge, it was do you know the subject at the level at which you're teaching it?

Do you know, can you explain why to divide fractions you invert and multiply? And what they found was that if you're taught by a teacher who scores one standard deviation higher than average on this test, you will make about two weeks more progress per year in math, which is not nothing. But given that the outstanding teacher, a teacher who is overall one standard deviation above the mean, produces 50% more progress, an extra six months of learning per year, it seems that either we haven't got a handle on what this capability is, or the effects aren't as strong as we thought they were. 

You know, given a choice, I'll take the teacher who's got the better math knowledge, but is it worthwhile improving their math knowledge? That is a question that is, as far as I'm concerned, still open. And we have one insight of this from an elementary school study that we did in England and it was just called, the report is available, it's called “Effective Teachers of Numeracy.” We looked to see what, which teachers in elementary schools were the most effective at teaching math. And we saw that the ones who were the most effective were the ones who saw math as connected.

So they didn't teach decimals and fractions and percentages as different topics, they taught them as being different ways of representing numbers that weren't integral. So, one of the things I would often ask kids to do is to is to put a half, 50% a half, 25%, and 0.36 on a number line. And, students are often surprised by that.

Because the idea, they've done number lines of fractions, they've done number lines of decimals, they've done number lines of percentages, but they haven't seen that all these are different ways of thinking about numbers that aren't integral. And so the teachers who were most effective were the ones who saw that kind of connection.

And in fact, we do know how to help them develop that kind of connectedness through professional development. So at the time in England, we had these what are called 20 day courses. So teachers could get day release, it's like, one day, a whole day release once every two weeks for a year and we worked on mathematical ideas with them, and that did seem to make a difference to how they taught math.

So it is, it is an example of changing teachers’ thinking about mathematics, that did seem to translate through into more effective practice in the classroom.

[00:55:54] Anna Stokke: And that makes sense. I mean, one thing I know for sure, you cannot teach what you don't know. We don't need studies to tell us that. If you don't know how to add fractions, you're not going to be able to teach someone how to add fractions. 

[00:56:05] Dylan Wiliam: Right.

[00:56:06] Anna Stokke: Beyond that, I do know that teachers need good resources. And I think that in a lot of cases that's not happening, like in some cases there aren't even any textbooks or any teacher's manuals and if math wasn't your strongest subject, to me, this isn't fair.

This isn't a fair place for teachers to be in, right? They do need really good resources.

[00:56:29] Dylan Wiliam: I agree with that, but I think I accept you can't teach what you don't know. The question for me is, how long ago did you first, did you first know it? So I think we have to accept that the teachers don't have the math knowledge we might wish, but good textbooks can provide that in a kind of just-in-time approach.

And I think what really distresses me is that even when textbooks are available, teachers are still downloading lesson resources from the internet. And the problem there is that they're just thinking about cool stuff for their students to do. And as I like to point out, a collection of resources, learning resources, is no more a curriculum than a pile of bricks is a house.

And the fact is that what we have is unsystematic development of capacity because we don't have clear learning sequences. And that's why I think textbooks are important. The problem in North America is that the textbooks are largely produced primarily from a commercial perspective. What we need to be looking at is what happens in Japan and Singapore, where there's a very strong input from government agencies about how to design really effective textbooks, so they tend to be thinner, they tend to be leaner, and just much better focused on what we know builds effective knowledge in our students. 

[00:57:51] Anna Stokke: So I'll ask one final question. So you've talked a lot about ensuring that teachers are constantly improving classroom practice. And we've talked about some of those things today. So if a district leader asked you what things can they do to ensure that that's happening, that teachers are constantly improving classroom practice, what would you advise them to do?

[00:58:16] Dylan Wiliam: It depends how much political room for maneuver they have. So, if it was a district that had a kind of step-and-lane compensation system, whereby teachers get an annual pay rise just for getting older, I would like to replace that by a requirement that a teacher demonstrates increases in skill or capacity.

So the idea, it's something for something. So the idea is, here's my proposal, and it may be too radical for many districts, every six months or so, a teacher would meet with a supervisor. The supervisor would say, “What are you planning on getting better at over the next six months?” The teacher would outline that.

The supervisor would say, “What's the evidence that that will make a difference?” So they'd have to engage with the research in some way. And then the supervisor would ask, “What evidence are you going to bring me in six months time to support your claim that you got better?” And so it could be videos of classroom practice, it could be test scores, it could be questionnaires with students.

But the idea is that the teacher chooses what to work on, provided they can say it's likely to make a difference, and they choose the evidence base that's going to support those claims. And then the supervisor says, “How can I help? What support do you need from me to do that?” And I got that idea from being a provost.

Because when you're a university provost, you end up line managing people who do work that you do not understand. So you say to the professor of organic chemistry, you know, “What's your plan for the year? How will I know you've been successful?” “Well, maybe six articles in refered journals and at least so many millions in grant income.”

​

“Okay. How can I help?” This idea that past three years of practice, and let's, let's support teachers closely in the first three years, but after three years, they should be ready to decide their own professional learning agenda, and we structure that, and we build a system whereby the teachers are responsible for making their own decisions about the ways that they get better, with a view to maximizing the benefit for their students.

[01:00:26] Anna Stokke: Very logical advice. I love it. Thank you so much for joining me today. I really enjoyed the conversation. There's going to be so many great things for teachers to walk away with from this. So I really appreciate you joining me.

[01:00:41] Dylan Wiliam: Thank you, it's been fun. 

[01:00:43] Anna Stokke: More in just a moment. As always, we've included a resource page for this episode that has links to articles and books mentioned in the episode, including a link to the BBC documentary on Dylan's work. I'll have another great episode coming out on April 12th.

If you enjoy this podcast, please consider showing your support by leaving a five-star review on Spotify or Apple Podcasts. Chalk and Talk is produced by me, Anna Stokke, transcript and resource page by Jazmin Boisclair, social media images by Nicole Maylem Gutierrez.

Subscribe on your favourite podcast app to get new episodes delivered as they become available. You can follow me on X for notifications or check out my website, annastokke.com, for more information. This podcast received funding through a University of Winnipeg Knowledge Mobilization and Community Impact grant funded through the Anthony Swaity Knowledge Impact Fund.