Ep 42. Math Academy: Optimizing student learning
with Alex Smith and Justin Skycak

[00:30:06] Justin Skycak: Yeah, so one, one thing we should probably clarify about our perspective on, on, on rote memorization is that we are not saying that, that a student should, for instance, memorize times tables or trig values without knowing what multiplication means or what trig functions represent. It's like, get the meaning first.  

 

But at some point, you have to really hammer these times tables or these trig values, these trig identities, these derivative rules, you have to hammer them into your memory. It's not enough to understand what they are, it's also about executing them. It's like, it's not enough to understand how to do a backflip, you actually have to do the backflip. And to do that, you have to practice repetitively over and over, really hammering this into place.  

 

So, we are not suggesting that students should be made to memorize things that they like, memorize symbols that they don't even know what it means, like memorize a times table without understanding that multiplication is repeated addition. 

 

No, it's like half the student do multiplication problems using repeated edition first but then move on to, to really hammer this information into memory. So, it's quick, instant recall building that automaticity that's needed.  

​

[00:31:21] Anna Stokke: Yeah, as someone who's been involved in this sort of conceptual understanding versus procedure debate for many years, I really don't know anyone who actually thinks that students should memorize stuff like times tables without knowing what multiplication means, but I hear a lot of people accuse people of that. 

 

I actually have seeing people claim that understanding is enough. 

  

[00:31:47] Alex Smith: One kind of personal anecdote I've got is that I did some, um, some like times tables remediation lessons at my son's local school. I was given; it was a remediation group. So, you know, the students were typically struggling with their times tables and in the UK at the end of year four, every student has to do like a times tables assessment. 

 

Automaticity is what you need to actually be good at this to actually be successful. This, this, um, assessment is five seconds, any times table, but weighted towards the difficult ones, like the six, sevens and eights. And I was giving the students that were really kind of struggling with this, with their times tables. 

 

And what I just found quite kind of ironic about that whole debate is that even the students are really struggling. They all know what multiplication was. They all got it just writing down what the times tables were. They were just doing repeated additions like, right. They know what it is. It's like, now let's get to the hard business of actually ingraining this in their memory. 

 

So, it surprises me that, uh, for timetables, this comes up as much as it, as it does.  

 

[00:32:45] Justin Skycak: It's good to practice this, like, level of conceptual understanding before you move on to memorizing timetables. But if you stay too long in that area, this thing that's supposed to be scaffolding you up to memorize your timetables, it actually becomes a crutch because then you, you just want to rely on it the whole time. 

 

[00:33:11] Anna Stokke: So, Alex, I am sure you have heard of the concrete pictorial abstract approach, which is sometimes fairly popular in younger grades for teaching arithmetic. Now, as someone who has created a fourth-grade math course for Math Academy, what do you think about that?  

 

[00:33:28] Alex Smith: So, CPA, Concrete Pictorial Abstract, sometimes referred to as CRA, Concrete Representational Abstract, it kind of means the same thing. 

 

So, the idea is that it uses physical objects and visual aids to build a child's intuitive understanding of abstracts topics. Might be place value, might be fractions, uh, those, those kinds of things typically employed at the, when building arithmetic skills and knowledge. And the ultimate goal is to use the concrete and pictorial stages as scaffolding to move on to the abstract stage. 

 

Abstract typically means using things without visual pictures or physical objects. So, I'll give an example. So, let's suppose you are teaching students to add fractions, you might start with the concrete approach. So, to give students a sense of how to add fractions, you might, uh, you might start with a physical object, so it might be a model of a pizza, but not a real pizza, you know, a pizza that can be with slices that can be taken away and added back in. And you can get some sort of sense how you might add fractions using these kind of model pizzas.  

 

Then once you've mastered that, you might then move on to pictorial instead of a physical model, you might be given pictures that's loosely based on the model you see, like circles divided into equally spaced parts, and asked to use those to add fractions. And then finally, you might move on to the abstract. And the abstract basically means you are doing the same kind of task but using purely mathematical symbols. 

 

So, no objects or pictures. In general, I think this is kind of like a reasonable basis, I'd say it's a good leading order approximation for beginning to teach arithmetic and some mathematical ideas to young children. Just in my experience of this approach, particularly with the fourth grade and fifth grade courses that we have at Math Academy, there's a couple of caveats I just kind of wanted to point out that might be useful for some listeners. 

 

So, the first thing I want to say, and I think one of your other list, one of your other speakers mentioned this as well, is important to move kids along as soon as they are ready. Uh, you know, so once they have mastered the concrete, move them to pictorial. Once they have mastered pictorial, move them on to abstract. 

 

It's like mastery learning again, you know, once they have, once they have mastered the prerequisite, they need to move on to the post requisite as soon as they are ready. I think what can happen in, in classrooms is they, is that they are not moved on at the right time and they might become overly dependent on the concrete or the pictorial stage, reluctant to move on to the abstract stage, which is, and the abstract stage is the ultimate goal. Mathematics is about manipulating symbols, you know, and doing so efficiently, efficient strategies to solve problems. It's at that point where what was meant to be scaffolding then becomes a crutch.  

 

Secondly, there's a point where the concrete stage isn't really needed anymore. I mean, some say that, you know, by the time it gets to around sort of seven or eight years old, you don't really need to be manipulating physical objects. You can move straight on to like a pictorial representation, and then of course the abstract. So, at this point it really becomes a PA instead of CPA. It's just, it's just Pictorial Abstract, dropping the C in, in the vast majority of, of cases.  

 

Now in Math Academy, obviously we are an online platform, so we, we can't do the concrete tape, you know, there, there is no way of doing physical objects. We only do pictorial abstracts, and the lack of the concrete hasn't hindered our students whatsoever. Uh, in fact, the vast majority of students on our fourth-grade course have completed the course successfully, and that involves mastering every single topic on the curriculum. And they have managed to do this without the concrete approach. 

 

So, I think it's, I think it's important to be aware that the concrete stage isn't always needed. And I think knowing that can actually save quite a lot of time. So just, just to go back to the point where about moving kids along exactly when they are ready and not delaying it, I mean, I have a, I have a six year old son and, you know, he needs to be moved on to the abstract, uh, stage as quickly as possible, really, because if he had his way, he just, he, he happily count beads. He loves counting. It's like, yeah, that's great. But you know, you are, you are more than ready for like the standard algorithm for addition. Now, you know, there's no need to be using blocks or bars or, or, or even pictures at that, at that point.  

 

And I think thirdly, and I think this is something that many people don't necessarily realize, is that it's important to realize that sometimes the abstract should come first. And the classic example here is like dividing fractions by fractions. Now the abstract method is super easy. Every kid that's got the right prerequisites, got mastered the prerequisites, gets it and gets it instantly. Flip the second fraction and multiply. What if they have got the prerequisites? They find this super easy. 

 

If you follow the concrete pictorial abstract approach to the latter, you are going to get them using models to do this before they are ready for the abstract procedure. And this is actually, in this particular example, is actually much more difficult and forcing kids to master that first, before moving on to like the abstract phase, is not really the best, the best option, I don't think.  

 

 

And just the final point I would make about that is that you really want to limit the number of physical objects or pictorial models that you use. I think, I think that less is certainly more. I mean, certainly at Math Academy, uh, in our fourth and fifth grade curriculums, I think we have got like one or two kind of consistent models that we use for, for manipulating fractions. 

 

And we have got number lines, which is a pretty universal way all the way up to university level math you need number lines, and that's pretty much it. That's all they really needs in their arsenal to be able to sort of tackle these to put their abstract understanding on a kind of concrete footing. As it were, they don't need ten different manipulatives or 10-20 different pictures, which I think there's a, there's a, seems to be like an emphasis to expose students to as many of these kinds of manipulatives and models as, as possible. And I think that's potentially quite a big mistake is picked the two most used two or three most useful run with that, chances are it's going to get the students to where they need to be.  

 

[00:39:41] Anna Stokke: Hundred percent. I will tell you a funny story, I wrote an op ed piece for the Globe and Mail. So, that's a, that's a Canadian newspaper and it was about exactly what you are saying, because I had also noticed this with my kids in school, that they were spending far too much time with concrete and pictures and not moving to the abstract and you really need to spend most of the time on the abstract, on the abstract part, realistically, like that's, you need to be able to work with fractions fluently.  

 

Otherwise, you are not going to be able to do any other math later, like algebra, etc. And anyway, so I wrote this article, and you were talking about pizza, and so, when you write one of these, someone else picks the title, and so, the title they chose was, Put Down That Pizza Slice, Math Teachers. So, it reminded me of that when you were talking about pizza.  

 

[00:40:34] Alex Smith: Oh, a great coincidence. 

 

[00:40:46] Anna Stokke: I am curious, how does Math Academy implement space practice? Like, what do the length of the intervals between practice of a topic look like?  

 

[00:40:56] Justin Skycak: Yeah, so space practice, space review is, is one of the cognitive strategies that we have actually leaned into the most. So, I spent a long-time building a, a spaced repetition system into math academies learning system that that takes our knowledge graph into account. 

 

So, I guess I should back up and just say like, okay, so spaced repetition, spaced repetition has been known for, for over a century. Ebbinghaus was, was the scientist who first kind of noticed that like, hey, when you learn something, your memory decays. But if you review it after a while, it decays slower. 

 

And so, you can wait longer until you review it again, and your memory will extend more. The way you increase your retention of something is by retrieving a fuzzy memory. And so, when you retrieve a fuzzy memory, it gets stronger, and it decays slower the next time. So, you have to, you can wait longer until it gets fuzzy, that same level of fuzziness again.  

 

Anyway, so this is, this is the idea of space repetition. The idea of repetition intervals kind of, they sort of, they depend a lot about the, the context of what's being learned and how well you're doing on your recalls. But, but roughly speaking, it's, it's something like maybe you wait a, about a day, the first time you review it, then a couple of days and like a week, a couple of weeks, a month, it gets to a point where the intervals are expanding sort of exponentially, roughly doubling, I'd say, is a good rule of thumb.  

 

So, each time you successfully recall something, maybe you wait about twice as long before attempting to recall it again. There's platforms for doing spaced repetition on flashcards that are unrelated, and so these each have their own spaced repetition schedule based on when you introduce the card, when you started reviewing it, whether you got the repetition correct or not, you successfully recalled the card.  

 

But in math, things get really complicated because we've got this big knowledge graph of concepts, and we are often implicitly practicing subskills. So, for instance, say, just imagine you stick one step linear equations and two step linear equations into a spaced repetition system, and you are doing spaced repetitions on these two topics, and let's say a repetition is just solving some problems, uh, from each topic. If you just do this naively do each repetition schedule in parallel without interacting them at all, then what's going to happen is you are, you are being very inefficient because every time you solve a two-step linear equation, you are implicitly solving a one step linear equation. 

 

The whole point of it, the way you solve a two-step linear equation is you actually, you do something to the equation and then that turns it into a one step linear equation that you know how to solve. So, every single time you solve a two step, you are implicitly solving a one step. It's like, okay, learn one step first. But as soon as you learn 2 steps, now you don't need to worry about reviews on 1 step anymore. You should just like throw that spaced repetition card away and just focus on the 2 step linear equations card. Because every time you do that, that is essentially a repetition on this 1 step.  

 

So, you can kind of imagine how this gets really complicated when you have this whole web of 3000-ish math topics connected by like 10,000 relationships. It just gets really, really complicated. So yeah, we had to devote a lot of time into figuring out how, how this repetition system is going to work. The general idea, actually, Jason was, was the one who first pointed it out when a kid was, was getting some silly reviews on, uh, some, some topics that you would reasonably infer they already know how to do and that they are already practicing in their lessons. Uh, the thing that he, he called it was an encompassing.  

 

So, topic A encompasses topic B, if topic B is used as a subskill in topic A. So, we ended up building this into encompassings into our spaced repetition system so that every time a student does a repetition on a higher-level topic, uh, the repetition like flows down the knowledge graph through the encompassings to affect the lower-level subscale topics that are being exercised.  

 

And we also generalize this to the idea of fractional encompassings. Sometimes you are not using the subscale in full, but you're using, like, a part of it. Or you are, you are, maybe you are using it in full, but only in a part of the problems in this higher-level topic. So, I spent a lot of time just kind of building this algorithm and this, this system to track these space repetitions through the, the knowledge graph. And we call it fractional implicit repetition, which has a catchy acronym FIRE.  

 

[00:45:29] Anna Stokke: Okay. So, you are trying to be as efficient as possible with the space practice.  

 

[00:45:34] Justin Skycak: So, yeah, efficiency is the name of the game with Math Academy. Every single task that we choose for a student to do it is the result of an optimization problem that's going behind the scenes. 

 

Optimization problem is how do you get the student making the most progress in the knowledge graph knocking out as much review implicitly as possible? So, we are trying to minimize the amount of work that the student has to do to get through the course, and there will still be plenty of review, but the idea is that we are choosing the reviews to knock out a lot of lower-level concept, a low-level skill, low level topic reviews as possible. 

 

And I should also mention that the space repetition process, in addition to being calibrated to this knowledge graph, it's also calibrated to how well the student does on their learning tasks. If you blow a task out of the water, you get everything correct, every question perfect. That's going to count for more repetition credit than if you only do like decently on it. Like maybe you get most of the questions right, but, but you still you miss some. And if you get a lesson halted, like, uh, that's not going to count as a repetition that you need to actually do that again, same for a review. And so, this is also tailored not just to the levels of student performance, but also to the levels of intrinsic difficulty in topics. 

 

Some topics are just more difficult than others. That's just how math goes, right? And so, we factor that into our spaced repetition algorithms, too. When a topic has a higher level of intrinsic difficulty, the repetitions are more frequent, so students will get more practice on it. Every topic has its own sort of spaced repetition schedule for the average student, and every student has their own spaced repetition schedule based on their performance on a topic, and it also takes into account performance on prerequisite topics and overall accuracy. 

 

So, it's just every single student for every single topic, there is a unique space repetition schedule that, that, that encodes all the context of the situation, tries to optimize just, just the right amount of review. So, they, they don't forget it, but at the same time, trying to free up time for them to learn new material instead of spending all their time reviewing. 

 

[00:47:47] Anna Stokke: Okay. So how about the testing effect? That's a really well studied effect that is very effective for learning. So, how do you leverage the testing effect in Math Academy?  

 

[00:47:59] Justin Skycak: Let me just start by explaining what that is for. for listeners. So, the best way to extend your memory duration of any information that you have consumed is to test yourself on it, meaning quizzing yourself, trying to, being made to pull it out of your brain without looking at a reference.  

 

And so, this stands in stark opposition to other strategies like re-reading or just transcribing notes again. There's, there's a lot of students I, I, I think who, who, who get into this mindset of like, oh, let me just rewrite my notes or like write what's in the textbook transcribe alongside, or just re-read over and over the same paragraph. 

 

And they think somehow it's, it's, it's getting into their brain, but it's, well, I guess it is technically getting into their brain, but it's not staying there because what happens is that they feel that there's this comfortable sense of fluency from having that information in their working memory, but what they don't realize, it's not enough for the information to be in your working memory, that information dissipates shortly after you stop rehearsing it, and you actually have to get it to encode into long term memory and the way you encode into long term memory is by retrieving from long term memory. 

 

It's kind of like weightlifting where your long-term memory is sort of like a muscle that you have to exercise and the act of lifting the weight is the act of retrieving the information. The testing effect, also known as retrieval practice, this is, this is also a centerpiece of, of our pedagogy, in that as soon as a student goes through a, a worked example, what's up next is, is practice problems, where they actually have to pull, uh, just pull, pull what they have learned out of their brain and then apply it to this new practice problem in a slightly different, uh, sort of context. 

 

As, as a student goes sees a topic multiple times throughout the course on Math Academy, we are gradually backing away even further from reference material and relying more on assisted retrieval. So, it starts out like when they are solving problems, they have the worked example to refer to if they need to. 

 

It's kind of like the spotter at a gym. The spotter can help you lift the weight if you are really getting crushed by it, but you shouldn't be relying on the spotter. But after a student goes through the lesson and moves on to a review, and the review, the worked example is not easy for them to reference. 

 

They can still like click back to the to the topic and then kind of scroll through to find like, oh, this problem is kind of similar to this work example, but it's, it is intentionally a little bit more annoying to go look up the reference material because we try to, we are trying to wean students off of that. And so, they are trying to get them just to more pull this information out of their brain when these review problems that they are solving. And then the third stage is that after a student has learned some, some material and done some, some review on it, they get quizzed on it. And so, quizzes on Math Academy happen about every 150 minutes of, of work. 

 

And we measure work in, in XP or experience points where one, one XP is equivalent to roughly one minute of work for a, for an average student at that level. It's coming every, um, maybe every, every two and a half hours. So, this, this is not like in a typical classroom where you might get a quiz every two weeks at most, or maybe every two months for a, for a student who's, who's using math academy along a, a, like a class schedule, say like 50 minutes a day on Math Academy, you are going to get a couple quizzes a week.  

 

And so, these quizzes, they are like 15-minute quizzes, but they are, they are packed with quite a few questions. You might, in lower grades, you can, you can get like maybe 20 questions on a 15-minute quiz. By this point, by the time, that a student enters the quiz, they should be strong enough on the material where they can just pull it out of the brain in a reasonably quick time frame and really start building this level of automaticity that they need. 

 

And higher-level courses like calculus, it might be more like, uh, maybe six or seven questions for a, for a 15-minute quiz. It kind of just, we calibrate amount of questions in a quiz to the amount of time that we expect the questions to take. But, but the idea is we are always focused on retrieval practice and we are gradually stripping away reference material and trying to have students retrieve under some time constraints to really build automaticity. 

 

[00:52:13] Anna Stokke: The students, using the program, need to be pretty good at working independently, am I right?  

​

[00:52:19] Justin Skycak: If you just sit a student in front of a computer and that's it and you don't check in on them, then yeah, you are kind of, you are depending on their being a responsible, studious, like, I will do what I am supposed to do, kind of student. Um, and going through worked examples properly, just not being adversarial in any way. You are depending on that. 

 

And as an adult, if you have a kid that's on Math Academy, or if you are a teacher and you have students on math academy, you can take steps to kind of corral more adversarial students into practicing the correct behaviors. So, something that we always recommend for, for younger students, especially for younger students who, who come on the system is like even, even a lot of times you get a younger student who's just excited about the material and they just want to solve problems, and they just skip the worked example. They don't read anything; they don't read the tutorial; they don't read the worked example. They are just like; I want to solve problems. And so, they'll go to the problem and then they're unsure how to solve it. And they are like, oh no, I am stuck. I don't know what to do. And then they get distracted.  

 

If they have a parent or a teacher or tutor, somebody sitting next to them kind of modeling the process, like, hey, let's read through this introduction, let's go through the work example. Like, let's read through it. Okay, now see if you can reproduce this worked example, like Alex said earlier, close the laptop, here let me write the problem down for you. Can you, can you solve it for me? And then you move them on to problems. Sometimes you have to model the process for them so they, so that they understand what it is that they are, they are supposed to be doing. 

 

And that's, that's one thing that we are, that's kind of on our road map is, is this kind of in-task coaching where we kind of detect based on a student's behavior during the, during the task if, if they are rushing or guessing or not reading or maybe they get a question wrong and they just click continue right away whereas they should really just look at the explanation to see what they got wrong so they can improve on the next attempt, there's a, there's a lot of areas where, where students can, can fall off the rails. That's kind of on our, on our road map to bake into the system. But for right now, we recommend that, yeah, a parent or a teacher needs to kind of model how to use the system properly for kids.  

 

And oftentimes, um, incentives may also be needed. I remember back when I was teaching using the system, there was one student who just like they just didn't want to do the work. One of the things that really worked in that case was setting up an incentive structure where, like, I would check in on the student pretty frequently during class, just try to keep them chugging through the material, talk about it a bit, but really worked was, was talking to their, their parent and having their, their parent, um, set up this kind of incentive structure at home was like, hey, if you do all your Math Academy work every week, we can go buy you a book on the weekend. 

 

She was really into reading like fantasy novels. And so just, just that incentive was enough to get her really excited about doing the work and, and, and doing well on the system. Even if you, you, you put a student on math academy and they are kind of adversarial sometimes that they don't want to do the work or maybe they are not exercising the right behavior and going through the system, you can kind of corral them in the right direction often by, by modeling the right behavior, holding them accountable for learning, setting up a good incentive structure.  

 

[00:55:37] Anna Stokke: Following up on that, adults do tend to be better at independent learning. And I understand that actually you have a lot of adult users. 

 

So maybe, Alex, can you talk a bit about that?  

 

[00:55:51] Alex Smith: The online program was originally designed for young students, but we always wanted to include like the university level courses too, but I mean, because at the end of the day, you know, if students are done with calculus at eighth grade, they need some other material to move on to. 

​

And we originally thought there wasn't really going to be much of a market for adults, you know, they are too busy with their jobs, their own children, to have them, uh, other, other things, you know, too many distractions to seriously learn maths. And then all of a sudden, you know, a huge explosion of interest around, you know, machine learning, AI, particularly all this large language models, stuff like ChatGPT really sparked a lot of people's interest in the technology behind it. 

 

So, the people that were curious about the technology, they began looking into it, and they began to realize that there's a lot of advanced math behind these models. They started to come to us to help get them machine learning ready, and so we actually created a math for machine learning course, especially for that purpose. And we have also got a machine learning one course coming out hopefully early in 2025.  

 

In most cases, many of these adult students, they hadn't done any formal math education for many years, five, ten, twenty years. And even if they had, it was only up to high school level, you know, and so some serious work was needed on their foundations. 

 

To give you an idea, some couldn't remember how to add fractions, many could only remember very basic algebra. We decided it was important to create a learning path from very foundational material, like middle school, high school material, all the way up to university level math. It was better suited to adult students, you know, so to guide them through all of that middle and high school material, to allow them to study the university level material. 

 

Uh, because we felt it just kind of wasn't right saying to adult students, well, you know, you can start off at fourth grade math. You know, it's like, yeah, I mean, although a lot of adults said, yeah, I don't mind, but there's a lot of stuff in, in that kind of, in the traditional learning path, middle school, high school, that isn't necessarily, isn't really that necessary for them to do. So, we felt it was important to kind of find like a streamlined path for them. That's what the mathematical foundation sequence was aimed to do. It's, it's three courses, uh, that basically start off at like fractions, geometry, the essential geometry, the essential trig to calculus, linear algebra, basic vectors and matrices and stuff, to get them onto the, the university map as quickly as possible. 

 

These foundation, uh, courses servers, they are prerequisite courses for our high-level content, which is like the Mathematics machine learning course. We have also got linear algebra, methods of proof, multivariable calculus, probability statistics. And then we have also got abstract algebra, differential equations, discrete math coming, coming to, and that now unsurprisingly, we actually have more adult students than we have young students, and many of them have made like outstanding progress.  

 

So just to give an example, um, we had a student who's, who's quite a vocal fan of math academy. So, he started, I think about six months ago, he was saying, I didn't know what completing the square was when I first started with math academy. So, completing the square is like an algebra one technique. And in five to six months, he's completed I think he placed into Foundations 2, so he's completed Foundations 2, Foundations 3. He's now successfully completed Mathematics for Machine Learning and is now like halfway through Multivariable Calculus. 

 

Now don't get me wrong, he's put in a lot of work to get to that point, but I think that's pretty extraordinary to have achieved that in say like six months.  

 

[00:59:26] Justin Skycak: One thing that I want to say is a common reaction, especially from adults using our system, is that they get to a point where they have this real emotional experience of seeing some math on a system that they had tried to learn previously and they, they are really intimidated by it because previously it did not go well at all and it just felt so out of reach and they were just kind of like, well, I guess I am just a dummy because I don't understand any of this math, but when they see it on our system, they, it's this, it's just a totally different feeling where they say, like, wait, that's, that's all it is, like it's, it just comes down to these prerequisites I know, that's, that's all that I was missing. I was like, I thought it was dumb, but it turns out I was just missing the prerequisites.  

 

And then they like look at like all the time that they spent, uh, in the past just kind of like banging their head on these problems when they really could have just been filling in their prerequisite knowledge and it, and, and it gets so much easier to learn. That's a very common, common reaction with our adults.  

 

[01:00:32] Alex Smith: That's why it's so important to go back and fix those missing prerequisites. 

 

If, if you are missing prerequisites and yet you are trying to build on top of that, it's almost like having a structure with kind of shaky foundations. It's like, it might hold up for a bit, but you try to build anything on top of that, and things are going to come sort of crashing down sort of quite quickly. 

 

I mean, we, we have had adult students that have said, you know, we need to go back and master trig and algebra and all that stuff. Why can't I just say, yes, you really do need to go back and, you know, again, not just familiarity, it's actual fluency with those topics, because you are going to need those in spades when you get to math and machine learning or multivariable calculus.  

 

So, we really can't emphasize how important it is to go back and fix those missing prerequisites. But if you do, that's what happens, you find yourself progressing through the material at ease. I mean, we often liken like a lesson on Math Academy to like a good session at the gym. You know, it does require effort, you do feel kind of like sweaty and tired and exhausted afterwards, but you are at the level where you can actually tackle that, and you have everything in your toolbox to do it. 

 

[01:01:49] Anna Stokke: And I want to just mention another group of students that I think Math Academy is quite popular with, and that's maybe advanced students, right? I think advanced students in general, at least where I live, are a neglected group of students. They often don't have a lot of extra programming in schools, and I don't really think that's fair. 

 

Like, I think that advanced students deserve challenging math. Just like struggling students deserve to get caught up. So, talking about that a bit. So, you mentioned that Math Academy can get eighth graders taking AP calculus BC. So, can you elaborate a bit on that?  

 

[01:02:35] Justin Skycak: I should say that when a student is advanced and sitting in a class and just cover the classes, just rehashing things that the student has already learned, then yeah, that's, that's not a good use of time for the student. And, and I think, yeah, it's, it's true. Every student needs to really just have instruction that is appropriate for, for them. A student doesn't need to be advanced or even at grade level to benefit from individualized instruction. 

 

I don't want to turn off people who don't think of themselves as advanced learners because the benefits of filling in your prerequisite knowledge and having like instruction adapted to your, your pace of learning and your needs it really it, it benefits all students even if a student is below grade level if they are being below grade level just comes down to the fact that they have missing knowledge then then we can fill that knowledge up for them really quick. The only type of student that we really don't work for is a student who is just dead set on not learning, like, no matter what you do, they refuse to do problems. And that's more of a behavioral thing than a knowledge thing.  

 

So back to this, this eighth graders taking AP Calc. BC story. So, we actually originally started as this nonprofit school program founded by Jason and Sandy Roberts. One of their kids, Colby, was on the fourth-grade math field day team, his parents were coaching that team. Their kid and his friends were all really excited about learning math, and so they did the standard fourth grade field day stuff. But the kids were so excited that they didn't want to just stop at fourth grade, something that they would often be, uh, asking Jason and Sandy is like, what's the highest level of math? 

 

So, Jason and Sandy would have to say, well, like it goes really, really high. But for your purposes. Let's just say it's calculus, because that's what seniors in high school take if they are on the honors track. And the next, uh, question was, of course, awesome. When, when do we get to learn it? Can we learn it now? Can we learn calculus tomorrow? Like, they are just so excited about it.  

 

So, Jason and Sandy were teaching the kids, a bunch of these same kids, a bunch of advanced math, even through fifth, the following year, fifth grade, got up through a bunch of high school math, got to the point where they could start learning calculus. And one thing led to another, and this kind of turned into an official school program that was not just a pullout class, but turned into a, like, every day kids would go to Math Academy class. There were other cohorts that came in in following years. What this ended up turning into, was, we would get, uh, students in sixth grade, they'd be solid on their arithmetic, and they might know what a variable is, but they don't really know how to solve equations or anything. They are, they are kind of like an early pre-algebra level.  

 

We would scaffold them up all the way, teach them all of high school math within the next two years, uh, within sixth grade and seventh grade. So, pre-algebra, algebra one, geometry, algebra two, pre-calculus. And in eighth grade they'd be ready to take calculus. So then, they would take calculus. They would take the AP Calculus BC exam. We got to the point where most of the students who took the AP Calc. BC exam in eighth grade passed the exam, and most who passed got a perfect five out of five on the exam. A couple things that I should say is, like, these are not national talent search students. 

 

How the kids were selected is just like they, they score at or above the 90th percentile on a middle school math placement exam, which is typically taken by all fifth graders in the district around February or March. And, and then they are invited to join the program. It's, it's a seventh-grade math skills test, so it provides a somewhat high skilling, but it's not really designed to identify math aptitude. 

 

This is this is also in the Pasadena Unified School District, where about two thirds of the student population qualifies for a federal free and reduced lunch program, and about 44 percent of all K through 12 students are, are educated in private schools compared to the California average of 11%. So, this is not a particularly talented group of students. It's not a biased group of like, oh, well, these are just the top students of the nation. Just think of like a standard, a standard school and kids that are in the standard honors class, um, they can be accelerated way, way, way higher than they are, are currently being.  

 

When Jason and Sandy were, were teaching, they, they were doing this all, all manually and, and achieving, achieving very good results. But these results got even better once we had students actually working on the math academy system. Jason got tired of, of, of the kids, saying like, I forgot to do my homework or, oh, I forgot a pencil or like all these excuses for not doing work. So, he just built a system where he could pick problems for those to do, then all I have to do is log in at home and do the problems online.  

 

It would automatically grade the problems and keep track of all the, the kids stats., keep track of the class accuracy and, and various topics. And so just over time, this kind of evolved into, into a system that, that did more and more of this teaching work. In summer of 2019, that's about when, when Jason pulled me in to make this, this system a fully automated platform that would actually select learning tasks for students. So, we built this, this, uh, automated task selection algorithm and, and just continued refining it. And by the time the pandemic hit in 2020, there was the big question of like, well, how, how are we going to maintain this level or level of efficiency from manual instruction. 

 

And the answer was like, well, we have this, this sort of halfway baked task selection algorithm. Let's just get it all in place over the summer and put the whole school program on it. And that's what we did. And that's how our AP Calc. BC scores really skyrocketed was from putting them on the system.  

​

[01:08:26] Anna Stokke: Wow, that's pretty amazing. 

 

But what are they doing after that? Is there, they still have grade 9, 10, 11, 12? 

 

[01:08:34] Justin Skycak: So, in 9th through 12th grade, actually what they would do is they'd learn a bunch of undergraduate math. We have PhD level math and instructors who teach the 9th through 12th graders, and they learn linear algebra, multivariable calc., probability statistics, real analysis, abstract algebra, algebra.  

 

They, they go through all this, all this content, and they are also, um, often working on some independent math projects, too. In terms of full outcomes for the students, it's still pretty early, uh, so that the first cohort is still in their junior year of college, and, and so they haven't really hit their, like, careers yet. 

 

We have been hearing a lot of really cool things from them. One kid is doing an accelerated master's degree in school. There's some other kids who got into MIT and Caltech. There's another kid who is currently a senior in high school, and he actually did an internship at Caltech his, uh, summer of his sophomore year and then worked there on a research project for his junior year, and now he, he actually let me know a couple of weeks ago that he, he got a paper published as a high schooler, a solo author. paper in a legit journal. And it's, it's interesting to just see you can see his author affiliations. He's like Pasadena High School and California Institute of Technology.  

 

[01:09:56] Anna Stokke: That's incredible. 

 

What's next for Math Academy? And Justin, maybe you can go first.  

​

[01:10:01] Justin Skycak: Well, I'll say a particular and a general, particular thing that I am really excited about is our upcoming machine learning course and machine learning course sequence and we are actually going to have a programming course too.  

 

In general, what's next? just we are coming for everything. Machine learning, proof-based math. We have a methods of proof course. We have a linear algebra course that's more concrete than proof based, but we are going to have a, a, uh, an abstract linear algebra course in the future on par with Axler's linear algebra. We don't have a real analysis course in the system yet, but we are going to, we are going to have competition math. We are going to have in-task coaching to guide students through the learning process. We want to become the ultimate math learning platform.  

 

And anything that we don't currently have under our umbrella, it doesn't mean that that's not on our road map. We are, we are coming to engulf math education, do it the way that it ought to be done to empower the next generation of students with, with the ability to learn as much as they can and make of that mathematical knowledge throughout the rest of their lives and their careers. 

 

[01:11:09] Alex Smith: So, I'll mention a few specific things. So, the first thing for me is to finish our undergraduate curriculum. So, I mentioned some of the courses we have got planned. We are shooting for the start of the next academic year to have all of those in place. So, differential equations, discrete math, abstract algebra, uh, Justin mentioned the machine learning course. 

 

We might even have a second machine learning course to try to get all of these courses kind of in place up until sort of quite recently, we have had a sort of like, not exactly a shoestring budget as that would be unfair, but you really had to be quite careful in the early stage with how we allocate resources. 

 

But now that we are getting a bit more popular, we have got a little bit more time, a little bit more, a few more resources to actually hire some more mathematicians to help finish off some of these courses. One of the things we are going to do is like automaticity training. Like for example, like with, with math facts, providing like kind of spaced repetition and other pedagogical techniques that we, that we typically employ to ensure that students are automatic with their, with their math facts. We'll start off with math facts and then we will, we will continue building on that. 

 

We currently have in the system like multi part problems. We are going to fuse those with coding exercises. Well, one thing we have, we have really, we have realized is that particularly the adult students, they really like the kind of like the, the multi part problems because multi part problems kind of draw in questions from various different topics and usually put it in like a contextual, like concrete setting, like a practical problem. 

 

Uh, and the adult students in particular seem to like that. So, we definitely want to get a lot more of those, um, in the system so they can actually see where their math can be, the math and learning can be applied. And another big thing, uh, is like math competition. So, we want, we want to build a curriculum, uh, around competition math, you know, uh, even building up all the way up to like, you know, things like AMC and Olympiad, eventually, and have like various leagues, regional leagues, national leagues, international leagues, where students can actually kind of go on to learn the math and then compete against each other. Quite a few people asking about complex analysis as well, which is my personal favourite area of math, so I'd love to see that in the system at some point. 

 

It's probably a little bit niche, so it's not exactly top of the list of priorities, but, uh, but I think really, really packing out some of that undergraduate material.  

 

[01:13:26] Anna Stokke: I really enjoyed this conversation and thanks so much for coming on and sharing your work with me and my listeners. It's been a really great conversation, and I really enjoyed talking to you both today. 

Thank you so much.  

 

[01:13:38] Justin Skycak: Thank you so much for having us. We are absolutely honored to be here with you. And yeah, it was a great conversation for us as well.  

 

[01:13:45] Alex Smith: Absolutely. Thank you, Anna, for inviting us on. It's been fantastic. Really appreciate it. 

 

[01:13:57] Anna Stokke: As always, we have included a resource page that has links to articles and books mentioned in the episode. If you enjoy this podcast, please consider showing your support by leaving a five-star review on Spotify or Apple Podcasts. Chalk and Talk is produced by me, Anna Stokke, transcript and resource page by Jasmine Boisclair and Deepika Tung. 

 

Subscribe on your favorite podcast app to get new episodes delivered as they become available. You can follow me on X, BlueSky or LinkedIn for notifications, or check out my website, www.AnnaStokke. com for more information. This podcast received funding through a University of Winnipeg Knowledge Mobilization and Community Impact Grant, funded through the Anthony Swaity Knowledge Impact Fund.