This paper describes a framework to teach AI tutors, called the Apprentice Learner (AL). The idea is the following. Suppose you want to develop an automated tutor for some kind of math problem. That’s a lot of work, usually, because you don’t want the tutor to just give out the solution, but instead interact with the user. That’s complicated by the fact that there are multiple possible solutions, and multiple possible misconceptions that students can have, etc. You usually want a tutor that is model-tracing complete, which means: every correct way to solve the problem is captured by the tutor, and it can detect all errors. So this paper proposes an interaction design that allows people to more easily “train” model tracing-complete AI tutors.
There are two levels of indirection here: their Apprentice Learner helps a human tutor develop an AI tutor that will in turn tutor a student. So their user study, in the end, asks the question of whether human tutors are quicker at developing tutors using the AL. Their main insight is that these two levels are fundamentally different. You don’t just want a simulated learner that you manually teach, that will then teach a human, because that learner will take a lot of time to explore all solution paths and mistakes that a human learner could make. Instead, the AL provides you the possibility of pointing out all potential next steps all at once, mapping all possible mistakes all at once, among other interactions that make sense to teaching a tutor, but not to teaching a student.
Their solution has a lot of parts, from programming-by-demonstration to a behavior graph that maps possible actions at each intermediate solution state. Honestly, I didn’t try their software, and without doing so most of it was quite abstract. But I think I’d need to create a web interface for some tutoring task if I want to test something different from their “add two numbers” task. Anyway, good for them for making everything available - it’s hard to find actual software that you can play with in this Intelligent Tutoring Systems space.