Ranked by overall performance: the mean of appropriate scaffolding and appropriate rigor under the evaluation-aware prompt, the score the latency and cost charts below plot. Best score per column in pink. Human tutors, scored the same way at the same decision points, get 0.458 (appropriate scaffolding), 0.182 (appropriate rigor), and 0.496 (avoids over-scaffolding). Human tutors should be considered a naturalistic reference, not a ceiling. Annotators deliberately flagged moments where tutoring could have gone better, so the dataset concentrates on missed opportunities rather than ideal practice.
We ran language models ("LM") as tutors over 520 teacher-identified key moments from real math tutoring sessions, split evenly between moments that called for scaffolding (making the problem more accessible) and moments that called for a push for rigor (asking the student to do harder thinking). Each score is the share of moments where the model did what the moment called for, without over-scaffolding. Reasoning lists the thinking/effort/reasoning parameters each model was called with. The plain prompt just tells the model to use what it knows about great tutoring to tutor well; the evaluation-aware prompt spells out the scaffolding-vs-rigor trade-off.
The TutorMoments pipeline, step by step. Use controls to walk through a replay.
Ask a good tutor for help and you'll likely get a question back: "what do you know about what the problem is asking?" The tutor isn't being unhelpful by asking a question back. Instead, they are diagnosing what type of support may be needed. Language models, though, are trained to be a "helpful assistant". TutorMoments-Preview assesses whether the LM's helpful assistant character traits interfere with the core skills of a tutor: diagnosing the right level of support vs push to provide at a given moment in order to maintain the productive struggle that learning research has long tied to stronger understanding.
Most tutoring benchmarks reward one behavior across the board without asking whether that was the right move given the student's behavior at the time. For example, "never give away the answer", or "always reward scaffolding". TutorMoments characterizes tutoring as judgment calls made in context. Expert teachers annotated real human-student transcripts to select key learning moments where tutors faced a choice between introducing scaffolds or pushing for rigor. For every key moment, several teachers annotated why this was a scaffolding / rigor situation, what actions the human tutor took, and the resulting effectiveness. The majority label of whether it is a scaffolding or push for rigor moment becomes the ground truth an LM tutor is scored against. The result is a behavioral profile of each model: when it faces different tutoring situations, what pedagogical choices does it make, and do they fit the moment?
Told only to "tutor well," models tend to over-help — giving too much support and rarely pushing students to do deeper thinking. Spelling out the trade-off in the prompt lifts every score, but models still differ widely in how reliably they make that shift.
Share of tutor actions across twelve pedagogical moves, for each model and for the human tutors in the same transcripts (dashed line). Prompting for the evaluation shifts models toward asking students to justify their answers — while humans spread their choices across more varied strategies, including stepping back and letting the student work.
The working paper's seven models show figures from about 100 moments per model and prompt; the runs since were classified with the same model, prompt and scheme over all 520 moments, so their intervals are narrower.
To measure how much models adapt their behavior to scaffolding versus rigor situations, we look at the KL divergence between the distributions of actions in the type of moments. A higher KL divergence means that the model is using different moves in scaffolding situations versus rigor situations. The dotted line is the human tutors at the same moments.
KL divergence is asymmetric, so each dot is the mean of the two directions, KL(S‖R) and KL(R‖S): how far a tutor's moves in scaffolding moments are from its moves in rigor moments, and the reverse. Every series is at full sample, about 260 moments per situation, so these values are not comparable to the July working paper's KL table, whose add-one smoothing over ~50 moments per situation pulls every value toward zero (human tutors 0.18 there). Paper models from the action-taxonomy release's classifications of their full replays; later models from their own runs.
In the time since the benchmark was introduced, new low latency-high performance models have emerged. A human has a latency of 0.5–1 second on a video call, and face-to-face speech latency is around 100–200 ms. We measure overall performance (mean of appropriate scaffolding and appropriate rigor, evaluation-aware prompt) against median time to first answer token (TTFAT). TTFAT is the time until the first token of the reply that the student actually sees. The time to first token (TTFT), which many latency trackers report, stops at the first token of any kind, including reasoning tokens.
Education technology is a highly price sensitive industry. We measure overall performance (the same score as above) against model cost by multiplying the model's list price against the actual tokens generated. The cost is uncached: every prompt token is priced at the full input rate and reasoning tokens count at the output rate. Provider prompt caching can cut the input share substantially, so read this as a ceiling.
The explorer shows ten moments selected from the benchmark. Each shows the session up to the cut point, then what the human tutor actually did next beside a model's replay of the same moment. The human tutor's actions are as the expert annotators labelled them; each model's are as the benchmark's scorer judged them, by the same rule as the leaderboard. The moments are all de-identified and any names used are random surrogates.
1Allen Institute for AI 2Insource Services 3University of Washington 4MIT 5Stanford 6Step Up Labs
This project has been made possible in part through support from the Gates Foundation and the Learning Commons.