Eight posts ago the claim was that the AI-education industry is building the wrong product — chatbots students ignore, while the thing that actually moves exam scores is an LLM grading written answers against a rubric, wrapped in spaced cumulative review. Now there's a running system to argue with instead of a claim to nod at. This is the capstone of the assessment-first series: what got built, how to run it, and where the bet breaks.
Run it in five minutes
doerkit is a full course — six statistics lessons from OpenStax OER, quizzes, cumulative review, a dosage dashboard:
git clone https://github.com/michaeltuszynski/doerkit && cd doerkit
npm install
export ANTHROPIC_API_KEY=sk-ant-...
npm run dev # http://localhost:8734
Pick a name, read a lesson, take its quiz. Write a real answer to a constructed-response question and watch it get graded against the rubric with feedback in about a second; write "the median because reasons" and watch it get partial credit with a specific note on what's missing. Fail the 90% review bar, get nudged to come back tomorrow instead of cramming. Open /dashboard and see your own dosage. The grader is regression-tested by the sibling repo, including against the prompt-injection answers a real student would try.
That's the whole thesis, executable. The LLM never chats, never does the student's work, never assigns a grade directly — it judges rubric criteria as booleans and code computes the rest.
What eight posts actually shipped
Two repositories, both MIT, both green in CI, both tagged v1.0:
rubric-bench — regression testing for any LLM judge. Golden sets, run scoring, drift diffs, an adversarial suite, tone metrics. The general-purpose one; useful well beyond education.
doerkit — the platform: grading engine, lessons, mixed-format quizzes, interleaved spaced review, telemetry.
The findings that surprised me, collected: a frontier model shrugged off first-generation prompt injections that a cheaper model fell for, so grader security lives in the model-prompt pair and moves when you swap either. Grader severity and grader warmth are separable knobs: you can be kind without inflating grades, which means a cold grader is a defect, not rigor. And the boring cumulative-review feature carried the biggest effect size in the source study, beating both the AI grader and the chatbot everyone demos.
What a real deployment would still need
The honest gap between "runs on my laptop" and "runs a gateway course," so nobody mistakes this for the second thing:
LMS integration: LTI 1.3, roster sync, gradebook. Unglamorous, mandatory, and deliberately absent here.
Auth and multi-tenancy: the demo trusts a self-typed name. A real one needs SSO, real accounts, and per-institution isolation.
A FERPA data agreement: the moment student-keyed telemetry leaves a laptop it's regulated education data, with all the procurement that implies.
Human-rater validation: post 3 regression-tests grading consistency, not agreement with instructors. A pilot needs an inter-rater study against real graded work.
An RCT: everything here rests on one observational pilot at one selective school. The design is a hypothesis with strong priors, not proof.
None of these are hard research problems. They're the difference between a portfolio and a product, and pretending otherwise is how edtech demos oversell.
Where the whole bet breaks
The strongest counterargument to this series is selection. The students who complete more lessons and pass all three reviews are the ones who were going to ace the final anyway; the Dartmouth data brackets the effect between 0.71 SD (over-adjusted) and 1.30 SD (selection-inflated) precisely because it can't fully separate the platform from the motivation. I believe the effect is real and meaningful — the cross-format contrast, where constructed-response dosage tracked scores and multiple-choice didn't within the same students, is hard to explain by motivation alone, but "real and meaningful" is a defensible position, not a settled one. Anyone who tells you AI tutoring has proven 1.3-SD gains is selling.
And there's a tension the series surfaced without resolving: disabling constructed response in the pilot raised completion rates, because writing answers is more work than clicking. The highest-efficacy format may carry an engagement tax. The whole bet is that the tax is worth paying and that better grader tone shrinks it, but that's the open question a real study exists to answer, not one this code settles.
The actual takeaway
If you build one thing from these eight posts, don't make it an education product. Make it the eval suite. Every team putting an LLM judge into production — grading, triage, moderation, ranking — has the exact problem post 3 solved and mostly doesn't know it yet: their judge's behavior is an untested production dependency that changes when the model updates. Golden sets, drift diffs, adversarial cases, tone guards. That pattern outlives statistics, outlives edtech, and outlives whatever model you're calling this quarter.
The chatbot got two years of the industry's attention. The quiz engine moved the exam scores. Both repos are public, both are yours to fork, and the code is the argument.
SOCIAL SHARE CARD GENERATOR