Context
At MathGPT I built a pipeline that turns one calculus problem into three student-facing artifacts: a two-page LaTeX case-study worksheet (student sheet plus answer key) compiled to PDF, a set of concept flashcards, and a step-by-step solution walkthrough the browser renders one step at a time. A student picks a textbook question or pastes their own problem, and a Next.js app runs the pipeline server-side and presents all three results. I worked on a three-intern team with weekly syncs with the project lead. I owned the pipeline end to end, and the flashcard and case-study prompts were written together with the team.
The core design constraint: we need accurate math every time, but an LLM's answer can't always be trusted. Everything downstream follows from that.
Architecture
Five tracked stages. A generator solves the problem with read-only corpus tools and maps it to textbook sections. A critic verifies it (next section). Three stages then fan out in parallel: the LaTeX case study compiled with Tectonic, the concept cards, and the practice deck. The JSON artifacts must pass a zod contract before they count, and a failed validation feeds the validator's errors into exactly one retry. The case study is gated on a real LaTeX compile instead, also with one retry, and Stage 1 is checked for the files it should have produced. The fan-out stages fail independently, so a bad deck can't take down a good worksheet.
Grounding comes from OpenStax Calculus Volume 1: 45 sections and 195 learning objectives, extracted by page rendering with visual verification because standard PDF text extraction silently drops inline math. When a problem falls outside the corpus, Stage 1 emits no primary section and the downstream stages refuse to run instead of inventing a citation. A nine-table MySQL schema caches generated content keyed by textbook section and holds the template banks. A uniqueness constraint on the cache table enforces one card per type per concept, so the rule holds even if application code forgets it. The walkthrough is cached the same way: the first run in a section generates it, and later runs replay the stored JSON, except for picked-template runs, which bypass the cache in both directions. When a client-supplied template bank holds a problem for the section, the run animates that problem and its pre-verified answers instead of inventing its own. A picked template also skips Stage 1 and the critic outright, three model calls down to zero, because a human already vouched for that problem. Every gate on generated content still runs, since the audit vouches for the problem and never for model output.
The runtime is TypeScript end to end: Next.js with React 19, the pipeline running server-side inside the app. Python shows up in four offline corpus build tools, roughly 300 lines between them, and never serves a request.
The critic
The verifier is the same model in a fresh context, and its independence is structural rather than instructed. The critic's re-solve fires before the generator has produced anything, concurrently with it, so the drafts it must not see do not yet exist. Only after solving does it get shown the drafts, with its own solution replayed as a prior turn. On a mismatch the run fails and ships nothing. An unreadable critic reply gets one retry and then fails the run, because treating an unknown verdict as a pass would let an uncertified answer through. The critic's edits come back as a patch restricted to an explicit field whitelist. A whole-document reply is still accepted as a fallback, checked for dropped keys rather than field by field, so the patch is the preferred shape rather than the only one.
Five hard problems
Making independence structural. The first design sent the critic one message containing both the problem and the drafts, with an instruction to solve before reading. That rule is impossible to obey when the drafts are already in context, and no prompt wording fixes it. The fix was to fire the re-solve as its own call before the generator's output exists. A second failure appeared when the critic was asked to update a 15KB mapping document. It would either re-emit the whole thing or abbreviate it and silently drop fields that later stages depend on. Hence the whitelist patch.
No raw LaTeX ever reaches a student. Model output flows to three render paths: MathJax-typeset prose, plain text, and labels baked into SVG where LaTeX can't render at all. The original detector missed 11 of 15 realistic samples. The shipped one strips correctly delimited math spans and then flags any backslash command or stray brace. A sibling rule in the same family, the one that keeps LaTeX out of figure labels where it cannot render at all, has already rejected real model output in a live run. A later pass added a gate for spelled-out Greek in concept-card prose, with the word list measured rather than chosen: infinity, delta, and epsilon are deliberately exempt, because gating them rejected four correct cards.
Plots that can't silently lie. Letting the model supply coordinates puts
its arithmetic on the critical path with nothing to catch a bad point, and
model-supplied SVG is unverifiable. Instead the model declares a plot as a
closed-vocabulary ASCII expression like 8*x - x^2. A hand-written tokenizer
and precedence-climbing parser (247 lines, no eval, no dependencies) compiles
it at validation time, and an expression outside the vocabulary fails the
stage. The renderer computes every plotted point itself.
Case studies that are consistently good. The bad outputs kept failing the same three ways: a reskin of the source problem with new names, a story that is pure decoration, or a case too hard for the audience. The fix is a prompt that reads like a spec. It generates the worksheet from the learning objective rather than from the surface of the source problem, and an anti-reskin check asks whether a student who saw both would call the new case a different application of the same idea. Difficulty has a budget: three connected stages, at most five tasks, a 15 to 20 minute target, and no stage that is one-step substitution. A strip test catches decorative context, because a scenario that changes nothing when you remove it gets redesigned. And every worksheet compiles against the same fixed preamble, specified in the prompt and compiled with Tectonic, so the format stays put.
Valid LaTeX of the wrong thing. Importing a linear algebra bank meant converting AsciiMath, so I wrote a parser for it and the importable count went from 38 to 212. The failures that mattered were not syntax errors. A browser review turned up four cases that compiled cleanly and rendered the wrong mathematics: DNE typeset as D times N times E, a bare percent sign silently eating the rest of an expression, a matrix inside parentheses dropping every term after it. MathJax reported zero errors before the fixes and zero after. No mechanical gate can see this class, so the check has to be a human reading rendered output.
What it costs
Every run writes a per-stage token log. Profiling one run showed roughly 74% of output tokens were model reasoning, estimated from kept-output bytes since the API doesn't report the split. Three changes followed: per-stage reasoning effort, the critic's patch format replacing full document re-emission, and a rolling prompt-cache breakpoint in the tool loop that ended quadratic context growth. Comparable runs got 39% cheaper and dropped from 14.8 minutes to 7.5, though that compares consecutive runs on different problems, not a controlled A/B. The deck path later moved to the Batch API at half price, which surfaced a defect of its own: the batch call omitted the per-stage effort setting and ran at the API default instead. On the same three templates that was $0.32 against $0.13 for the three-deck batch, about $0.107 a deck against $0.043.
Where it stands
The review side grew its own tooling. Every stored card and deck is stamped with the template, book, section, and learning objective it came from, so a reviewer can trace any artifact back to its source. A rejection is tagged as either a content or a display problem, and a display rejection sends the artifact back to the model with its stored JSON and the reviewer's note, where an unchanged reply is the signal that the renderer is at fault rather than the content. One classifier now covers LaTeX formatting across every generated surface, and a bulk exporter reconciles against an independent count and refuses to write a short export.
1,000+ automated tests across 73 files: unit, integration against live MySQL, and two end-to-end suites that drive the full pipeline against mocked LLM replies and a real LaTeX compile. The negative controls are on disk and must fail: corrupted critic replies, invalid card and deck payloads, duplicate-key and phantom-foreign-key rejections. It's deployed and used for internal review, with the student run path disabled on the hosted instance, and the generated content has been handed to MathGPT's engineering team for integration.
The repository is private client work, so there's no public link, but I'm happy to walk through the verification architecture in detail.