Teaching a child in <1000 ms: the architecture behind a real-time tutor

The next generation of children will learn to read and do math with AI. Whether that actually works for a five-year-old comes down to the engineering: a child can't wait for a slow reply, can't read a chat interface, and can't unhear anything a model gets wrong. We wanted to share some of the things we've learned from building a real-time AI tutor. This article focuses on our approach to latency, and how it influenced our architectural decisions.

A 2-second pause in conversation feels different to a child than to a developer, or even to an adult on the phone speaking to an automated agent. A couple of seconds is enough for a child's attention to wander and for learning to stop.

Good teachers manage this without pausing to think. They acknowledge a child immediately, even when they hold the answer back to let the child work. Teaching is matching the right approach to the current moment, and most approaches aren't answers.

When we set out to build an AI tutor for children ages 4-9, we wanted to build a tutor that actually teaches and not just a chatbot that responds quickly. We knew the constraint underneath would be hard, and that it wasn't optional: sub-second response on every turn. Most agents trade off speed for quality through reasoning budgets. Our architecture has to ground the tutor in pedagogy and respond to the child in real-time.

A teacher is constantly deciding how to engage a student, whether to say something, draw on the whiteboard, play a game, or change topics entirely. The standard pattern for an agent today is a tool loop. The LLM outputs one or more tool calls, waits for them to execute, observes the results, and decides what to do next. So the straightforward way to build a teaching agent is to make a tool for each action a teacher could take.

But the tool loop has a latency problem. Frontier models take 2–3 seconds to produce their first token, then decode at around 30 tokens per second. Our actions average a few dozen tokens. Add round-trip latency and audio playback, and a standard loop means 3-4 seconds of downtime between each sentence or change on the screen.

In one of our earlier playtests we watched it happen in real time. A six-year-old boy waited for the agent to think, then asked:

Another child in the same round of playtests figured out she only needed to pay attention part of the time and could still keep up. Latency had taught her to tune the tutor out. That was also the moment she stopped learning.

The convenient fix would be a smaller, faster model. That's where a scope problem shows up. Teaching is a broad task. A tutor might pick between dozens of actions in a single lesson, and the hardest call is often to withhold the answer and give a hint, ask a smaller question, or let the child struggle just enough that the insight is theirs when it lands.

Smaller models struggled to follow instructions across that breadth. An early version of our agent that used one was responsive yet constantly giving the answer away. Every time it did, it took away the moment where the learning happens.

So we built a custom harness to balance instruction following, latency, and a flexible action space. The model streams multiple actions in a single response. An interpreter parses and executes each action while the model is still generating the next ones. The child only has to wait for the first action about 30 tokens in, not for the whole response to complete.

Separating generation from execution buys us two more things. We can change which actions are available depending on the situation. For instance, when a question is on screen the agent gets instructions and options for scaffolding rather than answering. And we can validate each action without a latency hit on the happy path. Only if the stream produces an invalid action do we interrupt and re-generate, otherwise execution never pauses.

None of this is free. Owning the loop means we've had to build our own observability and tracing instead of leaning on a framework. And we're swimming against the current: frontier models are heavily post-trained on the tool-use pattern. If future models get fast enough, our harness is designed to be replaced by the simpler loop.

Lesson: Agent frameworks are building toward background work, where the tradeoff between speed and thinking is easy. Real-time learning sits at the other extreme. Teaching at conversation speed means owning the loop ourselves.

A real teacher both reflects on what a student just did and anticipates what they'll do next. Teach the same lesson a hundred times and you see the patterns. But you also know this child, where they've been stalling, what excites them, what's likely to trip them up today. You start the lesson with a plan and adjust it on the fly.

We call the agent that interacts with the child the converser. Our early experiments showed that a smaller action space led to better instruction following, so we built a second agent, the planner, to review the conversation against the lesson's objectives and manage the converser's context.

The first version ran synchronously, which of course was too slow. Plans that expired after a fixed number of turns weren't reliable. Neither was having the converser ask for a new plan. What worked was an asynchronous planner that runs while the child is thinking or talking, the same way a teacher reflects and anticipates in the gaps of a conversation. Those gaps are where the judgment calls get made: challenge the child or let them succeed, stay on the concept or move on. A teacher makes them on intuition; a model has to reason its way there, and running async is what buys it the time.

Async also means two agents running at once, both reading and writing shared state without coordinating. So we store every turn, every tap, and every UI update as an immutable event on an append-only log. Either agent reads and appends without waiting on the other.

That trajectory format enables another kind of anticipation. Whenever the converser asks a closed-ended question (i.e. coming up with a fill in the blank question, playing I Spy, completing an equation etc.), the harness hypothesizes the child's likely answers and pre-generates a response to each one on its own branch, forked from the trajectory. When the child answers, we match it to a branch and play the response without waiting on a fresh model call.

The tradeoff is cost, and the occasional miscall. The planner runs on a more capable, more expensive model, and it runs on every turn. And a prediction is still a prediction. Sometimes a child who was ready to be pushed gets handed an easy win instead. It's harder to evaluate whether a converser error was a mistake or downstream of a faulty plan. We don't yet have a clean signal for when to trust the plan versus what's happening live in the moment.

Lesson: The child interacts with the app in real time while the agents run in discrete generations, so leverage the time the child thinks or talks. Let the planner reflect the past and anticipate the future while the converser handles the present, and the slow pedagogical reasoning happens concurrently with the real-time exchange. When the next move is predictable, generate it before the child even answers.

Most AI products build guardrails in serial with a model call or agent turn. A user won't notice when the token stream goes through a content filter and a developer is willing to wait for a CLI tool call to be auto-reviewed.

There's nowhere to hide in a real-time conversation with a five-year-old. Nor is there an undo: a child can't unhear what the tutor said. The safety system has to gate any action, on every turn.

Our safety classifier is an LLM that takes ~500-1000ms to run. Waiting to run the converser until that check completes adds a second of delay to every turn that we can't afford. Here’s another advantage of decoupling generation from execution in our harness.

The safety classifier blocks execution without blocking generation. As soon as the child finishes speaking, we dispatch both the classifier and a small model to generate the converser's first action in parallel. That model reacts quickly with an eager response that mirrors or acknowledges what the child said ("you like dinosaurs! me too"). Think of it as the verbal equivalent of nodding along before you speak.

While a rules-based check would be faster and cheaper, it wouldn't survive the ways a five-year-old actually talks. Every category we add to the safety policy adds tokens and requires re-tuning a non-deterministic classifier. Sometimes a transcription error spooks the classifier and triggers a false positive. We review these cases and use them to improve how the agent understands the child.

By the time that eager action has generated, the classifier has usually returned safe. That check unblocks the converser to generate while the eager action executes. The child hears one continuous turn despite the multiple model calls.

But the harder problem than latency is what to do when that reflexive action is the wrong choice. Mirroring is great for everyday conversation with a child. Other times it's the opposite of what pedagogy suggests.

Take a child who mentions, mid-lesson, that a classmate called them a bad name. The same reflex that turns "I like dinosaurs" into "you like dinosaurs! me too" would echo the mean name back to the child.

So whenever the safety classifier flags the child's turn, we throw out the eager action. The converser is handled different guidance for this turn: don't repeat the name, acknowledge it must have felt bad, and suggest speaking to a grown-up.

Note: Our safety systems are governed by policies developed with child-development experts. How our safety system works in detail will be a separate article in and of itself.

Lesson: Gate execution on the safety check rather than generation to prevent a latency hit. Replace the reflexive response with guidance tailored to the child's situation whenever the check fails.

You can't build an AI tutor for children by picking the right model, prompting it, and calling it a day. Building an AI tutor for children requires much more. It's about engineering a real-time system that gives enough time to be both factually and pedagogically right — the time to withhold an answer when the answer diminishes learning, the time to choose the next action before the child finishes their thought, the time to second-guess a quick reflex before the child hears it.

These pieces seem small in isolation, but the magic isn’t until you have all these pieces work together that you a tutor that thinks ahead, recovers gracefully, and feels like it's with the child, and not catching up to them.

If these problems sound interesting to you, or if you want to learn more about what it actually takes to build a real-time AI tutor for kids, we'd love to talk. We're hiring.