Per-student RAG: building AI mentors that actually know the student

Per-student RAG: building AI mentors that actually know the student

At Smart Grade AI, every student gets their own AI mentor — a chatbot they can ask questions of, that knows their actual performance history, surfaces specific weakness patterns, and recommends targeted practice. Building this taught us that "RAG" is too loose a term to be useful. What matters is the structure of what you retrieve, the scope of what you embed, and the contract the model is held to when it generates.

The product

A student finishes a graded assessment. The grading engine produces structured JSON: per-question scores, per-criterion feedback, marked-up answer text, and rubric-aligned justifications. The AI mentor ingests that JSON into a student-scoped vector store. When the student later asks "where am I weakest in calculus?" the mentor retrieves grounded context from their actual graded assessments and produces specific, citable advice.

The product fails the moment the mentor produces generic, ChatGPT-flavoured advice. "Practice more problems" is not what we shipped this for. Specific weakness identification with citations to the student's own work is what we shipped this for.

The architecture

One vector store per student

Each student gets their own embedded namespace. Embeddings are scoped tightly: a student's own assessments, never anyone else's. This isn't just a privacy concern (though it is one). It's also a quality concern — a global vector store with millions of student answers would surface noisy near-neighbours from other students that distract from the student's own performance.

The per-student model also gives us a clean operational story for deletion and data portability. A student leaving the system means deleting one namespace.

What gets embedded

The naive thing is to embed the raw student answer text. We tried this and the retrievals were noisy.

What works better: embed structured artefacts derived from the graded assessment, not the raw answer. Specifically, we embed:

Per-question feedback summaries generated during grading. These describe what the student did, what they missed, and why the score is what it is.

Rubric-aligned criterion judgements. "Student demonstrates understanding of integration by parts but mishandles boundary conditions."

Concept tags extracted during grading. These give the retriever something coarse-grained to match against.

The reason this works better than embedding raw answers: the embedded representation is already at the semantic level the retriever wants to operate on. Retrieving "the student struggles with boundary conditions" is more useful than retrieving "the student wrote: solving from 0 to pi..."

Retrieval is structured, not just similarity-based

A student question like "where am I weakest in calculus?" doesn't just trigger a cosine search. It triggers a structured query: filter to calculus-tagged feedback, retrieve top-K by similarity, then aggregate scores across all retrieved items to identify pattern-level weaknesses.

The model isn't reasoning over individual retrieved chunks. It's reasoning over an aggregate view that's been computed from the retrieved chunks. That distinction matters. Most RAG systems retrieve and stuff into a prompt. We retrieve, compute, and only then prompt.

The generation contract

The mentor's system prompt holds it to a strict contract:

Every claim about the student's performance must be grounded in retrieved content.

If the retrieved content doesn't support the claim the student is asking about, the mentor says so directly. "I don't have enough graded work in geometry to give you a confident answer here" is a valid response.

Citations are required. The student should be able to ask "why do you say that?" and get back a pointer to the specific assessment that grounds the claim.

This contract is what separates a useful mentor from a generic chatbot. The contract is enforced by a combination of prompt engineering, structured output validation, and a verification pass that checks claims against retrieved content before the response goes to the student.

What surprised us

The volume per student is small

A student might have 20 graded assessments by mid-semester. That's a tiny vector store. We over-engineered for scale we didn't need and would have done better starting with simpler in-memory indices and migrating as the per-student volume grew.

Cold-start is real

A new student has zero embedded content. The mentor is useless until they've done at least a few assessments. We handle this with a clearly-flagged "I don't have much of your work to learn from yet" mode that asks the student general questions rather than pretending to know them. The product cost of the cold-start period is real and worth surfacing in the UI rather than hiding.

Teachers want the same thing

Once we had the per-student mentor working, teachers asked for the inverse — a per-class view that aggregates the same signals across all students in a class. We built it as the AI Teacher Assistant. The architecture is similar: embed graded feedback, retrieve with structured queries, compute aggregates, then generate with a strict grounding contract. Same primitives, different scope.

Embedding cost was lower than expected

We were nervous about per-student embedding cost at scale. In practice, the cost is small because the volume per student is small. The bigger cost is the LLM inference for the mentor itself, not the embedding pipeline.

What I'd tell a team building per-user RAG

1. Scope per-user from day one. It's much harder to retrofit isolation than to start with it. 2. Embed structured artefacts, not raw user content. The retriever wants semantic-level inputs. 3. Retrieve, compute, then prompt. Don't just stuff retrieved chunks into a context window. Process them first. 4. Enforce a grounding contract. "Every claim cites a source or the model says it doesn't know" is the single most important constraint for product quality. 5. Plan for cold-start. New users have no data. Decide how the product behaves before they generate any, and surface that in the UI. 6. Don't over-engineer for scale you don't have. Per-user RAG often has small per-user data volume. Start simple.

The deeper lesson is that "RAG" is a technique, not a product. The product is "an AI mentor that actually knows the student." Getting from one to the other is mostly product engineering — defining what gets embedded, how it gets retrieved, what the model is allowed to say — not retrieval engineering. The retrieval part is the easy bit.