Voice-first recruiting: how we built AI-led screening interviews on Vapi

Voice-first recruiting: how we built AI-led screening interviews on Vapi

At Josh AI we run AI-led screening interviews over voice. Candidates get a phone call (or web call). An AI interviewer asks them rubric-aligned questions, scores their answers, and produces a structured report for the recruiter. This is the architecture deep-dive on how the system works, what's hard, and what we learned shipping it.

Why voice

We considered text. Text is easier to build. Text is cheaper to run. Text gives you fewer failure modes. We chose voice anyway because the product reality is that recruiters and candidates respond to voice the way they don't to chat. Voice surfaces hesitation, confidence, and communication style — signals that matter for hiring and that vanish in a text channel.

The cost is that voice is a much harder engineering problem. Almost every interesting failure mode we hit was downstream of the voice channel.

The architecture

There are four major components:

1. The call layer (Vapi)

Vapi handles the phone-call layer: PSTN connectivity, WebRTC for browser interviews, real-time transcription, and turn-taking. We chose Vapi because it gave us a managed call layer with hooks into our own LLM stack, rather than locking us into a single provider's model. We bring the brains; they bring the pipes.

The integration is straightforward in shape but fiddly in detail. Each interview is a Vapi session configured with our own LLM endpoint, our own prompt, and a structured set of tool calls the model can make (start question, end question, escalate to human, mark interview complete).

2. The interview workflow (LangGraph)

The interview logic itself is a LangGraph workflow. States include: greeting, rubric question, follow-up, transition, closing. Each state has its own prompt template, its own allowed tool calls, and its own exit conditions. The graph structure makes it possible to reason about interview flow as a state machine rather than a single sprawling prompt.

Why a state machine instead of one big system prompt? Because interviews have phases with different rules. The greeting phase needs to be warm and low-pressure. The technical-question phase needs to be precise about what counts as a complete answer. The closing phase needs to be brisk. Trying to encode all of that in a single system prompt produces a model that hedges everywhere and commits to nothing. State-machine prompts let each phase be opinionated.

3. Proctoring (MediaPipe + TF.js)

For web interviews, we run MediaPipe and TF.js client-side for lightweight video proctoring. We're not trying to be a high-security exam proctoring system — that's a different product. We're trying to flag obvious anomalies (no face present, multiple faces, off-camera glances during technical questions) so the recruiter has a signal during review.

Doing this client-side matters for two reasons:

Privacy. Video doesn't leave the candidate's machine. We only get derived signals.

Cost. Server-side video processing for thousands of interviews would be expensive. Pushing it to the client is essentially free.

The downside is that we trust the client's computation, which is fine for the threat model (flagging anomalies) but wouldn't be fine for higher-stakes proctoring (preventing them).

4. Scoring (transcript + rubric)

After the interview ends, we have a complete transcript. The scoring stage runs a separate LLM workflow that takes the transcript, the rubric, and the candidate's prior context, and produces per-criterion scores with justifications.

Scoring is decoupled from the interview itself. The interviewer agent doesn't know its own scoring rubric in detail — it just knows which question to ask and when to move on. The scorer evaluates the transcript independently. Decoupling these matters because it means the same transcript can be re-scored under a different rubric (which recruiters actually do — "score this interview again under the senior bar instead of the mid bar").

The hard problems

Turn-taking

This is the single hardest problem in voice AI. Knowing when the candidate has finished speaking, knowing when to interrupt politely, knowing how to handle long pauses without sounding broken — none of this is solved by the call layer alone. We tune it per-question-type. A "tell me about yourself" question gets generous silence handling. A short factual question gets a tighter cutoff.

Latency

Every second of latency between a candidate finishing speaking and the AI responding feels like an eternity. Our budget for follow-up generation is around 400ms. That's why our hybrid routing (covered in a separate post) leans local for this specific call: a smaller model that always responds in 300ms beats a bigger model that occasionally responds in 1500ms.

Language

Candidates in our market code-switch constantly between English, Hindi, Urdu, and regional languages. The interviewer needs to handle this gracefully without losing the rubric-aligned structure. We added a language-detection layer upstream of the interview workflow that informs the prompt templates which mixed-language conventions to expect.

Suspicion and fake-candidate scoring

This is the one that surprised us most. A meaningful fraction of inbound applications are bots, fake candidates, or coached candidates reading from a script. We added a scoring layer that flags transcripts with suspicion signals (response latency patterns inconsistent with live speech, lexical patterns suggesting reading, multiple sessions from suspicious IP ranges). It runs alongside the substantive scoring and surfaces a separate "candidate authenticity" score to the recruiter.

Configurable tone

Different cultures and different recruiters want different interviewer tone. We exposed a configurable tone control — formal, conversational, brisk, warm — that adjusts prompt templates without changing the underlying logic. This was small but recruiter-loved.

What I'd tell a team starting today

1. Bring your own model. Don't lock yourself into a voice platform's bundled LLM. The platform's job is the pipes; your job is the brains. Keep them separable. 2. Use a state machine for interview flow. Single big prompts can't enforce phase-appropriate behaviour. Graphs can. 3. Decouple scoring from interviewing. The scorer should be able to re-run independently. 4. Plan for turn-taking from day one. It is the single biggest UX lever and the hardest to retrofit. 5. Measure interview drop-off rate as your north star. It's the metric that captures "is this product actually working" better than any per-call metric.

Voice is a heavier engineering investment than text. For our product, it was worth it. For yours it might not be. The honest answer is: build the text version first, prove the loop, then layer voice on top once you have a working baseline. We did it in the other order and would not recommend it.