Hybrid LLM routing in production: when Ollama beats Groq (and when it doesn't)

Hybrid LLM routing in production: when Ollama beats Groq (and when it doesn't)

At Josh AI we route LLM calls between a local Ollama deployment and Groq's cloud API. This is not a "look how clever we are" story. It's a "we had to do this to keep the product reliable" story. Here's how the routing actually works and what the operational reality looks like.

Why we needed a fallback strategy

Groq is fast and cheap and we like it. It is also a single external dependency. When Groq has an incident — and like every cloud provider, it occasionally does — every screening interview that's mid-flight at that moment sees latency spike or requests fail. Recruiters notice, candidates notice, and the operational headache is real.

The naive answer is "fall back to OpenAI." But OpenAI is another network hop with its own SLO, its own pricing, and its own failure modes. You're trading one external dependency for another and hoping they don't fail in correlated ways. (They occasionally do.)

So we added a local fallback: Ollama running on our Azure VM, holding a smaller model with a much narrower job. The routing decision is per-request and considers the request type, the request's latency budget, and a runtime health signal from each provider.

The routing logic

In pseudo-code, it looks something like this:

`python def route(request: LLMRequest) -> Provider: # Health gate: skip any provider currently tripped candidates = [p for p in providers if not p.circuit_breaker.tripped]

# Job-specific routing if request.kind == "rerank_candidates": # Latency-critical, narrow task — prefer local return prefer(candidates, [ollama, groq]) if request.kind == "rubric_evaluation": # Quality-critical, wider task — prefer cloud return prefer(candidates, [groq, ollama_fallback_with_warning]) if request.kind == "live_interview_followup": # Hard real-time — prefer local, never wait return prefer(candidates, [ollama], require_under_ms=400)

# Default return prefer(candidates, [groq, ollama]) `

A few things are doing a lot of work here:

Circuit breakers per provider. A provider that's been failing recently is removed from the candidate set for a cooldown window. We use a simple windowed-error-rate breaker, not anything fancy.

Job-specific routing. Not every LLM call is equally tolerant of a smaller local model. Reranking candidates from a vector search? A 7B model does fine. Generating a rubric evaluation that a recruiter will read? You want the cloud-tier model.

Latency budget as a hard constraint. During a live voice interview, the follow-up question generation has a hard latency budget. We'd rather use a smaller model that responds in 300ms than a bigger model that responds in 1200ms. Quality matters, but latency violations are user-visible in a way model-quality degradation isn't.

When Ollama actually beats Groq

This is the part that surprised us.

For narrow, repeatable, structured-output tasks with tight latency budgets, a well-prompted local 7B model is sometimes a better product choice than a cloud frontier model. Specifically:

Reranking after vector search. The input is structured, the output is a small number of integers, the task is narrow. A local model with a tightly engineered prompt is fast, predictable, and never has a bad day because of someone else's incident.

Intent classification on incoming WhatsApp messages. "Is this candidate asking to opt out, asking a clarifying question, or sending us spam?" Three-way classification doesn't need a frontier model. Local-first is correct.

Generating follow-up interview questions during live voice flows. Latency budget dominates. A smaller model that always returns in 300ms beats a bigger model that occasionally returns in 1500ms.

When Groq still wins decisively

Open-ended rubric evaluation that gets shown to a human. Quality of phrasing matters. Subtle reasoning matters. Use the cloud-tier model.

Anything where the output is unbounded or weakly structured. Local models drift, repeat, and miss edge cases more than frontier models. Cloud is worth the cost.

First-time-user-facing outputs. First impressions matter. Eat the cloud cost on the first request a user sees from a new flow.

The metric that actually matters

When we first deployed this, I was tracking per-request latency and per-provider success rate. Those are necessary but not sufficient. The metric that ended up mattering most was end-to-end task completion rate — what fraction of recruitment campaigns finish without a human-visible LLM error. That metric forced us to think about correlated failures, retry budgets, and the cost of fallback degradation in a way the per-request metrics did not.

The failure mode that took us by surprise

Cold start.

Ollama has to load the model into memory on first request after a process restart. On our Azure VM that's a 4–6 second cold start. The first request after a deploy or a process recycle was always slow, sometimes timing out under load. The fix was boring: a warm-up health-check that fires after process startup and keeps the model resident. Nothing clever, just operational discipline.

The interesting bit is what this implied for our routing logic: we couldn't just route based on Ollama "being up." We had to route based on Ollama being up AND warm. That added a third state to the breaker (cold) with its own routing implications.

What I'd tell a team starting today

1. Don't build routing complexity speculatively. Start with one cloud provider. Add a local fallback only when you have a concrete reliability problem you've measured. 2. Per-job routing pays off. A single global "use cheaper model first" heuristic misses too much nuance. Routing by request kind is more code but more correct. 3. Cold starts are real. Plan for them. The cleverest routing logic in the world doesn't help if your fallback takes 5 seconds to warm up. 4. Measure end-to-end task completion, not per-request success. The former forces you to think about user-visible reliability. The latter is easy to game and not what you actually care about.

We've now been running this hybrid setup in production for several months. Groq still handles most of our traffic. Ollama catches us on the bad days. The end-to-end reliability of the product is meaningfully better than it was when we depended on Groq alone, and the operational cost has been manageable. That's the win.