Back to Blog

Building a 96%-accurate Hebrew OCR pipeline: lexical post-processing as the secret weapon

Notes from Introva: how we got from 78% to 96% accuracy on Hebrew historical text OCR. The model wasn't the bottleneck — the post-processing layer was. A 23,000-word lexical database did most of the heavy lifting.

Khursheed Ahmed
  • OCR
  • Hebrew
  • NLP
  • Production
  • Lexical Processing
  • Introva

Building a 96%-accurate Hebrew OCR pipeline: lexical post-processing as the secret weapon

The work below was done at Introva on a Hebrew OCR pipeline for thousands of multi-column historical texts. We got from around 78% character-level accuracy with an out-of-the-box approach to 96% with a custom layout-detection stage and an aggressive lexical post-processor. I want to share what actually moved the needle, because the lessons generalise.

The starting point

Out of the box, Google Vision API on our corpus gave roughly 78% accuracy on a held-out sample. That sounds reasonable until you realise what 22% character errors look like in practice: unreadable paragraphs, broken citations, words that turned into nonsense, and a downstream pipeline that couldn't trust anything.

The texts themselves were the hard part:

  • Multi-column layouts with footnotes, marginal notes, and decorative dividers.
  • Pre-modern Hebrew orthography with vowel pointing in some places and not others.
  • Mixed scripts and abbreviations, many of which a generic OCR model has never seen.
  • Variable scan quality, ranging from clean scans to faded microfilm.

    • Where we did not get the gains

    We tried fine-tuning. We tried more aggressive image preprocessing. We tried prompt-tuning a vision LLM. Some of these helped a little. None of them came close to the lexical post-processing step.

    The lesson, which I now believe is general: for OCR on a domain with a finite vocabulary and rich linguistic structure, your post-processor is more important than your model.

    The architecture that worked

    The pipeline has four stages:

    1. Image preprocessing

    Standard stuff. Deskew, binarise, denoise, scale to a consistent DPI. Nothing exotic. Worth the effort but not where the accuracy gains come from.

    2. Layout detection

    We wrote custom layout-detection logic to identify columns, footnote regions, and marginal notes before sending text to OCR. This matters more than people realise. A multi-column page that gets OCR'd as a single column produces text that's interleaved and unusable. Our layout detector partitions the image first, runs OCR per region, then assembles the result.

    The detection itself is mostly heuristic: column detection from vertical whitespace projections, footnote detection from font-size heuristics, and a small classifier for the harder cases. We didn't need anything more sophisticated.

    3. Raw OCR via Google Vision

    The actual character recognition is Google Vision API. We tried alternatives — it was the best for our specific script profile after our preprocessing.

    4. Lexical post-processing (the secret weapon)

    This is where most of the accuracy gain came from. We built a post-processor backed by a 23,000-word lexical database covering the vocabulary of our corpus. The post-processor does:

  • Word-level correction. Every OCR'd token gets checked against the lexical database. If it's not in the database, we look for the closest valid word within a small edit distance and a phonetic-distance threshold.
  • Context-aware correction. When a word has multiple plausible corrections, we use surrounding tokens to disambiguate.
  • Abbreviation expansion. Pre-modern texts use abbreviations that OCR systems mangle. The post-processor recognises them by shape and expands them.
  • Citation normalisation. Many of these texts cite other texts. We integrated the Sefaria API to validate and normalise citation strings — turning OCR'd "Berakhot 32a" into a verified canonical reference.

    • The lexical database is the asset

    If you take one thing from this post: for any narrow OCR domain, build the lexical database first. It is the longest-lived asset in the project. It outlives model upgrades. It outlives reprocessing campaigns. It outlives provider changes.

    Ours has roughly 23,000 entries today and is the single biggest contributor to our accuracy. We seeded it from existing digital editions of the corpus, then expanded it through a feedback loop where unrecognised tokens that appeared with sufficient frequency in fresh OCR runs got reviewed and added.

    What zero-hallucination actually means in this context

    We describe the pipeline as zero-hallucination. That phrase needs a definition.

    What we mean is: the post-processor never invents text that isn't supported by the raw OCR signal. If a word can't be confidently corrected against the lexical database, we surface it as low-confidence rather than guessing. We never paper over uncertainty with a generated alternative.

    This is the opposite of what an LLM-based post-processor would do by default. We tried LLM post-processing and rejected it for exactly this reason: an LLM cleaning up OCR output is too willing to invent plausible-sounding text that isn't actually what was scanned. For a corpus where downstream use depends on textual fidelity, that's unacceptable. The lexical-database approach is more boring and more correct.

    The accuracy numbers

  • Raw Google Vision on our corpus: ~78% character accuracy
  • Plus image preprocessing: ~83%
  • Plus layout detection: ~88%
  • Plus lexical post-processing: ~96%

    • The marginal value of each stage went up as we added the next one. Lexical post-processing alone, applied to raw output, wouldn't have given us the same gain — it needs the layout step to give it well-structured input. The stages compose.

    What I'd tell a team building OCR for a narrow domain

    1. Spend your engineering time on the post-processor, not the model. Almost everyone gets this wrong. The model is a commodity. The post-processor is the asset. 2. Build the lexical database early. It's the longest-lived artefact in the project and the highest-leverage investment. 3. Get layout detection right before you tune anything else. Bad layout assembly poisons every downstream step. 4. Define hallucination explicitly and design against it. "The post-processor will never invent text" is a constraint, not a vibe. Bake it into the architecture. 5. Reserve LLMs for tasks where invention is acceptable. Summarisation, search, Q&A. Not OCR cleanup.

    The work is unglamorous. Building a 23,000-word lexical database is not the kind of thing that shows up on a portfolio in flashy form. But it's what took us from 78% to 96%, and 96% is what made the pipeline actually useful.

    Related Posts

    Why we chose DuckDB over pgvector for embedded semantic search at Josh AI

    An architecture decision write-up: we evaluated pgvector, FAISS, and DuckDB for our candidate-search vector index at Josh AI. DuckDB won. Here's how we got there, the tradeoffs we accepted, and what we'd change in v2.

    7 min read

    Hybrid LLM routing in production: when Ollama beats Groq (and when it doesn't)

    Notes from production: we route between Ollama (local) and Groq (cloud) using confidence-gated fallback at Josh AI. Here's the routing logic, the metrics that actually matter, and the failure modes that took us by surprise.

    8 min read

    Voice-first recruiting: how we built AI-led screening interviews on Vapi

    An architecture deep-dive on Josh AI's voice interview system. Vapi for the call layer, MediaPipe + TF.js for proctoring, transcript-driven rubric scoring, and the hard problems that show up only in production.

    9 min read
    Back to All Posts