Probabilistic Indexing

Most search engines for archives start from a flat text file: one transcription per page, written down by a handwriting model that picked its top guess and discarded everything else. That works on clean material. It is the wrong tool for 16th to 19th century manuscripts, where the top guess is wrong often enough that the discarded alternatives are where the real word lived.

archAIc is built on a different starting point. The HTR engine does not hand us a transcription. It hands us a lattice: a directed graph of competing word and character hypotheses, each labelled with a probability derived from the visual posterior over the ink strokes. Every path through the lattice is one possible reading of the line. The probabilities encode what the model actually believes, not what it was forced to pick. We index every plausible path, weighted by its probability. This is Probabilistic Indexing (PrIx), the foundation everything else in archAIc sits on.

What PrIx actually is

A PrIx index is not a list of words on a page. It is a list of (word, page, position, probability) rows, where the same ink position can carry several rows for the several words the engine considered. The probability comes from the HTR posterior, smoothed across the lattice and pruned at a threshold low enough to keep useful alternatives and high enough to drop noise.

A query at retrieval time is matched against the index probabilistically. Each candidate page is scored by the sum (or, for multi-word queries, the joint probability) of its matching spots, weighted by the PrIx confidence at each spot. The result is a ranked list of pages where the ranking reflects how much evidence the engine actually has for the query word being on the page, not whether its top guess happened to match.

A concrete example: a notarial book ingested for a Spanish municipal archive contains the word Licencia on page 312, written in a hand where the second c is ambiguous. The 1-best transcription written by the HTR engine has Licenzia with confidence 0.62. PrIx keeps both readings. Licencia sits in the lattice at 0.37; Licenzia sits at 0.62; a few more variants share the rest of the mass. A user searching for Licencia gets page 312 back, ranked below pages where the same word transcribed cleanly at 0.9, but still in the results. A one-best system loses that page entirely. Multiply by an archive of three million pages and the effect is not subtle.

What this buys, in numbers

Across 23 trial projects on real heritage collections we measured Average Precision gains of +3.4 to +16.8 percentage points over the equivalent 1-best baseline. The gain is never negative. The largest gains come from the hardest material: noisy ink, unfamiliar hands, damaged paper. Cleanest collections see a gain of a few points. The worst collections see a gain of a sixth of an AP point, which moves them from “barely usable” to “actually findable.”

The reason is straightforward. PrIx exposes the second and third best readings of every word as fallbacks. When the first reading was wrong, one of the others is often right, and the page that would otherwise have been invisible to a search is now ranked just behind the easy hits.

Two indexes, one engine

PrIx feeds two physical indexes inside archAIc, each tuned to a different query shape.

The keyword index stores the probabilistic spots in LMDB, with an FST over the keyword vocabulary and roaring bitmaps over page sets. A single-word lookup returns in well under a millisecond on typical hardware. Multi-word queries combine the per-word bitmaps with confidence-weighted scoring. Sub-millisecond keyword latency is what lets a curator type and see live results, and what lets an MCP agent issue dozens of probing queries per second without budget anxiety.

The semantic index stores per-passage embeddings in Qdrant, both dense (a sentence transformer fine-tuned for archival vocabulary) and sparse (a TF-IDF style vector for rare-word recall). The interesting part is how the embeddings are computed: each line is embedded as a Gaussian distribution in vector space, derived from the probability-weighted set of paths through that line’s word graph. A line with one dominant reading is a point in vector space. A line with two equally likely readings is a small cloud. A line where the engine is genuinely unsure is a larger cloud. The page-level embedding aggregates the line Gaussians and carries the same confidence shape into scoring at query time.

Both indexes are built once at ingest. Updates after a curator fixes a word are sub-second.

Soft document frequency

archAIc’s keyword scoring uses a soft document-frequency formulation: a word’s IDF weight is computed from the probability-weighted number of pages where the word appears, not from a binary “is the word on the page” count. The effect is that words like real, which are common in absolute terms but only certain on a small subset of their occurrences, get the IDF they deserve given the actual confidence of the index. Rare-but-confident words score higher; common-but-fuzzy words score lower. The user does not see the formula, only the result: better rankings on multi-word queries that mix common and rare terms.

Confidence propagates

The probability on a single PrIx spot does not stay at the spot. It is the starting point of a chain that runs up through every layer of archAIc.

• Word confidence aggregates into line confidence when a transcription path is scored.
• Line confidence aggregates into passage confidence for semantic scoring, and into the Gaussian variance of the passage embedding for retrieval ranking.
• Passage confidence aggregates into DLU confidence, the badge an archivist sees on a Document Logical Unit’s review screen.
• DLU confidence enters the score of every extracted entity and event, because each extraction carries the line confidence of the line it was anchored to.
• A search result is finally ranked by a blend of keyword score, semantic score, and the confidence on the path of evidence behind it.

Nothing in the chain is invented or recalibrated. It is the same probability the HTR engine assigned to the ink, propagated honestly all the way up to the answer the user reads. When archAIc shows a confidence badge next to a fact, the number is the product of the visual evidence, not a vibe.

What it changes downstream

Once PrIx is in place, everything else in the system gets easier. The semantic index is computed against probability-weighted line embeddings, which is what makes the meaning layer survive noisy transcriptions. The DLU layer reads passages through the same lattice, so a boundary detector that uses passage embeddings is also using PrIx under the hood. The entity-extraction LLM is shown the multi-hypothesis text for ambiguous words, so it can recover names the 1-best transcription dropped. Confidence badges, anchor scores, drawer highlights and review queues are all reading off the same underlying probability.

archAIc is the layer that turns that probability into product surface. PrIx is the layer that produces it in the first place.