Deploy a collection

This walkthrough takes you from a folder of PrIx transcription output to a running archAIc node serving that collection. It assumes Docker is installed and you have the archAIc compose project on the host.

archAIc ships as a set of Docker images. You place your corpus on disk, set a handful of variables in .env, and bring the stack up with docker compose — the lexical and semantic indexes build themselves on first start. There are no build commands to run by hand.

1. Lay out the corpus

archAIc reads a single corpus root — the directory CORPORA_PATH points at. Inside it, PrIx output and page images sit under two parallel trees, one folder per collection, one per libro:

corpora/                                # the directory CORPORA_PATH points at
├── config.ini
├── INDEX/                              # PrIx output — the transcription
│   └── {collection}/{libro}/*.txt      #   one or more .txt files per libro
├── optimized_jpg/                      # page images
│   └── {collection}/{libro}/NNNN.jpg   #   one JPEG per page
├── staging/                            # optional — TSDO page JSON for GT editing
├── corpora.db                          # SQLite — created on first run
└── gd-index/                           # LMDB lexical index — the engine writes it

You provide INDEX/ and optimized_jpg/ (and, optionally, staging/). The engine creates corpora.db and gd-index/ itself on first start. Drop every collection under INDEX/ and optimized_jpg/ — one folder per collection. The engine discovers everything beneath those two roots; collections and libros are never registered by hand.

What each file is

• INDEX/.../*.txt — the PrIx transcription. Each line is one word hypothesis with eleven space-separated fields:

  <word> <log-prob> <page> <line> <x> <y> <w> <h> <line-x> <line-y> <line-w>

  Multiple hypotheses per region are expected — that probabilistic spread is what archAIc indexes. The engine reads these directly; no preprocessing.
• optimized_jpg/.../NNNN.jpg — one JPEG per page, named by page number.
• staging/ — optional. TSDO page JSON, needed only if curators will edit ground-truth transcriptions in the UI.

Tip

Before deploying, check that each .txt is non-empty and every line has 11 fields. A truncated .txt from an interrupted PrIx run is the most common ingest failure.

2. Configure .env

Copy the template and edit it:

cp .env.example .env
${EDITOR:-nano} .env

The variables to set for a fresh collection:

Variable	Set it to
APP_NAME	A short slug for the archive — drives the public URL path and /var/www/<APP_NAME>.
CORPORA_PATH	Absolute host path to the corpora/ directory from step 1.
KWS_BACKEND_URL	http://rust-engine:3030 — the default Rust engine. (Use http://cpp-engine:7010 only for the legacy cpp profile.)
GD_BUILD_INDEX	1 on the first start, and after PrIx data changes; 0 for every start after that.
OLLAMA_URL, OLLAMA_MODEL	Optional — only if you want the LLM Q&A surface. See LLM setup.

If you build the images yourself rather than using delivered ones, .env also needs a read-scoped GITHUB_TOKEN for the private dependency repos. The full variable reference is in .env.example and the environment-variable section of the overview.

3. Start the stack with a profile

docker-compose.yml exposes named profiles, so you start only the services you need:

Profile	Services	Use it for
rust	PHP UI + Rust engine + Qdrant + MCP server	The default for new deployments.
cpp	PHP UI + C++ engine	Legacy, search-only — no semantic, DLU, sync, or GDPR.
both	the rust services plus the C++ engine	A/B comparison of the two engines.
observability	Grafana + Prometheus + Tempo + Loki + Promtail	Metrics, logs, traces — run alongside rust.

Bring up the default profile:

docker compose --profile rust up -d
docker compose --profile rust ps        # service status
docker compose logs -f rust-engine      # follow the first-run build

To add the observability stack later, start it alongside the running engine:

docker compose --profile observability up -d

4. The first-run build

With GD_BUILD_INDEX=1, the engine runs a one-time pipeline on first start — you do not trigger it manually:

1. Ingest — reads the PrIx .txt files under INDEX/, discovering collections, libros, pages, and word hypotheses.
2. Lexical build — writes the LMDB index under gd-index/ (word→postings, FST vocabulary). Roughly 17 GB of PrIx input compresses to a ~3 GB index in about 10 minutes on four cores.
3. Semantic build — computes dense and sparse vectors per page and upserts them into Qdrant. This runs automatically once the lexical index is ready, provided Qdrant is reachable.

The engine does not serve queries until the build finishes — docker compose ps shows rust-engine healthy when it is ready. Expect:

Corpus	Pages	CPU (4 cores)	GPU (CUDA)
Small	≤ 10K	~5 min	~1 min
Medium	~100K	~30 min	~5 min
Large	≥ 300K	~2 hours	~15 min

Once the build completes, set GD_BUILD_INDEX=0 in .env so later restarts load the existing index read-only instead of rebuilding. Flip it back to 1 only when the PrIx output is regenerated or the engine schema changes.

5. Smoke-test

With the stack healthy, confirm each surface answers:

# Engine health
curl -fsS http://localhost:${RUST_PORT:-3030}/health

# Lexical (KWS) search
curl -fsS "http://localhost:${RUST_PORT:-3030}/v1/query?q=archive&limit=5" | head -c 500

# Index stats — a non-zero passage count confirms the semantic build landed
curl -fsS http://localhost:${RUST_PORT:-3030}/v1/index-stats

# PHP UI
curl -fsS http://localhost:${PHP_PORT:-6008}/${APP_NAME}/ -o /dev/null -w '%{http_code}\n'

If index-stats reports zero passages, Qdrant is unreachable or the semantic pass has not run — check docker compose logs -f rust-engine.

Next

• LLM setup — add Ollama for natural-language Q&A and curator autofill.
• Deployment overview — deployment models, the full compose service list, and the environment-variable reference.
• The Document Logical Unit layer — segmenting libros into deeds, letters, and sessions — is built after deployment from the curator UI; it is not part of the ingest pipeline.