LLM setup

archAIc’s natural-language answer surface and the curator-facing autofill rely on a local LLM. The default backend is Ollama running qwen3:32b. Both pieces are optional: KWS and semantic search work without an LLM, and so do every IIIF export, every standards-shaped REST endpoint, and the MCP search verbs. The LLM is only required for the natural-language Q&A endpoints and for curator-assist autofill on the DLU editor.

This page covers installing Ollama, pulling the model, and pointing archAIc at it. The graduation path to vLLM (multi-tenant or throughput-bound deployments) is mentioned at the end.

What the LLM is used for

Surface	Needs LLM?
Lexical keyword search (KWS, BBox, flat, word cloud)	No
Semantic search (dense + sparse RRF)	No
IIIF, Linked Art, ISAD(G), EAD, METS, PREMIS exports	No
DLU CRUD, manual entity / event / relation editing	No
MCP search and read verbs (search, get_page, get_dlu, etc.)	No
Natural-language Q&A (ask_documents, natural_language_search)	Yes
Curator autofill on DLU editor	Yes
Auto-segmentation suggestion explanations	Yes

The privacy stance is simple: all data stays inside your network. archAIc uses Ollama on the customer’s own server, no model API call leaves the customer’s perimeter, and the engine is fully air-gappable on request. No telemetry is shipped to Anthropic, OpenAI, or tranSkriptorium.

Install Ollama on the customer’s server

The canonical install is the upstream script on a Linux server with GPU drivers already present. On Ubuntu 22.04 or later:

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama
ollama --version

For docker-compose deployments, run Ollama as a sidecar container instead:

# Append this to your docker-compose.yml — the canonical file does not include an Ollama service.
ollama:
  image: ollama/ollama:latest
  restart: unless-stopped
  ports:
    - "127.0.0.1:11434:11434"     # localhost only
  volumes:
    - ./ollama-data:/root/.ollama
  environment:
    OLLAMA_HOST: 127.0.0.1:11434
    OLLAMA_NUM_PARALLEL: 8        # concurrent generation slots
    OLLAMA_KEEP_ALIVE: 60m        # keep the model resident
  # For GPU passthrough on a Linux host with the Nvidia container toolkit:
  deploy:
    resources:
      reservations:
        devices:
          - capabilities: ["gpu"]

Bind to 127.0.0.1 only. Ollama has no native authentication, so anything reachable on a public interface is unauthenticated LLM access. If you need TLS (because archAIc and Ollama are on different hosts), front Ollama with nginx + a bearer-token check. The canonical nginx config and the rationale live in the engineering spec spec_llm_deployment.md.

Pull the model

ollama pull qwen3:32b
ollama list                       # confirm the model is registered
ollama run  qwen3:32b "hello"     # one-shot smoke test

qwen3:32b is the default because it gives the best autofill quality / footprint trade-off we have measured on heritage workloads. Other models work: qwen2.5:32b, llama3.3:70b, mistral-small:24b. Pick by VRAM budget and benchmark on your own corpus.

Footprint

Component	RAM / VRAM	Notes
qwen3:32b on GPU	≈ 20 GB VRAM	Recommended. Single-digit DLUs per minute on autofill.
qwen3:32b on CPU	≈ 22 GB RAM	Functional but slow: around 10 minutes per page on a typical server CPU. Acceptable for offline batches, painful for live curator use.
Smaller models (qwen3:14b, qwen2.5:14b)	≈ 10–12 GB	Faster, lower quality on entity extraction. Useful for prototyping.

The canonical archAIc box for an LLM-bearing deployment is a GB10-class mini-workstation (Nvidia DGX Spark, ASUS Ascent GX10, around 128 GB unified memory). One host, archAIc + Qdrant + Ollama co-resident, around 3–4k USD hardware, fully sovereign.

Point archAIc at Ollama

Two environment variables in your .env:

OLLAMA_URL=http://ollama:11434             # service name inside docker-compose,
                                           # or http://127.0.0.1:11434 on bare metal
OLLAMA_MODEL=qwen3:32b
OLLAMA_AUTH=                               # bearer token if Ollama is fronted by a proxy

If OLLAMA_URL is unset or unreachable, gd serve logs LLM disabled. Natural-language Q&A and autofill will return 503 and starts cleanly. Every non-LLM surface stays up. Bring Ollama online later and restart rust-engine to light up the LLM features.

Verify

# Ollama itself
curl -fsS http://127.0.0.1:11434/api/tags

# archAIc sees it
curl -fsS http://localhost:3030/health | grep llm

# End-to-end: ask a question
curl -fsS -X POST http://localhost:3030/v1/llm/answer \
    -H "Content-Type: application/json" \
    -d '{"question":"who appears in this volume"}'

The first byte should arrive within a few seconds even on slow hardware once streaming is enabled. If the call sits silent for more than 30 seconds and then returns a 504, your reverse proxy is buffering instead of streaming; see the streaming notes in the engineering spec.

When you outgrow Ollama

Trigger conditions for moving off single-host Ollama:

• More than one archAIc tenant sharing one GPU host.
• Throughput-bound on a single GB10 (queue depths growing on autofill batches).
• Need per-tenant LLM cost tracking, quotas, or rate limits.

The graduation path is vLLM behind LiteLLM: vLLM replaces Ollama as the inference server (PagedAttention plus continuous batching gives 2–30× the throughput on the same hardware, OpenAI-compatible API, multi-GPU support), and LiteLLM acts as the gateway with per-tenant keys, quotas, and spend tracking. The cfg_llm_profile.provider config value reserves a vllm slot, but only the OllamaClient ships as a concrete client today. Ollama-compatible endpoints work out of the box; a dedicated OpenAI / vLLM client is a planned follow-up (see spec_llm_deployment.md).

Cloud LLM APIs (OpenAI, Anthropic, Azure OpenAI, AWS Bedrock) are supported through cfg_llm_profile and configurable through the admin UI, but explicitly not recommended for archives bound by GDPR or sovereignty constraints. Use them only for demos, low-volume deployments, or when the customer has explicitly accepted the trade-off in writing.