Models and inference contracts
Beta.2 adds Knowledge Consolidation and indexing resource budgets; see those guides for their complete YAML, API/library contracts and boundaries.
The first beta guide documents the current evidence/publication, operation, scoped-access and runtime contracts. It supplements the compatibility interfaces described below.
Model choices belong to each corpus's config. knowledge -c kb.yml models reports roles and build features without loading weights or printing credentials. --validate-config checks policies and structure. Model loading and provider requests exercise the actual inference backend.
For an existing index, follow the rollout and rollback steps in the model change guide.
Default local roles
| Role | Shipped default | Runs where |
|---|---|---|
| Text/raw media embedding | EmbeddingGemma 2 Q8_0 plus matching multimodal projector | In-process llama.cpp, Metal on the tested M3 build |
| RAG generation | Qwen2.5 1.5B Instruct Q4_K_M | In-process |
| Summaries/labels/entities/relationships | Optional generation enrichment; off by default | Selected generation backend |
| Image description/OCR | Not configured | Optional HTTP vision endpoint |
| Audio transcription | Not configured | Optional whisper.cpp or OpenAI-compatible HTTP endpoint |
| Learned reranking | Not configured | Optional HTTP reranker |
GGUF weights load as runtime files. Native image/audio support requires the model's matching projector and the multimodal feature. Tested embedding output is 768D; other model spaces have their dimensions registered dynamically. Incompatible GGUF architectures/pooling/projectors fail at runtime. Arbitrary chat GGUFs are not automatically valid embedding models.
All-HTTP roles
The complete example is configs/remote-models.yml. Provider services must implement the selected contracts; URLs are full operation URLs. Use HTTPS, api_key_env references and security.network.allow_remote_https: true for a provider, or explicitly allow loopback HTTP for a local service.
models:
enrich: false
embedding:
shared_space: my-joint-encoder-v1
http:
url: https://models.example.com/text/embeddings
model: joint-text-encoder
revision: v1
protocol: openai
api_key_env: MODEL_API_KEY
media_embedding:
url: https://models.example.com/media/embeddings
model: joint-media-encoder
revision: v1
format: base64
api_key_env: MODEL_API_KEY
chat:
url: https://models.example.com/v1/chat/completions
model: my-instruct-model
vision:
url: https://models.example.com/v1/chat/completions
model: my-vision-model
transcription:
url: https://models.example.com/v1/audio/transcriptions
protocol: openai
model: my-transcription-model
revision: v1
language: en
api_key_env: MODEL_API_KEY
reranker:
url: https://models.example.com/rerank
model: my-cross-encoder
security:
network: {allow_remote_https: true, allow_loopback_http: false}
These hostnames/model names are placeholders, not bundled provider accounts. You can configure different hosts/models for each role. Endpoints and secrets cannot be selected by source content or search requests. models.enrich explicitly controls extraction for either native or HTTP generation; when omitted, earlier behavior applies (HTTP chat enriches, native generation uses its enrich flag).
Text, chat, vision and reranking
Text embeddings support OpenAI-style {model,input:[text...]} with indexed data[].embedding, or Ollama /api/embed with embeddings. Native text prefixes are configured explicitly; use the correct document/query prefixes for the selected model. OpenAI-style responses are reordered by index and incomplete/duplicate indices are rejected. Zero/nonfinite/inconsistent vectors are rejected.
Chat/vision support OpenAI-compatible chat completions or Ollama chat. Vision is asked for JSON {description,ocr_text} and receives actual image bytes. Generation requests structured extraction/grounded answers, so the selected model/server must support that behavior. Citations are checked for known source IDs; this does not prove answer entailment. Reranking accepts {model,query,documents,top_n} and returns results: [{index,relevance_score}]; this is the app's explicit adapter contract, not a promise that every reranking API uses it.
Remote image/audio embeddings
models.media_embedding supports a single media item per request:
format |
input in {model,input} |
|---|---|
base64 |
{mime_type: "image/png", data_base64: "..."} (audio uses its own MIME) |
data_url |
"data:image/png;base64,..." |
openai_content |
Content array containing image_url or input_audio |
The server returns one vector as {embedding:[...]}, {embeddings:[[...]]}, or OpenAI-style {data:[{index:0,embedding:[...]}]}. These are adapter contracts, not a universal multimodal embeddings standard. A provider-specific adapter can implement the Models trait when its API differs. Remote ASR, vision captions and audio embeddings are separate capabilities.
Remote media vectors require models.embedding.shared_space and a text embedding backend. The identifier asserts that both encoders produce compatible vectors in one semantic space. Compatibility depends on the encoding, beyond vector dimensions. Give unrelated encoders separate corpus configs or expose them through separate federation sources. Select either a native projector or a remote media encoder. Remote raw representations use names such as remote_image, remote_audio and remote_video_frame. Local ones use native_*. BM25, enrichment and text-only RAG context use textual representations such as captions and transcripts. Federation guide
shared_space must be versioned when semantics change. Model identity/revision/prefixes also control processing/cache invalidation. An HTTP model's weights cannot be checksummed by the app: set a meaningful revision and pin the model server yourself. Native sha256: revisions verify weight bytes.
Audio transcription
models.transcription.protocol is whisper_cpp (default) or openai. Both send multipart file plus response_format=verbose_json; OpenAI-compatible mode requires model. Optional language and api_key_env are supported. Whisper's transcription[].offsets milliseconds and standard segments[].start/end seconds are parsed into millisecond locators. Endpoints that do not return verbose JSON need a custom adapter. Video segment timestamps are translated back to the original clip timeline.
There is no speech synthesis, speaker diarization or continuous streaming API. These are separate tasks from ingesting audio into knowledge.
Apple system models
On Apple silicon, select the optional Swift provider for on-device generation, sentence embeddings or image descriptions and OCR. Each selected role can be combined with other HTTP or native roles. Apple configuration and verification