Models and inference contracts

Beta.2 adds Knowledge Consolidation and indexing resource budgets; see those guides for their complete YAML, API/library contracts and boundaries.

The first beta guide documents the current evidence/publication, operation, scoped-access and runtime contracts. It supplements the compatibility interfaces described below.

Model choices belong to each corpus's config. knowledge -c kb.yml models reports roles and build features without loading weights or printing credentials. --validate-config checks policies and structure. Model loading and provider requests exercise the actual inference backend.

For an existing index, follow the rollout and rollback steps in the model change guide.

Default local roles

Role Shipped default Runs where
Text/raw media embedding EmbeddingGemma 2 Q8_0 plus matching multimodal projector In-process llama.cpp, Metal on the tested M3 build
RAG generation Qwen2.5 1.5B Instruct Q4_K_M In-process
Summaries/labels/entities/relationships Optional generation enrichment; off by default Selected generation backend
Image description/OCR Not configured Optional HTTP vision endpoint
Audio transcription Not configured Optional whisper.cpp or OpenAI-compatible HTTP endpoint
Learned reranking Not configured Optional HTTP reranker

GGUF weights load as runtime files. Native image/audio support requires the model's matching projector and the multimodal feature. Tested embedding output is 768D; other model spaces have their dimensions registered dynamically. Incompatible GGUF architectures/pooling/projectors fail at runtime. Arbitrary chat GGUFs are not automatically valid embedding models.

All-HTTP roles

The complete example is configs/remote-models.yml. Provider services must implement the selected contracts; URLs are full operation URLs. Use HTTPS, api_key_env references and security.network.allow_remote_https: true for a provider, or explicitly allow loopback HTTP for a local service.

models:
  enrich: false
  embedding:
    shared_space: my-joint-encoder-v1
    http:
      url: https://models.example.com/text/embeddings
      model: joint-text-encoder
      revision: v1
      protocol: openai
      api_key_env: MODEL_API_KEY
  media_embedding:
    url: https://models.example.com/media/embeddings
    model: joint-media-encoder
    revision: v1
    format: base64
    api_key_env: MODEL_API_KEY
  chat:
    url: https://models.example.com/v1/chat/completions
    model: my-instruct-model
  vision:
    url: https://models.example.com/v1/chat/completions
    model: my-vision-model
  transcription:
    url: https://models.example.com/v1/audio/transcriptions
    protocol: openai
    model: my-transcription-model
    revision: v1
    language: en
    api_key_env: MODEL_API_KEY
  reranker:
    url: https://models.example.com/rerank
    model: my-cross-encoder
security:
  network: {allow_remote_https: true, allow_loopback_http: false}

These hostnames/model names are placeholders, not bundled provider accounts. You can configure different hosts/models for each role. Endpoints and secrets cannot be selected by source content or search requests. models.enrich explicitly controls extraction for either native or HTTP generation; when omitted, earlier behavior applies (HTTP chat enriches, native generation uses its enrich flag).

Text, chat, vision and reranking

Text embeddings support OpenAI-style {model,input:[text...]} with indexed data[].embedding, or Ollama /api/embed with embeddings. Native text prefixes are configured explicitly; use the correct document/query prefixes for the selected model. OpenAI-style responses are reordered by index and incomplete/duplicate indices are rejected. Zero/nonfinite/inconsistent vectors are rejected.

Chat/vision support OpenAI-compatible chat completions or Ollama chat. Vision is asked for JSON {description,ocr_text} and receives actual image bytes. Generation requests structured extraction/grounded answers, so the selected model/server must support that behavior. Citations are checked for known source IDs; this does not prove answer entailment. Reranking accepts {model,query,documents,top_n} and returns results: [{index,relevance_score}]; this is the app's explicit adapter contract, not a promise that every reranking API uses it.

Remote image/audio embeddings

models.media_embedding supports a single media item per request:

format input in {model,input}
base64 {mime_type: "image/png", data_base64: "..."} (audio uses its own MIME)
data_url "data:image/png;base64,..."
openai_content Content array containing image_url or input_audio

The server returns one vector as {embedding:[...]}, {embeddings:[[...]]}, or OpenAI-style {data:[{index:0,embedding:[...]}]}. These are adapter contracts, not a universal multimodal embeddings standard. A provider-specific adapter can implement the Models trait when its API differs. Remote ASR, vision captions and audio embeddings are separate capabilities.

Remote media vectors require models.embedding.shared_space and a text embedding backend. The identifier asserts that both encoders produce compatible vectors in one semantic space. Compatibility depends on the encoding, beyond vector dimensions. Give unrelated encoders separate corpus configs or expose them through separate federation sources. Select either a native projector or a remote media encoder. Remote raw representations use names such as remote_image, remote_audio and remote_video_frame. Local ones use native_*. BM25, enrichment and text-only RAG context use textual representations such as captions and transcripts. Federation guide

shared_space must be versioned when semantics change. Model identity/revision/prefixes also control processing/cache invalidation. An HTTP model's weights cannot be checksummed by the app: set a meaningful revision and pin the model server yourself. Native sha256: revisions verify weight bytes.

Audio transcription

models.transcription.protocol is whisper_cpp (default) or openai. Both send multipart file plus response_format=verbose_json; OpenAI-compatible mode requires model. Optional language and api_key_env are supported. Whisper's transcription[].offsets milliseconds and standard segments[].start/end seconds are parsed into millisecond locators. Endpoints that do not return verbose JSON need a custom adapter. Video segment timestamps are translated back to the original clip timeline.

There is no speech synthesis, speaker diarization or continuous streaming API. These are separate tasks from ingesting audio into knowledge.

Apple system models

On Apple silicon, select the optional Swift provider for on-device generation, sentence embeddings or image descriptions and OCR. Each selected role can be combined with other HTTP or native roles. Apple configuration and verification

Read Markdown →