# Models and inference contracts

Beta.2 adds [Knowledge Consolidation](https://tsok.org/knowledge/docs/consolidation/) and [indexing resource budgets](https://tsok.org/knowledge/docs/indexing-resources/); see those guides for their complete YAML, API/library contracts and boundaries.

The [first beta guide](https://tsok.org/knowledge/docs/retrieval/) documents the current evidence/publication, operation, scoped-access and runtime contracts. It supplements the compatibility interfaces described below.


Model choices belong to each corpus's config. `knowledge -c kb.yml models` reports roles and build features without loading weights or printing credentials. `--validate-config` checks policies and structure. Model loading and provider requests exercise the actual inference backend.

For an existing index, follow the rollout and rollback steps in the [model change guide](https://tsok.org/knowledge/docs/model-changes/).

## Default local roles

| Role | Shipped default | Runs where |
|---|---|---|
| Text/raw media embedding | EmbeddingGemma 2 Q8_0 plus matching multimodal projector | In-process llama.cpp, Metal on the tested M3 build |
| RAG generation | Qwen2.5 1.5B Instruct Q4_K_M | In-process |
| Summaries/labels/entities/relationships | Optional generation enrichment; off by default | Selected generation backend |
| Image description/OCR | Not configured | Optional HTTP vision endpoint |
| Audio transcription | Not configured | Optional whisper.cpp or OpenAI-compatible HTTP endpoint |
| Learned reranking | Not configured | Optional HTTP reranker |

GGUF weights load as runtime files. Native image/audio support requires the model's matching projector and the `multimodal` feature. Tested embedding output is 768D; other model spaces have their dimensions registered dynamically. Incompatible GGUF architectures/pooling/projectors fail at runtime. Arbitrary chat GGUFs are not automatically valid embedding models.

## All-HTTP roles

The complete example is [configs/remote-models.yml](https://github.com/tsok-org/source-code/blob/main/apps/knowledge/configs/remote-models.yml). Provider services must implement the selected contracts; URLs are full operation URLs. Use HTTPS, `api_key_env` references and `security.network.allow_remote_https: true` for a provider, or explicitly allow loopback HTTP for a local service.

```yaml
models:
  enrich: false
  embedding:
    shared_space: my-joint-encoder-v1
    http:
      url: https://models.example.com/text/embeddings
      model: joint-text-encoder
      revision: v1
      protocol: openai
      api_key_env: MODEL_API_KEY
  media_embedding:
    url: https://models.example.com/media/embeddings
    model: joint-media-encoder
    revision: v1
    format: base64
    api_key_env: MODEL_API_KEY
  chat:
    url: https://models.example.com/v1/chat/completions
    model: my-instruct-model
  vision:
    url: https://models.example.com/v1/chat/completions
    model: my-vision-model
  transcription:
    url: https://models.example.com/v1/audio/transcriptions
    protocol: openai
    model: my-transcription-model
    revision: v1
    language: en
    api_key_env: MODEL_API_KEY
  reranker:
    url: https://models.example.com/rerank
    model: my-cross-encoder
security:
  network: {allow_remote_https: true, allow_loopback_http: false}
```

These hostnames/model names are placeholders, not bundled provider accounts. You can configure different hosts/models for each role. Endpoints and secrets cannot be selected by source content or search requests. `models.enrich` explicitly controls extraction for either native or HTTP generation; when omitted, earlier behavior applies (HTTP chat enriches, native generation uses its `enrich` flag).

### Text, chat, vision and reranking

Text embeddings support OpenAI-style `{model,input:[text...]}` with indexed `data[].embedding`, or Ollama `/api/embed` with `embeddings`. Native text prefixes are configured explicitly; use the correct document/query prefixes for the selected model. OpenAI-style responses are reordered by index and incomplete/duplicate indices are rejected. Zero/nonfinite/inconsistent vectors are rejected.

Chat/vision support OpenAI-compatible chat completions or Ollama chat. Vision is asked for JSON `{description,ocr_text}` and receives actual image bytes. Generation requests structured extraction/grounded answers, so the selected model/server must support that behavior. Citations are checked for known source IDs; this does not prove answer entailment. Reranking accepts `{model,query,documents,top_n}` and returns `results: [{index,relevance_score}]`; this is the app's explicit adapter contract, not a promise that every reranking API uses it.

### Remote image/audio embeddings

`models.media_embedding` supports a single media item per request:

| `format` | `input` in `{model,input}` |
|---|---|
| `base64` | `{mime_type: "image/png", data_base64: "..."}` (audio uses its own MIME) |
| `data_url` | `"data:image/png;base64,..."` |
| `openai_content` | Content array containing `image_url` or `input_audio` |

The server returns one vector as `{embedding:[...]}`, `{embeddings:[[...]]}`, or OpenAI-style `{data:[{index:0,embedding:[...]}]}`. These are adapter contracts, not a universal multimodal embeddings standard. A provider-specific adapter can implement [the Models trait](https://tsok.org/knowledge/docs/library/) when its API differs. Remote ASR, vision captions and audio embeddings are separate capabilities.

Remote media vectors require `models.embedding.shared_space` and a text embedding backend. The identifier asserts that both encoders produce compatible vectors in one semantic space. Compatibility depends on the encoding, beyond vector dimensions. Give unrelated encoders separate corpus configs or expose them through separate federation sources. Select either a native projector or a remote media encoder. Remote raw representations use names such as `remote_image`, `remote_audio` and `remote_video_frame`. Local ones use `native_*`. BM25, enrichment and text-only RAG context use textual representations such as captions and transcripts. [Federation guide](https://tsok.org/knowledge/docs/federation/)

`shared_space` must be versioned when semantics change. Model identity/revision/prefixes also control processing/cache invalidation. An HTTP model's weights cannot be checksummed by the app: set a meaningful revision and pin the model server yourself. Native `sha256:` revisions verify weight bytes.

### Audio transcription

`models.transcription.protocol` is `whisper_cpp` (default) or `openai`. Both send multipart `file` plus `response_format=verbose_json`; OpenAI-compatible mode requires `model`. Optional `language` and `api_key_env` are supported. Whisper's `transcription[].offsets` milliseconds and standard `segments[].start/end` seconds are parsed into millisecond locators. Endpoints that do not return verbose JSON need a custom adapter. Video segment timestamps are translated back to the original clip timeline.

There is no speech synthesis, speaker diarization or continuous streaming API. These are separate tasks from ingesting audio into knowledge.

## Apple system models

On Apple silicon, select the optional Swift provider for on-device generation, sentence embeddings or image descriptions and OCR. Each selected role can be combined with other HTTP or native roles. [Apple configuration and verification](https://tsok.org/knowledge/docs/apple-models/)
