# Configuring knowledge

Beta.2 adds [Knowledge Consolidation](https://tsok.org/knowledge/docs/consolidation/) and [indexing resource budgets](https://tsok.org/knowledge/docs/indexing-resources/); see those guides for their complete YAML, API/library contracts and boundaries.

The [first beta guide](https://tsok.org/knowledge/docs/retrieval/) documents the current evidence/publication, operation, scoped-access and runtime contracts. It supplements the compatibility interfaces described below.


Use [configs/reference.yml](https://tsok.org/knowledge/knowledge.yml) to review the complete grouped layout. Its active profile reads an Open Knowledge Format folder, indexes into an encrypted SQLite database, and serves lexical retrieval over TLS and MCP. The sample sets strict metadata, a path-specific chunk rule, and a CPU-only resource cap. Commented blocks show alternate sources, shared PostgreSQL storage, optional models, and integration settings. The root [knowledge.yml](https://github.com/tsok-org/source-code/blob/main/apps/knowledge/knowledge.yml) is the native model quickstart and includes a different set of explicit overrides. Both files reference credentials by path or environment variable.

| Section | What it controls |
|---|---|
| `source` | Authoritative OKF folder or SQL projection. |
| `sources` | Supplemental SQLite, manifest, and S3 inputs for a SQLite index. |
| `storage` | SQLite/SQLCipher path, or PostgreSQL index connection, bundle, quotas, and native vector projection. |
| `models` | Per-knowledge-base embedding, enrichment, generation, vision, transcription, and reranking roles. |
| `chunking` | Global chunk policy and ordered path overrides. |
| `retrieval` | Fusion, ranking, graph expansion, and query memory budgets. |
| `compression` | Optional experimental TurboQuant projection for compatibility search on SQLite. |
| `video`, `pdf`, `urls` | Decoder settings, media limits, and reference-fetching policy. |
| `indexing` | Watch and batching policy, leases, retry behavior, concurrency, and resource budgets. |
| `consolidation` | Evidence-backed background proposals and model selection. |
| `security` | Outbound model network policy. |
| `server` | TLS, client authentication, OpenAPI, MCP, and service budgets. |
| `federation` | Shares, grants, trusted peers, Mounts, and replica budgets. |

The sample activates MCP and SQLCipher-backed service state. Use a build with `mcp` and `sqlcipher`, and provision the referenced files before starting the listener. Config paths and commands below are relative to the repository root. [Local TLS and key setup](https://tsok.org/knowledge/docs/security/)

```sh
knowledge --config knowledge.yml --validate-config
knowledge --config knowledge.yml index
knowledge --config knowledge.yml serve
```

`validate-config` is also a subcommand. Validation checks types, unknown keys, policies and the source directory/index. It does not open SQLite, load model weights, resolve secret values or contact servers. Runtime startup performs those checks. Reload configuration by restarting the relevant process. Automatic source refresh requires an active watcher or reconciler. Run `check-server` separately to preflight the configured service credentials and TLS material.

## Choosing a file

1. `--config PATH` / `-c PATH` selects exactly that file.
2. Otherwise `KNOWLEDGE_CONFIG` selects exactly one file.
3. Otherwise the working directory's `knowledge.yml` or `knowledge.yaml` is used. Having both is an error; choose explicitly.
4. With neither YAML file, `knowledge.json` is the compatibility fallback.

Each process selects one file. YAML takes precedence over a legacy JSON file beside it. Relative paths resolve from the selected file, including model weights, databases, TLS material and secret references. Bare decoder command names such as `ffmpeg` are looked up on `PATH`. Unknown keys, duplicate YAML keys, anchors, aliases, merge keys and multiple YAML documents are rejected. Booleans are `true` and `false`; `yes`/`on` are strings. YAML parsing is bounded to 1 MiB and depth 32. No includes, shell commands or environment interpolation are evaluated.

The grouped layout works in YAML and JSON. Earlier flat JSON configurations continue to work, and their field layout can also be read as YAML. Do not mix grouped and legacy keys in one file. Switching the supplied JSON to the equivalent YAML preserves model identity and indexing policy.

## Source and storage

Choose the source and the index independently. Use exactly one of `source.path` and `source.backend` when explicitly selecting a source. A database backend maps paths and text or bytes into a virtual OKF bundle. An upstream PostgreSQL connection can also listen for notifications. [Authoritative SQL mappings](https://tsok.org/knowledge/docs/okf-backends/)

| Combination | Configuration and constraints |
|---|---|
| Folder OKF + SQLite | Set `source.path` and `storage.database`. Folder authoring is available when explicitly enabled and authorized. |
| SQL OKF + SQLite | Set `source.backend.type: sqlite` or `postgres` and map tables/views. Use a distinct SQLite index path. Source authoring belongs to the upstream application. |
| Folder or SQL OKF + PostgreSQL index | Set `storage.postgres` and use a primary OKF bundle. Keep supplemental inputs, URL fetching, consolidation, and federation disabled. For serving, set `server.watch: false` and disable authoring, MCP Tasks and MCP Events. Run the reconciler and workers separately. |
| Supplemental inputs + SQLite | Use `sources.<id>.input` and `sources.<id>.limits` alongside the primary OKF source. An enrollment has an independent namespace, refresh interval and size budget. |

For PostgreSQL storage, `storage.postgres.connection` configures verified database TLS and a dedicated login. `bundle` is the durable namespace. `max_projection_bytes` defaults to 64 MiB. `provider_quota` caps concurrently leased indexing jobs across bundles in one database. `native_vector` enables pgvector on an empty bundle and requires a compatible configured embedding model. The service uses `storage.database` for its local authentication and replay state. [PostgreSQL index options and role setup](https://tsok.org/knowledge/docs/postgres-index/)

| Key under `storage.postgres` | Default or constraint |
|---|---|
| `bundle` | Required, 1–63 lowercase ASCII letters, digits, and hyphens. |
| `connection` | Required `host`, `database`, `user`, `password` secret reference, and PEM `ca_file`. `port` defaults to 5432. |
| `max_projection_bytes` | 67108864, allowed range 1 KiB to 256 MiB. |
| `provider_quota.group` and `max_running` | Optional as a pair. Group names use the bundle naming rules. Capacity is 1–10,000 leased jobs. |
| `native_vector.dimensions` and `ef_search` | Optional. Dimensions are required when enabled, 1–2000. The search budget defaults to 100, allowed range 1–1000. |

The supplemental SQLite connector turns ordinary business rows into generated documents. Use `source.backend` when the rows already represent an authoritative OKF tree. [Connected sources and link syntax](https://tsok.org/knowledge/docs/sources/)

```yaml
source:
  format: open_knowledge_format
  path: ./knowledge
  strict_metadata: true
storage:
  database: ./state/index.sqlite
  encryption_key: {env: KNOWLEDGE_DATABASE_KEY}
```

`source.format` only accepts `open_knowledge_format`. `source.path` contains root `index.md`, Markdown concepts with optional YAML frontmatter, references, and media assets. `strict_metadata` defaults to false in the library and is true in the shipped secure config; strict mode requires a regular root index and valid concept frontmatter. Unknown authored metadata is preserved. The reader checks the format and invariants it needs for indexing. Full OKF conformance is a separate check.

`storage.database` defaults to `.okf/index.sqlite`. `storage.encryption_key` is optional for a CLI/library index; when set, SQLCipher is required and plaintext fallback is refused. Use exactly one `file` or `env` reference. The database key, server tokens, and MCP state keys use random 32-byte values encoded as 64 hex characters. PostgreSQL password references resolve the database login password. `server.require_encryption` defaults to true. File keys must be private regular files; see [SECURITY.md](https://tsok.org/knowledge/docs/security/). Apply filesystem encryption separately to the authored source, and PostgreSQL volume and backup controls separately to the shared index.

## Models, by role

For a model change on an indexed corpus, check the rollout, backfill and rollback steps. [Changing models](https://tsok.org/knowledge/docs/model-changes/)

| Role | Configuration | Purpose |
|---|---|---|
| Text embeddings | `models.embedding.gguf`, `models.embedding.http`, or `models.apple.embedding` | Select one encoder and its document/query prefixes. |
| Direct image/audio embeddings | `models.embedding.gguf.projector_path` or `models.media_embedding` | Use compatible native weights/projector or an explicit joint remote embedding space. |
| Native generation | `models.generation` | RAG answers; optional summaries, labels and grounded entity/relationship extraction |
| HTTP generation | `models.chat` | Generation/extraction endpoint; native generation takes precedence when both are configured |
| Vision | `models.vision` | Image captions and OCR as searchable text |
| Speech-to-text | `models.transcription` | Timestamped transcript via whisper.cpp or OpenAI-compatible HTTP |
| Learned reranking | `models.reranker` | Candidate reranking with `retrieval.rerank: http`. |
| Apple system roles | `models.apple` | Configure embedding, generation, and vision independently. Select one provider per role. |

Embedding produces search vectors. Configure speech transcription, image captioning, or OCR to supply textual evidence for RAG. Apple roles require the local helper and available system models. [Apple model configuration](https://tsok.org/knowledge/docs/apple-models/) No particular model name is hard-coded into the pipeline. Model architecture, pooling, prefixes, projector compatibility and context limits must match the selected runtime and weights. The included native adapter is tested with EmbeddingGemma 2; not every GGUF is an embedding or multimodal model.

### Native embedding and generation

```yaml
models:
  embedding:
    gguf:
      path: ./models/embedding.gguf
      revision: sha256:REPLACE_WITH_ACTUAL_SHA256
      projector_path: ./models/projector.gguf # Optional raw image/audio support.
      projector_revision: sha256:REPLACE_WITH_ACTUAL_SHA256
      context_size: 2048
      gpu_layers: 0 # CPU; set >0 only with a compatible accelerated build.
      threads: 4
    document_prefix: "title: none | text: " # These prefixes are model-specific.
    query_prefix: "task: search result | query: "
  generation:
    path: ./models/instruct.gguf
    context_size: 8192
    max_tokens: 512
    gpu_layers: 0
    threads: 4
    enrich: false
```

`revision` identifies caches; `sha256:...` additionally verifies native weight bytes at load. Use the real checksum, not the placeholder above. Native embedding defaults: context 2048, GPU layers 99, threads 4; generation: context 8192, max tokens 384, GPU layers 99, threads 4, enrichment false. CPU deployments should explicitly set **both** `gpu_layers` values to zero. Native weights load inside the process, requiring the `embedded-gguf` build feature; projector use additionally requires `multimodal`.

### HTTP models and audio transcription

```yaml
models:
  embedding:
    http:
      url: http://127.0.0.1:8080/v1/embeddings
      model: my-embedding-model
      revision: weights-v1
      protocol: openai # openai | ollama
  vision:
    url: http://127.0.0.1:8081/v1/chat/completions
    model: my-vision-model
  transcription:
    url: http://127.0.0.1:8082/inference
    revision: whisper-tiny.en-v1
  reranker:
    url: http://127.0.0.1:8083/rerank
    model: my-cross-encoder
security:
  network:
    allow_loopback_http: true
    allow_remote_https: false
retrieval:
  rerank: http
```

These are examples of services you must run separately, not services launched by `knowledge`. Endpoint URLs are full operation URLs, not base URLs. HTTP endpoints accept `revision`, `protocol` (default `openai`) and optional `api_key_env: VARIABLE_NAME`; credentials are read at request time and are not stored in config. Endpoint URLs cannot contain credentials, query strings or fragments. The reranker has the contract documented in [README.md](https://tsok.org/knowledge/). `transcription.protocol` selects `whisper_cpp` (default) or `openai`; OpenAI-compatible mode requires a model name. See [the full model contracts](https://tsok.org/knowledge/docs/models/).

Network defaults are loopback HTTP allowed and remote HTTPS denied; the shipped secure profile explicitly denies loopback HTTP. Remote providers require `allow_remote_https: true`; remote plaintext HTTP is never permitted. `extra_ca_file` optionally adds a trusted CA. TLS verification, no proxies and no redirects apply to model requests. Knowledge derives an embedding-space ID from the encoder configuration unless you supply `shared_space`. Give an incompatible replacement a new explicit ID when using that setting. [Changing models](https://tsok.org/knowledge/docs/model-changes/#set-the-new-model-identity)

## Chunking, retrieval and compression

```yaml
chunking:
  strategy: markdown
  max_chars: 1800
  max_tokens: null
  overlap_chars: 180
  rules:
    - prefix: procedures
      chunks: {strategy: sentence, max_chars: 400, overlap_chars: 40}
retrieval:
  vector_backend: float32
  lexical_weight: 1.0
  vector_weight: 1.0
  rrf_k: 60
  candidates: 40
  rerank: mmr
  rerank_candidates: 20
  mmr_lambda: 0.7
  graph_predicates: []
  graph_outgoing_only: false
  graph_context_chunks: 10
  max_snapshot_chunks: 50000
  max_snapshot_bytes: 134217728
compression:
  bits: 3
  seed: 1263423319
  exact_rerank: true
```

Choose `fixed`, `recursive`, `sentence`, or `markdown`. Character sizes count Unicode scalar values and overlap must be smaller than the chunk size. `max_tokens` is an optional conservative chunk budget. Rules apply from broad to specific path, followed by a document's `processing.chunking` extension. The file and path policy controls chunking independently of the query-time context budget. [Chunking and evidence](https://tsok.org/knowledge/docs/retrieval/)

Retrieval defaults are shown above, with `rerank: none` as the default. The `coverage`, `mmr`, and `http` rerankers are optional. Candidate budgets are 1–400. The snapshot defaults are 50,000 chunks and 128 MiB of materialized retrieval inputs. PostgreSQL also applies its separate projection byte budget. Query fields control mode, limit, filters, graph hops, and reranking. `graph_predicates` and `graph_outgoing_only` set traversal defaults, and `graph_context_chunks` bounds expansion to at most 400 chunks. [Retrieval contracts](https://tsok.org/knowledge/docs/retrieval/)

Compression is off when omitted; enabling it maintains TurboQuant codes during indexing. Bits must be 2–5, including one residual-sign bit. This codec accepts 4–2048 dimensions; omit compression for larger compatible full-vector encoders. `vector_backend: turboquant` requires a complete compressed index. This experimental CPU backend retains float32 vectors. The SQLite compatibility `search` surface supports TurboQuant. Publication-aware OpenAPI and MCP retrieval use the retained-artifact ranking stack, and PostgreSQL native candidates have their own `storage.postgres.native_vector` setting. [TurboQuant measurements and limits](https://github.com/tsok-org/source-code/blob/main/apps/knowledge/reports/TURBOQUANT.md)

## Server and indexing defaults

The commented default file includes TLS, read/admin token references and the listen address. Additional server parameters:

| Path under `server` | Default | Meaning |
|---|---|---|
| `listen` | `127.0.0.1:8443` | Socket address. Give each instance its own port. |
| `transport.max_connections` | 128 | Live TCP connections, including TLS negotiation, HTTP requests and streams. Range 1–4096. |
| `transport.tls_handshake_timeout_secs` | 10 | TLS negotiation deadline. Range 1–60. |
| `transport.header_timeout_secs` | 10 | HTTP/1.1 header deadline after transport setup. Range 1–60. |
| `transport.shutdown_grace_secs` | 5 | Grace period for HTTP connections during shutdown. Range 1–120. |
| `require_encryption` | true | Require SQLCipher for the embedded index or the PostgreSQL service’s local state. |
| `allow_insecure_loopback` | false | Explicit local-only HTTP development mode |
| `max_body_bytes` | 65536 | Request body ceiling |
| `max_query_chars` | 8000 | Query character ceiling |
| `max_context_chars` | 16000 | RAG context character ceiling |
| `max_parallel_requests` | 2 | Concurrent model/retrieval work budget |
| `requests_per_minute` | 120 | Rate per listener for each of three buckets covering authenticated requests, public or rejected requests, and OAuth authentication attempts. |
| `request_timeout_secs` | 60 | HTTP response deadline; actual work retains its concurrency permit |
| `watch` / `poll` | true / false | Run a watcher for the SQLite index. Poll mode uses periodic reconciliation. Set watch to false when serving the PostgreSQL index. |

For static path-scoped access use `server.principals`. Each entry defines a name, secret reference, capabilities, and OKF path prefixes. A prefix of `""` grants whole-bundle scope. The built-in `read_token` has whole-bundle read access. Set `admin_token: null` to omit the built-in administrator. `server.mtls` controls trusted client certificates and peer mappings. `server.oauth.issuers` configures inbound access-token validation and `server.oauth.login` configures outbound interactive login profiles. [TLS, OAuth, OIDC, and proof rules](https://tsok.org/knowledge/docs/federation-security/)

The [MCP guide](https://tsok.org/knowledge/docs/mcp/) documents every `server.mcp` group. Configure `http` and `stdio` independently. HTTP OAuth uses a separate `/mcp` resource audience. `features` gates skills, apps, subscriptions, authoring, and tasks. `tools.allow` uses short names such as `retrieve`. The opt-in Events draft uses a separate cursor key. Signed webhooks require enrolled HTTPS targets and a namespaced Server Card identity. [MCP Events](https://tsok.org/knowledge/docs/mcp/#mcp-events-sep-3415)

`certificate`, `private_key`, `read_token`, `admin_token`, exact `allowed_hosts` and `allowed_origins` are described in [SECURITY.md](https://tsok.org/knowledge/docs/security/). Run `knowledge check-server` to verify runtime key material and TLS configuration before starting the listener. Omit the server section for CLI-only use.

| Path under `indexing` | Default | Meaning |
|---|---:|---|
| `timeout_secs` | 120 | Model HTTP request timeout |
| `scan_interval_ms` | 2000 | Hash reconciliation interval; minimum 50 |
| `debounce_ms` | 300 | Native notification window measured from the first event |
| `change_batching` | See guide | Scan pacing and per-path revision settling for the automatic watcher. [Indexing resource budgets](https://tsok.org/knowledge/docs/indexing-resources/#frequently-edited-okf-files) |
| `lease_secs` | 180 | Persistent worker lease; minimum 3 |
| `max_attempts` | 5 | Processing retry attempts |
| `retry_base_secs` | 2 | Retry backoff base |
| `embedding_batch_size` | 16 | Text embedding batch; 1–2048 |
| `max_file_bytes` | 67108864 | Per-source byte ceiling |
| `max_scan_files` | 10000 | Recognized files per snapshot |
| `max_scan_entries` | 50000 | Traversed directory entries |
| `max_scan_bytes` | 536870912 | Total source bytes per snapshot |
| `pipeline_revision` | `"1"` | Bump to invalidate processing after custom implementation changes |

Durations use the units in their names and byte fields take integers. Automatic reconciliation uses `change_batching.min_scan_interval_ms` to pace scans, `quiet_ms` to coalesce edits, and `max_delay_ms` to admit persistently changing paths. Its defaults are 1000, 1500, and 30000 ms. In the PostgreSQL index profile, lease, timeout, and attempt caps are 3600 seconds, 300 seconds, and 100 attempts respectively. `resources.max_concurrent_jobs` is a per-process limit of 1–8. `resources.max_concurrent_source_polls` separately limits supplemental polls on SQLite. CPU/GPU placement and work/rest pacing are admission controls. [Resource settings and boundaries](https://tsok.org/knowledge/docs/indexing-resources/)

## Separate knowledge bases

Give each corpus its own config file with its own `source.path`, database, encryption key, model roles and API port. For example:

```sh
knowledge -c research.yml serve
knowledge -c support.yml serve
knowledge -c support.yml search "maintenance interval"
```

These are separate processes/indexes; each loads its configured native models into memory. HTTP backends can share an independently hosted model server. Use configured federation Shares and Mounts for cross-corpus retrieval. Use a distinct SQLite state path per instance or a distinct bundle name in a shared PostgreSQL index.

## Additional role and adapter controls

`models.embedding.shared_space` explicitly identifies a compatible joint text/media vector space. `models.media_embedding` selects an HTTP raw-media encoder (`url`, `model`, `revision`, `api_key_env`, and `format: base64 | data_url | openai_content`). It requires a text embedding backend and shared-space assertion, and cannot be combined with a native projector. `models.enrich: true | false` overrides automatic enrichment for either local or HTTP generation. See [models](https://tsok.org/knowledge/docs/models/) for request/response contracts and compatibility limits.

`models.transcription` additionally accepts `protocol: whisper_cpp | openai`, optional `model`, `language` and `api_key_env`. No credential values are placed in YAML. Model choices/revisions, chunk rules, adapter settings and the data root can differ for each knowledge base.

`video`, `pdf` and `urls` are opt-in sections. All parameters, defaults, source/temporary-file boundaries and refresh semantics are in [formats](https://tsok.org/knowledge/docs/formats/). URL fetching and model-network policies are separate: enabling a provider does not make source URL fetching unrestricted.

`server.allow_source_writes` defaults to false; the shipped secure profile explicitly enables it for the admin token. `server.max_upload_bytes` defaults to 33,554,432 decoded bytes (up to 128 MiB); it has a separate base64-adjusted request ceiling. Ordinary query/page bodies retain `server.max_body_bytes`. See [the write API](https://tsok.org/knowledge/docs/http-api/) for conditional hashes, deletion and capture semantics.

## Federation

`configs/federation-query.yml` is a retrieval request passed to `federation retrieve`. Select the consumer application profile with `-c configs/federation-consumer.yml`.

The top-level `federation` block is optional and disabled by default. It configures publisher identity, Shares, grants, trusted peers, Mounts, query/sync limits and opt-in foreign consolidation. Authentication lives in `server.mtls` and `server.oauth`. See the [field and operations guide](https://tsok.org/knowledge/docs/federation/), [security profiles](https://tsok.org/knowledge/docs/federation-security/), and [owner](https://github.com/tsok-org/source-code/blob/main/apps/knowledge/configs/federation-owner.yml) / [consumer](https://github.com/tsok-org/source-code/blob/main/apps/knowledge/configs/federation-consumer.yml) examples. Model roles and the indexing governor remain per knowledge instance.


## Check a profile before deployment

```sh
knowledge -c configs/reference.yml --validate-config
knowledge -c configs/reference.yml check-server
```

The first check validates the YAML, selected source and compatibility rules. The second reads the referenced key and TLS material and checks the serving configuration. A runtime smoke test still needs the intended source, storage, and model providers. Use [the test guide](https://github.com/tsok-org/source-code/blob/main/apps/knowledge/docs/testing.md) for connector, native model, OpenAPI, MCP, and PostgreSQL exercises.

All 13 application profiles passed structural validation, including the root quickstart and the grouped reference. The review also activated the reference’s authoritative SQLite and PostgreSQL mappings, the three supplemental connectors, PostgreSQL serving with HNSW, Apple and HTTP model alternatives, OAuth issuer settings, a namespaced MCP Server Card, and signed MCP Events in separate configurations. The active reference also passed the TLS and key preflight. The loader rejected simultaneous folder and SQL sources, simultaneous GGUF and HTTP text encoders, missing MCP Events key, and PostgreSQL service combinations with the inline watcher or supplemental connectors.
