Configuring knowledge
Beta.2 adds Knowledge Consolidation and indexing resource budgets; see those guides for their complete YAML, API/library contracts and boundaries.
The first beta guide documents the current evidence/publication, operation, scoped-access and runtime contracts. It supplements the compatibility interfaces described below.
Use configs/reference.yml to review the complete grouped layout. Its active profile reads an Open Knowledge Format folder, indexes into an encrypted SQLite database, and serves lexical retrieval over TLS and MCP. The sample sets strict metadata, a path-specific chunk rule, and a CPU-only resource cap. Commented blocks show alternate sources, shared PostgreSQL storage, optional models, and integration settings. The root knowledge.yml is the native model quickstart and includes a different set of explicit overrides. Both files reference credentials by path or environment variable.
| Section | What it controls |
|---|---|
source |
Authoritative OKF folder or SQL projection. |
sources |
Supplemental SQLite, manifest, and S3 inputs for a SQLite index. |
storage |
SQLite/SQLCipher path, or PostgreSQL index connection, bundle, quotas, and native vector projection. |
models |
Per-knowledge-base embedding, enrichment, generation, vision, transcription, and reranking roles. |
chunking |
Global chunk policy and ordered path overrides. |
retrieval |
Fusion, ranking, graph expansion, and query memory budgets. |
compression |
Optional experimental TurboQuant projection for compatibility search on SQLite. |
video, pdf, urls |
Decoder settings, media limits, and reference-fetching policy. |
indexing |
Watch and batching policy, leases, retry behavior, concurrency, and resource budgets. |
consolidation |
Evidence-backed background proposals and model selection. |
security |
Outbound model network policy. |
server |
TLS, client authentication, OpenAPI, MCP, and service budgets. |
federation |
Shares, grants, trusted peers, Mounts, and replica budgets. |
The sample activates MCP and SQLCipher-backed service state. Use a build with mcp and sqlcipher, and provision the referenced files before starting the listener. Config paths and commands below are relative to the repository root. Local TLS and key setup
knowledge --config knowledge.yml --validate-config
knowledge --config knowledge.yml index
knowledge --config knowledge.yml serve
validate-config is also a subcommand. Validation checks types, unknown keys, policies and the source directory/index. It does not open SQLite, load model weights, resolve secret values or contact servers. Runtime startup performs those checks. Reload configuration by restarting the relevant process. Automatic source refresh requires an active watcher or reconciler. Run check-server separately to preflight the configured service credentials and TLS material.
Choosing a file
--config PATH/-c PATHselects exactly that file.- Otherwise
KNOWLEDGE_CONFIGselects exactly one file. - Otherwise the working directory's
knowledge.ymlorknowledge.yamlis used. Having both is an error; choose explicitly. - With neither YAML file,
knowledge.jsonis the compatibility fallback.
Each process selects one file. YAML takes precedence over a legacy JSON file beside it. Relative paths resolve from the selected file, including model weights, databases, TLS material and secret references. Bare decoder command names such as ffmpeg are looked up on PATH. Unknown keys, duplicate YAML keys, anchors, aliases, merge keys and multiple YAML documents are rejected. Booleans are true and false; yes/on are strings. YAML parsing is bounded to 1 MiB and depth 32. No includes, shell commands or environment interpolation are evaluated.
The grouped layout works in YAML and JSON. Earlier flat JSON configurations continue to work, and their field layout can also be read as YAML. Do not mix grouped and legacy keys in one file. Switching the supplied JSON to the equivalent YAML preserves model identity and indexing policy.
Source and storage
Choose the source and the index independently. Use exactly one of source.path and source.backend when explicitly selecting a source. A database backend maps paths and text or bytes into a virtual OKF bundle. An upstream PostgreSQL connection can also listen for notifications. Authoritative SQL mappings
| Combination | Configuration and constraints |
|---|---|
| Folder OKF + SQLite | Set source.path and storage.database. Folder authoring is available when explicitly enabled and authorized. |
| SQL OKF + SQLite | Set source.backend.type: sqlite or postgres and map tables/views. Use a distinct SQLite index path. Source authoring belongs to the upstream application. |
| Folder or SQL OKF + PostgreSQL index | Set storage.postgres and use a primary OKF bundle. Keep supplemental inputs, URL fetching, consolidation, and federation disabled. For serving, set server.watch: false and disable authoring, MCP Tasks and MCP Events. Run the reconciler and workers separately. |
| Supplemental inputs + SQLite | Use sources.<id>.input and sources.<id>.limits alongside the primary OKF source. An enrollment has an independent namespace, refresh interval and size budget. |
For PostgreSQL storage, storage.postgres.connection configures verified database TLS and a dedicated login. bundle is the durable namespace. max_projection_bytes defaults to 64 MiB. provider_quota caps concurrently leased indexing jobs across bundles in one database. native_vector enables pgvector on an empty bundle and requires a compatible configured embedding model. The service uses storage.database for its local authentication and replay state. PostgreSQL index options and role setup
Key under storage.postgres |
Default or constraint |
|---|---|
bundle |
Required, 1–63 lowercase ASCII letters, digits, and hyphens. |
connection |
Required host, database, user, password secret reference, and PEM ca_file. port defaults to 5432. |
max_projection_bytes |
67108864, allowed range 1 KiB to 256 MiB. |
provider_quota.group and max_running |
Optional as a pair. Group names use the bundle naming rules. Capacity is 1–10,000 leased jobs. |
native_vector.dimensions and ef_search |
Optional. Dimensions are required when enabled, 1–2000. The search budget defaults to 100, allowed range 1–1000. |
The supplemental SQLite connector turns ordinary business rows into generated documents. Use source.backend when the rows already represent an authoritative OKF tree. Connected sources and link syntax
source:
format: open_knowledge_format
path: ./knowledge
strict_metadata: true
storage:
database: ./state/index.sqlite
encryption_key: {env: KNOWLEDGE_DATABASE_KEY}
source.format only accepts open_knowledge_format. source.path contains root index.md, Markdown concepts with optional YAML frontmatter, references, and media assets. strict_metadata defaults to false in the library and is true in the shipped secure config; strict mode requires a regular root index and valid concept frontmatter. Unknown authored metadata is preserved. The reader checks the format and invariants it needs for indexing. Full OKF conformance is a separate check.
storage.database defaults to .okf/index.sqlite. storage.encryption_key is optional for a CLI/library index; when set, SQLCipher is required and plaintext fallback is refused. Use exactly one file or env reference. The database key, server tokens, and MCP state keys use random 32-byte values encoded as 64 hex characters. PostgreSQL password references resolve the database login password. server.require_encryption defaults to true. File keys must be private regular files; see SECURITY.md. Apply filesystem encryption separately to the authored source, and PostgreSQL volume and backup controls separately to the shared index.
Models, by role
For a model change on an indexed corpus, check the rollout, backfill and rollback steps. Changing models
| Role | Configuration | Purpose |
|---|---|---|
| Text embeddings | models.embedding.gguf, models.embedding.http, or models.apple.embedding |
Select one encoder and its document/query prefixes. |
| Direct image/audio embeddings | models.embedding.gguf.projector_path or models.media_embedding |
Use compatible native weights/projector or an explicit joint remote embedding space. |
| Native generation | models.generation |
RAG answers; optional summaries, labels and grounded entity/relationship extraction |
| HTTP generation | models.chat |
Generation/extraction endpoint; native generation takes precedence when both are configured |
| Vision | models.vision |
Image captions and OCR as searchable text |
| Speech-to-text | models.transcription |
Timestamped transcript via whisper.cpp or OpenAI-compatible HTTP |
| Learned reranking | models.reranker |
Candidate reranking with retrieval.rerank: http. |
| Apple system roles | models.apple |
Configure embedding, generation, and vision independently. Select one provider per role. |
Embedding produces search vectors. Configure speech transcription, image captioning, or OCR to supply textual evidence for RAG. Apple roles require the local helper and available system models. Apple model configuration No particular model name is hard-coded into the pipeline. Model architecture, pooling, prefixes, projector compatibility and context limits must match the selected runtime and weights. The included native adapter is tested with EmbeddingGemma 2; not every GGUF is an embedding or multimodal model.
Native embedding and generation
models:
embedding:
gguf:
path: ./models/embedding.gguf
revision: sha256:REPLACE_WITH_ACTUAL_SHA256
projector_path: ./models/projector.gguf # Optional raw image/audio support.
projector_revision: sha256:REPLACE_WITH_ACTUAL_SHA256
context_size: 2048
gpu_layers: 0 # CPU; set >0 only with a compatible accelerated build.
threads: 4
document_prefix: "title: none | text: " # These prefixes are model-specific.
query_prefix: "task: search result | query: "
generation:
path: ./models/instruct.gguf
context_size: 8192
max_tokens: 512
gpu_layers: 0
threads: 4
enrich: false
revision identifies caches; sha256:... additionally verifies native weight bytes at load. Use the real checksum, not the placeholder above. Native embedding defaults: context 2048, GPU layers 99, threads 4; generation: context 8192, max tokens 384, GPU layers 99, threads 4, enrichment false. CPU deployments should explicitly set both gpu_layers values to zero. Native weights load inside the process, requiring the embedded-gguf build feature; projector use additionally requires multimodal.
HTTP models and audio transcription
models:
embedding:
http:
url: http://127.0.0.1:8080/v1/embeddings
model: my-embedding-model
revision: weights-v1
protocol: openai # openai | ollama
vision:
url: http://127.0.0.1:8081/v1/chat/completions
model: my-vision-model
transcription:
url: http://127.0.0.1:8082/inference
revision: whisper-tiny.en-v1
reranker:
url: http://127.0.0.1:8083/rerank
model: my-cross-encoder
security:
network:
allow_loopback_http: true
allow_remote_https: false
retrieval:
rerank: http
These are examples of services you must run separately, not services launched by knowledge. Endpoint URLs are full operation URLs, not base URLs. HTTP endpoints accept revision, protocol (default openai) and optional api_key_env: VARIABLE_NAME; credentials are read at request time and are not stored in config. Endpoint URLs cannot contain credentials, query strings or fragments. The reranker has the contract documented in README.md. transcription.protocol selects whisper_cpp (default) or openai; OpenAI-compatible mode requires a model name. See the full model contracts.
Network defaults are loopback HTTP allowed and remote HTTPS denied; the shipped secure profile explicitly denies loopback HTTP. Remote providers require allow_remote_https: true; remote plaintext HTTP is never permitted. extra_ca_file optionally adds a trusted CA. TLS verification, no proxies and no redirects apply to model requests. Knowledge derives an embedding-space ID from the encoder configuration unless you supply shared_space. Give an incompatible replacement a new explicit ID when using that setting. Changing models
Chunking, retrieval and compression
chunking:
strategy: markdown
max_chars: 1800
max_tokens: null
overlap_chars: 180
rules:
- prefix: procedures
chunks: {strategy: sentence, max_chars: 400, overlap_chars: 40}
retrieval:
vector_backend: float32
lexical_weight: 1.0
vector_weight: 1.0
rrf_k: 60
candidates: 40
rerank: mmr
rerank_candidates: 20
mmr_lambda: 0.7
graph_predicates: []
graph_outgoing_only: false
graph_context_chunks: 10
max_snapshot_chunks: 50000
max_snapshot_bytes: 134217728
compression:
bits: 3
seed: 1263423319
exact_rerank: true
Choose fixed, recursive, sentence, or markdown. Character sizes count Unicode scalar values and overlap must be smaller than the chunk size. max_tokens is an optional conservative chunk budget. Rules apply from broad to specific path, followed by a document's processing.chunking extension. The file and path policy controls chunking independently of the query-time context budget. Chunking and evidence
Retrieval defaults are shown above, with rerank: none as the default. The coverage, mmr, and http rerankers are optional. Candidate budgets are 1–400. The snapshot defaults are 50,000 chunks and 128 MiB of materialized retrieval inputs. PostgreSQL also applies its separate projection byte budget. Query fields control mode, limit, filters, graph hops, and reranking. graph_predicates and graph_outgoing_only set traversal defaults, and graph_context_chunks bounds expansion to at most 400 chunks. Retrieval contracts
Compression is off when omitted; enabling it maintains TurboQuant codes during indexing. Bits must be 2–5, including one residual-sign bit. This codec accepts 4–2048 dimensions; omit compression for larger compatible full-vector encoders. vector_backend: turboquant requires a complete compressed index. This experimental CPU backend retains float32 vectors. The SQLite compatibility search surface supports TurboQuant. Publication-aware OpenAPI and MCP retrieval use the retained-artifact ranking stack, and PostgreSQL native candidates have their own storage.postgres.native_vector setting. TurboQuant measurements and limits
Server and indexing defaults
The commented default file includes TLS, read/admin token references and the listen address. Additional server parameters:
Path under server |
Default | Meaning |
|---|---|---|
listen |
127.0.0.1:8443 |
Socket address. Give each instance its own port. |
transport.max_connections |
128 | Live TCP connections, including TLS negotiation, HTTP requests and streams. Range 1–4096. |
transport.tls_handshake_timeout_secs |
10 | TLS negotiation deadline. Range 1–60. |
transport.header_timeout_secs |
10 | HTTP/1.1 header deadline after transport setup. Range 1–60. |
transport.shutdown_grace_secs |
5 | Grace period for HTTP connections during shutdown. Range 1–120. |
require_encryption |
true | Require SQLCipher for the embedded index or the PostgreSQL service’s local state. |
allow_insecure_loopback |
false | Explicit local-only HTTP development mode |
max_body_bytes |
65536 | Request body ceiling |
max_query_chars |
8000 | Query character ceiling |
max_context_chars |
16000 | RAG context character ceiling |
max_parallel_requests |
2 | Concurrent model/retrieval work budget |
requests_per_minute |
120 | Rate per listener for each of three buckets covering authenticated requests, public or rejected requests, and OAuth authentication attempts. |
request_timeout_secs |
60 | HTTP response deadline; actual work retains its concurrency permit |
watch / poll |
true / false | Run a watcher for the SQLite index. Poll mode uses periodic reconciliation. Set watch to false when serving the PostgreSQL index. |
For static path-scoped access use server.principals. Each entry defines a name, secret reference, capabilities, and OKF path prefixes. A prefix of "" grants whole-bundle scope. The built-in read_token has whole-bundle read access. Set admin_token: null to omit the built-in administrator. server.mtls controls trusted client certificates and peer mappings. server.oauth.issuers configures inbound access-token validation and server.oauth.login configures outbound interactive login profiles. TLS, OAuth, OIDC, and proof rules
The MCP guide documents every server.mcp group. Configure http and stdio independently. HTTP OAuth uses a separate /mcp resource audience. features gates skills, apps, subscriptions, authoring, and tasks. tools.allow uses short names such as retrieve. The opt-in Events draft uses a separate cursor key. Signed webhooks require enrolled HTTPS targets and a namespaced Server Card identity. MCP Events
certificate, private_key, read_token, admin_token, exact allowed_hosts and allowed_origins are described in SECURITY.md. Run knowledge check-server to verify runtime key material and TLS configuration before starting the listener. Omit the server section for CLI-only use.
Path under indexing |
Default | Meaning |
|---|---|---|
timeout_secs |
120 | Model HTTP request timeout |
scan_interval_ms |
2000 | Hash reconciliation interval; minimum 50 |
debounce_ms |
300 | Native notification window measured from the first event |
change_batching |
See guide | Scan pacing and per-path revision settling for the automatic watcher. Indexing resource budgets |
lease_secs |
180 | Persistent worker lease; minimum 3 |
max_attempts |
5 | Processing retry attempts |
retry_base_secs |
2 | Retry backoff base |
embedding_batch_size |
16 | Text embedding batch; 1–2048 |
max_file_bytes |
67108864 | Per-source byte ceiling |
max_scan_files |
10000 | Recognized files per snapshot |
max_scan_entries |
50000 | Traversed directory entries |
max_scan_bytes |
536870912 | Total source bytes per snapshot |
pipeline_revision |
"1" |
Bump to invalidate processing after custom implementation changes |
Durations use the units in their names and byte fields take integers. Automatic reconciliation uses change_batching.min_scan_interval_ms to pace scans, quiet_ms to coalesce edits, and max_delay_ms to admit persistently changing paths. Its defaults are 1000, 1500, and 30000 ms. In the PostgreSQL index profile, lease, timeout, and attempt caps are 3600 seconds, 300 seconds, and 100 attempts respectively. resources.max_concurrent_jobs is a per-process limit of 1–8. resources.max_concurrent_source_polls separately limits supplemental polls on SQLite. CPU/GPU placement and work/rest pacing are admission controls. Resource settings and boundaries
Separate knowledge bases
Give each corpus its own config file with its own source.path, database, encryption key, model roles and API port. For example:
knowledge -c research.yml serve
knowledge -c support.yml serve
knowledge -c support.yml search "maintenance interval"
These are separate processes/indexes; each loads its configured native models into memory. HTTP backends can share an independently hosted model server. Use configured federation Shares and Mounts for cross-corpus retrieval. Use a distinct SQLite state path per instance or a distinct bundle name in a shared PostgreSQL index.
Additional role and adapter controls
models.embedding.shared_space explicitly identifies a compatible joint text/media vector space. models.media_embedding selects an HTTP raw-media encoder (url, model, revision, api_key_env, and format: base64 | data_url | openai_content). It requires a text embedding backend and shared-space assertion, and cannot be combined with a native projector. models.enrich: true | false overrides automatic enrichment for either local or HTTP generation. See models for request/response contracts and compatibility limits.
models.transcription additionally accepts protocol: whisper_cpp | openai, optional model, language and api_key_env. No credential values are placed in YAML. Model choices/revisions, chunk rules, adapter settings and the data root can differ for each knowledge base.
video, pdf and urls are opt-in sections. All parameters, defaults, source/temporary-file boundaries and refresh semantics are in formats. URL fetching and model-network policies are separate: enabling a provider does not make source URL fetching unrestricted.
server.allow_source_writes defaults to false; the shipped secure profile explicitly enables it for the admin token. server.max_upload_bytes defaults to 33,554,432 decoded bytes (up to 128 MiB); it has a separate base64-adjusted request ceiling. Ordinary query/page bodies retain server.max_body_bytes. See the write API for conditional hashes, deletion and capture semantics.
Federation
configs/federation-query.yml is a retrieval request passed to federation retrieve. Select the consumer application profile with -c configs/federation-consumer.yml.
The top-level federation block is optional and disabled by default. It configures publisher identity, Shares, grants, trusted peers, Mounts, query/sync limits and opt-in foreign consolidation. Authentication lives in server.mtls and server.oauth. See the field and operations guide, security profiles, and owner / consumer examples. Model roles and the indexing governor remain per knowledge instance.
Check a profile before deployment
knowledge -c configs/reference.yml --validate-config
knowledge -c configs/reference.yml check-server
The first check validates the YAML, selected source and compatibility rules. The second reads the referenced key and TLS material and checks the serving configuration. A runtime smoke test still needs the intended source, storage, and model providers. Use the test guide for connector, native model, OpenAPI, MCP, and PostgreSQL exercises.
All 13 application profiles passed structural validation, including the root quickstart and the grouped reference. The review also activated the reference’s authoritative SQLite and PostgreSQL mappings, the three supplemental connectors, PostgreSQL serving with HNSW, Apple and HTTP model alternatives, OAuth issuer settings, a namespaced MCP Server Card, and signed MCP Events in separate configurations. The active reference also passed the TLS and key preflight. The loader rejected simultaneous folder and SQL sources, simultaneous GGUF and HTTP text encoders, missing MCP Events key, and PostgreSQL service combinations with the inline watcher or supplemental connectors.