Configuring knowledge

Beta.2 adds Knowledge Consolidation and indexing resource budgets; see those guides for their complete YAML, API/library contracts and boundaries.

The first beta guide documents the current evidence/publication, operation, scoped-access and runtime contracts. It supplements the compatibility interfaces described below.

Use configs/reference.yml to review the complete grouped layout. Its active profile reads an Open Knowledge Format folder, indexes into an encrypted SQLite database, and serves lexical retrieval over TLS and MCP. The sample sets strict metadata, a path-specific chunk rule, and a CPU-only resource cap. Commented blocks show alternate sources, shared PostgreSQL storage, optional models, and integration settings. The root knowledge.yml is the native model quickstart and includes a different set of explicit overrides. Both files reference credentials by path or environment variable.

Section What it controls
source Authoritative OKF folder or SQL projection.
sources Supplemental SQLite, manifest, and S3 inputs for a SQLite index.
storage SQLite/SQLCipher path, or PostgreSQL index connection, bundle, quotas, and native vector projection.
models Per-knowledge-base embedding, enrichment, generation, vision, transcription, and reranking roles.
chunking Global chunk policy and ordered path overrides.
retrieval Fusion, ranking, graph expansion, and query memory budgets.
compression Optional experimental TurboQuant projection for compatibility search on SQLite.
video, pdf, urls Decoder settings, media limits, and reference-fetching policy.
indexing Watch and batching policy, leases, retry behavior, concurrency, and resource budgets.
consolidation Evidence-backed background proposals and model selection.
security Outbound model network policy.
server TLS, client authentication, OpenAPI, MCP, and service budgets.
federation Shares, grants, trusted peers, Mounts, and replica budgets.

The sample activates MCP and SQLCipher-backed service state. Use a build with mcp and sqlcipher, and provision the referenced files before starting the listener. Config paths and commands below are relative to the repository root. Local TLS and key setup

knowledge --config knowledge.yml --validate-config
knowledge --config knowledge.yml index
knowledge --config knowledge.yml serve

validate-config is also a subcommand. Validation checks types, unknown keys, policies and the source directory/index. It does not open SQLite, load model weights, resolve secret values or contact servers. Runtime startup performs those checks. Reload configuration by restarting the relevant process. Automatic source refresh requires an active watcher or reconciler. Run check-server separately to preflight the configured service credentials and TLS material.

Choosing a file

  1. --config PATH / -c PATH selects exactly that file.
  2. Otherwise KNOWLEDGE_CONFIG selects exactly one file.
  3. Otherwise the working directory's knowledge.yml or knowledge.yaml is used. Having both is an error; choose explicitly.
  4. With neither YAML file, knowledge.json is the compatibility fallback.

Each process selects one file. YAML takes precedence over a legacy JSON file beside it. Relative paths resolve from the selected file, including model weights, databases, TLS material and secret references. Bare decoder command names such as ffmpeg are looked up on PATH. Unknown keys, duplicate YAML keys, anchors, aliases, merge keys and multiple YAML documents are rejected. Booleans are true and false; yes/on are strings. YAML parsing is bounded to 1 MiB and depth 32. No includes, shell commands or environment interpolation are evaluated.

The grouped layout works in YAML and JSON. Earlier flat JSON configurations continue to work, and their field layout can also be read as YAML. Do not mix grouped and legacy keys in one file. Switching the supplied JSON to the equivalent YAML preserves model identity and indexing policy.

Source and storage

Choose the source and the index independently. Use exactly one of source.path and source.backend when explicitly selecting a source. A database backend maps paths and text or bytes into a virtual OKF bundle. An upstream PostgreSQL connection can also listen for notifications. Authoritative SQL mappings

Combination Configuration and constraints
Folder OKF + SQLite Set source.path and storage.database. Folder authoring is available when explicitly enabled and authorized.
SQL OKF + SQLite Set source.backend.type: sqlite or postgres and map tables/views. Use a distinct SQLite index path. Source authoring belongs to the upstream application.
Folder or SQL OKF + PostgreSQL index Set storage.postgres and use a primary OKF bundle. Keep supplemental inputs, URL fetching, consolidation, and federation disabled. For serving, set server.watch: false and disable authoring, MCP Tasks and MCP Events. Run the reconciler and workers separately.
Supplemental inputs + SQLite Use sources.<id>.input and sources.<id>.limits alongside the primary OKF source. An enrollment has an independent namespace, refresh interval and size budget.

For PostgreSQL storage, storage.postgres.connection configures verified database TLS and a dedicated login. bundle is the durable namespace. max_projection_bytes defaults to 64 MiB. provider_quota caps concurrently leased indexing jobs across bundles in one database. native_vector enables pgvector on an empty bundle and requires a compatible configured embedding model. The service uses storage.database for its local authentication and replay state. PostgreSQL index options and role setup

Key under storage.postgres Default or constraint
bundle Required, 1–63 lowercase ASCII letters, digits, and hyphens.
connection Required host, database, user, password secret reference, and PEM ca_file. port defaults to 5432.
max_projection_bytes 67108864, allowed range 1 KiB to 256 MiB.
provider_quota.group and max_running Optional as a pair. Group names use the bundle naming rules. Capacity is 1–10,000 leased jobs.
native_vector.dimensions and ef_search Optional. Dimensions are required when enabled, 1–2000. The search budget defaults to 100, allowed range 1–1000.

The supplemental SQLite connector turns ordinary business rows into generated documents. Use source.backend when the rows already represent an authoritative OKF tree. Connected sources and link syntax

source:
  format: open_knowledge_format
  path: ./knowledge
  strict_metadata: true
storage:
  database: ./state/index.sqlite
  encryption_key: {env: KNOWLEDGE_DATABASE_KEY}

source.format only accepts open_knowledge_format. source.path contains root index.md, Markdown concepts with optional YAML frontmatter, references, and media assets. strict_metadata defaults to false in the library and is true in the shipped secure config; strict mode requires a regular root index and valid concept frontmatter. Unknown authored metadata is preserved. The reader checks the format and invariants it needs for indexing. Full OKF conformance is a separate check.

storage.database defaults to .okf/index.sqlite. storage.encryption_key is optional for a CLI/library index; when set, SQLCipher is required and plaintext fallback is refused. Use exactly one file or env reference. The database key, server tokens, and MCP state keys use random 32-byte values encoded as 64 hex characters. PostgreSQL password references resolve the database login password. server.require_encryption defaults to true. File keys must be private regular files; see SECURITY.md. Apply filesystem encryption separately to the authored source, and PostgreSQL volume and backup controls separately to the shared index.

Models, by role

For a model change on an indexed corpus, check the rollout, backfill and rollback steps. Changing models

Role Configuration Purpose
Text embeddings models.embedding.gguf, models.embedding.http, or models.apple.embedding Select one encoder and its document/query prefixes.
Direct image/audio embeddings models.embedding.gguf.projector_path or models.media_embedding Use compatible native weights/projector or an explicit joint remote embedding space.
Native generation models.generation RAG answers; optional summaries, labels and grounded entity/relationship extraction
HTTP generation models.chat Generation/extraction endpoint; native generation takes precedence when both are configured
Vision models.vision Image captions and OCR as searchable text
Speech-to-text models.transcription Timestamped transcript via whisper.cpp or OpenAI-compatible HTTP
Learned reranking models.reranker Candidate reranking with retrieval.rerank: http.
Apple system roles models.apple Configure embedding, generation, and vision independently. Select one provider per role.

Embedding produces search vectors. Configure speech transcription, image captioning, or OCR to supply textual evidence for RAG. Apple roles require the local helper and available system models. Apple model configuration No particular model name is hard-coded into the pipeline. Model architecture, pooling, prefixes, projector compatibility and context limits must match the selected runtime and weights. The included native adapter is tested with EmbeddingGemma 2; not every GGUF is an embedding or multimodal model.

Native embedding and generation

models:
  embedding:
    gguf:
      path: ./models/embedding.gguf
      revision: sha256:REPLACE_WITH_ACTUAL_SHA256
      projector_path: ./models/projector.gguf # Optional raw image/audio support.
      projector_revision: sha256:REPLACE_WITH_ACTUAL_SHA256
      context_size: 2048
      gpu_layers: 0 # CPU; set >0 only with a compatible accelerated build.
      threads: 4
    document_prefix: "title: none | text: " # These prefixes are model-specific.
    query_prefix: "task: search result | query: "
  generation:
    path: ./models/instruct.gguf
    context_size: 8192
    max_tokens: 512
    gpu_layers: 0
    threads: 4
    enrich: false

revision identifies caches; sha256:... additionally verifies native weight bytes at load. Use the real checksum, not the placeholder above. Native embedding defaults: context 2048, GPU layers 99, threads 4; generation: context 8192, max tokens 384, GPU layers 99, threads 4, enrichment false. CPU deployments should explicitly set both gpu_layers values to zero. Native weights load inside the process, requiring the embedded-gguf build feature; projector use additionally requires multimodal.

HTTP models and audio transcription

models:
  embedding:
    http:
      url: http://127.0.0.1:8080/v1/embeddings
      model: my-embedding-model
      revision: weights-v1
      protocol: openai # openai | ollama
  vision:
    url: http://127.0.0.1:8081/v1/chat/completions
    model: my-vision-model
  transcription:
    url: http://127.0.0.1:8082/inference
    revision: whisper-tiny.en-v1
  reranker:
    url: http://127.0.0.1:8083/rerank
    model: my-cross-encoder
security:
  network:
    allow_loopback_http: true
    allow_remote_https: false
retrieval:
  rerank: http

These are examples of services you must run separately, not services launched by knowledge. Endpoint URLs are full operation URLs, not base URLs. HTTP endpoints accept revision, protocol (default openai) and optional api_key_env: VARIABLE_NAME; credentials are read at request time and are not stored in config. Endpoint URLs cannot contain credentials, query strings or fragments. The reranker has the contract documented in README.md. transcription.protocol selects whisper_cpp (default) or openai; OpenAI-compatible mode requires a model name. See the full model contracts.

Network defaults are loopback HTTP allowed and remote HTTPS denied; the shipped secure profile explicitly denies loopback HTTP. Remote providers require allow_remote_https: true; remote plaintext HTTP is never permitted. extra_ca_file optionally adds a trusted CA. TLS verification, no proxies and no redirects apply to model requests. Knowledge derives an embedding-space ID from the encoder configuration unless you supply shared_space. Give an incompatible replacement a new explicit ID when using that setting. Changing models

Chunking, retrieval and compression

chunking:
  strategy: markdown
  max_chars: 1800
  max_tokens: null
  overlap_chars: 180
  rules:
    - prefix: procedures
      chunks: {strategy: sentence, max_chars: 400, overlap_chars: 40}
retrieval:
  vector_backend: float32
  lexical_weight: 1.0
  vector_weight: 1.0
  rrf_k: 60
  candidates: 40
  rerank: mmr
  rerank_candidates: 20
  mmr_lambda: 0.7
  graph_predicates: []
  graph_outgoing_only: false
  graph_context_chunks: 10
  max_snapshot_chunks: 50000
  max_snapshot_bytes: 134217728
compression:
  bits: 3
  seed: 1263423319
  exact_rerank: true

Choose fixed, recursive, sentence, or markdown. Character sizes count Unicode scalar values and overlap must be smaller than the chunk size. max_tokens is an optional conservative chunk budget. Rules apply from broad to specific path, followed by a document's processing.chunking extension. The file and path policy controls chunking independently of the query-time context budget. Chunking and evidence

Retrieval defaults are shown above, with rerank: none as the default. The coverage, mmr, and http rerankers are optional. Candidate budgets are 1–400. The snapshot defaults are 50,000 chunks and 128 MiB of materialized retrieval inputs. PostgreSQL also applies its separate projection byte budget. Query fields control mode, limit, filters, graph hops, and reranking. graph_predicates and graph_outgoing_only set traversal defaults, and graph_context_chunks bounds expansion to at most 400 chunks. Retrieval contracts

Compression is off when omitted; enabling it maintains TurboQuant codes during indexing. Bits must be 2–5, including one residual-sign bit. This codec accepts 4–2048 dimensions; omit compression for larger compatible full-vector encoders. vector_backend: turboquant requires a complete compressed index. This experimental CPU backend retains float32 vectors. The SQLite compatibility search surface supports TurboQuant. Publication-aware OpenAPI and MCP retrieval use the retained-artifact ranking stack, and PostgreSQL native candidates have their own storage.postgres.native_vector setting. TurboQuant measurements and limits

Server and indexing defaults

The commented default file includes TLS, read/admin token references and the listen address. Additional server parameters:

Path under server Default Meaning
listen 127.0.0.1:8443 Socket address. Give each instance its own port.
transport.max_connections 128 Live TCP connections, including TLS negotiation, HTTP requests and streams. Range 1–4096.
transport.tls_handshake_timeout_secs 10 TLS negotiation deadline. Range 1–60.
transport.header_timeout_secs 10 HTTP/1.1 header deadline after transport setup. Range 1–60.
transport.shutdown_grace_secs 5 Grace period for HTTP connections during shutdown. Range 1–120.
require_encryption true Require SQLCipher for the embedded index or the PostgreSQL service’s local state.
allow_insecure_loopback false Explicit local-only HTTP development mode
max_body_bytes 65536 Request body ceiling
max_query_chars 8000 Query character ceiling
max_context_chars 16000 RAG context character ceiling
max_parallel_requests 2 Concurrent model/retrieval work budget
requests_per_minute 120 Rate per listener for each of three buckets covering authenticated requests, public or rejected requests, and OAuth authentication attempts.
request_timeout_secs 60 HTTP response deadline; actual work retains its concurrency permit
watch / poll true / false Run a watcher for the SQLite index. Poll mode uses periodic reconciliation. Set watch to false when serving the PostgreSQL index.

For static path-scoped access use server.principals. Each entry defines a name, secret reference, capabilities, and OKF path prefixes. A prefix of "" grants whole-bundle scope. The built-in read_token has whole-bundle read access. Set admin_token: null to omit the built-in administrator. server.mtls controls trusted client certificates and peer mappings. server.oauth.issuers configures inbound access-token validation and server.oauth.login configures outbound interactive login profiles. TLS, OAuth, OIDC, and proof rules

The MCP guide documents every server.mcp group. Configure http and stdio independently. HTTP OAuth uses a separate /mcp resource audience. features gates skills, apps, subscriptions, authoring, and tasks. tools.allow uses short names such as retrieve. The opt-in Events draft uses a separate cursor key. Signed webhooks require enrolled HTTPS targets and a namespaced Server Card identity. MCP Events

certificate, private_key, read_token, admin_token, exact allowed_hosts and allowed_origins are described in SECURITY.md. Run knowledge check-server to verify runtime key material and TLS configuration before starting the listener. Omit the server section for CLI-only use.

Path under indexing Default Meaning
timeout_secs 120 Model HTTP request timeout
scan_interval_ms 2000 Hash reconciliation interval; minimum 50
debounce_ms 300 Native notification window measured from the first event
change_batching See guide Scan pacing and per-path revision settling for the automatic watcher. Indexing resource budgets
lease_secs 180 Persistent worker lease; minimum 3
max_attempts 5 Processing retry attempts
retry_base_secs 2 Retry backoff base
embedding_batch_size 16 Text embedding batch; 1–2048
max_file_bytes 67108864 Per-source byte ceiling
max_scan_files 10000 Recognized files per snapshot
max_scan_entries 50000 Traversed directory entries
max_scan_bytes 536870912 Total source bytes per snapshot
pipeline_revision "1" Bump to invalidate processing after custom implementation changes

Durations use the units in their names and byte fields take integers. Automatic reconciliation uses change_batching.min_scan_interval_ms to pace scans, quiet_ms to coalesce edits, and max_delay_ms to admit persistently changing paths. Its defaults are 1000, 1500, and 30000 ms. In the PostgreSQL index profile, lease, timeout, and attempt caps are 3600 seconds, 300 seconds, and 100 attempts respectively. resources.max_concurrent_jobs is a per-process limit of 1–8. resources.max_concurrent_source_polls separately limits supplemental polls on SQLite. CPU/GPU placement and work/rest pacing are admission controls. Resource settings and boundaries

Separate knowledge bases

Give each corpus its own config file with its own source.path, database, encryption key, model roles and API port. For example:

knowledge -c research.yml serve
knowledge -c support.yml serve
knowledge -c support.yml search "maintenance interval"

These are separate processes/indexes; each loads its configured native models into memory. HTTP backends can share an independently hosted model server. Use configured federation Shares and Mounts for cross-corpus retrieval. Use a distinct SQLite state path per instance or a distinct bundle name in a shared PostgreSQL index.

Additional role and adapter controls

models.embedding.shared_space explicitly identifies a compatible joint text/media vector space. models.media_embedding selects an HTTP raw-media encoder (url, model, revision, api_key_env, and format: base64 | data_url | openai_content). It requires a text embedding backend and shared-space assertion, and cannot be combined with a native projector. models.enrich: true | false overrides automatic enrichment for either local or HTTP generation. See models for request/response contracts and compatibility limits.

models.transcription additionally accepts protocol: whisper_cpp | openai, optional model, language and api_key_env. No credential values are placed in YAML. Model choices/revisions, chunk rules, adapter settings and the data root can differ for each knowledge base.

video, pdf and urls are opt-in sections. All parameters, defaults, source/temporary-file boundaries and refresh semantics are in formats. URL fetching and model-network policies are separate: enabling a provider does not make source URL fetching unrestricted.

server.allow_source_writes defaults to false; the shipped secure profile explicitly enables it for the admin token. server.max_upload_bytes defaults to 33,554,432 decoded bytes (up to 128 MiB); it has a separate base64-adjusted request ceiling. Ordinary query/page bodies retain server.max_body_bytes. See the write API for conditional hashes, deletion and capture semantics.

Federation

configs/federation-query.yml is a retrieval request passed to federation retrieve. Select the consumer application profile with -c configs/federation-consumer.yml.

The top-level federation block is optional and disabled by default. It configures publisher identity, Shares, grants, trusted peers, Mounts, query/sync limits and opt-in foreign consolidation. Authentication lives in server.mtls and server.oauth. See the field and operations guide, security profiles, and owner / consumer examples. Model roles and the indexing governor remain per knowledge instance.

Check a profile before deployment

knowledge -c configs/reference.yml --validate-config
knowledge -c configs/reference.yml check-server

The first check validates the YAML, selected source and compatibility rules. The second reads the referenced key and TLS material and checks the serving configuration. A runtime smoke test still needs the intended source, storage, and model providers. Use the test guide for connector, native model, OpenAPI, MCP, and PostgreSQL exercises.

All 13 application profiles passed structural validation, including the root quickstart and the grouped reference. The review also activated the reference’s authoritative SQLite and PostgreSQL mappings, the three supplemental connectors, PostgreSQL serving with HNSW, Apple and HTTP model alternatives, OAuth issuer settings, a namespaced MCP Server Card, and signed MCP Events in separate configurations. The active reference also passed the TLS and key preflight. The loader rejected simultaneous folder and SQL sources, simultaneous GGUF and HTTP text encoders, missing MCP Events key, and PostgreSQL service combinations with the inline watcher or supplemental connectors.

Read Markdown →