knowledge beta evidence contract
This guide originated with beta.1. Beta.2 adds consolidation, indexing budgets and schema 5. Beta.3 adds federation and schema 6. Existing evidence and operation contracts remain supported.
The first beta turns the OKF index into a versioned knowledge source. OpenAPI 3.2.1 is the HTTP contract. An external MCP gateway can map the declared operations/resources; knowledge does not run a second MCP protocol implementation.
Supported contract
A retrieval identifies its immutable publication. Publications also have globally distinct publication_key values; prefer these for stored snapshot references across restores/forks. Numeric IDs identify ordering within a database timeline. Every hit includes an evidence ID, knowledge URI, source revision/hash, processing recipe, representation and explicit locator. Evidence IDs refer to retained data, not a mutable position in the current chunk list. Edits, moves and rechunking create new artifacts; old IDs continue to resolve until explicit retention purges them. Every request uses current configured authorization, including historical reads. Purged or unauthorized evidence returns unavailable; it is never silently replaced with a newer excerpt.
index_stale compares a returned revision to the currently observed desired revision. content_freshness separately evaluates OKF stale_after; missing freshness is unknown. Neither a relevance score nor source-supplied verification metadata proves truth. Source text and model-derived annotations remain untrusted content for consumers, including agents.
The database has logical source (versions, remote snapshots), artifact (artifacts, evidence, binary_artifacts, stage/cache records), and projection (documents, chunks/FTS/vector tables, graph, publications) layers. Publication membership uses validity intervals, sharing unchanged concepts across revisions instead of copying the whole corpus for every publication. These layers are in one SQLite database for this beta. It is not safe to discard the entire database as if it held only indexes: historical URL bytes, generated evidence and revisions may have no remaining external copy.
Agent/application workflow
- Discover
/v1/capabilitiesand/openapi.json. - Create/update/move/remove/capture through
POST /v1/changes, with a caller-scopedidempotency_key. Updates and removals require exact SHA-256 preconditions. Capture creates an OKF page, navigation index, original HTML and allowed assets. Changes use the configured upload-body budget, separately from small retrieval/query bodies; large batches must fit that bounded request. - Poll
GET /v1/operations/{id}untilpublished.indexingmeans the source was committed, not that retrieval has caught up. With the watcher disabled, the host must calldrainor runknowledge work. - Use
/v1/retrievewithafter_operation, or pinpublication_id./v1/selectreturns catalog resources withrole: concept | navigation | asset, preserving OKF’s distinction between Markdown concepts, reserved navigation files and referenced assets. It supports deterministic JSON-pointer metadata filters and pagination without model calls. - Use
/v1/context/buildfor bounded evidence independently of answer generation. Resolve IDs using/v1/evidence/{id}./v1/resources/{version_id}streams retained source bytes;/v1/media/{evidence_id}serves the exact sampled media or original associated with evidence. Both support one HTTP byte range and conditional ETags after authorization. - Poll
/v1/events?after=CURSOR&limit=100; persistnext_cursor. Operations are private to their caller/admin. Events and historical results still obey current path scopes. Publication sequence/activity is collection metadata; scopes do not promise to hide traffic timing. Events are retained in this beta, not an infinite replay guarantee across future retention-policy changes.
The checked OpenAPI contract includes gateway hints and stable operation IDs. Dependency-free Python and TypeScript clients are generated by scripts/generate-clients.py; use a trusted local CA for TLS. Compatibility source-writing endpoints are deprecated and do not issue the durable operation contract; use /v1/changes for beta integrations. Gateways must forward caller credentials and should begin with the x-knowledge-gateway.readOperations profile. Expose authoring/admin operations only when explicitly required. Gateway annotations do not grant privileges.
Example context request:
{
"search": {
"query": "How often is Orion calibrated?",
"mode": "hybrid",
"filters": {"metadata": {"/type": "Procedure"}},
"graph_hops": 2,
"predicates": ["depends_on"],
"rerank": "mmr",
"strict": false
},
"context": {"max_tokens": 4096, "max_per_source": 3, "neighbors": 1}
}
HTTP token accounting conservatively charges UTF-8 bytes. The response declares token_accounting; this is deliberately not advertised as an exact count for an arbitrary consumer model. The Rust TokenCounter interface accepts the consumer tokenizer for exact accounting. max_per_source limits text passages; media attachments are governed by modality and retrieval limits. Text-only consumers omit modalities; media-capable consumers can request supported representation families and fetch their associated resources. Media references are evidence attachments, not textual proof or an estimate of a consumer's image/audio token cost.
Consistency and durable changes
Source writes are journaled before application. The journal stores bounded exact staged bytes, preconditions, progress and results. A restart can distinguish already-applied writes by their hashes and resume safely. recover_operations is invoked by the CLI/watcher; embedded hosts can invoke it through the expert engine. An idempotency key reused with different content fails. A new key does not bypass create/update preconditions.
Batches are recoverable sequences, not multi-file filesystem transactions. External editors can race filesystem changes. A conflict leaves completed changes visible in results; resolve it explicitly instead of silently overwriting unrelated changes. Capture is fetched under the configured URL allowlist before the expanded changes are journaled, so an HTTP retry with an existing operation does not refetch and overwrite the retained capture. Remove a journaled capture with conditional removals of its page/HTML/assets; compatibility capture endpoints also support their checked-directory removal format.
States: applying, indexing, published, scan_failed, index_failed, conflict, superseded. resume recovers interrupted application/scans or requeues the caller's failed indexing jobs. Native model work already executing may finish after an HTTP deadline, but capacity remains bounded. Operation publication receipts are retained at the first matching publication even if later edits occur before the caller polls.
Publications select immutable artifacts. Retrieval copies a bounded authorized view before model awaits, including graph inputs. Reranking cannot splice new graph data into older hits. SQLite read snapshots alone would not provide this across a remote await or across filesystem mutations.
Model-space changes stage new artifacts while the old space serves. Cutover installs the complete target view transactionally after all desired sources are ready. If a target job fails, the old view remains available. Superseded writers cannot claim jobs for the replacement configuration. The old encoder must remain available for vector queries during migration; its recorded recipe is validated against current network policy. Retain old weights/provider revisions until cutover. Explicit shared-space IDs are operator assertions of compatibility, not automatic alignment of unrelated encoders.
Processing and retrieval
Each source records extraction, chunking, embedding and enrichment stage runs, recipe IDs and dependency/output identities. Decoder samples, transcripts, captions, embeddings and annotations are cached independently. Changing a transcription model does not require decoding unchanged video again. Changing a reranker does not invalidate content embeddings. Source-specific fingerprints avoid queuing text documents for unrelated ASR changes. Native weights/projectors are identified from bytes; file relocation and thread/GPU tuning do not redefine their semantic identity. Remote model revisions remain operator assertions.
Scans stage one file at a time into the archive, retaining only source metadata in the scan list. Failed scan budgets do not remove the previous index. File, folder, decoder-output, HTTP and query budgets still apply. A source is limited to 10,000 chunks across representations/referenced inputs; excessive expansion fails before publication rather than silently truncating. Moves require a supported source extension.
Markdown chunking recognizes structural boundaries; fixed/recursive/sentence strategies, subgroup rules and per-document overrides remain available. Optional chunking.max_tokens uses a conservative byte-token bound. Custom Chunker implementations can apply exact model tokenization and must retain exact original UTF-8 offsets.
The beta retrieval path uses exact cosine vectors from retained artifacts and rank fusion with BM25. Its temporary FTS view contains only authorized, filtered, pinned evidence, keeping hidden/historical documents out of IDF statistics. This costs CPU/memory per query; use the measured workload envelope below. Compatibility retrieval retains sqlite-vec/TurboQuant experiments. Compression is disabled in the primary sample and remains experimental.
Configure retrieval.max_snapshot_chunks (default 50,000) and max_snapshot_bytes (default 128 MiB). Exceeding these fails rather than silently truncating candidate coverage. These are accounted payload bounds, not an exact allocator/RSS limit. Filters run before candidate ranking and remote reranking. Compatible embedding spaces are queried with their matching encoder; scores across incompatible vectors are never directly compared. Hybrid mode reports unavailable vector channels; strict: true or vector-only retrieval fails instead of degrading.
Graph expansion applies typed predicate/direction and path budgets, then selects useful related passages by query coverage. Model-extracted names are scoped by concept and type. Same names in separate documents remain distinct; curated, evidence-backed same_as resolutions are versioned and reversible. Assertions preserve origin and review state; source-declared and model-extracted claims are not automatically authoritative. Global community summarization remains an explicitly separate potential processing recipe, not an always-on cost for ordinary lookup.
Library and resource ownership
Use the opaque Knowledge facade with typed contracts. Backends composes small embedding, annotation, vision, speech, answer and ranking traits; applications need only supply capabilities they use. Engine and Store remain expert integration surfaces. Reconfiguration means constructing a validated new handle and scanning; mutation of an expert Engine's public fields is not a coordinated live-reconfiguration API.
The host owns Tokio, logging, shutdown and task lifetime. reconcile moves scanning off an async host's executor thread; synchronous administrative methods should be called from a blocking worker in GUI applications. A trusted host can obtain expert access; restricted handles enforce their own scopes and do not expose configuration/secret references.
Configured GGUF roles load on demand. A missing optional generator does not disable lexical/catalog access. A native model provider serializes native work and retains one native role at a time, evicting unused roles before loading another. This trades reload latency for bounded residency on small Macs. Separate knowledge bases/provider objects have separate residency; embedding hosts must budget their aggregate memory. Index workers are serialized per Engine clone group; SQLite serializes publication writes. Model/decoder HTTP services can provide process isolation. In-process native calls are trusted native code and are not preemptively killable or sandboxed by SQLite.
Authorization and egress
Legacy read/admin secrets continue to work. Optional named principals have distinct secret references, capability lists and component-based path prefixes. They use the beta API; compatibility data routes are denied to custom scoped principals. No source metadata can grant capabilities or choose privileged model/network endpoints. Current configured credentials define revocation; change credentials/scopes and restart the service to apply a new policy. Knowledge does not implement a separate identity provider or per-request external policy reload.
server:
principals:
- name: research-agent
token: {env: KNOWLEDGE_RESEARCH_TOKEN}
capabilities: [read, write, capture]
prefixes: [agents/research]
Global admin/purge authority requires whole-bundle scope. Hex credentials are canonicalized so case changes cannot bypass secret-reuse checks. Prefixes govern reads, writes, graph context, historical evidence, operations and events. Historical scopes use the retained original path: a move alone does not revoke archival access. Change scopes or purge retained history when revocation is required. Moves preserve bytes and do not rewrite relative links; absolute bundle-relative OKF references are more stable. Source path controls and exact-origin URL allowlists remain enforced. Model egress uses the per-knowledge-base operator-configured security.network policy; URL capture has its separate allowlist. No automatic remote fallback is introduced. Protect host processes and source folders independently: OKF files are portable plaintext unless the filesystem itself is encrypted. SQLCipher covers the database/retained content, not plaintext authoring files, models, process memory, GPU memory or filesystem backups.
Migration, backup, restore and retention
Schema 3 upgrades through schema 4 and backfills active artifacts/evidence/vectors into the publication archive. The current binary adds consolidation tables in schema 5 and federation policies, projections, replica control and graph tables in schema 6. Before migration from an older version, a consistent private *.pre-beta-v<old-version>-<random>.sqlite backup is created using the current encryption key. Newer unsupported schemas are rejected. Opening with an older binary after migration is unsupported; restore the backup instead.
knowledge doctor
knowledge backup /absolute/private/backup.sqlite
knowledge restore --help
knowledge rebuild
knowledge retain --publications 10 # dry run
knowledge retain --publications 10 --apply
knowledge gc # dry run
knowledge gc --apply
knowledge check
restore DATABASE ROOT requires a new database path and an existing empty destination root. It copies the database with the same SQLCipher key, materializes observed source revisions, explicitly rebinds the root and retains bundle/evidence IDs. Pending journal applications are reset for recovery. Retain/provide the corresponding key separately; a database backup is not a key backup. Unobserved external edits are not part of a database snapshot.
Retention preserves active/desired versions, retained publications, operation receipts and current resolution/proof roots. Old citation availability is subject to that explicit policy. Pause indexing before applying GC; it refuses to race a running job. GC clears rebuildable caches and unreferenced decoder/URL snapshots, while retained evidence roots associated bytes. It does not shrink the allocated SQLite file automatically; backup and perform maintenance in a controlled window. Events and operation receipts are retained in this beta, so retention can preserve more history than the requested publication count. Diagnose disk usage instead of assuming that count is a strict storage quota.
Diagnostics use bounded, capability-scoped reads and distinguish existing binary references from missing concepts. Strict scans reject an unsupported root format before queuing any source. check diagnoses supported OKF version, required frontmatter, reserved navigation files, dangling links and freshness. Unknown producer types/fields remain accepted. Broken links are advisory, as OKF allows them. Executable attestation metadata is preserved but never automatically executed. Strict ingestion's root-index requirement is an application profile, not a universal OKF requirement.
Validation and beta boundaries
See beta validation for actual commands/results, raw HTTP responses, native embedding evidence, retrieval metrics and limitations. The gold corpus is small and synthetic; perfect scores on it are not a production-quality estimate. Exact vector retrieval and authorized-view FTS construction prioritize correctness within declared resource budgets. Large-corpus ANN, GPU worker isolation, global summary generation and production-scale load qualification require separate measured deployment work. Linux/Windows/CUDA/Vulkan runtime execution is not claimed from an M3 test run; the portable CI matrix remains supplied for those environments.