Run Knowledge Federation
Knowledge 0.3.0-beta.3 can query and selectively replicate a Share from another instance. Both instances retain their own OKF bundle, SQLite databases, model configuration and credentials. The owner controls the Share; the consumer controls which local principals can use a Mount. The default mode is live retrieval. A replica requires an explicit owner grant and a SQLCipher build. An independent copy is a separate export operation.
sequenceDiagram
participant Agent as App or MCP gateway
participant Bob as Consumer knowledge
participant Cache as Encrypted replica
participant Alice as Owner knowledge
Agent->>Bob: Workspace, query, allowed sources and optional pins
Bob->>Bob: Check caller, mount audience and query egress
alt Live mount
Bob->>Alice: TLS request to enrolled Share
Alice->>Alice: Check grant, project current OKF evidence, rank
Alice-->>Bob: Signed evidence and view receipt
else Replica mount
Bob->>Alice: Check the current grant and serving revision
Alice-->>Bob: Signed authorization receipt
Bob->>Cache: BM25 and compatible local vectors
Cache-->>Bob: Ranked evidence with original citations
end
Bob->>Bob: Combine ranks by lineage, add permitted graph context
Bob-->>Agent: Cited context, snapshot vector and source coverage
Start two local instances
The owner and consumer profiles use separate fictional Alice and Bob bundles. Alice's private subtree and bundle navigation are not published. The profiles start with BM25 and graph retrieval; add either instance's own models.embedding block for vector retrieval. No model download or external provider is needed for the example.
From the repository directory, build knowledge with server,sqlcipher enabled. Provision credentials into fresh directories:
mkdir -p .knowledge-federation
knowledge -c configs/federation-owner.yml init-security --output .knowledge-federation/alice
knowledge -c configs/federation-consumer.yml init-security --output .knowledge-federation/bob
knowledge federation init-key .knowledge-federation/alice/secrets/publisher.key
(umask 077; openssl rand -hex 32 > .knowledge-federation/alice/secrets/bob-peer.token)
(umask 077; cp .knowledge-federation/alice/secrets/bob-peer.token .knowledge-federation/bob/secrets/alice-peer.token)
The examples refer to the files produced by init-security. Run the checked profiles, rather than the separate knowledge.secure.yml files the command emits. The token copy is a shortcut for this same-host demo; exchange a real peer credential through a secure channel. For another host, install a certificate for its DNS name and keep its private key on the owner.
Run knowledge -c configs/federation-owner.yml federation identity. In a local copy of the consumer profile in the same configs directory, replace publisher_id and public_key with the two values you verified. The Ed25519 public key is separate from the CA certificate. /.well-known/knowledge is a discovery endpoint, not automatic trust enrollment.
knowledge -c configs/federation-owner.yml index
knowledge -c configs/federation-consumer.yml index
# Keep these running in separate terminals:
knowledge -c configs/federation-owner.yml serve
knowledge -c configs/federation-consumer.yml serve
# The CLI can also query Bob without his HTTP service:
knowledge -c configs/federation-consumer.yml federation retrieve configs/federation-query.yml
For an application, POST the equivalent JSON to Bob's /v1/federation/retrieve or wrap it as {"search": ..., "context": {"max_tokens":4096}} for /v1/federation/context. Bob's read-token is on the lab mount audience. Alice's bob-peer token is for Alice's Share routes and cannot read Alice's normal catalog. Neither token becomes an administrator on the other side.
Choose what travels
| Control | Effect |
|---|---|
include, exclude |
Complete OKF path prefixes. Use include: [""] only when intentionally considering the whole bundle. |
metadata, metadata_fields |
Exact JSON-pointer predicates narrow membership, e.g. "/type": Procedure. Only listed fields can be projected or queried. Source tags never grant permissions. |
representations |
Exact representation or family prefix, e.g. transcript includes timestamped transcript chunks. Default text families exclude native media vectors and raw originals. |
pages, time_ranges |
Path-specific page numbers and fully contained millisecond intervals. A chunk that crosses a boundary is omitted. |
permissions |
Share and Grant must both allow the action. Search returns authorized excerpts; catalog and citation lookup require read_evidence. |
download_assets, original_assets |
Grant access to associated binary samples, and separately to full originals. Originals cannot be enabled with partial page/time scope. |
replicate, export |
Permit a managed replica or an independent copy, respectively. One does not imply the other. |
processing |
local_only allows in-process and loopback models. configured_providers also requires a matching HTTPS origin in allowed_model_origins for remote processing. The consumer can further restrict egress. |
derive, reshare, reshare_publishers |
Opt into controlled foreign consolidation and explicit onward copy to named recipient publishers. |
history, history_views |
Optional bounded history within the same policy epoch. Historical reads still intersect current membership. Default retains the current view only. |
max_lease_secs, max_offline_secs |
Owner bounds for online authorization and cooperative offline use. Offline is zero by default. |
Use separate Shares when different recipients need different content scopes. Policies restrict projected representations; they do not redact sentences inside an allowed excerpt. Keep sensitive text out of the source representation or author a separate sanitized page. Navigation files, private asset URLs, unapproved graph endpoints and automatically imported/forked excerpts are omitted from Share projection. A local promoted consolidation is eligible only if its current parents also fit the Share.
Owner changes can be installed through federation share policy.yml and federation grant policy.yml, or the admin PUT routes. enabled: false revokes. File-managed policy changes reconcile when the engine opens the config; removing a file-managed entry disables it. An unchanged YAML entry preserves a persisted API edit. Peers and mounts are operator configuration: changing trusted URLs, keys or audience requires reopening the config. Do not source these values from OKF metadata or an agent prompt.
Retrieve, cite and traverse
The planner limits source fan-out, concurrency, response bytes, candidates and wall time. It prefers configured replicas, then mount weight and stable alias order. Empty sources selects eligible mounts in the named workspace; explicit aliases are validated. required_sources, strict and optional pins let callers reject partial coverage. A pin contains publisher, grant, exported revision and policy epoch; a changed source fails the pin instead of silently serving different content. A snapshot vector is a set of independent observations, not a transaction across owners.
Each source ranks its own compatible BM25/vector channels. The consumer compares reciprocal rank positions with configurable mount weights. It caps the contribution of copies sharing a collection lineage, representation and artifact text/hash; the same document visible through live and replica mounts does not get two votes. Separate independent collections stay separate even when their text matches. coverage reranks by query term overlap. Federated mmr uses a common local term space to diversify the shortlist; http uses one configured reranker only if each contributing Share permits that endpoint. No peer embedding distances are compared across model spaces.
coverage.selected, completed, unavailable and omitted_by_budget tell the application what was searched. A failed peer never triggers implicit egress or a recursive search. A mounted evidence locator includes its typed origin, mount, exported revision and policy epoch. Resolve through /v1/federation/evidence with {workspace, reference}; pass a mounted reference containing the returned mount and origin. Use /v1/federation/asset for bounded authorized ranges. A raw knowledge:// citation is an identifier; it contains no peer credential or host address.
Local bridge assertions connect explicitly mounted or local evidence references. A bridge has a predicate, a proof reference and a workspace. Install one with federation bridge bridge.yml or the admin bridge endpoint. Traversal rechecks both endpoints and the proof, honors graph permission and caller audience, and stays within the original query's deadline. It visits known nodes and does not ask peers to route onward. Peer graphs preserve attributable evidence references; they do not merge two entities just because names match.
Replicate, pause and revoke
If you change the publisher's or consumer's model, plan for Share coverage and replica vector backfill. Model changes in federation
Set mode: replica with the exact grant_id; set query_egress: false to keep replica searches off the owner's search endpoint. Online replica retrieval still sends a small lease check. It does not send the query text to the owner. knowledge serve and knowledge watch run synchronization automatically, independently of the OKF watcher. Embedders can run Knowledge::federation_until with their own cancellation token.
knowledge -c configs/federation-consumer.yml federation sync alice-orion
knowledge -c configs/federation-consumer.yml federation status
For recovery tests or operational control, run stage and publish separately. Stage verifies a signed Share inventory and pages in only new or changed evidence, reusing locally verified bodies. An unchanged revision can renew through the scoped change feed without fetching bodies. If a cursor is compacted, rotated or from a different epoch, the consumer reconciles against the scoped inventory. Staging, transfer bandwidth, batch size, per-partition vector bytes and deadline are bounded. Staging does not start a model. Publishing uses the existing background governor, batches text embeddings and publishes FTS and vectors in one SQLite transaction. It can defer when indexing is paused or at its admission limit. Unchanged compatible embeddings are reused.
Partitions are encrypted with separate keys under the database's federation-cache directory. The primary database contains policy/control metadata rather than replica bodies. A replica stores the exported text and permitted metadata, never a peer SQLite database. It recomputes vectors with the consumer's exact configured embedding space. Text representations of PDFs, images, audio and video are searchable; native media vector search is provided by the owner in live mode. Replica indexing does not automatically fetch large media or follow URLs. A media asset can be resolved on demand only with the appropriate current grant.
Received and serving checkpoints are separate. Known scope changes, revocations and mismatched revisions close serving eligibility before backfill. The worker checks leases separately from model work. A crash between stage and publish leaves a resumable encrypted stage. Online requests fail when the current grant does not match the serving revision. Offline serving requires both the owner's and mount's finite limits, a previously observed online authorization and a process-local monotonic deadline. A restart requires a new online check. A policy denial, bad signature or certificate validation error does not authorize offline fallback.
There is no recall of already returned or independently exported bytes. Ciphertext/WAL/backup retention and physical erasure are separate operator controls. A cooperative replica hides known revoked data; do not distribute its encryption key with a shared backup.
Consolidate and export deliberately
Foreign derivation is off by default. Enable ordinary consolidation, federation.consolidation.enabled, its explicit workspaces, and allow_derivation on each participating mount. The Share and Grant must both allow derive. federation derive request.yml takes a workspace and bounded typed evidence parents. The output starts as a proposal, records source hashes/view pins and validates claim support against exact excerpts. Approve the output hash with review-derivation; approved digests stay in encrypted managed storage. Each read rechecks all parents. include_derived: true adds an approved digest to RAG context only when all its parents are in the current query's authorized evidence. No background or ordinary OKF publication grants onward access. Explicit resharing requires reshare, export and the named recipient in every origin's allowlist.
knowledge federation export <mount> <new-folder> [--assets] requires the owner's export grant and whole-bundle local authority. It creates an independent OKF folder with fresh navigation, quoted excerpts, permitted graph/asset links and namespaced origin metadata. No old navigation, executable metadata or arbitrary external resource fetch is carried over. --assets downloads only granted assets and verifies their full content hashes. fork also creates a new Ed25519 signing key at --signing-key <new-file>; point a new database at the exported OKF root and use the reported collection_lineage to preserve common lineage. Keep the signing key outside the OKF folder. Restoration as the same authority uses the original enrolled key and branch; key rotation requires explicit re-enrollment.
The protocol and storage reference, OpenAPI 3.2.1 contract, Rust facade, authentication guide and validation report describe the supported interfaces and measured limits. The MCP gateway remains external and must preserve the caller's authorization and mount scope.