Embed knowledge in Rust
Beta.2 adds Knowledge Consolidation and indexing resource budgets; see those guides for their complete YAML, API/library contracts and boundaries.
Knowledge is an opaque facade over the same core used by the OpenAPI service. Engine and Store remain explicit expert interfaces. Opening a library handle starts no HTTP listener, signal handler, watcher or global logging configuration. The host owns Tokio, task lifetime, logging and shutdown.
[dependencies]
knowledge = { path = "/path/to/knowledge", default-features = false, features = ["sqlcipher"] }
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
Use no native feature for HTTP models or lexical-only access. Add embedded-gguf,multimodal for native CPU inference, plus metal, cuda or vulkan for one configured accelerator. server is optional. This crate is not published to crates.io and provides no C/Swift/JNI ABI.
use knowledge::{Knowledge, Result, contracts::{ContextQuery, Query}, search::SearchMode};
async fn example() -> Result<()> {
let kb = Knowledge::from_file("knowledge.yml")?;
kb.reconcile().await?;
kb.drain().await?;
let context = kb.context(ContextQuery {
search: Query {
query: "How often is the sensor calibrated?".into(),
mode: SearchMode::Hybrid,
..Default::default()
},
..Default::default()
}).await?;
println!("{}", context.text); // Pass this context/citations to the host's agent or model.
Ok(())
}
context_with takes a TokenCounter for the consumer model. Default context uses a conservative UTF-8 byte budget and declares that accounting. Results include immutable evidence IDs, publication keys, exact source/representation provenance and freshness. select, evidence, events, changes and operation provide deterministic access and durable authoring. Prefer a publication_key when persisting a snapshot reference: numeric IDs are convenient within one database timeline, while keys avoid collisions across restore/fork histories.
Durable source-first authoring
use knowledge::{Knowledge, Result, operations::{Changes, Change}, authoring::PageInput};
use serde_json::json;
async fn save(kb: &Knowledge) -> Result<()> {
let receipt = kb.changes(Changes {
idempotency_key: "inspection-note-001".into(),
changes: vec![Change::Page(PageInput {
path: "notes/maintenance.md".into(),
content: "Inspect the seal every month.".into(),
metadata: json!({"type":"Procedure","title":"Seal inspection"}),
expected_sha256: None,
})],
}).await?;
kb.drain().await?;
let operation = kb.operation(&receipt.id)?;
assert_eq!(operation.state, "published");
Ok(())
}
Updates/removals need current SHA-256 preconditions. Change::Capture prepares allowlisted URL content/assets, then journals the expanded OKF writes. Batches are recoverable sequences, not filesystem transactions. restricted(Access) creates a narrower facade for a trusted host to hand to its components; it does not create an OS sandbox. Source metadata cannot grant privileges. The trusted host can request source_writer()? for low-level writes or engine()? for expert integration; those paths require administrative access. server.allow_source_writes gates HTTP, not trusted host code.
Custom models and chunkers
Knowledge::new(config, Arc<dyn Models>) injects a backend. backends::Backends composes independent Embeddings, Annotation, Vision, Speech, Answers and Ranking interfaces. Uninstalled capabilities return explicit errors. Native built-ins load roles on demand and serialize/evict native roles to limit residency. See beta resource/processing semantics.
Engine::with_chunker installs a custom tokenizer-aware or domain chunker. Chunks must preserve exact UTF-8 source ranges/text. Use Knowledge::new for ordinary integrations; use Engine directly when orchestrating a custom chunker. Model semantic changes require versioned identities. Native identities include weight/projector bytes; remote revisions remain operator assertions.
Lifecycle and recovery
reconcile dispatches scanning to a blocking worker. Synchronous administrative methods (backup, rebuild, diagnostics, retention) should be called from a blocking host task when used by a GUI. Retrieval snapshot construction also performs bounded synchronous SQLite work; run the facade on a worker runtime rather than a GUI event loop. drain processes currently available jobs; delayed retries still need a worker/watch loop. Configured writers are fenced when a different processing configuration supersedes them. Deploy new configuration with a new validated handle, retain the serving encoder during shadow migration, and stop the superseded watcher.
async fn watch(kb: knowledge::Knowledge, stop: tokio::sync::oneshot::Receiver<()>) -> knowledge::Result<()> {
let engine = kb.engine()?.clone();
knowledge::watch::watch_until(engine, false, async { let _ = stop.await; }).await
}
watch_until is host-controlled; the CLI watch helper owns Ctrl-C handling. Native C/C++ inference is not preemptively killable: cancellation/deadlines can leave already executing work running until completion, with permits retained. Journal/lease recovery handles interrupted work. Clones share provider handles and local locks; independent processes do not share the filesystem lock.
cargo run --no-default-features --example beta_library
The earlier low-level writer example remains available. See validation for regression tests and a separate consumer-crate check. The beta guide covers backup/restore, retention, publication consistency and security boundaries.
Federated retrieval from an embedded host
Open a config with enrolled mounts. Hand a restricted handle to each caller; the mount audience and ordinary local path permissions are evaluated independently. A Knowledge handle does not start a network listener or sync worker by itself.
# async fn example() -> knowledge::Result<()> {
use knowledge::{Knowledge, contracts::{Access, Query}, federation::FederatedQuery};
let host = Knowledge::from_file("consumer.yml")?;
let agent = host.restricted(Access {
principal: "research-agent".into(),
capabilities: vec!["read".into()],
prefixes: vec!["notes".into()],
admin: false,
})?;
let result = agent.federated_retrieve(FederatedQuery {
workspace: "lab".into(),
sources: vec!["alice-orion".into()],
required_sources: vec!["alice-orion".into()],
query: Query { query: "Orion calibration interval".into(), ..Default::default() },
..Default::default()
}).await?;
let _ = result.snapshots;
# Ok(()) }
federated_context packs evidence and graph context. federation_evidence and federation_asset re-authorize typed references. Whole-bundle hosts can stage, publish, synchronize or export a mount, maintain bridge assertions, and run foreign consolidation and review. Run federation_until(cancel) in a host-owned task for scheduled synchronization. Replica and foreign derivation storage requires sqlcipher; a consumer may use live mounts with neither server nor native model features. See federation configuration.