Embed knowledge in Rust

Beta.2 adds Knowledge Consolidation and indexing resource budgets; see those guides for their complete YAML, API/library contracts and boundaries.

Knowledge is an opaque facade over the same core used by the OpenAPI service. Engine and Store remain explicit expert interfaces. Opening a library handle starts no HTTP listener, signal handler, watcher or global logging configuration. The host owns Tokio, task lifetime, logging and shutdown.

[dependencies]
knowledge = { path = "/path/to/knowledge", default-features = false, features = ["sqlcipher"] }
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }

Use no native feature for HTTP models or lexical-only access. Add embedded-gguf,multimodal for native CPU inference, plus metal, cuda or vulkan for one configured accelerator. server is optional. This crate is not published to crates.io and provides no C/Swift/JNI ABI.

use knowledge::{Knowledge, Result, contracts::{ContextQuery, Query}, search::SearchMode};

async fn example() -> Result<()> {
    let kb = Knowledge::from_file("knowledge.yml")?;
    kb.reconcile().await?;
    kb.drain().await?;
    let context = kb.context(ContextQuery {
        search: Query {
            query: "How often is the sensor calibrated?".into(),
            mode: SearchMode::Hybrid,
            ..Default::default()
        },
        ..Default::default()
    }).await?;
    println!("{}", context.text); // Pass this context/citations to the host's agent or model.
    Ok(())
}

context_with takes a TokenCounter for the consumer model. Default context uses a conservative UTF-8 byte budget and declares that accounting. Results include immutable evidence IDs, publication keys, exact source/representation provenance and freshness. select, evidence, events, changes and operation provide deterministic access and durable authoring. Prefer a publication_key when persisting a snapshot reference: numeric IDs are convenient within one database timeline, while keys avoid collisions across restore/fork histories.

Durable source-first authoring

use knowledge::{Knowledge, Result, operations::{Changes, Change}, authoring::PageInput};
use serde_json::json;

async fn save(kb: &Knowledge) -> Result<()> {
    let receipt = kb.changes(Changes {
        idempotency_key: "inspection-note-001".into(),
        changes: vec![Change::Page(PageInput {
            path: "notes/maintenance.md".into(),
            content: "Inspect the seal every month.".into(),
            metadata: json!({"type":"Procedure","title":"Seal inspection"}),
            expected_sha256: None,
        })],
    }).await?;
    kb.drain().await?;
    let operation = kb.operation(&receipt.id)?;
    assert_eq!(operation.state, "published");
    Ok(())
}

Updates/removals need current SHA-256 preconditions. Change::Capture prepares allowlisted URL content/assets, then journals the expanded OKF writes. Batches are recoverable sequences, not filesystem transactions. restricted(Access) creates a narrower facade for a trusted host to hand to its components; it does not create an OS sandbox. Source metadata cannot grant privileges. The trusted host can request source_writer()? for low-level writes or engine()? for expert integration; those paths require administrative access. server.allow_source_writes gates HTTP, not trusted host code.

Custom models and chunkers

Knowledge::new(config, Arc<dyn Models>) injects a backend. backends::Backends composes independent Embeddings, Annotation, Vision, Speech, Answers and Ranking interfaces. Uninstalled capabilities return explicit errors. Native built-ins load roles on demand and serialize/evict native roles to limit residency. See beta resource/processing semantics.

Engine::with_chunker installs a custom tokenizer-aware or domain chunker. Chunks must preserve exact UTF-8 source ranges/text. Use Knowledge::new for ordinary integrations; use Engine directly when orchestrating a custom chunker. Model semantic changes require versioned identities. Native identities include weight/projector bytes; remote revisions remain operator assertions.

Lifecycle and recovery

reconcile dispatches scanning to a blocking worker. Synchronous administrative methods (backup, rebuild, diagnostics, retention) should be called from a blocking host task when used by a GUI. Retrieval snapshot construction also performs bounded synchronous SQLite work; run the facade on a worker runtime rather than a GUI event loop. drain processes currently available jobs; delayed retries still need a worker/watch loop. Configured writers are fenced when a different processing configuration supersedes them. Deploy new configuration with a new validated handle, retain the serving encoder during shadow migration, and stop the superseded watcher.

async fn watch(kb: knowledge::Knowledge, stop: tokio::sync::oneshot::Receiver<()>) -> knowledge::Result<()> {
    let engine = kb.engine()?.clone();
    knowledge::watch::watch_until(engine, false, async { let _ = stop.await; }).await
}

watch_until is host-controlled; the CLI watch helper owns Ctrl-C handling. Native C/C++ inference is not preemptively killable: cancellation/deadlines can leave already executing work running until completion, with permits retained. Journal/lease recovery handles interrupted work. Clones share provider handles and local locks; independent processes do not share the filesystem lock.

Run the typed beta example:

cargo run --no-default-features --example beta_library

The earlier low-level writer example remains available. See validation for regression tests and a separate consumer-crate check. The beta guide covers backup/restore, retention, publication consistency and security boundaries.

Federated retrieval from an embedded host

Open a config with enrolled mounts. Hand a restricted handle to each caller; the mount audience and ordinary local path permissions are evaluated independently. A Knowledge handle does not start a network listener or sync worker by itself.

# async fn example() -> knowledge::Result<()> {
use knowledge::{Knowledge, contracts::{Access, Query}, federation::FederatedQuery};
let host = Knowledge::from_file("consumer.yml")?;
let agent = host.restricted(Access {
    principal: "research-agent".into(),
    capabilities: vec!["read".into()],
    prefixes: vec!["notes".into()],
    admin: false,
})?;
let result = agent.federated_retrieve(FederatedQuery {
    workspace: "lab".into(),
    sources: vec!["alice-orion".into()],
    required_sources: vec!["alice-orion".into()],
    query: Query { query: "Orion calibration interval".into(), ..Default::default() },
    ..Default::default()
}).await?;
let _ = result.snapshots;
# Ok(()) }

federated_context packs evidence and graph context. federation_evidence and federation_asset re-authorize typed references. Whole-bundle hosts can stage, publish, synchronize or export a mount, maintain bridge assertions, and run foreign consolidation and review. Run federation_until(cancel) in a host-owned task for scheduled synchronization. Replica and foreign derivation storage requires sqlcipher; a consumer may use live mounts with neither server nor native model features. See federation configuration.

Read Markdown →