# Embed knowledge in Rust

Beta.2 adds [Knowledge Consolidation](https://tsok.org/knowledge/docs/consolidation/) and [indexing resource budgets](https://tsok.org/knowledge/docs/indexing-resources/); see those guides for their complete YAML, API/library contracts and boundaries.

`Knowledge` is an opaque facade over the same core used by the OpenAPI service. `Engine` and `Store` remain explicit expert interfaces. Opening a library handle starts no HTTP listener, signal handler, watcher or global logging configuration. The host owns Tokio, task lifetime, logging and shutdown.

```toml
[dependencies]
knowledge = { path = "/path/to/knowledge", default-features = false, features = ["sqlcipher"] }
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
```

Use no native feature for HTTP models or lexical-only access. Add `embedded-gguf,multimodal` for native CPU inference, plus `metal`, `cuda` or `vulkan` for one configured accelerator. `server` is optional. This crate is not published to crates.io and provides no C/Swift/JNI ABI.

```rust,no_run
use knowledge::{Knowledge, Result, contracts::{ContextQuery, Query}, search::SearchMode};

async fn example() -> Result<()> {
    let kb = Knowledge::from_file("knowledge.yml")?;
    kb.reconcile().await?;
    kb.drain().await?;
    let context = kb.context(ContextQuery {
        search: Query {
            query: "How often is the sensor calibrated?".into(),
            mode: SearchMode::Hybrid,
            ..Default::default()
        },
        ..Default::default()
    }).await?;
    println!("{}", context.text); // Pass this context/citations to the host's agent or model.
    Ok(())
}
```

`context_with` takes a `TokenCounter` for the consumer model. Default context uses a conservative UTF-8 byte budget and declares that accounting. Results include immutable evidence IDs, publication keys, exact source/representation provenance and freshness. `select`, `evidence`, `events`, `changes` and `operation` provide deterministic access and durable authoring. Prefer a `publication_key` when persisting a snapshot reference: numeric IDs are convenient within one database timeline, while keys avoid collisions across restore/fork histories.

## Durable source-first authoring

```rust,no_run
use knowledge::{Knowledge, Result, operations::{Changes, Change}, authoring::PageInput};
use serde_json::json;

async fn save(kb: &Knowledge) -> Result<()> {
    let receipt = kb.changes(Changes {
        idempotency_key: "inspection-note-001".into(),
        changes: vec![Change::Page(PageInput {
            path: "notes/maintenance.md".into(),
            content: "Inspect the seal every month.".into(),
            metadata: json!({"type":"Procedure","title":"Seal inspection"}),
            expected_sha256: None,
        })],
    }).await?;
    kb.drain().await?;
    let operation = kb.operation(&receipt.id)?;
    assert_eq!(operation.state, "published");
    Ok(())
}
```

Updates/removals need current SHA-256 preconditions. `Change::Capture` prepares allowlisted URL content/assets, then journals the expanded OKF writes. Batches are recoverable sequences, not filesystem transactions. `restricted(Access)` creates a narrower facade for a trusted host to hand to its components; it does not create an OS sandbox. Source metadata cannot grant privileges. The trusted host can request `source_writer()?` for low-level writes or `engine()?` for expert integration; those paths require administrative access. `server.allow_source_writes` gates HTTP, not trusted host code.

## Custom models and chunkers

`Knowledge::new(config, Arc<dyn Models>)` injects a backend. `backends::Backends` composes independent `Embeddings`, `Annotation`, `Vision`, `Speech`, `Answers` and `Ranking` interfaces. Uninstalled capabilities return explicit errors. Native built-ins load roles on demand and serialize/evict native roles to limit residency. See [beta resource/processing semantics](https://tsok.org/knowledge/docs/retrieval/).

`Engine::with_chunker` installs a custom tokenizer-aware or domain chunker. Chunks must preserve exact UTF-8 source ranges/text. Use `Knowledge::new` for ordinary integrations; use Engine directly when orchestrating a custom chunker. Model semantic changes require versioned identities. Native identities include weight/projector bytes; remote revisions remain operator assertions.

## Lifecycle and recovery

`reconcile` dispatches scanning to a blocking worker. Synchronous administrative methods (`backup`, `rebuild`, diagnostics, retention) should be called from a blocking host task when used by a GUI. Retrieval snapshot construction also performs bounded synchronous SQLite work; run the facade on a worker runtime rather than a GUI event loop. `drain` processes currently available jobs; delayed retries still need a worker/watch loop. Configured writers are fenced when a different processing configuration supersedes them. Deploy new configuration with a new validated handle, retain the serving encoder during shadow migration, and stop the superseded watcher.

```rust,no_run
async fn watch(kb: knowledge::Knowledge, stop: tokio::sync::oneshot::Receiver<()>) -> knowledge::Result<()> {
    let engine = kb.engine()?.clone();
    knowledge::watch::watch_until(engine, false, async { let _ = stop.await; }).await
}
```

`watch_until` is host-controlled; the CLI `watch` helper owns Ctrl-C handling. Native C/C++ inference is not preemptively killable: cancellation/deadlines can leave already executing work running until completion, with permits retained. Journal/lease recovery handles interrupted work. Clones share provider handles and local locks; independent processes do not share the filesystem lock.

Run [the typed beta example](https://github.com/tsok-org/source-code/blob/main/apps/knowledge/examples/beta_library.rs):

```sh
cargo run --no-default-features --example beta_library
```

The earlier [low-level writer example](https://github.com/tsok-org/source-code/blob/main/apps/knowledge/examples/embedded_library.rs) remains available. See [validation](https://github.com/tsok-org/source-code/blob/main/apps/knowledge/reports/beta/VALIDATION.md) for regression tests and a separate consumer-crate check. [The beta guide](https://tsok.org/knowledge/docs/retrieval/) covers backup/restore, retention, publication consistency and security boundaries.

## Federated retrieval from an embedded host

Open a config with enrolled mounts. Hand a restricted handle to each caller; the mount audience and ordinary local path permissions are evaluated independently. A `Knowledge` handle does not start a network listener or sync worker by itself.

```rust,no_run
# async fn example() -> knowledge::Result<()> {
use knowledge::{Knowledge, contracts::{Access, Query}, federation::FederatedQuery};
let host = Knowledge::from_file("consumer.yml")?;
let agent = host.restricted(Access {
    principal: "research-agent".into(),
    capabilities: vec!["read".into()],
    prefixes: vec!["notes".into()],
    admin: false,
})?;
let result = agent.federated_retrieve(FederatedQuery {
    workspace: "lab".into(),
    sources: vec!["alice-orion".into()],
    required_sources: vec!["alice-orion".into()],
    query: Query { query: "Orion calibration interval".into(), ..Default::default() },
    ..Default::default()
}).await?;
let _ = result.snapshots;
# Ok(()) }
```

`federated_context` packs evidence and graph context. `federation_evidence` and `federation_asset` re-authorize typed references. Whole-bundle hosts can stage, publish, synchronize or export a mount, maintain bridge assertions, and run foreign consolidation and review. Run `federation_until(cancel)` in a host-owned task for scheduled synchronization. Replica and foreign derivation storage requires `sqlcipher`; a consumer may use live mounts with neither `server` nor native model features. See [federation configuration](https://tsok.org/knowledge/docs/federation/).
