Security in knowledge
Beta.2 adds Knowledge Consolidation and indexing resource budgets; see those guides for their complete YAML, API/library contracts and boundaries.
The first beta guide documents the current evidence/publication, operation, scoped-access and runtime contracts. It supplements the compatibility interfaces described below.
knowledge 0.2 adds an authenticated HTTPS service and encrypted SQLite storage around the Open Knowledge Format ingestion/retrieval application. The default knowledge.yml uses these controls. Earlier okf.*.json files and benchmark databases remain as legacy sample/test configurations; they have not been silently encrypted or removed.
The source of authored knowledge is still an OKF folder, with a root index.md, Markdown concept files, YAML frontmatter, links, and referenced assets. SQLite stores the original source versions and derived indexes. The API queries this index. Authenticated admin authoring can write supported data inside the configured OKF root; capture URLs must satisfy the configured fetch policy. Requests cannot select another root, execute SQL or override model endpoints.
flowchart LR
F[OKF folder on protected volume] --> W[Bounded capability-based scanner]
W --> P[Parse metadata and chunk content]
P --> M[Native embedding model]
M --> C[Float32 vectors and optional TurboQuant codes]
C --> D[(SQLCipher database)]
P --> D
D --> A[Authenticated knowledge API]
A --> T[TLS 1.3]
T --> U[Client with read or admin token]
Start the service
This workspace has a generated .knowledge-secure/ profile. On a fresh checkout, initialize it once:
python3 scripts/fetch-model.py
sh scripts/build-mac.sh
knowledge init-security
knowledge index
knowledge serve
init-security refuses to overwrite an existing output directory or credentials. It creates independent random 256-bit database/read/admin secrets, a local CA certificate, a 90-day localhost server certificate/key, protected client files, and derived .knowledge-secure/knowledge.secure.yml and compatibility .json configurations. The default knowledge.yml references this private directory and an encrypted database there. No OS trust-store settings are changed; no service is installed or left running by setup.
The default endpoint is https://localhost:8443. serve also runs the OKF watcher and persistent job worker. Ctrl-C and Unix SIGTERM start graceful shutdown. server.watch=false serves an already-maintained index without starting ingestion. The API binds loopback by default. LAN deployment requires explicitly configuring the listen address, exact allowed hosts, a certificate valid for the intended hostname, and client trust.
Run knowledge check-server under the service account to verify the resolved config, private key permissions, TLS key pair, and distinct bearer, database and enabled MCP secrets. The command leaves SQLite closed. A successful authenticated HTTPS /healthz request verifies the listener and its database connection.
The connection cap applies before TLS negotiation and holds for the lifetime of the socket, including SSE. HTTP header and request deadlines apply separately. Public and rejected requests have their own rate budget, so exhausting it does not spend authenticated request tokens. OAuth attempts use a separate rate and concurrency budget. Duplicate Host or Origin values and ambiguous preflight headers fail closed. TLS deployments require exact HTTPS browser origins. Explicit plaintext loopback development mode also accepts an exact loopback HTTP origin.
A client must support TLS 1.3 and trust the local CA. Modern SDKs/generated clients can use .knowledge-secure/secrets/ca.pem. Some macOS system Python/curl builds do not support TLS 1.3; do not bypass validation or lower server requirements to make those clients work. Our real TLS smoke test uses Node's current TLS implementation; the Rust tests use reqwest/rustls.
For a curl build with TLS 1.3 support, generated client configuration keeps bearer values out of command arguments:
curl --config .knowledge-secure/secrets/read.curl https://localhost:8443/v1/bundle
curl --config .knowledge-secure/secrets/read.curl https://localhost:8443/openapi.json
curl --config .knowledge-secure/secrets/read.curl \
--header 'Content-Type: application/json' \
--data '{"query":"sensor calibration","mode":"hybrid","vector_backend":"turboquant"}' \
https://localhost:8443/v1/search
curl --config .knowledge-secure/secrets/admin.curl --request POST \
https://localhost:8443/v1/admin/reconcile
Do not use -k/insecure certificate verification. Client configuration files contain credentials and are private files too.
API and authorization
The OpenAPI document uses OpenAPI 3.2.1, checked against the official current 3.2 schema. It is compiled into the executable; startup does not fetch docs or scripts from a CDN. knowledge openapi prints it without opening databases, loading models or reading keys. /openapi.json serves it behind authentication, with the actual listener URL.
| Endpoint | Role | Purpose |
|---|---|---|
GET /healthz |
Read/admin | Readiness and database encryption state |
GET /openapi.json |
Read/admin | OpenAPI contract |
GET /v1/bundle |
Read/admin | Explicit OKF v0.2 format and relative indexed files |
GET /v1/status |
Read/admin | Counters; internal error details removed |
GET /v1/documents?path=… |
Read/admin | Parsed metadata/chunks; internal processing config removed |
GET /v1/graph?from=…&hops=… |
Read/admin | Graph nodes, edges and provenance |
POST /v1/search |
Read/admin | Lexical/vector/hybrid retrieval and reranking |
POST /v1/context |
Read/admin | Context with citation mappings |
POST /v1/ask |
Read/admin | Optional generation using retrieved context |
POST /v1/admin/reconcile |
Admin | Scan the configured OKF folder and queue changes |
POST /v1/admin/retry |
Admin | Requeue failed jobs |
Search requests use the fields in OpenAPI. Context/answer bodies wrap search options:
{"search":{"query":"Where is the backup drive?","mode":"hybrid","graph_hops":2},"max_chars":6000}
Every data route requires one bearer header. Query-string tokens and cookies are not authentication mechanisms. Read tokens cannot perform administrative writes. The admin token also has read access. Secret values are hashed for comparison with constant-time digest checks, and read/admin/database keys must be distinct.
This is a single-corpus read/admin model, not per-document ACLs, OAuth/OIDC or multi-tenant isolation. Every read credential authorizes the full configured knowledge corpus.
Encryption at rest and key handling
SQLCipher encrypts database pages containing original content, metadata, summaries, chunk text, FTS indexes, float32 vectors, compressed codes, graph facts and caches. WAL pages use the same encrypted database mechanism; shared-memory bookkeeping is not a separate content store. Temporary SQL storage is configured in memory. SQLCipher memory wiping, SQLite defensive mode, untrusted schema handling and secure deletion are enabled. Dynamic extension loading is never enabled; sqlite-vec is linked and registered statically.
Every connection receives the database key before reading its schema and verifies that it can authenticate the database. Missing/wrong keys, a plaintext file supplied as encrypted storage, unavailable SQLCipher support and corrupt/incompatible data fail explicitly. There is no fallback that quietly creates a plaintext database. A random raw 32-byte key is expected, encoded as 64 hexadecimal characters; short human passwords are rejected. The SQLCipher API describes the underlying key and encryption mechanisms.
Secrets are references in configuration, never literal config fields:
"security": {
"database_key":{"env":"KNOWLEDGE_DB_KEY"},
"require_valid_okf_metadata":true,
"network":{"allow_remote_https":false,"allow_loopback_http":false}
}
Use either env or file, not both. Secret/private-key files must be regular files with owner-only Unix permissions; symlinks are rejected. New credential files and databases are created with mode 0600, and new private directories with 0700. Secret buffers owned by the application use zeroizing storage. The executable disables core dumps on Unix.
Protection boundary: SQLCipher does not encrypt the editable OKF source folder, model weights, plaintext reports, existing legacy databases or arbitrary backups. Protect source files and local credentials with FileVault/an encrypted volume and appropriate account permissions. Bootstrap key files are convenient local development credentials: a copy of the whole private directory includes the database key and defeats protection against theft of that entire directory. Production keys should be kept separately, for example injected from a protected OS/secret-management facility. Environment variables/private files do not protect against a compromised running account or administrator reading process memory. Losing the database key prevents recovery of the encrypted database; the OKF sources can be reindexed into a new one.
Setup creates a separate encrypted index from the OKF source files. It does not migrate/delete the old plaintext databases. Remove old sensitive copies/backups through your normal retention process once independently verified. Live key rotation, encrypted backup/export tooling and OS keychain integration are not implemented.
Encryption in transit and outbound calls
The server accepts TLS 1.3 only and HTTP/1.1. Certificates and private keys are loaded locally. Trust/hostname verification remains enabled on clients. Explicit server.allow_insecure_loopback=true permits a development HTTP listener only on a loopback address; non-loopback plaintext listeners are rejected. It is false by default.
The native embedding and generation models run in-process, so those calls do not traverse a network. Optional HTTP model clients validate certificate/hostname trust, require at least TLS 1.2 for HTTPS, never follow redirects, ignore environment proxies, and pin localhost resolution to loopback. Remote HTTPS endpoints require security.network.allow_remote_https=true. Remote plaintext HTTP is always rejected. The secure profile also rejects loopback HTTP; enabling allow_loopback_http is an explicit local development exception. extra_ca_file adds a local model service CA without disabling verification. URL userinfo, query-string credentials/fragments and source-supplied model URLs are rejected/not executed.
Limits, browser handling and input processing
Defaults include a 64 KiB request body, 8,000 query characters, 16,000 context characters, 50 results maximum, four graph hops, two simultaneous HTTP/work requests, and a global 120-request/minute token bucket. Schema/runtime validation rejects unknown request fields. There is a request deadline, a header-read deadline and a bounded HTTP header buffer. Keep-alive is disabled. Timed-out native work retains its work permit until it actually finishes, preventing retries from building an unbounded detached-inference queue. Native inference is not forcibly killed halfway through a library call.
Allowed HTTP hosts are checked explicitly. Browser origins are denied by default; configured origins must match exact HTTPS origins, with bounded CORS preflights. CORS does not replace bearer authorization. Responses use no-store, nosniff, restrictive CSP/frame/referrer headers and HSTS over HTTPS. API errors do not expose internal exceptions, endpoints or paths. Structured HTTP audit events contain route templates, method, status, duration and request IDs; no credentials, raw query strings, prompts or answers are logged.
The watcher uses capability-relative file opens, skips symlinks and non-regular files, and limits total files, directory entries and bytes per scan. Reads are bounded even when a file grows during a scan. A failed scan preserves the previous index rather than treating unreadable files as deletions. Defaults: 10,000 files, 50,000 entries, 512 MiB total scan bytes, and the existing 64 MiB per-file limit.
The secure profile requires a regular root index.md; Markdown concepts must have a valid YAML mapping and a nonempty string type, except reserved index/log files. Invalid replacements retain the previous publication with an explicit failure. YAML parsing uses a maintained safe-Rust parser with bounds on metadata bytes, nesting, nodes/events and aliases; duplicate keys are rejected. Filesystem inclusion and environment interpolation are disabled. All unknown authored fields remain metadata. This is strict frontmatter enforcement, not a complete OKF trust/attestation/conformance validator. relations and processing.chunking remain application extensions.
Native model/media libraries remain trusted components; there is no process sandbox for adversarial codecs, no comprehensive media decoder audit, and no internet-scale connection/DDoS protection. This loopback-first application should not be described as a certified public multi-tenant service. Authentication and encryption do not fix model hallucination or OCR errors; those remain documented in the earlier quality report.
Evidence and reproduction
- Automated checks: authentication, roles, Host/Origin policies, body/query limits, timeouts retaining permits, real TLS trust/version/plaintext rejection, SQLCipher/WAL/wrong-key checks, parser budgets, strict OKF failure preservation, compression lifecycle, plus original ingestion tests and real native inference.
- Real native service run: original OKF bundle → native embeddings → encrypted SQLite → authenticated TLS hybrid/graph search using both backends, with response-schema validation.
- OpenAPI validation: official current schema, schema URLs and document checksum.
- Locked dependency audit: zero known RustSec vulnerabilities at the recorded check; one informational unmaintained
pastedependency remains throughnalgebra → simba. It is a build-time macro dependency, not an exposed parser. The runtime YAML/PEM libraries flagged by the initial audit were replaced. RustSec does not constitute a complete audit of bundled native C/C++ or operating-system libraries. - TurboQuant results: measured storage/accuracy/latency tradeoffs and explicit retention of the float32 reference index.
cargo test --release --features metal,multimodal -- --include-ignored
cargo clippy --features metal,multimodal --all-targets -- -D warnings
cargo test --no-default-features
python3 scripts/security-smoke.py
python3 scripts/validate-openapi.py
cargo run --release --features metal,multimodal --example benchmark_compression
The API smoke script needs Node for a modern TLS client. OpenAPI/schema response validation additionally uses Python jsonschema and referencing; the official schemas are cached for offline reruns. Neither is required by the Rust application at runtime. Test services are stopped and temporary test credentials removed when tests finish. The retained local default credentials are private and are not included in the report.
Source authoring, remote data and native decoders
Source mutations require an admin credential and server.allow_source_writes; read tokens remain read-only. Paths are capability-scoped to the configured corpus, reject hidden/traversal/device names and symlinks, and existing files require SHA-256 preconditions. Capture publication stages a complete subfolder before rename. Whole-capture deletion checks an app marker and current tree hash. See API semantics, including source changes that commit before indexing or an HTTP timeout. API/library writer locks do not serialize independent filesystem editors.
Source URL fetching has an independent, disabled-by-default policy. Exact origins, HTTPS, DNS/address checks, pinned resolution, no redirects/proxies/cookies, MIME limits and streamed byte ceilings constrain requests. Explicitly permitted loopback HTTP is a local-service exception; remote private/special networks remain blocked. Downloaded HTML is parsed without executing code. Its scripts/styles/hidden elements are excluded from extracted text. Text still remains untrusted knowledge and can contain prompt injections; the app's generation instructions and citation checks do not make hostile documents trustworthy.
FFmpeg/Poppler receive staged original bytes in private temporary directories. Output is bounded and subprocess deadlines/kill-on-drop apply; stderr is drained without storing content. FFmpeg gets restricted format/protocol lists and no shell. The decoders/native model runtime still execute as the host account and are not OS-level sandboxes. An encrypted DB does not encrypt these temporary files, editable sources, runtime memory or co-located credentials. Supply encrypted temporary/source volumes and host isolation where needed.
Remote model credentials are environment references. Model endpoints and referenced-resource fetch policies are independent. A service using remote model roles sends the corresponding content to those configured services; native defaults perform inference locally. Embedding-space identifiers assert model compatibility rather than providing a security boundary.