Connected sources

To store the entire authoritative OKF bundle in a database, use a database-backed OKF bundle. This guide covers supplemental inputs.

The OKF folder remains the authored graph. A Knowledge index can also project multiple external sources into the same SQLite index. Each enrollment has its own identity, stable record keys, polling interval and budgets. A connector supplies the bytes for an indexed record. The existing pipeline handles extraction, chunking, embeddings, keyword indexing, graph edges and publication. Embedding, summarization and media extraction depend on the model and format settings of the receiving knowledge base.

flowchart LR
    OKF["OKF bundle"] --> F["Folder watcher + reconciliation"]
    SQL["SQLite tables or views"] --> S["Independent, bounded connector polls"]
    MAN["Local files or allowlisted URL manifest"] --> S
    S3["S3-compatible bucket prefix"] --> S
    F --> Q["Versioned records and durable SQLite jobs"]
    S --> Q
    Q --> W["Configurable indexing workers"]
    W --> P["Search, citations, graph and publication history"]
    P --> C["CLI, Rust library, MCP and OpenAPI"]

Try two sources together

From the repository root, create a read-only source database and a manifest containing one local file. The sample indexes them alongside the existing OKF example. It works with lexical search before models are configured.

mkdir -p .okf/source-example/assets
sqlite3 .okf/source-example/lab.sqlite <<'SQL'
CREATE TABLE IF NOT EXISTS devices(id INTEGER PRIMARY KEY, title TEXT, notes TEXT, team TEXT);
CREATE TABLE IF NOT EXISTS manuals(id INTEGER PRIMARY KEY, content TEXT, device_id INTEGER);
INSERT OR REPLACE INTO devices VALUES (1, 'Workbench sensor', 'Quadrature calibration every 30 days.', 'lab');
INSERT OR REPLACE INTO manuals VALUES (2, 'Manual describes failover and maintenance.', 1);
SQL
printf 'Optical interferometry reference material.\n' > .okf/source-example/assets/notes.txt
cat > .okf/source-example/manifest.yml <<'YAML'
items:
  - key: optical-notes.txt
    path: notes.txt
YAML

knowledge -c configs/sources.yml validate-config
knowledge -c configs/sources.yml index
knowledge -c configs/sources.yml sources list
knowledge -c configs/sources.yml search quadrature --mode lexical
knowledge -c configs/sources.yml search interferometry --mode lexical

See the complete sample configuration. YAML and JSON paths resolve relative to the configuration file.

SQLite tables and views

Each enrolled input opens a separate database in read-only mode and reads the configured projections in one SQLite transaction. A row becomes a generated Markdown document. Each text_columns field becomes a section, title_column supplies an optional label, and metadata_columns populate the fields metadata map. Integer and text IDs should be stable and unique within each table or view. Use a view to define joins, select a tenant or subset of rows, normalize column names, convert data types, or build a composite primary key. Configuration takes literal table and column identifiers. It has no embedded SQL and performs no writes to the source database.

The row key is <table>/<id>.md, with non-alphanumeric text ID characters percent-encoded within the ID component. Integer and text values that produce the same key collide, causing that refresh to fail. Use homogeneous ID types in a projection. Fields accept UTF-8 text, finite numbers or NULL. For BLOBs, expose authorized paths or URLs in a separate manifest. A declared links mapping makes a non-NULL foreign ID an edge to a row in another enrolled projection within the connector. The target projection should use the same ID representation as the foreign ID.

Manifests

A YAML or JSON manifest lists stable logical keys with exactly one of path or url per entry.

items:
  - key: reference-card.pdf
    path: manuals/reference-card.pdf
  - key: field-notes.html
    url: https://docs.example.org/lab/field-notes.html

Local paths are relative to the connector's local_root. Use a dedicated readable directory containing only data approved for indexing. Paths are confined to that directory. Keep the index, source databases, secrets and configuration outside it. The manifest is limited to 1 MiB. Publish it with an atomic rename so a poll can read a complete inventory. To avoid mixed revisions across a manifest with several local files, write the new files first and then replace the manifest to point to their new paths under the same keys.

URL items require urls.enabled: true and an exact HTTPS origin in urls.allowed_origins. The existing fetch policy handles address checks, transport and caching, with urls.refresh_secs setting the network revalidation cadence. Use stable URLs with a response content type compatible with the key extension. URLs containing query strings, fragments or credentials are rejected. Only explicitly listed URLs are fetched. Links and processing directives found inside a connected document cannot trigger recursive URL fetches or override chunk policy. Use operator-configured chunking rules instead.

S3-compatible buckets

Default builds and the platform build script include S3. For custom builds, add the s3 feature. An omitted endpoint uses the AWS regional endpoint. For a self-hosted bucket, configure an HTTPS origin. HTTP requires an explicit opt-in and a literal loopback IP for local development. Requests use SigV4, direct connections, standard TLS validation and path-style bucket addressing. Provide read-only credentials through the named environment variables. Set session_token_env for temporary session credentials and restart with fresh credentials when they rotate. Use prefix-scoped bucket listing and object read permissions. Grant any required object decryption separately.

Use a dedicated bucket or a trailing-slash prefix. Records use full object keys, including that prefix. Each poll consumes all listing pages within the record budget. Supported objects are downloaded conditionally at their listed ETag. Unchanged ETags and lengths reuse retained bytes. Unsupported object extensions are skipped and still count toward the inventory limit. An object change between listing and download or a failed page retains the previous snapshot. A completed multi-page listing is a reconciliation input. Concurrent writes can occur between pages, so use the watcher’s repeat observation for deletions or publish immutable keys with a manifest when the source needs a single coherent release. ListObjectsV2 · GetObject · Arrow object_store

Refer to connected records from OKF

Use knowledge-source://<connector-id>/<record-key> in a Markdown link, a concept's sources[].resource, or a Knowledge relations[].target. These are stable graph identifiers. The connector and its table/prefix are selected by operator configuration. The URI itself never executes a query or enrols a data source.

---
type: Procedure
title: Inspect the bench sensor
sources:
  - resource: knowledge-source://lab/devices/1.md
    relationship: reference
relations:
  - predicate: documented_by
    target: knowledge-source://lab/manuals/2.md
---

Follow the [calibration record](knowledge-source://lab/devices/1.md).

Use knowledge::sources::reference to encode special characters in keys. Citation locators include source_origin with the connector ID, record key, logical URI and S3 ETag when available. Current and historical evidence use Knowledge's existing citation APIs. Access is evaluated against each evidence path, including when following graph links. Grant _knowledge_sources/lab to scope a consumer to that connector. An unrestricted bundle reader or share can see all indexed connectors. The reserved namespace is unavailable for authoring through the filesystem scanner or API. Backup retains captured connector bytes in the database. Restore materializes authored files in the OKF folder and keeps connector records in the restored database. A backup contains source content, so protect it with the same controls as the index. OKF specification

Watching, maintenance and embedding

Entry point Behavior
knowledge index Reconciles the authored folder and every enrolled connector, then drains runnable jobs. Reports failed sources and exits nonzero after processing healthy sources.
knowledge sources list Shows configured IDs, connector types, intervals and last successful refresh timestamps.
knowledge sources refresh [id] Takes a snapshot of one or all configured connectors and queues changes. Use knowledge work to drain those jobs on demand.
knowledge watch Runs the native folder watcher, connector polls and worker pool in the foreground. Supports Ctrl-C and Unix SIGTERM.
knowledge serve Runs HTTP(S) with OpenAPI and configured MCP. With server.watch: true, it also runs the watcher and worker pool. Run it in the foreground under your service manager.
knowledge mcp-stdio Connects to an existing index for local MCP reads. Use another process for ongoing indexing.
Rust Knowledge Use reconcile for the folder, refresh_source for a connector, drain for the queue, or watch_until with a caller-owned cancellation token. Use new to inject a custom model provider.

Snapshots commit in one transaction per connector. A missing manifest, failed query, partial listing, oversized record or timeout preserves the connector's last successful state. Removing a record from a complete inventory can remove it from current search. Watch polls coalesce hot updates using indexing.change_batching and defer replacement while the old job is running. Initial records can queue immediately. Later edits and disappearances are compared on successful polls. The observed delay can include a full polling interval. Explicit CLI/library refreshes bypass the watcher’s settling window. Retries and publication history remain managed by the shared indexing queue.

Control Default Range
limits.poll_interval_secs 30 1–86,400
limits.timeout_secs 30 1–300
limits.max_records 1,000 1–100,000
limits.max_snapshot_bytes 64 MiB 1 byte–128 MiB
indexing.resources.max_concurrent_source_polls 2 1–8
indexing.resources.max_concurrent_jobs 1 1–8

Per-record bytes also obey indexing.max_file_bytes. Polls are bounded separately from inference and run after the previous poll completes. The timeout interrupts SQLite query execution and caps asynchronous network work. OS filesystem operations can still hold an occupied poll slot if the backing filesystem stalls. Connector reads and inference budgets are per Knowledge engine. Use one supervised watcher per index as the deployment default. Generation checks prevent overlapping refreshes from committing an older snapshot over a newer one. Use the worker configuration to overlap extraction and I/O. Background model calls are serialized while pacing is enabled. Setting 100% duty and zero minimum idle allows concurrent calls within the worker cap when the provider supports them.

Removing a connector from configuration stops polling after restart and preserves its current records. To remove the records from the current publication, remove the enrollment and run knowledge sources forget <id>. The ID remains retired to block stale processes from resurrecting it. Historical citations follow the configured retention and purge policies. To replace the connector with a different physical source, assign a new ID and update the OKF links.

Extending the connector design

Bounded snapshots suit small and medium inputs where a complete scan is affordable. For higher-volume databases, introduce a connector checkpoint and normalized upsert/delete changes. Commit the checkpoint with the queued versions in the same SQLite transaction. A PostgreSQL connector should pair a consistent initial snapshot with logical decoding and persist the consumed LSN. It needs replica identity for deletes, an explicit publication boundary, and monitoring for retained WAL and lag. A SQLite connector can use a long-lived read connection's data_version as a hint to skip unchanged polls, or an application-owned outbox for incremental change capture. Its data_version is connection-local and cannot serve as a durable restart checkpoint. PostgreSQL logical decoding · SQLite data_version

Bucket events can wake a scoped poll and reduce delay. Events can be duplicated or reordered, so use them as hints and keep periodic inventory reconciliation. Additional connectors such as a sitemap, Git snapshot, or application outbox should provide stable record keys, explicit source scope, bounded payloads and a clear delete contract before sharing the same ingestion path. S3 event delivery

Read Markdown →