Knowledge Consolidation

Implemented in 0.3.0-beta.2, following the Dreaming design analysis. Consolidation reads published evidence, produces a reviewable proposal and can promote approved content into an ordinary OKF concept. It does not change model weights. The original evidence remains the basis for verification and cited context.

Configuration

The primary knowledge.yml enables extractive proposals for the fictional sample; the library default is disabled. Extractive mode selects exact excerpts and needs no analysis model. Model mode can summarize and connect evidence, including inferred/hypothesis claims, but requires review before the result enters retrieval. See the runnable local model profile.

consolidation:
  enabled: true
  strategy: model                    # extractive | model
  interval_secs: 600                 # Automatic runs in watch/serve.
  source_prefixes: [systems, procedures] # [""] includes the whole bundle.
  proposal_prefix: derived
  max_sources_per_run: 4
  max_evidence_per_source: 4
  max_input_bytes: 8192
  max_output_bytes: 16384
  max_claims: 6
  timeout_secs: 180
  max_attempts: 3
  recipe_revision: "1"
  model:
    http:
      url: http://127.0.0.1:11434/api/chat
      protocol: ollama              # ollama | openai
      model: your-local-analysis-model
      revision: your-pinned-revision
security:
  network:
    allow_loopback_http: true

Use model.gguf instead of model.http for in-process native generation; both together are rejected. GGUF has the same fields as models.generation, including model path/revision, context size, output tokens, threads and GPU layers. Paths resolve relative to the configuration file. Without either override, model mode uses the configured generation/chat capability. An explicit model override has separate residency; budget it together with any resident foreground model. A missing analysis capability fails the proposal attempt without disabling retrieval. Remote/provider endpoints require the existing network policy and optional api_key_env; no fallback to a provider happens automatically. The models CLI command shows analysis selection and effective built-in thread/offload caps.

max_input_bytes counts serialized evidence supplied as the model input; instructions, provider framing and output tokens are additional. Choose a generation context that can fit the input and output token budget. Whole evidence chunks must fit: overly large chunks produce a diagnostic instead of being silently marked processed after reading a prefix. Configure smaller chunks or increase this limit. max_output_bytes validates returned JSON; the HTTP transport also retains its existing 16 MiB response ceiling. Claim count and supporting-reference count are separately bounded.

Groups follow source-path order within the configured prefixes; automatic graph/community grouping and freshness inheritance from all parents remain further processing work. Review temporal claims against the original concept metadata. The worker advances by evidence ID, not merely by file: subsequent runs cover remaining chunks of a large source. Each run is bounded by source, chunk and input limits; a digest is selective, not a promise to reproduce every fact from its inputs. The extractive baseline uses up to 320 bytes per selected claim, with UTF-8 boundaries preserved. Model mode receives source text, IDs, paths and representation kinds. It does not receive raw image/audio bytes; configure vision/OCR/ASR during indexing to make those textual representations available. Direct media embeddings alone are not transcripts or descriptions.

Workflow

knowledge index
knowledge consolidate               # One bounded run; does not scan/index first.
knowledge consolidate list          # Latest 100 summaries and output hashes.
knowledge consolidate show RUN_ID   # Full inputs, output, state and error.
knowledge consolidate review RUN_ID --hash OUTPUT_HASH --action approve
knowledge consolidate promote RUN_ID --hash OUTPUT_HASH
knowledge work                      # Needed when the watcher is not running.

Review also supports reject and retract. Promotion requires whole-bundle admin/write authority, a current proposal, and its exact output hash. Review records the authenticated principal and an append-only private review event; it does not claim the reviewer is a human or that source-supplied verified metadata is authenticated. Review is an explicit knowledge-management action, not a prompt instruction that grants a model permission to approve itself.

watch/serve run consolidation on the configured interval. It yields admission to ready indexing jobs and shares the indexing resource governor. Missed timer ticks are skipped rather than replayed as a burst. index/work process indexing only; an embedded host owns its own scheduling and can call consolidate when appropriate. Disabled, paused, busy, resource-deferred or fully processed collections do not create another proposal; disabled explicit runs return an explanatory error.

States are running, proposed, failed, approved, rejected, invalidated, promoting, promoted. Expired interrupted runs can retry; attempts are bounded. A run key binds exact evidence/input bytes and analysis recipe, and concurrent attempts are fenced by attempt number. Failed outputs stay outside the index. Changing analysis recipes/revisions schedules new work; changing the timer or resource caps does not redefine analysis content. Superseded source versions get new input identities.

Proposal text is stored in consolidation_runs, inside the configured SQLite/SQLCipher database, not as draft files inside the watched root. consolidation_inputs records its original evidence/version dependencies. No autonomous promotion is enabled. Promotion writes derived/<run-id>.md (or your configured prefix) through the durable changes API, with sources, claim footnotes, analysis identity, review identity and input IDs in x-knowledge-derivation. It disables referenced-URL fetching for that generated concept. Standard sources produce graph derived_from edges. The extension fields are application metadata, not additions to the OKF standard.

Promotion is retryable even if a restart interrupts recording its operation receipt. The operation owns stable exact bytes and source preconditions. Poll its publication receipt; promoting does not promise the index has caught up. Parent checks and serving eligibility close the race with a concurrent parent change: an invalid derivative is excluded even if its filesystem write already occurred. Managed derived concepts cannot be moved through the changes API because that would detach their dependency policy; retract and create a new consolidation instead. An external edit to generated bytes invalidates that registered digest.

Evidence, authorization and RAG

Every claim requires known input evidence and an exact nonempty supporting quote. Allowed kinds are extracted, summarized, computed, inferred, hypothesis. These checks prove reference/quotation consistency, not that a paraphrase follows logically, a calculation is sanctioned or a hypothesis is true. Curators must evaluate those distinctions. Automatic deterministic calculation/attestation and independent semantic verification are separate future processing recipes.

Serving a promoted digest requires access to its output path and all input paths. Every parent must still be the current published/desired version. Parent edits/removal, retraction or output tampering make that digest unavailable to retrieval, selection, evidence/resource reads and graph context, including requests pinned to older publications. Original historical evidence remains available under its own access/retention policy. Promoted output files remain ordinary plaintext OKF files; protect the authoring filesystem separately.

Retrieval limits repeated derived results that share parent versions. Context building expands supporting originals ahead of the generated digest, with citations to their immutable evidence IDs and byte-budget checks. This does not make a digest independent corroboration or guarantee an aggregate quality improvement. Derived text can still affect lexical statistics; full lineage-neutral indexing and broad quality qualification remain evaluation work. Consumers can select only original concept types using metadata filters when their workflow requires it.

All proposal dependencies are retention roots in this implementation, including rejected/failed runs, to keep review/recovery auditable. This increases retained storage; there is no automatic proposal-expiry or strict disk quota. Database backups include the proposal ledger. Schema 4 upgrades to schema 5 with a consistent private same-key pre-migration backup. Older binaries reject schema 5; restore that backup to roll back. Interrupted analysis in a restored database becomes retryable after its recorded lease expires. Raw source deletion alone does not erase its retained evidence or proposal excerpts. Avoid collecting secrets in knowledge in the first place; model instructions are not a secret detector.

OpenAPI and library

The OpenAPI contract and generated clients include:

Method/path Purpose
POST /v1/admin/consolidation/run Run one bounded group; returns a proposal or null when idle/deferred
GET /v1/admin/consolidation/proposals Latest proposal summaries
GET /v1/admin/consolidation/proposals/{id} Full run/proposal
POST …/{id}/review `{output_hash, action: approve
POST …/{id}/promote {output_hash}; source writes must also be enabled

All require whole-bundle admin authority and remain outside the default read-only MCP gateway profile. Long runs can outlive an HTTP deadline while retaining a bounded work slot; poll the proposal list instead of assuming the timeout cancelled native work. Callers cannot supply arbitrary prompts or privileged endpoint overrides through these routes.

The Rust Knowledge facade exposes consolidate, consolidation_proposals, consolidation_proposal, review_consolidation and promote_consolidation. Custom model providers implement the small backends::Analysis capability or Models::analyze(instruction, input). Answer generation keeps its own RAG schema; analysis has a separate structured-output contract.

See validation evidence for executed recovery, access, resource, HTTP and real local-model checks. The earlier research proposal describes broader future ideas; this release implements the bounded proposal/digest workflow, not autonomous code changes, model training or automatic truth arbitration.

Read Markdown →