Indexing resource budgets

0.3.0-beta.2 adds a shared cooperative governor for indexing and consolidation. The goal is controlled throughput during large imports while retaining foreground access. CPU/GPU percentage quotas and hard allocation limits require operating-system or model-server support; configuration does not invent those capabilities.

indexing:
  scan_interval_ms: 10000
  embedding_batch_size: 8
  resources:
    cpu_threads: 2
    max_gpu_layers: 8
    max_concurrent_jobs: 1
    max_concurrent_source_polls: 2
    max_embedding_batch_size: 8
    duty_cycle_percent: 25
    min_idle_ms: 250
    max_jobs_per_minute: 20
    max_rss_mb: null

The primary sample uses two CPU threads, eight offloaded layers, one indexing job, batches up to eight, 50% work/rest pacing, 100 ms minimum idle and 30 admissions/minute. Library defaults are two threads, one job, batches up to eight, 50%, 100 ms, no job-rate/RSS cap, and no additional GPU-layer cap. Existing per-model limits continue to apply; the smaller cap wins.

Parameter Default / range Actual control
cpu_threads 2 / 1–256 Caps built-in llama.cpp text, multimodal/projector and generation threads, including batch threads
max_gpu_layers null / nonnegative integer Caps native offloaded layers; 0 disables built-in GPU inference; null retains each model's configured value
max_concurrent_jobs 1 / 1–8 Maximum indexing jobs across an Engine and its clones. Watcher, index, work and library drain use this worker count
max_concurrent_source_polls 2 / 1–8 Maximum overlapping connector snapshots. See connected sources.
max_embedding_batch_size 8 / 1–1024 Caps the existing indexing.embedding_batch_size for uncached text embeddings
duty_cycle_percent 50 / 1–100 Work/rest scheduling ratio at background inference and job boundaries
min_idle_ms 100 / 0–60000 Minimum rest at those boundaries, including small jobs
max_jobs_per_minute null / 1–60000 Smooth admission spacing shared by indexing and consolidation; does not release a whole minute's budget in a burst
max_rss_mb null / 1–1048576 MiB Defers starting a job when measured process RSS exceeds the threshold; available on macOS/Linux

At 25% duty, a 200 ms background inference call schedules at least 600 ms of rest before the next call, or the configured minimum if larger. Pacing uses elapsed wall time, including model/network waits, rather than measured hardware busy time. Job pacing also covers extraction/storage work. Both constraints can reduce throughput further than the simple ratio. With pacing enabled, the governor serializes background inference. With 100% duty and zero minimum idle, a concurrent provider can overlap inference up to the worker cap. Built-in native model residency still serializes inference.

These are per-Engine/clone-group controls. Independent applications, separate providers and other processes must budget their aggregate resource use. Persisted pause state is shared by handles using the same database. Change resource/model settings by reopening the handle or restarting the service; mutating expert Engine fields is not a supported live reconfiguration mechanism. Changing scheduling limits does not invalidate embeddings.

CPU, GPU and memory boundaries

CPU threads limit native inference parallelism, not every thread in Tokio, SQLite, the operating system or a custom backend. FFmpeg decoding already uses one thread; decoder subprocesses additionally receive one-thread OpenMP/BLAS environment settings. Rust extraction and filesystem scanning remain bounded by job concurrency and existing file/folder/output budgets. Configure scan frequency and max_scan_files, max_scan_entries, max_scan_bytes, max_file_bytes, PDF pages and video frame/audio limits for large collections. Scanning continues while processing is paused so new desired revisions are observed and stale derivatives can be excluded.

GPU layers control offload placement, not GPU utilization percentage or a VRAM allocation quota. A partly offloaded model can still briefly occupy the GPU fully. Work/rest pacing reduces sustained background activity at call boundaries; it cannot interrupt a kernel or divide one long native call into time slices. Use max_gpu_layers: 0 when indexing must not use the GPU. The shared built-in provider applies these thread/offload caps to foreground inference too; create a separately budgeted provider/process when foreground needs different placement.

max_rss_mb is a soft admission threshold, not a reservation or hard memory ceiling. It does not unload a resident model, prevent one admitted job from exceeding the threshold, count GPU/driver memory comprehensively or account for child processes. If resident weights keep RSS above the threshold, processing remains deferred until resources/configuration change. Unsupported platforms reject a configured RSS threshold instead of pretending to enforce it; leave it null there. CPU, offload, pacing, concurrency and batching controls remain portable.

HTTP model services control their own CPU/GPU/RAM. Knowledge can bound request concurrency, batches and pacing, but cannot impose native-layer or thread limits on a remote service. Configure that server separately. Custom Rust model backends likewise own their internal parallelism. For hard host ceilings, run workers under suitable OS/container CPU/memory controls and isolate model services; a hard memory limit can terminate a worker, so use durable jobs and recovery. This release does not automatically install a service or alter host quotas.

Shared PostgreSQL job limits

PostgreSQL workers apply per-process concurrency and pacing. Configure a shared storage.postgres.provider_quota group to limit simultaneously leased indexing jobs across bundles and machines using the same index database. The quota spans worker processes and is independent of their individual lane counts. It leaves query-time inference and the model server's own memory and rate controls separate. PostgreSQL cluster setup

Pause, resume and inspect

knowledge indexing status
knowledge indexing pause
knowledge indexing resume

Pause is persisted in SQLite and survives restart. It prevents new job admission and stops background inference at subsequent call boundaries; native work already executing may finish. Pending jobs retain their data and do not consume attempts while waiting for admission. Foreground retrieval/model calls bypass background pacing, while retaining built-in model placement limits and model capacity constraints. A foreground request can still wait behind an already-running native call; this is not preemptive priority scheduling.

OpenAPI route Contract
GET /v1/admin/indexing/resources Effective budget configuration, pause state, current RSS if supported, policy explanations
POST /v1/admin/indexing/pause `{paused: true

Both require whole-bundle admin authority. Rust hosts use Knowledge::indexing_resources() and pause_indexing(bool). The model inventory shows clamped native threads/layers. Resource settings are not accepted from OKF metadata or untrusted HTTP callers.

knowledge index/work honor pacing and wait for rate-limited work. If pause or RSS admission leaves queued work, the CLI reports pending jobs; resume or adjust the budget rather than requeueing already valid jobs. The watcher keeps checking deferred work without consuming job attempts. Bulk imports remain persistent in SQLite rather than creating an unbounded inference task per notification.

See the validation report for timing assertions, batch/concurrency checks, memory admission, persisted pause and native-model evidence. Do not compare the earlier ungoverned beta benchmark directly with a throttled profile without recording the limits and build configuration.

Frequently edited OKF files

The automatic folder watcher coalesces revisions independently for each path. A file that changes repeatedly does not delay updates to neighboring files. The previous published snapshot remains searchable until a replacement is ready. Intermediate revisions held during settling are not stored as source versions or sent to models.

indexing:
  scan_interval_ms: 2000
  debounce_ms: 300
  change_batching:
    min_scan_interval_ms: 1000
    quiet_ms: 1500
    max_delay_ms: 30000
Setting Effect
min_scan_interval_ms Minimum pause after a folder scan, in milliseconds. Both native events and periodic reconciliation respect this limit. Range 50–3600000.
quiet_ms Time a changed path must keep the same observed version before it is queued. Set to zero for immediate admission on the next scan. The same window protects a file that briefly disappears during a save.
max_delay_ms Age of an unsettled change at which the watcher may sample a readable snapshot even if the file is still changing. Must cover both other intervals and be at most 86400000.

The watcher measures time from observation. The next scan, file read stability, a live indexing lease and the resource governor can add delay. Continuous edits may therefore produce occasional snapshots. An active job for the same path keeps its lease while later snapshots settle. A confirmed deletion removes the path from search. Native notifications initiate a bounded scan window and periodic scans repair missed events. The watcher still hashes files within the existing scan budgets.

A fresh database can queue its first snapshot immediately. On restart, the watcher observes the folder again and rebuilds its in-memory settling state. The explicit knowledge index command, administrative reconciliation and journaled OKF authoring apply their existing immediate reconciliation behavior. Watcher scan logs include a deferred count when paths are waiting for admission.

On the M3 Mac, a native watcher run applied 60 atomic saves about 30 ms apart. With a 100 ms minimum scan interval, a 350 ms quiet window and a 1000 ms maximum batch delay, SQLite retained two new versions of the hot file. The unrelated page published during the burst and the final hot revision published after it. This was a lexical-only fixture, so the measurement counts admitted snapshots and does not estimate inference throughput.

Read Markdown →