Configuration¶

All runtime configuration lives in db.env (the sourced shell-env format), with tokens, allowed IPs, and tenant directories stored as plain-text files under $DB_ROOT.

db.env¶

Placed in the working directory where you run shard-db (usually build/bin/db.env). Loaded once at startup.

Variable	Default	Description
`DB_ROOT`	`./db`	Root directory for all data, indexes, metadata, and per-tenant subdirectories.
`PORT`	`9199`	TCP listen port.
`TIMEOUT`	`0`	Query timeout in seconds. `0` = disabled. Enforced via cooperative cancellation in scan loops.
`LOG_DIR`	`./logs`	Directory for async logs (`info-YYYY-MM-DD.log`, `error-YYYY-MM-DD.log`, `slow-YYYY-MM-DD.log`).
`LOG_LEVEL`	`3`	`0`=off · `1`=errors · `2`=warnings · `3`=info · `4`=debug.
`LOG_RETAIN_DAYS`	`7`	Auto-prune logs older than N days. `0` = keep forever.
`INDEX_PAGE_SIZE`	`4096`	B+ tree page size in bytes (power of 2, 1024–65536).
`THREADS`	`0` (auto)	Parallel-for worker pool — drives every parallel hot path (shard scans, indexed find/count/aggregate fan-out, parallel index builds, bulk-insert phase 2). `0` = `4 × nproc`, minimum 4.
`IO_THREADS`	`0` (auto)	Separate I/O thread pool size for cache-bypassing operations (O_DIRECT scans, bulk-fetch). `0` = `4 × nproc`. I/O threads wait on page faults and benefit from oversubscription without starving CPU-bound queries.
`WORKERS`	`0` (auto)	Server-worker pool that accepts connections + dispatches request handlers. `0` = auto (CPU count, minimum 4).
`POOL_CHUNK`	`0` (auto)	parallel_for submission chunk size. `0` = `nproc`. Tasks are enqueued in chunks of this many; larger chunks reduce queue-lock contention but serialise concurrent submitters. Rarely needs tuning.
`GLOBAL_LIMIT`	`100000`	Default `limit` applied when a query omits one. Per-query `limit` is not clamped — pass any value to override.
`MAX_CONCURRENT_QUERIES`	`0` (auto)	Hard cap on queries in-flight simultaneously. `0` = `max(4, min(nproc, 32))`. When the cap is reached, additional requests get `{"error":"server at capacity","max_concurrent_queries":N}` and the client should retry. Worst-case query-buffer RAM = `MAX_CONCURRENT_QUERIES × QUERY_BUFFER_MB`.
`MAX_REQUEST_SIZE`	`33554432` (32 MB)	Maximum JSON request size per line. Oversized requests get `{"error":"Request too large (max N bytes)"}`. Every connection allocates a read buffer of this size, so total per-conn memory = N × `MAX_REQUEST_SIZE`.
`FCACHE_MAX`	`4096`	Entry capacity for each of `kfcache` and `segcache`. Strict allow-list: `{4096, 8192, 12288, 16384}`. Invalid values fall back to default with a warning. Also derives `BT_CACHE_MAX = FCACHE_MAX / 4`; see Tuning.
`BT_CACHE_MAX`	derived	Not configurable as of 2026.05.1. Derived as `FCACHE_MAX / 4` (so `{1024, 2048, 3072, 4096}`). Setting it in db.env emits a stderr warning and is ignored.
`QUERY_BUFFER_MB`	`256`	Per-query intermediate buffer cap. With `MAX_CONCURRENT_QUERIES` bounding fan-in, worst-case RAM stays predictable. Auto-tunes upward on big-RAM hosts with low slot counts (floor: 256 MB).
`INDEX_BUILD_BUDGET_MB`	`1024`	Peak per-pass memory budget for `reindex` / multi-field `add-index` / `migrate`'s phase-2 rebuild. Floor 64 MB. Multi-field builds group fields into passes that fit this cap; an oversized single field still runs alone. See Tuning → INDEX_BUILD_BUDGET_MB.
`DURABILITY_SYNC_MS`	`1000`	Target maximum dirty age / periodic attempt interval for cached mmap writes. `0` disables periodic sync; nonzero values must be at least 50 ms. Dirty ordinary evictions sync first, failed syncs remain dirty and are retried, and failures are written to the error log. This is not a WAL, transactional durability, a hard real-time bound, or a guarantee that directory-entry metadata is durable.
`DB_ODIRECT_BUF_MB`	`32`	O_DIRECT buffer size per worker in MB. Each parallel worker reads shard data in chunks of this size using cache-bypassing pread. Peak O_DIRECT RAM is approximately `2 × DB_ODIRECT_BUF_MB × IO_THREADS`. Larger chunks reduce syscall overhead on fast NVMe; smaller chunks reduce peak memory.
`DISABLE_LOCALHOST_TRUST`	`0`	Default: 127.0.0.1/::1 bypasses auth (assumes a trusted loopback proxy). Set to `1` for strict mode (tokens required even same-host).
`TOKEN_CAP`	`1024`	Open-addressed bucket count for the token store. Bump to 4096+ if you run thousands of tokens across scopes.
`SLOW_QUERY_MS`	`500`	Log queries slower than N ms to `slow-*.log` and the in-memory ring (`stats` endpoint). `0` = disable. Minimum 100 ms.
`RANDOM_SEQ_COST_RATIO`	`8`	Planner cost model: random-read penalty vs sequential scan. Higher values prefer full-scan; the planner chooses fetch-and-check when `matches < live_count / RANDOM_SEQ_COST_RATIO`. Tuning knob for workload-specific I/O patterns.
`WARMUP`	`async`	Cache warmth on daemon startup. `async` = joinable background thread (first queries race cache population); `sync` = block startup until complete; `off` = skip. Embedded mode defaults to `off` unless a valid `WARMUP=` line is explicitly present. Primes OS page cache for kf + index shards.
`AUTO_VACUUM`	`0`	`1` = enable a background thread that periodically polls `vacuum-check`'s recommendation logic and runs plain `vacuum` on objects that meet the thresholds. Never auto-runs `--compact` or `--splits` — both need an exclusive objlock for a long rebuild window and stay manual.
`AUTO_VACUUM_INTERVAL_SEC`	`3600`	Auto-vacuum poll cadence in seconds. Floor 60. Sleep is sliced into 1-second chunks so SIGTERM brings the thread down within a second.
`VACUUM_RECOMMEND_TOMBSTONE_PCT`	`10`	Tombstone ratio at which `vacuum-check` flags an object for cleanup (`deleted * 100 ≥ total * N`). Also drives auto-vacuum when enabled — same threshold for both manual and automated paths.
`VACUUM_RECOMMEND_MIN_DELETED`	`1000`	Absolute floor on `deleted` count below which `vacuum-check` does not recommend cleanup, even if the percentage clears. Prevents tiny objects from triggering vacuum overhead that exceeds the work saved.
`AUTO_RESHARD_ENABLE`	`0`	`1` = enable a background thread that, once per calendar day during `AUTO_RESHARD_HOUR`, grows an object's `splits` when its live record count outgrows the recommended sizing table (see Tuning → Recommended splits). Runs a full reshard (`vacuum --splits=N`), which holds the exclusive objlock for the duration — unlike `AUTO_VACUUM`, which never touches `--splits`.
`AUTO_RESHARD_HOUR`	`3`	Server-local hour (`0`-`23`) the auto-reshard sweep is allowed to run in, once per calendar day.
`AUTO_RESHARD_THROTTLE_MS`	`0`	Milliseconds to pause after each successful reshard before the same sweep tick considers its next candidate object. `0` = no pacing (reshards still run strictly one at a time either way; this only adds a gap between them).
`TLS_ENABLE`	`0`	`1` = require TLS 1.3 on `PORT`; plaintext clients rejected at handshake. See Operations → Deployment → Native TLS.
`TLS_CERT` / `TLS_KEY`	(empty)	Server cert + private key paths (PEM). Required when `TLS_ENABLE=1`.
`TLS_CA`	(empty)	Client-side CA bundle for verifying the server (defaults to OS trust store).
`TLS_SKIP_VERIFY`	`0`	Client-side: `1` skips server cert verify (dev only — emits stderr warning).

Example:

# db.env
export DB_ROOT="../db"
export PORT=9199
export TIMEOUT=30
export LOG_DIR="../logs"
export LOG_LEVEL=3
export LOG_RETAIN_DAYS=14
export INDEX_PAGE_SIZE=4096
export THREADS=0
export IO_THREADS=0
export WORKERS=0
export GLOBAL_LIMIT=100000
export MAX_CONCURRENT_QUERIES=0
export MAX_REQUEST_SIZE=33554432
export FCACHE_MAX=4096
# BT_CACHE_MAX is no longer configurable — derived as FCACHE_MAX / 4
export QUERY_BUFFER_MB=256
export INDEX_BUILD_BUDGET_MB=1024
export DURABILITY_SYNC_MS=1000
export DB_ODIRECT_BUF_MB=32
export TOKEN_CAP=1024
export DISABLE_LOCALHOST_TRUST=0
export SLOW_QUERY_MS=500
export RANDOM_SEQ_COST_RATIO=8

# Startup warmup — primes caches on startup
export WARMUP=async

# Auto-vacuum — opt-in. Same thresholds drive `vacuum-check` recommendations.
export AUTO_VACUUM=0
export AUTO_VACUUM_INTERVAL_SEC=3600
export VACUUM_RECOMMEND_TOMBSTONE_PCT=10
export VACUUM_RECOMMEND_MIN_DELETED=1000

# Auto-reshard — opt-in, grows splits automatically when an object outgrows
# its sizing (off by default)
export AUTO_RESHARD_ENABLE=0
export AUTO_RESHARD_HOUR=3

# Native TLS — leave TLS_ENABLE=0 unless terminating TLS in-process
export TLS_ENABLE=0
export TLS_CERT=""
export TLS_KEY=""
export TLS_CA=""
export TLS_SKIP_VERIFY=0

Every variable is optional; defaults apply when the file or a specific export is missing. Changes require a server restart.

Tenant directories — `dirs.conf`¶

Every data query must include a dir parameter (e.g., "dir":"acme"). That directory must be listed in $DB_ROOT/dirs.conf — one tenant name per line.

# $DB_ROOT/dirs.conf
default
acme
engineering

The default dir is auto-registered on first use. Add tenants with a plain edit + server restart, or at runtime via create-object (which will create and register the tenant path automatically if it's new).

Queries for unregistered dirs return {"error":"Unknown dir: <name>"}.

See Concepts → Multi-tenancy for the isolation model.

API tokens — `tokens.conf`¶

$DB_ROOT/tokens.conf — one token per line. Any request with "auth":"<token>" that matches a line is accepted from any IP.

# $DB_ROOT/tokens.conf
ajb9dsuf87sa8df7asdfasdf
another-service-account-token

Tokens are loaded at startup and refreshed when add-token / remove-token JSON modes are used (see Operations → Deployment).

IP allowlist — `allowed_ips.conf`¶

$DB_ROOT/allowed_ips.conf — one IP per line. Requests from these IPs bypass the token check.

# $DB_ROOT/allowed_ips.conf
127.0.0.1
::1
10.0.1.17

Localhost (127.0.0.1 and ::1) is trusted by default — you don't need to add it. Use the allowlist for sidecar processes or trusted services on other hosts.

Precedence¶

A request is authenticated if either:

The client IP is on allowed_ips.conf, or
The request carries "auth":"<token>" and the token is in tokens.conf.

Both lists live under $DB_ROOT so they travel with the data root.

Schema — `schema.conf` and per-object `fields.conf`¶

$DB_ROOT/schema.conf — one line per object: dir:object:splits:max_key:2:streams[:auto_key=<mode>]. Auto-managed by create-object — don't edit by hand. The literal 2 is the engine-version slot, kept on disk for forward compatibility; this version refuses any other value at load (legacy v1 objects must be upgraded via 2026.05.4's ./migrate first). streams is the number of seg-write streams (derived from nproc at create time, immutable for the object's life unless vacuum self-heals a streams-mismatch).
$DB_ROOT/<dir>/<object>/fields.conf — typed field definitions, one per line: name:type[:size|P,S][:default=...]. Also auto-managed (via create-object, add-field, remove-field, rename-field).

See Concepts → Typed records for the on-disk layout and all type definitions.

Storage layout¶

$DB_ROOT/
  .shard-db.lock                   # flock guard — prevents two daemons sharing this root
  tokens.conf                      # Global API tokens
  allowed_ips.conf                 # Trusted IPs
  dirs.conf                        # Allowed tenant directories
  schema.conf                      # Object catalog (one line per object)
  <dir>/                           # Per-tenant directory
    tokens.conf                    # Per-tenant tokens (optional)
    <object>/
      tokens.conf                  # Per-object tokens (optional)
      fields.conf                  # Typed field schema
      metadata/
        sequences/                 # Per-named-sequence counter files
      data/                        # v2 slotcask engine — see Concepts → Storage model
        kf/
          NNN.kf                   # Keyfile shards (24B header + 24B slot array)
        streams/
          NNN/                     # One subdir per write stream (nproc-derived)
            NNNNNN.dat             # Append-only segment files (128 MB rotation)
      indexes/
        index.conf                          # List of indexed fields
        <field>/                            # Per-field directory (per-shard btree layout)
          NNN.idx                           # Sharded B+ tree files, index_splits_for(splits) of them
        <a>+<b>/                            # Composite index — '+' joined name
          NNN.idx
      backup/
        <YYYYMMDD-HHMMSS>/                  # Per-backup snapshots created by `backup` mode
          ...
      files/                       # Stored files (put-file)
        <filename>                 # Flat — basename is the lookup key
  logs/
    info-YYYY-MM-DD.log
    error-YYYY-MM-DD.log
    slow-YYYY-MM-DD.log

The slotcask engine separates keys from values:

data/kf/NNN.kf — keyfile shards. Each is a 24-byte file header (SKF1 magic + version + live count + tombstone count) followed by a packed array of 24-byte slot headers. Slot count per shard is tiered on splits — 1M at splits ≤ 16, 256K at splits ≤ 128, 128K at splits ≤ 1024, 64K at splits ≤ 4096. Shards auto-resplit in-place (doubling capacity) at 75 % fill, up to a per-shard ceiling of 16M slots.
data/streams/NNN/NNNNNN.dat — append-only segment files. Each stream has its own subdirectory; segments rotate at 128 MB. Writers in different streams don't contend. The number of streams is fixed at create-object time from nproc (≤ 8 → nproc; ≤ 16 → 8; else 16).

Each indexed field is split into index_splits_for(splits) btree files — non-linear curve in src/db/types.h: 8→2, 16→4, 32→4, 64→8, 128→16, 256→16, 512→32, 1024→64, 2048→64, 4096→128. Writes route by record hash to a single idx-shard; reads fan out across all shards in parallel.

Pre-2026.05.5 installs with legacy v1 (probe-into-slot, data/NNN.bin shard files with Zone A + Zone B layout) must first upgrade to 2026.05.4 and run that release's ./migrate to convert objects to the slotcask layout — 2026.05.5+ refuses v1 objects at load.

Next¶

Operations → Deployment — systemd unit, native TLS or reverse-proxy TLS, log rotation.
Operations → Tuning — when to raise THREADS, FCACHE_MAX, etc.

Configuration¶

db.env¶

Tenant directories — dirs.conf¶

API tokens — tokens.conf¶

IP allowlist — allowed_ips.conf¶