Skip to content

shard-db 2026.08.1

Automatic startup migration, ACID-hardened writes (atomic indexed writes and crash-safe object rebuild), a new opt-in auto-reshard feature, and a broad set of concurrency and input-validation fixes accumulated since 2026.07.3. No on-disk format changes.

Highlights

  • Automatic startup migration — replaces the standalone ./migrate binary with a version-gated $DB_ROOT/.version check on both daemon and embedded paths; runs a full secondary-index reindex in-process when needed. See Migration.
  • Atomic indexed writes — every indexed write path (single + bulk) now survives a crash mid-write via a commit-intent marker and abort-sidecar recovery, closing the window where a kill could leave the index out of sync with the record.
  • Crash-safe object rebuild — vacuum-splits and edit-field rebuilds now run inside an atomic rename transaction (RebuildTxn) with fail-closed recovery, instead of risking inconsistent on-disk state on a crash.
  • Auto-reshard — new opt-in background thread (AUTO_RESHARD_ENABLE) that grows an object's shard count automatically as it grows, once per day in a configurable window.
  • Dead v1 storage engine and rebuild-kf removed; a broad set of concurrency, query-protocol, and input-validation fixes (below).

Migration

On startup, shard-db compares $DB_ROOT/.version with the compiled-in release. When an older or previously unversioned database is opened, the only migration performed by this release is a full secondary-index reindex; the marker is written only after that reindex succeeds. Matching versions start without migration, and newer database markers are rejected to prevent downgrades. Both cmd_server() (daemon) and shard_db_open() (embedded) call the same shared migration seam.

The documented minimum source release is 2026.07.3. It is recorded for operator guidance but is not enforced in this release because earlier releases did not write .version; unversioned and older roots therefore use the reindex-and-stamp path.

The fixed-slot/variable-length conversion and compaction migrations are not run by this release. Legacy v1 objects still require the historical 2026.05.4 migration path before they can be opened.

The standalone ./migrate binary is no longer built or shipped. Use ./shard-db reindex for an explicit on-demand rebuild. New ./shard-db version subcommand prints the compiled-in version without a running server.

Durability

  • Atomic indexed writes across crash boundaries — a durable commit-intent marker is written before every indexed mutation; a post-marker index-apply failure writes a matching abort sidecar, applies the inverse diff, and rejects the mutation instead of publishing an index/record mismatch. Covers single insert/update/delete and all bulk variants.
  • Crash-safe object rebuild — RebuildTxn (.preparing → .active → .done, atomic renames) makes vacuum-splits and edit-field rebuilds crash-safe; startup recovery fails closed on any ambiguous on-disk state instead of guessing which copy is authoritative. The DB root is flock'd exclusively during recovery in both daemon and embedded modes.
  • Atomic single-key partial updates — concurrent partial updates to different fields of the same key no longer race and silently drop one writer's change.
  • Atomic index publication — index shard writes now publish atomically.
  • Durable write hardening — keyfile sync failures on single-record writes are now propagated instead of swallowed.
  • rebuild-kf removed — it could repoint a live kf entry to the wrong value on hash collisions. Replaced by strict live-reference validation (slotcask_validate_live_refs) run before every full rebuild, which aborts and restores pre-rebuild data on any invalid reference. Operators who upgraded from an affected pre-2026.07.1 build without running that release's repair must run the last release containing rebuild-kf against a backup before upgrading past this release.

Concurrency fixes

  • Read-path use-after-free: JSON get/exists/count/describe-object and the entire NQL dispatch path could race a concurrent rebuild/vacuum freeing the underlying object. Also fixes a warmup-thread UAF of the same class, an enum_values leak, and a post-delete range-scan regression.
  • Warmup thread UAF against concurrent vacuum/rebuild.
  • Auto-reshard/auto-vacuum background threads are now joined before kfcache/slotcask shutdown (previously detached, could touch freed state mid-teardown).
  • B-tree cache eviction race (use-after-unmap under high parallelism) plus 3 memory leaks (cmd_edit_fields, dispatch_json_query, bulk-insert auto-key buffer).
  • kfcache/bitmap-cache lock-order inversion closed.
  • kfcache/segcache entries are now validated by file identity (dev/ino) on every hit, closing a rebuild-rename aliasing race that could leak a stale cache entry into a freshly-rebuilt object.
  • B-tree mutations are now serialized per path, preventing delete resurrection during bulk-merge rebuilds.
  • btree_bulk_merge no longer writes a duplicate leaf entry on an exact (value, hash) tie.
  • bt_cache/kfcache/segcache/bitmap-cache rwlocks are now writer-preferring on glibc/Linux, preventing writer starvation under sustained read pressure.

New: auto-reshard

Opt-in background thread (AUTO_RESHARD_ENABLE, AUTO_RESHARD_HOUR, AUTO_RESHARD_THROTTLE_MS) that walks every object once per day during the configured hour and grows its shard count to match its live record count. See configuration.md.

Query protocol

  • NQL quoted literals — adds SQL-style ''/"" escaping and a "..." wrapper delimiter, so values with spaces or embedded quotes are expressible. Breaking: an unterminated filter-string literal (e.g. name eq 'Alice with no closing quote) is now a hard parse error instead of silently truncating and matching.
  • NQL --filter flag on aggregate — explicit flag for filters the positional heuristic misclassified (IN/BETWEEN/parenthesized expressions); positional parsing still works for simple filters.
  • NQL reads no longer take a per-object lock — find/count/ aggregate now match JSON's lock-free read behavior instead of blocking behind a held schema write lock.
  • Aggregate requests eligible for the top-N streaming path with format:"csv" no longer silently returned JSON instead of CSV.
  • Per-key CAS on array-form bulk-update JSON records (if condition, evaluated under the kf-shard write lock; a CAS miss or malformed if skips only that record).
  • vacuum/recount on a missing or unopenable object now return an error instead of reporting fake success.

Input validation hardening

Several previously-silent-failure paths across the criteria parser, NQL parser, and server dispatch now return structured errors instead of misbehaving quietly: unknown criteria/aggregate operators (previously defaulted to equality/count()), malformed NQL --limit/--offset/ --order-by, uppercase DESC in aggregate ordering, and missing required fields on add-index/get-file-path/bulk-insert/bulk-delete/negative offset.

Removed

  • Dead v1 ucache/shard-file storage engine — this binary has only ever created v2 (slotcask) objects since 2026.05.5's v1→v2 migration requirement; the unreachable v1 probe-into-slot code path is removed. The stats/stats-prom ucache fields are retained, permanently reporting zero, so existing dashboards don't break.
  • rebuild-kf — see Durability above.

Fixes

  • JSON-escaped varchar values are decoded consistently before storage, indexing, and criteria comparison — partial-update, bulk-update, cursor pagination, and multi-get CSV export no longer store literal escape bytes or build mismatched index keys. Malformed escapes and decoded NUL bytes are now rejected at write time. regex/not_regex values are excluded (raw POSIX-regex source, not JSON string content).
  • get + fields returned "Not found" on v2 objects — the dispatch branch still read through the dead v1 layout; fixed with a new cmd_get_fields on the v2 slotcask registry.
  • decode_field stack-buffer overflow on composite fields (field1+field2) whose combined content exceeded the fixed 4096-byte stack buffer — now a dynamically-growing heap buffer.
  • count silently returned a count instead of erroring when the object couldn't be opened on the non-indexed fallback path — now returns {"error":"object not open"}.

npm binding

npm/binding.gyp was missing src/db/durability.c, which bitmap.c, btree.c, index.c, and slotcask.c (all already in the source list) call into — an unbuilt native addon with undefined-reference link errors. Fixed by adding the file; npm package shard-db bumped 1.0.10 → 1.0.11 to ship this release's fixes (embedded auto-migration included) to Node.js consumers.

Upgrade notes

./shard-db stop
# replace build/bin/ contents
./shard-db start

Startup runs a one-time full reindex if $DB_ROOT/.version is older than 2026.08.1, or missing on a non-empty root; a newer .version marker is refused (no downgrades). The standalone ./migrate binary is no longer built or shipped — legacy v1 objects still require the historical 2026.05.4 ./migrate path before this binary can open them.