shard-db 2026.07.1¶

Compact VARCHAR storage, Natural Query Language (NQL), kf-corruption recovery (rebuild-kf), and an add-index/remove-index locking fix. This release covers everything merged since 2026.06.3. Run ./migrate after upgrading — it converts existing objects to VARIABLE format, compacts all segments, and repairs any kf corruption. Wire-compatible with 2026.06.3.

Highlights¶

Compact VARCHAR storage — segment files now store records as exactly 24 + key_len + value_len bytes, with trailing all-zero fields trimmed on every write. Typical varchar workloads see 50–90% smaller segment files. Run ./migrate once to compact existing records.
NQL wire protocol — the server now accepts human-readable find/count/aggregate commands on the same TCP port as JSON. Auto-detected by the first character of the request line. CLI gains a first-class find subcommand.
rebuild-kf recovery command — repairs corrupted/dangling kf (keyfile) entries by rescanning every segment file and re-deriving each live record's kf slot from scratch. Fixes a vacuum/compact guard bug that could delete a donor segment file even when the kf entry pointing to it failed to update, leaving records unreachable by key lookup despite still existing on disk.
add-index/remove-index exclusive locking — both commands now take the exclusive per-object lock (objlock_wrlock), closing a race where a concurrent insert's on-the-fly index update could land on the same index file mid-rebuild and either get silently discarded or corrupt the B-tree pages being written. Longer-running rebuilds (e.g. multi-field) were more exposed to this than single-field ones.
Several correctness/segfault fixes surfaced while building the above (see Fixes).

Storage: Compact VARCHAR¶

VARIABLE-format segments¶

shard-db previously padded every record to a fixed slot_size (the sum of all field byte lengths). Records are now stored as exactly 24 + key_len + value_len bytes — no zero-padding. For typical varchar workloads where most field values are short, this reduces segment file sizes by 50–90%.

Trailing-zero trim on every write¶

typed_encode_trim_len() walks the typed field layout from the end and computes the shortest prefix that fully encodes all non-zero fields. Zero fields at the tail are omitted; the decoder fills them back as zero on read.

Safe for all field types:

Type	Zero value
`varchar`	empty string
`int` / `long` / `short`	`0`
`double`	`0.0`
`bool`	`false`
`date`	epoch (`00000000`)
`datetime`	epoch

Happens automatically on every INSERT / UPDATE / bulk-insert after upgrading. No schema changes needed.

`compact` command¶

Rewrites all existing segment files for an object, applying the trim to every live record. Needed to reclaim space from records written by older binary versions.

# CLI (server must be stopped)
./shard-db compact default stories

# JSON API (server running, requires rwx permission)
./shard-db query '{"mode":"compact","dir":"default","object":"stories"}'

`./migrate` runs compact automatically¶

Phase 2 of the migrate tool iterates all objects and calls compact on each one. Run once after upgrading:

./shard-db stop
./migrate          # converts format + compacts all objects
./shard-db start

Idempotent — safe to re-run if interrupted.

Features: NQL wire protocol¶

NQL on the wire¶

The server auto-detects NQL vs. JSON by the first character of each request line:

{ → JSON query (existing protocol, unchanged)
f / c / a → NQL (find / count / aggregate)

Response framing (\0\n terminator, pipelining) is identical to JSON. Both formats are accepted on the same port, from the same clients, with no configuration change.

# Raw TCP — auto-detected
echo 'find default users "age > 25" --limit 5' | nc localhost 9199

NQL filter grammar¶

A recursive-descent parser produces the same CriteriaNode * tree the JSON engine uses — identical planner paths, identical performance characteristics.

or-expr  = and-expr ("or"  and-expr)*
and-expr = atom     ("and" atom)*
atom     = "(" or-expr ")" | predicate

AND binds tighter than OR. Parentheses override precedence. Max nesting depth: 16.

All 38 operators are supported, including symbolic aliases (=, !=, >, <, >=, <=), string ops (contains, starts, ends, like, case-insensitive i-variants), length ops (len_eq, len_between, …), field-vs-field ops (eq_field, lt_field, …), and existence checks (exists, not_exists).

BETWEEN correctly disambiguates its and keyword from the logical connector: age between 18 and 65 and status = active parses as (age BETWEEN 18 AND 65) AND (status = active).

CLI `find` subcommand¶

./shard-db find <dir> <obj> [filter] [--order-by field[:dir]] [--limit N] [--offset N] [--fields f1,f2] [--format json|rows|csv|dict]

Previously find was JSON-only via ./shard-db query '{...}'. The new subcommand accepts an optional NQL filter string directly.

count and aggregate retain their existing positional JSON API and gain NQL auto-detection — if the criteria argument doesn't start with [ or {, it is parsed as NQL.

Examples¶

# find
./shard-db find default users 'age > 25 and status = active' --limit 20
./shard-db find default users 'status in (active,pending)' --fields name,email --format csv
./shard-db find default users 'deleted_at not_exists' --order-by created_at:desc

# count
./shard-db count default orders 'total between 100 and 999 and paid = true'
./shard-db count default users   # all records — O(1) metadata path

# aggregate
./shard-db aggregate default orders 'status = active' sum(amount),count() \
  --group-by region --having 'count > 10' --order-by sum_amount:desc

# TCP wire
echo 'find default users "age > 25" --limit 5' | nc localhost 9199
# With auth (non-trusted IP)
echo 'find tenant users "age > 25" --auth mysecrettoken' | nc remotehost 9199

Full reference: docs/query-protocol/nql.md.

kf corruption recovery¶

cmd_vacuum's light path (slotcask_compact_segs → compact_one_stream_varlen, part of this release's variable-length records work) merges sparse donor segment files into denser recipients, then deletes the donor. compact_migrate_records_varlen returned success even when a record's kf update was silently skipped — including when the segment file a kf entry currently pointed at was already missing or corrupted. The kf entry still referenced the donor file, but the donor got deleted anyway, leaving a dangling pointer. Symptom: records intermittently unreachable by key despite existing on disk; bulk-insert paths that need pre_commit_needs_old=1 (triggered by any btree index) failing with some_records_dropped.

The fix:

compact-kf guard — the donor file is no longer deleted if any record's kf update couldn't be confirmed during migration.
rebuild-kf — new command that repairs already-corrupted kf entries by rescanning all segment files for an object and rebuilding kf slots from what's actually on disk.

{"mode":"rebuild-kf","dir":"<dir>","object":"<obj>"}

./shard-db rebuild-kf <dir> <obj>

Idempotent — a clean object reports {"status":"ok","repaired":0}.

Automatic repair:

./migrate runs rebuild-kf automatically (phase 2/3, before compact) on every upgrade, and writes a .kf_rebuild_done sentinel so it isn't repeated on every subsequent embedded startup.
Embedded (npm) clients auto-run rebuild-kf once per db_root on first use, gated by the same sentinel — npm users get the repair with zero manual steps. Also exposed directly as shardDb.rebuildKf(dir, object) for on-demand repair.

Full reference: docs/query-protocol/schema-mutations.md#rebuild-kf.

add-index / remove-index locking fix¶

add-index and remove-index unlink and rebuild live index files in place (.tg/.idx/.bm) — the same destructive pattern vacuum/truncate/reindex use — but were classified under the shared per-object read lock (same class as insert) instead of the exclusive write lock those other rebuild commands correctly take. The btree cache keys entries by path string with no inode check, so a concurrent insert's on-the-fly index maintenance could land on the same path mid-rebuild and either have its write silently discarded (landing on the soon-to-be-unlinked old file) or corrupt the B-tree structure the rebuild was actively writing. Caught via trigram search returning drastically undercounted results despite index files containing plausible byte volume — worse the longer a rebuild ran, since longer (multi-field) rebuilds give concurrent writers a bigger window to interleave.

The fix: add-index and remove-index now take objlock_wrlock (exclusive), serializing them against all concurrent writes to the same object — identical to how vacuum/truncate already behave. See Concepts → Concurrency and the Lock model summary.

Practical effect: add-index/remove-index now queue behind in-flight writes and block new writes while running, rather than racing them. Rebuilding a large index under sustained write load will take noticeably longer to start (waiting for in-flight writes to drain) but is now guaranteed correct. Run large rebuilds in a maintenance window where possible.

Fixes¶

reindex_seg_cb segfault on compacted objects — the reindex segment scanner ignored vlen and read field values at a fixed offset, but compact writes records shorter than ts->total_size (trailing zero fields trimmed). Any add-index/reindex on a compacted object could read past the end of the record and segfault. Fixed with a per-worker zero-padded buffer used whenever vlen < ts->total_size.
compact dispatch double-dir path bug — fixed a path-construction bug in the compact JSON dispatch that duplicated the directory segment.
recover_one_stream realloc leak on OOM — a failed buffer grow during crash-recovery directory scanning leaked the prior allocation instead of freeing it (the realloc() return value overwrote the original pointer directly). Only triggers under OOM during crash recovery; fixed to match every other realloc call site in slotcask.c.
export-schema dropped its unused version field — nothing ever read or validated it; removed rather than left to go stale.
npm packaging — nql.c was missing from binding.gyp's source list, breaking NQL support in embedded/npm builds; added. npm package version corrected from an erroneous 1.1.2 back to sequential 1.0.7.

Upgrade notes¶

Migration required to reclaim disk space and repair any kf corruption from earlier versions:

./shard-db stop
./migrate          # phase 1: convert fixed→variable; phase 2: rebuild-kf + compact
./shard-db start

New records written after upgrading are already stored in compact form — ./migrate is only needed to rewrite the backlog of fixed-format records and repair any accumulated kf corruption.

If any index was rebuilt under concurrent write load on a prior version and results look suspect (e.g. search undercounting), force-rebuild it after upgrading:

./shard-db add-index <dir> <obj> <field> -f

Embedded (npm) consumers: rebuild the native addon (npm run build:release in npm/, or bump to the published 1.0.7+ package) to pick up the C-level fixes.

Wire-compatible with 2026.06.3 — no client changes needed beyond the native addon rebuild above for embedded users.