Skip to content

shard-db 2026.05.5

Breaking-cleanup release. The legacy v1 (probe-into-slot) storage engine is removed; slotcask is the only supported layout.

The B+ tree on-disk sort order also changes — entries are now ordered by (value, hash) (magic 'BTRH') instead of value alone. This closes a silent-no-op bug in btree_delete on duplicate-value clusters and restores O(log N) descent for index deletes. Existing btrees must be rebuilt via ./migrate once on upgrade.

This release also lands edit-field (same-type schema mutations) and auto-key (server-generated UUID / sequence keys) — both v2-only, both shipped on main between 2026.05.4 and the cut.

Highlights

v1 storage engine removed

Every v1 code path was dropped: addr_from_hash / compute_addr helpers, the Zone A / Zone B probe code in cmd_get / cmd_insert / cmd_update / cmd_delete, the v1 vacuum / reindex / bulk paths, the text counts file (kf headers are the source of truth on slotcask), and the SHARD_ALLOW_V1_CREATE test opt-in. ~12 000 lines of dispatch + fallback code retired.

  • The storage_version JSON field is no longer part of the create-object API. Any client that still sends it gets the documented storage_version is not configurable error.
  • schema.conf line format is unchanged on disk: dir:object:splits:max_key:2:streams[:auto_key=...]. The literal 2 slot is preserved for forward compatibility with future engine versions, and the daemon refuses any other value at load with a pointer to the 2026.05.4 migrate step (see Upgrading below).
  • Schema struct loses its storage_version field; every site that branched on it now runs the slotcask path unconditionally.
  • SlotcaskSchemaInfo loses storage_version too — slotcask_registry_get is the only registry path.

B+ tree (value, hash) sort

Indexed btrees previously sorted entries by value only. Within a duplicate-value cluster (e.g. every record with status="paid"), hashes appeared in insertion order, so btree_delete's (value, hash) descent landed at the wrong slot when both the leaf bsearch and internal-page routing relied on the tuple. The pre-2026.05.5 fallback was a chain walk over every leaf containing the cluster — O(N) in the worst case, and the silent-no-op condition still triggered if the entry happened to sit on a leaf the walk had already passed.

This release flips the on-disk invariant: every entry sorts by (value, hash) lexicographically. Internal pages now carry both the separator's value and its hash, so descent routes directly to the unique leaf holding the target tuple. The magic number rolls from 'BTRG' (0x42545247) to 'BTRH' (0x42545248); the daemon refuses to open older files with a pointer to ./migrate.

./migrate runs unconditionally on first start after upgrade — it's just ./shard-db reindex under a daemon-lifecycle wrapper, idempotent on already-BTRH installs. No data shard touch; only <obj>/indexes/ btrees are rewritten. Plan ~3–5 s per million indexed entries per shard. See Upgrading below for the operator steps.

edit-field + auto-key

Both shipped on main between 2026.05.4 and this cut. See Schema mutations for the edit-field surface (same-type transforms; cross-type explicitly refused), and auto_key=uuid / auto_key=seq(<name>) for server-generated keys at insert.

Slotcask CRUD write-path optimisation

Four changes land together against the slotcask update/delete and bulk-upsert hot paths. No architectural shifts — the tombstone contract, kf-wrlock ownership, and reader-side hash verification are all unchanged. Net effect on real-disk 10M-record workloads:

Metric pre post delta
BULK INSERT JSON 10M 2.45 M/s 3.17–3.30 M/s +33%
BULK INSERT CSV 10M 2.88 M/s 4.01–4.20 M/s +45%
Single UPDATE p50 134 µs 54 µs 2.5× faster
Single DELETE p50 130 µs 50 µs 2.6× faster
Bulk UPDATE x10000 270 k/s 748–913 k/s 3× faster
Bulk DELETE x10000 81 k/s 649–1623 k/s 8–20× faster
Parallel UPDATE 5×10k 38 k/s 102–112 k/s 2.7× faster

Indexed invoice 1M (14 indexes) sees +13–26% on indexed bulk paths.

What changed under the hood:

  • Single slotcask_update / slotcask_delete — collapse the redundant double kfcache_acquire cycle on the test-only entry points. Production paths (upsert_slow_path, slotcask_delete_with_hooks) already had the one-cycle pattern.
  • Bulk-upsert Phase 5 tombstones — sort OLD slots by (stream_id, file_id) and walk in runs under one segcache rdlock per file. Cuts segcache acquire/release pairs from N to (unique files). Mirrors the bulk-delete Phase 3 batching already in place.
  • Parallel field-fanout in every CRUD pre_commit — v2_update_pre_commit, v2_delete_pre_commit, and v2_bulk_upd_pre_commit_bulk now dispatch per-indexed-field delete_index_entry / write_index_entry via parallel_for + update_idx_fn. The worker was promoted from storage.c-static to index.c so all three hooks share it. For a 14-idx workload, field wall time drops from N × ~1 µs serial to ~1 µs parallel.
  • Arena-allocated index keys — new build_index_key_from_record_into(out, out_cap, ...) no-alloc variant; each pre_commit allocates one arena (per-call for single-op hooks, per-worker for bulk hooks) sized nidx × INDEX_KEY_MAX (4096 B/slot), replacing 2 × nidx mallocs per record with one upfront pair. Jumbo varchar indexes (>4096 B) fall back to the malloc'd variant.

The audit checked but did not change: lock ordering between rotation_lock and the kf wrlock (no inversion — rotation_lock is always fully released before any wait for the kf wrlock); bulk fast path Phase 3 batching (already optimal); insert paths (single + bulk already at the memcpy/hash floor — no structural wins available).

Upgrading

From any 2026.05.1 – 2026.05.4 install (already on slotcask): swap binaries, then run ./migrate once to rebuild btrees in the new (value, hash) sort order.

./shard-db stop
# replace shard-db, shard-cli, migrate with the 2026.05.5 binaries
./migrate                                  # idempotent — reindexes B+ trees
./shard-db start

./migrate reads db.env from the current directory, starts the daemon itself, runs ./shard-db reindex, and stops the daemon. Plan ~3–5 s per million indexed entries per shard. Running it on an already-migrated install is a no-op rebuild.

If you run shard-db under systemd / supervisord / docker, swap the ./shard-db stop and ./shard-db start lines for your service manager's equivalents (e.g. systemctl stop shard-db / systemctl start shard-db).

From a pre-2026.05.1 install with legacy v1 objects on disk: this binary refuses v1 at load with the error Object [X] is in legacy v1 storage; this binary supports v2 only. Install 2026.05.4 first and run ./migrate to convert v1 → v2, then upgrade. Two-step path:

# step 1 — install 2026.05.4, run its migrate
./shard-db stop
# replace shard-db / shard-cli / migrate with the 2026.05.4 artifacts
./migrate                                  # converts v1 → slotcask
./shard-db start
# verify schema.conf entries all end in ":2:<streams>"
./shard-db stop

# step 2 — upgrade to 2026.05.5
# replace shard-db / shard-cli / migrate with the 2026.05.5 artifacts
./migrate                                  # rebuilds B+ trees as BTRH
./shard-db start

What did not change

  • Wire protocol stays JSON, byte-for-byte compatible with 2026.05.4 except for the create-object input validation noted above. All read response shapes (bare values, csv/dict format, find cursors) are unchanged.
  • All 38 search operators, joins, aggregates, bulk paths, TLS, the shard-cli TUI, and per-tenant / per-object auth are untouched.
  • On-disk layout for slotcask data objects (kf shards, stream segments) is byte-identical — only <obj>/indexes/*.idx files are rewritten by ./migrate.