shard-db 2026.07.2¶

Three new field types (ipv4, ipv6, datetimems), auto_create preservation across upserts, security hardening (path traversal, varchar overflow, CLI injection, secure random), performance improvements (lock-free hash lookup, IN-list precompute, inline record buffer), and internal refactoring (query.c split, type descriptor table). The ./migrate step is retained from 2026.07.1 — it is idempotent and will be reviewed for removal in 2026.08. Wire-compatible with 2026.07.1.

Highlights¶

ipv4 field type — raw 4-byte IPv4 address, network byte order. Parsed from dotted-quad string. Byte-lexicographic order matches numeric IPv4 order.
ipv6 field type — raw 16-byte IPv6 address, network byte order. Parsed from canonical IPv6 string. Byte-lexicographic order matches numeric IPv6 order.
datetimems field type — millisecond-precision calendar datetime (8 bytes: int32 date + uint32 ms-of-day). Wire format is the 17-digit string "yyyyMMddHHmmssfff".
auto_create preservation — :auto_create fields now stamp once on genuine fresh inserts and are preserved (not re-stamped) on upserts. Previously, every upsert silently reset the create timestamp.
Security hardening — object name path traversal, varchar overflow rejection, CLI shell injection (posix_spawn), and secure random source hardening.
Performance — lock-free fast path for slotcask_lookup_by_hash, precomputed IN-list varchar lengths, inline buffer for read_record_ref.
Internal refactoring — monolithic query.c split into 8 per-concern translation units; centralized field-type descriptor table (type_desc).

New field types¶

`ipv4`¶

Raw 4-byte IPv4 address stored in network byte order. Parsed from dotted-quad strings (e.g. 192.168.1.1); malformed input encodes as all-zero bytes. Byte-lexicographic ordering matches numeric IPv4 order, so range queries and index walks produce correct results without special-casing.

addr:ipv4        # 4 bytes on disk

`ipv6`¶

Raw 16-byte IPv6 address stored in network byte order. Parsed from canonical IPv6 strings (e.g. 2001:db8::1); malformed input encodes as all-zero bytes. Byte-lexicographic ordering matches numeric IPv6 order.

addr:ipv6        # 16 bytes on disk

`datetimems`¶

Millisecond-precision calendar datetime. 8 bytes on disk: int32 date (yyyyMMdd) + uint32 ms-of-day (0..86399999). Wire format is the 17-digit string "yyyyMMddHHmmssfff". Supports :auto_create and :auto_update.

created_at:datetimems              # 8 bytes on disk
created_at:datetimems:auto_create  # stamped on first insert

Distinct from datetime (second precision) and timestamp (epoch milliseconds). Use datetimems when you need calendar-packed values with sub-second precision but want to preserve the human-readable date decomposition.

auto_create field behavior¶

:auto_create fields now stamp a default value once on genuine fresh inserts and are preserved on all subsequent upserts (single insert, bulk insert, bulk-insert-delimited). Previously, every upsert silently reset the create timestamp to the current time, which was incorrect for records where the original creation time should be immutable.

This is a behavior change — existing objects with :auto_create fields will see the correct semantics immediately after upgrading. No schema changes needed. Records written by older binaries that had their create timestamp overwritten will retain the overwritten value; only new upserts going forward preserve the existing timestamp.

Security hardening¶

Object name path traversal¶

Object names were interpolated into filesystem paths without validation. A tenant-scoped token could pass object:"../other_tenant/obj" to access sibling tenant data. Added is_valid_object() which rejects names containing /, \, leading dot, .., control characters, or exceeding 255 bytes. Enforced at every request entry point.

Varchar overflow rejection¶

typed_encode_defaults previously clamped over-length varchar content to fit the declared field size and inserted it anyway with no error. Now rejects with an explicit error on insert, bulk-insert, and CSV/delimited paths when content exceeds the configured size. This is a behavior change — previously silently truncated values will now fail with an error message indicating the overflow.

CLI shell injection (posix_spawn)¶

run_capture in shard-cli built shell command strings via snprintf with single-quoted user input. Any value containing a single quote could break into arbitrary shell execution. Replaced popen with posix_spawn using explicit argv — no shell involved.

Secure random source¶

Replaced per-call fopen("/dev/urandom") with fill_random() using getentropy(2) as primary and /dev/urandom as fallback. All generation sites now propagate failure:

gen_uuid4_raw returns -1 on failure (was void, returned all-zeros)
Bulk-insert paths refuse batch on generator failure
typed_encode_defaults rejects inserts when defaults fail to generate
Rebuild backfill aborts on generator failure

Performance¶

Lock-free `slotcask_lookup_by_hash`¶

Switched from kfcache_acquire() (full table mutex + per-slot rwlock) to kfcache_acquire_direct() with per-shard ref for the warm-hit lock-free path. Transparent fallback on cold miss. Reduces contention on the hot lookup path under high-concurrency workloads.

IN-list varchar precompute¶

IN-list varchar lengths are now precomputed at criteria compile time instead of being re-measured per record during criteria matching. Reduces per-record overhead for queries with in/not_in operators on varchar fields.

Inline record-ref buffer¶

read_record_ref avoids malloc/free per record for records that fit in an inline buffer. Reduces allocation pressure on the common case of small to medium-sized records.

Internal refactoring¶

`query.c` split into per-concern translation units¶

The 28,289-line monolithic query.c (~40% of the daemon source) has been split into 8 files by concern:

File	Responsibility
`query.c`	Criteria matching, planner core, find/count orchestration
`query_aggregate.c`	Aggregate operations, group-by, having, top-N, hash tables
`query_join.c`	Join planning, resolution, lookup, and result emission
`query_plan.c`	Compiled criteria, `match_typed*`, criteria tree parser, planner (`plan_filter`), `cmd_explain`
`query_find.c`	Scan helpers, fetch, keys, exists, CSV output, file ops
`query_bulk.c`	Bulk insert/delete/update
`query_maint.c`	Maintenance ops (vacuum, backup, recount, rebuild-kf, reindex, etc.)
`query_schema.c`	Schema mutations (create/drop object, edit/add/remove fields/indexes)
`query_internal.h`	Shared cross-TU prototypes, types, and enums

Mechanical move, behavior-preserving. Binary size unchanged.

Type descriptor table¶

Introduced type_desc.c / type_desc.h as a single source of truth for field-type facts (name, integer width for aggregate fast path). Replaces parallel switch statements scattered across call sites. _Static_assert ensures every FieldType enum value has a row.

Linker garbage collection¶

Release builds now compile with -ffunction-sections -fdata-sections and link with --gc-sections (Linux) / -dead_strip (macOS), allowing the linker to drop unreferenced code and data.

Fixes¶

Aggregate group_by hash-table resize — agg_ht_resize used the wrong hash function for int-keyed buckets (agg_hash instead of agg_hash_int), causing duplicate buckets after resize. Silent data corruption on large groupings with int/long/short keys.
IN-list uuid/time memory leak — free_compiled_criteria did not free in_uuid / in_time IN-list arrays, leaking memory on every query that used those operator forms.
slotcask_registry_get single-flight — prevented redundant concurrent opens of the same shard on startup with parallel requests.

Upgrade notes¶

The ./migrate binary is retained from 2026.07.1 and is idempotent. It will be reviewed for removal in 2026.08 — any migration steps that are no longer needed will be dropped at that time.

./shard-db stop
./migrate          # idempotent — safe to re-run if interrupted
./shard-db start

New records written after 2026.07.1 are already in compact form — ./migrate is only needed if you haven't run it since upgrading to 2026.07.1.

Behavior changes to be aware of:

auto_create preservation — upserts no longer reset the create timestamp. Existing records with overwritten timestamps retain their values; only new upserts going forward preserve the original.
Varchar overflow rejection — over-length varchar values that were previously silently truncated will now fail with an error. Ensure your data fits the declared field sizes.
Object name validation — object names containing /, \, .., or control characters are now rejected at the server level.

Wire-compatible with 2026.07.1 — no client changes needed.