Skip to content

Query protocol overview

shard-db speaks one wire protocol: newline-delimited JSON over TCP. Every request is a JSON object on a single line terminated by \n; every response is terminated by \0\n (NUL byte + newline).

Wire format

Request

{"mode":"...","dir":"...",...}\n
  • One JSON object per line. No comments, no trailing comma.
  • Maximum per-request size = MAX_REQUEST_SIZE (default 32 MB). Oversized requests are rejected with {"error":"Request too large (max N bytes)"}.
  • You can pipeline multiple requests on the same connection — the server processes them in order.

Response

{"status":"ok",...}\n\0\n
  • JSON object (or array) followed by \0\n.
  • Clients read until \0 to frame a response. The trailing \n is separator between responses; most clients can ignore it.
  • For pipelined requests, responses come back in order, each terminated by \0\n.

Minimal request shape

Every request has:

{
  "mode": "<command-name>",
  "dir": "<tenant>",
  "object": "<object-name>",
  "...": "mode-specific fields"
}

Except for global modes (stats, db-dirs, vacuum-check, auth admin, create-object) that don't need object, or modes that explicitly accept keys:[...] instead of a single key.

Authentication

From trusted IPs (allowed_ips.conf + default localhost), no credential is required. Otherwise, include a token:

{"mode":"get","dir":"default","object":"users","key":"u1","auth":"<token>"}

Tokens carry both scope (global / per-tenant / per-object) and permission (r / rw / rwx). See Concepts → Multi-tenancy for the full model and Getting started → Configuration for file layout.

Native TLS

Set TLS_ENABLE=1 (plus TLS_CERT / TLS_KEY) in db.env to require TLS 1.3 on PORT. Plaintext clients are rejected at handshake. Reverse-proxy termination (nginx stream, HAProxy, stunnel) remains supported as the alternative. Details: Operations → Deployment.

Per-request timeout

Any query can override the global TIMEOUT (db.env) for itself:

{"mode":"find", ..., "timeout_ms": 200}

Applies to find, count, aggregate, bulk-delete, bulk-update. 0 or absent → falls back to the global. Use it to give specific callers tighter deadlines without reconfiguring the server.

Modes at a glance

Data operations

  • get, insert, update, delete, exists, not-exists, size — per-record CRUD and existence checks
  • find, count, keys, fetch — queries

Aggregation and joins

  • aggregate — count/sum/avg/min/max with group_by + having
  • Joins in find — inner/left, by PK or indexed field

Bulk

Conditional writes

  • CAS — if_not_exists, if:[...]

Schema

Indexes

Files

Sequences (monotonic counters)

  • sequence — action: init | next | current | reset; optional batch for bulk allocation

Diagnostics

Auth admin (server scope: trusted-IP or global rwx token)

  • add-token, remove-token, list-tokens, add-ip, remove-ip, list-ips, add-dir, remove-dir

get — the simplest round-trip

// Request
{"mode":"get","dir":"default","object":"users","key":"u1"}

// Response
{"key":"u1","value":{"name":"Alice","email":"a@x.com","age":30}}

Multi-get:

// Request
{"mode":"get","dir":"default","object":"users","keys":["u1","u2","u3"]}

// Response
[{"key":"u1","value":{...}},{"key":"u2","value":{...}}]
// (missing keys are omitted)

With field projection:

// Request
{"mode":"get","dir":"default","object":"users","key":"u1","fields":"name,email"}

// Response
{"key":"u1","value":{"name":"Alice","email":"a@x.com"}}

Pagination patterns

Pattern Use when Speed
offset + limit UI-style page numbers O(matches) — fine for shallow pages, expensive deep
cursor (returned by fetch) Stream through an entire object O(limit) — constant cost per page

Error shape

{"error":"<machine-readable-string>","..."}

Common errors:

Error Cause
Unknown dir: <name> dir not in dirs.conf.
Object [<name>] not found. Use create-object first. Object doesn't exist.
Request too large (max N bytes) Request exceeded MAX_REQUEST_SIZE.
condition_not_met CAS check on insert/update/delete failed.
invalid filename Upload/download filename violated rules.
file exists Upload with if_not_exists:true hit an existing file.

See Reference → Error codes for a full list.

Clients

Python

import socket, json

def query(request, host="localhost", port=9199):
    s = socket.socket()
    s.connect((host, port))
    s.sendall((json.dumps(request) + "\n").encode())
    s.shutdown(socket.SHUT_WR)
    data = b""
    while True:
        chunk = s.recv(65536)
        if not chunk: break
        data += chunk
    s.close()
    text = data.decode().strip().rstrip("\x00").strip()
    return json.loads(text) if text else None

Node.js

const net = require("net");

function query(request, host = "localhost", port = 9199) {
  return new Promise((resolve, reject) => {
    const s = net.connect(port, host);
    let data = "";
    s.on("connect", () => { s.write(JSON.stringify(request) + "\n"); s.end(); });
    s.on("data", chunk => data += chunk.toString());
    s.on("end", () => {
      try { resolve(JSON.parse(data.replace(/\0/g, "").trim())); }
      catch { resolve(data); }
    });
    s.on("error", reject);
  });
}

Pipelining

Send multiple requests on one socket — responses come back in order, each terminated by \0\n. See the README's Python/JavaScript sections for a pipeline() helper.

Shell (diagnostic only)

echo '{"mode":"get","dir":"default","object":"users","key":"u1"}' | nc -q1 localhost 9199

Where to go

  • find — the most-used query mode; all 38 operators, joins, sorting, projection, cursor pagination.
  • aggregate — group-by + having.
  • CAS — conditional writes.
  • Bulk — bulk-insert, bulk-update, bulk-delete.
  • Schema mutations — evolve objects without downtime.