Port the Java HA sender to the Rust, Python, and C/C++ clients

.dockerignore

@@ -0,0 +1,14 @@
+# Keep the Docker build context small. The demo images only need
+# python/blotter.py, blotter-demo/feed.py and blotter-demo/entrypoint.sh; everything below is
+# build output or large data that would otherwise be uploaded to the daemon.
+.git/
+**/target/
+**/build/
+rust/target/
+c/build/
+go/
+*.csv.gz
+**/__pycache__/
+**/*.pyc
+**/venv/
+**/.venv/

README.md

@@ -1,5 +1,21 @@
 # Sample Sender with Multiple Hosts for HA Ingestion
 
+## Client implementations
+
+This project is a QuestDB high-availability sender. The **Java** implementation in this
+repository is the **reference**. It is ported, in parity, to the three clients built on the
+same C/Rust core (the "CRusty" clients):
+
+- [`rust/`](./rust) - Rust port.
+- [`python/`](./python) - Python port, plus a pandas/polars ingestion and egress demo.
+- [`c/`](./c) - C/C++ port.
+
+Each mirrors the Java design (CSV replay loop, per-worker senders, the `qwp` / `qwpudp` /
+`ilp` transports, failover, store-and-forward, and the probe). Per-language build steps and
+client differences (for example, auto-flush is present in Java and Python but not in Rust or
+C/C++) are documented in that folder's `README.md`. The rest of this document describes the
+reference Java implementation.
+
 ## Compile
 
 `mvn  -DskipTests clean package`
@@ -46,6 +62,98 @@ java -jar target/ilp_sender-1.0-SNAPSHOT.jar \
 QWP. That is a different timeout from `--connect-timeout-ms` (below), which bounds a single
 connect attempt.
 
+## Pacing: `--delay-ms` vs `--rate`
+
+Two mutually exclusive ways to throttle generation:
+
+- `--delay-ms` (default `50`): a fixed `Thread.sleep(delay)` after **every row**, per worker.
+  Simple, but a poor throttle at speed: `sleep()` has millisecond granularity (and often
+  over-sleeps), and the fixed per-row cost caps throughput. `--delay-ms 1` yields well under
+  1000 rows/s; `--delay-ms 0` runs flat out (millions/s).
+- `--rate` (default `0` = off): a **target aggregate rate in rows/second across all workers**.
+  Each worker paces itself to its share (`rate / num-senders`) against a deadline schedule,
+  sending rows back-to-back and sleeping only when it runs *ahead* of schedule (coalescing many
+  rows into one sleep). This hits high targets a per-row delay never could: `--rate 300000`
+  holds ~300k rows/s regardless of `--num-senders`.
+
+When `--rate > 0` it **takes precedence** and `--delay-ms` is ignored (a warning is printed if
+both are set). Measured on a local QuestDB: `--rate 5000` → 4.03 s for 20k rows (~4966/s);
+`--rate 300000` → 2.03 s for 600k rows (~296k/s), steady per-second progress at the target.
+
+### Regenerating the replay CSV
+
+Two scripts regenerate the replay CSV. **Both always export from the public demo box**
+(`https://demo.questdb.io/exp`) on purpose: the demo is *continuously ingesting live market
+data*, so every export is a fresh, recent-price snapshot rather than a stale file. You then
+**recreate the table locally** from that CSV and replay it for internal demos. The sender
+always writes into a table named **`trades`** locally, regardless of which demo table the CSV
+came from.
+
+- **`regenerate_csv.sh`** — pulls the crypto **`trades`** table (BTC-USDT, ...). Its
+  timestamp is microseconds.
+  ```
+  ./regenerate_csv.sh                   # -> trades.csv, 1,000,000 most-recent rows
+  ./regenerate_csv.sh trades.csv.gz     # gzip output (matches the --csv default's .gz)
+  ./regenerate_csv.sh trades.csv 250000 # custom row count
+  ```
+- **`regenerate_csv_fx.sh`** — pulls the FX **`fx_trades`** table (EURCHF, AUDNZD, ...).
+  Two schema differences from the crypto table, both handled in the query
+  `select timestamp, symbol, side, price, quantity as amount from fx_trades order by timestamp desc limit N`:
+  its size column is **`quantity`**, aliased to **`amount`** so it matches the `trades` schema
+  the sender writes; and its designated timestamp is **`TIMESTAMP_NS`** (nanoseconds), not
+  micros (see [Nanosecond vs microsecond timestamps](#nanosecond-vs-microsecond-timestamps)).
+  ```
+  ./regenerate_csv_fx.sh                      # -> fx_trades.csv, 1,000,000 rows
+  ./regenerate_csv_fx.sh fx_trades.csv.gz     # gzipped
+  ./regenerate_csv_fx.sh fx_trades.csv 250000 # custom row count
+  ```
+
+Both export exactly `symbol, side, price, amount, timestamp` (the columns the sender reads),
+download to a temp file first so a failed fetch never clobbers a good CSV, and gzip-compress
+when the output path ends in `.gz` (the loader auto-detects `.gz`). Override `DEMO_HOST` only if
+you mirror the demo tables elsewhere.
+
+### Nanosecond vs microsecond timestamps
+
+The crypto `trades` table exports microsecond timestamps; the FX `fx_trades` table exports
+**nanosecond** timestamps (`TIMESTAMP_NS`, e.g. `...192508297Z`). This only ever matters with
+`--timestamp-from-file`; in the default replay mode the sender stamps `now()` and ignores the
+file timestamp entirely, so precision is irrelevant.
+
+**The generator sends timestamps at nanosecond resolution** (via `at(long, NANOS)`), and
+QuestDB stores them at the **target column's** resolution: a micros `TIMESTAMP` column silently
+truncates the extra digits, a `TIMESTAMP_NS` column keeps them. Cross-resolution replay is
+therefore safe in both directions and lands every row. Validated against a local QuestDB, 20k
+crypto + 20k FX rows each way (all 40k landing, no errors):
+
+| Local `trades` column | Replayed source | Stored as |
+| --- | --- | --- |
+| `timestamp` (micros) | crypto micros `...790999Z` | `...790999Z` |
+| `timestamp` (micros) | FX nanos `...192508297Z` | `...192508Z` (truncated to micros) |
+| `timestamp_ns` (nanos) | crypto micros `...790999Z` | `...790999000Z` |
+| `timestamp_ns` (nanos) | FX nanos `...192508297Z` | `...192508297Z` (**full nanos preserved**) |
+
+So if your local table is `TIMESTAMP_NS`, the FX source's nanosecond precision is preserved
+end-to-end; if it is micros, the sub-microsecond digits are truncated on store (lossless to the
+column's resolution, never an error). To recreate the local table at a given resolution,
+pre-create it before ingesting, e.g.:
+
+```sql
+-- microsecond trades table
+create table trades (symbol symbol, side symbol, price double, amount double,
+  trade_id varchar, timestamp timestamp) timestamp(timestamp) partition by day wal;
+
+-- nanosecond trades table (swap the timestamp type)
+create table trades (symbol symbol, side symbol, price double, amount double,
+  trade_id varchar, timestamp timestamp_ns) timestamp(timestamp) partition by day wal;
+```
+
+**If the table does not exist, the server auto-creates it as `TIMESTAMP_NS`** (nanosecond),
+because the generator sends nanosecond timestamps: the designated timestamp column is created
+at the resolution of the incoming value. Verified end-to-end: dropping `trades` and replaying
+the FX chunk with `--timestamp-from-file` recreated it with a `TIMESTAMP_NS` column holding the
+full `...192508297Z` value. Pre-create the table (above) if you want micros instead.
+
 ## Transport (`--protocol`)
 
 - `--protocol qwp` (default): QWP over WebSocket. Adds store-and-forward (un-acked

blotter-demo/README.md

@@ -0,0 +1,177 @@
+# One-command live blotter demo
+
+A self-contained Docker demo: QuestDB with **LIVE VIEW**, the unreleased
+**questdb 5.0 Python client**, a synthetic price feed, and the terminal blotter
+rendering on top of a live view. Your colleague needs **nothing but Docker** — no
+Python, no Rust, no client build.
+
+## Run it
+
+```bash
+cd blotter-demo
+docker compose build            # one-time, slow: builds QuestDB + the client from source
+docker compose run --rm demo    # QuestDB starts, then the blotter draws in your terminal
+```
+
+`run` (not `up`) is deliberate — it gives the blotter a real TTY for the in-place
+redraw. **Ctrl+C** quits the blotter; the feed stops with it. Afterwards:
+
+```bash
+docker compose down             # stop QuestDB
+```
+
+## Running it again
+
+The slow part is the one-time `docker compose build`. Once the images exist, every
+`docker compose run --rm demo` skips the build entirely and just starts containers,
+so repeat runs are fast — the only cost is a few seconds of JVM startup as QuestDB
+cold-starts (the `--rm`/`down` flow disposes the container each time, so the table,
+live view, and history start fresh on every run — intended for a demo). A rebuild
+only happens if you edit a file a Docker layer depends on, and even then only the
+cheap final layers re-run — never the Rust/Maven builds — unless you change the
+pinned client/server commits.
+
+If a `docker compose down` reports "network still in use", a run container lingered
+from a Ctrl+C; clear it with `docker compose down --remove-orphans`.
+
+## Removing everything at the end
+
+This demo creates four kinds of Docker artifact: two built images
+(`blotter-demo-questdb`, `blotter-demo-demo`), a network and containers, build
+cache (the Rust + Maven layers — the bulk of the disk), and base images it pulls
+(`python:3.12-slim`, `eclipse-temurin:25-jdk`, `eclipse-temurin:25-jre`).
+
+**Removing an image is not destructive the way deleting a volume is** — any image
+can be re-pulled from Docker Hub on demand, so the worst case of over-removing is a
+re-download later. No data is lost. The demo defines no named volumes.
+
+### Step 1 — containers, network, and the two built images (scoped, safe)
+
+```bash
+docker compose down --rmi local --remove-orphans
+```
+
+Touches only this project. `--rmi local` deletes the two images the demo built;
+`--remove-orphans` clears any run container left by a Ctrl+C.
+
+### Step 2 — the base images this demo pulled
+
+Docker does not record when an image was pulled, so "was it already there before?"
+can only be answered by snapshotting the image list **before** the first build.
+
+**Before your first `docker compose build`**, record what already exists:
+
+```bash
+docker image ls --format '{{.Repository}}:{{.Tag}}' | sort > ~/blotter-demo-images-before.txt
+```
+
+**At cleanup**, remove each base image only if it was NOT in that snapshot (i.e.
+only if this demo pulled it):
+
+```bash
+for img in python:3.12-slim eclipse-temurin:25-jdk eclipse-temurin:25-jre; do
+  grep -qxF "$img" ~/blotter-demo-images-before.txt || docker rmi "$img"
+done
+```
+
+If you did **not** take the snapshot, you cannot tell which were pre-existing.
+Either leave them, or just remove all three — re-pulling is harmless (no data loss),
+it only costs a download if another project needs them later:
+
+```bash
+docker rmi python:3.12-slim eclipse-temurin:25-jdk eclipse-temurin:25-jre
+```
+
+### Step 3 — reclaim the build cache (the big disk win)
+
+```bash
+docker builder prune -f
+```
+
+`docker builder prune` removes only BuildKit's cached build layers. It **never
+deletes images, containers, or volumes** — those are safe. But it is **system-wide,
+not scoped to this demo**: it clears the build cache of every project, so other
+projects' next builds recompute their layers (slower, but nothing is lost — cache
+only). There is no per-project build-cache prune. If this demo is your only heavy
+Docker build, it is effectively scoped anyway.
+
+### Not recommended
+
+`docker system prune -af` reclaims everything Docker is not currently using **across
+all projects**, not just this demo — only run it if you genuinely mean machine-wide.
+
+## What it does
+
+1. **`questdb`** — built from source at the commit that first ships `LIVE VIEW`
+   (`90a1b54c…`) with the web console (`-P build-web-console`). The server is
+   reachable from the host at **http://localhost:19000** (console + `/exec`); this
+   host port is deliberately 19000, not 9000, so it never collides with a local
+   QuestDB. The blotter reaches the server internally over the compose network.
+2. **`demo`** — builds the 5.0 client (`ea54b6f…`, Rust 1.91.1 + Cython), then:
+   - waits for QuestDB,
+   - `curl` creates `core_price_demo` (base table) and `core_price_lv` (live view),
+   - runs `feed.py` in the background (~2000 rows/s of synthetic crypto/FX bids),
+   - runs `blotter.py` in the foreground, polling the live view at 10 Hz.
+
+The live view:
+
+```sql
+CREATE LIVE VIEW IF NOT EXISTS core_price_lv
+FLUSH EVERY 5s IN MEMORY 5s START FROM NOW AS
+SELECT timestamp, symbol, bid_price, avg(bid_price) OVER w AS moving_avg
+FROM core_price_demo
+WINDOW w AS (PARTITION BY symbol ORDER BY timestamp ANCHOR DAILY '00:00');
+```
+
+`START FROM NOW` is required by this build (the view tracks data from creation
+time onward). A live view serves results from memory, so ingested rows appear
+immediately; `FLUSH EVERY` is only the disk-commit cadence and does not gate what
+the blotter sees. Visible motion comes from the **feed** flushing to the base
+table often.
+
+The blotter queries the live view directly — `select * from core_price_lv limit -20`
+— so it streams the newest 20 rows (timestamp, symbol, bid, moving average),
+refreshed at `--rate` Hz.
+
+## Knobs
+
+- **Feed rate / flush frequency.** `FEED_FLUSH_HZ` (default 20, i.e. a flush every
+  50 ms) and `FEED_ROWS_PER_FLUSH` (default 100) on the `demo` service set how
+  fast rows land and how often they are flushed. 20 flushes/s keeps the blotter
+  visibly moving. Prices are illustrative 2026 levels, not live market data.
+- **Blotter refresh.** `--rate` Hz in `entrypoint.sh` (default 10) sets how often
+  the terminal redraws.
+- **What the blotter shows.** Defaults to the last 20 rows of the live view
+  (`--table core_price_lv --limit -20`). Adjust the count with `--limit`, or pass
+  `--query "<sql>"` in `entrypoint.sh` to run any SQL against the view verbatim.
+
+## Pins and durability
+
+Both sources are pinned by commit SHA for reproducibility:
+
+| Component | Repo | Commit |
+| --- | --- | --- |
+| QuestDB server | `questdb/questdb` | `90a1b54c98b10fad5304b1ad817a69cda25e52ad` |
+| Python client | `questdb/py-questdb-client` | `ea54b6f474062c144aa6395facad42e77c99e6f6` |
+
+A pinned SHA is fetchable from GitHub only while it stays reachable from a ref.
+The questdb commit is expected to merge to `main`, so it survives. The **client
+commit lives on `jh_experiment_new_ilp`, which may be deleted before it merges** —
+if that happens the build's `git checkout` will fail. Run `./vendor.sh` to
+snapshot the client source into `blotter-demo/vendor/` and follow its printed instructions
+to switch the Dockerfile to the local tarball.
+
+## When the client ships / questdb merges
+
+- Once `LIVE VIEW` lands in a nightly, delete `questdb.Dockerfile` and set
+  `image: questdb/questdb:nightly` on the `questdb` service.
+- Once the client is on PyPI, replace the whole `client-build` stage with
+  `pip install questdb==<version>`.
+
+## Sharing without a rebuild
+
+To hand a colleague an image that needs no build at all, either push to a registry
+(`docker compose build && docker compose push`, after adding `image:` names) or
+`docker save`/`docker load` a tarball. Both are architecture-specific — an amd64
+image runs under emulation on Apple Silicon. Letting them `docker compose build`
+locally avoids that by building for their arch.

blotter-demo/demo.Dockerfile

@@ -0,0 +1,46 @@
+# Blotter demo app: the unreleased questdb 5.0 Python client (built from source),
+# plus the feed + blotter scripts. Built once, then runs with zero host deps.
+#
+# The client is pinned by SHA. NOTE: unlike the questdb server commit, this branch
+# (jh_experiment_new_ilp) may be DELETED before it merges, which would make the SHA
+# unfetchable from GitHub. If that happens, run blotter-demo/vendor.sh to snapshot the
+# source locally and switch the client-build stage to COPY it in (see vendor.sh).
+ARG CLIENT_COMMIT=ea54b6f474062c144aa6395facad42e77c99e6f6
+ARG RUST_VERSION=1.91.1
+
+FROM python:3.12-slim AS client-build
+ARG CLIENT_COMMIT
+ARG RUST_VERSION
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        git curl build-essential pkg-config ca-certificates \
+    && rm -rf /var/lib/apt/lists/*
+# Rust >= 1.91.1: the bundled c-questdb-client (questdb-rs 7.0.0) requires it.
+RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \
+        | sh -s -- -y --default-toolchain "${RUST_VERSION}" --profile minimal
+ENV PATH="/root/.cargo/bin:${PATH}"
+
+WORKDIR /src
+RUN git clone https://github.com/questdb/py-questdb-client.git .
+RUN git checkout "${CLIENT_COMMIT}"
+RUN git submodule update --init --recursive
+# Build-time deps must stay in the isolated build env (do NOT use
+# --no-build-isolation; that pins the wrong numpy and fails the cython compile).
+RUN pip install --no-cache-dir -U pip "cython>=3.1.2" "setuptools>=80.9.0" numpy
+RUN pip wheel --no-deps --no-cache-dir -w /wheels .
+
+FROM python:3.12-slim
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        curl ca-certificates \
+    && rm -rf /var/lib/apt/lists/*
+COPY --from=client-build /wheels /wheels
+RUN pip install --no-cache-dir /wheels/*.whl polars pyarrow
+
+WORKDIR /app
+COPY python/blotter.py /app/blotter.py
+COPY blotter-demo/feed.py /app/feed.py
+COPY blotter-demo/entrypoint.sh /app/entrypoint.sh
+RUN chmod +x /app/entrypoint.sh
+
+ENV QDB_ADDR=questdb:9000 \
+    QDB_HTTP=questdb:9000
+ENTRYPOINT ["/app/entrypoint.sh"]