docs: introduce Connect section, connect-string reference, QWP stubs#444
Open
bluestreak01 wants to merge 53 commits into
Open
docs: introduce Connect section, connect-string reference, QWP stubs#444bluestreak01 wants to merge 53 commits into
bluestreak01 wants to merge 53 commits into
Conversation
Consolidate how-applications-talk-to-QuestDB content under a single Connect supersection. Rewrite the ingestion overview as QWP-native "Connect to QuestDB", add a comprehensive connect-string reference at documentation/client-configuration/, and scaffold the Wire Protocols sub-section under documentation/protocols/. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
🤖 Component Converter ReminderA component in We are creating markdown correspondents of every path (e.g. questdb.com/docs/quick-start/ → questdb.com/docs/quick-start.md) for LLM consumption. Quick Check
💡 This is a friendly reminder, not a blocker. Ignore if not applicable. |
|
🚀 Build success! Latest successful preview: https://preview-444--questdb-documentation.netlify.app/docs/ Commit SHA: 60855d8
|
UDP documentation is deferred. Removes the stub page, sidebar entry, and links from ingress, egress, overview, and connect-string pages.
Closed
- Add "Why implement a QWP client" pitch and "Client lifecycle" narrative so new implementers can orient before diving into encoding details. - Spell out sequence numbering (server-assigned by receive order, not in the wire header), Gorilla first-DoD anchor, decimal scale formula (value = unscaled / 10^scale), and VARCHAR offset endianness — closes silent-wrong-guess risks for one-shot client generation. - Collapse Symbol section to WebSocket-only (per-table dict is UDP) and drop the now-stranded per-table example. - Document the practical WebSocket frame cap: http.recv.buffer.size (default 2 MiB) is the real ceiling, not the 16 MB protocol limit; exceeding it returns close code 1009 MESSAGE_TOO_BIG. - Fill out durable-ack semantics: watermark trails OK, empty messages trivially durable, reconnects discard in-flight tracking. - Note X-QWP-Client-Id may influence version selection. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/questdb/documentation into docs/connect-and-qwp-scaffolding # Conflicts: # documentation/protocols/qwp-ingress-websocket.md
- Add "Why implement a QWP query client" pitch and "Client lifecycle" narrative paralleling the ingress doc; surfaces the java-questdb-client reference impl link upfront. - Document the practical WebSocket frame cap on /read/v1: client-to-server frames (QUERY_REQUEST in particular) are bounded by http.recv.buffer.size (default 2 MiB), not the 16 MiB protocol limit; oversized frames are rejected with close code 1009 MESSAGE_TOO_BIG. - Clarify X-QWP-Max-Batch-Rows only asks for smaller batches than the server default (clamps to server's hard limit). - Tighten NULL sentinel docs: FLOAT/DOUBLE sentinel is *any* NaN (incl. 0.0/0.0); IPv4 0.0.0.0 and all-ones GEOHASH cannot round-trip as non-null. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…sage Brokers up to Connect - Add a "Use QWP for new clients" tip callout to ilp/overview.md naming QWP's wins (binary, type-rich, faster, failover, store-and-forward) and framing ILP as the path for InfluxDB / Telegraf / Kafka / Flink users who already emit ILP. - Shorter callout on ilp/columnset-types.md reframing the page as "extensions on top of the InfluxDB type model" and noting QWP exposes the full QuestDB type system natively (no suffix encoding, no casts). - Operator-facing callout on ilp/advanced-settings.md flagging this page as the legacy ILP tuning surface and pointing new deployments at QWP. - Sidebar: lift Java Embedded and Message Brokers out from under ILP (they're protocol-agnostic delivery mechanisms, not ILP sub-pages). Final Connect order: Overview, Connect string, Date to Timestamp, Client Libraries, Message Brokers, Compatibility Protocols, Java Embedded, Wire Protocols. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…eader) The egress endpoint /read/v1 is asymmetric on the wire: server-to-client frames carry the 12-byte QWP header, but client-to-server frames start directly with msg_kind — no QWP header. Including the header makes the server read 0x51 (the ASCII 'Q' of "QWP1") as an unknown msg_kind and close the WebSocket with code 1006, partway through send. Verified against server (QwpEgressUpgradeProcessor.dispatchEgressMessage calls peekMsgKind at offset 0 of the WS frame body) and Java reference client (QwpEgressIoThread.sendQueryRequest writes msg_kind as the first byte, no header). The upstream wire-egress.md spec is wrong on this point and should be filed separately. - Rewrite Message structure section with two ASCII diagrams (server-to- client with header, client-to-server without) and a warning callout naming the symptom and the reason (server keeps the header for RESULT_BATCH's flags + payload_length; client control frames have no analogous need). - Fix Example 1: drop the bogus 12-byte header from the QUERY_REQUEST hex dump; RESULT_BATCH / RESULT_END below unchanged. - Client lifecycle step 4: inline note that the binary frame body starts directly with msg_kind for client-to-server frames. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/questdb/documentation into docs/connect-and-qwp-scaffolding
New page (documentation/connect/agents.md) covering how AI agents operate QuestDB across three angles: protocols, tooling, and operational practices. Positioning: - QWP egress is the recommended path for SQL execution (DDL + streaming SELECT), with native client libraries when available and the protocol spec for clean-room implementations. - QWP ingress is the recommended path for all writes (bulk and sustained), including local-file uploads — the recipe explicitly calls out the failure mode where agents reach for read_parquet/read_csv/COPY, which require server-side filesystem access. - REST is positioned for schema discovery and small ad-hoc queries that fit in a single HTTP response. - PGWire and /imp are intentionally not recommended (superseded by QWP). - No MCP framing: an MCP server would just wrap REST + QWP without adding capability, so the page tells agents to use the underlying protocols directly. Includes a Recipes section seeded with the local-file upload recipe; links to the existing Getting Started > AI Coding Agents page for the tooling quickstart and the QuestDB / TSBS Claude skills. Sidebar: - Add Connect > Agents (between Client Libraries and Message Brokers). - Move Date to Timestamp inside Client Libraries (cross-cutting reference for all language clients). - Move Connect string inside Client Libraries as the first item (config schema shared by every QWP client). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/questdb/documentation into docs/connect-and-qwp-scaffolding
Adds imports to code examples, fixes auto_flush=off (unsupported on WS), completes the egress accessor table with all QwpColumnBatch methods, removes unreleased LONG_ARRAY references, documents timestamp_ns via ChronoUnit.NANOS, DoubleArray clear/reshape lifecycle, array column reading on egress, and execute() blocking semantics.
Every page reachable from the Connect sidebar now declares an explicit slug that mirrors its sidebar nesting. File paths are unchanged (redirects to be added in a follow-up); internal links are updated to the new URLs so the production build stays green. URL mapping: - /docs/ingestion/overview -> /docs/connect/overview - /docs/client-configuration/connect-string -> /docs/connect/clients/connect-string - /docs/ingestion/clients/* -> /docs/connect/clients/* - /docs/ingestion/message-brokers/* -> /docs/connect/message-brokers/* - /docs/ingestion/ilp/* -> /docs/connect/compatibility/ilp/* - /docs/query/pgwire/* -> /docs/connect/compatibility/pgwire/* - /docs/query/rest-api -> /docs/connect/compatibility/rest-api - /docs/ingestion/import-csv -> /docs/connect/compatibility/import-csv - /docs/query/export-parquet -> /docs/connect/compatibility/export-parquet - /docs/ingestion/java-embedded -> /docs/connect/java-embedded - /docs/protocols/* -> /docs/connect/wire-protocols/* Changes: - 36 frontmatter slug additions on Connect pages. - ~306 internal markdown link rewrites across ~90 files in documentation/ (pages, partials, admonitions). - Hardcoded URLs updated in src/components/Resources/index.tsx, src/modules/integration/index.tsx, shared/clients.json, shared/ilp_clients.json. - Fixed pre-existing broken link: 5 client pages had [Configuration string](/docs/.../configuration-string/), now point at the real connect-string page. External inbound links still point at the old URLs and will 404 until redirects are configured (planned follow-up). The production build (CONTEXT=preview yarn build) now succeeds with zero broken-link errors. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Go client page was still the legacy HTTP/ILP, insert-only reference. Rewrite it to document the QWP (WebSocket) path, mirroring the in-flight Java client page as the per-language template: client creation, QWP ingestion, the new QwpQueryClient query API, error handling, store-and-forward, and multi-host failover. Exhaustive connect-string keys, protocol details, and HA concepts are deep-linked to the already-landed connect-string, protocols, and high-availability pages rather than duplicated. The page is written so an autonomous coding agent can build a correct Go application from this page alone. The caveats that otherwise cause data loss, corruption, or panics are stated inline where code is copied: the asynchronous ingestion error model, query-batch buffer aliasing, single-goroutine concurrency, the comma-ok QwpSender assertion, store-and-forward blocking and HALT, and Exec not being retried across a reconnect. Simple single-host idioms are the default; the extra rules for multi-host failover are consolidated into a short checklist at the top of the failover section and signposted from the top of the page, so neither audience pays for the other. Also add a documentation changelog entry. The featured example blocks use RemoteRepoExample and render once the matching Go client examples land on go-questdb-client/main. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The RemoteRepoExample component read example.code without optional chaining, so when an example was absent from the remote-repo-example plugin data it threw "Cannot read properties of undefined (reading 'code')" and failed static-site generation for the entire locale. This contradicted the plugin's own design: plugins/remote-repo-example already skips a missing example file with a warning and continues, treating an absent example as a tolerable, transient state (docs can land before the upstream example does). Guard the lookup: when the example is missing, log a warning in the same style as the plugin and render nothing, so the block appears once the example lands upstream instead of breaking the build. This unblocks the Go QWP client page, which references qwp-ingest/go and qwp-query/go before they exist on go-questdb-client/main. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Establishes a transport-agnostic home for at-least-once / exactly-once
semantics so QWP client docs no longer depend on legacy ILP pages
scheduled for removal.
- New documentation/concepts/delivery-semantics.md covering the three
replay paths (client retry, multi-host failover, SF restart), the
designated-timestamp + DEDUP recipe for exactly-once, and when
at-least-once alone is acceptable.
- Cross-link from connect-string.md DEDUP warning and SF replay note,
client-failover/concepts.md ingress section, and
store-and-forward/concepts.md .ack-watermark section.
- rust.md: replace the legacy /docs/connect/compatibility/ilp/overview
anchor with the new concepts page.
- connect-string.md: rename the misleading {#egress-flow} anchor on the
ingress Durable ACK section to {#durable-ack}; add a stable
{#egress-failover} anchor on the actual egress-failover subsection.
Repoint client-failover/configuration.md references that previously
landed on durable-ack content while claiming to document egress
failover.
- sidebars.js: insert Delivery semantics in Core Concepts after
Deduplication.
- review-client skill: add checklist item 18 (no content dependencies on
legacy ILP pages) and a style-guide exception clarifying that legacy
links don't count as coverage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A client-documentation review of the Go QWP page surfaced several correctness and completeness gaps, each verified against the go-questdb-client source and the QWP spec rather than inferred: - Fix a broken cross-reference: the "durable ACK keys" link pointed at connect-string#egress-flow, an anchor that does not exist. The section is #durable-ack; the reader was being dropped at the page top. - Document how to write NULL on ingest. The client has no null setter; a cell is null when its column setter is omitted before At/AtNow, and a column introduced on a later row is backfilled with null for earlier buffered rows. This was previously undocumented. - Warn that store-and-forward / failover replay is at-least-once. Without DEDUP UPSERT KEYS on the target table, replay produces duplicate rows. This hazard was prominent in the connect-string reference but absent from the client page. - State that there is no per-transition connection callback. Connect, disconnect, reconnect, and failover are not delivered as events; reconnect is observable only via the counters and terminal failures via the error handler. - Add the failover_max_duration_ms key (default 30000, 0 = unbounded) and the WithQwpQueryFailoverMaxDuration option to the query-failover table. This documents behaviour added in go-questdb-client PR #62; the doc is accurate once that PR merges. The temporary plugins/remote-repo-example/index.js preview change is intentionally left uncommitted. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds checklist items 19-22 (Enterprise/OIDC + mTLS coverage, exhaustive bind-parameter type enumeration, cross-page capital-markets schema consistency, field-level error diagnostic surface). Removes go.md from the legacy-ILP list now that it's a QWP page. Switches the Step 4 output format from grouped-by-section to ordered by severity (Missing → Partial → Covered) so action items surface first; checklist section becomes a parenthetical tag per finding instead of a sub-heading. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Grounded against the implementation at c-questdb-client. - Replace bare OIDC pointer with an explicit "not implemented" note covering token acquisition, passing via `token=`, and rebuild-on-expiry. - Add explicit "mTLS not supported" callout under TLS. - Add a Concurrency subsection (single-owner Sender, decoupled Buffer, distinct sender_id per slot). - Replace the sample column-setter list + crate-docs handwave with a complete table covering all 20 typed setters, the NULL variants, and a worked `column_f64_opt` example. Surface the IPv4 / LONG_ARRAY client-encodes-but-server-rejects caveat with a footnote. - Correct the `sf_durability` rows: only `memory` ships; `flush` and `append` are reserved (setting them today fails sender construction). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…load rejection - Add ### Backpressure on flush() under ## Flushing: the two stacked caps (max_in_flight wire window, sf_max_total_bytes queue cap), block semantics, sf_append_deadline_millis as the bounded deadline, SubmitTimedOut on timeout, never-drops invariant, and memory ≡ disk symmetry. Verified against c-questdb-client qwp_ws.rs:355-399 and qwp_ws_sfa_queue.rs:891-910. - Add a :::caution covering the PayloadExceedsByteCapacity error path: a single payload exceeding sf_max_bytes is rejected immediately, does not enter the backpressure wait. The Rust analog of Java's PAYLOAD_TOO_LARGE sentinel. - Update the existing Flushing intro to forward to the new section. - Fix the sf_append_deadline_millis row description in the SF tuning keys table: applies in both memory and disk modes (the SFA queue is shared), not only in `append` mode (which does not ship). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…lation Closes the field-level documentation gap on QwpWsSenderError. Three new subsections under the field table: - **Message stability** — message is opaque. Server-supplied text comes from QuestDB's SQL error pipeline (reworded across versions) and is capped at 1024 bytes by the wire spec; WebSocket protocol violations produce a client-synthesized "ws-close[<code>]: <reason>". Application code must dispatch on category and status, never pattern-match on message. - **PII / secret safety** — message may quote application payload (an offending value from a schema/parse rejection) or a server-supplied WebSocket close reason. Treat as potentially PII-bearing; sanitize before forwarding to external error trackers. The other fields are structural metadata and safe to forward as-is. - **Correlating with server-side logs** — the WebSocket upgrade does not carry a server-issued request/connection ID. The closest correlation tuple is (message_sequence, from_fsn, to_fsn). Document the bug-report tuple (connection start time, X-QWP-Client-Id, triple). Explicit "there is no globally unique handle." Also annotates table rows: category note about programmatic dispatch; message_sequence note about per-connection scope and reconnect reset; from_fsn/to_fsn note that it's the client-side span. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A review-client pass over the Go QWP client documentation found three actively-misleading statements and several coverage gaps. go.md corrections: - The result-batch caution claimed Float64Array, Int64Array, and the *Range accessors alias the receive buffer. They return caller-owned copies; only Str and Binary alias. Rewrote the caution accordingly. - request_durable_ack=on is rejected by the Go connect-string parser today; added a caution instead of instructing readers to set it. - The custom-trust-store section pointed at tls_roots keys the Go client rejects. Replaced it with accurate trust-store/mTLS behavior, a static-token OIDC acquisition/expiry note, and a worked TLS+auth+multi-host example. go.md gap fills: - Full SenderError field table (status byte, message sequence, message length/stability/PII guidance). - QwpQueryError.RequestId and the 0x09 WRITE_ERROR row. - Called out BINARY/ARRAY/IPv4 as non-bindable and documented the DECIMAL64/256 and GEOHASH result-read paths. - Aligned the in-page schema (CREATE TABLE column names, decimal example table) so examples no longer contradict each other. - Added a counter-polling example for connection observability. connect-string.md: - Documented the egress/query-client keys (compression, compression_level, initial_credit, max_batch_rows, buffer_pool_size) so the Go page's references resolve. - Noted that tls_roots support varies by client. Excludes the local-only plugins/remote-repo-example/index.js preview change, which is marked to be reverted before merge. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Cross-client schema misalignment
|
The SenderError field table committed in d302cf3 described MessageSequence as "the correlation key for support tickets and server-log matching". The Rust QWP page (commit 7d0efa2) instead documents the same wire field as connection-scoped and reset on reconnect, with an explicit statement that the protocol surfaces no server-issued request or connection id. The two client pages told a reader porting between them contradictory stories; the cross-client follow-up posted on PR #444 flagged this. Verified against the Go client and the QWP spec that Rust's framing is the accurate one for Go too: SenderError.MessageSequence carries the raw per-connection wireSeq (qwp_sf_send_loop.go), fsnAtZero is rebound per connection, and only the client-set X-QWP-Client-Id header exists on the upgrade — there is no server-issued request id. Reworded the MessageSequence row to state it resets on reconnect and is meaningful only within one connection, and added a correlation note after the table (closest handle is the (MessageSequence, FromFsn, ToFsn) tuple plus connection start time) mirroring the Rust page so the two now tell the same story. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…pported Addresses two QWP-client review gaps on the connect-string reference: the Authentication section was silent on OIDC, and the existing mTLS pointer conflated server-cert trust with client identity. OIDC is now documented as a pass-through (static bearer in `token=`, app rotates), and mTLS is stated flatly as unsupported by the server in both the Authentication and TLS sections. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bind-parameter table, adds null-by-omission docs, error diagnostic fields (FSN, message sequence), sender_id uniqueness, sf_append_deadline, and data model consistency. Adds builder-to-connect-string equivalence note on all three pages. Moves mTLS non-support from Rust to the connect-string TLS section. Fixes misleading "mutual TLS" reference.
OIDC is a server-side concern, not a client API. Removes OIDC partial, inline OIDC guidance, and library hints from Java, Go, and Rust pages. Removes OIDC subsection from connect-string reference. Recommends token auth over basic auth for Enterprise on all pages. Fixes Go production example to use token instead of username/password and explains the target key. Fixes broken connect-string links (slug vs filesystem path).
Single-endpoint failover does not retry (rotates, not retries). Query client enters terminal state after failover exhaustion (must recreate). Ingress/egress failover asymmetry documented. onFailoverReset is mid- stream only. target=replica needs N+1 replicas. Go production example uses three endpoints. All findings verified against client source code.
… claims Adds end-to-end example combining ingestion (builder + TLS + token + multi-host + connection events + DoubleArray) and querying (connect string + recreate-on-failure pattern + onFailoverReset with correct QwpServerInfo accessors). Enterprise features marked with inline comments. Adds builder enterprise example in Creating the Client. Reverts unverified single-endpoint and terminal-state additions from Go and Rust pages (not verified against those client codebases).
Verified against c-questdb-client ia_qwp_ws branch. Adds warning that sf_dir is strongly recommended for multi-host (flush blocks without it), env_logger example for observing reconnect events, and full tested failover example with retry pattern around flush() and must_close().
Adds storeAndForwardDir and senderId to the ingestion builder example. Updates trailing comments to explain SF vs memory-only behavior during outages.
…on caution Java: adds store-and-forward to the full enterprise example with builder API, connection events, and egress reconnect-on-failure pattern. Removes outdated "requires QuestDB 9.2.0" caution on decimals (QWP itself requires QuestDB 10). Rust: adds SF recommendation for multi-host, env_logger hint for observing reconnect events, and full tested failover example with flush() retry pattern and must_close().
…ample Verified by agent testing against Go client. Documents egress terminal state after failover exhaustion with recreate-on-failure pattern. Adds single-endpoint failover warning. Notes ingress/egress reconnect asymmetry. Full tested example: ingestion with SF + options API + observability counters, and querying with QwpFailoverReset handling. Enterprise features marked with inline comments.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Consolidates how-applications-talk-to-QuestDB content under a single Connect supersection. The native QWP clients, the compatibility protocols (ILP / PGWire / REST), and the wire-protocol specifications now share one home. The connect-string reference at
documentation/client-configuration/connect-string.mdanchors the section as the comprehensive knob doc.What's new
Sidebar restructure
Connectsection absorbs Client Libraries, Compatibility Protocols, and Wire Protocols.Connect → Client Libraries.Connect → Compatibility Protocols.Connect → Wire Protocolsfor the client-implementer audience.Connect → Overview (
documentation/ingestion/overview.md)Connect → Connect string reference (
documentation/client-configuration/connect-string.md, new)*Applies to: ingress.*/*Applies to: ingress and egress.*).sf_dirpath handling (no shell expansion, parent must exist).sender_idvalidation (allowed characters, lock-collision semantics).Wire Protocols scaffolding (
documentation/protocols/, new)Touch-ups
#first-party-clients→#client-librariesincapacity-planning.md,monitoring-alerting.md,datatypes/overview.md, andsrc/components/Resources/index.tsx.date-to-timestamp-conversionfrontmatter description from a bare language list to an actual summary.Project handoff (
ONBOARDING.md)Bundle B / A / C team handoff document captured at repo root — assignments, coordination rules, first-prompt templates for the remaining work streams (Wire-protocol content, Client failover, Store-and-forward concepts). Reviewers can opt to keep or remove from this PR.
Out of scope (intentionally)
Connect → Compatibility Protocols → ILP. The connect-string reference is QWP-native; ILP-specific knobs (protocol_version,retry_timeout,request_timeout,request_min_throughput) are not duplicated.Known follow-ups
sf_durability=flush|append, theon_*_errorfamily) so LLM auto-completion is less likely to surface them as available.ingestion/clients/*→clients/*) and HTTP-side URL redirects in a separate restructure PR.Test plan
yarn build— 353 pages generated, no broken links / anchorstarget/zoneframing matches the QuestDB Enterprise feature storyONBOARDING.mdshould remain at repo root🤖 Generated with Claude Code