diff --git a/reference/mqtt/overview.md b/reference/mqtt/overview.md index c09157fb..7eec70b8 100644 --- a/reference/mqtt/overview.md +++ b/reference/mqtt/overview.md @@ -55,7 +55,7 @@ Harper supports multi-level topics for both publishing and subscribing: -A durable session retains a client's subscription list and any unacknowledged messages across disconnects. When the client reconnects with the same client ID, it picks up from where it left off — including any messages published while it was offline. +A durable session retains a client's subscription list and any unacknowledged messages across disconnects. When the client reconnects with the same client ID, Harper catches up subscribed topics from retained audit history, picking up from where it left off. Durable sessions in Harper are persisted as records in the `hdb_durable_session` system table, indexed by client ID. The session record holds the list of subscriptions (topic + QoS) and, per topic, the position in the transaction log from which a reconnect catches up. Because durable sessions are records rather than in-memory state, an abandoned session sits idle with no runtime cost until the client reconnects or the record is deleted. @@ -73,7 +73,7 @@ mqtt.connect('mqtts://harper.example.com:8883', { }); ``` -**Catch-up on reconnect** — When the client reconnects, Harper replays missed messages on subscribed topics by reading the audit log. For this to work, audit logging must be enabled on the tables backing the subscribed topics. See [Transaction Logging](../database/transaction.md) and [`logging.auditLog`](../logging/configuration.md#loggingauditlog). Catch-up applies to QoS 1 and 2 subscriptions. A QoS 0 subscription is kept with the session and resumes live, without the messages published while the client was away. +**Catch-up on reconnect** — When the client reconnects, Harper replays missed messages on subscribed topics by reading retained audit history. For this to work, audit logging must be enabled on the tables backing the subscribed topics. See [Transaction Logging](../database/transaction.md) and [`logging.auditLog`](../logging/configuration.md#loggingauditlog). Catch-up applies to QoS 1 and 2 subscriptions. A QoS 0 subscription is kept with the session and resumes live, without the messages published while the client was away. **When a session can no longer catch up** — As of v5.4.0, Harper checks each subscription's saved position against the database's history before replaying from it. When audit retention has since removed history the replay would need, Harper resets the session instead of replaying only part of what was missed: @@ -85,6 +85,10 @@ A position saved before the database was restored or copied, or saved on another Harper keeps a quiet topic's position current while the client is connected, and again when it disconnects, so a subscription that receives nothing is not reset merely because other tables' retention advanced. A position only moves past a message once that message, and every other message from the same transaction, has been acknowledged; an out-of-order acknowledgement can mean some messages are delivered again after a reconnect. A client that sends SUBSCRIBE again for a topic its session already holds, as many clients do on every connect, continues from the saved position, so resubscribing loses nothing. A new connection with the same client ID on the same Harper thread takes the session over, and the older connection is closed (MQTT v5 reason code `0x8E`); a clean-start connect closes it the same way. These checks apply to RocksDB databases; LMDB sessions catch up as before. + + +Durable subscriptions with QoS 1 or 2 explicitly include superseded record versions during both live delivery and reconnect catch-up. This preserves delivery of intermediate retained publications (`retain: true`), even though ordinary table subscriptions now default to current-state delivery. Non-retained publications remain independent messages. Catch-up is limited by audit-log retention. See [Superseded record updates](../resources/resource-api.md#superseded-record-updates). + **Session expiry** — In MQTT v5, the `sessionExpiryInterval` property on `CONNECT` controls how long the session is retained after the client disconnects. With `sessionExpiryInterval: 0` (or a clean session connect), Harper deletes the session record at disconnect. Connecting with the same client ID and `clean: true` also explicitly deletes any existing durable session. **Distributed durable sessions** — Harper does not currently maintain a single logical durable session across multiple cluster nodes; a durable session record lives on the node where it was created. Clients that may reconnect to different nodes (e.g., behind a load balancer with round-robin DNS) should use sticky routing or terminate on a fixed node. @@ -107,7 +111,7 @@ Harper is designed for distributed, low-latency message delivery. Messages are d In a distributed cluster, messages may arrive out of order due to network topology. The behavior depends on whether the message is retained or non-retained: -- **Retained messages** (published with `retain: true`, or written via PUT/upsert) maintain eventual consistency across the cluster. Harper keeps the message with the latest timestamp as the winning record state. An out-of-order earlier message will not be re-delivered to clients; the cluster converges to the most recent state. +- **Retained messages** (published with `retain: true`, or written via PUT/upsert) maintain eventual consistency across the cluster. Harper keeps the message with the latest timestamp as the winning record state. Ordinary subscriptions skip superseded updates. Durable QoS 1/2 subscriptions include superseded versions, so their delivered history can include older updates even though the stored record converges to the most recent state. - **Non-retained messages** are always delivered to local subscribers when received, even if they arrive out of order. Every message is delivered, prioritizing completeness over strict ordering. **Non-retained messages** are suited for applications like chat where every message must be delivered. **Retained messages** are suited for sensor readings or state updates where only the latest value matters. diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 36e7a844..da064a25 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -238,15 +238,44 @@ Called for MQTT subscribe commands. Returns a `Subscription` — an `AsyncIterab All properties are optional: -| Property | Description | -| -------------------- | --------------------------------------------------------------------------------------------------------- | -| `includeDescendants` | Include all updates with an id prefixed by the subscribed id (e.g. `sub/*`) | -| `startTime` | Start from a past time (catch-up of historical messages). Cannot be used with `previousCount`. | -| `databaseGeneration` | Resume from `startTime` as a checked position. See [Resuming from a position](#resuming-from-a-position). | -| `previousCount` | Return the last N updates/messages. Cannot be used with `startTime`. | -| `omitCurrent` | Do not send the current/retained record as the first update. | -| `rowFilter` | Synchronous JavaScript predicate applied to authoritative row values. | -| `eventFilter` | Synchronous JavaScript predicate for events that may not carry an authoritative row. | +| Property | Description | +| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `includeDescendants` | Include all updates with an id prefixed by the subscribed id (e.g. `sub/*`) | +| `startTime` | Resume after a local audit-log cursor; all events at that cursor are excluded. Cannot be used with `previousCount`. | +| `databaseGeneration` | Resume from `startTime` as a checked position. See [Resuming from a position](#resuming-from-a-position). | +| `previousCount` | Return up to the last N accepted updates/messages, capped at 1,000. Cannot be used with `startTime`. | +| `includeSuperseded` | Include record mutations superseded by a later version. Defaults to `false`, or `true` when `rawEvents` is enabled. An explicit value overrides that default. | +| `rawEvents` | Return raw audit events rather than reconstructing full records. Defaults to `false`. | +| `omitCurrent` | Do not send the current/retained record as the first update. | +| `rowFilter` | Synchronous JavaScript predicate applied to authoritative row values. | +| `eventFilter` | Synchronous JavaScript predicate for events that may not carry an authoritative row. | + +#### Superseded record updates + + + +By default, table subscriptions skip record mutations whose version differs from the current stored record's version. This applies both to catch-up with `startTime` or `previousCount` and to live delivery. If a record was updated at versions 2, 3, and 4 before catch-up begins, and version 4 is still current, the subscription delivers only version 4. Deletion events require a retained tombstone with a matching version; older updates or deletions superseded by a later write are skipped. If the record or its tombstone has been removed, record mutations are skipped because no current version remains to compare. Published messages are independent events and are not suppressed by this version check. The comparison uses the stored primary entry's metadata version, which publishing also advances. A publication can therefore cause a preceding record mutation to be skipped; this is the existing live-delivery rule, not a separate timestamp tracking only record mutations. + +`startTime` is a local audit-log cursor, not a record version. Audit events expose this cursor as `localTime`, which multiple events in a transaction can share. Resuming with `startTime` excludes all events at that cursor, so callers must account for shared cursors before checkpointing. + +`previousCount` counts accepted events: superseded or filtered events do not consume the requested count. The request is capped at 1,000 events, and scanning stops after 10,000 in-scope audit entries or the end of retained history, so fewer events may be returned. + +Before v5.3.0, non-raw catch-up could include superseded mutations even though non-raw live delivery suppressed them. Consumers that need historical transitions can opt in: + +```javascript +const subscription = await Product.subscribe({ + startTime: lastLocalTime, + includeSuperseded: true, +}); + +for await (const event of subscription) { + await processEvent(event); +} +``` + +With `includeSuperseded: true`, record values are reconstructed for each historical version, unless `rawEvents` is enabled. This option applies to live delivery too. Delivery still depends on retained audit history and subscription filters; it does not provide an exactly-once guarantee. + +`rawEvents: true` preserves its existing default of including superseded events. Set `includeSuperseded: false` explicitly to apply the same version and missing-record checks while retaining raw event payloads. #### Resuming from a position diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index d7186c99..4dda6765 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -24,7 +24,7 @@ Read these before upgrading a production node or cluster. - **Each node rewrites `/ssh/config` once at startup** to add `# BEGIN/END harper ssh key ` lines around each key's block. An ssh config damaged by the pre-5.3 block-matching bug is not repaired automatically: run `get_ssh_key`, then `delete_ssh_key`, then `add_ssh_key` with the returned sealed key and the correct `host`/`hostname`. See [SSH Key Blocks Are Marked in the ssh Config](#ssh-key-blocks-are-marked-in-the-ssh-config). - **npm installs that use S3 export or import must add `@aws-sdk/client-s3` and `@aws-sdk/lib-storage`.** The AWS SDK is now an optional peer dependency. Without it, `export_to_s3` and `import_from_s3` return `501` with the install command. The official Docker image still includes it ([harper#2609](https://github.com/HarperFast/harper/pull/2609)). See [`export_to_s3`](/reference/v5/operations-api/operations#export_to_s3). - **Restoring a backup taken with `exclude_blobs` requires `allow_engine_only`.** Such a restore previously succeeded with only a log warning and could leave records pointing at the wrong blobs. A scripted engine-only restore, including one into a new `target_database`, fails until the flag is added ([harper#2646](https://github.com/HarperFast/harper/pull/2646)). -- **Subscription catch-up delivers only the current version of each record by default.** Pass `includeSuperseded: true` to `Table.subscribe` to also receive the retained history. `rawEvents: true` still includes every event, and durable MQTT subscriptions still receive every publication ([harper#2767](https://github.com/HarperFast/harper/pull/2767)). +- **Subscription catch-up delivers only the current version of each record by default.** Pass `includeSuperseded: true` to `Table.subscribe` to also receive the retained history. `rawEvents: true` still includes every event, and durable MQTT subscriptions with QoS 1 or 2 still receive every retained publication except one gap closed in 5.4 ([harper#2767](https://github.com/HarperFast/harper/pull/2767)). See [Subscription Catch-up](#subscription-catch-up). - **`searchByIndex` and a custom index's `search()` take an options object.** A positional fifth argument to `searchByIndex` now throws a `TypeError` instead of silently permitting the scan it was meant to forbid ([harper#2187](https://github.com/HarperFast/harper/pull/2187)). - **Startup waits at most `deployment.startupInstallTimeout` for component installs** before opening listeners. The default is 10 minutes, and `0` restores the previous unbounded wait. An install that runs past the limit finishes in the background, and when it succeeds Harper requests a restart to load it ([harper#2784](https://github.com/HarperFast/harper/pull/2784)). - **Some requests Harper used to accept are now refused:** `set_configuration` with unrecognized parameters ([harper#2272](https://github.com/HarperFast/harper/pull/2272)); a deploy or `drop_component` whose root config change `HARPER_CONFIG` or `HARPER_SET_CONFIG` would undo at the next start, with a `409` ([harper#2801](https://github.com/HarperFast/harper/pull/2801)); and `Table.clear()` on a table with an active full-text declaration, with a `501` ([harper#2616](https://github.com/HarperFast/harper/pull/2616)). @@ -78,6 +78,12 @@ Every replication retry now follows one jittered backoff schedule. Subscription `update_node` now works; the documented operation was never registered. A request that changes only `revoked_certificates` or `shard` updates the node record locally without resetting replication ([harper-pro#916](https://github.com/HarperFast/harper-pro/pull/916)). See [Update Node](/reference/v5/replication/clustering#update-node). A clone's wait for the initial sync is now bounded by the data size the leader reports and resumes across restarts, so a large clone no longer times out on a fixed budget or starts over ([harper-pro#661](https://github.com/HarperFast/harper-pro/pull/661)). +## Subscription Catch-up + +Table subscriptions now skip superseded record updates during catch-up with `startTime` or `previousCount`, matching live delivery. A record updated at versions 2, 3, and 4 is delivered only at version 4 if that version is still current. The comparison uses the stored primary metadata version: a later publication can suppress a preceding mutation, and removing a record or tombstone makes its mutations ineligible. **JavaScript consumers that need retained mutation history, including these cases, must set `includeSuperseded: true`.** Raw-event subscriptions retain their existing default; `rawEvents: true` includes superseded events unless `includeSuperseded: false` is explicit. + +MQTT durable subscriptions with QoS 1 or 2 explicitly include superseded versions in live delivery and reconnect catch-up, preserving intermediate retained publications — except that a saved position could still skip an unacknowledged message sharing a transaction with one already acknowledged; 5.4 closes that gap (see [Durable MQTT Sessions No Longer Replay Short](/release-notes/v5-lincoln/5.4#durable-mqtt-sessions-no-longer-replay-short)). Non-retained published messages remain independent and are not suppressed by the table version check. See [Superseded record updates](/reference/v5/resources/resource-api#superseded-record-updates) and [MQTT durable sessions](/reference/v5/mqtt/overview#durable-sessions). + ## Node Identity `node.hostname` and `replication.hostname` must now be a bare hostname or IP literal. This value is the node's identity — it becomes the node's TLS certificate common name and the host that replication advertises to peers and dials to reach the node — so a URL or `host:port` value corrupted certificate matching and replication without surfacing an error: a node configured as `http://host:9926` advertised and dialed a host literally named `http` ([harper#2218](https://github.com/HarperFast/harper/issues/2218)).