Add cursor-based continuation to `/query` so paging depth is unbounded and cost is flat

[thehabes]

Summary

skip cannot page past the configured maximum. Once #301 lands, a request past it is a 400 instead of a repeating page — honest, but it also means a client is told it may not read past 100000 records. That is a correct contract and a worse one than clients think they have today.

Keyset (cursor) continuation removes the ceiling. /query already has a unique, indexed sort key once #300 lands, so a cursor is just "resume after this _id":

db.find({ ...props, _id: { $gt: lastId } }).sort({ _id: 1 }).limit(limit + 1)

Cost is flat with depth, because Mongo seeks the index rather than counting past skip documents. Depth is unbounded, because there is no offset to cap.

skip stays. Decision recorded on the parent thread: keep both modes and document them — skip as the bounded, random-access mode, cursors as the unbounded, sequential one.

Why this matters

It is what makes the skip rejection defensible. Rejecting skip > 100000 with no alternative narrows what the API can do. Rejecting it while handing back a rel="next" link that pages indefinitely is a straight improvement. These two issues are better shipped close together than far apart.

Deep skip is expensive even inside the cap. Mongo walks and discards every skipped document. The parent draft measured /query at 340-445 ms across skip=0 through skip=99000, so this is not currently a crisis on /query — but it is linear work that a cursor makes constant, and it is the shape that matters once collections grow.

Clients cannot build a cursor themselves, and should not. idNegotiation() deletes _id from every response body (controllers/utils.js:70) and only reattaches it as id when the @context matches a known mapping. So the server must mint the token and hand it back. That is the right design anyway: an opaque, server-minted token keeps the encoding an implementation detail, and a client that only follows rel="next" needs no code change when the encoding changes.

Proposed change

The token

Add a cursor query parameter to /query. It is opaque to clients and appears only in the server's rel="next" link:

Link: <https://store.rerum.io/v1/api/query?limit=100&cursor=eyJfaWQiOnsiJHMiOiI2NmYuLi4ifX0>; rel="next"

The token encodes the last _id of the page just served. Base64url of a small JSON envelope is sufficient. Two requirements:

• Encode the BSON type, not just the value. newID() returns new ObjectId().toHexString(), so modern RERUM _id values are strings. Legacy v0 documents on the dev collection carry embedded-object _id values (roughly 6400; production has none), and some legacy documents may carry true ObjectId values. BSON type ordering gives a total order across all three, so $gt paging is correct — but a token that round-trips a bare hex string will compare against the wrong type and silently skip or repeat a block of documents at each type boundary. This is the detail most likely to be gotten wrong, and it will only fail on dev, which is where clients are developed.
• Validate and reject a malformed token with 400. Same principle as the rest of limit and skip silently guess at invalid input, and an over-maximum skip returns the same page forever #301: do not guess.

Interaction with skip

• cursor and skip together is a 400. They are two different positioning schemes and combining them has no coherent meaning.
• limit applies to both modes and is clamped identically.
• rel="next" on /query should carry a cursor, not a skip, once this lands. A client that already follows the link inherits unbounded depth with no change on its side. That transparency is the argument for shipping Paged responses carry no rel="next", so no client can tell a full page from the last page #302 first and this second.
• skip keeps working, keeps its maximum, and is documented as the random-access mode for jumping to a known offset within the cap.

Scope

/query only. See the note below on /search.

Notes

• /search is deliberately excluded. Under the current in-memory paging (controllers/search.js:282), a cursor could only encode an offset into the merged array. Same cost, same ceiling, dressed up as a cursor — worse than not shipping one, because it would advertise a guarantee the implementation does not provide. Revisit after /search paginates in application memory, so every page costs the server everything up to that page #309, at which point Atlas Search's own searchAfter / searchSequenceToken paging is the genuine equivalent. Availability on our cluster tier and server version needs confirming before that goes in any plan — do not assume it.
• Depends on /query paginates with no sort, so page boundaries rest on MongoDB natural order #300 (needs a sort key) and Paged responses carry no rel="next", so no client can tell a full page from the last page #302 (needs somewhere to put the token).
• A keyset cursor is stable under insert in a way skip is not: a document inserted before the cursor position does not shift the remaining pages. Given RERUM's mark-deleted-never-remove policy, this makes cursor walks meaningfully more correct than offset walks over a live collection, which is a second reason to prefer them beyond cost.
• This is Query and version endpoints load unbounded results into memory #252's third recommendation. Worth linking there when filed.
• HEAD /query should accept cursor too, so both verbs stay interchangeable. See HEAD /query answers an empty page with 404 where POST /query answers 200 [] #304.

Acceptance criteria

• /query accepts an opaque cursor parameter and returns the page following it
• The rel="next" link on /query carries a cursor, and a client following only that link pages past the skip maximum and terminates correctly
• Cursor paging returns every matching document exactly once across mixed _id types, verified on the dev collection where string, ObjectId, and embedded-object _id values coexist
• A cursor walk and a skip walk of the same query return the same records in the same order, within the range where skip is legal
• /query latency is flat with respect to depth under cursor paging, measured to the depth skip cannot reach
• A malformed or unparseable cursor returns 400
• cursor combined with skip returns 400
• skip continues to work within its maximum, and both modes are documented in public/API.html and the OpenAPI contract