Paged responses carry no `rel="next"`, so no client can tell a full page from the last page

[thehabes]

Summary

Nothing in a paged response says whether more results exist. A client has to infer it from the number of records it received, and both available inferences are wrong:

• Stopping when results.length < limit silently drops records, because a clamped limit produces a short page that is not a final page.
• Stopping when results.length === 0 and advancing by results.length never terminates, because past the skip maximum every request returns the same page.

The fix is to stop making clients guess. Emit Link: <…>; rel="next" (RFC 8288) while another page exists and omit it when the result set is exhausted. Clients follow next and stop when it is absent, which makes every length-based stop condition irrelevant in one move.

This is the centerpiece of the paging work. It is the only item that requires per-controller changes rather than a change to the shared getPagination() helper.

Why this matters

Both client shapes fail today, in opposite directions. From the parent draft, both run against the same query on the same deployment:

# stop on short page, limit=1000
req #1: skip=0 -> 500 docs
=> 500 records (client believes the walk is complete; the query matches more than 100000)

# stop on empty page, limit=100, entered at skip=99800
req #3: skip=100000 -> 100 docs, first=4c0ef019
req #4: skip=100100 -> 100 docs, first=4c0ef019
req #5: skip=100200 -> 100 docs, first=4c0ef019
...never terminates

With rel="next" the first client gets a next link on its 500-record page and keeps going; the second gets no next link at the end and stops. Neither has to know what the server's maximums are.

A client that wants to defend itself currently cannot. No response header carries the applied limit/skip or the maximums. The only Link emitted is the JSON-LD context (utils.js:163). HEAD /query returns the Content-Length of the page it would have sent, not a total count, and answers an empty page with 404 where POST answers 200 [], so it cannot be used to probe ahead. GET /v1/api lists endpoint descriptions and nothing else. The OpenAPI contract does not mention limit or skip at all.

We documented the problem instead of fixing it. public/API.html warns three times (lines 507, 612, 730) that "your application may experience strange behavior with large limits, such as ?limit=1000". The strange behavior is the silent truncation. Nothing enforces the warning, and a client that sets its own page size above 500 fails silently from that point on.

Affected lines

File	Line	Current
utils.js	163	configureLDHeadersFor() assigns Link as a single string — the context link only
controllers/crud.js	87	/query — .limit(limit), needs limit + 1
controllers/history.js	91	HEAD /query — same
controllers/search.js	211, 216	$limit: limit + skip per branch, needs + 1
controllers/search.js	282, 368	merged.slice(skip, skip + limit) — trim point for /search, /search/phrase
controllers/search.js	446, 540, 672	Same shape in the unmounted searchFuzzily, searchWildly, searchAlikes
controllers/gog.js	36, 167	/gog/*InManuscript, default 50

Proposed change

Know whether another page exists

Over-fetch one record and trim it before serialization. The extra record is never returned; its existence is the signal.

• /query and HEAD /query: db.find(props).sort({_id:1}).limit(limit + 1).skip(skip)
• /search and friends: $limit: limit + skip + 1 per branch, then slice as today
• /gog/*InManuscript: the equivalent in each aggregation

Emit the header

Link: <http://store.rerum.io/v1/context.json>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json",
      <https://store.rerum.io/v1/api/query?limit=100&skip=100>; rel="next"

Points worth getting right:

• Append, do not replace. configureLDHeadersFor() (utils.js:163) currently assigns Link as a single string. The context link must survive.
• Absolute URLs from process.env.RERUM_PREFIX, which is already the deployment's base (https://store.rerum.io/v1). Do not build them from the request host.
• next is absent on the final page. Absence is the termination signal, so it has to be genuinely absent, not an empty value.
• rel="prev" and rel="first" are cheap to add and make the header a complete RFC 8288 set. rel="last" requires a total count and should not be attempted.
• /query and /search are POST endpoints. The next URL carries the pagination parameters; the client re-sends the same request body to it. This is the ordinary pattern for POST-based search APIs, but it needs saying explicitly in the documentation or clients will expect a GET.
• Access-Control-Expose-Headers is already * (app.js:52), so browser clients can read the header without further CORS work.

Fix the published example

Revise pagedQuery at public/API.html:555 to follow rel="next", and replace the three "strange behavior" warnings with the actual maximums and the actual mechanism.

Notes

• Additive and non-breaking. Existing clients ignore the header and behave exactly as they do now. This is the one item in the set that can ship without client coordination, which is an argument for landing it before limit and skip silently guess at invalid input, and an over-maximum skip returns the same page forever #301 so that the skip rejection has somewhere to send people.
• Depends on /query paginates with no sort, so page boundaries rest on MongoDB natural order #300. A next link over a non-deterministic order promises more than the server can keep.
• For /search, rel="next" is honest about page boundaries but the pool it pages over is still missing documents until /search merge sorts on a field that does not exist and drops documents whose _id is an embedded object #307 lands. Both are in the minimum set for a client to walk /search completely.
• Once Add cursor-based continuation to /query so paging depth is unbounded and cost is flat #303 lands, the /query next URL should carry a cursor instead of a skip. Clients that follow the link get unbounded depth for free, without changing their code. Design the header so that swap is invisible to a conforming client.
• The Link response header needs to be declared in openapi/contracts/core-provider.openapi.yaml; see The pagination contract is undocumented, misdocumented, and absent from the OpenAPI contract #305.
• Query and version endpoints load unbounded results into memory #252 is still open and its third recommendation is cursor-based pagination, which this issue is the first half of.

Acceptance criteria

• Paged responses carry Link: …; rel="next" while more results exist, and omit it on the final page
• The JSON-LD context link is still present alongside rel="next"
• The over-fetched record is never returned to the client, at any limit including the maximum
• /query, HEAD /query, /search, /search/phrase, and both /gog/*InManuscript endpoints all emit it
• The unmounted searchFuzzily, searchWildly, and searchAlikes are updated to match, so the pattern is not carried forward when they are routed
• A client following only rel="next" walks a result set to completion and terminates, with no length-based stop condition
• Regression tests cover rel="next" presence, its absence on the final page, and the untrimmed extra record never appearing
• public/API.html publishes a pagedQuery example that follows rel="next"