Make paged reads on `/query` and `/search` honest and self-describing

[thehabes]

Detailed verification report. This is the full investigation this issue and its sub-issues were written from.

---

Summary

/query and /search accept limit and skip, clamp both to server maximums, and then return the clamped page with no indication that clamping happened.
A client cannot tell a full page from a truncated one, and cannot tell "no more results" from "you asked for more than I will give you."

Both endpoints share getPagination() (controllers/utils.js:21), so this is one contract with one fix site, not two. The same helper also paginates the two Gallery of Glosses endpoints (controllers/gog.js:36,167, default 50, JWT-gated), so they inherit whatever changes here. This issue covers that shared contract. Where /search performs its paging — in application memory rather than in MongoDB — is a separate concern, tracked in the /search paging issue.

Three distinct failures fall out of that silence:

• A client that stops when results.length < limit silently drops records.
• A client that stops when results.length === 0 and advances by results.length never terminates once skip passes its maximum, because every subsequent request returns the same page.
• The deployment's configured maximums are not in effect at all — .env and the code disagree on the variable names.

The second failure is the one to look at first: the non-terminating client is the paged-query recipe published in our own API documentation (public/API.html:555), and the loop is reachable on production today with ordinary typed queries.

Every behavior below was verified against the public deployments https://store.rerum.io and https://devstore.rerum.io, not only locally.

Why this matters

Pagination is a contract. The server promises that a client can walk a result set to completion by advancing skip until the results run out. Today the server breaks that promise in whichever direction the client happens to lean:

A length < limit stop condition loses data. The client asks for limit=1000, the server decides on 500, returns 500, and says nothing. 500 < 1000 reads as "last page." The walk terminates early and the client reports success on a partial result set. Nothing on the wire distinguishes this from a genuine final page. This is how the problem was found: a client with a configurable page size above 500 walked a result set, received 500, and stopped.

A length === 0 stop condition hangs. Past the skip maximum every request returns an identical page. A client that concatenates and advances by results.length re-fetches that same page forever, growing its accumulator without bound. This is not a hypothetical client shape — it is the exact pagedQuery function we publish as the recommended pattern, and it is reachable on the production data set today.

We already knew, and documented it as folklore. public/API.html warns three times that "your application may experience strange behavior with large limits, such as ?limit=1000" (lines 507, 612, 730). The strange behavior is this truncation. A documented warning not to trigger a bug is not a fix, and nothing enforces it — a client sets its own MAX_LIMIT above the server's and the failure is silent from that point on.

Clients have no way to discover the limits. No response header carries the applied limit/skip or the server maximums. The only Link header emitted is the JSON-LD context (utils.js:163). HEAD /query returns Content-Length for the page it would have sent, not a total count (controllers/history.js:86), and it answers an empty page with a 404 where POST /query answers 200 [], so it cannot be used to probe ahead either. GET /v1/api lists endpoints and nothing else, and the checked-in OpenAPI contract does not mention limit or skip at all. A well-written client that wants to defend itself cannot.

Evidence

Verified 2026-09-02 against three deployments, all read-only POST /query and POST /search requests:

• local — the pm2 instance at localhost:3001, which reads the same collection as devstore.rerum.io (the rerum-test cluster, annotationStore.alpha)
• devstore — https://devstore.rerum.io
• store — https://store.rerum.io, production

Independently re-run twice on 2026-09-03, read-only, against localhost:3001 for source lines and endpoint behavior, with the clamp, skip-repeat, and HEAD Content-Length checks re-run against devstore.rerum.io and store.rerum.io. Every reproducible number below held: the clamp values on all three deployments, the repeating page past skip=100000 (…4c0ef016 at skip=99999, …4c0ef019 at skip=100000 through skip=500000 locally; …645fb6b6 / …645fb6b7 on production), the HEAD /query Content-Length repeat (1883 local, 1891 devstore, 1404 store), the production reachability table, every parsing edge case, and the tiling checks. The environment-variable fix was verified end-to-end: setting RERUM_MAX_QUERY_LIMIT=10 / RERUM_MAX_QUERY_SKIP=20 in the local .env and restarting pm2 made limit=1000 return exactly 10 documents and skip=20, 25, and 100 return the identical page; the .env change was then reverted and pm2 restarted again to restore the original 500 / 100000 caps. Two citations were corrected along the way: the published pagedQuery function in public/API.html begins at line 555, not 556, and the getPagination() test block is __tests__/utils.test.js:329-354, not 329-355.

Limit is clamped silently, both endpoints, all deployments

Requested limit	/query returned	/search returned
100	100	100
500	500	500
1000	500	500
5000	500	500

Identical on local, devstore, and store (limit=1000 returns 500 on all three, both endpoints). Nothing in the response headers describes pagination: the clamped response carries Allow, Content-Type, Content-Length, an ETag, the JSON-LD context Link, and the CORS headers — no applied limit or skip, no maximum, no total.

Skip is clamped to a fixed page that then repeats forever

Querying {"__rerum.APIversion":{"$exists":true}} with limit=2 and comparing the first returned id:

skip	local / devstore	store (production)
9999	…e8255df7	…a7acc9f3
10000	…e8255f68	…a7acc9f4
10001	…e8255d5e	…a7acc9f5
99999	…4c0ef016	…645fb6b6
100000	…4c0ef019	…645fb6b7
100001	…4c0ef019	…645fb6b7
150000	…4c0ef019	…645fb6b7
500000	…4c0ef019	—

The effective skip maximum is 100000 on every deployment, including production. Consecutive pages at skip=100000 and skip=100100 with limit=100 return byte-identical id lists (same md5 over the ids). Any client advancing past 100000 receives the same records indefinitely.

HEAD /query behaves identically: skip=100000 and skip=150000 report the same Content-Length for the same repeated page on every deployment (1883 local, 1891 devstore, 1404 store).

/search applies the same clamp, but it slices the page from its merged in-memory set (see the /search issue), so the repeated page only appears for a term with more than 100000 merged matches. No term tried on the dev collection reaches that — line, the, and of all return an empty page at skip=100000, 100100, and 150000 — so the documented client terminates on /search today. The mechanism is identical; only the data keeps it out of reach.

The loop is reachable on production with realistic queries

Probing store.rerum.io with limit=1 at increasing skip:

Query	skip=10000	skip=50000	skip=100000
{"@type":"oa:Annotation"}	1	1	1
{"__rerum.generatedBy":{"$exists":true}}	1	1	1
{"__rerum.APIversion":{"$exists":true}}	1	1	1
{"@type":"Annotation"}	1	1	0
{"type":"Annotation"}	0	0	0
{"type":"Thing"}	0	0	0

A document at skip=100000 means the query has more matches than the cap. {"@type":"oa:Annotation"} — every IIIF 2.1 annotation — is over the cap on production right now. {"@type":"Annotation"} is between 50000 and 100000 and growing toward it.

The documented client does not terminate

The published pagedQuery from public/API.html:555 was run verbatim — only a request counter, an iteration guard, and logging added — against the local deployment at the real caps (500 / 100000).

Query {"type":"Thing"} with limit=100, 29 matching documents. Terminates correctly:

req #1: skip=0  -> 29 docs, first=6aab775e
req #2: skip=29 ->  0 docs
=> 29 records, 29 distinct

Query {"__rerum.APIversion":{"$exists":true}} with limit=100, entered at skip=99800 so the cap is reached cheaply. Never terminates:

req #1: skip=99800  -> 100 docs, first=912509cc
req #2: skip=99900  -> 100 docs, first=4c0eef00
req #3: skip=100000 -> 100 docs, first=4c0ef019
req #4: skip=100100 -> 100 docs, first=4c0ef019
req #5: skip=100200 -> 100 docs, first=4c0ef019
req #6: skip=100300 -> 100 docs, first=4c0ef019
req #7: skip=100400 -> 100 docs, first=4c0ef019
req #8: skip=100500 -> 100 docs, first=4c0ef019
GUARD TRIPPED after 8 requests
=> 800 records accumulated, 300 distinct

From request #3 on, every page is the same page. Without the guard this does not stop and the accumulator grows without bound. The same run against a deployment with the caps temporarily lowered to limit=10 / skip=20 shows the same shape from skip=20 on: 12 requests, 110 records accumulated, 29 distinct.

Note that the published example never terminates on this query at any page size, because its only stop condition is an empty page. At limit=1000 it receives 500-record pages, advances by 500, reaches the cap after 200 requests, and then repeats forever exactly as above.

The other common client shape fails the other way. Running the same function with its stop condition changed to page.length < limit — the shape the reporting client used — at limit=1000 against the same query:

req #1: skip=0 -> 500 docs
=> 500 records (client believes the walk is complete)

500 < 1000, so the client stops and reports a complete walk of 500 records. The query matches more than 100000.

The configured maximums are inert

.env in this working tree (the deploy workflows write the servers' .env from the DEV_FULL_ENV / PROD_FULL_ENV secrets in the same shape) sets:

MAX_QUERY_LIMIT=500
MAX_QUERY_SKIP=10000

The code reads different names (controllers/utils.js:12-13):

const MAX_QUERY_LIMIT = Number.parseInt(process.env.RERUM_MAX_QUERY_LIMIT ?? 500, 10)
const MAX_QUERY_SKIP = Number.parseInt(process.env.RERUM_MAX_QUERY_SKIP ?? 100000, 10)

env-loader.js loads keys verbatim and does no prefixing, so neither RERUM_-prefixed variable is ever set and both fall back to the code defaults. limit appears configured only because 500 happens to be both the intended value and the default.

Confirmed three ways:

• Both public deployments clamp at exactly the code defaults, 500 and 100000 (tables above). If MAX_QUERY_SKIP=10000 is set on either server, it is being ignored. If a configured skip cap of 10000 were honored, skip=9999, skip=10000 and skip=10001 would return the same page; on every deployment they return three different pages.
• Adding the prefixed names to the local .env and restarting takes effect immediately: RERUM_MAX_QUERY_LIMIT=10 / RERUM_MAX_QUERY_SKIP=20 makes limit=1000 return 10 documents and skip=20, 25, 100 return an identical page. This both proves the diagnosis and validates the rename as a fix.
• The existing unit tests for getPagination() (__tests__/utils.test.js:329-354) assert only that a huge limit is clamped "below Number.MAX_SAFE_INTEGER". That passes for any cap under any variable name, which is why the mismatch was never caught. Nothing documents the keys either: .env is not tracked and the repository has no .env template, the 251 memory reins #262 description that introduced them does not name them, and they appear in neither CONTRIBUTING.md, README.md, nor .github/copilot-instructions.md.

The operator-visible consequence: someone tightened skip to 10000 in .env and the server has been allowing 100000 ever since.

Parsing edge cases

Both endpoints, from clampNonNegativeInt() (controllers/utils.js:15). Every one of these returns 200:

limit=0     -> 100      limit=abc   -> 100      limit=10abc -> 10
limit=-5    -> 100      limit=1e3   -> 1        limit=0x10  -> 100
limit=250.7 -> 250      limit=      -> 100

skip=-5     -> 0        skip=abc    -> 0        skip=1e3    -> 1
skip=2.9    -> 2        skip=       -> 0

limit=1e3 is the worst of these: Number.parseInt("1e3", 10) is 1, so the client receives a single record. A length < limit client stops after one object and reports a completed walk. The same parse makes skip=1e3 land at offset 1 rather than 1000. Non-numeric and non-positive values silently become the defaults rather than being rejected.

A repeated parameter is silently reduced to its first value, from the same parseInt call. Express 5's simple query parser (the default this app uses — app.set('query parser') is never called) hands getPagination() an array, and Number.parseInt(["100","200"], 10) coerces it to the string "100,200" and parses 100:

?limit=100&limit=200      -> 100 records
?limit=200&limit=100      -> 200 records
?skip=0&skip=1000&limit=1 -> offset 0

A bracketed parameter also silently becomes the default: ?limit[a]=5 returns 100 records, because the simple parser keeps limit[a] as a literal key and req.query.limit is never set. Under the extended parser it would instead arrive as the object {a:"5"}, parseInt would give NaN, and the fallback would produce the same 100 — so this one is insensitive to the parser choice, but only by accident. Neither case is an error today; both are cases where a client's intent and the server's behavior differ with nothing on the wire to say so.

Documented default is wrong

public/API.html:506 states the response is "limited to 10 records" by default. The actual default is 100 (getPagination(req.query, 100), called at controllers/crud.js:77 and controllers/search.js:274). A request with no limit parameter returns 100.

The machine-readable contract does not describe pagination

openapi/contracts/core-provider.openapi.yaml is the checked-in contract for this API — synced to the shared spec repository by .github/workflows/sync-rerum-shared-openapi.yml and guarded by __tests__/openapi_sync_artifacts.test.js. It defines /api/query (POST and HEAD), /api/search, and /api/search/phrase, and none of the four declares a limit or skip parameter. The words do not appear anywhere in the file, and neither does any pagination-related response header. The only reusable query parameters it defines are ExpansionGenerator and ExpansionCreator, for /id/{id}/expanded.

So a client generated from the contract cannot page at all, and a client written by hand against the contract has no reason to believe paging exists. Whatever shape the fix takes, the parameters, their maximums, and any Link header need to land here as well as in public/API.html — otherwise the honest behavior is still undiscoverable through the artifact that is supposed to be authoritative.

HEAD /query is not a count

HEAD /query?limit=2 returns Content-Length: 11832, which is exactly the byte size of the body POST /query?limit=2 returns. limit=100 gives 68396 and limit=1000 gives 949640 (the clamped 500-record page). It reports the size of one page, subject to the same clamping, and says nothing about how many records match.

It also disagrees with POST about what the end of a result set looks like. For the same request body and the same pagination parameters:

POST /query?limit=2 {"type":"NoSuchTypeAtAll"}    -> 200, []
HEAD /query?limit=2 {"type":"NoSuchTypeAtAll"}    -> 404

POST /query?limit=2&skip=100 {"type":"Thing"}     -> 200, []   (29 matches, so this is past the end)
HEAD /query?limit=2&skip=100 {"type":"Thing"}     -> 404

queryHeadRequest() treats an empty page as "no objects in the database matching the query" (controllers/history.js:99-101) rather than as an exhausted page. So the one endpoint that could cheaply tell a client where a result set ends signals that with a status code the paged endpoint never uses, and a client cannot use HEAD to probe ahead of POST without special-casing the mismatch. A HEAD response is also supposed to carry the headers its GET/POST counterpart would send; a 404 against a 200 is not that. #96 already questions whether these HEAD handlers should exist at all.

What is working correctly

/search pagination tiles consistently — five sequential limit=100 pages return exactly the same records in the same order as a single limit=500 request, and the tail terminates (on the dev collection, searching line: skip=3700 returns 100, skip=3800 returns 25, skip=3900 returns 0). Page boundaries and membership are stable, which is what this issue is about.

/query tiles as well: five sequential limit=100 pages from skip=50000 return the same 500 ids in the same order as one limit=500 request, and repeating a request returns the same page. One caveat for the fix: /query applies no sort, so page boundaries rest on MongoDB natural order. That holds in practice because RERUM marks deletions rather than removing documents, but it is not a guarantee the server makes, and a rel="next" implementation should sort on _id so that consecutive pages are deterministic by construction.

That is not the same as /search being correct. It pages in application memory, its results are not ordered by relevance, and its merge step silently drops some matching documents before paging even begins — so the "honest" tail is honest only relative to a pool that has already lost records. All three are covered in the separate /search issue.

Affected lines

File	Line	Current
controllers/utils.js	12-13	Reads RERUM_MAX_QUERY_LIMIT / RERUM_MAX_QUERY_SKIP; .env sets the unprefixed names
controllers/utils.js	15-19	clampNonNegativeInt() silently substitutes fallbacks and caps
controllers/utils.js	21-28	getPagination() returns clamped values with no report of clamping
controllers/crud.js	77, 87	/query paging
controllers/history.js	89, 91	HEAD /query paging; no total count
controllers/history.js	99-103	HEAD /query returns 404 on an empty page where POST /query returns 200 []
controllers/gog.js	36, 167	/gog/fragmentsInManuscript and /gog/glossesInManuscript paging, default 50; same helper, same clamping
controllers/search.js	274, 282	/search paging
controllers/search.js	360, 368	/search/phrase paging
utils.js	163	configureLDHeadersFor() emits only the context Link
__tests__/utils.test.js	329-354	getPagination() tests pin neither the configured cap nor the env key names
openapi/contracts/core-provider.openapi.yaml	210, 233, 241, 267	/api/query, HEAD /api/query, /api/search, /api/search/phrase declare no limit or skip parameter
public/API.html	506	Documents a default of 10; actual default is 100
public/API.html	507, 612, 730	Warns about "strange behavior" instead of the server being honest
public/API.html	555	Published pagedQuery example is the non-terminating shape

Proposed change

The goal is that a correct client never has to guess. Ordered by how much data loss each item prevents.

1. Reject an out-of-range skip instead of clamping it

Silently returning a page the client did not ask for, forever, has no defensible reading. Return 400 naming the maximum:

{
  "message": "The skip value 150000 exceeds the maximum of 100000.",
  "status": 400
}

This alone converts the non-terminating loop into an immediate, legible failure.

2. Advertise the next page with a Link header

Link: <…>; rel="next" (RFC 8288) when another page exists, omitted when the result set is exhausted. Clients follow next and stop when it is absent, which makes every length-based stop condition irrelevant.

The server can know this cheaply by over-fetching one record and trimming it before serialization:

• /query: db.find(props).limit(limit + 1).skip(skip)
• /search: $limit: limit + skip + 1 per branch, then slice as today

Access-Control-Expose-Headers: * is already set, so browser clients can read it. Note that configureLDHeadersFor() currently assigns Link as a single string — rel="next" needs to be appended to the existing context link rather than replacing it.

3. Make an over-maximum limit visible

Either reject it with a 400 the way skip is handled above, or keep clamping as a safety valve and report the applied value in a response header. Clamping is reasonable server behavior; doing it in silence is not. Rejecting is the more honest of the two and is the recommendation, though it is the more breaking of the two — worth a decision on this thread.

4. Fix the environment variable names

Either rename the keys in .env on both deploy servers to the RERUM_-prefixed form, or change controllers/utils.js to read the unprefixed names. Whichever way, the two must agree, CONTRIBUTING.md should document the keys alongside the rest of the .env contents, and a unit test should set the key and assert the cap it produces — the current tests would pass under either name. Reading both names with the prefixed one winning would avoid a coordinated deploy, if that is preferred.

5. Reject unparseable and non-positive limit

limit=abc, limit=-5, and limit=0 should be 400s, not a silent 100. limit=1e3 (and skip=1e3) should either parse as 1000 or be rejected; returning one record is the worst available option. A repeated parameter (?limit=100&limit=200) should be rejected too — Number.parseInt currently coerces the array to a comma-joined string and takes the first value, which is a guess dressed up as a result. Validating the raw parameter as a decimal integer string, and rejecting anything that is not a string, covers all of these in one place. ?limit[a]=5 is the one case that cannot be caught: under the simple parser the server never receives a limit key at all, so it is indistinguishable from a request that omitted the parameter.

6. Publish the limits

Add the effective maximums and default to the GET /v1/api response so clients can configure themselves against the deployment they are actually talking to rather than a hardcoded guess. Today that response is a plain list of endpoint descriptions and carries no pagination information.

7. Declare limit and skip in the OpenAPI contract

Add a shared PageLimit / PageSkip parameter pair under components/parameters in openapi/contracts/core-provider.openapi.yaml and reference it from /api/query (POST and HEAD), /api/search, and /api/search/phrase, with the maximums in the schema and a 400 response for out-of-range values once items 1, 3, and 5 land. If rel="next" ships, describe the Link response header there too. Without this the contract keeps describing an API that has no pagination.

8. Make HEAD /query agree with POST /query

An empty page is not a 404. HEAD /query should return 200 with Content-Length: 2 (the [] body POST would send) so that a client can probe with either verb and read the same signal. This is a small change inside queryHeadRequest(), and it is worth resolving alongside #96 rather than independently of it.

Notes

• Fixing getPagination() covers /query, HEAD /query, /search, /search/phrase, and the two /gog/*InManuscript endpoints together, which is what keeps them consistent by construction. The rel="next" work is the only part needing per-controller changes.
• public/API.html needs updating in the same PR: correct the documented default from 10 to 100, replace the "strange behavior" warnings with the actual maximums, and revise the pagedQuery example to follow rel="next" (or at minimum to stop on a short page rather than only on an empty one).
• Items 1, 3, 5, and 8 change response codes for requests that currently return 200 (or, for 8, a 404 that becomes a 200). That is a breaking change for any client relying on the clamping, and it should land on dev first with a note to known client maintainers.
• The two issues compound in one place worth noting here: because /search pages in application memory, the skip maximum is also what bounds how many documents that endpoint will pull into the API process for a single request. Lowering the cap helps both issues.
• History: getPagination() and the caps arrived in 251 memory reins #262 ("251 memory reins", merged 2026-05-07) as the fix for RERUM memory grows unbounded #251; that PR's description standardizes pagination across the CRUD, search, history, and Gallery of Glosses controllers but never names the environment variables. Query and version endpoints load unbounded results into memory #252 is still open: its first recommendation (a default and maximum limit on query endpoints) is what 251 memory reins #262 shipped, and its third (cursor-based pagination) is the rel="next" work proposed here; its getAllVersions and /history items are separate. HEAD history - since - query #96 questions whether HEAD /query should exist.

Acceptance criteria

• skip above the maximum returns 400 rather than a repeated page
• limit above the maximum is either rejected or reported in a response header — never silently applied
• Responses carry Link: …; rel="next" while more results exist, and omit it on the final page
• limit=abc, limit=-5, and limit=0 return 400
• .env keys and controllers/utils.js agree on the maximum names, verified by observing the configured cap take effect on devstore
• A unit test sets the maximum via the environment and asserts the resulting cap
• /query, /search, and the other getPagination() callers (HEAD /query, /gog/*InManuscript) behave identically for every case above
• public/API.html documents the real default, the real maximums, and a paged example that terminates correctly
• A repeated limit or skip parameter is rejected rather than silently reduced to its first value
• HEAD /query and POST /query return the same status for the same request, including on an empty page
• openapi/contracts/core-provider.openapi.yaml declares limit and skip on /api/query, HEAD /api/query, /api/search, and /api/search/phrase, with the maximums and any new response codes
• Regression tests cover the clamp boundaries and the rel="next" presence and absence