[RUM] New 400s on the direct-to-instance operations API (:9925) — a rejected poll re-fires every ~60s for 30 minutes

[dawsontoth]

Summary

Datadog RUM shows a step-up in HTTP 400s on the direct-to-instance operations API — POST https://<instance-host>:9925/. 26 in the last 24h against a ~1–3/day baseline.

This is a different code path from #1569, which tracks 400s on the proxied POST /HDBInstance/{id}/operation on fabric.harper.fast. These go straight to the customer's instance host on port 9925, i.e. the non-forceFabricConnect client. Filing separately for that reason; they may well share a root cause.

Surfaced by the automated daily RUM review (app f590deee-…).

Datadog findings

@type:resource @resource.status_code:400 @resource.url_path:"/" @resource.url_port:9925

Window	400s	Sessions
24h → now	26	3
48h → 24h ago	2	1
7d → 48h ago	6	4
14d → 7d ago	14	6

• Reach: 3 sessions across 3 distinct customer orgs / 4 instance hosts (IDs and hostnames omitted — customer resources). So not one bad instance.
• All on v2.156.4 (current prod).
• Server behavior: fast, uniform rejections — median ~122ms, content-type: application/json. A validator/parser saying no, not work that failed partway.
• The matching JS error stream is AxiosError: Request failed with status code 400, 26 events — up from 2 in the prior 24h.

The shape that matters

Three distinct clusters, and the repetition is the point:

1. Applications page — /{org}/{cluster}/apps — 8 errors, re-issuing the same rejected operation every ~60s for ~31 minutes (03:00 → 03:31 UTC). It never gave up.
2. Instance overview — /{org}/{cluster}/instance/{id} — 16 errors in 55 seconds (10:59:23 → 11:00:19).
3. Browse / data — /{org}/{cluster}/databases/data/<Table> — 2 errors, 33s apart, against two different clusters in one session.

In every cluster the errors arrive in near-simultaneous pairs (~5–40ms apart), so two subscribers appear to be issuing the same rejected request.

RUM cannot see the request body, so the operation name is not directly observable.

Why this is worth fixing

A 400 will be rejected identically forever, but nothing in the client treats it that way:

• pollUnlessForbidden (src/react-query/pollUnlessForbidden.ts) is 403-only, so the refetchInterval timer keeps firing.
• The retry predicate was also 403-only, so each doomed tick cost up to 4 requests on the callers using default exponential backoff.
• Each one is a console.error + a toast through the global handler in queryClient.ts.

#1601 addresses the retry half (stop retrying 400s) and deliberately leaves the poll timer running, because halting it would freeze a poll whose 400 came from state that is still settling. The remaining question — should a sustained 400 stop the timer, and which call sites can safely do that — is still open, and is the same question #1569 raises.

Candidate root causes

Not confirmed. Because these are direct-to-instance calls (port 9925, not the fabric proxy), the most likely candidates are operations whose availability depends on the target's Harper build:

1. An operation the instance's Harper version doesn't implement, on a route that is nav-gated but not route-gated (a bookmark or back-nav mounts the component and starts its poll anyway).
2. get_analytics-style calls carrying metric names or bucket_ms that a stricter/older validator rejects.
3. The apps page's ~60s component/deployment poll sending an argument the instance rejects.

The ~60s cadence in cluster 1 is a useful fingerprint for whoever picks this up — it should identify the query directly.

Suggested next steps

1. Identify the ~60s poll on the applications page and the burst source on the instance overview page; confirm the operation name server-side.
2. Version-gate the route (not just the nav link) if it turns out to be a capability mismatch.
3. Decide the poll-timer policy for a sustained 400 (see [RUM] New 400s on POST /HDBInstance/{id}/operation — ~41% of that endpoint's calls in one session, and a 400 never stops the poll #1569 and fix(react-query): stop retrying polls the server deterministically rejects #1601).

Filed by the automated daily Datadog RUM review. No PII; customer org/cluster/instance identifiers and hostnames deliberately omitted.