perf: fast feature-frame builds and response-body release in the chunked OGC path

[thodson-usgs]

Summary

A design review of the async parallel chunking stack (planning → fan-out → pagination → shaping/combining → transport), with each candidate improvement measured against real Water Data queries. Two changes survived; everything else is listed below with the numbers that ruled it out. A deprecation of the one dead function a package-wide sweep found rides along.

1. perf(shaping): vectorized feature-frame fast paths.

Flat feature properties — every Water Data and NGWMN collection — build through pd.DataFrame rather than pd.json_normalize; nested values are found by scanning only object-dtype columns after the cheap build. An all-2D-point page builds its geometry in one vectorized geopandas.points_from_xy call over two flat coordinate lists, rather than per-feature GeoDataFrame.from_features. The spatial fast path keeps nested properties as raw dicts, which is what from_features does; the plain path still normalizes them with underscores.

Non-point or malformed geometry, null coordinates, and a geometry property column each fall back to the previous path. This CPU runs on the fan-out's event loop, so it is on every chunked call's critical path.

2. perf(transport): free page bodies once parsed, where transport owns them.

Every per-chunk aggregate stored for resume shared its first page's decompressed body, so a one-page-per-chunk fan-out held the whole download in memory until the call finished. paginate likewise pinned its first page's body for the whole walk.

A page body is now freed after it is parsed and reported, but only when transport owns the client — client is None, which covers the fan-out's ambient client. A caller who supplies their own client= keeps every page body, because those bytes are theirs to read. An aggregate is a copy with an empty body regardless of ownership; status, headers, URL and elapsed are unchanged.

Behavior change (in NEWS): the raw aggregate reached through partial_response — on ChunkInterrupted, FanOutInterrupted, and the ChunkedCall they carry — no longer carries body bytes. What it carried was one arbitrary page's fragment, not the query's data; use the returned frame.

The scope is narrower than that may read:

• Response metadata is unaffected. BaseMetadata.__init__ keeps url, query_time, header and comment and discards the response object; no subclass stores it (self.response appears nowhere in the package). A caller reading metadata sees no change.
• partial_response is the only reachable surface — the interruption and resume path, plus the waterdata/nearest.py wrapper that forwards it.
• Nothing on main documented the body. Both partial_response docstrings say "raw aggregate", where raw contrasts with finalized (not yet passed through finalize into a (df, metadata) pair), not with empty; ADR 0006 had no body clause at all. This PR adds that clause, recording both the aggregate contract and the page-body lifetime rule.

3. chore(utils): deprecate format_datetime.

A dead-function sweep over the package found exactly one orphan: it shaped qw-service responses, lost its last caller in 491eb5c3, and appears in no docs or demos. It is a public name, so it warns through _deprecation.warn_deprecated with a 2027-09-15 horizon rather than disappearing.

Benchmarks

Re-measured at the current branch head against a main worktree, on an M3 Pro, Python 3.12, pandas 3.0.5, geopandas 1.1.3. Frame builds are the median of seven runs over ten property columns, with traced peak taken in separate runs.

Frame build	main	This PR
100k flat point features	548 ms · 64.1 MiB	101 ms · 20.8 MiB	5.4x, −68% peak
50k nested point features	294 ms · 33.7 MiB	50 ms · 11.2 MiB	5.9x, −67% peak
100k features without geometry	110 ms · 58.8 MiB	84 ms · 15.5 MiB	1.3x, −74% peak
50k polygon features	479 ms · 32.2 MiB	477 ms · 32.2 MiB	unchanged (fallback)
100k flat plain frames	211 ms	75 ms	2.8x

Full call	main	This PR
Replayed 93k-row 12-chunk call, zero latency	0.84 s	0.42 s	−50% (unfanned −55%)
Peak heap, same replay, sequential arrival	165.8 MB	114.0 MB	−31%
Peak heap, same replay, concurrent arrival	165.8 MB	149.9 MB	−10%
Peak heap, unfanned 2-page walk	211.3 MB	210.5 MB	unchanged
End-to-end wall, replay with recorded latencies, fanned	4.28 s	4.19 s	−2.1%
End-to-end wall, replay with recorded latencies, unfanned	9.91 s	9.35 s	−5.7%
End-to-end wall, live cache-hot 12-chunk get_daily, median of 7 trials/arm	1.33 s	0.95 s	−29%

The unfanned peak is one 40 MB page's JSON decode spike, which neither change touches — it is bounded by a single page. The live row was measured on an earlier revision of this branch and has not been re-run; the replay CPU-path A/B measures the same −50% now as it did then, so it should still hold.

Read the two latency rows as the honest end-to-end figure: once real network latency is in play, a cold query gains a few percent. The CPU and peak-heap rows are what this PR is for — they are what a cached query, a local re-parse, or a memory-bound fan-out actually sees.

Correctness: outputs verified identical (assert_frame_equal) against from_features and json_normalize for full, missing, mixed, null and malformed geometry, polygons, nested and sparse properties, colliding property names, 3-D coordinates, single-feature pages, and a properties-level id column. 1200 offline tests at 99.02% branch coverage, plus a focused pandas-2 run; mypy --strict, ruff, xenon, complexipy and import-linter pass.

Evaluated and rejected (with numbers)

• HTTP/2 (server negotiates h2 via ALPN): consistently slower — per-request median 0.33 s (h1) → 0.42 s (h2), in both pool-of-32 and single-multiplexed-connection modes, over 15 interleaved cold-window trials.
• orjson decode (the practical "Rust" option): a further −15 pp on the CPU path on top of this PR, but ≤1% end-to-end with realistic latencies — not worth a new dependency. A custom Rust/C extension is strictly worse: pure-Python package, conda-forge feedstock, and the remaining CPU tail is already sub-second per 100k rows.
• Columnar frame construction (dict-of-lists instead of DataFrame(records)): 1.08x best case — pandas 3's record path is already good.
• Pipelined page prefetch (fetch page N+1 while building frame N): the inter-page CPU gap is ~0.4 s per boundary on 40 MB pages (~3% of a 2-page pull, less after this PR) — not worth the cancellation complexity.
• Speculative parallel pagination within a chunk: impossible — pagination is cursor-based (cursor=<opaque>, and numberMatched is absent for daily), so page N+1's URL cannot be built ahead of page N.
• Persistent client/event loop across calls: pre-first-request overhead measured at 5–6 ms; nothing to save.
• Streaming JSON parse for the transient decode spike: incremental parsers are slower, and the spike is bounded by one page.
• Fan-out dispatch layer: no change needed — 12 parallel chunks complete within ~50 ms of the slowest single request.

MRE

Offline, deterministic, no API key or quota. Compares the fast path against the fallback it replaces, on flat and on nested properties:

import time

import pandas as pd
from pandas.testing import assert_frame_equal

import dataretrieval.ogc.shaping as shaping


def features(n, nested):
    props = {
        "monitoring_location_id": "USGS-01646500",
        "parameter_code": "00060",
        "statistic_id": "00003",
        "value": "1",
        "approval_status": "Approved",
        "qualifier": None,
    }
    if nested:
        props = {**props, "qualifiers": {"code": "A", "note": "n"}}
    return [
        {
            "id": f"id-{i}",
            "properties": {**props, "time": f"2020-{1 + i % 12:02d}-{1 + i % 28:02d}"},
            "geometry": {"type": "Point", "coordinates": [-77.0 - i * 1e-6, 38.9]},
        }
        for i in range(n)
    ]


def old_spatial(feats):  # the path this PR keeps as the fallback
    df = shaping._geo_feature_frame(feats)
    df["id"] = [f.get("id") for f in feats]
    return df[["id"] + [c for c in df.columns if c != "id"]]


for label, n, nested in [("flat", 100_000, False), ("nested", 50_000, True)]:
    feats = features(n, nested)
    t0 = time.perf_counter(); old = old_spatial(feats); t_old = time.perf_counter() - t0
    t0 = time.perf_counter(); new = shaping._spatial_feature_frame(feats); t_new = time.perf_counter() - t0
    assert_frame_equal(old, new)
    print(
        f"{label:6} {n:>7} rows: from_features={t_old * 1000:6.0f}ms  "
        f"fast path={t_new * 1000:6.0f}ms  ({t_old / t_new:.1f}x, identical output)"
    )

Typical output on an M-series laptop (single run, six property columns, so faster than the table's ten-column median):

flat    100000 rows: from_features=   551ms  fast path=    83ms  (6.6x, identical output)
nested   50000 rows: from_features=   275ms  fast path=    51ms  (5.4x, identical output)

Methodology notes

• Frame-build and replay benchmarks compare this branch against a git worktree of main, one arm per process, rather than monkeypatching the old functions back in.
• Replay benchmarks drive the real fan-out/pagination machinery over 14 recorded API responses (152 MB) through httpx.MockTransport, sleeping each request's recorded wall time — exact pairing, no quota, no cache noise.
• Memory measured with tracemalloc peak, one arm per process, with response bodies freshly allocated per request.
• Live A/B trials interleaved arm order and rotated distinct time windows, because the API caches by data window.

🤖 Generated with Claude Code

https://claude.ai/code/session_014ppUeFxCydyd4n1k6TFXuq