Skip to content

Cache /historical per symbol; 503 + Retry-After on PSX rate limits - #5

Merged
mtauha merged 9 commits into
mainfrom
feat/historical-cache
Sep 30, 2026
Merged

mtauha merged 9 commits into
mainfrom
feat/historical-cache

Conversation

@mtauha

@mtauha mtauha commented Sep 30, 2026

Copy link
Copy Markdown
Member

Summary

On 2026-09-29 (11:57–12:32 UTC) GET /stocks/{symbol}/historical returned 324 × 500. One client was crawling ~210 symbols at 2–3 req/s. The combined traffic from our single egress IP made PSX answer 429, and PSXRateLimitError had no handler, so the catch-all turned it into a 500.

This PR:

  • caches /historical per symbol;
  • serves the last cached copy when PSX refuses or is unreachable;
  • stops calling PSX for 60 s after a 429;
  • maps PSXRateLimitError to 503 psx_unavailable with Retry-After: 60 on every endpoint.

Behaviour

  • One entry per symbol. PSX always returns a symbol's full history, so one cached entry serves every start/end range, sliced in the API. The SDK is called with cache=False, which makes this layer the only cache.
  • Freshness.
    • Mon–Fri 09:00–17:00 PKT: 30 min, capped at 17:00.
    • Any other time: until the next weekday 09:00 PKT.
  • Headers. X-Cache: HIT | MISS | STALE. Age is sent on HIT and STALE. meta.cached is true on HIT and STALE.
  • Two tiers.
    • An in-process LRU (64 MB) sits in front of an optional Redis-compatible server (REDIS_URL, e.g. Aiven for Valkey).
    • Redis fails open: short timeouts, one warning per 30 s backoff window, never a 500.
    • Without REDIS_URL the cache is in-memory only.
  • Concurrency. Concurrent misses for one symbol share a single PSX fetch, using a fixed pool of 256 striped locks.
  • Unreadable cache entries. Corrupt or wrong-schema entries are treated as misses and overwritten.
  • Date validation. start/end must be YYYY-MM-DD. Invalid dates and start > end now return 422 bad_request instead of 500.
  • Row order. Row order and missing-value handling are unchanged from 0.2.x: newest first, a NaN close gives null, and a null volume gives 502.

Configuration

Variable Default Meaning
REDIS_URL unset Redis-compatible server URI. Set via fastapi cloud env. Only the hostname is ever logged.
HISTORICAL_CACHE_MARKET_TTL 1800 Trading-hours freshness in seconds.

Release notes

  • sync-openapi.yml now runs on v* tags instead of every push to main, so the public docs update only when a release is made.
  • After merge: tag v0.3.0. The psxdata-docs update follows the release.
  • Version bumped to 0.3.0. There's a CHANGELOG entry, and the README has new Caching and Configuration sections.

Testing

139 tests pass (60 existing + 79 new); ruff and mypy are clean. New test modules:

  • tests/api/test_cache_freshness.py: window edges, weekends, UTC.
  • tests/api/test_cache_codec.py: round-trip, malformed envelopes.
  • tests/api/test_cache_store.py: LRU, fail-open Redis with fakeredis, tiering.
  • tests/api/test_cache_historical.py: hit, miss, expiry, stale-on-429 and cooldown, stale-on-unavailable, single-flight, restart via L2.
  • tests/api/test_cache_factory.py: env parsing, bad REDIS_URL without leaking the secret, lifespan.
  • tests/api/test_historical_endpoint.py: headers, slicing, 422, 503, Redis down, OpenAPI.

A local smoke test against live PSX gave MISS, then HIT, then 422 on start=20260920.

@mtauha
mtauha merged commit c65774a into main Sep 30, 2026
4 checks passed
@mtauha
mtauha deleted the feat/historical-cache branch September 30, 2026 12:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant