Cache /historical per symbol; 503 + Retry-After on PSX rate limits

.github/workflows/sync-openapi.yml

@@ -2,7 +2,7 @@ name: Sync OpenAPI spec to psxdata-docs
 
 on:
   push:
-    branches: [main]
+    tags: ["v*"]
 
 jobs:
   sync:

CHANGELOG.md

@@ -7,6 +7,26 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 
 ---
 
+## [0.3.0] — 2026-09-30
+
+### Added
+
+- `GET /stocks/{symbol}/historical` is now cached. Each symbol's full history is fetched from PSX once and reused for every `start`/`end` range. It stays fresh for 30 minutes during PSX trading hours (Mon–Fri 09:00–17:00 PKT) and until the next trading-day open otherwise. Responses carry `X-Cache: HIT | MISS | STALE`, an `Age` header on cached responses, and `meta.cached: true` when served from cache.
+- When PSX rate-limits the API or is unreachable, `/historical` serves the last cached copy (`X-Cache: STALE`) instead of failing.
+- Optional `REDIS_URL` environment variable: any Redis-compatible server (e.g. Aiven for Valkey) that keeps the cache across restarts and instances. Without it the cache is in-memory only. `HISTORICAL_CACHE_MARKET_TTL` (seconds, default `1800`) tunes trading-hours freshness.
+
+### Fixed
+
+- PSX rate-limiting (HTTP 429 from PSX) surfaced as `500 internal_error` on every endpoint (324 such responses on `/historical` on 2026-09-29). It now returns `503 psx_unavailable` with `Retry-After: 60`, and `/historical` stops calling PSX for 60 seconds after a 429.
+- Invalid `start`/`end` dates, or `start` after `end`, on `/historical` returned `500`; they now return `422 bad_request`.
+
+### Changed
+
+- `start`/`end` on `/historical` must be ISO dates (`YYYY-MM-DD`).
+- The OpenAPI spec is pushed to the docs site on release tags (`v*`) instead of every push to `main`, so public docs change only when a release is made.
+
+---
+
 ## [0.2.1] — 2026-09-26
 
 ### Fixed

README.md

@@ -98,6 +98,8 @@ Every response wraps its payload in a consistent envelope.
 }
 ```
 
+`meta.cached` is `true` when the response was served from the API's cache (currently `/stocks/{symbol}/historical`; see [Caching](#caching)).
+
 **Error response**
 ```json
 {
@@ -113,7 +115,7 @@ Every response wraps its payload in a consistent envelope.
 | 404 | `not_found` | Symbol or index does not exist |
 | 429 | `rate_limited` | Exceeded 60 requests/minute per IP |
 | 502 | `upstream_data_error` | Upstream PSX data failed validation |
-| 503 | `psx_unavailable` | PSX website unreachable |
+| 503 | `psx_unavailable` | PSX website unreachable, or PSX is rate-limiting the API (sent with a `Retry-After` header) |
 | 500 | `internal_error` | Unexpected server error |
 
 ---
@@ -124,6 +126,43 @@ Every response wraps its payload in a consistent envelope.
 
 ---
 
+## Caching
+
+`GET /stocks/{symbol}/historical` is served from a cache. PSX always returns a symbol's full history, so the API fetches it once and slices it to your `start`/`end` range.
+
+| When the data was fetched | Stays fresh until |
+| ------------------------- | ----------------- |
+| Mon–Fri 09:00–17:00 PKT (trading hours) | 30 minutes later, but no later than 17:00 |
+| Any other time | The next weekday 09:00 PKT |
+
+Every `/historical` response says where it came from:
+
+| Header | Values |
+| ------ | ------ |
+| `X-Cache` | `HIT` — fresh cached copy · `MISS` — fetched from PSX just now · `STALE` — PSX refused or was unreachable, so the last cached copy was served |
+| `Age` | Seconds since the data was fetched from PSX (on `HIT` and `STALE`) |
+
+`meta.cached` is `true` for `HIT` and `STALE`. If PSX is rate-limiting and no cached copy exists, the API returns `503 psx_unavailable` with `Retry-After: 60`.
+
+---
+
+## Configuration
+
+All settings are optional environment variables.
+
+| Variable | Default | Description |
+| -------- | ------- | ----------- |
+| `REDIS_URL` | unset | Redis-compatible server for the `/historical` cache, e.g. an [Aiven for Valkey](https://aiven.io/valkey) service URI (`rediss://default:<password>@<host>:<port>`). Keeps the cache across restarts and instances. When unset, or when the server is unreachable, the API falls back to an in-memory cache and keeps working. |
+| `HISTORICAL_CACHE_MARKET_TTL` | `1800` | How long, in seconds, `/historical` data stays fresh during PSX trading hours. |
+
+```bash
+docker run -p 8000:8000 -e REDIS_URL="rediss://default:<password>@<host>:<port>" mtauha/psxdata-api
+```
+
+On FastAPI Cloud, set it as an app environment variable (`fastapi cloud env`) rather than committing it.
+
+---
+
 ## Docker
 
 ```bash

api/cache/__init__.py

@@ -0,0 +1 @@
+"""Response cache for GET /stocks/{symbol}/historical."""

api/cache/codec.py

@@ -0,0 +1,84 @@
+"""Serialisation of cached /historical entries: a versioned, gzip-compressed JSON envelope.
+
+Rows are stored as arrays under a single ``columns`` list, which is roughly 20% smaller than
+repeating the keys on every row.
+"""
+from __future__ import annotations
+
+import gzip
+import json
+import logging
+import zlib
+from dataclasses import dataclass
+from datetime import datetime
+from typing import Any
+
+logger = logging.getLogger(__name__)
+
+FORMAT_VERSION = 1
+COLUMNS: tuple[str, ...] = ("date", "open", "high", "low", "close", "volume", "is_anomaly")
+
+
+@dataclass(frozen=True)
+class CacheEntry:
+    symbol: str
+    fetched_at: datetime
+    fresh_until: datetime
+    rows: list[dict[str, Any]]
+
+
+def _json_default(value: Any) -> Any:
+    # numpy / pandas scalars expose .item() returning the native Python value
+    if hasattr(value, "item"):
+        return value.item()
+    raise TypeError(f"Object of type {type(value).__name__} is not JSON serialisable")
+
+
+def encode(entry: CacheEntry) -> bytes:
+    payload = {
+        "v": FORMAT_VERSION,
+        "symbol": entry.symbol,
+        "fetched_at": entry.fetched_at.isoformat(),
+        "fresh_until": entry.fresh_until.isoformat(),
+        "columns": list(COLUMNS),
+        "rows": [[row.get(col) for col in COLUMNS] for row in entry.rows],
+    }
+    raw = json.dumps(payload, default=_json_default, separators=(",", ":")).encode()
+    return gzip.compress(raw, compresslevel=6)
+
+
+def _aware(value: Any) -> datetime:
+    parsed = datetime.fromisoformat(value)
+    if parsed.tzinfo is None:
+        raise ValueError("cache timestamp is not timezone-aware")
+    return parsed
+
+
+def _row(values: Any) -> dict[str, Any]:
+    if not isinstance(values, list) or len(values) != len(COLUMNS):
+        raise ValueError("cache row has the wrong shape")
+    if not isinstance(values[0], str):
+        raise TypeError("cache row date is not a string")
+    return dict(zip(COLUMNS, values))
+
+
+def decode(blob: bytes) -> CacheEntry | None:
+    """Return the entry, or None for anything that is not a complete v1 envelope."""
+    try:
+        payload = json.loads(gzip.decompress(blob))
+        if payload["v"] != FORMAT_VERSION:
+            logger.warning("Ignoring cache entry with unknown format version %r", payload["v"])
+            return None
+        if payload["columns"] != list(COLUMNS):
+            raise ValueError("cache entry has unexpected columns")
+        if not isinstance(payload["symbol"], str) or not isinstance(payload["rows"], list):
+            raise TypeError("cache entry has malformed fields")
+        return CacheEntry(
+            symbol=payload["symbol"],
+            fetched_at=_aware(payload["fetched_at"]),
+            fresh_until=_aware(payload["fresh_until"]),
+            rows=[_row(values) for values in payload["rows"]],
+        )
+    except (OSError, EOFError, zlib.error, ValueError, KeyError, TypeError) as exc:
+        logger.warning("Ignoring unreadable cache entry (%s)", type(exc).__name__)
+        return None

api/cache/factory.py

@@ -0,0 +1,76 @@
+"""Build the /historical cache from environment configuration.
+
+``REDIS_URL`` (optional) points at any Redis-compatible server, e.g. Aiven for Valkey. It is
+never logged, only its hostname.
+"""
+from __future__ import annotations
+
+import logging
+from collections.abc import Mapping
+from datetime import timedelta
+from urllib.parse import urlparse
+
+import redis
+from redis.backoff import NoBackoff
+from redis.retry import Retry
+
+from api.cache.freshness import DEFAULT_MARKET_TTL
+from api.cache.historical import HistoricalService
+from api.cache.store import MemoryLRU, RedisStore, TieredStore
+
+logger = logging.getLogger(__name__)
+
+REDIS_TIMEOUT_SECONDS = 0.5
+
+
+def market_ttl_from_env(env: Mapping[str, str]) -> timedelta:
+    raw = env.get("HISTORICAL_CACHE_MARKET_TTL")
+    if raw is None:
+        return DEFAULT_MARKET_TTL
+    try:
+        seconds = int(raw)
+    except ValueError:
+        seconds = 0
+    if seconds <= 0:
+        logger.warning(
+            "Ignoring invalid HISTORICAL_CACHE_MARKET_TTL=%r; using %ds",
+            raw, int(DEFAULT_MARKET_TTL.total_seconds()),
+        )
+        return DEFAULT_MARKET_TTL
+    return timedelta(seconds=seconds)
+
+
+def build_redis_client(url: str | None) -> redis.Redis | None:
+    if not url:
+        return None
+    try:
+        return redis.Redis.from_url(
+            url,
+            socket_connect_timeout=REDIS_TIMEOUT_SECONDS,
+            socket_timeout=REDIS_TIMEOUT_SECONDS,
+            health_check_interval=30,
+            retry=Retry(NoBackoff(), 0),
+        )
+    except ValueError as exc:
+        logger.warning(
+            "REDIS_URL could not be parsed (%s); using in-memory cache only", type(exc).__name__
+        )
+        return None
+
+
+def build_historical_service(env: Mapping[str, str]) -> HistoricalService:
+    url = env.get("REDIS_URL")
+    client = build_redis_client(url)
+    if client is None:
+        logger.info("No cache backend configured; /historical uses in-memory cache only")
+    else:
+        host = urlparse(url or "").hostname or "unknown-host"
+        try:
+            client.ping()
+            logger.info("Cache backend connected: %s", host)
+        except (redis.RedisError, OSError) as exc:
+            logger.warning(
+                "Cache backend unreachable at startup: %s (%s)", host, type(exc).__name__
+            )
+    store = TieredStore(MemoryLRU(), RedisStore(client))
+    return HistoricalService(store, market_ttl=market_ttl_from_env(env))

api/cache/freshness.py

@@ -0,0 +1,37 @@
+"""Freshness policy for cached /historical entries, tied to PSX trading hours.
+
+Past rows never change; only today's row moves while the market is open. Inside a padded
+trading window an entry stays fresh for ``market_ttl`` (capped at the window close); outside
+it, an entry stays fresh until the next window opens.
+"""
+from __future__ import annotations
+
+from datetime import date, datetime, time, timedelta, timezone
+from zoneinfo import ZoneInfo
+
+PKT = ZoneInfo("Asia/Karachi")
+WINDOW_OPEN = time(9, 0)
+WINDOW_CLOSE = time(17, 0)
+DEFAULT_MARKET_TTL = timedelta(minutes=30)
+
+
+def _is_weekday(day: date) -> bool:
+    return day.weekday() < 5
+
+
+def fresh_until(fetched_at: datetime, market_ttl: timedelta = DEFAULT_MARKET_TTL) -> datetime:
+    """Return the UTC instant after which data fetched at ``fetched_at`` is stale."""
+    if fetched_at.tzinfo is None:
+        raise ValueError("fetched_at must be timezone-aware")
+    local = fetched_at.astimezone(PKT)
+    today = local.date()
+    window_open = datetime.combine(today, WINDOW_OPEN, tzinfo=PKT)
+    window_close = datetime.combine(today, WINDOW_CLOSE, tzinfo=PKT)
+
+    if _is_weekday(today) and window_open <= local < window_close:
+        return min(local + market_ttl, window_close).astimezone(timezone.utc)
+
+    day = today if _is_weekday(today) and local < window_open else today + timedelta(days=1)
+    while not _is_weekday(day):
+        day += timedelta(days=1)
+    return datetime.combine(day, WINDOW_OPEN, tzinfo=PKT).astimezone(timezone.utc)