Fix HTTP 429 rate limit errors for USCensusPEP_Annual_Population and USCensusPEP_Sex

[niveditasing]

Summary

Fixes recurring P0 production failures caused by HTTP 429 (Too Many Requests) rate limiting from www2.census.gov for USCensusPEP_Annual_Population and USCensusPEP_Sex.

Root Cause

1. Broken Status-Code Retry Logic: @retry wrapped requests.get() without checking response.status_code. When Census returned HTTP 429 or 503, no exception was thrown, bypassing retries and immediately aborting the import.
2. Missing Caching for Immutable Historical Data: Historical files (1900–2021) were re-downloaded from scratch on every weekly scheduled run, blasting 220+ requests in seconds. In USCensusPEP_Sex, cleanup() deleted cached historical data every 30 days.
3. Unthrottled Probing / Downloads: 0-delay HEAD requests probing candidate years (up to 112 HEAD requests per year) triggered WAF throttling.

Changes Made

• Persistent GCS Caching:
  • USCensusPEP_Annual_Population: Added persistent caching via gcs_folder/usa_annual_population_source_files (symlinked to mounted GCS bucket cache in Cloud Batch). If a historical file is present, it copies locally without making an HTTP request.
  • USCensusPEP_Sex: Modified cleanup() to preserve immutable historical files (pre-2021) indefinitely.
• Granular 429/5xx Retries with Exponential Backoff:
  • Added per-file retry logic inspecting status codes 429, 500, 502, 503, 504.
  • Respects Retry-After response header when present, and applies jittered exponential backoff.
• Polite Delays & Session Reuse:
  • Reused requests.Session with browser User-Agent header.
  • Added time.sleep(1.0) between uncached file downloads and time.sleep(0.2)–0.5 between HEAD probes.
• Dynamic Bounded Scanning:
  • Bounded candidate future years to min(2030, dt.now().year + 1).

Testing

• python3 -m unittest discover -v -s scripts/us_census/pep/annual_population -p "*test*.py" -> PASS
• python3 -m unittest discover -v -s scripts/us_census/pep/us_pep_sex -p "*test*.py" -> PASS
• Verified with yapf --style=google.

[gemini-code-assist]

Code Review

This pull request enhances the data downloading process in the US Census PEP preprocessing scripts by introducing robust URL probing, exponential backoff, and persistent caching to handle rate limits and avoid redundant downloads. The review feedback highlights two critical improvements: first, the regex pattern for identifying historical files in the cache cleanup process is too restrictive and could lead to the unintended deletion of historical files; second, temporary files created during download may leak on disk if exceptions occur, which should be resolved by wrapping the file operations in try-finally blocks.