statistics_poland: remove golden validation and config_override, adopt node_mcf pattern, and add freshness checks

[abhishekjaisw]

Description

Updates statvar_imports/statistics_poland to align with the standardized Data Commons import configuration, hardening guidelines, and CRA review recommendations:

1. Golden Validation & Data Removal:
   • Removed brittle GOLDENS_CHECK rules (check_goldens_summary_report and check_goldens_observations) from validation_config.json.
   • Deleted unused golden_data/ directory (golden_observations.csv, golden_summary_report.csv).
2. Removed Config Override:
   • Removed config_override from manifest.json (including use_autopush_dc_api: false), allowing standard executor defaults to resolve global schema.
3. Updated Source Files:
   • Replaced golden_data/*.csv with validation_config.json in manifest.json source_files.
4. Adopted node_mcf Pattern:
   • Replaced static stat_var_mcf with flexible node_mcf: "StatisticsPoland_output*.mcf" (aligning with PR #2232). Because all 108 StatVars are already merged into Data Commons globally (recon-lookup-schema = 109), stat_var_processor.py avoids emitting duplicate local schema MCF files; using the wildcard pattern gracefully handles resolved global statvars without missing file errors.
5. Download Script Resilience & Hardening (download_input_data.py):
   • Added automatic HTTP session retry with exponential backoff on 429/5xx status codes and increased read timeout to 60s.
   • Fixed HTTP status code logging bug (resp is not None check) so real status codes (403, 404, 429, 500) are logged instead of None.
   • Replaced fixed range(10) pagination with dynamic pagination (while True:) to consume all variable pages from GUS API (guidelines.md:107).
   • Prevented silent undercounting/corruption in population rollups: raised RuntimeError on failed variable metadata calls, unmapped demographic slices, and observation request failures.
   • Cleaned up dead imports (import traceback).
   • Replaced unsafe KeyError swallowing with final_df.reindex(columns=...) for deterministic column alignment.
   • Implemented atomic file generation via temporary .tmp files and os.replace.
   • Structured entry point def main(argv=None): with clean exit propagation.
6. Enhanced Validation Rules (validation_config.json):
   • Updated check_all_statvars_freshness (SQL_VALIDATOR with per-StatVar window query asserting MaxDate >= '2025' AND total_svs > 0).
   • Retained check_max_date_consistent (MAX_DATE_CONSISTENT), verifying all 108 StatVars uniformly end at year 2025.
   • Retained check_deleted_records_percent at threshold: 0.1 (0.1%, capped at <= 0.5%).
   • Omitted redundant default checks (LINT_ERROR_COUNT, MISSING_REFS_COUNT, EMPTY_IMPORT_CHECK), which run automatically in the executor.
7. Hermetic Unit Test Suite (download_input_data_test.py):
   • 11 hermetic unit tests covering template loading, offline fallback, request sanitization, HTTP 403 status logging, /variables dynamic pagination, error propagation on unmapped slices and download failures, and end-to-end data transformation (100% pass in 1.4s).
   • Cleaned up unused import (import io).
8. Documentation:
   • Updated README.md with operational guidance, updated freshness boundary (MaxDate >= '2025'), and testing instructions; enforced line length ceiling (<= 100 chars).

---

Dev Cloud Batch Verification (us-west1)

• Job ID: statistics-poland-abhishekjaisw-20261005-095516
• Job UID: statistics-poland-26db8b91-8f0c-4e2500
• Job Console Link: https://console.cloud.google.com/batch/jobsDetail/regions/us-west1/jobs/statistics-poland-abhishekjaisw-20261005-095516/details?project=datcom-infosys-dev
• Job Logs: https://console.cloud.google.com/batch/jobsDetail/regions/us-west1/jobs/statistics-poland-abhishekjaisw-20261005-095516/logs?project=datcom-infosys-dev
• GCS Staging Output: https://console.cloud.google.com/storage/browser/datcom-import-test/statvar_imports/statistics_poland/statistics_poland/2026_10_05T02_58_24_936831_07_00/
• Status: SUCCEEDED (ImportStatus.STAGING, execution time: 243s)
• Rows Processed: 42,228 observations (+1,836 new 2025 observations added, 0 deleted observations)

Validation Output (validation_output.csv)

Validation Rule	Validator	Status	Details
check_all_statvars_freshness	SQL_VALIDATOR	PASSED	Verified all 108 StatVars satisfy MaxDate >= '2025' AND total_svs > 0
check_max_date_consistent	MAX_DATE_CONSISTENT	PASSED	108/108 rows share uniform MaxDate: 2025 (0 failed)
check_deleted_records_percent	DELETED_RECORDS_PERCENT	PASSED	0 deleted records (0.0%, threshold: 0.1)
check_empty_import	EMPTY_IMPORT_CHECK	PASSED	num_nodes: 42228, num_rows: 42228 (default validator)
check_missing_refs_count	MISSING_REFS_COUNT	PASSED	0 missing references (default validator)
check_lint_error_count	LINT_ERROR_COUNT	PASSED	0 lint errors (default validator)

Differ Summary (differ_summary.json)

{
    "current_obs_count": 42228,
    "previous_obs_count": 40392,
    "current_schema_count": 0,
    "previous_schema_count": 65,
    "added_obs_count": 1836,
    "deleted_obs_count": 0,
    "modified_obs_count": 0,
    "added_schema_count": 0,
    "deleted_schema_count": 65,
    "obs_diff_count": 1836,
    "schema_diff_count": 65
}

(Note: deleted_schema_count: 65 reflects pruning redundant local schema definitions that now exist in canonical global Data Commons schema; deleted_obs_count: 0 confirms 100% of data observations are preserved).

[gemini-code-assist]

Code Review

This pull request enhances the robustness of the Statistics Poland data download script by implementing a requests session with automatic retries, backoff, and improved error handling. It also increases request timeouts and sleep intervals to mitigate transient network issues. Additionally, the PR removes several golden data files, updates the import manifest to use wildcard MCF files and reference validation_config.json, and cleans up the validation configuration by removing golden-check rules and configuration overrides. There are no review comments, so no feedback is provided.