Skip to content

CDC_Mortality_UnderlyingCause - #2210

Open
goutamabrol wants to merge 18 commits into
datacommonsorg:masterfrom
goutamabrol:cdc_mortality
Open

goutamabrol wants to merge 18 commits into
datacommonsorg:masterfrom
goutamabrol:cdc_mortality

Conversation

@goutamabrol

@goutamabrol goutamabrol commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Summary

PR checklist - link

Adds the CDC_Mortality_UnderlyingCause import under statvar_imports/us_cdc/county_mortality/ for CDC WONDER county-level mortality data.

Data Source: CDC WONDER Underlying Cause of Death (Database D158), covering all ~3,143 US counties from 2018 to 2024.
Coverage: Maps 118 active StatVars (116 with 2024 data + 2 rare causes with 2023 data) across 137 mapped causes in the ICD-10 113 list.
Downloader (download.py): Automates session agreements, form-based year discovery, query partitioning (75k-row cap), and 31-min silent backoff on HTTP 429.
Mappings: Dynamic 5-digit FIPS zero-padding (geoId/{CountyCode1}) in PV map with no hardcoded county lookup tables.
Automation (manifest.json): Configured for Cloud Batch execution with download.py, stat_var_processor.py, and monthly cron.
Validation (validation_config.json): Enforces 0.1% historical deletion threshold and dynamic date freshness (SQL_VALIDATOR) for standard and rare causes.
Testing: 16/16 unit tests pass locally (download_test.py).

Cloud Batch job (test) - link

Validation - link

CRA Report - link

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new automated importer for CDC WONDER County-Level Mortality Data, including a download script, metadata, mapping files, unit tests, and validation configurations. The reviewer feedback highlights two important improvements for the download script: first, making the is_state_downloaded check more robust to ensure all intermediate years are covered rather than just the first and last; second, modifying the downloader to log errors and continue processing other states instead of immediately terminating the script with logging.fatal when a single state fails.

Comment thread statvar_imports/us_cdc/county_mortality/download.py Outdated
Comment thread statvar_imports/us_cdc/county_mortality/download.py Outdated
@goutamabrol goutamabrol changed the title CDC_Mortality_Count CDC_Mortality_UnderlyingCause Sep 11, 2026
Comment thread statvar_imports/us_cdc/county_mortality/county_mortality_metadata.csv Outdated
Comment thread statvar_imports/us_cdc/county_mortality/county_mortality_pvmap.csv Outdated
Comment thread statvar_imports/us_cdc/county_mortality/manifest.json Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants