USFEMA_FloodInsuranceClaims: Optimize download and transformation pipeline - #2217
Open
kartik-s21 wants to merge 1 commit into
Open
kartik-s21 wants to merge 1 commit into
kartik-s21 wants to merge 1 commit into
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request re-engineers the FEMA flood insurance claims import pipeline, replacing a slow, sequential, row-by-row legacy processor with a highly optimized, vectorized, multi-process chunked pipeline. It introduces direct bulk CSV downloading with paginated API fallback, updates the manifest and documentation, and adds comprehensive unit tests. Feedback suggests further optimizing the generation of StatVar names in process.py by replacing .apply() with .map() using pre-computed unique flood zone names to avoid Python function call overhead when generating millions of observations.
kartik-s21
force-pushed
the
usfema-flood-claims-optimization
branch
2 times, most recently
from
September 10, 2026 10:15
4d7f6d6 to
56746b1
Compare
kartik-s21
force-pushed
the
usfema-flood-claims-optimization
branch
4 times, most recently
from
September 11, 2026 10:57
58c6d1a to
3848d33
Compare
| "source_files": [ | ||
| "input_file/fema_nfip_claims.csv" | ||
| "input_file/fema_nfip_claims.csv", | ||
| "manifest.json", |
Contributor
There was a problem hiding this comment.
we are not adding manifest.json
kartik-s21
force-pushed
the
usfema-flood-claims-optimization
branch
from
September 11, 2026 12:15
c6a8334 to
7bb1d11
Compare
…ipeline - Re-engineer download pipeline with direct bulk CSV streaming and API pagination fallback. - Replace sequential row iteration in process.py with multi-process vectorized pipeline. - Use concurrent.futures.ProcessPoolExecutor with spawn context for safe multiprocessing. - Add FIPS zero-padding for tract and county places to prevent data loss for states 01-09. - Enforce atomic publishing for observations, MCF, TMCF, and counters. - Improve error handling with non-zero exit on empty aggregations. - Replace eval() with ast.literal_eval() for secure mapping parsing. - Update manifest.json node_mcf pattern and source_files. - Expand README documentation and add comprehensive unit test suite.
kartik-s21
force-pushed
the
usfema-flood-claims-optimization
branch
from
September 11, 2026 12:38
7bb1d11 to
0c411ef
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updated the download and data processing pipeline for
USFEMA_FloodInsuranceClaimsto eliminate multi-hour runtime bottlenecks and improve error resilience.Changes
fema_download.py):~60sdownload time).process.py):concurrent.futures.ProcessPoolExecutorandmultiprocessing.get_context('spawn')..tmpstaging withos.replace) for.csv,.tmcf,.mcf, and counters.eval()withast.literal_eval()for safe mapping parsing.manifest.json,README.md):node_mcfto"output/*.mcf"and includes"manifest.json"insource_files.README.mdwith prerequisites, layout, CLI documentation, and testing steps.process_test.py,fema_download_test.py):