Mongolia_Employment: Fix duplicate region formation and missing references in registered unemployment - #2216
Open
kartik-s21 wants to merge 60 commits into
Open
kartik-s21 wants to merge 60 commits into
kartik-s21 wants to merge 60 commits into
Conversation
Code fix unenergy
- Filter out erroneous duplicate indented Region code 511 (' Ulaanbaatar') in common_download_script.py for registered_unemployed_by_education_level_region_gender_month.csv
- Pass table.get('query') in common_download_script.py for EMPLOYMENT_TABLES
- Add defense-in-depth sanitization in run.sh to strip duplicate indented Ulaanbaatar rows from input file if present
- Add unit test in common_download_script_test.py verifying query configuration
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request adds a query filter to the Mongolian registered unemployed data download configuration to exclude duplicate Ulaanbaatar entries, along with a corresponding unit test. It also updates the execution script to strip duplicate indented Ulaanbaatar entries from the downloaded CSV using sed. Feedback was provided regarding the portability of sed -i across macOS and Linux, suggesting the use of a temporary file instead.
kartik-s21
changed the title
…rg#2216 - Use co-located temporary file, atomic move, non-empty check, and exit trap in run.sh - Forward table.get('query') uniformly across all domain table loops in common_download_script.py - Add unit test verifying runtime query parameter propagation in common_download_script_test.py - Add MAX_DATE_CONSISTENT and SQL_VALIDATOR date freshness rules to validation_config.json
…ate_freshness from validation_config.json
balit-raibot
approved these changes
Sep 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Import validation for
Mongolia_Employmentfailed withFound 7366 missing references in input 5 (registered_unemployed_by_education_level_region_gender_month`).Root Cause
DT_NSO_0400_002V5.pxcontains both code5("Ulaanbaatar") and duplicate indented code511(" Ulaanbaatar").wikidataId/Q23430.511contains conflicting counts (234,0,..), causing 430 mismatched duplicate SVObs.stat_var_processor.pyflagged 10 StatVars with#ErrorDuplicateSVObsand dropped them from the generated MCF, including newly defined schema StatVarCount_Person_Unemployed_UpperSecondaryEducation.Count_Person_Unemployed_UpperSecondaryEducationfailed reference existence checks in validation (3,683 rows × 2 passes = 7,366 missing references).Solution
Бүсincommon_download_script.pyforregistered_unemployed_by_education_level_region_gender_month.csvto exclude code511and keep all 27 valid regions.table.get('query')forEMPLOYMENT_TABLESincommon_download_script.py.run.shbefore running input 5 to strip any indented duplicate Ulaanbaatar rows from existing input files.common_download_script_test.py.Verification
stat_var_processor.py:error-mismatched-svobs: 430 -> 0error-statvar-with-dup-svobs: 10 -> 0dropped-invalid-statvars: 10 -> 0Count_Person_Unemployed_UpperSecondaryEducation)country/MNG+ 22 aimags/capital, matching golden data)python3 -m unittest statvar_imports/mongolia_imports/common_download_script_test.py(7/7 passed).Documentation