oecd_regional_education |Increasing deletion threshold - #2209
Ashwani725 wants to merge 12 commits into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
There was a problem hiding this comment.
Code Review
This pull request updates the OECD regional education import manifest to include a node MCF file and a validation configuration file, and introduces a new validation configuration file to check the percentage of deleted records. The reviewer noted a mismatch in the validation configuration where the rule description specifies a 5% threshold, but the parameter is set to 4.
…ces, and add NUTS 2024 regions
The PR is raised to Allow up to 5% deleted records in OECD regional education validation.
Root Cause
OECD transitioned to dataflow 2.5 (OECD.CFE.EDS:DSD_REG_EDU@DF_ATTAIN(2.5)), introducing Eurostat NUTS 2024 regional restructuring (retiring obsolete territorial units such as PT16, PT17, NL31, and NL33) and pruning historical zero-value estimates. This resulted in 9,531 legitimate record deletions (~3.90% of total observations).
##Test Result:
Test Cloud batch
Test GCS Bucket