You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Data: sample_facet_masks omits ~29,917 located samples with no tree membership (built from membership, not samp_geo) #306
sample_facet_masks (and the sample_facet_membership it derives from) omit ~29,917 located samples that have coordinates but no hierarchical facet membership. Any count or filter computed from the masks therefore silently undercounts the true located universe.
A located sample that asserts no concept in any of the three SKOS trees (material / context / object_type) never appears in membership, so it gets no mask row at all — rather than a row with all-zero masks. It's invisible to the bitmask path.
Why it matters now
It's latent today (the masks are only used for filtering, where a no-membership sample legitimately matches no tree node). But the #305 plan computes facet counts and the located-sample baseline from a masks-based index. Those counts must reflect the full located universe (incl. zero-membership samples, e.g. the "source" facet count and any "samples in view" total), so this gap would produce convincing-but-wrong numbers.
Fix
Build the per-pid index starting from samp_geo (the located universe), LEFT JOIN the aggregated masks, and emit zero masks for samples with no hierarchical membership:
samp_geo LEFT JOIN (aggregated masks) -> COALESCE(mask, 0)
Carry this into the new complete index described in #305 Phase 1 (pid, source, material_mask, context_mask, object_type_mask, build_id, schema_version), and add a validator gate asserting index.pid_count == facets_v3.pid_count.
Verification
Validator: pid-set equality between the new index and facets_v3; source equality; mask≡membership for samples that DO have membership.
Confirm the 29,917 previously-missing pids appear with all-zero masks.
Found during the #305 plan review (Codex). Related: #305, #299 (introduced the masks), #304.
🤖 rbotyee (Claude, operated by @rdhyee) — closing-week sweep (7/17 board): parking with an honest status so a future reader (human or AI) has state in 30 seconds.
Known data gap, root-caused: sample_facet_masks is built from tree membership, omitting ~29,917 located samples with no tree membership. Fix = rebuild masks from samp_geo; needs a pipeline run. Documented as a known issue for the archival substrate. — rbotyee 🤖 (Claude, for @rdhyee)
Activity
isamplesorg#305/isamplesorg#306: keep membership_build_id format-comp…
isamplesorg#304/isamplesorg#305 Phase 2: multi-filter facet counts at…
isamplesorg#304/isamplesorg#305 Phase 2: address Codex review (honest…
isamplesorg#304/isamplesorg#305 Phase 2: fix masks-catch recoupling +…
isamplesorg#304/isamplesorg#305 Phase 2: correct coverage-handshake c…
docs: plan of record for #305 facet-count coherence
#306/#305: complete per-pid facet index (sample_facet_index)
#306/#305: harden index validator + fingerprint (Codex round 1)
#306/#305: close index validator fail-open gaps (Codex round 2)
#306/#305: force-emit full fast-path bundle with index (Codex round 3)
#306/#305: prove index-only rebuild RE-EMITS its bundle (Codex round 4)
Merge pull request #307 from rdhyee/feat/306-complete-facet-index
rdhyee commentedon Jul 22, 2026
Known data gap, root-caused: sample_facet_masks is built from tree membership, omitting ~29,917 located samples with no tree membership. Fix = rebuild masks from samp_geo; needs a pipeline run. Documented as a known issue for the archival substrate. — rbotyee 🤖 (Claude, for @rdhyee)