Meta: facet-count correctness & coherence (counts beyond the single-filter cube)

@ekansa

Why this issue exists

Facet-count correctness keeps surfacing as separate bug reports (#296, #304, and others from collaborators), but they share one root cause and one fix lever. This is the higher-level tracker so the team can see the whole picture in one place and not re-diagnose each report from scratch.

It effectively succeeds #234 (now closed) for the counts dimension: #234's "coherent filter semantics across all surfaces" roadmap shipped, but the count-coherence frontier outlived it.

The through-line (plain English)

The Explorer can show correct, cross-filtered facet counts instantly for exactly one active filter — that case is served by a precomputed lookup ("the cube", #290/#298). Everything beyond one filter currently falls off that fast path:

More than one filter at the zoomed-out view → other facets revert to their unfiltered global totals (this is Interactive Explorer: Facet counts wrong when filtering by multiple facet values #304).
Viewport-scoped counts, active search, and flat mode also fall through to live paths with their own rough edges.

So the mental model is: the single-filter cube is the spine; the frontier is "counts when more than one thing is true at once."

Crucially, the map/point filtering is correct in all these cases — it's specifically the count numbers that mislead. And zooming in makes counts correct (the live membership path cross-filters by all active dims). The breakage is specific to the global/zoomed-out view.

The shared fix lever

We already shipped the tool to fix this: the sample_facet_masks bitmask (#299) computes multi-filter results in one ~6M-row columnar scan, instead of the ~39M-row sample_facet_membership scan that originally forced the global multi-filter count path to be stubbed to the baseline. The plan is to un-stub the global multi-filter count path on top of the masks.

Children / related

Interactive Explorer: Facet counts wrong when filtering by multiple facet values #304 — facet counts wrong when filtering by multiple facet values (the live bug; keep riffing there). ← primary
explorer: facet counts — remaining work after #157/#226 #230 — facet counts: remaining work after globe: Phase 1 unification — Specimen Type filter, SKOS labels, portable predicate, v2 facets #157/Explorer: unify click semantics — click opens detail card, external link is one-hop-away #226 (the concrete leftovers list)
explorer: material facet semantics — single "first non-root" value vs membership ("anywhere in array") counts #276 — material facet semantics: single "first non-root" value vs membership ("anywhere in array") counts (a semantics decision that shapes what the counts even mean)
explorer: clearing a search does not deterministically restore viewport-scoped counts (needs repro) #286 — clearing a search doesn't deterministically restore viewport-scoped counts (needs repro)

What "done" looks like

Facet counts reflect all active selections (multi-filter), at any zoom, served fast (masks-backed) rather than reverting to baseline.
A single, stated semantics contract for what a count means (resolves explorer: material facet semantics — single "first non-root" value vs membership ("anywhere in array") counts #276), applied consistently across map / table / facets.
The individual children above closed against that contract.

cc @ekansa @akthom — filing this so the count issues you've reported have one coordinating home; the specific bug stays in #304.

@ekansa

Plan of record (Claude + Codex, joint)

Full write-up committed to the repo: PLAN_305_facet_counts.md (code citations, native benchmarks, SQL, validator gates). Condensed version below.

Core idea — count per node FROM the #299 bitmasks, no 39M-row membership scan

For a count dimension D over a qualifying pid set Q: bitwise-join Q against facet_node_bits (≤63 rows/dim) and GROUP BY concept_uri:

... JOIN node_bits nb ON nb.facet_type='<D>'
    AND (s.<D>_mask & (1::BIGINT << nb.bit_index)) <> 0
GROUP BY nb.concept_uri

Masks are one row/pid → COUNT(*) = distinct pids (no COUNT(DISTINCT) needed); ancestor closure → parent ≥ child for free. Semantics: OR within a dim, AND across dims, exclude the target dim (matches the existing cube + UI contract).

Feasibility — confirmed by native DuckDB benchmarks (real 6M-pid data)

All three tree dims via one 56-node join: ~0.69s; Eric's Material+Specimen-Type case: ~0.02s. Worst case ≈ 336M cheap bit-tests → WASM cold+warm benchmark stays a hard release gate.

Two firm rules

Source → add as a plain VARCHAR column to a new complete per-pid index (pid, source, 3 masks, build_id, schema_version), not a mask and not a semi-join.
Never fall back to the unfiltered baseline under active filters — that is this bug. On miss/error show "—"/unavailable/stale, and apply all dimension counts atomically after one stale-guard check.

Prerequisite data bug → #306

The current masks are built from membership, so they omit ~29,917 located samples with no tree membership (6,026,242 vs 5,996,325). The new index must start from samp_geo (zero-mask those pids). Filed as #306; lands in Phase 1.

Phases

Artifact + semantics — complete per-pid index (incl. Data: sample_facet_masks omits ~29,917 located samples with no tree membership (built from membership, not samp_geo) #306 fix + source), fingerprint the pid/source universe, AI-free validator, publish under a new immutable tag, ratify the count contract as membership/"anywhere in tree" (explorer: material facet semantics — single "first non-root" value vs membership ("anywhere in array") counts #276 — already shipped semantics).
Fix Interactive Explorer: Facet counts wrong when filtering by multiple facet values #304 at global view — generalized direct-control filter snapshot (today's describeCrossFilters() zeroes tree selections at global view = root cause); route multi-filter counts through the histogram; keep the single-filter cube as a pure optimization; benchmark cold+warm WASM; unavailable-not-baseline on failure.
Extend (not unify yet) — add viewport (padded bbox) + search (search_pids), characterization-test vs the membership path, then retire it.
Verify + clean up — predicate-gen unit tests, validator in CI, Playwright (Eric's URL, multi-value, source+tree, impossible combos, global↔viewport, superseded requests), remove the baseline fallback, Codex review, staging verify, smoke gate.

Top risks

WASM latency / connection starvation (unmeasured in-browser).
Stale/incomplete index generations (Data: sample_facet_masks omits ~29,917 located samples with no tree membership (built from membership, not samp_geo) #306 + membership-only build_id).
Honesty & async state-transition failures (baseline-as-truth + non-atomic repaint).

cc @ekansa @akthom

isamplesorg#305/isamplesorg#306: keep membership_build_id format-comp…

isamplesorg#304/isamplesorg#305 Phase 2: multi-filter facet counts at…

isamplesorg#304/isamplesorg#305 Phase 2: address Codex review (honest…

isamplesorg#304/isamplesorg#305 Phase 2: fix masks-catch recoupling +…

isamplesorg#304/isamplesorg#305 Phase 2: correct coverage-handshake c…

isamplesorg#304/isamplesorg#305 Phase 2: make the facetIndexReady cov…

isamplesorg#304/isamplesorg#305 Phase 3: extend mask-index counts to …

isamplesorg#304/isamplesorg#305 Phase 3: fix global-search honesty fa…

docs: plan of record for #305 facet-count coherence

#306/#305: complete per-pid facet index (sample_facet_index)

#306/#305: harden index validator + fingerprint (Codex round 1)

#306/#305: close index validator fail-open gaps (Codex round 2)

#306/#305: force-emit full fast-path bundle with index (Codex round 3)

#306/#305: prove index-only rebuild RE-EMITS its bundle (Codex round 4)

Merge pull request #307 from rdhyee/feat/306-complete-facet-index

Merge pull request #308 from rdhyee/feat/304-multifilter-counts

Merge pull request #309 from rdhyee/feat/304-viewport-search-counts

@rdhyee

🤖 rbotyee (Claude, operated by @rdhyee) — closing-week sweep (7/17 board): parking with an honest status so a future reader (human or AI) has state in 30 seconds.

Meta-tracker parked: the single-filter cube shipped (#157/#226/#317-era) and multi-filter counts work via the mask index. Remaining semantics questions (counts beyond the single-filter cube) are documented here and in #230/#276; revisit only with a data rebuild. — rbotyee 🤖 (Claude, for @rdhyee)

#340: never show baseline facet counts under an active search

coherence: stop four more surfaces asserting numbers they don't have

isamplesorg#343 round 5: snapshot banners on the remaining current-fa…

invariant round 2: guarantee a successor on throw; make the verifier …

Navigation Menu

Repository navigation

Meta: facet-count correctness & coherence (counts beyond the single-filter cube) #305

Why this issue exists

The through-line (plain English)

The shared fix lever

Children / related

What "done" looks like

Plan of record (Claude + Codex, joint)

Core idea — count per node FROM the #299 bitmasks, no 39M-row membership scan

Feasibility — confirmed by native DuckDB benchmarks (real 6M-pid data)

Two firm rules

Prerequisite data bug → #306

Phases

Top risks

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Meta: facet-count correctness & coherence (counts beyond the single-filter cube) #305

Description

Why this issue exists

The through-line (plain English)

The shared fix lever

Children / related

What "done" looks like

Activity

rdhyee commented on Jun 19, 2026

Plan of record (Claude + Codex, joint)

Core idea — count per node FROM the #299 bitmasks, no 39M-row membership scan

Feasibility — confirmed by native DuckDB benchmarks (real 6M-pid data)

Two firm rules

Prerequisite data bug → #306

Phases

Top risks

rdhyee commented on Jul 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions