Skip to content

Compute HistogramBuilder/CategoryBuilder real statistics via SQL instead of a Python-side sample #144

Open

Description

@yhong123

Context

Raised by @tim-band during review of #137:

  • HistogramBuilder.build_from_table: "I see this is the same as what I did... do you think it would be better to move the min/max into the bucket number query? Then can we sum within the query? I can't remember why I didn't; perhaps I was in a hurry?"
  • CategoryBuilder.build_from_table: "Looks like we can group by the column and count the values within this query."

Current behavior

HistogramBuilder.build_from_table pulls a random sample_size-row sample into Python and computes mean/stddev with statistics.mean/statistics.stdev, even though the bucket-count half of the same method already does an exact, full-table GROUP BY aggregate query (see the # comment block above bucket_expr in distribution_builders.py).

CategoryBuilder.build_from_table pulls a random sample into Python and counts features with collections.Counter, even though FeatureExtractor already has an .expression() method for building the equivalent SQL expression (used by HistogramBuilder via feature_expr = self.extractor.expression(self.column)) - so for an extractor that implements .expression(), the same GROUP BY <expression>, COUNT(*) pattern already used for the histogram bucket counts should work here too.

Why not just do it in #137

This is a real behavior change, not a pure refactor: switching from an approximate sample_size-row sample to an exact full-table aggregate changes the actual numbers (mean/stddev, category frequencies) that drive every downstream fidelity score, for every numeric/categorical column. It needs:

  • verifying AVG/STDDEV/GROUP BY semantics agree across Postgres/DuckDB/MSSQL (sample vs. population variance, NULL handling) the way dialects.py's StdDev class already does for other call sites,
  • deciding whether to keep a sample_size cap on the full-table aggregate query at all (arguably unnecessary once it's a single aggregate query rather than row-by-row fetch+Python-side computation),
  • auditing which FeatureExtractors used with CategoryBuilder actually have .expression() implemented - several currently raise NotImplementedError (CharacterBigramExtractor, WordExtractor, VowelConsonantPatternExtractor, the Email*Extractors) and would need either an implementation (see #<EmailDomainExtractor issue, filed separately>) or to keep the Python-sampling path as a fallback.

Given that, this deserves its own PR rather than scope-creeping #137.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions