You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Compute HistogramBuilder/CategoryBuilder real statistics via SQL instead of a Python-side sample #144
HistogramBuilder.build_from_table: "I see this is the same as what I did... do you think it would be better to move the min/max into the bucket number query? Then can we sum within the query? I can't remember why I didn't; perhaps I was in a hurry?"
HistogramBuilder.build_from_table pulls a random sample_size-row sample into Python and computes mean/stddev with statistics.mean/statistics.stdev, even though the bucket-count half of the same method already does an exact, full-table GROUP BY aggregate query (see the # comment block above bucket_expr in distribution_builders.py).
CategoryBuilder.build_from_table pulls a random sample into Python and counts features with collections.Counter, even though FeatureExtractor already has an .expression() method for building the equivalent SQL expression (used by HistogramBuilder via feature_expr = self.extractor.expression(self.column)) - so for an extractor that implements .expression(), the same GROUP BY <expression>, COUNT(*) pattern already used for the histogram bucket counts should work here too.
This is a real behavior change, not a pure refactor: switching from an approximate sample_size-row sample to an exact full-table aggregate changes the actual numbers (mean/stddev, category frequencies) that drive every downstream fidelity score, for every numeric/categorical column. It needs:
verifying AVG/STDDEV/GROUP BY semantics agree across Postgres/DuckDB/MSSQL (sample vs. population variance, NULL handling) the way dialects.py's StdDev class already does for other call sites,
deciding whether to keep a sample_size cap on the full-table aggregate query at all (arguably unnecessary once it's a single aggregate query rather than row-by-row fetch+Python-side computation),
auditing which FeatureExtractors used with CategoryBuilder actually have .expression() implemented - several currently raise NotImplementedError (CharacterBigramExtractor, WordExtractor, VowelConsonantPatternExtractor, the Email*Extractors) and would need either an implementation (see #<EmailDomainExtractor issue, filed separately>) or to keep the Python-sampling path as a fallback.
Given that, this deserves its own PR rather than scope-creeping #137.
Activity