You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Implement FeatureExtractor.expression() for email domain/TLD extractors #145
Raised by @tim-band during review of #137 (discussion): EmailDomainExtractor.expression() and EmailTopLevelDomainExtractor.expression() currently just raise NotImplementedError(). Tim sketched the actual per-dialect SQL:
Postgres: SELECT SPLIT_PART(email, '@', 2) AS domain FROM customer limit 20
MsSQL: SELECT SUBSTRING(email, CHARINDEX('@', email) + 1, LEN(email)) AS domain FROM customer WHERE CHARINDEX('@', email) > 1
and separately floated a more general AfterCharacter dialects.py function class that could serve both domain and TLD extraction (TLD being "after the last . of whatever's after @").
Why this doesn't bite today
EmailDomainExtractor/EmailTopLevelDomainExtractor are only ever used with CategoryBuilder (see EMAIL_PIPELINE in statistical_fidelity.py), and CategoryBuilder.build_from_table doesn't currently call .expression() at all - it samples rows and calls .extract() in Python. So the NotImplementedError is unreachable in the current architecture.
Why it's still worth doing
It becomes a real gap the moment #144 (moving CategoryBuilder onto SQL-side GROUP BY for extractors that support it) is implemented - at that point, the EMAIL profile's "domain"/"tld" pipeline stages would need these to work, or would have to keep a Python-sampling fallback specifically for them.
Proposal
Add dialect-aware SQL expression(s) to dialects.py (Postgres SPLIT_PART/MSSQL SUBSTRING+CHARINDEX for the "after @" case, reused for "after the last ." for TLD), matching the existing pattern of SecondsDifference/WordCount/SentenceCount. Consider Tim's generalized AfterCharacter(column, char) form if it cleanly covers both call sites.
Activity