Skip to content

Implement FeatureExtractor.expression() for email domain/TLD extractors #145

Open

Description

@yhong123

Context

Raised by @tim-band during review of #137 (discussion): EmailDomainExtractor.expression() and EmailTopLevelDomainExtractor.expression() currently just raise NotImplementedError(). Tim sketched the actual per-dialect SQL:

Postgres: SELECT SPLIT_PART(email, '@', 2) AS domain FROM customer limit 20
MsSQL: SELECT SUBSTRING(email, CHARINDEX('@', email) + 1, LEN(email)) AS domain FROM customer WHERE CHARINDEX('@', email) > 1

and separately floated a more general AfterCharacter dialects.py function class that could serve both domain and TLD extraction (TLD being "after the last . of whatever's after @").

Why this doesn't bite today

EmailDomainExtractor/EmailTopLevelDomainExtractor are only ever used with CategoryBuilder (see EMAIL_PIPELINE in statistical_fidelity.py), and CategoryBuilder.build_from_table doesn't currently call .expression() at all - it samples rows and calls .extract() in Python. So the NotImplementedError is unreachable in the current architecture.

Why it's still worth doing

It becomes a real gap the moment #144 (moving CategoryBuilder onto SQL-side GROUP BY for extractors that support it) is implemented - at that point, the EMAIL profile's "domain"/"tld" pipeline stages would need these to work, or would have to keep a Python-sampling fallback specifically for them.

Proposal

Add dialect-aware SQL expression(s) to dialects.py (Postgres SPLIT_PART/MSSQL SUBSTRING+CHARINDEX for the "after @" case, reused for "after the last ." for TLD), matching the existing pattern of SecondsDifference/WordCount/SentenceCount. Consider Tim's generalized AfterCharacter(column, char) form if it cleanly covers both call sites.

Depends on / pairs with #144.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions