Skip to content

feat(pii): load generator word pools from embedded data files - #608

Open
rhiannalitchfield wants to merge 1 commit into
dataiku:mainfrom
rhiannalitchfield:feat/pii-generator-embed-pools
Open

feat(pii): load generator word pools from embedded data files#608
rhiannalitchfield wants to merge 1 commit into
dataiku:mainfrom
rhiannalitchfield:feat/pii-generator-embed-pools

Conversation

@rhiannalitchfield

Copy link
Copy Markdown
Contributor

Purpose

Part of #465, and a follow-up to #594 where @hanneshapke asked how the name/city/location pools should be extended. This moves those pools out of the inline Go slices and into embedded data files, so growing them (or adding locale-aware variants later, pairing with #457) becomes a data-only change that never touches Go.

What Changed

  • New data/*.txt pool files (one value per line, # for comments/section headers), embedded with //go:embed in a new pools.go. The files are small and always required, so unlike the build-tag-gated model/UI embeds this one is unconditional.
  • pools.go loads each pool once at startup. Parsing lives in a testable parsePool(io.Reader) that trims whitespace and skips blank/comment lines; loadPool panics on a missing or empty embedded file so a packaging error fails loudly at startup rather than leaving a generator with an empty pool.
  • pii_generators.go now references the loaded pool vars. Generator logic, pickExcluding, and the never-echo-original guarantee from fix(pii): never echo original PII and generate guaranteed-invalid dummies #594 are unchanged, and every existing generator test passes as-is.
  • Externalized pools: first names, surnames, cities, streets, states, countries, company prefixes/suffixes, and the lowercase email local-part names. Format- and security-sensitive lists (RFC reserved email/URL domains, IBAN country codes, security-token prefixes) stay inline on purpose.
  • Each pool is expanded modestly to show the payoff; the additions are common, verifiable entries.
  • .gitattributes pins data/*.txt to eol=lf so the bytes embedded into the binary are identical regardless of the build platform's checkout settings.

Testing

  • parsePool is unit-tested for comment/blank/whitespace handling.
  • A table test asserts every loaded pool is non-empty with no parser leakage (no blank or # entries, no stray whitespace).
  • A test asserts the email pools stay lowercase ASCII tokens, which is what keeps EmailGenerator's first.last output matching the existing [a-z]+\.[a-z]+ check.
  • go test ./src/backend/pii/generators/ passes; golangci-lint reports 0 issues; gofmt clean.

No changeset, since those are added by the maintainers at release time.

Part of dataiku#465, follow-up to dataiku#594. Move the name, city, street, state,
country, and company word pools out of the inline slices in
pii_generators.go into newline-delimited data/*.txt files embedded via
//go:embed and loaded once at startup. Generator logic, the
never-echo-original guarantee, and every existing test are unchanged;
this only changes where the pools live.

Growing a pool (or adding locale-aware variants later, pairing with the
XLM-RoBERTa work in dataiku#457) is now a data-only edit that never touches Go.
Also expands each pool modestly to demonstrate the payoff. A loader with
a testable parsePool handles comments and blank lines, and the data
files are pinned to eol=lf so the embedded bytes are identical across
build platforms.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Warning

This PR touches 3+ distinct areas of the codebase.

Consider splitting into smaller, focused PRs — each covering a single semantic type.
This makes reviews easier and keeps the git history clean.

Categories found:

code:

  • src/backend/pii/generators/data/cities.txt
  • src/backend/pii/generators/data/company_prefixes.txt
  • src/backend/pii/generators/data/company_suffixes.txt
  • src/backend/pii/generators/data/countries.txt
  • src/backend/pii/generators/data/email_firstnames.txt
  • src/backend/pii/generators/data/email_surnames.txt
  • src/backend/pii/generators/data/firstnames.txt
  • src/backend/pii/generators/data/states.txt
  • src/backend/pii/generators/data/streets.txt
  • src/backend/pii/generators/data/surnames.txt
  • src/backend/pii/generators/pii_generators.go
  • src/backend/pii/generators/pools.go

chore:

  • .gitattributes

test:

  • src/backend/pii/generators/pools_test.go

@rhiannalitchfield

Copy link
Copy Markdown
Contributor Author

Heads up that the failing "Check PR scope" job is the same false positive tracked in #595, not something to split here. It counts three areas: code (the generators + the new data/*.txt pools), test (pools_test.go), and chore (the one-line .gitattributes change that pins the data files to eol=lf). That is a code change shipped with its tests plus a necessary config line, which is exactly the compliant shape the check should allow. Splitting would mean landing the pools without their tests, which CONTRIBUTING forbids.

Everything else is green, including Go Tests on the full Linux build.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant