feat(pii): load generator word pools from embedded data files - #608
feat(pii): load generator word pools from embedded data files#608rhiannalitchfield wants to merge 1 commit into
Conversation
Part of dataiku#465, follow-up to dataiku#594. Move the name, city, street, state, country, and company word pools out of the inline slices in pii_generators.go into newline-delimited data/*.txt files embedded via //go:embed and loaded once at startup. Generator logic, the never-echo-original guarantee, and every existing test are unchanged; this only changes where the pools live. Growing a pool (or adding locale-aware variants later, pairing with the XLM-RoBERTa work in dataiku#457) is now a data-only edit that never touches Go. Also expands each pool modestly to demonstrate the payoff. A loader with a testable parsePool handles comments and blank lines, and the data files are pinned to eol=lf so the embedded bytes are identical across build platforms. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Warning This PR touches 3+ distinct areas of the codebase. Consider splitting into smaller, focused PRs — each covering a single semantic type. Categories found: code:
chore:
test:
|
|
Heads up that the failing "Check PR scope" job is the same false positive tracked in #595, not something to split here. It counts three areas: code (the generators + the new Everything else is green, including Go Tests on the full Linux build. |
Purpose
Part of #465, and a follow-up to #594 where @hanneshapke asked how the name/city/location pools should be extended. This moves those pools out of the inline Go slices and into embedded data files, so growing them (or adding locale-aware variants later, pairing with #457) becomes a data-only change that never touches Go.
What Changed
data/*.txtpool files (one value per line,#for comments/section headers), embedded with//go:embedin a newpools.go. The files are small and always required, so unlike the build-tag-gated model/UI embeds this one is unconditional.pools.goloads each pool once at startup. Parsing lives in a testableparsePool(io.Reader)that trims whitespace and skips blank/comment lines;loadPoolpanics on a missing or empty embedded file so a packaging error fails loudly at startup rather than leaving a generator with an empty pool.pii_generators.gonow references the loaded pool vars. Generator logic,pickExcluding, and the never-echo-original guarantee from fix(pii): never echo original PII and generate guaranteed-invalid dummies #594 are unchanged, and every existing generator test passes as-is..gitattributespinsdata/*.txttoeol=lfso the bytes embedded into the binary are identical regardless of the build platform's checkout settings.Testing
parsePoolis unit-tested for comment/blank/whitespace handling.#entries, no stray whitespace).EmailGenerator'sfirst.lastoutput matching the existing[a-z]+\.[a-z]+check.go test ./src/backend/pii/generators/passes;golangci-lintreports 0 issues;gofmtclean.No changeset, since those are added by the maintainers at release time.