Skip to content

Add opt-in PostgreSQL IS predicate literal preservation - #114

Open
joelmarcotte wants to merge 3 commits into
mainfrom
joel.marcotte/keep-is-predicate
Open

joelmarcotte wants to merge 3 commits into
mainfrom
joel.marcotte/keep-is-predicate

Conversation

@joelmarcotte

@joelmarcotte joelmarcotte commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Add WithKeepISPredicate(bool) and keep_is_predicate, disabled by default.
  • For PostgreSQL only, preserve TRUE, FALSE, and NULL on the right of IS and IS NOT in both Obfuscate and ObfuscateAndNormalize.
  • Match keyword tokens across whitespace and comments. Leave both IS [NOT] DISTINCT FROM forms, ordinary literals, and positional binds on the existing replacement rules. Add the CLI flag and README guidance.

The existing keep_boolean and keep_null semantics remain intact. Enabling this option in DBM changes query signatures and requires a coordinated sample, metrics, controller, and proxy rollout. This PR changes only go-sqllexer; it does not change Agent, integrations-core, dd-source, or PgProxy.

Validation

  • go test ./... passed.
  • Changed Go files were formatted with gofmt; git diff --check passed.
  • Golden tests assert exact output for all four ReplaceBoolean/ReplaceNull combinations with the option off and on through both public processing paths. The off goldens were captured from origin/main before the change.
  • CLI smoke checks passed for obfuscate and obfuscate_and_normalize.

Performance

Added BenchmarkKeepISPredicate for queries with no predicate, one predicate, and mixed predicates through both processing paths. On an Apple M5 Pro with Go 1.27.1 (-benchmem -benchtime=300ms -count=5), option-off medians were within about 2% of main. Option-on added roughly 3–7% CPU time in Obfuscate and 2–3% in ObfuscateAndNormalize. Both settings had the same allocation count: 5 per Obfuscate call and 6 per ObfuscateAndNormalize call in these cases. The mixed normalized output used 16 more bytes with the option on because it retains literal text. CPU and allocation profiles showed no allocation in the new predicate check; the measured allocations remain in the lexer, output builder, and normalizer.

PostgreSQL comparison

An isolated local PostgreSQL 18.6 (Homebrew) instance used shared_preload_libraries=pg_stat_statements, compute_query_id=auto, standard_conforming_strings=on, pg_stat_statements.track=top, track_utility=on, track_planning=off, max=5000, and save=on.

pg_stat_statements reported SELECT $1 IS DISTINCT FROM $2 for raw SELECT TRUE IS DISTINCT FROM FALSE, and SELECT $1 IS NOT DISTINCT FROM $2 for raw SELECT TRUE IS NOT DISTINCT FROM FALSE. Preserving the right-hand FALSE would split raw-sample and representative-text signatures. This PR therefore defers both DISTINCT FROM forms. With the option enabled and boolean and positional parameter replacement on, both sources normalize to SELECT ? IS DISTINCT FROM ? (or SELECT ? IS NOT DISTINCT FROM ?). SELECT TRUE IS TRUE also matched its representative text while keeping the right-hand TRUE.

@joelmarcotte
joelmarcotte requested a review from a team as a code owner September 30, 2026 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant