Skip to content

fix(kyc): enforce unique constraint on nullable columns#3768

Open
davidleomay wants to merge 3 commits into
developfrom
fix/nullable-unique-indexes
Open

fix(kyc): enforce unique constraint on nullable columns#3768
davidleomay wants to merge 3 commits into
developfrom
fix/nullable-unique-indexes

Conversation

@davidleomay
Copy link
Copy Markdown
Member

Summary

  • PostgreSQL treats NULL != NULL in unique indexes, so the composite unique index on kyc_step (userData, name, type, sequenceNumber) silently allowed duplicate steps when type was NULL (e.g. FinancialData). Concurrent requests could create multiple FinancialData steps with the same sequence number.
  • The user_data unique index on (identDocumentId, nationality, accountType, kycType) was missing a NULL filter on nationality, allowing the same bypass.
  • Adds NULLS NOT DISTINCT (PG 15+) to the kyc_step index and nationalityId IS NOT NULL to the user_data index WHERE clause.

Test plan

  • Verified on dev DB: duplicate NULL-type inserts into kyc_step are now correctly blocked
  • Verified no existing duplicate rows on dev that would conflict with the new constraint
  • Migration runs cleanly (up and down)
  • Verify on staging before production deploy

…nd user_data indexes

PostgreSQL treats NULL != NULL in unique indexes, so the composite unique
index on kyc_step (userData, name, type, sequenceNumber) silently allowed
duplicate FinancialData steps when type was NULL. Similarly, the user_data
unique index was missing a NULL filter on the nationality column.

- Add NULLS NOT DISTINCT to kyc_step unique index (PG 15+)
- Add nationalityId IS NOT NULL to user_data unique index WHERE clause
@davidleomay davidleomay requested a review from TaprootFreak as a code owner May 26, 2026 19:51
@davidleomay
Copy link
Copy Markdown
Member Author

CLEAN UP DATA BEFORE MERGING!!

@davidleomay
Copy link
Copy Markdown
Member Author

Duplicate check results (production)

kyc_step

83 duplicate groups, 298 total rows — all with type = NULL:

name type duplicate_rows
FinancialData NULL 296
DfxApproval NULL 2

Find all conflicting rows:

SELECT ks.id, ks."userDataId", ks.name, ks.type, ks."sequenceNumber"
FROM kyc_step ks
WHERE EXISTS (
  SELECT 1 FROM kyc_step ks2
  WHERE ks2.id != ks.id
    AND ks2."userDataId" = ks."userDataId"
    AND ks2.name = ks.name
    AND COALESCE(ks2.type, '__NULL__') = COALESCE(ks.type, '__NULL__')
    AND ks2."sequenceNumber" = ks."sequenceNumber"
)
ORDER BY ks."userDataId", ks.name, ks."sequenceNumber", ks.id;

Delete duplicates (keep the row with the lowest id per group):

DELETE FROM kyc_step
WHERE id NOT IN (
  SELECT MIN(id)
  FROM kyc_step
  GROUP BY "userDataId", name, COALESCE(type, '__NULL__'), "sequenceNumber"
);

user_data

No duplicates found — migration is safe to run on this index.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants