fix(integrations): Guard inbound assignee sync against reordered webhooks

[vaind]

Integration webhooks are forwarded control silo → cell as WebhookPayload rows and drained without ordering: a failed delivery is rescheduled with exponential backoff and lands behind events that originally followed it, backlogged mailboxes drain through a thread pool that explicitly sacrifices ordering, and for some providers the drain skips a failed message outright.

Inbound assignee sync runs synchronously in the webhook handler, so mailbox order decides the outcome. All four handlers read the issue's full assignee snapshot rather than a delta, which is better than a delta but not sufficient — a stale snapshot applied last still wins, so reordering two assignment changes leaves the earlier one as the final state. Inbound status sync is not exposed the same way: it hands off to sync_status_inbound_task one call later, so ordering was already gone by then.

Each handler now passes its provider's own timestamp for the change (issue.updated_at, object_attributes.updated_at, issue.fields.updated), and ExternalIssue.provider_assignee_updated_at holds the newest one already applied. Anything strictly older is dropped. Provider time is only ever compared to provider time — delivery latency is the same order as a real user action, so comparing against arrival time would suppress legitimate follow-ups. The guard is inert when either side is missing, so a payload with no usable timestamp behaves exactly as it does today.

Covers GitHub, GitLab, Jira, and Jira Server; GitHub Enterprise inherits the GitHub handler. Unassignment is the same code path (empty snapshot → deassign) and needed no separate treatment. VSTS is deliberately left alone: its timestamp lives in System.ChangedDate, which is the same extraction #121084 adds, so touching it here would collide for no benefit.

Equal timestamps: snapshot vs delta

The three ordering PRs deliberately use two different comparisons, and the rule is about what the payload contains rather than about the provider:

• Snapshot payloads treat equal as fresh (< is stale) — this PR and fix(scm): Make PR lifecycle writes monotonic across reordered webhooks #121059. The payload carries the whole state, so re-applying a same-instant event is idempotent when it is a redelivery and correct-by-recency when it is a distinct change the provider's second-resolution clock cannot separate. Dropping it would be a real loss with nothing to repair it.
• Delta payloads treat equal as stale (<= is stale) — fix(integrations): Ignore reordered and replayed inbound status webhooks #121084. The payload carries a transition (closed, reopened, a from/to pair), so re-running one over an intervening human action is destructive.

Why assignment gets its own watermark

Sharing provider_status_updated_at looked plausible — both syncs are driven by the same issues webhook carrying the same updated_at — but they are independent state machines, and one watermark lets either censor the other. A status event at T2 would drop an assignment change at T1 even when that change is not late relative to any other assignment, and it is then lost permanently, because no corrective assignment event follows. Same-timestamp events make this concrete (Jira sends a single jira:issue_updated whose changelog can carry both a status and an assignee item under one fields.updated), but the censoring exists regardless of timestamp resolution. The differing comparisons above are the second reason: one column cannot carry both coherently.

Concurrency

The check and the write have to be one critical section. Two deliveries for the same issue can both read the pre-write watermark and both pass the staleness check; if the newer one commits first, the older one still overwrites the assignment while its own conditional watermark write correctly affects zero rows — leaving a stored assignee from the older event under a watermark from the newer one, which no later event carrying an intermediate timestamp can repair. A backlogged mailbox draining through the delivery thread pool is exactly what produces overlapping deliveries for one issue.

The staleness check therefore takes select_for_update on the issue rows and holds it through the assignment and the watermark write, as #121059 does. Both entry points were restructured so that critical section is cell-local — the affected groups and the users to assign are resolved beforehand, since hybrid cloud RPC inside a transaction is a banned pattern and both lookups are RPCs. What remains under the lock is DB writes, a cache invalidation and task enqueues; the issue_assigned signal that fans out to Sentry App webhooks is deferred by transaction.on_commit and runs after the lock is released.

Interaction with #121084

That PR solves the same hazard for status sync, and this one deliberately mirrors its shape: normalize provider time in the handler, keep the watermark on ExternalIssue, compare provider time to provider time, stay inert when either side is missing.

This is built against master and does not depend on it, but the two overlap and whoever merges second should expect it. Both add a field to ExternalIssue on adjacent lines, both touch migrations_lockfile.txt, and both add a migration off head 1149. This migration is numbered 1152 to sit clear of 1151, so if #121084 lands first the only fix-up is repointing this migration's dependency to its migration. parse_provider_event_time then exists in both utils/status_sync.py and utils/assignee_sync.py and is worth unifying once both have landed.

[github-actions]

This PR has a migration; here is the generated SQL for src/sentry/migrations/1152_externalissue_assignee_updated_at.py

for 1152_externalissue_assignee_updated_at in sentry

--
-- Add field assignee_updated_at to externalissue
--
ALTER TABLE "sentry_externalissue" ADD COLUMN "assignee_updated_at" timestamp with time zone NULL;

[github-actions]

This PR has a migration; here is the generated SQL for src/sentry/migrations/1152_externalissue_provider_assignee_updated_at.py

for 1152_externalissue_provider_assignee_updated_at in sentry

--
-- Add field provider_assignee_updated_at to externalissue
--
ALTER TABLE "sentry_externalissue" ADD COLUMN "provider_assignee_updated_at" timestamp with time zone NULL;

[sentry-warden]

Unguarded data["changelog"]["items"] access crashes on missing key

Direct dict access to data["changelog"]["items"] will raise KeyError if Jira sends a webhook where the changelog dict exists but omits the items key.

Evidence

• JiraIssueUpdatedWebhook.post returns early only when data.get("changelog") is missing or falsy, but a truthy changelog dict that lacks an items key proceeds past the guard.
• The endpoint's own diagnostic logging explicitly reads (data.get("changelog") or {}).get("items") or [], treating items as optional in the upstream payload.
• handle_assignee_change at line 62 in jira/utils/api.py unconditionally iterates data["changelog"]["items"] inside an any() generator with no fallback or existence check.
• Neither the webhook endpoint nor the handler wraps this call in try/except KeyError, so a malformed or edge-case payload raises an unhandled exception.
• The same pattern exists in jira_server/utils/api.py line 30, which is also modified in this PR.

_{Identified by Warden · sentry-backend-bugs · Q4V-Q5G}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 27b1dab. Configure here.}

[cursor]

Unassign signal runs under row lock

Medium Severity

_ordered_assignment now holds ExternalIssue select_for_update locks through deassign, but issue_unassigned still fires synchronously inside that critical section. Assign carefully defers issue_assigned with transaction.on_commit; unassign does not, so its Kafka snapshot work can run while the issue row lock is held and stall concurrent deliveries for the same issue.

Additional Locations (1)

• src/sentry/integrations/utils/sync.py#L270-L278

 

^{Reviewed by Cursor Bugbot for commit 27b1dab. Configure here.}