Skip to content

fix(spans_metrics): fall back to update on 409 during create - #650

Merged
riyazsh merged 1 commit into
mainfrom
riyazsh/spans-metrics-409-fallback
Jul 29, 2026
Merged

fix(spans_metrics): fall back to update on 409 during create#650
riyazsh merged 1 commit into
mainfrom
riyazsh/spans-metrics-409-fallback

Conversation

@riyazsh

@riyazsh riyazsh commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary

spans_metrics sets skip_resource_mapping=True, so state.destination["spans_metrics"] is never populated from a live LIST. First-run against a destination that already has a same-named metric (customer-created before sync-cli started managing the org) POSTs, gets HTTP 409 "metric already exists with that name", and loops forever — state stays empty, next dispatch retries the CREATE, same 409.

The v2 API keys spans_metrics by name (URL path param metric_id == metric name), so a 409 on POST means the same _id is on the destination. This PR makes create_resource catch 409, GET the existing record by id, hydrate state.destination, and delegate to update_resource (PATCH). Non-409 errors propagate unchanged.

Motivation

Observed in production against a real customer org where 18/18 spans_metrics failed every sync run with HTTP 409. The destination org had all 18 metric names pre-created by the customer (managed=false); sync-cli had no way to converge because it never saw them.

The same class of defect affects any resource type that combines:

  • skip_resource_mapping=True (no live LIST populating state)
  • create_resource doing a bare POST with no 409 fallback
  • A resource "id" that is a human-chosen name rather than a server-assigned UUID

I only fix spans_metrics here; other candidates (metric_percentiles, metrics_metadata, name-keyed order/config types) should be evaluated individually.

Test plan

  • pytest tests/unit/test_spans_metrics_409_fallback.py -v → 3 passed
    • test_create_resource_happy_path — POST succeeds, no fallback triggered
    • test_create_resource_409_falls_back_to_get_then_patch — POST 409 → GET-by-id → state.destination hydrated → PATCH called with id-scoped URL
    • test_create_resource_non_409_reraises — 500 propagates unchanged, no GET/PATCH
  • Existing spans_metrics behavior unchanged on the happy path (same return shape, same POST call)

Files changed

  • datadog_sync/model/spans_metrics.py — wrap create_resource POST in try/except; on 409, GET-by-id, hydrate state, delegate to update_resource
  • tests/unit/test_spans_metrics_409_fallback.py — new unit tests

The `spans_metrics` model has `skip_resource_mapping=True`, so
`state.destination` is never populated from a live LIST. First-run against
a destination that already has a same-named metric (customer-created before
sync-cli started managing the org) POSTs, hits HTTP 409 "metric already
exists with that name", and loops forever — state stays empty, next
dispatch retries the CREATE, same 409.

The v2 API keys spans_metrics by name (path param `metric_id` == metric
name), so a 409 on POST means the same `_id` is on the destination. On
409, GET the existing record by id, hydrate `state.destination`, and
retry as `update_resource` (PATCH). Non-409 errors propagate unchanged.

Unit tests cover: happy-path POST, 409 → GET → PATCH fallback, and non-409
reraise.
@riyazsh
riyazsh requested a review from a team as a code owner July 28, 2026 23:12
@riyazsh
riyazsh merged commit 3e58160 into main Jul 29, 2026
12 of 14 checks passed
@riyazsh
riyazsh deleted the riyazsh/spans-metrics-409-fallback branch July 29, 2026 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants