fix: Preserve admin segment lock order during discovery

[kfelternv]

Summary

Prevent a PostgreSQL deadlock in DPU discovery by making the discovery transaction acquire admin-network-segment advisory locks before it updates or locks machine-interface rows. Address allocation already uses this segment-lock-before-row-lock order.

Root cause

The 18-tray machine-a-tron lifecycle in #3207 exposed a pre-existing lock inversion:

• DPU discovery updated machine-interface rows, then waited for the admin-segment advisory lock during address reconciliation.
• Concurrent address allocation held the admin-segment advisory lock, then waited for machine-interface rows.

PostgreSQL repeatedly reported deadlock detected from the FOR UPDATE OF mi query in reconcile_admin_addresses_for_host, paired with another transaction waiting on pg_advisory_xact_lock.

Fix

The DPU discovery transaction now calls the existing load_and_lock_all_admin_segments helper before any machine-interface work. The helper is made public within the database crate so both paths use the same lock order.

No new locking primitive, abstraction, or dependency is introduced.

Before and after evidence

Removing only this two-file change reproduced repeated deadlocks in the 18-tray lifecycle. Restoring it made the same rack-scale path complete in 83.85 seconds:

test result: ok. 1 passed; 0 failed; finished in 83.85s
machine-a-tron-nvl72 | NVL72 | 18 expected | 18 managed | 18 Ready

The passing run contained no deadlock detected entry.

Verification

Hotfix revision: 2fcc29b83052d745d2fb39374cc2e812c342cd5b

Base revision: upstream main at b7154872b

Supporting checks passed:

cargo check -p carbide-api-core --no-default-features --lib
cargo fmt --all --check
git diff --check

Hands-on regression exercise:

REPO_ROOT="$PWD" \
DATABASE_URL=postgresql://postgres@127.0.0.1:55433 \
cargo test --release -p carbide-api-integration-tests --test lib \
  test_machine_a_tron_rack_integration -- --exact --nocapture

The focused lifecycle test currently lives in #3207. Its copy of this two-file fix has the same stable Git patch ID as this hotfix: 76226037bcbdaef26b31592f98bb7f3e6c1d8741.

The direct carbide-api-core unit-test target cannot build on this Apple Silicon host because its circular test-harness dev dependency re-enables the Linux-only default TPM feature. The no-default-features library check verifies the hotfix build without that unsupported target; the real PostgreSQL lifecycle above exercises the changed lock path.

Relationship to #3207

#3207 exposed the deadlock but does not own the production lock-order behavior. This hotfix is separate so the concurrency fix has a focused review, merge, and rollback path. #3207 remains draft and depends on this PR merging first.

Risk and scope

• Two production files: seven insertions and one visibility change.
• DPU discovery takes the existing admin-segment locks earlier in its transaction, which can move contention earlier but removes the observed lock inversion.
• No rack or simulator behavior changes.
• No API, schema, configuration, or dependency changes.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7cbea94b-358b-4f94-8890-54022dc21aba

📥 Commits

Reviewing files that changed from the base of the PR and between 2fcc29b and fcc8265.

📒 Files selected for processing (2)

• crates/api-core/src/handlers/managed_host.rs
• crates/api-core/src/tests/set_primary_interface.rs

---

Summary by CodeRabbit

• Bug Fixes
  • Improved interface and machine discovery handling for DPU-backed hosts by ensuring the system waits for the right locks before making changes.
  • Fixed primary interface repair flows so hosts without an Admin-segment primary are reconciled correctly and consistently.
  • Strengthened validation around interface updates to reduce race conditions during concurrent operations.

Walkthrough

load_and_lock_all_admin_segments is now public, and both machine discovery and primary-interface repair invoke it to acquire admin segment locks before continuing their transactional updates. The repair path is covered by a concurrency-based test that asserts the lock ordering.

Changes

DPU Segment Locking

Layer / File(s)	Summary
Expose segment lock function publicly crates/api-db/src/machine_interface.rs	load_and_lock_all_admin_segments changes from private to public async function, with unchanged body logic.
Invoke segment lock in DPU discovery flow crates/api-core/src/handlers/machine_discovery.rs	discover_machine adds a conditional branch that calls load_and_lock_all_admin_segments(&mut txn) when hardware_info.is_dpu() is true, before continuing with discovery.
Repair primary-interface flow crates/api-core/src/handlers/managed_host.rs	set_primary_interface_core adds a branch that acquires admin segment locks when there is no current Admin-segment primary, then proceeds with primary-interface updates.
Assert lock ordering in repair test crates/api-core/src/tests/set_primary_interface.rs	The repair test introduces a blocker transaction, concurrent API execution, advisory-lock polling, and a NOWAIT row-lock check to verify the updated repair path.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant discover_machine
  participant machine_interface
  participant Database

  Client->>discover_machine: discover_machine(hardware_info)
  discover_machine->>Database: begin txn
  alt hardware_info.is_dpu()
    discover_machine->>machine_interface: load_and_lock_all_admin_segments(&mut txn)
    machine_interface->>Database: lock admin segments
  end
  discover_machine->>Database: continue discovery updates
  discover_machine-->>Client: discovery completes

Loading

sequenceDiagram
  participant Test
  participant set_primary_interface_core
  participant machine_interface
  participant PostgreSQL

  Test->>PostgreSQL: open blocker txn and hold admin segment locks
  Test->>set_primary_interface_core: set_primary_interface(...)
  set_primary_interface_core->>machine_interface: load_and_lock_all_admin_segments(&mut txn)
  machine_interface->>PostgreSQL: wait on admin segment locks
  Test->>PostgreSQL: verify target row is not locked early
  Test->>PostgreSQL: release blocker txn
  set_primary_interface_core-->>Test: primary-interface repair completes

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: preserving admin segment lock order during discovery.
Description check	✅ Passed	The description is directly aligned with the changeset and explains the deadlock fix, locking order, and validation.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[kfelternv]

This hotfix was split out of #3207 after the rack-scale path exposed the production lock inversion. #3207 is back in draft and blocked on this PR; its focused 18-tray lifecycle is the real regression reproducer.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2fcc29b830

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

🔍 Container Scan Summary

Service	Total	Critical	High	Medium	Low	Other
boot-artifacts-aarch64	3	0	0	3	0	0
boot-artifacts-x86_64	3	0	0	3	0	0
forge-admin-cli-x86_64	271	13	34	91	7	126
machine-validation-runner	776	37	223	281	40	195
machine_validation	776	37	223	281	40	195
machine_validation-aarch64	776	37	223	281	40	195
nvmetal-carbide	776	37	223	281	40	195
TOTAL	3381	161	926	1221	167	906

Per-CVE detail lives in the per-service grype-* artifacts (JSON + SARIF). Severity counts only — no CVE IDs published here.