bridge: add guest-side reconnect loop for live migration#2698
Open
shreyanshjain7174 wants to merge 1 commit intomicrosoft:mainfrom
Open
bridge: add guest-side reconnect loop for live migration#2698shreyanshjain7174 wants to merge 1 commit intomicrosoft:mainfrom
shreyanshjain7174 wants to merge 1 commit intomicrosoft:mainfrom
Conversation
During live migration the vsock connection between the host and the GCS breaks when the VM moves to the destination node. The GCS bridge drops and cannot recover, leaving the guest unable to communicate with the new host. This adds a reconnect loop in cmd/gcs/main.go that re-dials the bridge after a connection loss. On each iteration a fresh Bridge and Mux are created while the Host state (containers, processes) persists across reconnections. A Publisher abstraction is added to bridge/publisher.go so that container wait goroutines spawned during CreateContainer can route exit notifications through the current bridge. When the bridge is down between reconnect iterations, notifications are dropped with a warning — the host-side shim re-queries container state after reconnecting. The defer ordering in ListenAndServe is fixed so that quitChan closes before responseChan becomes invalid, and responseChan is buffered to prevent PublishNotification from panicking on a dead bridge. Tested with Invoke-FullLmTestCycle on a two-node Hyper-V live migration setup (Node_1 -> Node_2). Migration completes at 100% and container exec works on the destination node after migration. Signed-off-by: Shreyansh Sancheti <shsancheti@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2669
Problem
During live migration the vsock connection between the host and the GCS (Guest Compute Service) breaks when the UVM moves to the destination node. The bridge inside the GCS drops and cannot recover —
ListenAndServereturns with an I/O error, and the GCS has no way to re-establish communication with the new host. This leaves the guest unable to process any further container lifecycle operations after migration.What this does
Adds a reconnect loop around the bridge lifecycle in
cmd/gcs/main.go. When the bridge connection drops (detected byListenAndServereturning), the GCS re-dials the host on the vsock command port and creates a freshBridge+Mux. TheHoststate (containers, processes, cgroups) persists across reconnections since it lives outside the bridge.A
Publisheris added tointernal/guest/bridge/publisher.goto solve the goroutine lifetime mismatch: container wait goroutines are spawned duringCreateContainerand outlive the bridge that created them. When a container exits, its wait goroutine callsPublisher.Publish()which routes the notification through whichever bridge is currently active. If no bridge is connected (during the reconnect gap), the notification is dropped — the host-side shim recovers by re-querying container state after reconnecting.The defer ordering in
ListenAndServeis fixed soquitChancloses beforeresponseChanbecomes invalid, preventing a panic whenPublishNotificationraces with bridge teardown.responseChanis buffered to absorb in-flight responses during shutdown.Design
The approach follows the existing
runWithRestartMonitorpattern already used incmd/gcs/main.gofor chronyd — a loop with exponential backoff that retries forever.Key design decisions:
Bridge+Muxis created per iteration. Channels and handler closures are scoped to eachListenAndServecall, so there is no stale state to reset.hcsv2.Hostholds containers in mutex-guarded maps. It is created once and reused across all bridge iterations. Container state survives the bridge drop.WaitForProcesswhich blocks on the container's actual exit status — the notification is a convenience, not the source of truth.ShutdownRequested()returns true and the loop breaks instead of reconnecting.Reference: Kevin Parsons' live migration POC for the reconnect concept. This implementation simplifies the POC down to the minimum — just the main loop + Publisher (~90 lines of new code).
Changes
cmd/gcs/main.gofor{}loop with exponential backoff (1s–30s, retry forever)internal/guest/bridge/bridge.goPublisherfield onBridge,ShutdownRequested()method, fixed defer ordering, bufferedresponseChan, priority select guard inPublishNotificationinternal/guest/bridge/bridge_v2.goPublisher.Publish()internal/guest/bridge/publisher.gointernal/guest/bridge/publisher_test.goTesting
Tested on a two-node Hyper-V live migration setup using the
TwoNodeInfratest module:Invoke-FullLmTestCycle -Verbose— deploys LM agents, creates a UVM with an LCOW container on Node_1, migrates to Node_2, verifies 100% completion on both nodes. Containerlcow-testsuccessfully migrated with pod sandbox intact.crictl exec— created a fresh LCOW pod on Node_1 with our custom GCS (deployed viarootfs.vhd), started a container that writes a file, exec'dcat /tmp/dummy.txtto verify bridge communication works end-to-end.go build,go vet, andgofmtclean on all modified packages.