Skip to content

Flaky test fix: Stop tracing the sender after the ack_sync check - #95

Open
vheathen wants to merge 1 commit into
ostinelli:masterfrom
vheathen:ack-sync-test-trace-window
Open

vheathen wants to merge 1 commit into
ostinelli:masterfrom
vheathen:ack-sync-test-trace-window

Conversation

@vheathen

@vheathen vheathen commented Oct 5, 2026

Copy link
Copy Markdown

three_nodes_ack_sync_ordered_delivery turns on 'receive' tracing of the scope's sender process, reconnects the slave node, waits until the node shows up in the scope's subcluster, turns tracing off, and only then looks for the traced {send_single, _, {'3.0', ack_sync, _, _}, _} message in check_trace_for_ack_sync_via_sender/0, which gives up after 2 seconds. The discover handler of syn_gen_scope queues the ack_sync to the sender and adds the node to nodes_map in the same handle_info call, so the node can be visible in the subcluster before the sender has taken the message out of its queue. A message that is still queued when tracing is turned off produces no receive trace event, and the check fails with {badmatch, false}.

The change moves erlang:trace(SenderPid, false, ['receive']) after the check. The sender stays traced until it has received the ack_sync, and the 2 second timeout of the check still bounds the wait. Nothing else changes.

Evidence, OTP 27.3.4.11. With the sender suspended from right after tracing starts until just before the check (erlang:suspend_process/1 and resume_process/1, a scratch edit not in this branch), the case fails 10 runs out of 10 with tracing turned off first and passes 10 out of 10 with the change. Run alone, one fresh ct_run per run, 60 times: 5 failures with {badmatch, false} before the change, 0 after it. The full syn_registry_SUITE passes with the change on OTP 25.3.2.8, 27.3.4.11 and 29.0.6.

three_nodes_ack_sync_ordered_delivery traces 'receive' on the scope's
sender process, reconnects a node, waits until the node shows up in the
scope's subcluster, turns tracing off and only then looks for the traced
ack_sync. The scope queues ack_sync to the sender and adds the node to
nodes_map in the same handle_info call, so the node can be visible
before the sender has taken the message from its queue. A message that
is still queued when tracing is turned off produces no receive event,
and the check fails with {badmatch,false}.

Turning tracing off after the check keeps the sender traced until it
receives ack_sync. The check's 2 second timeout still bounds the wait.

Suspending the sender from before the reconnect until just before the
check makes the test fail in 10 runs out of 10 when tracing is turned
off first, and pass in 10 out of 10 with this change. Run alone 60
times on OTP 27, the case failed 5 times before the change and never
after it.
@vheathen vheathen changed the title Stop tracing the sender after the ack_sync check Flaky test fix: Stop tracing the sender after the ack_sync check Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant