Skip to content

partition_and_heal fails on CI only — real partition leak or timing flake #72

Open

Description

@adiled

Symptom

sim/tests/partition_and_heal.rs fails on CI, passes locally every run (5/5).

At febc862 (PR #67, before any socket-admission work):

left: (2, 0)
right: (2, 2)
"a sigil only A has must survive the join"

Left is expected. Right is what happened: B reached a wane count of 2 for a sigil only A ever ticked. The join moved state across what the test holds to be a partition.

Why this matters more than a red test

The property under test is that nothing crosses a partitioned link. The test asserts it three ways — the tips must not move during the outage, both sides must converge on the higher value after heal, and neither inbox may drop a drained tone. If that property does not hold, a humd behind a network cut may be reading state it should not see.

That claim likely appears in the README. If the test is red, we do not currently know that it is true.

Two possibilities, and I cannot distinguish them from the failure alone

  1. Timing flake. The test polls a 10s convergence deadline and holds a 600ms partition window. CI's scheduling is slower and less predictable; a lost race looks exactly like this.
  2. A real leak. Something crosses the partition that should not.

It passes locally on every run I have done, which is weak evidence for (1) — a flaky test passes locally almost every time.

Suggested approach

Run it twenty times consecutively on CI-class hardware and count failures.

  • Fails even once → treat as real. Stop and find what is crossing.
  • Clean twenty times → timing. Tighten the assertion so CI cannot lose a race to it, and the file stops being a coin flip on every push.

Context

Deferred from the 0.33.0 release. The branch socket/admission-integrity (PR #70) does not touch the partition, wane, or heal paths, so whatever the cause is it predates that work.

Refs #66

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions