tests/test_pay.py::test_error_returns_blockheight fails intermittently on master with KeyError: 'raw_message':
Root cause is a race, not a broken assertion. Normally waitsendpay attaches while the payment is still pending, so the error is built from the live failure and data.raw_message carries the decrypted BOLT4 blob the test checks. But when the whole HTLC-fail round trip (peer's update_fail_htlc, both commitment dances, DB write) completes before lightningd dispatches the waitsendpay command, wait_payment() takes the PAYMENT_FAILED branch and rebuilds the error from the database - which does not persist the raw failure message:
/* FIXME: We don't store this! */
fail->msg = NULL;
(lightningd/pay.c, FIXME dates to 2019, d943d8a)
So raw_message vanishes from the error data exactly when the race is lost. This test is maximally exposed: single hop with the direct peer as the failer, i.e. the shortest possible failure round trip racing one local RPC dispatch.
Proposed fix: persist the raw fail message in the payment fail info (schema migration + wallet_payment_set_failinfo/wallet_payment_get_failinfo, then set fail->msg from it in wait_payment). That resolves the 2019 FIXME and makes the waitsendpay error shape consistent regardless of timing, rather than weakening the test, which exists precisely to check the raw blob contents.
Related but distinct: #7430 documents different fields going missing on the sendpay_failure notification path - same underlying theme of failure fields varying by code path.
Part of #9222.
tests/test_pay.py::test_error_returns_blockheightfails intermittently on master withKeyError: 'raw_message':Root cause is a race, not a broken assertion. Normally
waitsendpayattaches while the payment is still pending, so the error is built from the live failure anddata.raw_messagecarries the decrypted BOLT4 blob the test checks. But when the whole HTLC-fail round trip (peer's update_fail_htlc, both commitment dances, DB write) completes before lightningd dispatches thewaitsendpaycommand,wait_payment()takes the PAYMENT_FAILED branch and rebuilds the error from the database - which does not persist the raw failure message:(lightningd/pay.c, FIXME dates to 2019, d943d8a)
So
raw_messagevanishes from the error data exactly when the race is lost. This test is maximally exposed: single hop with the direct peer as the failer, i.e. the shortest possible failure round trip racing one local RPC dispatch.Proposed fix: persist the raw fail message in the payment fail info (schema migration +
wallet_payment_set_failinfo/wallet_payment_get_failinfo, then setfail->msgfrom it inwait_payment). That resolves the 2019 FIXME and makes thewaitsendpayerror shape consistent regardless of timing, rather than weakening the test, which exists precisely to check the raw blob contents.Related but distinct: #7430 documents different fields going missing on the
sendpay_failurenotification path - same underlying theme of failure fields varying by code path.Part of #9222.