Truncate large coro repr in retry log output by ernestprovo23 · Pull Request #9197 · dask/distributed

ernestprovo23 · 2026-02-22T15:41:22Z

Summary

When retry() falls back to str(coro) for log messages, truncate the representation to 200 characters to prevent excessively large logs
This was observed in P2P shuffles where functools.partial repr includes serialized binary data (see Fix excessive logging on P2P retry #8511)
Added test verifying truncation behavior

Closes #8529

Test plan

New test test_retry_truncates_large_coro_repr verifies 500-char repr is truncated
Existing retry tests unaffected (no truncation when repr is small)
Explicit operation parameter bypasses truncation (preserves full user-provided descriptions)

🤖 Generated with Claude Code

github-actions · 2026-02-22T16:29:37Z

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

31 files ± 0 31 suites ±0 10h 53m 10s ⏱️ - 7m 48s
4 074 tests + 2 3 968 ✅ + 2 104 💤 ±0 2 ❌ ±0
59 131 runs +30 56 696 ✅ +30 2 433 💤 ±0 2 ❌ ±0

For more details on these failures, see this check.

Results for commit 630991a. ± Comparison against base commit deb1004.

♻️ This comment has been updated with latest results.

jacobtomlinson

There is a recommended solution in the issue to use reprlib to help with this truncation instead of simply chopping the end of the string off #8529 (comment).

ernestprovo23 · 2026-03-07T00:23:03Z

thanks for catching that, updated to use reprlib.Repr() with maxother=200 instead of manual slicing. keeps the truncation logic cleaner and more standard.

ernestprovo23 · 2026-03-07T02:49:58Z

hey @jacobtomlinson — just checking in on this. the reprlib approach is in place per your suggestion (lines 385-390). the Windows CI failures look like pre-existing flakiness (investigating now), all Linux and macOS tests pass. happy to adjust anything else if needed.

jacobtomlinson

Left a couple more comments

jacobtomlinson · 2026-03-10T14:36:59Z

+                aRepr = reprlib.Repr()
+                aRepr.maxother = 200
+                operation = aRepr.repr(coro)


Let's clean this up into a simple one-liner.

Suggested change

aRepr = reprlib.Repr()

aRepr.maxother = 200

operation = aRepr.repr(coro)

operation = reprlib.Repr(maxother=200).repr(coro)

done — condensed to a one-liner.

turns out Repr() doesnt accept kwargs per mypy, maxother is an instance attribute. switched to a two-liner that sets it after init.

jacobtomlinson · 2026-03-10T14:37:50Z

+            jitter_fraction=0,
+        )
+
+    import logging


Can you move imports to the top of the file. Claude loves importing stuff randomly in the middle of code for some reason.

moved to the top, good catch.

jacobtomlinson · 2026-03-10T14:38:26Z

+    handler = logging.Handler()
+    handler.emit = lambda record: log_messages.append(record.getMessage())
+
+    logger = logging.getLogger("distributed.utils_comm")
+    logger.addHandler(handler)
+    try:
+        with pytest.raises(MyEx):
+            asyncio_run(f(), loop_factory=get_loop_factory())
+    finally:
+        logger.removeHandler(handler)


Can you just use pytest's caplog instead of doing all this?

yep, switched to caplog — much cleaner.

ernestprovo23 · 2026-03-30T23:12:18Z

Hi @fjetter — just checking in on this. The reprlib approach from the Mar 26 commit addresses the earlier feedback (using reprlib.Repr() with maxother set post-init to satisfy mypy, plus caplog in tests). The "not ci1" failures appear pre-existing and unrelated. Let me know if there's anything else you'd like adjusted.

crusaderky · 2026-05-07T10:05:46Z

@ernestprovo23 CI is failing; could you look into it before the next round of review?

ernestprovo23 · 2026-05-07T10:19:04Z

np. i will check out today

When retry() falls back to str(coro) for the log message, truncate the representation to 200 characters to prevent excessively large logs. This was observed in P2P shuffles where functools.partial repr includes serialized binary data. Closes dask#8529

Replace manual str[:200] + "..." truncation with stdlib reprlib.Repr() per reviewer feedback. reprlib handles truncation consistently and is a well-known stdlib pattern for this use case.

- Condense reprlib setup to single line per suggestion - Replace manual logging handler with pytest caplog fixture - Move logging import to top of test file

distributed/config.py sets propagate=False on the top-level 'distributed' logger and attaches its own handler. Records from distributed.utils_comm propagate up to 'distributed' but stop there, so pytest's caplog (attached to root) never sees them. Use monkeypatch to flip propagate=True on the 'distributed' logger for the test duration.

ernestprovo23 · 2026-05-19T01:55:58Z

@crusaderky CI fixed. The caplog capture was silently dropping records because distributed/config.py sets propagate=False on the top-level distributed logger (and attaches its own stderr handler there), so records from distributed.utils_comm stop at the distributed logger and never reach caplog's root handler. Used monkeypatch.setattr(logging.getLogger('distributed'), 'propagate', True) to flip it for the test duration. Verified locally; also rebased on main. Ready for another look.

crusaderky · 2026-05-19T09:54:50Z

@crusaderky CI fixed. The caplog capture was silently dropping records because distributed/config.py sets propagate=False on the top-level distributed logger (and attaches its own stderr handler there), so records from distributed.utils_comm stop at the distributed logger and never reach caplog's root handler. Used monkeypatch.setattr(logging.getLogger('distributed'), 'propagate', True) to flip it for the test duration. Verified locally; also rebased on main. Ready for another look.

We use captured_logger to avoid this issue. I fixed it for you.

ernestprovo23 requested a review from fjetter as a code owner February 22, 2026 15:41

jacobtomlinson requested changes Mar 3, 2026

View reviewed changes

jacobtomlinson requested changes Mar 10, 2026

View reviewed changes

ernestprovo23 added 5 commits May 18, 2026 21:50

Use reprlib for truncating large coro repr in retry logs

0590d81

Replace manual str[:200] + "..." truncation with stdlib reprlib.Repr() per reviewer feedback. reprlib handles truncation consistently and is a well-known stdlib pattern for this use case.

Address review feedback: one-liner reprlib, caplog, top-level imports

7f0bdda

- Condense reprlib setup to single line per suggestion - Replace manual logging handler with pytest caplog fixture - Move logging import to top of test file

Fix mypy error: Repr() doesn't accept maxother as kwarg

7aa9c06

ernestprovo23 force-pushed the fix-retry-excessive-logs branch from 40472e5 to ac79e3e Compare May 19, 2026 01:55

Review test

630991a

crusaderky approved these changes May 19, 2026

View reviewed changes

crusaderky merged commit c813e6d into dask:main May 19, 2026
33 of 37 checks passed

ernestprovo23 deleted the fix-retry-excessive-logs branch May 19, 2026 10:55

Uh oh!

Conversation

ernestprovo23 commented Feb 22, 2026

Summary

Test plan

Uh oh!

github-actions Bot commented Feb 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Unit Test Results

Uh oh!

jacobtomlinson left a comment

Choose a reason for hiding this comment

Uh oh!

ernestprovo23 commented Mar 7, 2026

Uh oh!

ernestprovo23 commented Mar 7, 2026

Uh oh!

jacobtomlinson left a comment

Choose a reason for hiding this comment

Uh oh!

jacobtomlinson Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

ernestprovo23 Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

ernestprovo23 Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

jacobtomlinson Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

ernestprovo23 Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

jacobtomlinson Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

ernestprovo23 Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

ernestprovo23 commented Mar 30, 2026

Uh oh!

crusaderky commented May 7, 2026

Uh oh!

ernestprovo23 commented May 7, 2026

Uh oh!

ernestprovo23 commented May 19, 2026

Uh oh!

crusaderky commented May 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Feb 22, 2026 •

edited

Loading