Repository navigation
Conversation
Errors raised from AsyncHTTPCrawlerStrategy._handle_http quoted the underlying aiohttp message, and InvalidURL's message is the whole proxy URL with the password in it. Mask user:pass@ in the wrapped message, and when the original carried credentials keep it out of the traceback. Refs unclecode#2326
ntohidi
approved these changes
Oct 12, 2026
ntohidi
left a comment
Collaborator
There was a problem hiding this comment.
Thanks @alkaz-nodemaven for this, and for checking it together with #2327.
What I verified
- Tests: merged with current
develop(which now includes #2327),test_http_proxy_error_redaction_2326.py,tests/proxy/test_http_proxy_url_encoding.pyandtest_http_timeout_unit_1894.pypass together (29). - No regressions:
tests/unit,tests/proxyandtests/general/test_http_crawler_strategy.pyshow no new failures against develop. - Still needed after #2327: with #2327, the
#and/passwords no longer causeInvalidURL, so_handle_httpno longer leaks them on develop. But a malformed proxyserver(bad host or port) still raisesInvalidURLcarrying the full URL with the password, and this PR masks it. It's good defense in depth. proxy_urlis set before thetry, so every handler can use it safely.
Small note (not blocking)
The test file would fit better in tests/proxy/, next to the #2327 test.
Approving. Thanks again!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refs #2326: the separate step mentioned there, hiding credentials in the wrapped error text. It does not touch
_format_proxy_url, so it is independent of #2327 (checked: the two merge cleanly and both test files pass together)._handle_httpre-raised aiohttp errors asHTTPCrawlerError/ConnectionError/ConnectionTimeoutErrorwithstr(e)in the message.InvalidURL's message is the whole proxy URL, so the password ended up in the error text, and also in the traceback through the chained original exception.Now every re-raise goes through
_wrap_error, which masksuser:pass@in the message (the exact proxy userinfo first, so a password with/or#is caught whole, then anyscheme://user:pass@pattern). When the original message carried credentials, its context is suppressed so the traceback does not print it under "During handling of the above exception". Errors without credentials keep their context as before.List of files changed and why
crawl4ai/async_crawler_strategy.py:_redact_proxy_credentialsand_wrap_error; the five re-raises in_handle_httpuse_wrap_error.tests/test_http_proxy_error_redaction_2326.py: unit tests for both helpers, plus_handle_httpagainst a closed local proxy port with passwordss3cret,s3cret#pw,pa/ss, asserting the password is in neither the message nor the formatted traceback.How Has This Been Tested?
tests/test_http_timeout_unit_1894.pyandtests/general/test_http_crawler_strategy.pystill pass.developwithout the change, the same_handle_httpcalls leaks3cret#pwandpa/ssin both message and traceback; with it, neither does.Checklist:
🤖 Generated with Claude Code