Repository navigation
[Bug]: HTTP crawler puts proxy credentials into the URL unencoded #2326
Description
crawl4ai version
0.9.4 (same code on develop at 1f68e5b)
Expected Behavior
AsyncHTTPCrawlerStrategy works through a proxy whose username or password contains #, /, ? or %, as the browser path already does (it passes username and password to Playwright as separate fields).
Current Behavior
_format_proxy_url (async_crawler_strategy.py:2634) builds f"{protocol}://{username}:{password}@{rest}" without percent-encoding. Results with aiohttp 3.14.3 against a local proxy:
| password | result |
|---|---|
plainpass, p@ss |
correct credentials sent |
p#ss, pa/ss, p?ss |
aiohttp.InvalidURL, request never sent |
p%41ss |
proxy receives pAss |
The InvalidURL message is the whole URL, and the except aiohttp.ClientError branch re-raises it as HTTPCrawlerError(f"HTTP client error: {str(e)}"), so the password ends up in the error text.
Is this reproducible?
Yes
Inputs Causing the Bug
ProxyConfig(server="http://host:port", username="user", password="p#ss") with AsyncHTTPCrawlerStrategy.
Steps to Reproduce
Pass the output of _format_proxy_url as proxy= to aiohttp.ClientSession.get, which is what the crawler does at async_crawler_strategy.py:2737.
Code snippets
from urllib.parse import quote
return f"{protocol}://{quote(username, safe='')}:{quote(password, safe='')}@{rest}"(both branches). Happy to send it as a PR to develop with a test.
OS
Windows 10
Python version
3.11
Activity
omarbounawarapy commentedon Oct 1, 2026
I'd like to take this. I reproduced it against a local proxy: passwords with
#,/or?raiseInvalidURL, and%gets decoded before it reaches the proxy. The fix is to percent-encode the username and password in_format_proxy_url. Should the password showing up in theHTTPCrawlerErrortext be fixed in the same PR, or separately?ntohidi commentedon Oct 12, 2026
Thanks for the clear report. Confirmed on
develop. Here is the root cause.Root cause
_format_proxy_urlputs the username and password into the proxy URL as they are (async_crawler_strategy.py#L2643-L2649):So
#,/and?end the user-info part of the URL early (aiohttp raisesInvalidURL), and%is read as an escape code (the proxy gets the wrong password).The password also reaches the error text:
InvalidURL's message is the full URL, and theexcept aiohttp.ClientErrorbranch wraps it asHTTPCrawlerError(f"HTTP client error: {str(e)}")(L2816-L2818). Withpassword="s3cret#pw"the error readshttp://user:s3cret#pw@127.0.0.1:9.Fix direction
Percent-encode the username and password with
quote(..., safe='')before building the URL. With that change the URL is valid, and the error above becomes a plain connection error with no password in it. #2327 does this and is under review.Hiding credentials in the wrapped error text is a separate safety step and can go in its own PR.
alkaz-nodemaven commentedon Oct 12, 2026
Thanks for confirming. I'll send the separate PR for the error text: mask proxy credentials in the messages
_handle_httpwraps intoHTTPCrawlerErrorand the connection/timeout errors, so a password doesn't reach logs through any error type, not onlyInvalidURL. It won't touch_format_proxy_url, so it doesn't overlap #2327.fix(http): keep proxy credentials out of HTTP crawler errors
ntohidi commentedon Oct 14, 2026
Fixed by #2327 and #2361, released in v0.9.5. Upgrade with
pip install -U crawl4ai. Release notes: https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.5.md