Percent-encode proxy credentials in the HTTP crawler proxy URL

[omarbounawarapy]

Percent-encodes the proxy username and password when building the URL for the HTTP crawler, so #, /, ? and % in credentials work. Tested against a local proxy. Closes #2326

[ntohidi]

Thanks @omarbounawarapy for the fix. Encoding the credentials in _format_proxy_url is the right place for it, since that's the only place that builds a proxy URL with credentials.

What I verified

• Tests: the 6 new tests pass. With develop's async_crawler_strategy.py, all 6 fail.

One change needed before we merge

Many users pass the proxy as an already-encoded URL string, for example PROXIES entries or ProxyConfig.from_string("http://user:p%40ss@host:8080"), where the real password is p@ss. from_string keeps the credentials as written (p%40ss) and doesn't decode them. So:

• develop: _format_proxy_url passes p%40ss through, and aiohttp decodes it, so the proxy receives p@ss (correct).
• this PR: it's encoded again to p%2540ss, so the proxy receives p%40ss (wrong password).

I checked this with yarl.URL(strategy._format_proxy_url(cfg)).password on both branches.

Suggested fix: decode the credentials in ProxyConfig.from_string with urllib.parse.unquote(username) and unquote(password). Then ProxyConfig always holds the real values, and _format_proxy_url encodes them exactly once. As a bonus, this also fixes the browser path: Playwright currently receives p%40ss from an encoded URL string.

Please add a test that ProxyConfig.from_string("http://user:p%40ss@host:8080") ends up sending p@ss.

Small note

The branch is based on an older develop. A rebase onto the current develop would be good.

Thanks again! Happy to approve once that's in.

[omarbounawarapy]

Thanks for the review. ProxyConfig.from_string now unquotes the username and password, so http://user:p%40ss@host:8080 gives p@ss, and _format_proxy_url encodes it once. I added a test for that case and rebased onto current develop.

[ntohidi]

Thanks @omarbounawarapy, that's exactly it.

What I verified

• Tests: all 7 tests pass, including test_encoded_proxy_string_is_sent_once_encoded. With develop's code, all 7 fail.
• No regressions: tests/unit and tests/proxy show no new failures against develop.
• Browser path: browser_manager.py passes proxy_config.password to Playwright as-is, so encoded proxy strings now also give the browser the right password.

Approving. Thanks again!