Repository navigation
[Bug]: crawler_configs is silently ignored for a single URL and on /crawl/stream #2287
Description
crawl4ai version
0.9.4
Expected Behavior
crawler_configs applies to every request that carries it: a /crawl with one URL, and /crawl/stream (or /crawl with stream: true). If some case can't support it, a 400 rather than a 200 that ran with a different config.
Current Behavior
Two gaps in the config-list support from #1852:
handle_crawl_requestuses the list only whenlen(urls) > 1(api.py#L726); with one URL it callsarun()withcrawler_config(#L749).stream_processdoesn't passcrawler_configsdown, andhandle_stream_crawl_requesthas no parameter for it (server.py#L1030-L1036, api.py#L889-L895).
Both answer 200, so the caller can't tell its per-URL settings were dropped. The single-URL case is pinned by tests/test_issue_1837_config_list.py::test_single_url_ignores_crawler_configs ("arun only takes one config"), but arun_many handles a one-URL list fine.
Is this reproducible?
Yes
Inputs Causing the Bug
- /crawl, one URL, crawler_configs: [{url_matcher: "*", css_selector: ".only-a"}]
- /crawl/stream, two URLs, the same listSteps to Reproduce
1. Page with two sections, .only-a and .only-b
2. POST /crawl with that one URL and the crawler_configs above
3. The markdown still contains the .only-b section
4. Same list on /crawl/stream: same resultCode snippets
import httpx
# URL: a page with
# <div class="only-a"><p>SECTION-A is the part a per-URL css_selector keeps.</p></div>
# <div class="only-b"><p>SECTION-B is the part it drops.</p></div>
per_url = [{"type": "CrawlerRunConfig",
"params": {"url_matcher": "*", "css_selector": ".only-a", "cache_mode": "bypass"}}]
r = httpx.post("http://localhost:11235/crawl", headers={"Authorization": f"Bearer {TOKEN}"},
json={"urls": [URL], "crawler_configs": per_url}, timeout=60)
md = r.json()["results"][0]["markdown"]["raw_markdown"]
print("SECTION-B" in md) # True: the per-URL css_selector was not appliedOS
Linux (Docker image built from develop @ 1f68e5b)
Python version
3.12 (the image's)
Browser
Chromium (Playwright, bundled)
Browser version
No response
Error logs & Screenshots (if applicable)
Local test page with .only-a / .only-b sections (CRAWL4AI_ALLOW_INTERNAL_URLS=true); A / B = that section is in the markdown:
[configs] /crawl, one URL, per-URL css_selector=.only-a -> HTTP 200: A=True B=True
[configs] /crawl/stream, two URLs, same list -> HTTP 200: A=True B=True; A=True B=True
Activity
Adam13y commentedon Oct 10, 2026
Thanks for the precise repro; your
.only-aand.only-bpage is already a great fixture. Known answer cases I could add: /crawl with one URL and css_selector.only-ain crawler_configs returns SECTION-A and not SECTION-B. /crawl/stream with two URLs and the same list gives that answer per URL. A list that cannot be honoured returns 400, not 200. test_single_url_ignores_crawler_configs would flip. Checks I proposed for fastbrowse caught a false success (fix in PR #240). Would a PR with these be useful once triaged?ntohidi commentedon Oct 12, 2026
Thanks for the clear report. Confirmed on
develop. Here is the root cause.Root cause
The schema says
crawler_configs"takes precedence over crawler_config" (schemas.py#L11-L17), but two paths drop it.One URL on
/crawl.handle_crawl_requestbuilds the config list only when there is more than one URL (api.py#L726). With one URL it callsarun, which takes a single config (api.py#L749).arun_manyhandles a one-URL list fine, so the length check is not needed.Streaming (
/crawl/stream, or/crawlwithstream: true).handle_stream_crawl_requesthas nocrawler_configsparameter (api.py#L889), andstream_processnever passes it (server.py#L1030). The non-streaming path does pass it (server.py#L996). Soarun_manyalways gets the singlecrawler_config(api.py#L967).In both cases the server returns 200, so the caller cannot tell the list was ignored.
An existing test checks for the single-URL behavior:
test_single_url_ignores_crawler_configs(test_issue_1837_config_list.py#L128).Fix direction
Send any request with
crawler_configstoarun_many, whatever the URL count, and pass the list through the streaming path with the same loading and checks. #2290 does this and is under review.talelboussetta commentedon Oct 12, 2026
Thanks @Adam13y. The fix for this is already open in #2290, and it has tests for these cases: one URL with a
crawler_configslist, the same list on/crawl/stream, and a list mixing PDF and browser entries being refused with a 400. It also carries a before/after run against a.only-a/.only-bpage. A second PR would overlap it, so I'd wait for #2290 to land and then open anything it doesn't cover.ntohidi commentedon Oct 12, 2026
The fix is in the
developbranch and will be released soon.Adam13y commentedon Oct 12, 2026
Thanks @ntohidi for the root cause and the quick fix, and @talelboussetta for the pointer. That makes sense, so I will not open a second PR. Once the release is out I will check the cases from my earlier comment against it and only raise anything it does not cover.
ntohidi commentedon Oct 14, 2026
Fixed by #2290, released in v0.9.5. Upgrade with
pip install -U crawl4ai. Release notes: https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.5.md