Skip to content

[Bug]: crwl always bypasses the cache, -c cache_mode=enabled is ignored #2365

Closed

Description

@ntohidi

--bypass-cache is declared as a flag with default=True, in both crwl and crwl crawl:

@click.option("--bypass-cache", "-bc", is_flag=True, default=True, help="Bypass cache when crawling")

So bypass_cache is always True, and this runs after -c / --crawler-config are applied:

if bypass_cache:
    crawler_cfg.cache_mode = CacheMode.BYPASS

Because of that, -c cache_mode=enabled (or a crawler config file that sets it) is silently ignored, and the CLI never reads or writes the cache.

Repro (0.9.4)

crwl https://example.com -o md -c cache_mode=enabled
crwl https://example.com -o md -c cache_mode=enabled

Both runs fetch the page again, and ~/.crawl4ai/crawl4ai.db never gets a crawled_data table. The same thing from Python works fine: CrawlerRunConfig(cache_mode=CacheMode.ENABLED) gives cache_status miss, then hit.

Fix idea

Make it default=False (keep the flag), or turn it into --bypass-cache/--use-cache. Then only override cache_mode when the user did not set it with -c. Today crwl bypasses the cache by default, so we probably want to keep that as the default and just let -c cache_mode=... win.

Found while looking at the community plugins in #2316. Both of them pass -c cache_mode=enabled and expect caching to work.

Activity

  1. added a commit that references this issue on Oct 14, 2026
  2. added
    ⚙ DoneBug fix, enhancement, FR that's completed pending release
    on Oct 14, 2026
  3. ntohidi commented on Oct 14, 2026

    @ntohidi
    SponsorCollaboratorAuthor

    Fixed by #2366, released in v0.9.5. Upgrade with pip install -U crawl4ai. Release notes: https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.5.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ⚙ DoneBug fix, enhancement, FR that's completed pending release🐞 BugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions