Repository navigation
fix: run fallback fetch after a single failed attempt - #2388
Open
JackyHsu-X wants to merge 1 commit into
Open
JackyHsu-X wants to merge 1 commit into
JackyHsu-X wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A fetch exception skips
fallback_fetch_functionwhenmax_retries=0and there is only one proxy/direct attempt. For example, a failed direct fetch returns an error without calling an explicitly configured fallback, while the same configuration with one retry reaches the fallback.Let that exception reach the existing fallback path when a fallback is available. This uses a directly constructed configuration and is separate from the config-cloning fix in #2385.
List of files changed and why
crawl4ai/async_webcrawler.py: preserve the early error path when neither another attempt nor a fetch fallback is available.tests/unit/test_fallback_fetch_without_retries.py: cover exception/403 recovery with zero and one retry, empty/raising fallbacks, no fallback, raw HTML, and successful fetches.docs/md_v2/advanced/anti-bot-and-fallback.md: clarify that a fallback does not require an extra retry after an exception.How Has This Been Tested?
arun()fromdevelopat2efcccc5d1838da1f7718813e9be5177ce579b49; 9 passed with the fix.git diff --checkand installed dependency checks pass.The fetch strategy is mocked; HTML processing and Markdown generation run through the real
AsyncWebCrawler.arun()path. No browser or live model call is needed. The full repository/browser integration suite was not run. The production module has existing formatting drift outside this change; its changed condition is unchanged by Black's range formatter.Checklist:
AI assistance was used for investigation, implementation, and validation. No independent human review of the full diff is claimed.