Repository navigation
Fix #3364: web search robustness - handle non-2xx responses and filter search engine self-links - #3365
Merged
Conversation
…ters search engine self-links - Add HTTP status code check to execute_web_search: return error if response is not 2xx instead of proceeding with potentially junk results (e.g., DuckDuckGo challenge pages) - Filter out links pointing back to the search engine host in extract_search_hits_from_generic_links to avoid returning search engine self-links when no proper search results are found - Add regression tests for both scenarios Refs #3364
3 tasks done
Yeachan-Heo
marked this pull request as draft
October 23, 2026 13:08
added 2 commits
October 23, 2026 13:13
…responses - Replace exact host matching with registrable domain comparison to properly filter search engine self-links across subdomains (e.g., html.duckduckgo.com vs duckduckgo.com) - Add error handling for 2xx responses with no search results (e.g., 202 challenge pages from blocked search providers) - Add regression tests for both 202 challenge page and mixed self-link scenarios - Remove unrelated Cargo.lock version bumps (revert to 0.1.6) Fixes #3364
…ine links The challenge page arrives as HTTP 202, which is_success() accepts, and its links point at duckduckgo.com while the request goes to html.duckduckgo.com, so the previous filter kept them. Only a 200 page is parsed now, and the generic fallback drops any duckduckgo.com link or link to the configured search host. Removes the registrable-domain heuristic and the 'no hits is an error' path (a real query can legitimately return nothing), and replaces tests that never exercised a self-link with ones that fail on the old code. Refs #3364
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the bug reported in issue #3364 where DuckDuckGo challenge pages (HTTP 202 or similar without proper search result markup) would return junk results pointing back to DuckDuckGo itself.
Changes
What this fixes
Refs #3364
Note: The optional provider proposal from #3364 (CLAWD_WEB_SEARCH_PROVIDER=keenable) is left for maintainer review as a separate product decision.