feat(google_search): fall back to SearXNG when Google walls every IP

[AnishSarkar22]

Description

• When Google walls or captchas every proxy IP, fetch_serp_html returns nothing and the scraper drops that page and stops paging. This adds a bundled SearXNG instance as a last-resort provider so a walled run returns organic results instead of an empty one.
• New platforms/google_search/searxng.py queries the instance's JSON API and builds a SerpItem directly, translating only what SearXNG actually honors: site:, exact-match quoting, languageCode → language, and plain quickDateRange units → time_range. Compound codes like m6 are dropped rather than approximated, and results are capped at 10 so a fallback page never looks anomalously large to a caller that is paging.
• Wired in at _serp_page_flow, the single seam every caller shares, so REST, MCP, the playground, and the chat subagent all get the fallback without surface-specific code.
• Skipped when it could only produce a guaranteed miss: focusOnPaidAds (aggregators serve no ads), a URL whose q could not be read, or no SEARXNG_URL configured — in which case behavior is exactly as before.
• Fallback pages are labelled rather than passed off as Google: items carry resultsProvider: "searxng", and _search_query_stamp gained a domain override so searchQuery.domain reads searxng instead of claiming google.com. The label is the provider name, not the configured host, which is usually a private address that should not reach API output.
• billable_units now skips items carrying resultsProvider. A fallback page costs no proxy render and no captcha solve, so it is not charged at the Google SERP rate; the pre-flight gate still reserves worst case, since a fallback cannot be predicted before trying.
• The google_search subagent prompt now requires disclosing fallback provenance and forbids describing those items as Google rankings or positions.
• Restores the searxng service to the prod, dev, and deps-only stacks on a pinned image with a healthcheck, and injects SEARXNG_URL (default http://searxng:8080) into backend and celery_worker. Deliberately no depends_on, so an unhealthy SearXNG can never block backend startup — it is a fallback, not a dependency.
• Adds docker/searxng/settings.yml and limiter.toml. Two entries there are load-bearing: json in search.formats, without which every fallback request 403s, and the google engine explicitly disabled, since this instance only runs after Google walled us and would query it from the same address.
• install.sh and install.ps1 create the searxng/ directory and fetch both config files, so a fresh self-hosted install is complete.
• Both .env.example files document SEARXNG_URL and SEARXNG_TIMEOUT_S with how to point at an external instance or disable the fallback, plus SEARXNG_PORT for the port the dev and deps-only stacks expose.
• web-search.mdx documents the bundled fallback and corrects the stale note claiming SearXNG was removed entirely.
• Two diagnostics for the failure modes that were previously silent: a warning when the unproxied single render path gives up, and a 403 from SearXNG that names the exact settings.yml knob to fix.
• Covered by 7 offline tests: results served, operator folding, disabled, ads-requested, unreadable query, the 403 log, and a Google success left unstamped and untouched. Verified end to end through the playground with Google forced to fail, confirming the three-stage log trail and resultsProvider reaching the run payload.

Motivation and Context

FIX #

Screenshots

API Changes

• This PR includes API changes

Change Type

• Bug fix
• New feature
• Performance improvement
• Refactoring
• Documentation
• Dependency/Build system
• Breaking change
• Other (specify):

Testing Performed

• Tested locally
• Manual/QA verification

Checklist

• Follows project coding standards and conventions
• Documentation updated as needed
• Dependencies updated as needed
• No lint/build errors or new warnings
• All relevant tests are passing

High-level PR Summary

This PR adds an optional SearXNG fallback mechanism for the Google Search scraper to provide organic search results when Google walls all proxy IPs. The fallback is controlled via environment variables (SEARXNG_FALLBACK_URL and timeout settings) and only activates after Google's retry attempts are exhausted. Fallback results are tagged with resultsProvider: "searxng" to distinguish them from native Google results, and they are not billed at the Google SERP rate. The implementation includes Docker configuration files for running a SearXNG instance, core fallback logic that maps aggregator responses to the existing SerpItem schema (organic results and suggestions only, no ads or PAA), and comprehensive unit tests to verify the fallback wiring and edge cases.

⏱️ Estimated Review Time: 30-90 minutes

💡 Review Order Suggestion

Order	File Path
1	surfsense_web/content/docs/how-to/web-search.mdx
2	docker/searxng/settings.yml
3	docker/searxng/limiter.toml
4	surfsense_backend/app/proprietary/platforms/google_search/searxng.py
5	surfsense_backend/app/proprietary/platforms/google_search/scraper.py
6	surfsense_backend/app/capabilities/google_search/scrape/schemas.py
7	surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_search/system_prompt.md
8	surfsense_backend/tests/unit/platforms/google_search/test_searxng_fallback.py

[vercel]

@AnishSarkar22 is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel.

A member of the Team first needs to authorize it.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: fd1e6278-ed06-4c6d-8318-4a479007c5a4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}