ScrapeUnblocker Ruby client

Official Ruby client for the ScrapeUnblocker web scraping API.

Every request is fully JavaScript-rendered in a real browser and routed through premium proxies, so it bypasses Cloudflare, DataDome, PerimeterX, Akamai, Kasada and similar anti-bot systems - from one simple call. You are only billed for successful requests.

Highest success rate on the market (95%+ on live production traffic)
Rendered HTML or parsed JSON - no per-site parsers to maintain
Zero dependencies (uses the standard library), typed errors

Install

gem install scrapeunblocker

Or in a Gemfile:

gem "scrapeunblocker"

Requires Ruby 2.7+.

Quickstart

require "scrapeunblocker"

su = ScrapeUnblocker::Client.new # reads SCRAPEUNBLOCKER_KEY, or Client.new(api_key: "YOUR_API_KEY")

# Rendered HTML for any URL
html = su.get_page_source("https://example.com")

# Structured JSON instead of HTML (products, listings, search results, ...)
product = su.get_parsed("https://www.amazon.com/dp/B08N5WRWNW")
puts product.page_type # "product"
p product.data

Get your API key at app.scrapeunblocker.com. The free trial does not require a credit card.

Authentication

Set an environment variable and the client picks it up:

export SCRAPEUNBLOCKER_KEY="YOUR_API_KEY"

su = ScrapeUnblocker::Client.new # reads SCRAPEUNBLOCKER_KEY

Fetch rendered HTML

html = su.get_page_source(
  "https://www.nordstrom.com/browse/women/clothing/dresses",
  proxy_country: "US", # route through a specific country
  time_sleep: 3        # wait extra seconds after load
)

Browser steps

Drive the page after it loads with an ordered list of browser actions - fill a form, click through, wait for content, then capture the rendered result. Steps run once in the real browser, in order.

html = su.get_page_source(
  "https://example.com/search",
  steps: [
    { action: "type", selector: "input#q", value: "web scraping", clear: true },
    { action: "press_key", value: "Enter" },
    { action: "wait_for", selector: "#results" },
    { action: "scroll", value: "bottom" }
  ]
)

Supported actions and their fields:

Action	Fields
`wait_for`	`selector`, `selector_type` (`css` default, `xPath`, `className`, `tagName`), `timeout_ms`
`wait_for_text`	`value` (text to await), `timeout_ms`
`wait`	`value` (milliseconds)
`click`	`selector`, `selector_type`, `timeout_ms`
`type`	`selector`, `selector_type`, `value`, `clear` (bool), `timeout_ms` - types like a human
`select`	`selector`, `selector_type`, `value`, `timeout_ms`
`press_key`	`value` - one of `Enter`, `Tab`, `Escape`, `Backspace`, `Delete`, `Space`, `ArrowUp`, `ArrowDown`, `ArrowLeft`, `ArrowRight`, `Home`, `End`, `PageUp`, `PageDown`
`scroll`	`value` - `"bottom"` or a pixel count

Steps are not idempotent (they submit forms, click buttons), so a call is not safe to blindly retry. If a step fails, the API answers HTTP 422 and the client raises ScrapeUnblocker::ValidationError; its body is the JSON describing which step failed (error: "step_failed", step_index, action, reason, selector, html).

List elements

Pass list_elements: true to get a JSON summary of the matched elements instead of the full HTML. The method then returns a parsed Hash ({"url", "count", "elements"}) rather than an HTML String.

result = su.get_page_source("https://example.com", list_elements: true)
puts result["count"]
result["elements"].each { |el| p el }

Get parsed JSON

result = su.get_parsed("https://www.walmart.com/ip/12345")
puts result.page_type # e.g. "product"
p result.data         # the fields

# If a parse ever comes back wrong, force a fresh set of rules:
fresh = su.get_parsed(url, refresh_rules: true, rules_hint: "price is missing")

Plugins

Ready-made scrapers for specific sites that return structured JSON. Each method takes the plugin's main input positionally and everything else as keyword options; options you leave out are not sent, so the API's own defaults apply. Every parameter is documented on the plugin pages.

products = su.target_search("coffee maker", sort: "price_low", max_results: 48)
milk = su.woolworths_product("888137")                     # one product, with nutrition
trends = su.google_trends(%w[coffee tea], geo: "US")       # a list is sent comma-separated
reviews = su.trustpilot_reviews("www.nike.com", stars: [4, 5])
jobs = su.career_site_jobs("https://jobs.ashbyhq.com/ramp", keyword: "engineer")

Category	Methods
Travel	`skyscanner.flights`, `skyscanner.hotels`, `skyscanner.carhire`, `booking_hotels`, `booking_reviews`, `airbnb_search`, `airbnb_room`, `airbnb_reviews`, `tripadvisor_list`, `tripadvisor_reviews`
Google	`google_local`, `google_images`, `maps_place_details`, `google_trends`, `google_shopping`, `google_jobs`
Search engines	`bing_search`, `bing_images`, `duckduckgo_search`, `yahoo_search`, `yahoo_images`, `brave_search`, `brave_images`
E-commerce	`amazon_product`, `amazon_search`, `amazon_bestsellers`, `amazon_reviews`, `walmart_search`, `target_search`, `bestbuy_search`, `homedepot_search`, `etsy_search`, `temu_search`, `temu_product`, `shein_search`, `zalando_search`, `aliexpress_search`, `alibaba_search`, `alibaba_product`, `alibaba1688_search`, `alibaba1688_product`, `tiktok_shop_search`, `tiktok_shop_product`, `tiktok_shop_store`, `oopbuy_search`
Classifieds & Resale	`ebay_search`, `vinted_search`, `leboncoin_search`
Groceries	`woolworths_search`, `woolworths_category`, `woolworths_specials`, `woolworths_product`, `coles_search`, `coles_category`, `coles_specials`, `coles_product`, `aldi_au_search`, `aldi_au_category`, `aldi_au_specials`, `aldi_au_product`
Real estate	`zillow_search`, `zillow_detail`, `redfin_search`, `redfin_detail`, `rightmove_search`, `idealista_search`, `idealista_detail`, `immobilienscout24_search`
Vehicles	`mobile_de`, `autoscout24_search`
Jobs	`indeed_jobs`, `glassdoor_jobs`, `glassdoor_reviews`, `glassdoor_salaries`, `naukri_jobs`, `wellfound_jobs`, `wellfound_job`, `wellfound_company`, `wellfound_company_search`, `career_site_jobs`
Reviews & Companies	`trustpilot_reviews`, `g2_reviews`, `g2_search`, `capterra_reviews`, `capterra_search`, `product_hunt`, `crunchbase_company`, `crunchbase_search`
Local & Maps	`yelp_search`, `yelp_business`, `yelp_reviews`, `naver_place_search`, `naver_place_details`
Apps	`google_play_app`, `google_play_search`, `app_store_app`, `app_store_search`, `app_store_reviews`
Video & Music	`youtube_channel`, `youtube_search`, `youtube_transcript`, `spotify_search`, `spotify_artist`, `spotify_album`, `spotify_track`, `spotify_playlist`
SEO & Ads	`similarweb_overview`, `ahrefs`, `semrush_overview`, `meta_ad_library`

Any plugin endpoint can also be called by path, with parameters named as the API names them - handy for an endpoint added after your gem version:

data = su.plugin("/marketplace/target-search", query: "tv", max_results: 24)

Google search (SERP)

serp = su.serp("web scraping api", pages_to_check: 2, proxy_country: "US")

Google Local (Maps)

local = su.google_local("coffee shops in chicago", proxy_country: "US", gl: "us")
local["results"].each { |biz| puts "#{biz['name']} #{biz['rating']} #{biz['address']}" }

Google Images

images = su.google_images("golden retriever puppy", proxy_country: "US", pages: 3)
images["results"].each { |img| puts "#{img['imageUrl']} #{img['sourceDomain']} #{img['title']}" }

# Only images Google found in the last day (for a daily scan of the same query)
new_images = su.google_images("eiffel tower", proxy_country: "US", time_range: "day")
# Or a fixed window: date_from / date_to (YYYY-MM-DD), each one optional
october = su.google_images("eiffel tower", date_from: "2026-10-01", date_to: "2026-10-07")

time_range takes hour, day, week, month or year and cannot be combined with date_from / date_to.

Meta Ad Library

ads = su.meta_ad_library("Nike", country: "US")
ads["results"].each { |ad| puts "#{ad['id']} #{ad['media_type']}" }

advertiser is required. Optional filters: country, active_status, media_type and max_ads. Unset filters are dropped and the API applies its own defaults.

Oopbuy goods search

goods = su.oopbuy_search("running shoes", channel: "1688", page: 1, page_size: 20, sort: "default")
goods["results"].each { |item| puts "#{item['title']} #{item['price']} #{item['url']}" }

channel is one of "1688" (default), "taobao" or "official". sort is one of "default", "price_asc", "price_desc" or "best_selling". page_size max is 60. Oopbuy trademark-blocks brand keywords at its own backend: those come back as a successful 200 with keywordRejected: true and an empty results array, not an error.

Amazon

Product and search data as a Hash, priced in the marketplace's own currency:

# One product by ASIN (or url: "https://www.amazon.de/dp/B0BSHF7WHW")
product = su.amazon_product(asin: "B0BSHF7WHW", marketplace: "amazon.com")
puts [product["title"], product["price"], product["currency"], product["rating"]].join(" | ")

# Keyword search
results = su.amazon_search("wireless headphones", sort: "price_asc")
results["results"].each do |item|
  puts [item["title"], item["price"], item["currency"], item["asin"]].join(" | ")
end

proxy_country defaults to the marketplace's home country (amazon.com -> US, amazon.de -> DE), so prices come back in the right currency with no configuration.

eBay search

items = su.ebay_search("iphone 13", marketplace: "ebay.com", condition: "used", sort: "newly_listed")

if items["exactMatches"]
  items["results"].each do |item|
    puts [item["title"], item["price"], item["currency"], item["condition"]].join(" | ")
  end
end

marketplace is any of the 19 regional eBay hosts (ebay.com default). condition is one of "new", "open_box", "refurbished", "used" or "for_parts"; sort is one of "best_match" (default), "newly_listed", "ending_soon", "price_asc" or "price_desc"; page_size is 60, 120 or 240. exactMatches is false when eBay found nothing for the keyword and answered with its own loosely-related suggestions instead, so check it before using the listings.

Cookies and the serving proxy

page = su.get_page_with_cookies("https://example.com")
puts page.html
p page.cookies
puts page.proxy

Images

bytes = su.get_image("https://example.com/photo.jpg")
File.binwrite("photo.jpg", bytes)

The bytes come back exactly as the image host serves them, in the original format (JPEG, PNG, WebP, AVIF, GIF, SVG, ...); nothing is re-encoded. Pass the image URL itself, not the page it sits on.

Skyscanner plugins

Flights, hotels and car hire as JSON:

locations = su.skyscanner.flight_locations("London")

flights = su.skyscanner.flights(
  origin: "London", dest: "New York",
  depart_date: "2026-09-01", adults: 1, currency: "USD"
)

hotels = su.skyscanner.hotels(destination: "Madrid", checkin: "2026-09-01", checkout: "2026-09-03")
cars = su.skyscanner.carhire(pickup: "Madrid", pickup_datetime: "2026-09-01T10:00", dropoff_datetime: "2026-09-03T10:00")

Error handling

Non-2xx responses raise typed errors, all subclasses of ScrapeUnblocker::Error.

begin
  html = su.get_page_source("https://example.com")
rescue ScrapeUnblocker::BlockedError
  # 403: the target blocked every bypass path (not billed)
rescue ScrapeUnblocker::PaymentRequiredError
  # 402: quota, credit limit, your budget limit, or a failed payment - fix billing
rescue ScrapeUnblocker::RateLimitError
  # 429: slow down
rescue ScrapeUnblocker::UpstreamOutageError
  # 503: the target site itself is down - retry later
end

Error	Status	Meaning
`InvalidRequestError`	400	Bad URL, unsupported scheme, or the API key header was not sent
`AuthenticationError`	401	Key not recognised - typo, stray whitespace, or a rotated key
`NoSubscriptionError`	401	Key is fine, but the account has no active plan
`PaymentRequiredError`	402	Billing block - base class for the four below
`QuotaExceededError`	402	The plan's requests for this period are used up
`CreditLimitExceededError`	402	Unpaid balance is past the account's credit limit
`BudgetExceededError`	402	This billing period's spend reached the monthly budget limit you set
`PaymentFailedError`	402	A card payment was declined three times
`BlockedError`	403	Blocked by bot protection on every path
`NotFoundError`	404	What you asked for does not exist - a plugin lookup found nothing
`TargetNotFoundError`	404 / 410	The target page itself does not exist; carries `origin_status`, `html`, `destination_url` (subclass of `NotFoundError`, billed)
`BrowserTimeoutError`	408	Our browser run timed out before the page was ready
`UnsupportedContentError`	415	The URL serves something other than HTML (`get_page_source`) or other than an image (`get_image`)
`ValidationError`	422	Missing or wrong-typed parameter; `body` holds the `detail` array
`RateLimitError`	429	Too many requests
`UpstreamOutageError`	503	The target origin is down
`ServerError`	5xx	Unexpected server error, including a 504 upstream timeout
`TimeoutError`	-	This client gave up locally before the API answered
`ConnectionError`	-	Could not reach the API

The target page does not exist (404 / 410)

When the site you scrape answers 404 or 410 itself, the API passes that status through with an X-Origin-Status header, and the client raises ScrapeUnblocker::TargetNotFoundError. It is the target's final answer, so it is never retried, and it is billed like any delivered page. The not-found page is on #html:

begin
  html = su.get_page_source("https://example.com/removed-listing")
rescue ScrapeUnblocker::TargetNotFoundError => e
  puts e.origin_status   # 404 or 410
  puts e.html            # the target's own not-found page (can be empty)
end

TargetNotFoundError subclasses NotFoundError, so rescue ScrapeUnblocker::NotFoundError catches it too. A 404 without X-Origin-Status is the API's own and stays a plain NotFoundError.

With get_parsed the body is the parsed-data JSON ({"data": {"page_type": "not_found", ...}}), so html is nil there; the raw JSON is on #body.

No structured data on the page

When get_parsed renders the page but can extract no structured data from it, the API still answers 200 and the client returns a normal ParsedPage: data is empty, data_extracted? is false, detail says why, and html carries the rendered page. The call is billed like any delivered page, so you already have the HTML - no second call needed:

page = su.get_parsed("https://example.com/some-page")
unless page.data_extracted?
  puts page.detail       # the API's explanation
  html = page.html       # the rendered page, parse it yourself
end

On a successful parse data_extracted? is true and html is nil. Up to 0.7.1 this case raised NoDataExtractedError (a 422); the API no longer sends that, and the class stays only so existing rescue clauses keep loading.

Transient failures (429, 502, 503, 504 and network errors) are retried automatically with exponential backoff. A 401 or 402 is never retried - it clears when the key or the billing state changes, not on another attempt. Neither is billed or counted against your quota, because the request is refused before anything is scraped.

Billing errors (402)

The four billing blocks share a status code and differ only in their message, so the client raises a dedicated error for each:

begin
  html = su.get_page_source("https://example.com")
rescue ScrapeUnblocker::QuotaExceededError
  # plan quota (plus any overage allowance) is used up for this period
rescue ScrapeUnblocker::CreditLimitExceededError
  # unpaid balance passed the account credit limit
rescue ScrapeUnblocker::BudgetExceededError
  # this period's spend reached the monthly budget limit set in your profile
rescue ScrapeUnblocker::PaymentFailedError
  # card declined three times - update the payment method
end

When more than one applies, the most serious wins: failed payment outranks credit limit, which outranks quota, which outranks your own budget limit. All four lift by themselves once the billing state changes - access returns within about a minute, and the API key stays the same. One catch worth knowing: subscribing to a new plan does not clear PaymentFailedError, because the old unpaid invoice stays open until it is paid.

BudgetExceededError (body User set budget exceeded) means this billing period's spend reached the monthly budget limit you set in your profile (EUR, excluding VAT). The key works again at the start of the next billing period, or within about a minute after you raise or remove the limit.

Full details for every status code: docs.scrapeunblocker.com/errors.

Configuration

ScrapeUnblocker::Client.new(
  api_key: nil,        # or SCRAPEUNBLOCKER_KEY env var
  base_url: "https://api.scrapeunblocker.com",
  timeout: 180,        # seconds; protected pages can be slow
  max_retries: 2
)

Links

License

MIT

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
lib		lib
test		test
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
Gemfile		Gemfile
LICENSE		LICENSE
README.md		README.md
Rakefile		Rakefile
scrapeunblocker.gemspec		scrapeunblocker.gemspec

Navigation Menu

Repository navigation

About

Repository files navigation

ScrapeUnblocker Ruby client

Install

Quickstart

Authentication

Fetch rendered HTML

Browser steps

List elements

Get parsed JSON

Plugins

Google search (SERP)

Google Local (Maps)

Google Images

Meta Ad Library

Oopbuy goods search

Amazon

eBay search

Cookies and the serving proxy

Images

Skyscanner plugins

Error handling

The target page does not exist (404 / 410)

No structured data on the page

Billing errors (402)

Configuration

Links

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

ScrapeUnblocker Ruby client

Install

Quickstart

Authentication

Fetch rendered HTML

Browser steps

List elements

Get parsed JSON

Plugins

Google search (SERP)

Google Local (Maps)

Google Images

Meta Ad Library

Oopbuy goods search

Amazon

eBay search

Cookies and the serving proxy

Images

Skyscanner plugins

Error handling

The target page does not exist (404 / 410)

No structured data on the page

Billing errors (402)

Configuration

Links

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages