A scraper that works out which detection layer is blocking it —
and escalates only as far as that layer requires.
The model · Documentation · Examples · White paper · Live report · Changelog
from scraper import Scraper
with Scraper(origin="https://example.com") as scraper:
soup = scraper.get_soup("https://example.com/")
print(soup.select_one("h1").text)Those four lines already reproduce a real browser's TLS and HTTP/2 fingerprint, hold one address per origin, pace themselves like a person reading, and refuse to follow decoy links.
A modern mitigation engine runs many largely independent detectors and folds them into one trust score. Admission behaves as a near-conjunction, so:
P(evade) ≲ min( p₁ , p₂ , p₃ , … , pₙ )
▲
└─ the binding layer. Every other layer is
wasted effort until this one stops being
the minimum.
The weakest layer bounds the outcome. If a strategy fails on address reputation, perfecting its TLS profile gains nothing — not a little, zero. So this library diagnoses which layer is binding before it changes anything.
What a detector reads decides whether it can be satisfied.
| The detector reads | Reproducible? | What actually moves it | |
|---|---|---|---|
| Emitted | an artifact the client sends — a TLS ClientHello, an HTTP/2 frame order, a header order |
yes | imitate it faithfully |
| Possessed | a property the client must hold — accumulated per-zone history, a private signing key | no | accrue it, rent it, or hold the key |
That second distinction produces the behaviour that most sets this library apart: when the binding layer reads a possessed property, it does not rotate. Rotating discards the very history the detector is measuring, so it holds the address still and slows down. And two layers raise instead of retrying, because they read a secret you either hold or do not.
Full treatment: the model. It comes from the paper this library implements — A Layered Model of Modern Web Bot Protection and the Structural Limits of Its Circumvention, included as a PDF.
| Diagnoses instead of reacting | A response is mapped onto the nineteen-layer model, not onto its status code. A 200 carrying a challenge is a failure; a 429 is a pacing problem, not a bad address; a 403 with error 1010 is about the automation channel, and rotating the exit changes nothing. |
| Names the vendor, not just the status | DataDome, Kasada, PerimeterX, Akamai, Imperva, DDoS-Guard, Sucuri, AWS WAF and F5 are recognised from their own headers and cookies, and each refusal maps to what it really means. A CDN is named without being blamed: a CloudFront header is on every response it serves. |
| Escalates on evidence | Four tiers, ordered by real cost. The cheapest one whose reach covers the binding layer is chosen, so a site needing only a header profile never pays for a browser launch. |
| Treats identity as indivisible | A clearance is bound to the address, User-Agent and TLS fingerprint that earned it, and Clearance.usable_by() refuses to replay it under any other — which makes the classic rotating-proxy failure structurally impossible. |
| Solves once and reuses | A browser runs for the challenge, its exact User-Agent is adopted, and everything after is a cheap request on the same identity until the cookie expires. |
| Accumulates rather than fakes | Gamma-distributed pacing, homepage warm-up, real referrer chains, one address per origin, capped concurrency — persisted between runs and shareable across scrapers through SharedState, because a process that forgets cannot accumulate. |
| Avoids the trap with no error response | safe_links enumerates only anchors a person could click; TopicGuard notices content that stopped being about the site. |
| Signs requests, to be welcome | RFC 9421 / Ed25519 Web Bot Auth. A valid signature skips the challenge machinery entirely, making it the cheapest tier there is. |
| Takes your word for the rest | check_response faults a 200 whose body says otherwise — the difference lives in a schema only the caller knows, and the loop then treats that verdict as its own. |
| Tells you why | scraper.explain(url) names the binding layer, the working tier, the learned pacing and the ladder available. Exceptions carry .layer, not just a status code. |
Every one of these goes through the ladder, the pacing and the memory described above.
| Method | |
|---|---|
get post head ping submit_form |
a requests.Response |
get_soup post_soup make_soup |
a PageSoup — null-safe BeautifulSoup, selectors never return None |
get_json post_json |
parsed JSON |
render render_soup |
a browser renders the page. For HTML that is a shell JavaScript fills in — not a tier |
get_file get_image |
download to disk; get_image needs the [image] extra |
unchanged |
revalidate against the stored ETag and skip a whole job. Opt-in: a 304 has no body, so doing it underneath get_soup would return an empty page and no error |
links |
safe_links over a page — the anchors a person could click |
explain knows |
what has been learned about this origin |
abort close |
cancel everything in flight; release exits and browsers. Any call also takes signal= to cancel just itself |
cost tier reaches
──── ────────── ───────────────────────────────────────────────────
0 archive everything — when a capture exists
10 direct TLS, HTTP/2 frames, header order, post-quantum keyshare
100 clearance + the JavaScript, Turnstile and automation-channel layers
1000 managed + the per-zone composite, at someone else's price
The planner picks the cheapest rung that covers the binding layer, and stops with an explanation when no configured rung does. Writing your own rung: tiers.
pip install lncrawl-scraperPython 3.9 and up; CI tests and builds on every version in that range.
| Extra | Pulls in | Needed for |
|---|---|---|
lncrawl-scraper[cdp] |
websockets | either bundled solver — both speak over one WebSocket |
lncrawl-scraper[botauth] |
cryptography | signed requests (Web Bot Auth) |
lncrawl-scraper[image] |
Pillow | get_image() |
lncrawl-scraper[all] |
all three |
A solver drives a browser you already have and never downloads one. Finding it is the
library's job: find_firefox() and find_chromium() look inside macOS application bundles,
the Windows program directories, and distribution and flatpak paths.
Impersonation is not an extra. Layers 2–5 are one barrier and an ordinary Python client fails all four in the first round trip, so a build without it would not be a degraded scraper but one that cannot reach a protected page.
Two settings change what this library can do. The rest adjust how it does it.
from scraper import BidiSolver, ExitKind, ExitSpec, Scraper, ScraperConfig
config = ScraperConfig(
# The only thing that moves layer 1: reputation is not something a client emits.
# Declare the kind honestly — claiming MOBILE for a datacenter range only stops
# this library from telling you that layer 1 is why nothing works.
exits=[ExitSpec(url="http://user:pw@residential.test:8000", kind=ExitKind.RESIDENTIAL)],
# The only thing that reaches the challenge layers.
browser=BidiSolver(),
)
with Scraper(origin="https://site.test", config=config) as scraper:
scraper.get("https://site.test/deep/page")
print(scraper.explain("https://site.test/deep/page"))site.test
binding layer : L9 Managed JavaScript challenge — reads a hybrid property, solve
tier : clearance
pacing : 4.2s mean interval
requests : 48 ok / 3 failed
clearance : 712s left
ladder : direct(10) clearance(100)
exits : residential
Two solvers ship: BidiSolver drives Firefox over WebDriver BiDi, CdpSolver drives
Chrome over the DevTools protocol. They clear a comparable share of challenged hosts and
disagree on which ones, so neither dominates — prefer BidiSolver, because a clearance binds
to the fingerprint that earned it and firefox is the impersonation profile that reaches the
most hosts. A tor-pool is configured the same way, with TorPoolSpec.
Failures name the layer and what would move it, because "403 after 3 retries" is the message that sends people to rewrite the part that was already working.
from scraper import Layer
from scraper.exceptions import Exhausted, Impassable
try:
scraper.get(url)
except Impassable as exc:
# Layers 18 and 19 read a secret. Nothing to retry; the message names the route.
print(exc.detail)
except Exhausted as exc:
# A bypass may exist; this configuration does not reach it.
if exc.layer is Layer.IP_REPUTATION:
print(exc.detail)
# "no configured exit clears the reputation layer — datacenter and Tor ranges
# are published, so rotating between them cannot help."| Page | |
|---|---|
| The model | The bound, and emit vs. possess. Start here. |
| Layers | The nineteen layers, and which are diagnosable. |
| Tiers | The escalation ladder; writing a tier. |
| Configuration | Every ScraperConfig field. |
| Behaviour | Pacing, warm-up, persistence, shared state. |
| Decoy content | The layer that returns no error. |
| Web-bot-auth | Signed requests and the key directory. |
| Diagnostics | explain(), exceptions, common conclusions. |
| Migration | Porting from 0.2.x. |
| Examples | Ten runnable programs, ordered to explain the design. |
This library is for retrieving publicly accessible content. It does not attempt authentication bypass, credential abuse, or circumvention of access controls protecting non-public data — layer 19 raises rather than trying, and layer 18 raises where a signature is mandated. Where a site publishes an API or an archive holds what you need, both are cheaper than anything else here and are supported first-class for that reason.
uv sync # deps + editable install
uv run poe lint # ruff + pyright
uv run poe test # pytest
uv run poe cov # with coverage
uv run poe docs # serve the documentation siteTests are offline: the pipeline talks to a two-method Transport, so tests/conftest.py's
FakeTransport covers every tier without a network. The modules that encode judgement —
diagnosis, planner, layers — are pure functions over primitives and are tested as such.
AGENTS.md has the architecture and
the invariants that break silently.
Verifying against real deployments
livetest/ runs the same paths against real Cloudflare deployments, using every host in
lightnovel-crawler's source index as the
corpus. It is not part of poe test — it needs the network, and some scenarios need a local
tor-pool and a real browser. See
livetest/README.md.
uv run poe live-allThe recorded output is the live report —
scenario results, which layer each client meets across the corpus, and what a Tor exit
actually costs. It is a standalone page, regenerated in place at livetest/report.html and
published with the docs.
Nearly every defect found before 1.0 was invisible to a stubbed transport, and two of them made whole features silently useless while every unit test passed. Anything the harness finds gets a unit test whose docstring says it was found live, so those docstrings are the record of which assumptions turned out to be wrong.
Extracted from lightnovel-crawler. Apache-2.0; see LICENSE.