I build web scrapers, publish them on Apify, and keep them running when the sites change underneath them. Each one reads a site the way it serves itself to a logged-out visitor and returns one clean row per thing you asked for. When it can't read something, the row says so and says why, rather than handing you an empty file.
The repos here are working examples: the problem, a Node and a Python script, and real output from a real run. The write-ups are on simple-actor.github.io.
How to see which websites rank on Google for a list of keywords (10 espresso searches, 117 results): Who ranks on Google for 10 espresso keywords, read in one run for 9 cents: Reddit in the top three for 5. Six searches came back with 2 to 7 rows, and a re-read minutes later gave different sites. Code: google-keyword-rank-checker
All of them, with prices: apify.com/simple.actor