[codex] Harden place extractor fixtures and preview enrichment#48
Conversation
📝 WalkthroughWalkthroughThis PR extracts the browser-side place DOM extraction logic into a standalone ChangesPlace Extractor and Enrichment
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 29b58efe7d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
michaelmwu
force-pushed
the
michaelmwu/improve-maps-scraping
branch
from
29b58ef to
289a1e9
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tests/test_place_scraper.py (1)
230-234: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider replacing string-matching JS assertions with the new fixture-execution harness.
These tests assert on literal substrings of
_PLACE_JS_EXTRACTOR(e.g.'const isAddressIcon = (icon) => {','const rowValue = (row) => {'). They'll break on harmless variable renames and won't catch a logic regression that keeps the same substrings. SincePlaceJsExtractorFixtureTestsnow provides real browser execution against fixtures, a fixture-based assertion (e.g. an address-icon-only HTML fixture) would more directly encode the intended behavior.As per coding guidelines, "Tests should encode intent, not just exercise code paths; a test is weak if it would keep passing after the relevant scraper behavior or business rule is broken."
Also applies to: 849-855
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_place_scraper.py` around lines 230 - 234, The current test in test_place_scraper relies on literal substring checks against _PLACE_JS_EXTRACTOR, which is brittle and doesn’t validate scraper behavior. Replace these string-matching assertions in test_place_js_extractor_avoids_broad_photo_selectors (and the related row/address substring checks) with a fixture-driven browser execution test using PlaceJsExtractorFixtureTests so the test encodes the intended behavior instead of implementation text. Use a focused HTML fixture that exercises the photo-selector/address-icon behavior and assert the extracted result, not the JS source contents.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tests/test_place_scraper.py`:
- Around line 230-234: The current test in test_place_scraper relies on literal
substring checks against _PLACE_JS_EXTRACTOR, which is brittle and doesn’t
validate scraper behavior. Replace these string-matching assertions in
test_place_js_extractor_avoids_broad_photo_selectors (and the related
row/address substring checks) with a fixture-driven browser execution test using
PlaceJsExtractorFixtureTests so the test encodes the intended behavior instead
of implementation text. Use a focused HTML fixture that exercises the
photo-selector/address-icon behavior and assert the extracted result, not the JS
source contents.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: e98604a8-c3d5-4630-ab4d-f0bdd1666229
📒 Files selected for processing (10)
src/gmaps_scraper/data/place_extractor.jssrc/gmaps_scraper/place_scraper.pytests/fixtures/place_pages/about.htmltests/fixtures/place_pages/limited_view.htmltests/fixtures/place_pages/overview.htmltests/fixtures/place_pages/reviews.htmltests/fixtures/place_pages/search_result.htmltests/fixtures/preview_payloads/ambiguous_rating_review_count.txttests/fixtures/preview_payloads/rating_review_count.txttests/test_place_scraper.py
1 participant
Summary
_PLACE_JS_EXTRACTORJavaScript intosrc/gmaps_scraper/data/place_extractor.jsand load it viaimportlib.resourceswhile preserving the Python symbol.ratingandreview_count, with fixtures for positive and ambiguous payloads.Why
The JS extractor was mostly protected by source-string assertions, which can pass even when selector behavior drifts. The new fixture harness evaluates the extractor in a real local browser page and asserts the structured snapshot. Preview payloads can also contain rating and review-count signals on limited/thin pages, so the scraper now backfills those fields when DOM data is missing.
Notes
.context/, but.context/is gitignored workspace memory and is not part of this PR.Validation
./scripts/lint.sh./scripts/typecheck.sh./scripts/test.sh(267 tests)gmaps_scraper/data/place_extractor.jsis included./Users/michaelwu/dev/favorite-places/site/data/cache/places.sqlitewith--skip-reviews --skip-about --llm-policy never; all returnedquality_flags: [],confidence: 1.0, and populated ratings/review counts.Summary by CodeRabbit
New Features
Bug Fixes