crawler: sitemap(), report(), keepQuery, seed normalization - #3
Merged
Conversation
…r context Seeds now go through the same URL normalization as crawled links, so `about/`, `/a#top` and `/posts?page=2` meet the link to the same page in one queue entry. Previously a seed carrying a query was fetched verbatim and written to a literal `posts?page=2/index.html`. `keepQuery` renders each query spelling as its own page (params sorted) but writes it only when the entry names a filename — a static host serves a path the same for every query, so there is nothing correct to write by default. RenderedPage gains `duration` (pacing excluded); PrerenderContext gains `skipped` and `files`; emitFile filenames resolve against outDir so `../` can keep a build artifact out of the deploy. Co-authored-by: Cursor <cursoragent@cursor.com>
A formatter over context.pages: every rendered page becomes a <url>, minus redirect stubs, non-HTML, query spellings and pages marked noindex (robots meta in the head, any attribute order, or X-Robots-Tag). Co-authored-by: Cursor <cursoragent@cursor.com>
JSON account of the run: pages with status, timing, output file, written flag and referrers; redirects; skipped pages with errors; files emitted by earlier integrations; totals. Co-authored-by: Cursor <cursoragent@cursor.com>
--sitemap <origin>, --sitemap-file, --report, --report-file, --keep-query. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three more small, high-leverage features for
prerender-crawlerahead of 0.2.0, plus a latent seed bug they surfaced. Folded into the pending 0.2.0 changeset rather than stacking a second minor.sitemap({ hostname })— every rendered page as a<url>, dynamic routes expanded; excludes redirect stubs, non-HTML, query spellings, andnoindexpages (<meta name="robots">any attribute order,X-Robots-Tag).filename,trailingSlash,filter,entryoptions. CLI--sitemap <origin>.report()— JSON account of the run: per-page status / duration / output file / written / referrers, redirects, skipped pages with errors, files from earlier integrations, totals.filename: "../…"keeps it out of the deploy. CLI--report,--report-file.keepQuery— render/posts?page=2apart from/posts(params sorted), follow its links and capture its data, but write it only when the entry names afilename— a static host serves a path the same for every query. CLI--keep-query.about/,/a#topor/posts?page=2now dedupes with the crawled link. Previously a seed with a query was fetched verbatim and written to a literalposts?page=2/index.html.RenderedPage.duration(pacing excluded);PrerenderContext.skipped/.files;emitFilefilenames resolve againstoutDir.Verified: 62 crawler + 19 solid tests, six concurrent full suite runs under load, example app build, and the CLI against the built example (
--sitemap --report-file ../r.json).Made with Cursor