Add option to only aggregate documentation from selected package versions - #107
Merged
Conversation
kdayday
changed the title
kdayday
changed the title
kdayday
marked this pull request as ready for review
April 23, 2026 00:36
Contributor
Author
|
@pfitzseb Hello, is it possible to get a review on this PR? I believe the remaining Windows errors are the same as currently exist on |
pfitzseb
self-requested a review
May 5, 2026 08:12
…sing Pass indexhtml_path through to the HTMLDocument method so warning logs include the file path instead of referencing an undefined variable. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
Author
|
Hi @pfitzseb and @mortenpi, it is possible to get a review on this? We're ready to publish our Sienna documentation using it. https://github.com/Sienna-Platform Thank you! |
…ent asset - Revert src/search/pagefind.jl to main. The npx/npm changes did not fix the Windows failure they targeted (Windows CI fails identically on this branch and on main, with `could not spawn '…/bin/npx' pagefind -V`, because the JLL ships npx.cmd on Windows), and they traded a scoped `dir =` kwarg for a process-global `cd`. Worth doing separately, with a fix that addresses the actual cause. - Drop the giturl -> GitHub Pages URL derivation. It fabricated repo names for any repo not ending in `.jl` (`org/SciMLDocs` -> `sciml.github.io/SciMLDocs.jl/`) and silently produced nothing for clone URLs with a trailing slash. The "See All Versions" entry is now added only when `all_versions_url` is set explicitly, and only for absolute http(s) URLs. - Move the "See All Versions" entry into the rewritten versions.js instead of shipping assets/default/see_all_versions.js and injecting a JSON config block into every HTML page. Every Documenter page already loads ../versions.js, and we already rewrite it, so no new client side asset and no per-page rewrite pass are needed. Documenter's own selector handler navigates to the selected option's value, so pointing that value at the absolute URL is enough -- the sentinel option, the capture-phase change handler and the selection-restore logic are all gone. Also fixes the `];;` left in the rewritten DOC_VERSIONS declaration, and the duplicate "See All Versions" option that appeared when the selector was already populated server-side (the old dedupe only checked the last option). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The snippet appended to versions.js only existed to append an <option> after Documenter had populated the selector. It doesn't need to run on the client at all: Documenter's versions.js never clears the selector, matches DOC_VERSIONS against existing options by their text, and navigates to the selected option's value. So an <option> written into the HTML at build time survives untouched and does the same job. The option is added in the Gumbo pass that already rewrites every page for the global navigation, so this adds no extra walk over the output. versions.js is back to only rewriting DOC_VERSIONS, and the JSON dependency it needed for escaping is gone. The entry now sits above the versions Documenter fills in rather than below them, and follows the selector's own navigation (same tab) instead of opening a new one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Rename the "include_versions patch" testset to "include_versions": it is a feature on the branch, not a patch on top of a release. - Drop the MultiDocRef constructor testset. It asserted that two keyword arguments end up in two fields, which was worth checking while the constructor fed the giturl -> Pages URL derivation; with that gone it tests the language. Both fields are still exercised through real logic by the uses_include_versions and see_all_versions_url testsets. - inject_see_all_versions_option! no longer returns a Bool that only its tests read; the tests assert on the markup it produces instead. - see_all_versions_url_for returns the first matching ref instead of the longest one. Two refs whose paths nest would already copy over each other in make_output_structure, so at most one can match. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Test does not wrap a testset body in a function, so the `return` used to skip the symlink test on Windows exited the enclosing "include_versions" testset instead, silently skipping every testset below it -- and still reporting green. Confirmed by forcing the branch: 45 assertions ran on Linux, 9 with the Windows path taken. Now 41 run there, with the symlink case reported as skipped. Also corrects the claim in inject_see_all_versions_option!'s docstring that Documenter navigates to the selected option's value. It appends the current page's path below the version directory to that value, HEADs the result, and only falls back to the bare value when the probe fails -- so all_versions_url should name a site root, which the MultiDocRef docstring now says too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1. Validate include_versions against what exists upstream. cp_select_versions now returns the versions it actually copied, warning about the rest, and the redirect stub and DOC_VERSIONS are built from that instead of the request. A package missing e.g. `stable` no longer gets a 404 landing page, a 404 dropdown entry, and -- because fix_canonical_url! derives the canonical directory from that stub -- a rel="canonical" pointing at a directory that was never copied. If none of the requested versions exist, all versions are copied and the ref stays usable, with a warning naming it. 2. Skip version directories whose symlink target is gone instead of throwing out of make_output_structure. Pruning old version directories by hand leaves `stable -> v1.2.3` dangling on gh-pages, and `cp(...; follow_symlinks = true)` raises on those. Root level files are now copied with follow_symlinks = true too, so a symlinked root file does not become a symlink to nothing. 3. Repoint Documenter's outdated-version markers at versions that were copied. warner.js compares DOCUMENTER_NEWEST against each directory's DOCUMENTER_CURRENT_VERSION and, on a mismatch, injects <meta name="robots" content="noindex"> plus a banner linking to ../$DOCUMENTER_STABLE. Left alone, keeping an older release de-indexed it and pointed the banner at a directory that is no longer there. DOCUMENTER_NEWEST is rewritten to the newest kept release and DOCUMENTER_STABLE (in each copied siteinfo.js) to a kept directory. Both are left alone when nothing kept is a release, since warner.js bails out before reading them in that case. 4. Honour hide_previews = false: previews/ is not a version directory, so cp_select_versions never copied it and the existing hide_previews handling had nothing to remove. 5. Don't inject the aggregate front end into generated redirect stubs. The injector skipped pages carrying Documenter's marker; the regenerated stubs carry MultiDocumenter's, so they were Gumbo round-tripped -- destroying the marker, emitting an invalid `<!DOCTYPE >`, and loading multidoc + pagefind JS and all CSS before the meta refresh could fire. is_generated_redirect() now matches both markers, which also stops the aggregate root index.html from being rewritten this way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two MultiDocRef keywords only meant anything together: all_versions_url
without include_versions was silently ignored, and its validation lived in a
helper that ran only when include_versions was also set, so a typo'd URL was
dropped without a word. One struct makes the coupling a type-level fact:
MultiDocRef(
upstream = ...,
path = "Pkg",
name = "Pkg.jl",
versions = VersionSelection(
["stable", "dev"];
all_versions_url = "https://org.github.io/Pkg.jl/",
),
)
Validation now happens in the constructor, where it cannot be bypassed, and
throws instead of warning -- a bad URL is a configuration mistake worth failing
on before any cloning or copying happens. An empty version list is rejected for
the same reason, which also means the struct's existence is the whole predicate:
uses_include_versions() and see_all_versions_url() are gone, along with their
tests.
Both keywords are new on this branch and unreleased, so there is nothing to
deprecate. Positional MultiDocRef construction keeps its pre-branch arity via an
extra outer constructor.
test/include_versions.jl becomes test/version_selection.jl to match.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SearchConfig.index_versions is one global list, but each ref decides which versions it publishes, so a VersionSelection that keeps e.g. only ["v1.0.0"] leaves that package out of the search index entirely -- published, linked from the navigation, and unreachable through the search box, with nothing about the build looking wrong. warn_unindexed_refs() names each such ref, along with the indexed versions it was checked against and the versions it does publish, so the fix is obvious from the message. It runs from walk_outputs (flexsearch, stork) and from PageFind, which builds a site-wide glob instead of going through walk_outputs. Note this can also fire without any VersionSelection, for a package whose gh-pages branch happens to use neither `stable` nor `dev` -- that package was already missing from search, just silently. Deriving index_versions from the selection instead of only warning is the better end state, but it changes how the default search engine is invoked for every build, so it belongs in its own PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Member
|
Sorry for the late response. I've tweaked the implementation a bit, so note that the API isn't compatible anymore with the original proposal. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds optional control over which version directories are copied into each aggregated doc tree, and wires a “See All Versions” entry into Documenter’s version
<select>that opens the upstream package site (GitHub Pages by default, or a custom URL) in a new tab. This helps when aggregating many repos with longgh-pageshistories so the combined site stays closer to GitHub Pages limits (e.g. the 1 GB cap).New
MultiDocRefoptionsinclude_versions::Union{Vector{String}, Nothing}— When non-nothingand non-empty (e.g.["stable", "dev"]), only those top-level version entries are copied fromupstream. All root-level files (not onlyindex.html/versions.js) are copied;.gitis skipped. Version dirs are copied withfollow_symlinks = trueso symlinks such asstable → v1.0.0become real directories in the aggregate.all_versions_url::Union{String, Nothing}— Absolute http(s) URL used for “See All Versions” wheninclude_versionsis in use. If unset andgiturlis a github.com clone URL, the URL is derived (e.g.https://github.com/org/Pkg.jl.git→https://org.github.io/Pkg.jl/). Non-GitHub hosts are not auto-derived; setall_versions_urlexplicitly. If neither yields a usable URL, version limiting still runs, but the extra selector option is not injected.Behavior when
include_versionsis setAfter the selective copy, the aggregate root
index.htmlis regenerated (not taken from upstream) so it meta-refreshes tofirst(include_versions), avoiding stale upstream redirects (e.g. old org URLs).versions.jsis rewritten soDOC_VERSIONSlists only the kept versions.Each processed doc HTML gets a small JSON config
<script>(before</body>) consumed byassets/default/see_all_versions.js. That script appends the sentinel option, opens the target URL in a new tab on choose, and restores the dropdown. It also replaces a legacy inline “see all versions” script if present.see_all_versions.jsis registered as a global default script (so it ships on every build), but it only does work when the config block exists (i.e. when injection ran for that ref).When
include_versionsis unsetBehavior matches main: full copy of
upstream, noversions.jsrewrite, no “See All Versions” injection for this feature path.Other changes (same branch)
PageFind/ NodeJS JLL: Hardennpx/npminvocation (executable path resolution and working directory) for Windows and newer Julia where CI previously failed to spawnpagefind.UndefVarErrorwas thrown when a malformed redirect meta tag is encountered during canonical URL fixing or sitemap generation.