[PARTH-TUSSLE]

Part of #8330

• Disallow /hugo-preview/ in production root robots.txt generated by Gatsby on master

Summary

Updates gatsby-config.js to include "/hugo-preview/" in policy.disallow alongside "/pr-preview/".

Because previews are served from the root domain (https://layer5.io/hugo-preview/pr-<N>/), web crawlers check the root https://layer5.io/robots.txt served from the default branch. This one-line change ensures that production Gatsby builds immediately instruct search engines not to crawl ephemeral Hugo previews.

Diff

diff --git a/gatsby-config.js b/gatsby-config.js
index b29b2c03..e5d10422 100644
--- a/gatsby-config.js
+++ b/gatsby-config.js
@@ -602,7 +602,7 @@ module.exports = {
               host: siteRootUrl,
               sitemap: `${siteRootUrl}/sitemap-index.xml`,
               policy: [
-                { userAgent: "*", allow: "/", disallow: ["/pr-preview/"] },
+                { userAgent: "*", allow: "/", disallow: ["/pr-preview/", "/hugo-preview/"] },
               ],
             },
           },

Summary by CodeRabbit

• Updates
  • Crawling is now disallowed for the /pr-preview/ and /hugo-preview/ paths, while the site root remains allowed.

[coderabbitai]

No actionable comments were generated in the recent review.

Walkthrough

The robots.txt policy now disallows /hugo-preview/ for all user agents. It continues to disallow /pr-preview/ and allow /.

Changes: Robots.txt policy

Layer / File(s)	Summary
Add Hugo preview path rule gatsby-config.js	The all-user-agent policy adds /hugo-preview/ to the disallowed paths. It continues to disallow /pr-preview/ and allow /.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Change: Bug fix

Possibly related issues

• layer5io/layer5#8330 — The issue describes adding /hugo-preview/ to the robots.txt disallow list, which this change does.

Possibly related PRs

• layer5io/layer5#8327 — This PR deploys Hugo previews under /hugo-preview/, the path disallowed by this change.

Suggested reviewers: leecalcote, saurabhraghuvanshii

Merge Risk: Minimal · up to a4e38

The change instructs compliant crawlers not to crawl Hugo preview paths while preserving existing crawler policy. No actionable merge-blocking risk was identified.

Pre-merge checks: 5

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title names the main change: adding /hugo-preview/ to the robots.txt disallow rules. The raw summary confirms gatsby-config.js now disallows both /pr-preview/ and /hugo-preview/ for all user agent…
Docstring Coverage		No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Title check

Explanation

The title names the main change: adding /hugo-preview/ to the robots.txt disallow rules. The raw summary confirms gatsby-config.js now disallows both /pr-preview/ and /hugo-preview/ for all user agents, and the objectives describe the same single-line change. The title is specific, concise, and uses a conventional commit prefix, so it matches the changeset.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 1 files.

Finishing Touches

Generate docstrings

• Commit to this branch
• Create a new PR

Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Warning

Some tools did not complete. Review the errors below.

ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

gatsby-config.js

Parsing error: The keyword 'const' is reserved

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

Recent review info

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 376eb227-a02b-4b23-ab70-9b14e17f89b2

Commits

Reviewing files that changed from the base of the PR and between 5487c16 and a4e3809.

Files selected for processing: 1

• gatsby-config.js

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

[carlosriosilva]

Nice work - a clean one-line fix with a clear explanation of why it matters.

What stood out:

• Previews live under the root domain, so crawlers read the root robots.txt from the default branch - adding /hugo-preview/ next to /pr-preview/ closes that gap exactly where it needs closing.
• The change is minimal and the reasoning in the PR body is clear.

No changes needed from me - approving.

[hortison]

Approved. Checked:

• Full diff: one line in gatsby-config.js, adds "/hugo-preview/" to the robots.txt disallow list alongside "/pr-preview/".
• Linked issue #8330 asks for exactly this change and confirms Hugo previews are served from the production origin at layer5.io/hugo-preview/pr-/, where only the root robots.txt counts.
• gatsby-plugin-robots-txt is the plugin behind this config (referenced in gatsby-config.js and package.json); the /hugo-preview/ path is confirmed in .github/workflows/deploy-hugo-preview.yml.
• Checks on head a4e3809 are all green: build, CodeRabbit, triage, DCO success; Handle Copilot PR skipped. The CI build passing covers the config change.
• CodeRabbit review found no actionable comments; no prior reviews and no unresolved review threads.
• Change is minimal, matches the issue exactly, and keeps the site root allowed while excluding the ephemeral preview paths.

[github-actions]

Preview removed because the pull request was closed.