fix: protect fenced code blocks and expand MDX includes for agent docs#16
Open
scottamain wants to merge 3 commits into
Open
fix: protect fenced code blocks and expand MDX includes for agent docs#16scottamain wants to merge 3 commits into
scottamain wants to merge 3 commits into
Conversation
The step-9 cleanup regex (/<[A-Z][a-zA-Z]*.../) removes MDX components from raw .md output. Inside bash heredocs, `<<EOF` is misread as a JSX tag named EOF because the pattern only requires an uppercase letter after `<`. On docs.modular.com this corrupted max/inference/image-generation.md: the stripper deleted from `cat <<EOF` through `<InstallOpenAI />`, leaving an unclosed ```bash fence and failing the markdown-code-fence-validity AFD check. Fix: tokenize fenced code blocks before step 9 and restore them afterward.
Self-closing tags like <InstallModular folder="..." /> were stripped entirely from .md output, while the HTML page renders the full install instructions. The afdocs markdown-content-parity check compares Turndown(HTML) against the .md endpoint, so agents saw missing setup steps. Expand _includes/install-modular.mdx and install-openai.mdx inline at build time, converting nested CodeBlock, ConditionalVersionDocs, Admonition, and Tabs markup. Also convert <figure> blocks with require()-based images to markdown with captions. Separate from the fenced-code-block protection fix — that addresses a corruption bug; this improves parity between HTML and .md content. Co-authored-by: Cursor <cursoragent@cursor.com>
…d tests Three improvements to the code-fence protection PR: 1. Restore InstallModular to the unwrapMdxComponents list so that non-self-closing <InstallModular>...</InstallModular> usage (if any site uses it as a wrapper) preserves inner content rather than having it silently deleted by the JSX stripper. The converter runs first and handles self-closing tags; unwrap acts as a safety net for the rest. 2. Cache include template reads in pluginContext. Previously convertInstallModularToMarkdown and convertInstallOpenAIToMarkdown called fs.existsSync + fs.readFileSync on every file processed. Now the templates are read once in postBuild and passed through pluginContext.includeTemplates. 3. Add test coverage (32 tests) for the new functions: fenced block protection round-tripping, the JSX stripper heredoc regression, CodeBlock/Admonition/ConditionalVersionDocs/Figure converters, InstallModular/InstallOpenAI expansion with and without pluginContext, and cleanMarkdownForDisplay compatibility without pluginContext. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two things in here:
Self-closing tags like
<InstallModular folder="..." />were stripped entirelyfrom .md output, while the HTML page renders the full install instructions.
The afdocs markdown-content-parity check compares Turndown(HTML) against the
.md endpoint, so agents saw missing setup steps.
This expands them, also converts nested CodeBlock, ConditionalVersionDocs, Admonition, and
blocks with require()-based images toTabs markup. Also convert
markdown with captions.
This improves parity between HTML and .md content.
Cleanup regex
(/<[A-Z][a-zA-Z]*.../)removes MDX components fromraw .md output. Inside bash code blocks,
<<EOFwas misread as a JSX tag namedEOF because the pattern only requires an uppercase letter after
<.On docs.modular.com this corrupted
max/inference/image-generation.md: thestripper deleted from
cat <<EOFthrough<InstallOpenAI />, leaving anunclosed ```bash fence and failing the markdown-code-fence-validity AFD check.