Add py-render custom code block formater

[waylan]

Description

This is similar to the previous markdown renderer (#1635) except that it runs Python code blocks in an isolated environment and renders the output as a result block. The Contributing Guide has been updated with an explanation of the feature.

AI Assistance Disclosure

• No AI tools were used in preparing this PR.
• If AI tools were used, I have disclosed which ones, and fully reviewed and verified their output.

Checklist

• This PR follows the contribution guidelines.
• The code follows the Code Style Guide.
• The commit message follows the Commit Message Style Guide.
• I have added or updated relevant docs, including release notes if applicable which follow the Documentation Style Guide.
• I have added or updated relevant tests.
• I have not requested, and will not request, an automated AI review for this PR.

[waylan]

[mitya57]

Are the backticks intended in Value of `fragment`, or it should be Value of <code>fragment</code> instead?

[waylan]

Good question. I had originally tried Value of <code>fragment</code> but the HTML gets escaped. I then tried the backticks and left them in as a compromise. I intended to file an issue with Superfences, but forgot about it. Thanks for the reminder.

[facelessuser]

I'm not sure I'm following. What is the issue?

[waylan]

Raw HTML in the title of a code block is escaped and Markdown rendering does not happen. Therefore, there is no way to include any markup in a title. For a simple example consider this plain superfence code block.

``` text { title='Content of <code>somefile.txt</code>' }
Text goes here...
```

It occurs to me that the above might not work as the raw HTML might trip up the parser. However, when using a custom formatter, I want to assemble a title which includes markup and can't because it is escaped.

[facelessuser]

Yeah, I think that was because when I originally put SuperFences together, the focus was getting fenced code to render in lists and other indented objects. Everything is run in a preprocessor and the stashed. It essentially becomes invisible to the parser except for the placeholder and unpacked at the end. We waited until the end as we don't have full context of whether the fenced code we parsed was under raw HTML or not.

I haven't looked into it, but I guess it could be possible to unpack them all in the treeprocessor, as we should be past the HTML processor at that point. We should have context of markdown=true and such to tell whether we are under an allowable HTML tag. But that means we need to consider changing how SuperFences stores content and give it more intelligence regarding the HTML tree than it currently has now.

Right now, we store fence HTML as strings, so we'd likely need to have them store content as etree Elements, or at least optionally for those who are okay with stuff in them getting processed.

So yeah, it was intentional, with the idea that code is handled and not seen until the end, unless you specifically go in and unpack them, but it is possible we could allow fenced code to be stored as etree Elements and unpack them before the inline treeprocessor, assuming no adverse effects.

[facelessuser]

It's probably fine to create a feature request, but I'll have to dig in and see how complicated it would be to pull off allowing that to get processed normally. I imagine people want access to all the normal inline stuff, so we'd need to ensure it gets inserted back into the tree at the proper time. I make no promises until I have a chance to dig into this as I haven't refactored this area in quite some time.

[waylan]

Turns out the issue is with Pygments, not Superfences. Supfences passes the title to the filename argument of HtmlFormatter and that value is HTML-escaped in the HTML returned by Pygments.

>>> from pygments import highlight
>>> from pygments.lexers import PythonLexer
>>> from pygments.formatters import HtmlFormatter
>>> print(highlight('# Code goes here', PythonLexer(), HtmlFormatter(filename='Content of <code>file.py</code>')))
<div class="highlight"><span class="filename">Content of &lt;code&gt;file.py&lt;/code&gt;</span><pre><span></span><span class="c1"># Code goes here</span>
</pre></div>

The only way to work around that would be to use a placeholder which could be swapped out later.

[facelessuser]

Yeah, there is some complexity in this area, so it's not something I'm committed to providing until I have time to investigate and determine all the potential issues.

It's likely possible to get something working; I'm just not 100% sure it's possible without side effects, but a thorough investigation will help make that more clear.

[waylan]

Turns out it was a simple fix for me to address in my custom formatter (see #1640). I just passed the title to md.htmlStash.store. I didn't even need to do the reverse swap. It was already taken care of by the normal handling of Pygment's output.

[facelessuser]

Apparently, I misunderstood what we were talking about. I thought it was desired that the inline code with ` would get parsed as code. I see now we just wanted to preserve raw HTML.