diff --git a/README.md b/README.md index 0a58a59..6292fd5 100644 --- a/README.md +++ b/README.md @@ -305,7 +305,7 @@ Paragraph with **bold**, *italic*, `code`, [links](https://example.com), and foo ) ``` -Supported Markdown includes headings, paragraphs, bold, italic, inline code, strikethrough, links, images, linked images, image captions, code blocks, blockquotes, ordered lists, unordered lists, horizontal rules, and footnotes. +Supported Markdown includes headings, paragraphs, bold, italic, inline code, strikethrough, superscript, subscript, links, images, linked images, image captions, code blocks, blockquotes, ordered lists, unordered lists, horizontal rules, footnotes, LaTeX math, pull quotes, and callouts. See [docs/markdown.md](docs/markdown.md) for the full reference with examples. When an `Api` instance is passed to `from_markdown`, local image paths are uploaded before the draft is created: diff --git a/docs/markdown.md b/docs/markdown.md new file mode 100644 index 0000000..f58f618 --- /dev/null +++ b/docs/markdown.md @@ -0,0 +1,166 @@ +# Markdown support + +`Post.from_markdown(markdown_content, api=None)` converts a Markdown document +into Substack's document format. Parsing is handled by +[markdown-it-py](https://github.com/executablebooks/markdown-it-py) (CommonMark) +plus a few plugins, so standard CommonMark works as you'd expect. This page +documents everything that maps to a Substack node. + +```python +from substack.post import Post + +post = Post(title="My Post", subtitle="", user_id=api.get_user_id()) +post.from_markdown(open("post.md").read(), api=api) +draft = api.post_draft(post.get_draft()) +``` + +Pass `api=` when your Markdown references local images so they can be uploaded +(see [Images](#images)); it is optional otherwise. + +## Text formatting + +| Markdown | Result | +|-------------------------------|-------------------| +| `**bold**` | **bold** | +| `*italic*` | *italic* | +| `***bold italic***` | ***bold italic*** | +| `` `inline code` `` | `inline code` | +| `~~strikethrough~~` | ~~strikethrough~~ | +| `^superscript^` | superscript | +| `~subscript~` | subscript | +| `[text](https://example.com)` | link | +| `` | autolinked URL | + +Note that subscript uses a single tilde (`~x~`) and strikethrough uses a double +tilde (`~~x~~`); both work in the same document. + +## Headings + +Levels 1–6, using `#` through `######`. Headings may contain inline formatting +and links. + +```markdown +# Heading level 1 +## Heading level 2 with **bold** and a [link](https://example.com) +``` + +## Paragraphs and line breaks + +Blank lines separate paragraphs. A single newline within a paragraph is treated +as a space (soft break), matching CommonMark. + +## Lists + +Bullet lists (`-`, `*`, or `+`) and ordered lists (`1.`), including nesting: + +```markdown +- Bullet one +- Bullet two + - Nested bullet + 1. Nested number + +1. Ordered one +2. Ordered two +``` + +## Blockquotes + +```markdown +> A blockquote. +> +> With multiple paragraphs. +``` + +## Code blocks + +Fenced code blocks, with an optional language for syntax highlighting, and +indented code blocks: + +````markdown +```python +print("hello") +``` +```` + +## Horizontal rule + +```markdown +--- +``` + +## Images + +A paragraph containing only an image becomes a captioned image. + +```markdown +![Alt text](https://example.com/image.png) +``` + +- **Caption:** use the CommonMark title slot — `![alt](url "caption text")`. +- **Link:** wrap the image in a link — `[![alt](url)](https://target.com)`. +- **Local upload:** if `api=` is passed and the `src` is a local path (not an + `http(s)` URL), the file is uploaded to Substack and the returned URL is used. + Local paths resolve relative to the current working directory. + +```markdown +![A chart](chart.png "Figure 1: quarterly results") +``` + +## Footnotes + +References become inline anchors; definitions become footnote blocks at the end, +numbered by order of first appearance. Labels may be numeric or named, and a +definition may contain block content such as lists or multiple paragraphs. + +```markdown +A claim that needs support.[^1] Another, with a named label.[^source] + +[^1]: The supporting detail, with a [link](https://example.com). +[^source]: Author, *Title* (2025). +``` + +A reference used more than once is emitted as a separate numbered anchor each +time, mirroring the Substack editor. Definitions that are never referenced are +dropped. + +## Math (LaTeX) + +Inline math with single dollars, block math with double dollars: + +```markdown +Einstein showed $E=mc^2$ inline. + +$$ +\int_0^\infty e^{-x} \, dx = 1 +$$ +``` + +Delimiters follow Pandoc's rules: the opening `$` must not be followed by +whitespace, and the closing `$` must not be preceded by whitespace or followed +by a digit. Ordinary dollar amounts (`$5 million to $10 million`) therefore +stay plain text. A label after a block (`$$ ... $$ (label)`) is accepted but +discarded, since Substack has no equation labels. + +## Pull quotes and callouts + +These use fenced-container syntax (`:::`), since they have no native Markdown +equivalent: + +```markdown +:::pullquote +A highlighted pull quote. **Formatting** works inside. +::: + +:::callout +A callout block, e.g. an aside or note. +::: +``` + +## Not supported + +- **Tables** — Substack has no table renderer or editor UI for them, so GFM + table syntax is not converted. To include tabular data, embed a chart (for + example via [Datawrapper](https://support.substack.com/hc/en-us/articles/15722290158100)) + and add it in the Substack editor. +- Widgets authored only in the Substack editor (buttons, polls, embeds, etc.) + have no Markdown equivalent. diff --git a/poetry.lock b/poetry.lock index c4e4949..626e87e 100644 --- a/poetry.lock +++ b/poetry.lock @@ -1061,23 +1061,23 @@ ws = ["websockets (>=15.0.1)"] [[package]] name = "mdit-py-plugins" -version = "0.4.2" +version = "0.6.1" description = "Collection of plugins for markdown-it-py" optional = false -python-versions = ">=3.8" +python-versions = ">=3.10" groups = ["main"] files = [ - {file = "mdit_py_plugins-0.4.2-py3-none-any.whl", hash = "sha256:0c673c3f889399a33b95e88d2f0d111b4447bdfea7f237dab2d488f459835636"}, - {file = "mdit_py_plugins-0.4.2.tar.gz", hash = "sha256:5f2cd1fdb606ddf152d37ec30e46101a60512bc0e5fa1a7002c36647b09e26b5"}, + {file = "mdit_py_plugins-0.6.1-py3-none-any.whl", hash = "sha256:214c82fb2ac524472ab6a5bcab1de80f73b50443e187f401bfd77efbc7c6481d"}, + {file = "mdit_py_plugins-0.6.1.tar.gz", hash = "sha256:a2bca0f039f39dbd35fb74ae1b5f998608c437463371f0ff7f49a19a17a114d0"}, ] [package.dependencies] -markdown-it-py = ">=1.0.0,<4.0.0" +markdown-it-py = ">=2.0.0,<5.0.0" [package.extras] code-style = ["pre-commit"] rtd = ["myst-parser", "sphinx-book-theme"] -testing = ["coverage", "pytest", "pytest-cov", "pytest-regressions"] +testing = ["coverage", "pytest", "pytest-cov", "pytest-regressions", "pytest-timeout"] [[package]] name = "mdurl" @@ -2290,4 +2290,4 @@ mcp = ["fastmcp"] [metadata] lock-version = "2.1" python-versions = "<4.0,>=3.10" -content-hash = "dc6eed4bd86907d6daf1c8f82f59d1f02a34ce4f08d09b40d2943a82bdd0d3ae" +content-hash = "5060b1c1d084c9538b2aca2084081085c532b022ec97cbb1582dfc6b193a9e22" diff --git a/pyproject.toml b/pyproject.toml index ce667a6..7a5a240 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -39,7 +39,7 @@ requests = "^2.32.0" python-dotenv = "^1.2.1" PyYAML = "^6.0" markdown-it-py = "^3.0" -mdit-py-plugins = "^0.4" +mdit-py-plugins = ">=0.5,<0.7" fastmcp = { version = "^3.1.1", optional = true } [tool.poetry.extras] diff --git a/substack/mdrender.py b/substack/mdrender.py index e04b97e..5926f9c 100644 --- a/substack/mdrender.py +++ b/substack/mdrender.py @@ -22,7 +22,11 @@ from markdown_it import MarkdownIt from markdown_it.tree import SyntaxTreeNode +from mdit_py_plugins.container import container_plugin +from mdit_py_plugins.dollarmath import dollarmath_plugin from mdit_py_plugins.footnote import footnote_plugin +from mdit_py_plugins.subscript import sub_plugin +from mdit_py_plugins.superscript import superscript_plugin from substack import nodes from substack.nodes import MarkType, NodeType @@ -31,11 +35,25 @@ "strong": {"type": MarkType.STRONG}, "em": {"type": MarkType.EM}, "s": {"type": MarkType.STRIKETHROUGH}, + "sup": {"type": MarkType.SUPERSCRIPT}, + "sub": {"type": MarkType.SUBSCRIPT}, } def _make_parser() -> MarkdownIt: - return MarkdownIt("commonmark").use(footnote_plugin).enable("strikethrough") + return ( + MarkdownIt("commonmark") + .use(footnote_plugin) + # Pandoc-style delimiters: no whitespace just inside the dollars and no + # digit just outside them, so paired currency amounts ("$5 ... $10") + # stay plain text instead of becoming math. + .use(dollarmath_plugin, allow_space=False, allow_digits=False) + .use(sub_plugin) + .use(superscript_plugin) + .use(container_plugin, name="pullquote") + .use(container_plugin, name="callout") + .enable("strikethrough") + ) def _coalesce(out_nodes: List[Dict]) -> List[Dict]: @@ -64,6 +82,8 @@ def _render_inline(node: SyntaxTreeNode, marks: List[Dict], ctx: Dict) -> List[D out.append(nodes.text(child.content, marks)) elif t == "code_inline": out.append(nodes.text(child.content, marks + [nodes.code_mark()])) + elif t == "math_inline": + out.append(nodes.latex_inline(child.content.strip())) elif t in _MARK_FOR: out.extend(_render_inline(child, marks + [_MARK_FOR[t]], ctx)) elif t == "link": @@ -159,10 +179,28 @@ def _render_block(node: SyntaxTreeNode, api, ctx: Dict) -> List[Dict]: if t == "ordered_list": return [nodes.ordered_list(_render_list_items(node, api, ctx))] + # "$$...$$ (label)" tokenizes as math_block_label; Substack has no equation + # labels, so it renders like an unlabeled block. + if t in ("math_block", "math_block_label"): + return [nodes.latex_block(node.content.strip())] + + if t == "container_pullquote": + return [nodes.pullquote(_render_container_body(node, api, ctx))] + + if t == "container_callout": + return [nodes.callout_block(_render_container_body(node, api, ctx))] + # footnote_block is handled separately in markdown_to_doc; ignore it here. return [] +def _render_container_body(node: SyntaxTreeNode, api, ctx: Dict) -> List[Dict]: + body: List[Dict] = [] + for child in node.children: + body.extend(_render_block(child, api, ctx)) + return body + + def _render_list_items(list_node: SyntaxTreeNode, api, ctx: Dict) -> List[Dict]: items = [] for li in list_node.children: diff --git a/substack/nodes.py b/substack/nodes.py index c5571ba..1d361e3 100644 --- a/substack/nodes.py +++ b/substack/nodes.py @@ -33,6 +33,10 @@ class NodeType: FOOTNOTE_ANCHOR = "footnoteAnchor" CAPTIONED_IMAGE = "captionedImage" CAPTION = "caption" + LATEX_BLOCK = "latex_block" + LATEX_INLINE = "latex" + PULLQUOTE = "pullquote" + CALLOUT_BLOCK = "calloutBlock" class MarkType: @@ -40,6 +44,8 @@ class MarkType: EM = "em" CODE = "code" STRIKETHROUGH = "strikethrough" + SUPERSCRIPT = "superscript" + SUBSCRIPT = "subscript" LINK = "link" @@ -146,3 +152,32 @@ def footnote(number: int, paragraphs: List[Dict]) -> Dict: "attrs": {"number": number}, "content": paragraphs or [paragraph()], } + + +def latex_block(expression: str) -> Dict: + return { + "type": NodeType.LATEX_BLOCK, + "attrs": {"persistentExpression": expression, "dirty": True}, + } + + +def latex_inline(expression: str) -> Dict: + return { + "type": NodeType.LATEX_INLINE, + "attrs": {"expression": expression, "persistentExpression": expression}, + } + + +def pullquote(paragraphs: List[Dict]) -> Dict: + return { + "type": NodeType.PULLQUOTE, + "attrs": {"align": None, "color": None}, + "content": paragraphs or [paragraph()], + } + + +def callout_block(paragraphs: List[Dict]) -> Dict: + return { + "type": NodeType.CALLOUT_BLOCK, + "content": paragraphs or [paragraph()], + } diff --git a/tests/substack/fixtures/full_features.expected.json b/tests/substack/fixtures/full_features.expected.json index ad7e75c..ecd9259 100644 --- a/tests/substack/fixtures/full_features.expected.json +++ b/tests/substack/fixtures/full_features.expected.json @@ -548,6 +548,120 @@ { "type": "horizontal_rule" }, + { + "type": "paragraph", + "content": [ + { + "type": "text", + "text": "A paragraph with inline math " + }, + { + "type": "latex", + "attrs": { + "expression": "E=mc^2", + "persistentExpression": "E=mc^2" + } + }, + { + "type": "text", + "text": ", H" + }, + { + "type": "text", + "text": "2", + "marks": [ + { + "type": "subscript" + } + ] + }, + { + "type": "text", + "text": "O subscript, and x" + }, + { + "type": "text", + "text": "2", + "marks": [ + { + "type": "superscript" + } + ] + }, + { + "type": "text", + "text": " superscript." + } + ] + }, + { + "type": "latex_block", + "attrs": { + "persistentExpression": "E=mc^2", + "dirty": true + } + }, + { + "type": "pullquote", + "attrs": { + "align": null, + "color": null + }, + "content": [ + { + "type": "paragraph", + "content": [ + { + "type": "text", + "text": "A pull quote with " + }, + { + "type": "text", + "text": "bold", + "marks": [ + { + "type": "strong" + } + ] + }, + { + "type": "text", + "text": " text." + } + ] + } + ] + }, + { + "type": "calloutBlock", + "content": [ + { + "type": "paragraph", + "content": [ + { + "type": "text", + "text": "A callout block with a " + }, + { + "type": "text", + "text": "link", + "marks": [ + { + "type": "link", + "attrs": { + "href": "https://example.com" + } + } + ] + }, + { + "type": "text", + "text": "." + } + ] + } + ] + }, { "type": "captionedImage", "content": [ diff --git a/tests/substack/fixtures/full_features.md b/tests/substack/fixtures/full_features.md index f2eca43..3cd028c 100644 --- a/tests/substack/fixtures/full_features.md +++ b/tests/substack/fixtures/full_features.md @@ -55,6 +55,20 @@ plain code block without a language --- +A paragraph with inline math $E=mc^2$, H~2~O subscript, and x^2^ superscript. + +$$ +E=mc^2 +$$ + +:::pullquote +A pull quote with **bold** text. +::: + +:::callout +A callout block with a [link](https://example.com). +::: + ![A captioned image](https://upload.wikimedia.org/wikipedia/commons/4/47/PNG_transparency_demonstration_1.png) [![Linked image alt](https://upload.wikimedia.org/wikipedia/commons/4/47/PNG_transparency_demonstration_1.png)](https://example.com/target) diff --git a/tests/substack/test_from_markdown_features.py b/tests/substack/test_from_markdown_features.py index 5a56323..fb0ac9c 100644 --- a/tests/substack/test_from_markdown_features.py +++ b/tests/substack/test_from_markdown_features.py @@ -174,3 +174,84 @@ def test_linked_image(self): attrs = block["content"][0]["attrs"] assert attrs["src"] == "https://i/x.png" assert attrs["href"] == "https://link" + + def test_latex_block(self): + post = make_post() + post.from_markdown("$$\nE=mc^2\n$$\n") + block = body(post)[0] + assert block["type"] == "latex_block" + assert block["attrs"]["persistentExpression"] == "E=mc^2" + assert block["attrs"]["dirty"] is True + + def test_labeled_latex_block_is_preserved(self): + post = make_post() + post.from_markdown("$$\nE=mc^2\n$$ (mass-energy)\n\nAfter the equation.") + blocks = body(post) + assert [block["type"] for block in blocks] == ["latex_block", "paragraph"] + assert blocks[0]["attrs"]["persistentExpression"] == "E=mc^2" + + def test_latex_inline(self): + post = make_post() + post.from_markdown("A paragraph with $E=mc^2$ inline.") + para = body(post)[0] + assert para["type"] == "paragraph" + latex = [n for n in para["content"] if n["type"] == "latex"] + assert len(latex) == 1 + assert latex[0]["attrs"]["expression"] == "E=mc^2" + assert latex[0]["attrs"]["persistentExpression"] == "E=mc^2" + + def test_currency_dollar_signs_are_not_parsed_as_latex(self): + post = make_post() + markdown = "Revenue grew from $5 million to $10 million this year." + post.from_markdown(markdown) + paragraph = body(post)[0] + assert [node["type"] for node in paragraph["content"]] == ["text"] + assert paragraph["content"][0]["text"] == markdown + + def test_superscript(self): + post = make_post() + post.from_markdown("E=mc^2^ here.") + para = body(post)[0] + sup = [n for n in para["content"] if n.get("marks")] + assert sup[0]["text"] == "2" + assert sup[0]["marks"] == [{"type": "superscript"}] + + def test_subscript(self): + post = make_post() + post.from_markdown("H~2~O here.") + para = body(post)[0] + sub = [n for n in para["content"] if n.get("marks")] + assert sub[0]["text"] == "2" + assert sub[0]["marks"] == [{"type": "subscript"}] + + def test_subscript_does_not_clash_with_strikethrough(self): + post = make_post() + post.from_markdown("~~struck~~ and H~2~O") + para = body(post)[0] + marks = { + m["type"] + for n in para["content"] + for m in n.get("marks", []) + } + assert marks == {"strikethrough", "subscript"} + + def test_pullquote(self): + post = make_post() + post.from_markdown(":::pullquote\nA **bold** quote.\n:::\n") + block = body(post)[0] + assert block["type"] == "pullquote" + assert block["attrs"] == {"align": None, "color": None} + para = block["content"][0] + assert para["type"] == "paragraph" + assert para["content"][1]["text"] == "bold" + assert para["content"][1]["marks"] == [{"type": "strong"}] + + def test_callout(self): + post = make_post() + post.from_markdown(":::callout\nA callout with a [link](https://example.com).\n:::\n") + block = body(post)[0] + assert block["type"] == "calloutBlock" + para = block["content"][0] + assert para["type"] == "paragraph" + link = [n for n in para["content"] if n.get("marks")][0] + assert link["marks"][0]["attrs"]["href"] == "https://example.com"