Ai manpage - #4899
Ai manpage#4899bdbaddog wants to merge 6 commits into
Conversation
The scons(1) man page is too large for a language model to read in full: the roff scons.1 is roughly 230k tokens and the generated HTML about 337k. Add doc/man/scons-ai.md, a condensed Markdown reference of roughly 113k tokens - about half the roff man page - which keeps every documented entry (each option, tool, builder, function, construction variable, configure context method and node attribute, with its description) but drops markup, cross-reference chrome, code examples, Note blocks and version history. The generator, bin/scons-man-to-ai.py, reads the checked-in DocBook source doc/man/scons.xml rather than built documentation, so it needs no doc build and works from a clean checkout. It resolves the parameter entities and XIncludes that supply the bulk of the content, which requires lxml (already a development dependency). The generated man page HTML is also accepted via --input for comparison against the documentation toolchain; both inputs produce the same set of entries. Wire it into doc/SConscript as the 'ai-manpage' alias, also reachable from the 'doc' alias. Because it builds from checked-in sources it is generated even when SKIP_DOC is set and the rest of the doc build is skipped. Also unwrap two <variablelist> elements in the Decider documentation that were nested inside a <para>. That nesting is schema-valid but is a 2-of-214 anomaly, and the DocBook XSL hoists the list out of the paragraph, leaving four empty <p></p> elements in the rendered HTML man page. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WZmNvpfrD52gbqknUecffc
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WZmNvpfrD52gbqknUecffc
Install the condensed AI manpage alongside the other rendered documentation formats, matching the HTML/PDF/EPUB/TEXT layout, and include it in the documentation tarball. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WZmNvpfrD52gbqknUecffc
|
Guess it's okay. Since the result isn't human-friendly, nor human-intended, it's hard to say much about it. How would this be provided? you planning to put it in doc/man even though it's "generated"? |
not sure. the .gen files are under doc/generated.. if we put it there and then referenced it in our agents.md maybe? |
I'd say that makes sense. |
Build the condensed AI manpage directly to build/doc/MD/scons-ai.md rather than writing it into doc/man and installing a copy. It is generated output, so it does not belong in the source tree; remove the checked-in copy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WZmNvpfrD52gbqknUecffc
Render scons-ai.md into doc/generated/scons-ai.md, matching the checked-in doc/generated/*.gen and *.mod files, instead of writing directly into the build tree. The build now installs a copy into build/doc/MD alongside the other rendered formats. Mark the generated file Precious/NoClean so 'scons -c' does not delete the checked-in copy. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Xrv6wmE3cfEhYmsTiZpnsZ
|
Have let this soak for a while, and I'm still not sure it's needed... the current agents are pretty good at picking out relevant snippets from a big file without trying to consume the whole thing. UpToYou, of course. |
|
From what I understand they're very good at markdown, and it's simpler to parse, but might be even better if we broke it up into sections as separate files and just had a top level TOC with links. The key is keeping the context window size down.. I don't see any way it would hurt.. I should add a pointer to this from the AGENTS.md as well and then merge it. |
2 participants
Added bin/scons-man-to-ai.py which translates the xml manpage into a minimized markdown file suitable for AI to use. (reduces from 230k (scons.1) or 337k (scons-man.html) to 113k tokens with scons-ai.md
Should this be built and checked into doc/man/scons-ai.md, or only built into build/doc/MD/scons-ai.md?
In the process of this work, found that there were a few variablelist's inside para's where there was nothing else in the para's so simplified those, to also simplify the markdown generation script
(Built with Claude Opus 5)
Contributor Checklist:
CHANGES.txtandRELEASE.txt(and read theREADME.rst).