feat(cli): add mex export (bundle scaffold to a single Markdown file) - #183
abhinav-phi wants to merge 3 commits into
Conversation
Concatenates every scaffold file the drift scanner discovers (DEFAULT_SCAFFOLD_PATTERNS through findScaffoldFiles) into a single Markdown document with a '## <path>' section header per source file, so what gets exported is exactly what mex check scans. Output goes to stdout by default, or to a path via --out (parent directories created). An empty scaffold fails with the setup guidance. Resolves mex-memory#56
src/export.ts writes one bundle file to a user-specified path — a brand-new file, never scaffold bytes — which is exactly the class the allowlist's own comment carves out. Registered by write call with its exemption, per the rule that a new writer names its scope.
|
CI caught this PR violating the wiki-architecture write pin: |
|
@theDakshJaitly @theyashasvipandey — done and green; requesting review. The CI run initially failed the wiki-architecture write-pin (this adds a |
|
hi this adds the export command that bundles the scaffold into a single markdown file the output is bounded and paths stay inside the scaffold happy to adjust the format if you want a different shape |
theDakshJaitly
left a comment
theDakshJaitly
left a comment
There was a problem hiding this comment.
The normal export works, but the destination needs protection against overwriting project state, and scaffold reads need explicit resource bounds. Both findings are inline.
Validation on current main with this PR applied: 59 existing tests, production build, workspace typecheck, and diff checks passed. Additional built-CLI probes reproduced the overwrite and resource-limit failures. Stdout and file output matched for the normal fixture.
| const target = resolve(config.projectRoot, opts.out); | ||
| mkdirSync(dirname(target), { recursive: true }); | ||
| writeFileSync(target, document, "utf-8"); |
There was a problem hiding this comment.
[P1] Refuse export destinations that overwrite project state
writeFileSync truncates any existing target, including the files being exported. In the built CLI, mex export --out .mex/ROUTER.md replaced the Router with the bundle, and --out .mex/config.json replaced configuration JSON with Markdown; both exited 0. An exports/scaffold.md symlink to the Router also overwrote it. This contradicts the new architecture exemption's claim that this writer creates a brand-new file and never writes scaffold bytes. Validate the destination and its aliases before writing, preserve existing scaffold/configuration state, and enforce the intended new-file behavior safely. Cover these rejected destinations with tests asserting that the original bytes survive. Also prevent output from becoming an input: repeated exports to .mex/context/bundle.md included the prior bundle and duplicated the scaffold.
There was a problem hiding this comment.
Addressed in 2abae42, with the destination validated before anything is written.
How it works now: --out is resolved (including through symlinks, via best-effort realpath that also resolves symlinked parent directories) and compared against every scaffold file, the project config (.mex/config.json, which discovery never lists, so it gets an explicit guard), and their realpath aliases. A collision throws before any write, so the original bytes always survive. An existing target carrying the bundle marker (# mex scaffold export) is treated as a previous output rather than project state: it is excluded from its own inputs and may be overwritten, which also fixes the repeat-export duplication (second run is now byte-identical to the first).
Tests in test/export.test.ts prove original bytes survive for all three reported cases: --out .mex/ROUTER.md, --out .mex/config.json, and a symlink alias. One environment note: the file-symlink fixture itself needs symlink privilege, so it errors with EPERM on my Windows box; the same alias-detection path is additionally covered by a directory-junction variant that runs everywhere, and the symlink case will run on Linux CI. The architecture test (pinned writers, guard table) still passes since no write call sites were added.
| for (const file of files) { | ||
| const relativePath = toPosix(relative(config.scaffoldRoot, file)); | ||
| bundle.push(`## ${relativePath}`, "", readFileSync(file, "utf-8").trimEnd(), ""); | ||
| } | ||
| const document = bundle.join("\n"); |
There was a problem hiding this comment.
[P2] Bound scaffold reads before retaining the full export
findScaffoldFiles provides discovery and deduplication, not file-count or byte limits. This loop reads every file into memory, retains the whole corpus, then allocates the joined document before either output mode can write anything. Large scaffold content can therefore terminate Node instead of producing a controlled error. In a deliberately constrained 64 MiB V8-heap diagnostic, the small fixture passed but two 40 MiB source files caused a fatal heap error during ReadFileUtf8; this is not a claim about Node's default heap threshold. MEX's bounded-input/output and retained-state rules require explicit file/count/aggregate limits checked before allocation, with a clear refusal and boundary tests. If larger exports are supported, use a bounded streaming design instead of retaining the full document.
There was a problem hiding this comment.
Addressed in 2abae42 with explicit limits checked before any content is retained: at most 1000 files, 1 MiB per file (via stat, never read first), and 8 MiB aggregate, each with a clear refusal error naming the limit.
Boundary coverage in test/export.test.ts: count 1000 passes while 1001 refuses (and writes nothing), per-file exactly 1 MiB passes while 1 MiB + 1 byte refuses, and aggregate exactly 8 MiB passes while 9 MiB refuses. Refusals are verified to leave no output file behind. I did not go the streaming route since the refusal path keeps the implementation (single joined document for both stdout and --out) intact; the limits sit far above any realistic hand-written scaffold while refusing far below heap-exhaustion territory.
P1: validate --out against scaffold files, the project config, and symlink aliases before writing; a previous bundle output (marker prefix) is excluded from its own inputs instead of refused. P2: enforce file-count, per-file, and aggregate byte limits before retaining content, with clear refusals and boundary tests.
2 participants
Resolves #56.
What
mex exportconcatenates the whole scaffold into one Markdown document, for pasting into tools that don't read files well:mex check— reusesfindScaffoldFiles+DEFAULT_SCAFFOLD_PATTERNS, so the export is exactly what the drift scanner sees, never a divergent file list.## <scaffold-relative-path>header, with content trimmed of trailing whitespace so the document stays clean.mex export→ stdout;mex export --out exports/scaffold.md→ file (parent directories created) plus a one-line count report.No scaffold files found. Run: mex setup.Tests
Three cases in
test/export.test.ts: full bundle (headers, content, deterministic order),--outwrite + count report, and missing-scaffold guidance.npm run typecheckgreen.