[mypyc] Design discussion: build-system-neutral C generation CLI (mypyc cgen) for projects not using setuptools

[trim21]

Motivation

mypycify() is currently the only supported entry point for compiling Python to C with mypyc, and it is tightly coupled to setuptools/distutils: it returns setuptools.Extension objects, patches build_ext, and requires setuptools to be importable (on Python 3.12+ distutils itself only exists through setuptools).

Projects that build their C extensions with meson, CMake, Bazel, scikit-build-core, or plain Makefiles cannot use mypyc without either dragging setuptools in as a secondary build backend or reverse-engineering mypyc's implicit conventions (shim generation, runtime file copying, include dirs, compiler flags, shared-lib naming, output paths).

This issue proposes a build-system-neutral CLI that performs mypyc's frontend + C generation and emits a machine-readable description of the extensions to build. The actual C compilation stays with the caller's build system.

Proposed CLI

mypyc cgen [--target-dir DIR] [--output-file FILE] [--multi-file] [--separate] SOURCE...

Behavior:

1. Run mypyc's frontend and C generation (type check + codegen, same as mypyc_build).
2. Write all generated C files into --target-dir (default build/mypyc): per-group .c/.h files, the runtime library (lib-rt), and per-module shims when a shared library is used.
3. Emit a JSON "build info" document to --output-file, or to stdout if not given.

Flags:

Flag	Meaning
SOURCE...	.py files / package directories to compile (same semantics as mypycify(paths))
--target-dir DIR	directory for generated C files (default build/mypyc)
--output-file FILE	write JSON here instead of stdout
--multi-file	one .c file per module within a group (compile-time/memory win, like mypycify(multi_file=True))
--separate	place each module in its own extension module (like mypycify(separate=True); enables incremental builds)

Deliberately not CLI flags: optimization/debug levels. The emitted cflags contain only flags required for the generated code to work correctly (warning suppression, feature macros like -DMYPYC_LOG_TRACE, the multi-file /GL- workaround on msvc); they never include optimization or debug levels. The consumer's build system owns those decisions, so we don't duplicate the knob at the CLI boundary. (mypycify keeps passing its own defaults, -O3 -g1, so the setuptools path is unchanged.)

JSON build info (schema v1)

{
  "schema_version": 1,
  "target_dir": "/abs/path/to/build/mypyc",
  "extensions": [
    {
      "module": "pkg.mod__mypyc",
      "out_path": "pkg/mod__mypyc.cpython-314-x86_64-linux-gnu.so",
      "sources": ["pkg/__native_pkg.mod.c", "init.c"],
      "include_dirs": ["/abs/.../mypyc/lib-rt", "/abs/path/to/build/mypyc"],
      "depends": ["pkg/__native_pkg.mod.h"],
      "cflags": ["-Werror", "-Wno-unused-function", "..."],
      "link_args": ["-shared"]
    }
  ]
}

Field semantics:

• schema_version: integer, bumped on incompatible changes.
• target_dir: absolute path all relative paths below are resolved against.
• extensions: one entry per extension module to build.
  • module: full dotted extension module name. For a package __init__ shim this follows the setuptools convention "pkg.__init__".
  • out_path: where the built file must end up, relative to the installation root (the equivalent of site-packages), including the platform extension suffix. Note this is a different base than sources/depends (which are relative to target_dir) — this distinction must be explicit in the schema since downstream tooling will join paths.
  • sources: .c files to compile, relative to target_dir (keeps the JSON portable across machines/containers).
  • include_dirs: absolute paths (they point into the mypyc installation, e.g. lib-rt, which cannot be made relative).
  • depends: headers/source files that must trigger a rebuild when they change, relative to target_dir (already transitively resolved from #include scanning, including cross-group export-table headers).
  • cflags: compiler flags required by the generated code (warnings, feature macros). Never includes optimization or debug levels — those are up to the caller's build system.
  • link_args: platform link flags needed to produce a loadable extension module, derived from Python's configured LDSHARED. May be empty; the caller is always responsible for the platform's standard extension linking (e.g. the Python import library on Windows) and for the Python.h include directory.

How a build system would consume this (meson example)

• run_command() or a custom_target() invokes mypyc cgen --output-file ... once per configuration;
• configure step parses the JSON and feeds each entry into py.extension_module(...) using sources, include_dirs, cflags, link_args;
• depends maps to meson's dependency tracking for incremental rebuilds.

A concrete working example (compiling src/bgm_tv_wiki/ast.py; the shim keeps the importable name, the shared library holds the generated code):

Full meson.build example

project('bgm-tv-wiki', 'c',
  default_options: ['c_std=c11', 'buildtype=release'],
  meson_version: '>= 1.5.0')

py = import('python').find_installation(pure: false)

py.install_sources(
  'src/bgm_tv_wiki/__init__.py',
  'src/bgm_tv_wiki/py.typed',
  subdir: 'bgm_tv_wiki',
)

# ---- 1. Run cgen at configure time: generate C sources + info.json ----
cgen_dir = meson.current_build_dir() / 'mypyc-cgen'

run_command(py, '-m', 'mypyc', 'cgen',
  '--target-dir', cgen_dir,
  '--output-file', cgen_dir / 'info.json',
  meson.project_source_root() / 'src' / 'bgm_tv_wiki' / 'ast.py',
  check: true)

# ---- 2. Extract build parameters from info.json (meson can't parse JSON) ----
json_get = '''
import json, sys
info = json.load(open(sys.argv[1]))
ext = next(e for e in info["extensions"] if e["module"] == sys.argv[2])
print("\\n".join(ext[sys.argv[3]]))
'''

cgen_info = cgen_dir / 'info.json'
native = 'bgm_tv_wiki.ast__mypyc'

native_sources = run_command(py, '-c', json_get, cgen_info, native, 'sources',
  check: true).stdout().strip().split('\n')
native_include_dirs = run_command(py, '-c', json_get, cgen_info, native, 'include_dirs',
  check: true).stdout().strip().split('\n')
native_cflags = run_command(py, '-c', json_get, cgen_info, native, 'cflags',
  check: true).stdout().strip().split('\n')
native_link = run_command(py, '-c', json_get, cgen_info, native, 'link_args',
  check: true).stdout().strip().split('\n')

# sources are relative to target_dir; make them absolute
native_sources = [cgen_dir / s for s in native_sources]
# include_dirs are absolute paths (into the mypyc installation); turn them into -I flags
foreach d : native_include_dirs
  native_cflags += ['-I' + d]
endforeach

# ---- 3. Build the extensions ----
# native: compile the mypyc-generated C (include_dirs already covers lib-rt
# and the cgen output directory). No py.dependency() needed — meson's python
# module provides Python.h and the import library automatically.
py.extension_module(
  'ast__mypyc',
  native_sources,
  subdir: 'bgm_tv_wiki',
  install: true,
  c_args: native_cflags,
  link_args: native_link,
)

# shim: forwards to the native module; only needs Python.h
shim_sources = run_command(py, '-c', json_get, cgen_info, 'bgm_tv_wiki.ast', 'sources',
  check: true).stdout().strip().split('\n')
py.extension_module(
  'ast',
  [cgen_dir / s for s in shim_sources],
  subdir: 'bgm_tv_wiki',
  install: true,
)

Notes on this example:

• The extension module names are hardcoded (bgm_tv_wiki.ast__mypyc, bgm_tv_wiki.ast). That works for a single-module build, where the shared library is named <module>__mypyc; multi-module builds get hash-named groups, so names must be read from the JSON.
• run_command executes cgen once at configure time; changing the Python source requires meson setup --reconfigure to regenerate. True incrementality would need custom_target, at the cost of moving JSON parsing into build time.
• The same JSON is directly usable from CMake add_library(MODULE ...), Bazel genrules, or a hand-written Makefile.

Why a CLI (not a Python API)

• Process isolation: the caller's build system can invoke it as a tool without importing mypyc into its own process, and without setuptools installed at all (the CLI path never touches distutils/setuptools).
• The JSON document is a stable, versioned contract between mypyc and any build backend; a Python API would tie callers to mypyc internals and Python packaging conventions.

Open questions

1. Naming/placement: mypyc cgen as a subcommand of the existing python -m mypyc entry point, or a separate console script?
2. Is the schema stable enough to commit to, particularly the "two path bases" convention (sources/depends vs target_dir vs out_path)?
3. Should there be an escape hatch for compiler-profile knobs (e.g. --cflags passthrough), or is "post-edit the JSON" sufficient?
4. Long-term relationship with mypycify: keep both (setuptools users keep mypycify; everyone else uses cgen), or eventually re-implement mypycify on top of cgen?

Happy to work on the implementation once the design is agreed upon.

[JukkaL]

Thanks for the ideas! We are planning to rethink the CLI and build-system integration, so that setuptools will no longer be a requirement.