From f2a037cf0f982a2b72c67163850d42efb1eb7c28 Mon Sep 17 00:00:00 2001 From: Lalatendu Mohanty Date: Fri, 10 Jul 2026 23:01:10 -0400 Subject: [PATCH] docs: add architecture overview docs Also add an AGENTS.md rule to keep the docs in sync when the architecture changes. Closes: #1247 Co-Authored-By: Claude Signed-off-by: Lalatendu Mohanty --- AGENTS.md | 6 + docs/concepts/architecture-overview.rst | 141 ++++++++++++++++++++ docs/concepts/bootstrapper-architecture.rst | 109 +++++++++++++++ docs/concepts/hooks-and-overrides.rst | 116 ++++++++++++++++ docs/concepts/index.rst | 10 ++ docs/concepts/package-settings.rst | 125 +++++++++++++++++ docs/concepts/resolver-architecture.rst | 127 ++++++++++++++++++ 7 files changed, 634 insertions(+) create mode 100644 docs/concepts/architecture-overview.rst create mode 100644 docs/concepts/bootstrapper-architecture.rst create mode 100644 docs/concepts/hooks-and-overrides.rst create mode 100644 docs/concepts/package-settings.rst create mode 100644 docs/concepts/resolver-architecture.rst diff --git a/AGENTS.md b/AGENTS.md index e246b2944..71a2bfd46 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -102,6 +102,12 @@ Look at these before writing code: - Use single backticks around function and class names in markdown (e.g. `req_ctxvar_context()`, `WorkContext`), double backticks in .rst (reStructuredText) - Use Sphinx `versionadded`, `versionremoved`, `versionchanged` directives for user-facing changes (get next version from last git tag) +- When code changes affect the architecture described in `docs/concepts/`, update the relevant doc: + - `architecture-overview.rst` — major subsystems, data flow, extension points, key data structures + - `bootstrapper-architecture.rst` — phase pipeline, class hierarchy, bootstrapper-phase interaction + - `resolver-architecture.rst` — resolution strategies, provider hierarchy, version filtering + - `hooks-and-overrides.rst` — override hook points, global hooks, plugin discovery + - `package-settings.rst` — settings loading flow, merge order, PackageBuildInfo facade ## Commit Messages diff --git a/docs/concepts/architecture-overview.rst b/docs/concepts/architecture-overview.rst new file mode 100644 index 000000000..b9ef2c776 --- /dev/null +++ b/docs/concepts/architecture-overview.rst @@ -0,0 +1,141 @@ +Architecture Overview +===================== + +Fromager rebuilds complete dependency trees of Python wheels from +source. This document maps the major subsystems and how they connect. + +.. seealso:: + + :doc:`bootstrap-vs-build` explains the two operating modes. + :doc:`bootstrapper-architecture` details the bootstrap engine. + :doc:`resolver-architecture` covers version resolution. + :doc:`hooks-and-overrides` describes the two plugin systems. + :doc:`package-settings` explains the settings layering. + +Major Subsystems +---------------- + +.. code-block:: text + + ┌───────────────────────────────────────────────────────┐ + │ CLI & Context │ + │ command parsing, WorkContext, settings │ + └───────────────────────────┬───────────────────────────┘ + │ + ▼ + ┌───────────────────────────────────────────────────────┐ + │ Bootstrap Engine │ + │ iterative DFS loop over phase pipeline │ + │ │ + │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ + │ │ Resolution │ │ Source │ │ Build │ │ + │ │ PyPI, graph │ │ Acquisition │ │ System │ │ + │ │ git URLs │ │ download, │ │ isolated │ │ + │ │ │ │ patch, Rust │ │ envs, │ │ + │ │ │ │ vendor │ │ wheels │ │ + │ └─────────────┘ └─────────────┘ └─────────────┘ │ + └───────────────────────────┬───────────────────────────┘ + │ + ┌────────────────┴────────────────┐ + ▼ ▼ + ┌──────────────────┐ ┌──────────────────┐ + │ Per-Package │ │ Global Hooks │ + │ Overrides │ │ post_bootstrap, │ + │ (stevedore) │ │ post_build │ + └──────────────────┘ └──────────────────┘ + +The **bootstrap engine** orchestrates the other subsystems. For each +package it resolves a version, downloads the source, builds a wheel, +and recurses into dependencies. **Resolution**, **source acquisition**, +and **build** are called as needed during each phase of the pipeline. + +**Extension points** intercept the pipeline at specific steps, allowing +per-package overrides (e.g. custom download logic) and global hooks +(e.g. post-build notifications). + +Data Flow +--------- + +A ``fromager bootstrap`` invocation flows through the subsystems in +this order: + +.. code-block:: text + + requirements.txt + │ + ▼ + ┌─ CLI & Context ───────────────────────────────────────┐ + │ parse args, load settings, create WorkContext │ + └───────────────────────────┬───────────────────────────┘ + │ + ▼ + ┌─ Bootstrap Engine ────────────────────────────────────┐ + │ for each package (iterative DFS): │ + │ resolve version ──► download source │ + │ ──► build wheel ──► extract deps │ + │ ──► recurse into dependencies │ + └───────────────────────────┬───────────────────────────┘ + │ + ▼ + ┌── Outputs ──┐ + │ graph.json │ + │ build-order │ + │ wheels/ │ + │ constraints │ + └─────────────┘ + +The ``build`` command uses the same source acquisition and build +subsystems but skips resolution and recursion -- it compiles a +single package given a name, version, and source URL. + +Extension Points +---------------- + +Fromager has two plugin systems, both using stevedore: + +.. list-table:: + :header-rows: 1 + :widths: 25 35 40 + + * - System + - Scope + - When it fires + * - **Per-package overrides** + - One package + - At each pipeline step (download, build sdist, build wheel, + dependency extraction, etc.) + * - **Global hooks** + - All packages + - After specific events (``post_bootstrap``, ``post_build``, + ``prebuilt_wheel``) + +Per-package overrides replace the default implementation of a step for +a specific package. Global hooks run in addition to the default logic +for every package. See :doc:`hooks-and-overrides` for the full +breakdown. + +Key Data Structures +------------------- + +Four data structures flow between subsystems: + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - Structure + - Role + * - ``WorkContext`` + - Central configuration and state: directory paths, constraints, + dependency graph, settings, and variant info. Passed to every + subsystem. + * - ``DependencyGraph`` + - In-memory directed graph of all resolved dependencies. + Serialized to ``graph.json``. Thread-safe. + * - ``PackageBuildInfo`` + - Per-package build configuration: environment variables, patches, + resolver settings, and build options. Derived from settings + files. + * - ``BuildEnvironment`` + - Isolated virtual environment for building one package. Created + per source build, cleaned up after completion. diff --git a/docs/concepts/bootstrapper-architecture.rst b/docs/concepts/bootstrapper-architecture.rst new file mode 100644 index 000000000..fa52825de --- /dev/null +++ b/docs/concepts/bootstrapper-architecture.rst @@ -0,0 +1,109 @@ +Bootstrapper Architecture +========================= + +The bootstrap command uses an iterative depth-first loop to resolve and +build an entire dependency tree. This document describes the phase +pipeline, class hierarchy, and interaction model at a high level. + +.. seealso:: + + :doc:`architecture-overview` maps the major subsystems. + :doc:`bootstrap-vs-build` covers the difference between bootstrap and + build modes. + +Bootstrap Phase Flow +-------------------- + +Every package passes through a sequence of phases. Source packages +traverse the full pipeline; prebuilt wheels skip the build phases. + +.. code-block:: text + + RESOLVE ──► START ──► PREPARE_SOURCE ─┬─► PREPARE_BUILD ──► BUILD ─┐ + │ │ │ + │ (prebuilt)└────────────────────────────┤ + │ │ + │ ┌────────────────────────────────┘ + │ ▼ + │ PROCESS_INSTALL_DEPS ──► COMPLETE + │ + (already seen) ──► drop + +Each phase returns a list of new items to push onto the work stack. +Dependency-discovery phases (``PREPARE_SOURCE``, ``PREPARE_BUILD``, +``PROCESS_INSTALL_DEPS``) also emit ``RESOLVE`` items for newly +discovered dependencies, which drives the recursive traversal. + +Phase Class Hierarchy +--------------------- + +All phases inherit from the ``Phase`` abstract base class, which defines +the contract every phase must satisfy: + +- ``run(bt)`` — execute the phase logic; return new items for the stack. +- ``background_work(bt)`` — optionally return a callable for background + I/O (used by ``Resolve`` and ``PrepareSource``). +- ``requires_exclusive_run`` — return ``True`` to drain the thread pool + before running (used by ``Build`` for exclusive builds). + +.. list-table:: + :header-rows: 1 + :widths: 25 75 + + * - Phase + - Role + * - ``Resolve`` + - Resolve available versions; fan out one ``Start`` per version + * - ``Start`` + - Add to dependency graph; deduplicate already-seen packages + * - ``PrepareSource`` + - Download source or prebuilt wheel; read build-system deps + * - ``PrepareBuild`` + - Install build-system deps; discover backend and sdist deps + * - ``Build`` + - Build sdist and/or wheel; update the local mirror + * - ``ProcessInstallDeps`` + - Run post-bootstrap hooks; extract install deps; record build order + * - ``Complete`` + - Clean up build directories (terminal phase) + +Each phase wraps a ``WorkItem`` dataclass that accumulates per-package +state (resolved version, build environment, build result, etc.) as it +moves through the pipeline. + +Bootstrapper and Phase Interaction +---------------------------------- + +The ``Bootstrapper`` class owns the work stack and thread pool. It runs +an iterative DFS loop: + +.. code-block:: text + + ┌─────────────────────────────────────────────────────┐ + │ Bootstrapper Loop │ + │ │ + │ while stack: │ + │ item = stack.pop() │ + │ │ + │ if item.requires_exclusive_run: │ + │ drain thread pool │ + │ │ + │ new_items = item.run(self) │ + │ ↑ phase blocks on its own bg_future │ + │ inside run() if it has one │ + │ │ + │ for each new_item in new_items: │ + │ submit background_work() to thread pool │ + │ stack.push(new_item) │ + └─────────────────────────────────────────────────────┘ + +The ``Bootstrapper`` provides services that phases call back into during +``run()``: dependency graph updates, seen-set tracking, build-order +recording, and work-item creation for discovered dependencies. + +Background I/O (version resolution, source downloads) runs in a thread +pool. When new items are pushed onto the stack, the ``Bootstrapper`` +submits their ``background_work()`` immediately so I/O overlaps with +the main thread processing other items. Each phase is responsible for +blocking on its own ``bg_future`` inside ``run()`` when it needs the +result. diff --git a/docs/concepts/hooks-and-overrides.rst b/docs/concepts/hooks-and-overrides.rst new file mode 100644 index 000000000..4b9170a85 --- /dev/null +++ b/docs/concepts/hooks-and-overrides.rst @@ -0,0 +1,116 @@ +Hooks and Overrides +=================== + +Fromager has two stevedore-based plugin systems that serve different +purposes: **per-package overrides** replace default behavior for a +specific package, while **global hooks** broadcast notifications after +events for every package. + +.. seealso:: + + :doc:`architecture-overview` maps the major subsystems. + :doc:`/reference/hooks` documents each override hook's arguments and + return values. :doc:`/customization` covers global hooks with code + examples. + +Comparison +---------- + +.. list-table:: + :header-rows: 1 + :widths: 20 40 40 + + * - Aspect + - Per-Package Overrides + - Global Hooks + * - **Scope** + - One named package + - Every package + * - **Cardinality** + - At most one override module per package + - Multiple plugins per hook + * - **Semantics** + - Replaces the default implementation + - Runs in addition to the default + * - **Return value** + - Used by the caller + - None (fire-and-forget) + * - **Typical use** + - Custom download, build, or resolution logic + - Publishing wheels, validation, logging + +Per-Package Overrides +--------------------- + +An override module provides alternative implementations for specific +build steps. Fromager looks up the module by the canonicalized package +name. If the module defines the requested method, it is called instead +of the default; otherwise the default runs. + +Override hooks fire at these points in the pipeline: + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - Category + - Hook points + * - **Resolution** + - ``get_resolver_provider`` — provide a custom resolvelib provider + * - **Source** + - ``download_source`` — download the source archive + + ``prepare_source`` — unpack and patch source + + ``expected_source_archive_name`` — filename for re-discovery + + ``expected_source_directory_name`` — directory for re-discovery + * - **Build** + - ``build_sdist`` — build a source distribution + + ``build_wheel`` — build a wheel + + ``add_extra_metadata_to_wheels`` — inject extra metadata + * - **Dependencies** + - ``get_build_system_dependencies`` — PEP 517 build-system deps + + ``get_build_backend_dependencies`` — build-backend deps + + ``get_build_sdist_dependencies`` — sdist build deps + + ``get_install_dependencies_of_sdist`` — install deps from sdist + * - **Environment** + - ``update_extra_environ`` — customize build environment variables + +Third-party packages register overrides via the +``fromager.project_overrides`` entry-point group in their +``pyproject.toml``, mapping a package name to a Python module. + +See :doc:`/reference/hooks` for the full API of each hook. + +Global Hooks +------------ + +Global hooks are event callbacks that fire for every package. All +registered plugins for a hook are called sequentially. They cannot +alter the build result — they are purely side-effecting notifications. + +.. list-table:: + :header-rows: 1 + :widths: 25 75 + + * - Hook + - When it fires + * - ``post_build`` + - After a wheel is built from source (not for prebuilt wheels) + * - ``prebuilt_wheel`` + - After a prebuilt wheel is downloaded (not built from source) + * - ``post_bootstrap`` + - After a package is bootstrapped, before its install dependencies + are processed + +Third-party packages register hooks via the ``fromager.hooks`` +entry-point group in their ``pyproject.toml``, mapping a hook name to +a callable. + +See :doc:`/customization` for examples and argument details. diff --git a/docs/concepts/index.rst b/docs/concepts/index.rst index fa1e32265..12c65518c 100644 --- a/docs/concepts/index.rst +++ b/docs/concepts/index.rst @@ -8,13 +8,23 @@ Key Topics These guides explain the fundamental concepts and design principles behind fromager: +* **Architecture Overview** - Major subsystems, data flow, extension points, and key data structures * **Bootstrap vs Build Modes** - Understand the difference between recursive discovery (bootstrap) and single-package builds (build) +* **Bootstrapper Architecture** - Phase pipeline, class hierarchy, and interaction model of the bootstrap engine +* **Resolver Architecture** - Resolution strategies, provider hierarchy, version filtering, and per-package configuration +* **Hooks and Overrides** - The two plugin systems: per-package overrides and global hooks +* **Package Settings** - Settings loading, merge order, and the PackageBuildInfo facade * **Dependency Types** - Learn about build-system, build-backend, build-sdist, and install dependencies .. toctree:: :maxdepth: 1 + architecture-overview bootstrap-vs-build + bootstrapper-architecture + resolver-architecture + hooks-and-overrides + package-settings dependencies Related Practical Guides diff --git a/docs/concepts/package-settings.rst b/docs/concepts/package-settings.rst new file mode 100644 index 000000000..2f0a0d7f3 --- /dev/null +++ b/docs/concepts/package-settings.rst @@ -0,0 +1,125 @@ +Package Settings +================ + +Fromager uses a layered settings system to configure how each package +is resolved, downloaded, patched, and built. Settings flow from YAML +files on disk through a merge pipeline into a single facade object +used by the build pipeline. + +.. seealso:: + + :doc:`/reference/config-reference` documents every configuration + field. :doc:`architecture-overview` maps the major subsystems. + +Settings Loading Flow +--------------------- + +.. code-block:: text + + CLI flags + (--settings-file, --settings-dir, --patches-dir) + │ + ▼ + Settings.from_files() + ├── Load settings.yaml ──► SettingsFile (global config) + └── Glob settings/*.yaml ──► PackageSettings (one per package) + │ + ▼ + Settings object (held by WorkContext) + │ + │ package_build_info("torch") (on demand, cached) + ▼ + PackageBuildInfo + (single facade used by the build pipeline) + +``PackageBuildInfo`` objects are created lazily the first time a +package is queried and cached for the rest of the run. + +Directory Structure +------------------- + +.. code-block:: text + + overrides/ + ├── settings.yaml # Global settings + ├── settings/ # Per-package settings + │ ├── torch.yaml + │ ├── flash-attn.yaml + │ └── ... + └── patches/ # Patches + ├── torch/ # Unversioned (all versions) + │ ├── 001-fix.patch + │ └── cpu/ # Variant-specific + │ └── 002-cpu.patch + └── torch-2.4.0/ # Version-specific + └── 001-ver.patch + +Package YAML filenames use the canonicalized package name (lowercase, +hyphens). All three directories are configurable via CLI flags. + +Merge Order +----------- + +Settings are merged from least-specific to most-specific. Later +layers override earlier ones: + +.. code-block:: text + + ┌──────────────────────────────────────────────────┐ + │ 1. Pydantic field defaults │ + │ (empty env, default build options, etc.) │ + │ │ + │ 2. Global settings.yaml │ + │ (SBOM config, global changelog) │ + │ │ + │ 3. Per-package YAML file │ + │ (env, resolver, build options, patches, ...) │ + │ │ + │ 4. Variant overrides (within package YAML) │ + │ (env vars, pre_built, wheel_server_url) │ + │ │ + │ 5. Version-specific patches and changelog │ + │ (patches/-/, changelog entries)│ + │ │ + │ 6. Override plugin hooks (runtime) │ + │ (update_extra_environ can mutate env vars) │ + └──────────────────────────────────────────────────┘ + +For environment variables specifically, the merge order within a +single ``get_extra_environ()`` call is: parallel-jobs settings, build +environment paths, package-level ``env``, then variant-level ``env``. +Entries can reference earlier values using ``$VAR`` template syntax. + +PackageBuildInfo Facade +----------------------- + +``PackageBuildInfo`` is the single interface the build pipeline uses to +query all package configuration. It wraps a ``PackageSettings`` and +adds variant awareness, patch discovery, and template resolution. + +.. list-table:: + :header-rows: 1 + :widths: 25 75 + + * - Category + - What it provides + * - **Identity** + - Package name, variant, whether a config file exists, override + plugin module + * - **Resolution** + - Sdist server URL, include wheels/sdists flags, platform + filtering, per-package cooldown override + * - **Build config** + - Environment variables, parallel jobs, exclusive build flag, + PEP 517 config settings, build directory + * - **Source** + - Download URL and filename templates with ``${version}`` + substitution, git submodule options + * - **Patches** + - Merged list of unversioned + version-specific + variant-specific + patch files, sorted by filename + * - **Metadata** + - Build tag (from changelog length), annotations, PURL config, + pyproject.toml overrides + +See :doc:`/reference/config-reference` for the full field reference. diff --git a/docs/concepts/resolver-architecture.rst b/docs/concepts/resolver-architecture.rst new file mode 100644 index 000000000..75bdfa370 --- /dev/null +++ b/docs/concepts/resolver-architecture.rst @@ -0,0 +1,127 @@ +Resolver Architecture +===================== + +Fromager resolves package versions from multiple sources using a +provider abstraction. This document describes the resolution +strategies, filtering, and key design decisions. + +.. seealso:: + + :doc:`architecture-overview` maps the major subsystems. + :doc:`bootstrapper-architecture` covers how resolution fits into the + bootstrap pipeline. + +Resolution Strategies +--------------------- + +During bootstrap, the resolver coordinator tries these sources in +priority order and accumulates results in a thread-safe session cache: + +.. code-block:: text + + ┌───────────────────────────────────────────────────────┐ + │ Resolution Priority Order │ + │ │ + │ 1. Session cache (versions found earlier) │ + │ │ miss │ + │ ▼ │ + │ 2. Previous dependency graph (prior run) │ + │ │ miss │ + │ ▼ │ + │ 3. Network (PyPI / GitHub / GitLab) │ + │ (per-package configurable) │ + │ │ age filter empties result │ + │ ▼ │ + │ 4. Cache server fallback │ + │ (multiple_versions mode only) │ + └───────────────────────────────────────────────────────┘ + + Git URL requirements (top-level only) bypass this + chain entirely: the repo is cloned and the version + is extracted from package metadata. + +Versions discovered from any source are merged into the session cache. +Subsequent requests for the same package skip the network entirely if +the cache already contains a matching version. + +Provider Hierarchy +------------------ + +All resolution sources implement the same provider interface, making +them interchangeable. The bootstrap engine and CLI commands interact +with providers through a common ``find_matches()`` method. + +.. list-table:: + :header-rows: 1 + :widths: 25 75 + + * - Provider + - What it queries + * - **PyPI** + - Any PEP 503 Simple API index (pypi.org, custom mirrors). + Filters by platform tags, Python version, and yanked status. + * - **GitHub tags** + - GitHub REST API for repository tags. Versions are parsed + from tag names using a configurable pattern. + * - **GitLab tags** + - GitLab REST API for repository tags. Includes upload + timestamps for age filtering. + * - **Cache** + - Fromager's own wheel server. No cooldown applied. Used as + a fallback when age filtering eliminates all candidates. + +Per-package settings in YAML can select which provider to use and +configure its parameters (index URL, tag pattern, etc.). Override +plugins can replace the provider entirely for a specific package. + +Version Filtering Window +------------------------- + +Two age-based filters can narrow the set of acceptable versions: + +.. code-block:: text + + ◄── too old ──┤ acceptable window ├── too new ──► + + │ │ + max_age cutoff cooldown cutoff + (reject before this) (reject after this) + +- **Cooldown** rejects releases that are too new. It ensures a + minimum time has passed since publication, guarding against + problematic releases. Configurable per package; bypassed for + exact version pins (``==``). + +- **Max age** rejects releases that are too old. Used primarily in + ``multiple_versions`` mode to limit the range of versions built. + +When both are active, only versions published within the window are +considered. If all candidates are filtered out, the behavior depends +on the mode: in single-version mode a warning is logged and all +candidates are kept; in ``multiple_versions`` mode the cache server +fallback is tried instead. + +Flat Resolution by Design +------------------------- + +Fromager deliberately does **not** use resolvelib's transitive +dependency resolution. Every provider's ``get_dependencies()`` method +returns an empty list. This means resolution is a simple "find the +best matching version" operation, not a full constraint solver with +backtracking. + +Transitive dependencies are handled by the bootstrap engine's own DFS +loop: after a package is built, its install dependencies are extracted +from the wheel metadata and pushed onto the work stack as new +``RESOLVE`` phase items. This separation keeps the resolver stateless +and makes each resolution call independent and thread-safe. + +Per-Package Configuration +------------------------- + +Package settings files (YAML) can configure the resolver for each +package independently: resolver type (PyPI, GitHub tags, GitLab tags, +hook-based, or blocked), index URL, cooldown, tag pattern, and +override plugins. + +See :doc:`package-settings` for how settings are loaded and merged.