Skip to content

fix(agentos): make Python packages import and install - #2028

Open
eersnington wants to merge 5 commits into
mainfrom
fix/python-packages
Open

eersnington wants to merge 5 commits into
mainfrom
fix/python-packages

Conversation

@eersnington

@eersnington eersnington commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Fixes #2027.

Python packages did not work on a fresh VM. Bundled pandas did not import, python.install(["pandas"]) ran out of CPU time, and Pyodide packages such as Pillow failed to load while pip reported success.

Before After
import pandas on a fresh VM ModuleNotFoundError imports 2.3.3, no network
python.install(["pandas"]) CPU-time budget exceeded after ~228 s succeeds in ~5 s, no network
python.install(["pillow"]) Failed to load Pillow, then Successfully installed pillow, exit 0 downloads from Pyodide's CDN, checked against the lockfile sha256; import PIL works in later executions
A wheel whose bytes do not match pyodide-lock.json installed EIO: ... does not match the sha256 pinned in pyodide-lock.json, pip exits 1
A package that does not load reported as installed ERROR: Failed to load Pyodide packages: pillow, exit 1
  • Bundled packages import on demand. A sys.meta_path finder in python-runner.mjs unpacks a bundled wheel and its bundled dependencies the first time the guest imports it. It uses only public Pyodide APIs: pyodide.lockfile, pyodide.unpackArchive(bytes, "whl", { extractDir }), and pyodide.loadedPackages. The unused AGENTOS_PYTHON_PRELOAD_PACKAGES knob is removed: only tests set it, which hid the bug.
  • Managed Pyodide paths serve fs.readFileSync. crates/execution/src/python.rs handles fs.readFileRangeSync, the ranged read the guest fs polyfill uses.
  • The sidecar owns package trust. Pyodide loads a lockfile package from its package cache before it downloads it. The sidecar seeds the bundled wheels (including micropip) into that cache, so bundled packages and pip bootstrap need no network. The sidecar also rejects any write into the cache whose sha256 does not match pyodide-lock.json, so a downloaded wheel is verified before Pyodide installs it.
  • Failed loads fail the install. Pyodide's loadPackage() logs a failed package and resolves anyway. The micropip compat loadPackage now rejects, and pip exits 1.
  • pip install does not copy bundled packages into the VM filesystem. They are importable in every process already.
  • Pyodide packages download from Pyodide's own versioned CDN by default. The runner passes packageBaseUrl to loadPyodide() only when AGENTOS_PYODIDE_PACKAGE_BASE_URL is set. Downloads go through the VM network policy.
  • Python VFS bridge cost follows the change, not the directory or file size. lookup does one lstat of the requested name instead of re-listing the parent directory, and write sends only the written range at its offset.
# Fresh VM, no install step, no network
import pandas
pandas.DataFrame({"a": [1, 2]})["a"].sum()  # 3

Python VFS bridge, release sidecar, median of 5 runs:

Operation in the guest Before After
Create a small file in a 200+ entry directory 76.1 ms 3.91 ms
Create a small file, directory growing 0 to 200 entries 33.3 ms 3.88 ms
Write 4 MiB in 64 KiB chunks 3,603 ms 218 ms
python.install(["openpyxl"]) end to end, PyPI download included 9,095 ms 6,951 ms
os.stat / read 4 MiB 0.18 ms / 3 ms 0.18 ms / 4 ms

@railway-app

railway-app Bot commented Oct 9, 2026

Copy link
Copy Markdown

This PR was not deployed automatically as @eersnington does not have access to the Railway project.

In order to get automatic PR deploys, please add @eersnington to your workspace on Railway.

@eersnington
eersnington marked this pull request as ready for review October 12, 2026 01:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

runtime.python fails with "command not found: python" on a default VM (0.2.21, 0.2.22); Pyodide-distribution packages don't install

1 participant