diff --git a/Cargo.lock b/Cargo.lock index e7667d62d6..ee9db6d86d 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -112,6 +112,7 @@ dependencies = [ "nix 0.29.0", "serde", "serde_json", + "sha2 0.10.9", "tempfile", "tokio", "tracing", diff --git a/crates/execution/CLAUDE.md b/crates/execution/CLAUDE.md index 839fa86e39..5c3e3bd912 100644 --- a/crates/execution/CLAUDE.md +++ b/crates/execution/CLAUDE.md @@ -226,7 +226,7 @@ The host-support ESM loader hook (`loader.mjs`) is generated from a Rust string - Warmup marker fingerprints for guest assets must include mutation data (`size` plus `mtime`/`mtime_nsec`), not just inode identity; in-place rewrites of Pyodide or WASM assets can preserve the inode and still need to invalidate prewarm stamps. - Pyodide bootstrap hardening in `node_import_cache.rs` must stay staged: `globalThis` guards can go in before `loadPyodide()`, but mutating `process` before `loadPyodide()` breaks the bundled Pyodide runtime under Node `--permission`. - Python RPC shims in `crates/execution/assets/runners/python-runner.mjs` should translate JS bridge failures into Python-native exceptions (`PermissionError`, `FileNotFoundError`, `OSError`) instead of leaking `JsException`, and Python `subprocess.run()` should inherit the VM cwd from sidecar process state rather than Pyodide's internal `/home/pyodide` working directory. -- Treat bundled Pyodide package loading and user-configured `AGENTOS_PYODIDE_PACKAGE_BASE_URL` as separate phases in `python-runner.mjs`: keep `loadPyodide(... packageBaseUrl)` plus the initial `pyodide.loadPackage("micropip")`/bundled preload path pinned to `/__agentos_pyodide`, then switch `pyodide._api.config.packageBaseUrl` afterward for user `micropip.install(...)` URLs. When guest Python needs HTTP wheels, patch `pyodide.http.pyfetch` through the Python `httpRequestSync` bridge so `micropip` obeys sidecar network policy and loopback exemptions instead of bypassing them. +- Pyodide reads `packageBaseUrl` only in `loadPyodide()`, and it checks its package cache before it downloads. Keep bundled wheels seeded in the package cache, and keep the sidecar's sha256 check on every write into the cache. When guest Python needs HTTP wheels, patch `pyodide.http.pyfetch` through the Python `httpRequestSync` bridge so `micropip` obeys sidecar network policy and loopback exemptions instead of bypassing them. - Guest runtime identity defaults must stay aligned across JS, WASM, and Python: keep `HOME` bound to the kernel user's homedir, keep `PWD` bound to the execution cwd, feed those values into the Pyodide bootstrap env, and make sure Python `execute()` requests still pass through `prepare_guest_runtime_env(...)` instead of bypassing the shared runtime-env assembly. - The shared runtime env contract also includes a stable guest `PATH` plus internal-env filtering: `prepare_guest_runtime_env(...)` should supply the canonical guest search path, `python-runner.mjs` should expose that `PATH` inside `os.environ`, and the WASM `AGENTOS_GUEST_ENV` payload must strip internal control vars like `AGENTOS_*` / `NODE_SYNC_RPC_*` before they reach guest-visible WASI env. - Pyodide `micropip` support must keep guest `js` / `pyodide_js` imports blocked for user Python code while exposing only a narrow internal compat surface to `micropip` and `pyodide.http`; widening that exception re-opens host escape hatches. diff --git a/crates/execution/Cargo.toml b/crates/execution/Cargo.toml index f7b05b573f..bff59311b7 100644 --- a/crates/execution/Cargo.toml +++ b/crates/execution/Cargo.toml @@ -32,6 +32,7 @@ flume = "0.11" nix = { version = "0.29", features = ["fs"] } serde = { version = "1.0", features = ["derive"] } serde_json = "1" +sha2 = "0.10" tokio = { version = "1", features = ["rt", "sync", "time"] } tracing = "0.1" diff --git a/crates/execution/assets/pyodide/README.md b/crates/execution/assets/pyodide/README.md index d5fb73cf19..0f98f3c7cb 100644 --- a/crates/execution/assets/pyodide/README.md +++ b/crates/execution/assets/pyodide/README.md @@ -21,17 +21,21 @@ Bundle size as vendored in this directory: - Offline package wheels: 8,347,517 bytes - Total: 20,631,138 bytes (19.68 MiB) -`python-runner.mjs` points `indexURL` at this local directory and defaults `packageBaseUrl` to the same bundled asset root so `pyodide.loadPackage()` and the built-in `micropip` bootstrap stay offline. +`python-runner.mjs` points `indexURL` at this local directory. + +The offline wheels need no install step: the first `import` of one of their top-level modules (for example `import pandas`) unpacks the wheel and its bundled dependencies. Dynamic package installs: -- `AGENTOS_PYODIDE_PACKAGE_BASE_URL` can override the package base used by Pyodide package resolution when a Python execution needs to install additional wheels from a network-visible host. -- The bundled `micropip` wheel is still loaded from the local asset directory first so package-manager bootstrap does not depend on external network access. +- Other `pyodide-lock.json` packages (for example Pillow) download from Pyodide's own CDN for the bundled runtime version, `https://cdn.jsdelivr.net/pyodide/v/full/`. Downloads go through the VM network policy, so the VM needs a network grant for that host. `AGENTOS_PYODIDE_PACKAGE_BASE_URL` overrides the base URL, for example with a mirror. +- The sidecar seeds the bundled wheels, including `micropip`, into the Pyodide package cache. Pyodide checks the cache first, so package-manager bootstrap and bundled packages do not depend on external network access. +- The sidecar rejects a write into the package cache unless its bytes match the sha256 that `pyodide-lock.json` pins, so a downloaded wheel is verified before Pyodide installs it. +- `pip install` fails with a non-zero exit when a requested package does not load. - `await micropip.install("https://.../package.whl")` goes through the Python runner's bridge-backed fetch path, which means network permissions are enforced by the agentos kernel rather than bypassing it. Debug timing output: - Set `AGENTOS_PYTHON_WARMUP_DEBUG=1` on a Python execution request to emit `__AGENTOS_PYTHON_WARMUP_METRICS__:` JSON lines on stderr. - The Rust execution engine emits a `phase:"prewarm"` line that reports whether warmup executed or reused the cached compile-cache path, plus the measured warmup duration in milliseconds. -- `python-runner.mjs` emits a `phase:"startup"` line just before guest code runs, including total startup time, `loadPyodide()` time, package-load time, package count, and whether the source was inline code, a file, or prewarm-only. +- `python-runner.mjs` emits a `phase:"startup"` line just before guest code runs, including total startup time, `loadPyodide()` time, and whether the source was inline code, a file, or prewarm-only. Startup targets: - Cold start target: first request in a fresh cache should keep the combined prewarm plus startup path under `3000ms` on commodity hardware. diff --git a/crates/execution/assets/runners/python-runner.mjs b/crates/execution/assets/runners/python-runner.mjs index 1a200f6825..2b86d9d071 100644 --- a/crates/execution/assets/runners/python-runner.mjs +++ b/crates/execution/assets/runners/python-runner.mjs @@ -1,4 +1,4 @@ -import { closeSync, createReadStream, mkdirSync, readFileSync, readSync, writeSync } from 'node:fs'; +import { closeSync, createReadStream, mkdirSync, readdirSync, readFileSync, readSync, writeSync } from 'node:fs'; import { readFile } from 'node:fs/promises'; import * as moduleBuiltin from 'node:module'; import { performance as realPerformance } from 'node:perf_hooks'; @@ -22,15 +22,12 @@ const PYTHON_PREWARM_ONLY_ENV = 'AGENTOS_PYTHON_PREWARM_ONLY'; const RETAIN_LANGUAGE_CONTEXT_ENV = 'AGENTOS_RETAIN_LANGUAGE_CONTEXT'; const PYTHON_WARMUP_DEBUG_ENV = 'AGENTOS_PYTHON_WARMUP_DEBUG'; const PYTHON_WARMUP_METRICS_PREFIX = '__AGENTOS_PYTHON_WARMUP_METRICS__:'; -const PYTHON_PRELOAD_PACKAGES_ENV = 'AGENTOS_PYTHON_PRELOAD_PACKAGES'; const PYTHON_VFS_RPC_REQUEST_FD_ENV = 'AGENTOS_PYTHON_VFS_RPC_REQUEST_FD'; const PYTHON_VFS_RPC_RESPONSE_FD_ENV = 'AGENTOS_PYTHON_VFS_RPC_RESPONSE_FD'; const FORWARD_KERNEL_STDIN_RPC_ENV = 'AGENTOS_FORWARD_KERNEL_STDIN_RPC'; const INTERNAL_ENV = globalThis.__agentOSPythonInternalEnv ?? Object.create(null); const ALLOW_PROCESS_BINDINGS = readRunnerEnv('AGENTOS_ALLOW_PROCESS_BINDINGS') === '1'; const STDIN_FD = 0; -const SUPPORTED_PRELOAD_PACKAGES = ['numpy', 'pandas']; -const SUPPORTED_PRELOAD_PACKAGE_SET = new Set(SUPPORTED_PRELOAD_PACKAGES); const DENIED_BUILTINS = new Set([ 'child_process', 'cluster', @@ -415,8 +412,6 @@ function emitPythonStartupMetrics({ prewarmOnly, startupMs, loadPyodideMs, - packageLoadMs, - packageCount, source, }) { if (readRunnerEnv(PYTHON_WARMUP_DEBUG_ENV) !== '1') { @@ -430,59 +425,11 @@ function emitPythonStartupMetrics({ prewarmOnly, startupMs, loadPyodideMs, - packageLoadMs, - packageCount, source, })}`, ); } -function parsePreloadPackages(value) { - if (value == null || value.trim() === '') { - return []; - } - - let parsed; - try { - parsed = JSON.parse(value); - } catch (error) { - throw new Error( - `${PYTHON_PRELOAD_PACKAGES_ENV} must be a JSON array of package names: ${formatError(error)}`, - ); - } - - if (!Array.isArray(parsed)) { - throw new Error(`${PYTHON_PRELOAD_PACKAGES_ENV} must be a JSON array of package names`); - } - - const packages = []; - const seen = new Set(); - - for (const entry of parsed) { - if (typeof entry !== 'string') { - throw new Error(`${PYTHON_PRELOAD_PACKAGES_ENV} entries must be strings`); - } - - const name = entry.trim(); - if (name.length === 0) { - throw new Error(`${PYTHON_PRELOAD_PACKAGES_ENV} entries must not be empty`); - } - - if (!SUPPORTED_PRELOAD_PACKAGE_SET.has(name)) { - throw new Error( - `Unsupported bundled Python package "${name}". Available packages: ${SUPPORTED_PRELOAD_PACKAGES.join(', ')}`, - ); - } - - if (!seen.has(name)) { - seen.add(name); - packages.push(name); - } - } - - return packages; -} - function parseOptionalFd(name) { const value = readRunnerEnv(name); if (value == null || value.trim() === '') { @@ -558,10 +505,13 @@ function createPythonBridgeRpcBridge() { async fsRead(path) { return this.fsReadSync(path); }, - fsWriteSync(path, content) { + // Without `offset` the content replaces the whole file; with it, the + // content is written at that byte offset. + fsWriteSync(path, content, offset) { requestSync('fsWrite', { path, contentBase64: normalizeWriteContent(content), + ...(offset === undefined ? {} : { offset }), }); }, async fsWrite(path, content) { @@ -782,10 +732,13 @@ function createPythonFdRpcBridge() { async fsRead(path) { return this.fsReadSync(path); }, - fsWriteSync(path, content) { + // Without `offset` the content replaces the whole file; with it, the + // content is written at that byte offset. + fsWriteSync(path, content, offset) { requestSync('fsWrite', { path, contentBase64: normalizeWriteContent(content), + ...(offset === undefined ? {} : { offset }), }); }, async fsWrite(path, content) { @@ -1557,6 +1510,11 @@ function installPythonKernelRpcShims(pyodide) { pyodide.runPython(PYTHON_KERNEL_RPC_SHIMS_SOURCE); } +// PEP 503 normalization, which is also how pyodide-lock.json keys packages. +function normalizePythonPackageName(name) { + return String(name).toLowerCase().replace(/[-_.]+/g, '-'); +} + function installPythonMicropipCompat(pyodide) { if (typeof pyodide?.registerJsModule !== 'function') { return; @@ -1604,9 +1562,22 @@ function installPythonMicropipCompat(pyodide) { lockfile_info: pyodide?._api?.lockfile_info, lockfile_packages: pyodide?._api?.lockfile_packages, }; + // Pyodide's loadPackage() logs a package that fails to load and resolves + // anyway. Reject instead so micropip.install() does not report success. + const loadPackage = async (names) => { + const loaded = await pyodide.loadPackage(names); + const loadedNames = new Set(Object.keys(pyodide.loadedPackages).map(normalizePythonPackageName)); + const missing = Array.from(names).filter( + (name) => !loadedNames.has(normalizePythonPackageName(name)), + ); + if (missing.length > 0) { + throw new Error(`Failed to load Pyodide packages: ${missing.join(', ')}`); + } + return loaded; + }; pyodide.registerJsModule('agentos_internal_pyodide_js', { loadedPackages: pyodide.loadedPackages, - loadPackage: pyodide.loadPackage?.bind(pyodide), + loadPackage, lockfileBaseUrl: pyodide?._api?.config?.packageBaseUrl ?? '', _api: pyodideApiCompat, }); @@ -1785,17 +1756,12 @@ function installPythonWorkspaceFs(pyodide, bridge) { node.mode = stat.mode; node.timestamp = Date.now(); - if (FS.isFile(stat.mode) && !node.agentOSDirty) { - node.agentOSRemoteSize = stat.size; - } } function createWorkspaceNode(parent, name, mode, dev, guestPath) { const node = MEMFS.createNode(parent, name, mode, dev); node.agentOSGuestPath = guestPath; - node.agentOSDirty = false; node.agentOSLoaded = FS.isDir(mode); - node.agentOSRemoteSize = 0; if (FS.isDir(mode)) { node.node_ops = workspaceDirNodeOps; node.stream_ops = workspaceDirStreamOps; @@ -1808,60 +1774,22 @@ function installPythonWorkspaceFs(pyodide, bridge) { return node; } - function syncDirectory(node) { - const guestPath = nodeGuestPath(node); - const entries = withFsErrors(() => bridge.fsReaddirSync(guestPath)); - const remoteNames = new Set(entries); - + // Drops cached children the VM no longer has, for example files another + // process deleted. Children that exist are created lazily by `lookup`. + function pruneMissingChildren(node, remoteNames) { for (const name of Object.keys(node.contents || {})) { if (remoteNames.has(name)) { continue; } - - const child = node.contents[name]; - if (FS.isDir(child.mode)) { + if (FS.isDir(node.contents[name].mode)) { memfsDirNodeOps.rmdir(node, name); } else { memfsDirNodeOps.unlink(node, name); } } - - for (const name of entries) { - const childPath = joinGuestPath(guestPath, name); - // lstat (don't follow) so a host symlink is created as a link node. - const stat = withFsErrors(() => bridge.fsLstatSync(childPath)); - const existing = node.contents[name]; - - if (existing) { - const existingIsDir = FS.isDir(existing.mode); - const remoteIsDir = Boolean(stat?.isDirectory); - if (existingIsDir !== remoteIsDir) { - if (existingIsDir) { - memfsDirNodeOps.rmdir(node, name); - } else { - memfsDirNodeOps.unlink(node, name); - } - } else { - existing.agentOSGuestPath = childPath; - updateNodeFromRemoteStat(existing, stat); - if (FS.isFile(existing.mode) && !existing.agentOSDirty) { - existing.agentOSLoaded = false; - } - continue; - } - } - - const mode = stat?.mode ?? (stat?.isDirectory ? 0o040755 : 0o100644); - const child = createWorkspaceNode(node, name, mode, 0, childPath); - updateNodeFromRemoteStat(child, stat); - } } function loadFileContents(node) { - if (node.agentOSDirty) { - return; - } - const stat = withFsErrors(() => bridge.fsStatSync(nodeGuestPath(node))); updateNodeFromRemoteStat(node, stat); const contentBase64 = withFsErrors(() => bridge.fsReadSync(nodeGuestPath(node))); @@ -1869,21 +1797,18 @@ function installPythonWorkspaceFs(pyodide, bridge) { node.contents = bytes; node.usedBytes = bytes.length; node.agentOSLoaded = true; - node.agentOSRemoteSize = bytes.length; } function persistFile(node) { const contents = node.contents ? MEMFS.getFileDataAsTypedArray(node) : new Uint8Array(0); withFsErrors(() => bridge.fsWriteSync(nodeGuestPath(node), contents)); - node.agentOSDirty = false; node.agentOSLoaded = true; - node.agentOSRemoteSize = contents.length; node.timestamp = Date.now(); } function makeStat(node, stat) { const mode = stat?.mode ?? node.mode; - const size = FS.isDir(mode) ? 4096 : (node.agentOSDirty ? node.usedBytes : (stat?.size ?? node.usedBytes ?? 0)); + const size = FS.isDir(mode) ? 4096 : (stat?.size ?? node.usedBytes ?? 0); const timestamp = new Date(node.timestamp || Date.now()); return { @@ -1915,6 +1840,7 @@ function installPythonWorkspaceFs(pyodide, bridge) { function propagateSetattrToHost(node, attr) { if (!attr) return; const payload = {}; + if (attr.size != null) payload.size = attr.size; if (attr.mode != null) payload.mode = attr.mode & 0o7777; // `os.chown(p, uid, -1)` keeps a side; never forward a negative sentinel // (it would saturate to 0 = root on the host). @@ -1949,21 +1875,21 @@ function installPythonWorkspaceFs(pyodide, bridge) { const workspaceFileNodeOps = { getattr(node) { - const stat = node.agentOSDirty - ? null - : withFsErrors(() => bridge.fsStatSync(nodeGuestPath(node))); - if (stat) { - updateNodeFromRemoteStat(node, stat); - } + const stat = withFsErrors(() => bridge.fsStatSync(nodeGuestPath(node))); + updateNodeFromRemoteStat(node, stat); return makeStat(node, stat); }, setattr(node, attr) { + // Truncating to a non-zero length keeps a prefix of the current + // contents, so load them before resizing the in-isolate copy. + if (attr?.size && !node.agentOSLoaded) { + loadFileContents(node); + } // Host first (see link setattr) so a failed host op leaves the in-isolate // node untouched. propagateSetattrToHost(node, attr); memfsFileNodeOps.setattr(node, attr); if (attr?.size != null) { - node.agentOSDirty = true; node.agentOSLoaded = true; } }, @@ -1974,29 +1900,36 @@ function installPythonWorkspaceFs(pyodide, bridge) { return memfsFileStreamOps.llseek(stream, offset, whence); }, read(stream, buffer, offset, length, position) { - if (!stream.node.agentOSLoaded && !stream.node.agentOSDirty) { + if (!stream.node.agentOSLoaded) { loadFileContents(stream.node); } return memfsFileStreamOps.read(stream, buffer, offset, length, position); }, write(stream, buffer, offset, length, position, canOwn) { - if (!stream.node.agentOSLoaded && !stream.node.agentOSDirty) { + if (!stream.node.agentOSLoaded) { loadFileContents(stream.node); } const written = memfsFileStreamOps.write(stream, buffer, offset, length, position, canOwn); - stream.node.agentOSDirty = true; - persistFile(stream.node); + // Send only the written range so a file written in many chunks costs + // O(bytes written), not O(file size) per chunk. + withFsErrors(() => + bridge.fsWriteSync( + nodeGuestPath(stream.node), + buffer.subarray(offset, offset + written), + position, + ), + ); + stream.node.timestamp = Date.now(); return written; }, mmap(stream, length, position, prot, flags) { - if (!stream.node.agentOSLoaded && !stream.node.agentOSDirty) { + if (!stream.node.agentOSLoaded) { loadFileContents(stream.node); } return memfsFileStreamOps.mmap(stream, length, position, prot, flags); }, msync(stream, buffer, offset, length, mmapFlags) { const result = memfsFileStreamOps.msync(stream, buffer, offset, length, mmapFlags); - stream.node.agentOSDirty = true; persistFile(stream.node); return result; }, @@ -2013,22 +1946,18 @@ function installPythonWorkspaceFs(pyodide, bridge) { propagateSetattrToHost(node, attr); memfsDirNodeOps.setattr(node, attr); }, + // Emscripten calls `lookup` only for names it has not cached, so one lstat + // of that name is enough. lookup(parent, name) { - syncDirectory(parent); - try { - return memfsDirNodeOps.lookup(parent, name); - } catch (error) { - if (!(error instanceof FS.ErrnoError) || error.errno !== ERRNO_CODES.ENOENT) { - throw error; - } - - const guestPath = joinGuestPath(nodeGuestPath(parent), name); - // lstat (don't follow) so a directly-looked-up host symlink is a link node. - const stat = withFsErrors(() => bridge.fsLstatSync(guestPath)); - const child = createWorkspaceNode(parent, name, stat.mode, 0, guestPath); - updateNodeFromRemoteStat(child, stat); - return child; + const guestPath = joinGuestPath(nodeGuestPath(parent), name); + // lstat (don't follow) so a directly-looked-up host symlink is a link node. + const stat = withFsErrors(() => bridge.fsLstatSync(guestPath)); + if (!stat) { + throw new FS.ErrnoError(ERRNO_CODES.ENOENT); } + const child = createWorkspaceNode(parent, name, stat.mode, 0, guestPath); + updateNodeFromRemoteStat(child, stat); + return child; }, mknod(parent, name, mode, dev) { const guestPath = joinGuestPath(nodeGuestPath(parent), name); @@ -2038,7 +1967,6 @@ function installPythonWorkspaceFs(pyodide, bridge) { } else if (FS.isFile(mode)) { node.contents = new Uint8Array(0); node.usedBytes = 0; - node.agentOSDirty = true; persistFile(node); } return node; @@ -2069,8 +1997,9 @@ function installPythonWorkspaceFs(pyodide, bridge) { } }, readdir(node) { - syncDirectory(node); - return memfsDirNodeOps.readdir(node); + const entries = withFsErrors(() => bridge.fsReaddirSync(nodeGuestPath(node))); + pruneMissingChildren(node, new Set(entries)); + return ['.', '..', ...entries]; }, symlink(parent, newName, oldPath) { const guestPath = joinGuestPath(nodeGuestPath(parent), newName); @@ -2086,9 +2015,7 @@ function installPythonWorkspaceFs(pyodide, bridge) { mount(mount) { const root = MEMFS.mount(mount); root.agentOSGuestPath = mount.mountpoint; - root.agentOSDirty = false; root.agentOSLoaded = true; - root.agentOSRemoteSize = 0; root.node_ops = workspaceDirNodeOps; root.stream_ops = workspaceDirStreamOps; return root; @@ -2351,16 +2278,85 @@ function installPythonVfsSitePackages(pyodide) { } } +// Bundled wheels (pandas, numpy, and their dependencies) are importable without +// an install step. `pyodide.loadPackage()` is async and an import statement +// cannot wait for it, so this finder unpacks the bundled wheel and its bundled +// dependencies synchronously the first time the guest imports one of its +// top-level modules. Extension modules then load through the regular dlopen +// path on import. +// +// Returns a predicate that tells whether a distribution is bundled. +function installPythonBundledPackages(pyodide, bundledDir) { + const lockfilePackages = pyodide?.lockfile?.packages; + if (typeof pyodide?.runPython !== 'function' || !lockfilePackages) { + return () => false; + } + + const bundledFiles = new Set(readdirSync(bundledDir)); + const packageByImport = new Map( + Object.entries(lockfilePackages).flatMap(([key, entry]) => + entry.imports.map((importName) => [importName, key]), + ), + ); + const sitePackages = pyodide.runPython('import site; site.getsitepackages()[0]'); + const unpackBundledPackage = (key) => { + const entry = lockfilePackages[key]; + if (Object.prototype.hasOwnProperty.call(pyodide.loadedPackages, entry.name)) { + return true; + } + if (!bundledFiles.has(entry.file_name) || !entry.depends.every(unpackBundledPackage)) { + return false; + } + const wheel = readFileSync(path.join(bundledDir, entry.file_name)); + pyodide.unpackArchive(new Uint8Array(wheel.buffer, wheel.byteOffset, wheel.byteLength), 'whl', { + extractDir: sitePackages, + }); + pyodide.loadedPackages[entry.name] = 'default channel'; + return true; + }; + const unpackForImport = (importName) => { + const key = packageByImport.get(importName); + return key !== undefined && unpackBundledPackage(key); + }; + + pyodide.globals.set('__agentos_unpack_for_import', unpackForImport); + try { + pyodide.runPython(` +def _agentos_install_bundled_finder(unpack_for_import): + import importlib, importlib.machinery, sys + + class AgentOsBundledPackageFinder: + def find_spec(self, fullname, path=None, target=None): + if path is not None or not unpack_for_import(fullname): + return None + importlib.invalidate_caches() + return importlib.machinery.PathFinder.find_spec(fullname) + + sys.meta_path.append(AgentOsBundledPackageFinder()) + +_agentos_install_bundled_finder(__agentos_unpack_for_import) +del _agentos_install_bundled_finder +`); + } finally { + pyodide.globals.delete('__agentos_unpack_for_import'); + } + + return (distributionName) => + bundledFiles.has(lockfilePackages[normalizePythonPackageName(distributionName)]?.file_name); +} + // `pip` / `python -m pip`: emulate the common pip CLI via Pyodide's micropip, // which fetches wheels through the runner's kernel-backed fetch (so egress is // governed by the VM network policy, never an ambient host fetch). Installed // packages are copied into the persistent VFS site-packages so they survive the // per-process interpreter and can be imported by a later `python` invocation. -async function runPythonPip(pyodide) { +// Bundled packages are importable in every process, so they are not copied. +async function runPythonPip(pyodide, isBundledPackage) { pyodide.globals.set('__agentos_vfs_site', PYTHON_VFS_SITE_PACKAGES); + pyodide.globals.set('__agentos_is_bundled_package', isBundledPackage); try { await pyodide.runPythonAsync(` -import os, shutil, site, sys +import importlib.metadata, os, shutil, site, sys _agentos_pip_args = sys.argv[1:] if _agentos_pip_args and _agentos_pip_args[0] == "install": import micropip @@ -2370,12 +2366,22 @@ if _agentos_pip_args and _agentos_pip_args[0] == "install": sys.exit(1) _agentos_sp = site.getsitepackages()[0] _agentos_before = set(os.listdir(_agentos_sp)) if os.path.isdir(_agentos_sp) else set() - await micropip.install(_agentos_pip_pkgs) + try: + await micropip.install(_agentos_pip_pkgs) + except Exception as _agentos_error: + print(f"ERROR: {_agentos_error}", file=sys.stderr) + sys.exit(1) # Persist whatever micropip extracted into the in-isolate site-packages into # the VFS-backed site-packages so it survives this process. os.makedirs(__agentos_vfs_site, exist_ok=True) _agentos_after = set(os.listdir(_agentos_sp)) if os.path.isdir(_agentos_sp) else set() - for _agentos_name in sorted(_agentos_after - _agentos_before): + _agentos_bundled = { + _agentos_file.parts[0] + for _agentos_dist in importlib.metadata.distributions(path=[_agentos_sp]) + if __agentos_is_bundled_package(_agentos_dist.name) + for _agentos_file in _agentos_dist.files or () + } + for _agentos_name in sorted(_agentos_after - _agentos_before - _agentos_bundled): _agentos_src = os.path.join(_agentos_sp, _agentos_name) _agentos_dst = os.path.join(__agentos_vfs_site, _agentos_name) if os.path.isdir(_agentos_src): @@ -2396,6 +2402,7 @@ else: } finally { try { pyodide.globals.delete('__agentos_vfs_site'); + pyodide.globals.delete('__agentos_is_bundled_package'); } catch (error) { writeStream( process.stderr, @@ -2440,15 +2447,15 @@ try { emitWarmupStage('startup'); emitWarmupStage(`python-rpc-bridge:${bridgePythonRpc ? 'on' : 'off'}`); const { indexPath, indexUrl } = resolveIndexLocation(requiredEnv(PYODIDE_INDEX_URL_ENV)); - const bundledPackageBaseUrl = normalizeBaseUrl(indexPath); - const packageBaseUrl = normalizeBaseUrl( - readRunnerEnv(PYODIDE_PACKAGE_BASE_URL_ENV) ?? bundledPackageBaseUrl, - ); + // Without a configured base URL, Pyodide downloads lockfile packages from its + // own CDN for the bundled runtime version. + const configuredPackageBaseUrl = readRunnerEnv(PYODIDE_PACKAGE_BASE_URL_ENV); + const packageBaseUrl = + configuredPackageBaseUrl == null ? undefined : normalizeBaseUrl(configuredPackageBaseUrl); const packageCacheDir = resolvePyodidePackageCacheDir(); emitWarmupStage(`package-cache-dir:${packageCacheDir}`); const prewarmOnly = readRunnerEnv(PYTHON_PREWARM_ONLY_ENV) === '1'; retainLanguageContext = readRunnerEnv(RETAIN_LANGUAGE_CONTEXT_ENV) === '1'; - const preloadPackages = parsePreloadPackages(readRunnerEnv(PYTHON_PRELOAD_PACKAGES_ENV)); const lockFileContents = await readLockFileContents(indexPath, indexUrl).catch((error) => { throw wrapPythonStartupError('lock file read', { indexPath, indexUrl }, error); }); @@ -2475,8 +2482,6 @@ try { prewarmOnly: true, startupMs: realPerformance.now() - startupStarted, loadPyodideMs: 0, - packageLoadMs: 0, - packageCount: 0, source: 'prewarm', }); process.exitCode = 0; @@ -2489,7 +2494,7 @@ try { const pyodide = await loadPyodide({ indexURL: indexPath, lockFileContents, - packageBaseUrl: bundledPackageBaseUrl, + packageBaseUrl, packageCacheDir, env: buildPythonRuntimeEnv(), stdout: writePyodideStdout, @@ -2501,7 +2506,6 @@ try { indexPath, indexUrl, packageBaseUrl, - bundledPackageBaseUrl, packageCacheDir, pyodideModuleUrl, }, @@ -2514,36 +2518,17 @@ try { restorePyodideShellCompat(); emitWarmupStage('after-load-pyodide'); const loadPyodideMs = realPerformance.now() - loadPyodideStarted; - let packageLoadMs = 0; installPythonStdin(pyodide); installPythonWorkspaceFs(pyodide, pythonVfsRpcBridge); installPythonVfsSitePackages(pyodide); installPythonGuestLoaderHooks(); - if (pyodide?._api?.config) { - pyodide._api.config.packageBaseUrl = bundledPackageBaseUrl; - emitWarmupStage(`pyodide-package-base:${pyodide._api.config.packageBaseUrl}`); - } - const canLoadPackages = typeof pyodide?.loadPackage === 'function'; - if (!canLoadPackages && preloadPackages.length > 0) { - throw new Error('Pyodide loadPackage() is required to preload Python packages'); - } - if (canLoadPackages) { + if (typeof pyodide?.loadPackage === 'function') { emitWarmupStage('before-load-micropip'); await pyodide.loadPackage(['micropip']); emitWarmupStage('after-load-micropip'); - if (preloadPackages.length > 0) { - emitWarmupStage('before-load-preload-packages'); - const packageLoadStarted = realPerformance.now(); - await pyodide.loadPackage(preloadPackages); - packageLoadMs = realPerformance.now() - packageLoadStarted; - emitWarmupStage('after-load-preload-packages'); - } - } - if (pyodide?._api?.config) { - pyodide._api.config.packageBaseUrl = packageBaseUrl; - emitWarmupStage(`micropip-package-base:${pyodide._api.config.packageBaseUrl}`); } + const isBundledPackage = installPythonBundledPackages(pyodide, indexPath); installPythonMicropipCompat(pyodide); installPythonKernelRpcShims(pyodide); installPythonGuestProcessHardening(); @@ -2566,12 +2551,10 @@ try { prewarmOnly: false, startupMs: realPerformance.now() - startupStarted, loadPyodideMs, - packageLoadMs, - packageCount: preloadPackages.length, source, }); if (moduleName === 'pip') { - await runPythonPip(pyodide); + await runPythonPip(pyodide, isBundledPackage); } else if (moduleName) { pyodide.globals.set('__agentos_module', moduleName); try { diff --git a/crates/execution/src/node_import_cache.rs b/crates/execution/src/node_import_cache.rs index 0cebdb3ece..c8ace3dc5a 100644 --- a/crates/execution/src/node_import_cache.rs +++ b/crates/execution/src/node_import_cache.rs @@ -1,4 +1,4 @@ -use std::collections::BTreeSet; +use std::collections::{BTreeMap, BTreeSet}; use std::env; use std::fs; use std::io; @@ -11,6 +11,8 @@ use agentos_runtime::RuntimeContext; use tokio::sync::Mutex as AsyncMutex; use tokio::time; +use crate::python::pyodide_cache_path; + pub(crate) const NODE_IMPORT_CACHE_DEBUG_ENV: &str = "AGENTOS_NODE_IMPORT_CACHE_DEBUG"; pub(crate) const NODE_IMPORT_CACHE_METRICS_PREFIX: &str = "__AGENTOS_NODE_IMPORT_CACHE_METRICS__:"; pub(crate) const NODE_IMPORT_CACHE_ASSET_ROOT_ENV: &str = "AGENTOS_NODE_IMPORT_CACHE_ASSET_ROOT"; @@ -21,7 +23,7 @@ const NODE_IMPORT_CACHE_SCHEMA_VERSION: &str = "1"; const NODE_IMPORT_CACHE_LOADER_VERSION: &str = "8"; // Upstream reached 104 while the reactor branch independently changed bundled // assets; use a new generation so no stale materialization survives the merge. -const NODE_IMPORT_CACHE_ASSET_VERSION: &str = "106"; +const NODE_IMPORT_CACHE_ASSET_VERSION: &str = "111"; const NODE_IMPORT_CACHE_DIR_PREFIX: &str = "agentos-node-import-cache"; const DEFAULT_NODE_IMPORT_CACHE_MATERIALIZE_TIMEOUT: Duration = Duration::from_secs(30); const NODE_IMPORT_CACHE_BLOCKING_JOB_RESERVATION_BYTES: usize = 64 * 1024; @@ -2699,6 +2701,7 @@ impl NodeImportCacheMaterialization { fs::create_dir_all(self.asset_root.join("denied"))?; fs::create_dir_all(self.asset_root.join("polyfills"))?; fs::create_dir_all(&self.pyodide_dist_path)?; + fs::create_dir_all(pyodide_cache_path(&self.pyodide_dist_path))?; fs::create_dir_all(&self.prewarm_marker_dir)?; write_file_if_changed(&self.loader_path, &render_loader_source())?; @@ -2755,9 +2758,13 @@ impl NodeImportCacheMaterialization { &self.pyodide_dist_path.join("python_stdlib.zip"), BUNDLED_PYTHON_STDLIB_ZIP, )?; + // Pyodide loads a lockfile package from its package cache before it + // downloads it, so seeding the cache keeps bundled packages offline. + let pyodide_cache_path = pyodide_cache_path(&self.pyodide_dist_path); for asset in BUNDLED_PYODIDE_PACKAGE_ASSETS { check_materialization_cancelled(&cancelled)?; write_bytes_if_changed(&self.pyodide_dist_path.join(asset.file_name), asset.bytes)?; + write_bytes_if_changed(&pyodide_cache_path.join(asset.file_name), asset.bytes)?; } check_materialization_cancelled(&cancelled)?; write_file_if_changed( @@ -2768,6 +2775,30 @@ impl NodeImportCacheMaterialization { } } +/// Returns the sha256 that the bundled `pyodide-lock.json` pins for a package +/// file, or `None` when the file is not a lockfile package. +pub(crate) fn bundled_pyodide_file_sha256(file_name: &str) -> Option<&'static str> { + static DIGESTS: OnceLock> = OnceLock::new(); + DIGESTS + .get_or_init(|| { + let lockfile: serde_json::Value = serde_json::from_slice(BUNDLED_PYODIDE_LOCK) + .expect("bundled pyodide-lock.json is valid JSON"); + lockfile["packages"] + .as_object() + .into_iter() + .flatten() + .filter_map(|(_, entry)| { + Some(( + entry["file_name"].as_str()?.to_owned(), + entry["sha256"].as_str()?.to_owned(), + )) + }) + .collect() + }) + .get(file_name) + .map(String::as_str) +} + fn check_materialization_cancelled(cancelled: &AtomicBool) -> Result<(), io::Error> { if cancelled.load(Ordering::Acquire) { return Err(io::Error::new( @@ -3757,6 +3788,7 @@ mod tests { NODE_IMPORT_CACHE_TEST_MATERIALIZE_DELAY_MS, NODE_WASM_RUNNER_SOURCE, }; use crate::host_node::node_binary; + use crate::python::pyodide_cache_path; use serde_json::Value; use std::collections::BTreeSet; use std::fs; @@ -3801,7 +3833,12 @@ mod tests { .env("AGENTOS_PYODIDE_INDEX_URL", pyodide_index_url) .env( "AGENTOS_PYODIDE_PACKAGE_CACHE_DIR", - pyodide_index_url.join("pyodide-package-cache"), + // Bundled wheels are seeded into the real distribution's package cache. + if pyodide_index_url == import_cache.pyodide_dist_path() { + pyodide_cache_path(pyodide_index_url) + } else { + pyodide_index_url.join("pyodide-package-cache") + }, ) .env("AGENTOS_PYTHON_CODE", code); @@ -4265,115 +4302,6 @@ print(json.dumps({ ); } - #[test] - fn materialized_python_runner_preloads_bundled_packages_from_local_disk() { - assert_node_available(); - - let import_cache = NodeImportCache::default(); - import_cache - .ensure_materialized() - .expect("materialize node import cache"); - - let pyodide_dir = tempdir().expect("create pyodide fixture dir"); - write_fixture( - &pyodide_dir.path().join("pyodide.mjs"), - r#" -export async function loadPyodide(options) { - return { - setStdin(_stdin) {}, - async loadPackage(packages) { - options.stdout(`packages:${packages.join(',')}`); - options.stderr(`base:${options.packageBaseUrl}`); - }, - async runPythonAsync(code) { - options.stdout(`code:${code}`); - }, - }; -} -"#, - ); - write_fixture( - &pyodide_dir.path().join("pyodide-lock.json"), - "{\"packages\":[]}\n", - ); - - let output = run_python_runner_with_env( - &import_cache, - pyodide_dir.path(), - "print('hello')", - &[("AGENTOS_PYTHON_PRELOAD_PACKAGES", "[\"numpy\",\"pandas\"]")], - ); - let stdout = String::from_utf8_lossy(&output.stdout); - let stderr = String::from_utf8_lossy(&output.stderr); - let expected_package_base = format!( - "base:{}{}", - pyodide_dir.path().display(), - std::path::MAIN_SEPARATOR - ); - - assert_eq!(output.status.code(), Some(0)); - assert_eq!( - stdout, - "packages:micropip\npackages:numpy,pandas\ncode:print('hello')\n" - ); - assert!( - stderr.contains(&expected_package_base), - "expected local package base path in stderr, got: {stderr}" - ); - } - - #[test] - fn materialized_python_runner_rejects_unknown_preload_packages() { - assert_node_available(); - - let import_cache = NodeImportCache::default(); - import_cache - .ensure_materialized() - .expect("materialize node import cache"); - - let pyodide_dir = tempdir().expect("create pyodide fixture dir"); - write_fixture( - &pyodide_dir.path().join("pyodide.mjs"), - r#" -export async function loadPyodide() { - return { - setStdin(_stdin) {}, - async loadPackage() { - throw new Error('loadPackage should not be called'); - }, - async runPythonAsync(_code) {}, - }; -} -"#, - ); - write_fixture( - &pyodide_dir.path().join("pyodide-lock.json"), - "{\"packages\":[]}\n", - ); - - let output = run_python_runner_with_env( - &import_cache, - pyodide_dir.path(), - "print('hello')", - &[("AGENTOS_PYTHON_PRELOAD_PACKAGES", "[\"requests\"]")], - ); - let stderr = String::from_utf8_lossy(&output.stderr); - - assert_eq!(output.status.code(), Some(1)); - assert!( - stderr.contains("Unsupported bundled Python package \"requests\""), - "unexpected stderr: {stderr}" - ); - assert!( - stderr.contains("Available packages: numpy, pandas"), - "unexpected stderr: {stderr}" - ); - assert!( - !stderr.contains("loadPackage should not be called"), - "runner should validate packages before calling loadPackage: {stderr}" - ); - } - #[test] fn materialized_python_runner_streams_multiple_stdin_reads_through_pyodide() { assert_node_available(); diff --git a/crates/execution/src/python.rs b/crates/execution/src/python.rs index cb51d6d7f7..a0bdb8da29 100644 --- a/crates/execution/src/python.rs +++ b/crates/execution/src/python.rs @@ -4,7 +4,9 @@ use crate::javascript::{ JavascriptExecutionEngine, JavascriptExecutionError, JavascriptExecutionEvent, JavascriptExecutionLimits, JavascriptSyncRpcRequest, StartJavascriptExecutionRequest, }; -use crate::node_import_cache::{NodeImportCache, NODE_IMPORT_CACHE_ASSET_ROOT_ENV}; +use crate::node_import_cache::{ + bundled_pyodide_file_sha256, NodeImportCache, NODE_IMPORT_CACHE_ASSET_ROOT_ENV, +}; use crate::runtime_support::{ env_flag_enabled, file_fingerprint, resolve_execution_path, warmup_marker_path, NODE_DISABLE_COMPILE_CACHE_ENV, NODE_FROZEN_TIME_ENV, @@ -14,6 +16,7 @@ use agentos_runtime::RuntimeContext; use base64::Engine as _; use serde::{Deserialize, Serialize}; use serde_json::{json, Value}; +use sha2::{Digest, Sha256}; use std::collections::BTreeMap; use std::fmt; use std::fs; @@ -27,7 +30,6 @@ const NODE_ALLOW_PROCESS_BINDINGS_ENV: &str = "AGENTOS_ALLOW_PROCESS_BINDINGS"; const NODE_GUEST_PATH_MAPPINGS_ENV: &str = "AGENTOS_GUEST_PATH_MAPPINGS"; const NODE_SYNC_RPC_DATA_BYTES_ENV: &str = "AGENTOS_NODE_SYNC_RPC_DATA_BYTES"; const PYODIDE_INDEX_URL_ENV: &str = "AGENTOS_PYODIDE_INDEX_URL"; -const PYODIDE_PACKAGE_BASE_URL_ENV: &str = "AGENTOS_PYODIDE_PACKAGE_BASE_URL"; const PYODIDE_PACKAGE_CACHE_DIR_ENV: &str = "AGENTOS_PYODIDE_PACKAGE_CACHE_DIR"; const PYODIDE_GUEST_ROOT: &str = "/__agentos_pyodide"; const PYODIDE_CACHE_GUEST_ROOT: &str = "/__agentos_pyodide_cache"; @@ -116,7 +118,12 @@ pub struct PythonVfsRpcRequest { pub gid: Option, pub atime_ms: Option, pub mtime_ms: Option, + /// `Setattr` length: truncate or extend the file to this many bytes. + pub size: Option, pub content_base64: Option, + /// `Write` byte offset: write `content_base64` at this offset instead of + /// replacing the whole file. + pub offset: Option, pub recursive: bool, pub url: Option, pub http_method: Option, @@ -221,8 +228,12 @@ struct PythonVfsBridgeRequestWire { #[serde(default, rename = "mtimeMs")] mtime_ms: Option, #[serde(default)] + size: Option, + #[serde(default)] content_base64: Option, #[serde(default)] + offset: Option, + #[serde(default)] recursive: bool, #[serde(default)] url: Option, @@ -1548,14 +1559,6 @@ fn build_python_internal_env( PYODIDE_INDEX_URL_ENV.to_string(), String::from(PYODIDE_GUEST_ROOT), ); - internal_env.insert( - PYODIDE_PACKAGE_BASE_URL_ENV.to_string(), - request - .env - .get(PYODIDE_PACKAGE_BASE_URL_ENV) - .cloned() - .unwrap_or_else(|| String::from(PYODIDE_GUEST_ROOT)), - ); internal_env.insert( PYODIDE_PACKAGE_CACHE_DIR_ENV.to_string(), String::from(PYODIDE_CACHE_GUEST_ROOT), @@ -1627,7 +1630,7 @@ fn add_python_guest_path_mapping( internal_env.insert(String::from(NODE_GUEST_PATH_MAPPINGS_ENV), serialized); } -fn pyodide_cache_path(pyodide_dist_path: &Path) -> PathBuf { +pub(crate) fn pyodide_cache_path(pyodide_dist_path: &Path) -> PathBuf { let base = pyodide_dist_path .parent() .and_then(|parent| { @@ -1741,7 +1744,9 @@ fn parse_python_bridge_sync_rpc_request( gid: wire.gid.map(|value| value as u32), atime_ms: wire.atime_ms.map(|value| value as u64), mtime_ms: wire.mtime_ms.map(|value| value as u64), + size: wire.size.map(|value| value as u64), content_base64: wire.content_base64, + offset: wire.offset.map(|value| value as u64), recursive: wire.recursive, url: wire.url, http_method: wire.http_method, @@ -2372,6 +2377,39 @@ fn python_javascript_sync_rpc_action( })), } } + // The guest `fs.readFileSync(path)` polyfill reads in ranged chunks. + "fs.readFileRangeSync" => { + let (Some(offset), Some(length)) = ( + request.args.get(1).and_then(Value::as_u64), + request.args.get(2).and_then(Value::as_u64), + ) else { + return Ok(None); + }; + if length > PYTHON_SYNC_RPC_DATA_BYTES as u64 { + return Err(PythonExecutionError::RpcResponse(format!( + "managed fs.readFileRangeSync length {length} exceeds {PYTHON_SYNC_RPC_DATA_BYTES} bytes" + ))); + } + let mut file = match fs::File::open(&host_path) { + Ok(file) => file, + Err(error) => { + return python_sync_rpc_fs_action_error(path, "open", error).map(Some); + } + }; + file.seek(SeekFrom::Start(offset)) + .map_err(PythonExecutionError::PrepareRuntime)?; + let mut bytes = Vec::new(); + file.take(length) + .read_to_end(&mut bytes) + .map_err(PythonExecutionError::PrepareRuntime)?; + if request.raw_bytes_args.contains_key(&usize::MAX) { + PythonJavascriptSyncRpcAction::RawSuccess(bytes) + } else { + PythonJavascriptSyncRpcAction::Success(Value::String( + v8_runtime::base64_encode_pub(&bytes), + )) + } + } "fs.statSync" | "fs.promises.stat" => match fs::metadata(&host_path) { Ok(metadata) => { PythonJavascriptSyncRpcAction::Success(python_host_stat_value(&metadata)) @@ -2414,6 +2452,24 @@ fn python_javascript_sync_rpc_action( } "fs.writeFileSync" | "fs.promises.writeFile" => { let contents = python_sync_rpc_bytes_arg(&request.args, 1)?; + // Pyodide stores each downloaded lockfile package in its package + // cache. Only bytes that match the sha256 pinned by the bundled + // pyodide-lock.json may enter the cache. + if matches!(path_kind.kind, PythonManagedPathKind::GuestCache) { + let file_name = host_path + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or(""); + let digest = format!("{:x}", Sha256::digest(&contents)); + if bundled_pyodide_file_sha256(file_name) != Some(digest.as_str()) { + return Ok(Some(PythonJavascriptSyncRpcAction::Error { + code: "EIO", + message: format!( + "EIO: {file_name} does not match the sha256 pinned in pyodide-lock.json, write '{path}'" + ), + })); + } + } if let Some(parent) = host_path.parent() { fs::create_dir_all(parent).map_err(PythonExecutionError::PrepareRuntime)?; } diff --git a/crates/execution/tests/python.rs b/crates/execution/tests/python.rs index 5c19eba79d..036f3bd6db 100644 --- a/crates/execution/tests/python.rs +++ b/crates/execution/tests/python.rs @@ -32,8 +32,6 @@ struct PythonStartupMetrics { prewarm_only: bool, startup_ms: f64, load_pyodide_ms: f64, - package_load_ms: f64, - package_count: usize, source: String, } @@ -86,16 +84,10 @@ fn parse_startup_metrics(stderr: &str) -> PythonStartupMetrics { prewarm_only: parse_boolean_metric(metrics_line, "prewarmOnly"), startup_ms: parse_float_metric(metrics_line, "startupMs"), load_pyodide_ms: parse_float_metric(metrics_line, "loadPyodideMs"), - package_load_ms: parse_float_metric(metrics_line, "packageLoadMs"), - package_count: parse_metric_value(metrics_line, "packageCount"), source: parse_string_metric(metrics_line, "source"), } } -fn parse_metric_value(metrics_line: &str, key: &str) -> usize { - parse_float_metric(metrics_line, key) as usize -} - fn parse_float_metric(metrics_line: &str, key: &str) -> f64 { let marker = format!("\"{key}\":"); let start = metrics_line.find(&marker).expect("metric key") + marker.len(); @@ -455,8 +447,6 @@ export async function loadPyodide() { assert!(!first_startup.prewarm_only); assert!(first_startup.startup_ms > 0.0); assert!(first_startup.load_pyodide_ms > 0.0); - assert_eq!(first_startup.package_load_ms, 0.0); - assert_eq!(first_startup.package_count, 0); assert_eq!(first_startup.source, "inline"); let (_second_stdout, second_stderr, second_exit_code) = run_python_execution( diff --git a/crates/native-sidecar/src/filesystem.rs b/crates/native-sidecar/src/filesystem.rs index cba1d96787..42f24d3908 100644 --- a/crates/native-sidecar/src/filesystem.rs +++ b/crates/native-sidecar/src/filesystem.rs @@ -936,10 +936,12 @@ where path )) })?; - vm.kernel - .write_file(&path, bytes) - .map(|()| PythonVfsRpcResponsePayload::Empty) - .map_err(kernel_error) + match request.offset { + Some(offset) => vm.kernel.pwrite_file(&path, offset, bytes), + None => vm.kernel.write_file(&path, bytes), + } + .map(|()| PythonVfsRpcResponsePayload::Empty) + .map_err(kernel_error) } PythonVfsRpcMethod::Stat => vm .kernel @@ -1040,8 +1042,8 @@ where .read_link(&path) .map(|target| PythonVfsRpcResponsePayload::SymlinkTarget { target }) .map_err(kernel_error), - // `setattr` carries any of mode/uid/gid/atime+mtime; apply each - // present field to the host VFS. + // `setattr` carries any of size/mode/uid/gid/atime+mtime; apply + // each present field to the host VFS. PythonVfsRpcMethod::Setattr => { (|| -> Result { // Mirror metadata into the host shadow only when the entry @@ -1051,6 +1053,9 @@ where // guest file (that resurrected empty content). let mirror = !is_non_root_mount_path(&vm.kernel, &path) && shadow_host_path_for_guest(&vm.cwd, &path).exists(); + if let Some(size) = request.size { + vm.kernel.truncate(&path, size).map_err(kernel_error)?; + } if let Some(mode) = request.mode { vm.kernel.chmod(&path, mode).map_err(kernel_error)?; if mirror { diff --git a/crates/native-sidecar/src/stdio.rs b/crates/native-sidecar/src/stdio.rs index 9a35fcd10a..edd6a98267 100644 --- a/crates/native-sidecar/src/stdio.rs +++ b/crates/native-sidecar/src/stdio.rs @@ -6379,7 +6379,9 @@ export async function loadPyodide() { gid: None, atime_ms: None, mtime_ms: None, + size: None, content_base64: Some(content_base64.to_owned()), + offset: None, recursive: false, url: None, http_method: None, diff --git a/crates/native-sidecar/tests/python.rs b/crates/native-sidecar/tests/python.rs index cdfef82092..5a7a34ccdf 100644 --- a/crates/native-sidecar/tests/python.rs +++ b/crates/native-sidecar/tests/python.rs @@ -1519,6 +1519,96 @@ print(json.dumps(results)) ); } +fn python_runtime_writes_vfs_files_in_place() { + assert_node_available(); + let mut sidecar = new_sidecar("python-vfs-writes"); + let cwd = temp_dir("python-vfs-writes-cwd"); + let connection_id = authenticate_wire(&mut sidecar, "conn-python"); + let session_id = open_session_wire(&mut sidecar, 2, &connection_id); + let vm_id = create_vm_with_root_filesystem( + &mut sidecar, + 3, + &connection_id, + &session_id, + GuestRuntimeKind::Python, + &cwd, + RootFilesystemDescriptor { + mode: RootFilesystemMode::Ephemeral, + disable_default_base_layer: false, + lowers: Vec::new(), + bootstrap_entries: vec![root_file( + "/tmp/seed.txt", + "hello world", + Some(RootFilesystemEntryEncoding::Utf8), + )], + }, + ); + + execute_inline_python( + &mut sidecar, + 4, + &connection_id, + &session_id, + &vm_id, + "proc-python-vfs-writes", + r#" +import os + +with open("/tmp/seed.txt", "r+") as handle: + handle.seek(6) + handle.write("WORLD") + +path = "/tmp/data.txt" +with open(path, "w") as handle: + for _ in range(10): + handle.write("0123456789" * 1000) + handle.flush() +with open(path, "r+") as handle: + handle.seek(5) + handle.write("XYZ") +with open(path, "a") as handle: + handle.write("END") +os.truncate(path, 20) +with open(path, "a") as handle: + handle.write("!") +"#, + ); + let (stdout, stderr, exit_code) = collect_process_output_with_timeout( + &mut sidecar, + &connection_id, + &session_id, + &vm_id, + "proc-python-vfs-writes", + Duration::from_secs(30), + ); + assert_eq!(exit_code, 0, "stdout: {stdout}\nstderr: {stderr}"); + + // A separate process starts with a fresh in-memory FS, so it reads what + // reached the VM filesystem. + execute_inline_python( + &mut sidecar, + 5, + &connection_id, + &session_id, + &vm_id, + "proc-python-vfs-reread", + "print(open('/tmp/seed.txt').read())\nprint(open('/tmp/data.txt').read())", + ); + let (stdout, stderr, exit_code) = collect_process_output_with_timeout( + &mut sidecar, + &connection_id, + &session_id, + &vm_id, + "proc-python-vfs-reread", + Duration::from_secs(30), + ); + assert_eq!(exit_code, 0, "stdout: {stdout}\nstderr: {stderr}"); + assert_eq!( + stdout, "hello WORLD\n01234XYZ890123456789!\n", + "stderr: {stderr}" + ); +} + fn python_runtime_supports_raw_tcp_and_udp_sockets() { assert_node_available(); @@ -2638,11 +2728,11 @@ sys.stdin.read() ); } -fn python_runtime_imports_bundled_numpy_without_network() { +fn python_runtime_imports_bundled_packages_on_first_import() { assert_node_available(); - let mut sidecar = new_sidecar("python-numpy-package"); - let cwd = temp_dir("python-numpy-package-cwd"); + let mut sidecar = new_sidecar("python-bundled-packages"); + let cwd = temp_dir("python-bundled-packages-cwd"); let connection_id = authenticate_wire(&mut sidecar, "conn-python"); let session_id = open_session_wire(&mut sidecar, 2, &connection_id); let (vm_id, _) = create_vm_wire( @@ -2654,68 +2744,14 @@ fn python_runtime_imports_bundled_numpy_without_network() { &cwd, ); - execute_inline_python_with_env( - &mut sidecar, - 4, - &connection_id, - &session_id, - &vm_id, - "proc-python-numpy", - "import numpy\nprint(numpy.__version__)", - HashMap::from([( - String::from("AGENTOS_PYTHON_PRELOAD_PACKAGES"), - String::from("[\"numpy\"]"), - )]), - ); - - let (stdout, stderr, exit_code) = collect_process_output_with_timeout( - &mut sidecar, - &connection_id, - &session_id, - &vm_id, - "proc-python-numpy", - Duration::from_secs(30), - ); - - assert_eq!(exit_code, 0); - assert!( - stderr.is_empty(), - "unexpected stderr from bundled numpy import: {stderr}" - ); - assert!( - stdout.lines().any(|line| line.trim() == "2.2.5"), - "expected numpy version in stdout, got: {stdout}" - ); -} - -fn python_runtime_imports_bundled_pandas_without_network() { - assert_node_available(); - - let mut sidecar = new_sidecar("python-pandas-package"); - let cwd = temp_dir("python-pandas-package-cwd"); - let connection_id = authenticate_wire(&mut sidecar, "conn-python"); - let session_id = open_session_wire(&mut sidecar, 2, &connection_id); - let (vm_id, _) = create_vm_wire( - &mut sidecar, - 3, - &connection_id, - &session_id, - GuestRuntimeKind::Python, - &cwd, - ); - - execute_inline_python_with_env( + execute_inline_python( &mut sidecar, 4, &connection_id, &session_id, &vm_id, - "proc-python-pandas", - "import pandas\nprint(pandas.__version__)", - HashMap::from([( - String::from("AGENTOS_PYTHON_PRELOAD_PACKAGES"), - String::from("[\"pandas\"]"), - )]), + "proc-python-bundled-packages", + "import pandas\nimport numpy\nprint(pandas.__version__, numpy.__version__)\nprint(pandas.DataFrame({'a': [1, 2]})['a'].sum())", ); let (stdout, stderr, exit_code) = collect_process_output_with_timeout( @@ -2723,19 +2759,16 @@ fn python_runtime_imports_bundled_pandas_without_network() { &connection_id, &session_id, &vm_id, - "proc-python-pandas", + "proc-python-bundled-packages", Duration::from_secs(30), ); - assert_eq!(exit_code, 0); + assert_eq!(exit_code, 0, "stderr: {stderr}"); assert!( stderr.is_empty(), - "unexpected stderr from bundled pandas import: {stderr}" - ); - assert!( - stdout.lines().any(|line| line.trim() == "2.3.3"), - "expected pandas version in stdout, got: {stdout}" + "unexpected stderr from bundled package import: {stderr}" ); + assert_eq!(stdout.trim(), "2.3.3 2.2.5\n3"); } fn python_runtime_supports_micropip_package_installation() { @@ -3383,6 +3416,110 @@ fn python_command_pip_installs_via_micropip() { ); } +fn python_command_pip_rejects_a_pyodide_wheel_that_does_not_match_the_lockfile() { + assert_node_available(); + + // The mirror serves a Pillow wheel whose bytes do not match pyodide-lock.json. + let mirror = temp_dir("python-cli-pip-tampered-mirror"); + fs::write( + mirror.join("pillow-11.3.0-cp313-cp313-pyodide_2025_0_wasm32.whl"), + b"not the wheel pinned by the lockfile", + ) + .expect("write tampered wheel"); + let (port, server) = spawn_static_file_server(mirror); + let mut sidecar = new_sidecar("python-cli-pip-missing"); + let cwd = temp_dir("python-cli-pip-missing-cwd"); + let connection_id = authenticate_wire(&mut sidecar, "conn-python"); + let session_id = open_session_wire(&mut sidecar, 2, &connection_id); + let vm_id = create_vm_with_metadata_and_permissions( + &mut sidecar, + 3, + &connection_id, + &session_id, + GuestRuntimeKind::Python, + &cwd, + HashMap::from([( + String::from("env.AGENTOS_LOOPBACK_EXEMPT_PORTS"), + serde_json::to_string(&vec![port.to_string()]).expect("serialize exempt ports"), + )]), + wire_permissions_allow_all(), + ); + + execute_python_cli_with_env( + &mut sidecar, + 4, + &connection_id, + &session_id, + &vm_id, + "proc-py-pip-missing", + "pip", + &["install", "pillow"], + HashMap::from([( + String::from("AGENTOS_PYODIDE_PACKAGE_BASE_URL"), + format!("http://127.0.0.1:{port}/"), + )]), + ); + + let (stdout, stderr, exit_code) = collect_process_output_with_timeout( + &mut sidecar, + &connection_id, + &session_id, + &vm_id, + "proc-py-pip-missing", + Duration::from_secs(90), + ); + let _ = server.join(); + assert_ne!(exit_code, 0, "stdout: {stdout}\nstderr: {stderr}"); + assert!( + !stdout.contains("Successfully installed"), + "stdout: {stdout}\nstderr: {stderr}" + ); + assert!( + stderr.contains("does not match the sha256 pinned in pyodide-lock.json"), + "expected a sha256 mismatch in stderr, got: {stderr}" + ); +} + +fn python_command_pip_installs_bundled_pandas_offline() { + assert_node_available(); + let mut sidecar = new_sidecar("python-cli-pip-bundled"); + let cwd = temp_dir("python-cli-pip-bundled-cwd"); + let connection_id = authenticate_wire(&mut sidecar, "conn-python"); + let session_id = open_session_wire(&mut sidecar, 2, &connection_id); + let (vm_id, _) = create_vm_wire( + &mut sidecar, + 3, + &connection_id, + &session_id, + GuestRuntimeKind::Python, + &cwd, + ); + + execute_python_cli( + &mut sidecar, + 4, + &connection_id, + &session_id, + &vm_id, + "proc-py-pip-pandas", + "pip", + &["install", "pandas"], + ); + let (stdout, stderr, exit_code) = collect_process_output_with_timeout( + &mut sidecar, + &connection_id, + &session_id, + &vm_id, + "proc-py-pip-pandas", + Duration::from_secs(60), + ); + assert_eq!(exit_code, 0, "stdout: {stdout}\nstderr: {stderr}"); + assert!( + stdout.contains("Successfully installed pandas"), + "stdout: {stdout}\nstderr: {stderr}" + ); +} + fn python_command_runs_inline_code() { assert_node_available(); let mut sidecar = new_sidecar("python-cli-inline"); @@ -3889,6 +4026,70 @@ fn python_pip_installs_persist_across_invocations() { ); } +#[test] +#[ignore = "needs network: downloads Pillow from the Pyodide CDN"] +fn python_pip_installs_a_pyodide_package_from_the_cdn() { + assert_node_available(); + let mut sidecar = new_sidecar("python-cdn-pip"); + let cwd = temp_dir("python-cdn-pip-cwd"); + let connection_id = authenticate_wire(&mut sidecar, "conn-python"); + let session_id = open_session_wire(&mut sidecar, 2, &connection_id); + let vm_id = create_vm_with_metadata_and_permissions( + &mut sidecar, + 3, + &connection_id, + &session_id, + GuestRuntimeKind::Python, + &cwd, + HashMap::new(), + wire_permissions_allow_all(), + ); + + execute_python_cli( + &mut sidecar, + 4, + &connection_id, + &session_id, + &vm_id, + "proc-cdn-pip-install", + "pip", + &["install", "pillow"], + ); + let (stdout1, stderr1, exit1) = collect_process_output_with_timeout( + &mut sidecar, + &connection_id, + &session_id, + &vm_id, + "proc-cdn-pip-install", + Duration::from_secs(120), + ); + assert_eq!(exit1, 0, "stdout: {stdout1}\nstderr: {stderr1}"); + + execute_python_cli( + &mut sidecar, + 5, + &connection_id, + &session_id, + &vm_id, + "proc-cdn-pip-import", + "python", + &[ + "-c", + "from PIL import Image; print(Image.new('RGB', (3, 2)).size)", + ], + ); + let (stdout2, stderr2, exit2) = collect_process_output_with_timeout( + &mut sidecar, + &connection_id, + &session_id, + &vm_id, + "proc-cdn-pip-import", + Duration::from_secs(60), + ); + assert_eq!(exit2, 0, "stdout: {stdout2}\nstderr: {stderr2}"); + assert_eq!(stdout2.trim(), "(3, 2)", "stderr: {stderr2}"); +} + fn python_rootfs_suite() { python_reads_and_writes_arbitrary_vm_paths(); python_pip_installs_persist_across_invocations(); @@ -3902,6 +4103,8 @@ fn python_cli_suite() { python_command_runs_interactive_repl(); python_command_runs_as_nested_child_process(); python_command_pip_installs_via_micropip(); + python_command_pip_rejects_a_pyodide_wheel_that_does_not_match_the_lockfile(); + python_command_pip_installs_bundled_pandas_offline(); } #[test] @@ -3928,6 +4131,7 @@ fn python_suite() { concurrent_python_processes_stay_isolated_across_vms(); python_runtime_mounts_workspace_over_the_kernel_vfs(); python_runtime_supports_file_delete_and_rename(); + python_runtime_writes_vfs_files_in_place(); python_runtime_supports_raw_tcp_and_udp_sockets(); python_runtime_supports_symlink_readlink_and_metadata(); workspace_files_are_shared_between_javascript_and_python_runtimes(); @@ -3938,8 +4142,7 @@ fn python_suite() { python_runtime_supports_interactive_input_prompts_and_multiple_streaming_writes(); python_runtime_close_stdin_triggers_input_eof_and_empty_read(); python_runtime_kill_process_terminates_blocked_stdin_reads(); - python_runtime_imports_bundled_numpy_without_network(); - python_runtime_imports_bundled_pandas_without_network(); + python_runtime_imports_bundled_packages_on_first_import(); python_runtime_supports_micropip_package_installation(); python_runtime_micropip_install_respects_network_permissions(); python_runtime_routes_dns_and_http_through_sidecar_bridge(); @@ -3984,6 +4187,7 @@ mod python_split { concurrent_python_processes_stay_isolated_across_vms, python_runtime_mounts_workspace_over_the_kernel_vfs, python_runtime_supports_file_delete_and_rename, + python_runtime_writes_vfs_files_in_place, python_runtime_supports_raw_tcp_and_udp_sockets, python_runtime_supports_symlink_readlink_and_metadata, workspace_files_are_shared_between_javascript_and_python_runtimes, @@ -3994,8 +4198,7 @@ mod python_split { python_runtime_supports_interactive_input_prompts_and_multiple_streaming_writes, python_runtime_close_stdin_triggers_input_eof_and_empty_read, python_runtime_kill_process_terminates_blocked_stdin_reads, - python_runtime_imports_bundled_numpy_without_network, - python_runtime_imports_bundled_pandas_without_network, + python_runtime_imports_bundled_packages_on_first_import, python_runtime_routes_dns_and_http_through_sidecar_bridge, python_runtime_routes_requests_through_sidecar_bridge, python_runtime_surfaces_network_permission_errors, @@ -4008,6 +4211,8 @@ mod python_split { python_command_runs_interactive_repl, python_command_runs_as_nested_child_process, python_command_pip_installs_via_micropip, + python_command_pip_rejects_a_pyodide_wheel_that_does_not_match_the_lockfile, + python_command_pip_installs_bundled_pandas_offline, python_reads_and_writes_arbitrary_vm_paths, python_pip_installs_persist_across_invocations, ); diff --git a/crates/native-sidecar/tests/service.rs b/crates/native-sidecar/tests/service.rs index c1a64b22e9..4ea1fe2f31 100644 --- a/crates/native-sidecar/tests/service.rs +++ b/crates/native-sidecar/tests/service.rs @@ -8200,7 +8200,9 @@ console.log(JSON.stringify({ status: "ok", summary })); gid: None, atime_ms: None, mtime_ms: None, + size: None, content_base64: None, + offset: None, recursive: false, url: None, http_method: None, @@ -8249,7 +8251,9 @@ console.log(JSON.stringify({ status: "ok", summary })); gid: None, atime_ms: None, mtime_ms: None, + size: None, content_base64: None, + offset: None, recursive: false, url: None, http_method: None, @@ -15446,7 +15450,9 @@ export async function loadPyodide() { gid: None, atime_ms: None, mtime_ms: None, + size: None, content_base64: None, + offset: None, recursive: false, url: None, http_method: None, @@ -15486,7 +15492,9 @@ export async function loadPyodide() { gid: None, atime_ms: None, mtime_ms: None, + size: None, content_base64: Some(String::from("aGVsbG8gZnJvbSBzaWRlY2FyIHJwYw==")), + offset: None, recursive: false, url: None, http_method: None, @@ -15538,7 +15546,9 @@ export async function loadPyodide() { gid: None, atime_ms: None, mtime_ms: None, + size: None, content_base64: None, + offset: None, recursive: false, url: None, http_method: None, diff --git a/docs/content/docs/python.mdx b/docs/content/docs/python.mdx index 7a4f0cbc17..b2a5f56715 100644 --- a/docs/content/docs/python.mdx +++ b/docs/content/docs/python.mdx @@ -70,6 +70,12 @@ Installs modify the VM-wide filesystem, so a package installed once is importabl by every later execution in that VM, in any language. Only one npm/Python mutation runs at a time per VM; a concurrent install fails with `execution_busy`. +pandas, numpy, python-dateutil, pytz, six, and click are bundled. They import +without an install step and need no network. Other packages download over the +network, so the VM needs a [network grant](/agentos/docs/permissions) for their +host: Pyodide builds of compiled packages such as Pillow come from +`cdn.jsdelivr.net`, and pure-Python wheels come from PyPI. + ## Background processes and web servers `spawn*` starts a long-lived process and returns a `pid`. From there you get