From 7b498195dc0bc5d87d9da8b6179e168f93185e13 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 7 Oct 2026 12:10:33 -0700 Subject: [PATCH 1/2] Document the commands provided by `Platforms/WASI/__main__.py` Along the way, fix some bugs and add a `path` command to make is easier to programmatically figure out where things are. The README should also be self-contained enough that one could point an LLM at it for working on WASI. --- Platforms/WASI/README.md | 223 ++++++++++++++++++++++++++++++++++++- Platforms/WASI/__main__.py | 24 ++++ Platforms/WASI/_build.py | 8 +- Platforms/WASI/_package.py | 22 +--- Platforms/WASI/_shared.py | 24 +++- 5 files changed, 278 insertions(+), 23 deletions(-) diff --git a/Platforms/WASI/README.md b/Platforms/WASI/README.md index 62c82924d437b15..f6045b458727b23 100644 --- a/Platforms/WASI/README.md +++ b/Platforms/WASI/README.md @@ -9,9 +9,227 @@ use WASM runtimes such as [wasmtime](https://wasmtime.dev/). **NOTE**: If you are looking for general information about WebAssembly that is not directly related to CPython, please see https://github.com/psf/webassembly. -## Build -See [the devguide on how to build and run for WASI](https://devguide.python.org/getting-started/setup-building/#wasi). +## Working on the WASI build + +This directory provides a CLI for building and packaging WASI builds for +distribution. + +Run all commands below from the root of the CPython source checkout. + +To see all the available commands, run: + +```shell +python3 Platforms/WASI --help +``` + +The `python3` interpreter used to run this CLI must be a Python version that +is receiving bugfixes. This requirement does not apply to the build Python, +which the CLI builds from the source checkout. + + +### Prerequisites + +There are some tools that must be available to successfully build. + +1. C compiler +2. `make` +3. WASI SDK +4. Wasmtime (or some other WASI runtime configured via `--host-runner`) + +The default runner requires Wasmtime be on `PATH`. + +The WASI SDK must be the same version as specified in config.toml. The search +for the WASI SDK is done via: + +1. `--wasi-sdk` CLI option +2. `WASI_SDK_PATH` environment variable +3. `/opt` where the WASI SDK has been unpacked from its tarball + +In the end, you will end up with a "build Python" which is a local build of +Python used for cross-builds. You will also have the WASI build. + + +### Development loop + +The common way to get started is to first do a full build: + +```shell +python3 Platforms/WASI build --quiet --logdir cross-build/logs -- --with-pydebug --config-cache +``` + +Once you have the build you can run the test you want. + +Bash: +```bash +"$(python3 Platforms/WASI path)/python.sh" -m test test_os +``` + +Fish: +```fish +set -l wasi_dir (python3 Platforms/WASI path); "$wasi_dir/python.sh" -m test test_os +``` + +If you are working on C code and need a rebuild: + +```shell +python3 Platforms/WASI make-host +``` + + +### Building + +In general, +[the devguide covers how to build and run for WASI](https://devguide.python.org/getting-started/setup-building/#wasi), +but we will cover some of the details here. + +The simplest way to get a pydebug WASI build is: + +```shell +python3 Platforms/WASI build -- --with-pydebug +``` + +This builds the build Python and the WASI build with `--with-pydebug` passed to +`configure` (as is anything that comes after `--`). The builds are placed in +the `cross-build/` directory of the source checkout, each in a subdirectory +matching the compiler triple for the build. + +You can do the two builds separately if you want: + +```shell +python3 Platforms/WASI build-python -- --with-pydebug +python3 Platforms/WASI build-host +``` + +This can be broken down even more to the separate `configure` and `make` steps: + +```shell +python3 Platforms/WASI configure-build-python -- --with-pydebug +python3 Platforms/WASI make-build-python +python3 Platforms/WASI configure-host +python3 Platforms/WASI make-host +``` + +Note that `configure-host` figures out to do a pydebug build by looking at the +build Python. + +There is a `--quiet` flag to redirect output from the underlying commands to a +directory. The `--logdir` flag controls where the log files go (which defaults +to `/tmp`). + +```shell +python3 Platforms/WASI build --quiet --logdir cross-build/logs +``` + + +### Packaging + +The `package` command is used to gather all the files necessary to make a +release and place them in an archive: + +```shell +python3 Platforms/WASI package +``` + +The files are gathered into a versioned directory inside `dist/` in the source +checkout. An archive containing that directory is placed alongside it in +`dist/`. The gathered directory includes a `bin/python3.wasmtime` file to ease +launching the interpreter. + +If you just need to gather the files for a release, you can use the `gather` +command: + +```shell +python3 Platforms/WASI gather +``` + +Both `package` and `gather` require a completed build and delete the entire +existing `dist/` directory before gathering files. + +There is no command just to archive the gathered files. + + +### Paths + +The `path` command prints the location of the build Python, WASI build, or +gathered distribution files: + +```shell +python3 Platforms/WASI path build-python +python3 Platforms/WASI path wasi +python3 Platforms/WASI path dist +``` + +With no location argument, `path` defaults to `wasi`. The `dist` location +returns the versioned distribution directory inside `dist/`, not the top-level +`dist/` directory. The `build-python` and `wasi` locations can be queried before +building; the `dist` location requires WASI build metadata to determine the +versioned directory name. + + +### Cleanup + +The `clean` command deletes the entire `cross-build/` and `dist/` directories, +including all build files, gathered distribution files, and archives: + +```shell +python3 Platforms/WASI clean +``` + + +### Testing + +To find out where the WASI build directory is, you can run: + +```shell +python3 Platforms/WASI path +``` + +From there, you can run the test suite. + +For Bash-like shells: + +```bash +make buildbottest -C "$(python3 Platforms/WASI path)" +``` + +For fish: + +```fish +make buildbottest -C (python3 Platforms/WASI path) +``` + +There is a `python.sh` file in the WASI build directory, so you can also run +tests that way. + +Bash: + +```bash +"$(python3 Platforms/WASI path)/python.sh" -m test +``` + +Fish: + +```fish +set -l wasi_dir (python3 Platforms/WASI path); "$wasi_dir/python.sh" -m test +``` + +If you want to test the files meant for distribution, the directory containing +the files can be found via `python3 Platforms/WASI path dist` and there is a +`bin/python3.wasmtime` shell script. + +Bash: + +```bash +"$(python3 Platforms/WASI path dist)/bin/python3.wasmtime" -m test +``` + +Fish: + +```fish +set -l dist_dir (python3 Platforms/WASI path dist); "$dist_dir/bin/python3.wasmtime" -m test +``` + ## Detecting WASI builds @@ -48,6 +266,7 @@ posix.uname_result( 'wasi' ``` + ### C code WASI SDK defines several built-in macros. You can dump a full list of built-ins diff --git a/Platforms/WASI/__main__.py b/Platforms/WASI/__main__.py index 58cbc44834dd284..dc1df16f829efea 100644 --- a/Platforms/WASI/__main__.py +++ b/Platforms/WASI/__main__.py @@ -68,6 +68,19 @@ def main(): package = subcommands.add_parser( "package", help="Package the host/WASI Python into an archive" ) + gather = subcommands.add_parser( + "gather", help="Gather all the files for distribution" + ) + path = subcommands.add_parser( + "path", help="Print the path to a build or distribution directory" + ) + path.add_argument( + "location", + nargs="?", + choices=("build-python", "wasi", "dist"), + default="wasi", + help="Directory whose path to print", + ) subcommands.add_parser( "clean", help="Delete files and directories created by this script" ) @@ -144,7 +157,9 @@ def main(): make_host, build_host, pythoninfo_host, + gather, package, + path, ): subcommand.add_argument( "--host-triple", @@ -191,9 +206,18 @@ def main(): _build.pythoninfo_wasi_python(context) case "clean": _build.clean_contents(context) + case "gather": + _package.gather(context) case "package": _package.gather(context) _package.archive(context) + case "path": + paths = { + "build-python": "build_python_path", + "wasi": "wasi_build_path", + "dist": "archive_dir", + } + print(getattr(context, paths[context.location])) case None: parser.print_help() case _: diff --git a/Platforms/WASI/_build.py b/Platforms/WASI/_build.py index f73f112bdf9eb17..8d9cc3c1bf281f5 100644 --- a/Platforms/WASI/_build.py +++ b/Platforms/WASI/_build.py @@ -113,6 +113,7 @@ def call(command, *, context=None, quiet=False, **kwargs): else: if (log_path := getattr(context, "log_path", None)) is None: log_path = pathlib.Path(tempfile.gettempdir()) + log_path.mkdir(parents=True, exist_ok=True) stdout = tempfile.NamedTemporaryFile( "w", encoding="utf-8", @@ -279,9 +280,10 @@ def make_wasi_python(context, working_dir): def clean_contents(context): """Delete all files created by this script.""" context.clean = True - if context.cross_build_path.exists(): - _shared.log("๐Ÿงน", f"Deleting {context.cross_build_path} ...") - shutil.rmtree(context.cross_build_path) + for path in [context.cross_build_path, context.dist_path]: + if path.exists(): + _shared.log("๐Ÿงน", f"Deleting {path} ...") + shutil.rmtree(path) @subdir("build_python_path") diff --git a/Platforms/WASI/_package.py b/Platforms/WASI/_package.py index d1d43e5da2843c1..862ffb30130d048 100644 --- a/Platforms/WASI/_package.py +++ b/Platforms/WASI/_package.py @@ -264,18 +264,6 @@ def config_symlink(config_path, context): return [(symlink, config_path) for symlink in symlinks] -def filename_stem(context): - """Calculate the stem of the archive file name.""" - version_info = context.wasi_build_details["language"]["version_info"] - version = f"python-{version_info['major']}.{version_info['minor']}.{version_info['micro']}" - if version_info["releaselevel"] != "final": - version += version_info["releaselevel"][0] + str( - version_info["serial"] - ) - - return f"{version}-{context.host_triple}" - - def copy_files(files, base): for dest, src in files: target = base / dest @@ -296,13 +284,13 @@ def gather(context): py_version = python_version(context) py_d_version = python_version(context, debug_ok=True) - dist = context.checkout / "dist" + dist = context.dist_path if dist.exists(): _shared.log("๐Ÿงน", f"Deleting {dist} ...") shutil.rmtree(dist) indent = " " - base = dist / filename_stem(context) + base = context.archive_dir _shared.log("๐Ÿ“", f"Copying files to {base} ...") _shared.log("๐Ÿ“", "bin/", spacing=indent * 2) @@ -365,12 +353,12 @@ def gather(context): def archive(context): - file_name = f"{filename_stem(context)}.tar.xz" - file_path = context.checkout / "dist" / file_name + file_name = f"{context.archive_stem}.tar.xz" + file_path = context.dist_path / file_name if file_path.exists(): _shared.log("๐Ÿงน", f"Deleting {file_path} ...") file_path.unlink() - to_compress = context.checkout / "dist" / filename_stem(context) + to_compress = context.archive_dir _shared.log("๐Ÿ—œ๏ธ", f"Archiving to {file_path} ...") mtime_format = "%Y-%m-%dT%H:%M:%SZ" if source_date_epoch := os.environ.get("SOURCE_DATE_EPOCH"): diff --git a/Platforms/WASI/_shared.py b/Platforms/WASI/_shared.py index 6c4cb156739d511..306aabd1f1c0977 100644 --- a/Platforms/WASI/_shared.py +++ b/Platforms/WASI/_shared.py @@ -24,6 +24,7 @@ class Context: def __init__(self): self.here = pathlib.Path(__file__).parent + self.orig_cwd = pathlib.Path.cwd() @functools.cached_property def checkout(self): @@ -180,10 +181,31 @@ def wasi_sdk_path(self): @functools.cached_property def log_path(self): if self._log_path is not None: - return self._log_path + if not (path := self._log_path).is_absolute(): + path = (self.orig_cwd / self._log_path).resolve() + return path return pathlib.Path(tempfile.gettempdir()) + @functools.cached_property + def dist_path(self): + return self.checkout / "dist" + + @functools.cached_property + def archive_stem(self): + version_info = self.wasi_build_details["language"]["version_info"] + version = f"python-{version_info['major']}.{version_info['minor']}.{version_info['micro']}" + if version_info["releaselevel"] != "final": + version += version_info["releaselevel"][0] + str( + version_info["serial"] + ) + + return f"{version}-{self.host_triple}" + + @functools.cached_property + def archive_dir(self): + return self.dist_path / self.archive_stem + def log(emoji, message, *, spacing=None): """Print a notification with an emoji. From 0407793e6fba964cfb4b1145d92b8ff9aecf7aa6 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 7 Oct 2026 12:21:29 -0700 Subject: [PATCH 2/2] Mention the WASI dev container image --- Platforms/WASI/README.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/Platforms/WASI/README.md b/Platforms/WASI/README.md index f6045b458727b23..a62efbda5366544 100644 --- a/Platforms/WASI/README.md +++ b/Platforms/WASI/README.md @@ -46,9 +46,16 @@ for the WASI SDK is done via: 2. `WASI_SDK_PATH` environment variable 3. `/opt` where the WASI SDK has been unpacked from its tarball -In the end, you will end up with a "build Python" which is a local build of -Python used for cross-builds. You will also have the WASI build. +Note that all prerequisites are included and configured appropriately in the +[WASI dev container image](https://github.com/python/cpython-devcontainers/pkgs/container/wasicontainer). +You can download it via: + +```shell +podman pull ghcr.io/python/wasicontainer:latest +``` +The `latest` image contains the WASI SDK versions required by all supported +CPython branches. ### Development loop @@ -58,6 +65,9 @@ The common way to get started is to first do a full build: python3 Platforms/WASI build --quiet --logdir cross-build/logs -- --with-pydebug --config-cache ``` +In the end, you will end up with a "build Python" which is a local build of +Python used for cross-builds. You will also have the WASI build. + Once you have the build you can run the test you want. Bash: