diff --git a/.github/workflows/container.yml b/.github/workflows/container.yml index 40d6603..4821fb3 100644 --- a/.github/workflows/container.yml +++ b/.github/workflows/container.yml @@ -9,7 +9,8 @@ name: Container # a prerelease is not tagged :latest. # - Actions -> Container -> Run workflow builds and pushes any released version. # - A pull request that changes the Dockerfile or this workflow builds and -# smoke-tests the current pyproject version, and never pushes. +# smoke-tests the newest version already on PyPI (a version-bump PR's own +# version is not released yet), and never pushes. # # The image's org.opencontainers.image.source label links the package to this # repository. Only GITHUB_TOKEN is used (no PAT, no stored secret). Every action @@ -73,7 +74,10 @@ jobs: latest="$INPUT_LATEST" push=true ;; *) - version=$(python3 -c "import tomllib; print(tomllib.load(open('pyproject.toml','rb'))['project']['version'])") + # Test the Dockerfile and workflow against the newest release on PyPI; + # the PR's own pyproject version may not be published yet. + version=$(curl --silent --fail https://pypi.org/pypi/flashspec/json \ + | python3 -c "import json, sys; print(json.load(sys.stdin)['info']['version'])") latest=false push=false ;; esac diff --git a/CHANGELOG.md b/CHANGELOG.md index 43dcfff..5591c30 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,33 @@ Versioning follows [Semantic Versioning 2.0.0](https://semver.org/). ## [Unreleased] +## [0.3.1] — 2026-10-26 — Optional selector, simpler quickstart, container image + +Development status stays Alpha. No measurement in this release: 0.3.1 does +not change the decoding path (an omitted selector is the same one-arm UCB1 +that one-drafter users already passed), so the README tables remain the +0.3.0 release run (bundle `3c43e08`). + +### Highlights +- `SpeculativeEngine(config, [drafter], target)`: with one arm, the bandit + selector argument is optional. The README quickstart no longer needs + `BanditConfig` or a selector, and it prints which drafter strategy ran. +- A GPU-oriented server image on GitHub Container Registry, + `ghcr.io/mattral/flashspec:`, built from the PyPI release and + smoke-tested on every publish (first published for 0.3.0). +- Engineering notes in the docs (`docs/lessons.md`), with figures drawn from + the result JSONs by `scripts/article/`. + +### Upgrade notes +- Nothing to change: every 0.3.0 call works as before. Passing a selector + with one arm is still allowed; omitting it with several arms raises a + `ValueError` that says which selector to pass. + +### Known issues +- Unchanged from 0.3.0 (first call compiles; fp16 greedy exact ties; batch + > 1 and other GPUs or model pairs not measured; bandit experimental; the + CUDA-graph "distinct sizes" warning). + ### Added - `SpeculativeEngine`'s `bandit` argument is optional (default `None`). When the resolved strategy has one arm (the default `drafter_strategy="auto"` with one model drafter, an explicit `"bandit"` strategy with one drafter, or a single explicit diff --git a/CITATION.cff b/CITATION.cff index 13df915..185772a 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -35,7 +35,7 @@ abstract: >- UCB1/Thompson draft selector is experimental: measured once over seven arms, it did not beat the best fixed arm. The Triton and assembly kernels are experimental and not on the generation critical path. -version: 0.3.0 +version: 0.3.1 date-released: "2026-10-26" license: Apache-2.0 repository-code: "https://github.com/Mattral/FlashSpec" diff --git a/README.md b/README.md index fd97a8d..56f9546 100644 --- a/README.md +++ b/README.md @@ -45,8 +45,8 @@ You need Python 3.11 or later, PyTorch 2.2 or later and transformers 4.40 or lat **Container image (GPU-oriented).** Each release is also published as a server image on GitHub Container Registry. It is built from the PyPI release on CUDA 13.0, with the torch build the GPU results were measured with, and it runs as a non-root user. It needs an NVIDIA GPU and the NVIDIA Container Toolkit; it starts on CPU too, but CPU is not where FlashSpec is faster. The default command serves the measured pair in fp16 on port 8000. ```bash -docker pull ghcr.io/mattral/flashspec:0.3.0 -docker run --gpus all -p 8000:8000 ghcr.io/mattral/flashspec:0.3.0 +docker pull ghcr.io/mattral/flashspec:0.3.1 +docker run --gpus all -p 8000:8000 ghcr.io/mattral/flashspec:0.3.1 ``` ## Quickstart @@ -153,7 +153,7 @@ python -m flashspec.serve --target Qwen/Qwen2.5-1.5B-Instruct --drafter Qwen/Qwe Or with the container image. Arguments replace its default command, so pass `--host 0.0.0.0`. The volume keeps downloaded checkpoints between runs: ```bash -docker run --gpus all -p 8000:8000 -v flashspec-hf:/home/flashspec/.cache/huggingface ghcr.io/mattral/flashspec:0.3.0 \ +docker run --gpus all -p 8000:8000 -v flashspec-hf:/home/flashspec/.cache/huggingface ghcr.io/mattral/flashspec:0.3.1 \ --target Qwen/Qwen2.5-1.5B-Instruct --drafter Qwen/Qwen2.5-0.5B-Instruct \ --device cuda:0 --dtype float16 --temperature 0 --host 0.0.0.0 ``` @@ -328,7 +328,7 @@ With `compile` on, both models run on static KV caches and the fixed-shape decod ## Status and limitations -FlashSpec is alpha, version 0.3.0. +FlashSpec is alpha, version 0.3.1. - **Measured scope.** One model pair (`Qwen/Qwen2.5-1.5B-Instruct` verified by `Qwen/Qwen2.5-0.5B-Instruct`), one Tesla T4, fp16, batch 1, 10 MT-Bench and 10 HumanEval prompts, run from `main` at git `3c43e08` with the defaults `compile="auto"` and `drafter_strategy="auto"` (which ran `composite-g6-f2`). No other GPU, model pair or batch size has been measured. Every H100 number in the docs is a design target. - **The first call compiles.** On CUDA the first `generate` compiles the forwards and captures CUDA graphs, which can take minutes; later calls reuse them. The tables exclude that call. diff --git a/deploy/Dockerfile b/deploy/Dockerfile index 4f906b7..7d6c08b 100644 --- a/deploy/Dockerfile +++ b/deploy/Dockerfile @@ -1,6 +1,6 @@ # FlashSpec server image: the released PyPI package on a CUDA 13.0 base. # -# docker build -f deploy/Dockerfile --build-arg FLASHSPEC_VERSION=0.3.0 -t flashspec deploy/ +# docker build -f deploy/Dockerfile --build-arg FLASHSPEC_VERSION=0.3.1 -t flashspec deploy/ # docker run --gpus all -p 8000:8000 flashspec # # GPU-oriented: torch is the CUDA 13.0 build FlashSpec's GPU results were diff --git a/deploy/docker-compose.yml b/deploy/docker-compose.yml index f578408..f6213e8 100644 --- a/deploy/docker-compose.yml +++ b/deploy/docker-compose.yml @@ -7,7 +7,7 @@ services: flashspec: - image: ghcr.io/mattral/flashspec:0.3.0 + image: ghcr.io/mattral/flashspec:0.3.1 ports: - "8000:8000" volumes: diff --git a/deploy/k8s/flashspec-deployment.yaml b/deploy/k8s/flashspec-deployment.yaml index 5db8069..3a40a83 100644 --- a/deploy/k8s/flashspec-deployment.yaml +++ b/deploy/k8s/flashspec-deployment.yaml @@ -4,7 +4,7 @@ metadata: name: flashspec labels: app: flashspec - version: "0.3.0" + version: "0.3.1" spec: replicas: 1 selector: @@ -19,7 +19,7 @@ spec: - name: flashspec # GPU-oriented image; the default command serves Qwen2.5-1.5B-Instruct # verified by Qwen2.5-0.5B-Instruct in fp16 on port 8000. - image: ghcr.io/mattral/flashspec:0.3.0 + image: ghcr.io/mattral/flashspec:0.3.1 imagePullPolicy: IfNotPresent ports: - containerPort: 8000 diff --git a/docs/install.md b/docs/install.md index c625e31..738288e 100644 --- a/docs/install.md +++ b/docs/install.md @@ -26,8 +26,8 @@ git clone https://github.com/Mattral/FlashSpec && cd FlashSpec pip install -e ".[dev,gpu]" # omit ",gpu" on Windows/macOS # Container image (Linux + NVIDIA GPU), published for each release: -docker pull ghcr.io/mattral/flashspec:0.3.0 -docker run --gpus all -p 8000:8000 ghcr.io/mattral/flashspec:0.3.0 +docker pull ghcr.io/mattral/flashspec:0.3.1 +docker run --gpus all -p 8000:8000 ghcr.io/mattral/flashspec:0.3.1 ``` **The container image** (`ghcr.io/mattral/flashspec:` and `:latest`) is @@ -48,7 +48,7 @@ built by `.github/workflows/container.yml` when a release is published: To build it yourself from a released version: ```bash -docker build -f deploy/Dockerfile --build-arg FLASHSPEC_VERSION=0.3.0 -t flashspec deploy/ +docker build -f deploy/Dockerfile --build-arg FLASHSPEC_VERSION=0.3.1 -t flashspec deploy/ ``` A development checkout, by platform: diff --git a/flashspec/__init__.py b/flashspec/__init__.py index 4fa855e..f681a26 100644 --- a/flashspec/__init__.py +++ b/flashspec/__init__.py @@ -42,5 +42,5 @@ "register", ] -__version__ = "0.3.0" +__version__ = "0.3.1" __author__ = "Min Htet Myet (Mattral)" diff --git a/pyproject.toml b/pyproject.toml index 3d8c639..f0a30a6 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "flashspec" -version = "0.3.0" +version = "0.3.1" description = "Lossless speculative decoding for HuggingFace causal LMs, with a static-cache CUDA-graph fast path; bandit draft selection and Triton kernels are experimental" readme = {file = "README.md", content-type = "text/markdown"} license = { file = "LICENSE" }