Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 72 additions & 0 deletions docs/release-notes/0.1.3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# ComputeArena CLI 0.1.3

This release fixes benchmarking with BaseRT 0.2.5. On 0.1.2, every default
run with the released BaseRT 0.2.5 ended, after the benchmark had completed,
with `BaseRT returned incompatible headline measurement metadata`, and no
report was saved. Nothing else changes.

## Install or upgrade

On macOS or Linux:

```sh
curl -LsSf https://computearena.ai/install.sh | sh
```

The same command upgrades an existing installation. Saved reports, the
installation signing key, login sessions, model receipts, and installed
runtimes are kept. Release archives and checksums are also available below for
manual installation.

BaseRT 0.2.5 itself needs no change. If BaseRT is not installed yet, or is
older than 0.2.5, `computearena basert install` fetches the current release.

## What changed

- **Runs with BaseRT 0.2.5 complete and submit.** BaseRT 0.2.5 runs each
prefill size at its own context capacity and records the largest one the
sweep used as the run's context size: 16384 for the default sweep, which
reaches PP16384. ComputeArena 0.1.2 expected the headline's 4096 there and
refused the run once it had finished. The client now expects the value
BaseRT 0.2.5 records. The measurement is unchanged: PP512 and TG128 still
run first in a freshly loaded model with a 4K reservation, every workload's
capacity, order, and repetitions are still checked before signing, and the
reports still normalize to `computearena-throughput/3`. computearena.ai
applies the same rule.

## Reports and compatibility

0.1.2 could not save a BaseRT 0.2.5 report, so there is nothing to resubmit
from it: run the benchmark again with 0.1.3. Reports saved by 0.1.0 to 0.1.2
with BaseRT 0.2.4 or llama.cpp are unaffected and remain submittable when
they contain the full default sweep. Benchmarks already published on
computearena.ai are not changed.

A BaseRT harness built from the development branch before the 0.2.5 release
reports version 0.2.5 but records the headline's capacity instead. 0.1.3 does
not sign its runs, and computearena.ai does not accept reports from it;
replace it with `computearena basert install`. BaseRT 0.2.4 keeps working as
before: its runs use the older protocol, the report records that, and the
client says so before the run.

## Supported systems

Prebuilt ComputeArena binaries are available for:

- macOS on Apple Silicon;
- Linux x86_64 with glibc 2.31 or newer; and
- Linux arm64 with glibc 2.31 or newer.

Windows and Intel Macs are not supported in this release. ComputeArena does not
bundle an inference runtime or model; install BaseRT or llama.cpp and obtain a
compatible model before benchmarking. Prebuilt BaseRT runtimes exist for macOS
on Apple Silicon and for Linux arm64 with CUDA; on Linux x86-64, use llama.cpp
or a BaseRT benchmark harness you built yourself.

The macOS binary is not signed with an Apple Developer ID. The release archive
is checksum-verified and carries a Sigstore keyless signature from the release
workflow. Browser-downloaded copies may need their quarantine attribute removed
as described in the README.

Accuracy benchmarks, MLX, vLLM, Windows, and Intel macOS support are not part
of v0.1.3.
Loading