Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions .github/workflows/local.yml
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
name: Local engines

# Agreement with Jev and throughput of typesafe-java-local's engines, as a table in the job summary.
# Agreement with Jev and throughput of typesafe-java-client-local's engines, as a table in the job summary.
# Model directories are cached: Laya fp32 (+ fp16 on macOS, for the GPU) and Qwen 4-bit, all onnx-community exports.
on:
workflow_dispatch:
push:
branches: [main]
paths: ['local/**', '.github/workflows/local.yml', '.github/scripts/local_summary.py']
paths: ['client-local/**', '.github/workflows/local.yml', '.github/scripts/local_summary.py']
pull_request:
paths: ['local/**', '.github/workflows/local.yml', '.github/scripts/local_summary.py']
paths: ['client-local/**', '.github/workflows/local.yml', '.github/scripts/local_summary.py']

jobs:
numbers:
Expand Down Expand Up @@ -48,23 +48,23 @@ jobs:
uvx --from huggingface_hub hf download Cloudflare/clef-flash --revision 17f0b0ad64efb65d273590632833508766b2aae6 tokenizer.json --local-dir $M/clef-flash

- name: Build
run: ./mvnw -B -q -pl local -am test-compile dependency:build-classpath -Dmdep.outputFile=$PWD/cp.txt -Dmdep.includeScope=test
run: ./mvnw -B -q -pl client-local -am test-compile dependency:build-classpath -Dmdep.outputFile=$PWD/cp.txt -Dmdep.includeScope=test

- name: Real-model tests
run: ./mvnw -B -pl local -am test -DexcludedGroups=acceptance -Dengine=laya -Dsurefire.failIfNoSpecifiedTests=false
run: ./mvnw -B -pl client-local -am test -DexcludedGroups=acceptance -Dengine=laya -Dsurefire.failIfNoSpecifiedTests=false

- name: Agreement with Jev
working-directory: local
working-directory: client-local
run: |
CP="$(cat ../cp.txt):target/classes:target/test-classes"
ENGINES="laya qwen"
if [ "$RUNNER_OS" = macOS ]; then ENGINES="$ENGINES laya-fp16-gpu"; fi
for e in $ENGINES; do
java -cp "$CP" io.github.dfa1.typesafe.local.JevComparison $e | tee -a ../jev.txt
java -cp "$CP" io.github.dfa1.typesafe.client.local.JevComparison $e | tee -a ../jev.txt
done

- name: Throughput
working-directory: local
working-directory: client-local
run: |
CP="$(cat ../cp.txt):target/classes:target/test-classes"
ENGINES=LAYA,QWEN
Expand Down
28 changes: 15 additions & 13 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,25 +7,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

- **Breaking: `JsonCodec` is now `Codec`, and it and `HttpTransport` speak `byte[]`** — `writeValueAsBytes`/`readValue(byte[], ...)` (no pretty-printing variant: the CLI's `--verbose` prints compact JSON), `Builder.codec(...)` (was `jsonCodec(...)`), and a `byte[]` request/response body, so JSON goes to and from the wire as UTF-8 with no `String` copy and a binary codec can implement the same SPI; `Jackson2Codec` now writes emoji and other non-BMP characters unescaped, like `Jackson3Codec`. (#35)
- **Breaking: modules split into model (`core`), serialization (`codec`), contract (`client`) and implementations (`client-*`), names aligned** — `core` keeps the model; `Codec` is the new `codec` module; `TypeSafeClient`, `TypeSafeException` and the decorators move to the new `client`, and `DefaultTypeSafeClient`/`ApiKey`/`HttpTransport` to the new `client-http`; `TypeSafeClient.builder()` becomes `DefaultTypeSafeClient.builder()`, and every module's artifact and package now match its directory (`client-http-jdk`, `client-http-okhttp`, `codec-jackson2`, `codec-jackson3`, `client-local`, `client-mapping`, `client-testkit`); see ADR 0003. (#35)
- **docs: local-model setup as three steps** — dependency, Laya download, code, then a model comparison; the optional Clef 4-bit script runs from its GitHub URL with `uv run`, and load errors link to the how-to.
- **`local`: hardening** — special tokens in caller text stay plain text (no prompt injection via `<|im_end|>`), the tokenizer cache is bounded, and Clef checks its weights' sha256 on first load.
- **`local`: shared request validation, `evaluateAsync` off virtual threads** — every engine rejects no questions or empty criteria with `BadRequest` (Laya used to NPE or answer a null choice), and `evaluateAsync` runs on one platform thread so native inference can't pin the JVM's virtual-thread carriers.
- **`local`: `LocalClefTypeSafeClient.loadOnGpu(...)`** — Clef-flash on ONNX Runtime's WebGPU backend (macOS on Apple Silicon), same answers as the CPU; Laya gets no GPU option, since its WebGPU answers drift between launches. (#19)
- **`client-local`: hardening** — special tokens in caller text stay plain text (no prompt injection via `<|im_end|>`), the tokenizer cache is bounded, and Clef checks its weights' sha256 on first load.
- **`client-local`: shared request validation, `evaluateAsync` off virtual threads** — every engine rejects no questions or empty criteria with `BadRequest` (Laya used to NPE or answer a null choice), and `evaluateAsync` runs on one platform thread so native inference can't pin the JVM's virtual-thread carriers.
- **`client-local`: `LocalClefTypeSafeClient.loadOnGpu(...)`** — Clef-flash on ONNX Runtime's WebGPU backend (macOS on Apple Silicon), same answers as the CPU; Laya gets no GPU option, since its WebGPU answers drift between launches. (#19)
- **docs: run Clef-flash on a Mac with MLX** — a how-to pointing the regular client at mlx-community's local Clef-flash server: about 0.55 s per request on an M5, much closer to Jev than Laya. (#14)
- **`local`: `LocalClefTypeSafeClient`** — Cloudflare's Clef-flash (Qwen3.5-9B + joint schema head) on Ollaya's ONNX graph over the upstream bf16 weights; one forward pass per request; `scripts/clef/quantize_q4.py` converts it to 4-bit weights (7.7 GB, 2–4 s per request on an M5 GPU). (#14)
- **`local`: `LocalLayaTypeSafeClient`, `LocalQwenTypeSafeClient`** — new module, one client class per model, evaluating in-process on ONNX Runtime from a local model directory (Laya fp32 from onnx-community, or Qwen2.5 as a prompted-LLM baseline), with a `Local engines` workflow publishing agreement-with-Jev and throughput tables. (#14)
- **`client-local`: `LocalClefTypeSafeClient`** — Cloudflare's Clef-flash (Qwen3.5-9B + joint schema head) on Ollaya's ONNX graph over the upstream bf16 weights; one forward pass per request; `scripts/clef/quantize_q4.py` converts it to 4-bit weights (7.7 GB, 2–4 s per request on an M5 GPU). (#14)
- **`client-local`: `LocalLayaTypeSafeClient`, `LocalQwenTypeSafeClient`** — new module, one client class per model, evaluating in-process on ONNX Runtime from a local model directory (Laya fp32 from onnx-community, or Qwen2.5 as a prompted-LLM baseline), with a `Local engines` workflow publishing agreement-with-Jev and throughput tables. (#14)
- **build: checkstyle engine 14.3.0** — the plugin's default 9.3 can't parse Java 21 pattern matching for switch; same rules. (#14)
- **Breaking: optional API key, `ApiKey` without a public constructor** — `TypeSafeClient.builder()` takes no argument and `.apiKey(key)` is optional (no key, no `Authorization` header, for local servers); `ApiKey` is a final class built with `of(String)`/`fromFile`/`fromDefaultFile`/`fromEnv`, and no longer exposes its value. (#18)
- **`jackson2`: ignore unknown fields, like `jackson3`** — a field the API adds to a response no longer fails decoding with `ResponseDecoding`; Jackson 2's default rejected it. (#18)
- **`codec-jackson2`: ignore unknown fields, like `codec-jackson3`** — a field the API adds to a response no longer fails decoding with `ResponseDecoding`; Jackson 2's default rejected it. (#18)
- **`core`: `Content.fields(...)` and `EvaluateRequest.Builder` keep their order** — both copied with `Map.copyOf`, which reshuffles keys per JVM run, so the same request serialized with fields/questions in a different order each run; they now keep the caller's iteration order.
- **`core`: `TokenCounter`** — running totals of input/output tokens from `EvaluateResponse#usage()`, added via `.decorateWith(tokens::decorate)`; thread-safe, shareable across clients.
- **`core`: `Builder.decorateWith(...)` naming and docs** — named to read as additive and to pair with the `decorate(...)` factories; `Builder` javadoc and the how-to explain decorator ordering.
- **`core`: `build()` rejects a second `RetryingTypeSafeClient`** — throws `IllegalStateException`, since stacked retries multiply attempts; `decorate(...)` now takes the delegate first on every decorator.
- **`client`: `TokenCounter`** — running totals of input/output tokens from `EvaluateResponse#usage()`, added via `.decorateWith(tokens::decorate)`; thread-safe, shareable across clients.
- **`client-http`: `Builder.decorateWith(...)` naming and docs** — named to read as additive and to pair with the `decorate(...)` factories; `Builder` javadoc and the how-to explain decorator ordering.
- **`client-http`: `build()` rejects a second `RetryingTypeSafeClient`** — throws `IllegalStateException`, since stacked retries multiply attempts; `decorate(...)` now takes the delegate first on every decorator.
- **Breaking: retries are opt-in** — `build()` makes one attempt per call; `Builder.maxRetries`/`initialBackoff` are gone (use `.decorateWith(RetryingTypeSafeClient::decorate)`, 5 retries from 500ms), and `Builder.decorator(...)` is renamed `decorateWith(...)`.
- **`core`: `Builder.decorator(...)`** (renamed `decorateWith(...)` below) — stacks decorators from the builder, last added outermost; `build(Function)` stays for a type-preserving outermost decorator.
- **`core`: `RetryingTypeSafeClient.decorate(...)`/`DeadlineTypeSafeClient.decorate(...)`** — static factories taking the client to wrap first, matching `MappingTypeSafeClient::decorate`; constructors are package-private.
- **`core`: `DeadlineTypeSafeClient` caps a call's total time, retries included** — fails with `TypeSafeException.Timeout` past the deadline; `RetryingTypeSafeClient` stops retrying once its future is done.
- **`core`: retry/backoff extracted into a `RetryingTypeSafeClient` decorator** — `DefaultTypeSafeClient` no longer retries itself, and `InternalServer` gains `retryAfter()`.
- **`client-http`: `Builder.decorator(...)`** (renamed `decorateWith(...)` below) — stacks decorators from the builder, last added outermost; `build(Function)` stays for a type-preserving outermost decorator.
- **`client`: `RetryingTypeSafeClient.decorate(...)`/`DeadlineTypeSafeClient.decorate(...)`** — static factories taking the client to wrap first, matching `MappingTypeSafeClient::decorate`; constructors are package-private.
- **`client`: `DeadlineTypeSafeClient` caps a call's total time, retries included** — fails with `TypeSafeException.Timeout` past the deadline; `RetryingTypeSafeClient` stops retrying once its future is done.
- **`client`: retry/backoff extracted into a `RetryingTypeSafeClient` decorator** — `DefaultTypeSafeClient` no longer retries itself, and `InternalServer` gains `retryAfter()`.

## [0.6.0] - 2026-09-30

Expand Down
Loading
Loading