Repository navigation
local: LocalClefTypeSafeClient.loadOnGpu; no GPU option for Laya - #19
Merged
Merged
Conversation
Clef-flash's 4-bit graph on WebGPU gives the same answers as the CPU on every launch and runs about 3.5x faster, so it gets a public loadOnGpu (macOS on Apple Silicon: the WebGPU backend ships only in ONNX Runtime's macOS native library). Laya stays CPU-only in the public API: on WebGPU its choice probabilities drifted between process launches (0.49-0.65 for an input the CPU answers 0.65), the failure mode that removed int8 Laya. Timings live in the how-to, not the javadoc. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #14: a public GPU option where the GPU is both faster and reliable.
LocalClefTypeSafeClient.loadOnGpu(Path): Clef-flash on ONNX Runtime's WebGPU backend (Metal). macOS on Apple Silicon only: the WebGPU backend ships only in ONNX Runtime's macOS native library, so elsewhere it throwsIllegalStateException.billingprobability for the same inputLaya's WebGPU drift is the same silent, hardware-dependent failure mode that removed int8 Laya, for a saving of tens of milliseconds. Its package-private
load(dir, gpu)stays for the test harness.Timings are in the how-to, not the javadoc. Docs, reference, explanation (why Laya has none), CLAUDE.md and CHANGELOG are updated.
./mvnw verifypasses;loadOnGpuwas also called from another package against the 4-bit model.🤖 Generated with Claude Code