Deprecate XNNPACK fp32 variants that have a faster, smaller quantized twin

[msluszniak]

From the v0.10.0 benchmarks in #1363 (my SM-S948B estate, Bartek's Pixel 10 / iPhone SE 3 / iPhone 17 runs).

17 XNNPACK fp32 variants have a quantized XNNPACK twin that is faster and smaller, and 15 LLM bf16/fp16 variants have one too. They are candidates for deprecation once output quality is checked. A further 63 fp32 variants have no quantized twin at all and need one exported first.

Speed and size are from SM-S948B. median ms is the whole pipeline, MB is download size. No accuracy data exists yet for any row here.

Tier A: deprecate after a quality check (24)

Vision (9)

variant	replacement	fp32 ms	quant ms	speedup	fp32 MB	quant MB
semantic-segmentation/fcn-resnet50-xnnpack-fp32	int8	500.2	117.9	4.24x	132	36
semantic-segmentation/fcn-resnet101-xnnpack-fp32	int8	669.6	203.3	3.29x	208	55
semantic-segmentation/deeplab-v3-resnet101-xnnpack-fp32	int8	791.0	242.4	3.26x	234	62
semantic-segmentation/deeplab-v3-resnet50-xnnpack-fp32	int8	522.3	161.9	3.23x	159	42
semantic-segmentation/deeplab-v3-mobilenet-v3-large-xnnpack-fp32	int8	92.4	41.1	2.25x	44	11
style-transfer/candy-xnnpack-fp32	int8	864.8	364.8	2.37x	7	2
style-transfer/mosaic-xnnpack-fp32	int8	700.4	357.7	1.96x	7	2
style-transfer/udnie-xnnpack-fp32	int8	799.2	436.1	1.83x	7	2
style-transfer/rain-princess-xnnpack-fp32	int8	897.2	557.4	1.61x	7	2

Segmentation is 777 MB of fp32 against 206 MB of int8. Needs mIoU. Style transfer needs a visual diff only.

LLMs (15)

Every non-quantized LLM variant in the registry has a quantized twin in the same family, so all 15 qualify. Sizes are the real .pte content-length on Hugging Face at v0.10.0. Only 2 of the 15 were benchmarked: the bf16 tier was abandoned mid-run because the first one took 34 min of device time.

bf16 (12), 49.8 GB against 18.2 GB

variant	replacement	unquant MB	quant MB	smaller	measured speedup
llm/qwen3-4-b-xnnpack-bf16	8-da4-w	8,048	2,682	3.00x	not run
llm/phi4-mini-xnnpack-bf16	8-da4-w	7,673	2,815	2.73x	not run
llm/llama3-2-3-b-xnnpack-bf16	spinquant	6,428	2,553	2.52x	not run
llm/qwen2-5-3-b-xnnpack-bf16	8-da4-w	6,174	2,092	2.95x	not run
llm/hammer2-1-3-b-xnnpack-bf16	8-da4-w	6,173	2,053	3.01x	not run
llm/qwen3-1-7-b-xnnpack-bf16	8-da4-w	3,444	1,304	2.64x	not run
llm/qwen2-5-1-5-b-xnnpack-bf16	8-da4-w	3,090	1,136	2.72x	not run
llm/hammer2-1-1-5-b-xnnpack-bf16	8-da4-w	3,089	1,135	2.72x	not run
llm/llama3-2-1-b-xnnpack-bf16	spinquant	2,473	1,136	2.18x	not run
llm/qwen3-0-6-b-xnnpack-bf16	8-da4-w	1,195	506	2.36x	not run
llm/qwen2-5-0-5-b-xnnpack-bf16	8-da4-w	989	417	2.37x	58.2x
llm/hammer2-1-0-5-b-xnnpack-bf16	8-da4-w	989	417	2.37x	not run

qwen2-5-0-5-b-xnnpack-bf16 ran at 0.5 tok/s against 27.1 for its own 8da4w twin, measured minutes apart on the same device, and it emitted all 63 tokens so it is not a truncated generation. The one fp16 pair costs only 2.2x, so 58x is not the price of precision: bf16 is off the fast kernel path. Treat this as a defect, not a tradeoff. The other 11 bf16 rows are unmeasured, but they share the export path, so the finding very likely generalizes.

fp16 (3), 6.6 GB against 2.0 GB

variant	replacement	unquant MB	quant MB	smaller	measured speedup
llm/bielik-v3-1-5-b-xnnpack-fp16	8-da4-w	3,194	923	3.46x	not run
llm/lfm2-5-1-2-b-xnnpack-fp16	8-da4-w	2,611	796	3.28x	not run
llm/lfm2-5-350-m-xnnpack-fp16	8-da4-w	845	278	3.04x	2.2x

fp16 is a weaker case than bf16 and should be judged on quality, not speed. 2.2x slower for 3x the download is a defensible tradeoff if the output is measurably better, which nobody has checked.

Tier B: probable, smaller margin (8)

variant	replacement	fp32 ms	quant ms	speedup	fp32 MB	quant MB
text-embeddings/distiluse-base-multilingual-cased-v2-xnnpack-fp32	8da4w	13.8	7.9	1.74x	544	396
speech-to-text/whisper-en-small-xnnpack-fp32	int8	1732.6	1034.1	1.68x	1133	452
speech-to-text/whisper-en-base-xnnpack-fp32	int8	397.5	256.9	1.55x	403	252
text-embeddings/paraphrase-multilingual-minilm-l12-v2-xnnpack-fp32	8da4w	13.2	8.9	1.48x	487	414
ocr/paddle-ppocrv6-small-xnnpack-fp32	xnnpack (default)	239.7	187.7	1.28x	31	24
speech-to-text/whisper-en-tiny-xnnpack-fp32	int8	143.7	118.5	1.21x	237	180
semantic-segmentation/lraspp-mobilenet-v3-large-xnnpack-fp32	int8	33.5	29.1	1.15x	13	4
classification/efficientnet-v2-s-xnnpack-fp32	int8	73.9	53.5	1.38x	86	23

The whisper .en models need WER on a held-out set. int8 is already the iOS default for them, so the fp32 build is opt-in traffic.

The two 8da4w embedders look mis-exported. 414 MB against 487 MB fp32, and 396 MB against 544 MB. A 4-bit weight variant that is only 1.2x to 1.4x smaller suggests most of the model was not quantised. Worth checking before treating 8da4w as the replacement.

Keep fp32 (4)

variant	why
semantic-segmentation/selfie-segmentation-xnnpack-fp32	under 1 MB, 5.6 ms
semantic-segmentation/selfie-segmentation-landscape-xnnpack-fp32	under 1 MB, 5.5 ms
keypoint-detection/blazeface-xnnpack-fp32	1 MB, 9.3 ms
voice-activity-detection/fsmn-vad-xnnpack-fp32	2 MB, 9.0 ms

Quantising these buys nothing and only risks accuracy.

Quantize first: 63 fp32 variants with no quantized twin (12.3 GB)

family	fp32 variants	total MB	slowest ms
kokoro (9 language bundles, 3 networks)	9	3080	2658
yolo26 (detection / instance-seg / pose)	33	2928	558
whisper multilingual (tiny / base / small)	3	1773	2121
sdxs-512-dreamshaper	1	1764	1217
text embedders (mpnet / minilm / clip-text)	5	1311	17
supertonic	1	401	905
rfdetr (detection / instance-seg / keypoint)	3	381	789
clip image	1	352	25
fastsam	2	336	548
ssdlite320-mobilenet-v3-large	1	14	22

Priority:

1. whisper multilingual tiny / base / small. whisper-small is the slowest non-LLM model in the estate at 2121 ms and 1133 MB. The .en twins already have int8, so the export path is known.
2. yolo26 + rfdetr + fastsam. 38 variants, 3.6 GB, and the entire detection / instance-segmentation / keypoint offering is fp32 only.
3. mpnet embedders. 437 MB each, fp32 only on XNNPACK. A Vulkan int8 exists but Vulkan is Android only.

Evidence caveat

The fp32-vs-quantized head-to-head exists on SM-S948B only. None of Bartek's three tables contain an xnnpack-fp32 row for any family that also has an XNNPACK quantized twin, so no second device confirms these ratios. That absence is itself signal that fp32 is not what those platforms default to.

iOS is not stranded by a removal: on iPhone 17 CoreML fp16 runs deeplab-v3-resnet50 at 103.1 ms and candy at 63.2 ms, both well ahead of XNNPACK int8 on the same device.

Proposed rule

Not "remove every non-quantized XNNPACK model", because 63 of them are the only build that exists for their model and 5 more are better off as fp32. Instead:

An unquantized build (fp32, fp16, bf16) does not ship where a quantized twin on the same backend is at least as fast and materially smaller, and where output quality has been shown to hold.

Steps

• Accuracy comparison against fp32 per candidate (mIoU for segmentation, WER for whisper, cosine similarity for embedders, visual diff for style transfer)
• Check HF download counts per variant so anything with real traffic gets a deprecation window
• Re-check the 8da4w text embedder exports
• Deprecate in the registry with a console warning for one minor, then remove
• Root-cause the bf16 kernel path before deprecating on speed alone; if bf16 is fixable it changes the argument for those 12 rows
• Export quantized variants for the priority families above

[msluszniak]

@momstouts67-eng You don't need to be reluctant to depending on vulkan variant. It is correct, please check my comment in the issue you mentioned. However I will export int8 variant to make sure that people that want to scope to only xnnpack will have viable option.