Texture-set overflow thrashes both backends: replace the slot ladder with texture arrays

[obiot]

Symptom

A scene stays smooth up to the multi-texture limit and then falls off a cliff the moment one more distinct texture enters the frame — reported at 16 → 17. The drop is far larger than one extra texture should cost, and it is not a hardware wall: nothing about the 17th texture is more expensive to sample than the 16th.

Mechanism

TextureCache.allocateTextureUnit() (packages/melonjs/src/video/texture/cache.js:68) scans for a free unit and, when there is none:

// No units available — flush the current batch and reset assignments
if (this.renderer.currentBatcher) {
    this.renderer.currentBatcher.flush();
}
this.units.clear();
this.usedUnits.clear();
...
emit(GPU_TEXTURE_CACHE_RESET);

The eviction policy is evict everything. One texture too many discards all N live assignments, and the emitted GPU_TEXTURE_CACHE_RESET makes every batcher drop its cached bindings too (webgl/batchers/material_batcher.js:88). QuadBatcher.addQuad holds a second copy of the same wipe when the assigned unit lands beyond the shader's sampler array (webgl/batchers/quad_batcher.js:281-292).

That is why the cliff is a cliff rather than a slope. Below the limit every texture keeps its unit for the whole frame and the scene batches. One over, and because submission order is world/z order — not texture order — an interleaved draw list makes the overflow recur: each wipe invalidates all N bindings, the next N draws each re-allocate and re-bind, and the next foreign texture wipes them again. Worst case degenerates toward a draw call per sprite plus a full re-bind cycle between them.

WebGPU has the same failure mode with a smaller blast radius. Its quad batcher resolves each quad to a slot in a pending segment of MAX_QUAD_TEXTURES = 8 (webgpu/pipeline/cache.js:15, webgpu/batchers/quad_batcher.js:207-233), and resetSegment() (:289) clears all eight slots on flush. It is submission-ordered for the same reason, so a round-robin over MAX_QUAD_TEXTURES + 1 textures flushes once per cycle there too, at a lower threshold. The difference is scope: WebGPU's reset stays inside one batcher, while WebGL's also clears the shared unit map and notifies every other batcher.

Root cause, stated plainly

Both backends emulate "many textures in one draw" with a slot ladder: N sampler bindings plus a switch on a per-quad id. They do that because neither shading language lets you index a sampler dynamically — GLSL ES 3.00 requires sampler-array indices to be constant expressions, and WGSL requires uniform control flow, which is why quad.wgsl:18-19 samples with textureSampleLevel(…, 0.0) instead of textureSample.

The ladder is the thing that has a capacity. Everything above — the wipes, the resets, the cliff — is capacity management for a structure that only exists to work around dynamic indexing.

Target: texture arrays

sampler2DArray is core in WebGL 2; texture_2d_array is core in WebGPU. One binding, N layers, and the layer index is fully dynamic, because it is a coordinate rather than control flow:

texture(uTextures, vec3(uv, layer))

This removes the ladder instead of managing it:

• Capacity rises by a large multiple. MAX_ARRAY_TEXTURE_LAYERS is ≥256 on GLES 3.0, but that is not the usable ceiling — see the memory constraint below. In practice a slot holds an array of N textures instead of one, so the ladder survives with capacity slots × N rather than slots. Overflow becomes rare enough that eviction stops being a hot path.
• No reordering required. Draws never need grouping by texture set, so the painter's-algorithm constraint — which makes any sort-by-texture scheme unsound for overlapping alpha-blended sprites — never arises.
• The vertex stream does not move. The per-quad aTextureId both backends already ship becomes the layer index unchanged.
• Mips come back on WebGPU. The textureSampleLevel(…, 0.0) workaround exists only because the slot index is non-uniform across a draw. A layer coordinate is not a branch, so implicit-derivative sampling works.
• Both shaders get simpler. buildMultiTextureFragment(n) stops generating a per-count variant and collapses to one static shader; quad.wgsl loses its eight hand-written cases.
• The backends converge rather than continuing to drift, which is the standing 20.0 direction.

Since 20.0 dropped WebGL 1, sampler2DArray is unconditionally available on master — no capability check, no fallback path for the API itself.

The constraints

Memory is the real ceiling. texStorage3D is immutable and allocates every layer eagerly at full size — a 2048² RGBA array with 256 layers is 4 GB, so the ≥256 layer limit is meaningless in practice. Layer counts stay modest (8-32), which means several arrays per class rather than one, and it is why the point above says the ladder survives holding arrays rather than disappearing. Any allocator needs a hard byte budget as a first-class input, not an afterthought.

Every layer of an array must share dimensions and format. Depth is not the problem; uniformity is.

Projects that already pack into atlases bucket well — atlas pages tend to be a handful of identical power-of-two sheets, which is precisely the shape an array wants. Loose images of arbitrary size do not. So the design work is:

1. Size-class bucketing — one array per (dimensions, format) class. Arrays are still banks, but far coarser ones, and switching between them is rare rather than per-overflow.
2. Layer admission — allocating a layer for a texture first seen mid-session, via texSubImage3D / copyExternalImageToTexture with a z offset, plus a growth policy when a class fills (reallocate-and-copy, or a second array in the same class).
3. What stays out — render targets, video textures with changing dimensions, and anything whose size is not known at admission.

Fallback

Textures that cannot be bucketed keep the existing slot ladder, so the ladder does not disappear — it stops being the common path. Two ladder improvements remain worthwhile on their own merits, and are worth landing first because they are small, independent, and useful even if the array work stalls:

• Bounded eviction. Replace the wipe in allocateTextureUnit with an eviction that frees only what it needs, and narrow GPU_TEXTURE_CACHE_RESET so overflow in one batcher stops invalidating every other one. Converts an N-binding loss into a 1-binding replacement.
• Reclaim the WebGPU sampler budget. Group 1 spends two bindings per slot (pipeline/cache.js:266-277) — 8 textures + 8 samplers, half of each base per-stage limit (16/16). But TextureStore.getSampler() already deduplicates samplers by (filter, addressModeU, addressModeV, mipmaps), so a segment resolves to one or two distinct sampler objects bound redundantly across 8 bindings. Selecting the sampler by a small id beside the texture switch reallocates that budget toward texture slots — on the order of 12 + 4 for the same 16 bindings. It is a trade: per-stage sampled-texture headroom drops from 8 to 4 while sampler headroom rises from 8 to 12, so it needs checking against what the lit family (color + normal, plus the map_d opacity pair) and ShaderEffect's extra samplers bind in the same fragment stage.

Not pursuing: bank-coherent submission. Sorting draws by texture set was the other way to bound the ladder's cost, and it is the one that collides with alpha-blended draw order — sound only for non-overlapping runs, depth-tested content, or an explicit opt-in. Texture arrays make it unnecessary, so it should not be built.

Explicitly not the fix

• LRU eviction. The obvious policy, defeated by exactly the access pattern that triggers this: a round-robin over N+1 textures evicts, every time, the one needed next. Helps the skewed case, worthless in the pathological one.
• Raising the ladder width. Many desktop GPUs report 32 MAX_TEXTURE_IMAGE_UNITS, so maxBatchTextures could rise where the device allows. That moves the cliff, it does not remove it, and it costs shader compile time and hurts on mobile. A knob.
• Telling users to pack their atlases better. Already the recommended practice and already done in the projects that hit this. The engine should degrade predictably when it is not enough.

Later

binding_array (bindless) would remove the ladder for the non-bucketable tail as well, with no uniformity constraint at all — but it is a WGSL feature with no WebGL 2 equivalent, so it can only ever help one backend, and not yet. Texture arrays are the portable answer and do not block it.

Testing

Most of the risk is testable without a GPU, which should drive how this is built.

Counters come first. There is no instrumentation for this today, so land flushes-per-frame and cache-resets-per-frame before any behavior changes — the "before" has to be measured rather than inferred, and they are the only way to state a pass condition. On WebGPU an equivalent assertion is already possible: createMockWebGPURenderer() (tests/helpers/webgpu-mock-renderer.js) records draws, drawIndexed, setPipeline and materialBinds, so "N textures produce one draw" is a direct unit assertion with no device.

The allocator is pure logic — test it as such. Bucketing, layer admission, growth and the byte budget are deterministic functions of an admission sequence. Feed a list of (w, h, format, filter, wrap) and assert both the resulting assignment and the total allocated bytes. That covers the memory ceiling, the uniformity constraint and the growth policy — the three expensive risks — before any GPU is involved. Budget assertions are the guard against the failure mode that hurts most: a bucketing policy that silently costs hundreds of MB.

Correctness: pixel readback, not screenshots. Draw N sprites in distinct solid colours, read the framebuffer back, assert each rect sampled its own layer. This catches the bug class that actually matters here — off-by-one layer index, UV remap after padding — and it is already the house idiom, with 19 spec files using gl.readPixels. No golden images to maintain, and no visual-regression harness needs to be introduced.

A visual test cannot show the goal. The rendered picture is identical whether the frame issued one draw call or four hundred, so screenshots prove "not broken", never "no longer thrashing". Correctness and batching need separate assertions; only the counters speak to the actual objective.

Out of scope for automation: real frame timing and actual VRAM consumption. Those stay manual, and any figure quoted should be no finer than the instrument resolves.

The claim to validate end-to-end: a scene above the ladder width stops producing cache resets once its textures live in an array.

Naming

MAX_QUAD_TEXTURES = 8 on WebGPU and maxBatchTextures = min(maxTextures, 16) on WebGL are two spellings of one concept. Whatever lands should give them one name in one place.