Skip to content

[Bug][CUDA][dlight] GPU Fallback calls sch.fuse on a zero loop extent and aborts with "Divide by zero" #20279

Description

@lackjava-creator

Environment

  • TVM version: main @ 48242ec33403f2b6e4fac6e763ca7a683fb9d5df (2026-09-03, reports 0.26.dev0); the same trigger fails on 0.26.0, and on 0.19.0 / 0.12.0 it surfaced as a ZeroDivisionError in Relay's pad_to_tensorcore.

  • Build / install: source build, USE_CUDA=ON, nvcc 12.2, LLVM 15

  • OS / Python: Ubuntu 24.04.3 LTS, x86_64, Python 3.10

  • GPU / driver: Tesla V100-DGXS-32GB, driver 535.309.01, sm_70

  • Target: cuda. opt_level 3 and 0 both fail. llvm is clean at both levels.

  • Pipeline coverage (relax.get_pipeline("default_build") performs no operator fusion, so all three are reported separately). Verified 2026-09-06 on 0.26.0, CUDA target — the abort happens at compile time, before any kernel launch:

    • get_pipeline("default_build") (no FuseOps/FuseTIR) — fails
    • get_pipeline("zero") (fuses) — fails
    • forced FuseOps + FuseTIRfails
      (also confirmed on main @ 48242ec at opt_level 3 and 0)

Minimal reproducer

Single Neg node with input X : float32[4,0] — a zero extent that is not on the leading axis. model.onnx + feed.npz + run.py attached.

import numpy as np, onnx, tvm
from tvm import relax
from tvm.relax.frontend.onnx import from_onnx
model = onnx.load("model.onnx"); feed = dict(np.load("feed.npz"))     # X: float32[4,0]
mod = from_onnx(model, shape_dict={k: list(v.shape) for k, v in feed.items()}, keep_params_in_input=False)
with tvm.transform.PassContext(opt_level=3):
    tvm.compile(relax.get_pipeline("default_build")(mod), target="cuda")   # <-- raises here

Companion note: the leading-axis zero-extent case does not fail here — it compiles and then faults at run time. The full matrix is: 0 on the leading axis (or a 1-D empty tensor) → runtime cudaErrorIllegalAddress; 0 on a non-leading axis → this compile-time abort.

Expected vs actual

  • Expected: compilation succeeds and the result is an empty float32 tensor of shape (4,0). numpy, onnxruntime, and TVM's own llvm target all produce that.
  • Actual:
InternalError: Check failed: pb->value != 0 (0 vs. 0) : Divide by zero

raised during tvm.compile, at both opt_level=3 and opt_level=0.

Root cause (if known)

dlight's GPU Fallback rule calls sch.fuse, and Fuse reconstructs the index of a fused loop as floordiv(floormod(fused, next_lower), lower). Here next_lower == 0, because it is the extent of the fused loop, which is 0. The zero is therefore the compiler's own loop extent, not a divisor coming from the model — an internal invariant that the scheduling rule breaks on itself. llvm is clean, so the rule is the producer.

Never fixed. On 0.12/0.19 the same trigger hit pad_to_tensorcore in the Relay pipeline (ZeroDivisionError); the site moved in (0.19, main], the root cause did not.

Why this is a bug (not tolerance / not undefined behaviour)

A zero extent on a non-leading axis is legal in ONNX and defined by numpy; TVM's llvm target handles it. The abort is not a validation message about the user's model — it is an ICHECK inside a scheduling rule about an expression the rule itself constructed.

Related, not duplicate. Open issue #19922 and closed #17745 hit the same ICHECK (pb->value != 0), but from has_reshape_pattern in the Relax analysis on the llvm target. That producer was fixed by merged PR #19958 (2026-07-09), and our llvm controls are clean on main @ 48242ec. Different producer, different target, still live. Closed #17938 is the same ICHECK string from a third producer (constant folding of an ONNX model).

How found

Found by EquiAutomaton (equivalence-graph differential testing, TVM CUDA vs TVM llvm vs onnxruntime). Depth-0 — a single Neg node.

Reproducer archive

TVM-C3-reproducer.zip

Triage

  • Needs triage

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-triagePRs or issues that need to be investigated by maintainers to find the right assignees to address ittype: bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions