Repository navigation
BindTemporaryResource hangs on new IDMLDevice after prior device removal during operator initializer dispatch #735
Description
Summary
IDMLBindingTable::BindTemporaryResource with a non-null buffer deadlocks CPU-side (never returns, no exception, no error HRESULT) on a freshly-created IDMLDevice when a prior IDMLDevice in the same process experienced device removal during operator initializer dispatch.
The deadlock is conditional on the cause of the prior device removal: device removal during inference does NOT cause this hang on the revival device. Only device removal during the initial operator initializer dispatch poisons subsequent device creation in the same process.
Environment
- DirectML.dll version: 1.15.4
- OS: Windows 11 Pro Version 25H2 (OS Build 26200.8117)
- GPU: NVIDIA RTX 4060 Ti 16GB
- GPU driver version: 595.97
- D3D12 feature level: 12 (12_2)
Reproduction Conditions
The deadlock requires all three of the following:
- A prior
IDMLDevicein the same process experienced device removal (GetDeviceRemovedReasonreturnsDXGI_ERROR_DEVICE_HUNG/0x887A0005) during anIDMLOperatorInitializerdispatch - A new
IDMLDeviceis created viaDMLCreateDeviceafter the prior device's removal (the underlyingID3D12Deviceis still alive — only the DML device died) - A binding table is created on the new device for an
IDMLOperatorInitializerwhoseGetBindingProperties().TemporaryResourceSize > 0, andBindTemporaryResourceis called with a non-nullDML_BUFFER_BINDING
When all three hold, BindTemporaryResource does not return. The calling thread blocks indefinitely.
Reproduction Sequence
// 1. First IDMLDevice runs a 2225-op operator initializer.
// The dispatch's ExecuteCommandLists returns success and the wait fence advances,
// but GetDeviceRemovedReason reports 0x887A0005 immediately afterward.
// (Specific model: RAFT optical flow, 2225 ops, tempSz=50,323,456 bytes,
// 8 persistent-buffer Conv ops at persistSz=86,528 each.)
// 2. Create new IDMLDevice on the same (still-alive) ID3D12Device
ComPtr<IDMLDevice> revivalDml;
DMLCreateDevice(d3d12Device.Get(), DML_CREATE_DEVICE_FLAG_NONE,
IID_PPV_ARGS(&revivalDml)); // succeeds
// 3. Build operator initializer for the same 2225 ops
ComPtr<IDMLOperatorInitializer> init;
revivalDml->CreateOperatorInitializer(2225, compiledOps, IID_PPV_ARGS(&init));
DML_BINDING_PROPERTIES bp = init->GetBindingProperties();
// bp.TemporaryResourceSize = 50,323,456
// bp.PersistentResourceSize = 692,224 (8 * 86,528)
// bp.RequiredDescriptorCount = 84
// 4. Allocate fresh resources on revival device
// (NOT reused from prior device — all freshly created)
ComPtr<ID3D12DescriptorHeap> heap = CreateDescriptorHeap(84);
ComPtr<IDMLBindingTable> bt = CreateBindingTable(init.Get(), heap.Get(), 84);
ComPtr<ID3D12Resource> tempBuf = CreateUAVBuffer(50'323'456);
// 5. Bind temporary resource — DEADLOCKS HERE, never returns
DML_BUFFER_BINDING bb = { tempBuf.Get(), 0, 50'323'456 };
DML_BINDING_DESC desc = { DML_BINDING_TYPE_BUFFER, &bb };
bt->BindTemporaryResource(&desc); // CPU hang
Activity
ALWAYSxREADY commentedon May 1, 2026
Conditions That Do NOT Trigger The Hang
I have verified the following cases all complete normally:
Every batch with TemporaryResourceSize > 0 triggered initial device removal. The 8-op subset has TemporaryResourceSize = 0 and survives trivially via the null-bind path. Determinism was confirmed with multiple runs at op counts 1112 and 2225.
The same TemporaryResourceSize > 0 condition controls whether the revival device's BindTemporaryResource deadlocks.
Eliminated Hypotheses
What I'd Find Useful
Additional Context
The model is a RAFT-style optical flow network compiled to DirectML with 2225 fused ops. The initial operator initializer's ExecuteCommandLists returns success and the wait fence advances, but GetDeviceRemovedReason reports DXGI_ERROR_DEVICE_HUNG (0x887A0005) immediately afterward.
Happy to provide a standalone C++ reproducer, PIX or DRED captures, or the compiled DML graph if useful — let me know what would help most.
ALWAYSxREADY commentedon May 1, 2026
Update: I tested whether ORT's DirectML EP avoids this deadlock by using a different DML device lifecycle. It does not — ORT's CreateSession hangs identically when a prior IDMLDevice in the same process experienced operator-initializer-caused device removal. ORT calls AppendExecutionProvider_DML1, which creates its own IDMLDevice via DMLCreateDevice. That device is poisoned per the conditions in the original issue, and BindTemporaryResource deadlocks during op initialization. This rules out the ONNX Runtime DML EP as a workaround for callers hitting this bug. Both the standalone DirectML API and ORT's DML EP share the same DirectML.dll codepath and exhibit identical behavior.