hwaccel init failure segfaults instead of falling back on 16.x (fixed incidentally in 17.0.0)

[adrianrfreedman]

This is already fixed in 17.0.0. I am filing it because the fix was incidental and has no changelog entry, so anyone hitting the crash on 16.x has no way to find out that upgrading resolves it. It also gives background for #2410.

Symptom

On 16.x, any hardware-accelerated decode whose initialisation fails takes the whole process down with SIGSEGV instead of falling back to software decoding, even with allow_software_fallback=True.

import av
from av.codec.hwaccel import HWAccel

c = av.open("high10.mp4", hwaccel=HWAccel(device_type="cuda", allow_software_fallback=True))
for frame in c.decode(c.streams.video[0]):
    break   # SIGSEGV on 16.x

Any stream NVDEC cannot decode will do it. I used 10-bit H.264, since NVDEC is 8-bit 4:2:0 only for H.264, and separately an 8-bit stream whose surface pool exceeds NVDEC's 32-surface limit. The trigger is the failure itself, not the particular stream.

Cause

_get_hw_format is installed as AVCodecContext.get_format, so libavcodec calls it from inside avcodec_send_packet, which PyAV runs under with nogil (av/codec/context.pyx). The callback runs with no GIL held.

In 16.x the loop counter was untyped:

i = 0                  # Python object
while pix_fmts[i] != -1:
    if pix_fmts[i] == private_data.hardware_pix_fmt:
        return pix_fmts[i]
    i += 1             # PyLong_FromLong / PyNumber_Add, with no GIL -> SIGSEGV

On the success path the hardware format is normally the first entry, so i += 1 never runs. On the failure path libavcodec calls the callback a second time with software-only formats, the loop runs to the end, and incrementing i boxes a Python integer with no GIL held.

That also explains why avcodec_open2 never crashed. It is called with the GIL held, so only the send_packet path is fatal.

Fix

Resolved in 17.0.0 by the Cython pure-Python-mode migration, which typed the counter (i: cython.int = 0) and removed the boxing. As far as I can tell that was a side effect of the port rather than a deliberate fix.

Verified on an L40S. 16.x segfaults, and typing the counter alone turns the same run into a clean software fallback. 17.0.0 and current main both fall back cleanly.

Suggestion

Nothing to fix in current main. Two small things that might be worth doing:

1. A changelog note against 17.0.0, so the crash is greppable for anyone still on 16.x.
2. Declare _get_hw_format nogil #2410, declaring the callback nogil so Cython rejects a future reintroduction of a Python object at compile time instead of at runtime.