`os.sendfile` leaks header buffer exports on validation errors

[KowalskiThomas]

Bug report

Bug description:

On macOS (and as far as I know FreeBSD), os.sendfile takes (among others) two headers and trailers parameters. Calling iov_setup acquires Py_buffer exports for headers, but several validation steps that come after can error and return before iov_cleanup is ever called.

The setup is done here:

cpython/Modules/posixmodule.c

Lines 12532 to 12550 in f74cdf8

	if (headers != NULL) {
	if (!PySequence_Check(headers)) {
	PyErr_SetString(PyExc_TypeError,
	"sendfile() headers must be a sequence");
	return NULL;
	} else {
	Py_ssize_t i = PySequence_Size(headers);
	if (i < 0)
	return NULL;
	if (i > INT_MAX) {
	PyErr_SetString(PyExc_OverflowError,
	"sendfile() header is too large");
	return NULL;
	}
	if (i > 0) {
	sf.hdr_cnt = (int)i;
	if (iov_setup(&(sf.headers), &hbuf,
	headers, sf.hdr_cnt, PyBUF_SIMPLE) < 0)
	return NULL;

The early returns after that never call iov_cleanup:

cpython/Modules/posixmodule.c

Lines 12551 to 12584 in f74cdf8

	#ifdef __APPLE__
	for (i = 0; i < sf.hdr_cnt; i++) {
	Py_ssize_t blen = sf.headers[i].iov_len;
	# define OFF_T_MAX 0x7fffffffffffffff
	if (sbytes >= OFF_T_MAX - blen) {
	PyErr_SetString(PyExc_OverflowError,
	"sendfile() header is too large");
	return NULL;
	}
	sbytes += blen;
	}
	#endif
	}
	}
	}
	if (trailers != NULL) {
	if (!PySequence_Check(trailers)) {
	PyErr_SetString(PyExc_TypeError,
	"sendfile() trailers must be a sequence");
	return NULL;
	} else {
	Py_ssize_t i = PySequence_Size(trailers);
	if (i < 0)
	return NULL;
	if (i > INT_MAX) {
	PyErr_SetString(PyExc_OverflowError,
	"sendfile() trailer is too large");
	return NULL;
	}
	if (i > 0) {
	sf.trl_cnt = (int)i;
	if (iov_setup(&(sf.trailers), &tbuf,
	trailers, sf.trl_cnt, PyBUF_SIMPLE) < 0)
	return NULL;

iov_cleanup is only called for trailers and headers after the syscall:

cpython/Modules/posixmodule.c

Lines 12602 to 12605 in f74cdf8

	if (sf.headers != NULL)
	iov_cleanup(sf.headers, hbuf, sf.hdr_cnt);
	if (sf.trailers != NULL)
	iov_cleanup(sf.trailers, tbuf, sf.trl_cnt);

In practice, this can happen if trailers is not valid for example. The function (legitimately) raises a TypeError, but:

• There's a leak: iov_setup allocated an iovec array and a Py_buffer array with PyMem_New. (A call to iov_cleanup would free them, but it is not called.)
• The input is left in an inconsistent state: PyObject_GetBuffer exported each header buffer. Without PyBuffer_Release, that export stays for the lifetime of the process, so for example a bytearray in headers can no longer be resized after the failure.

The same happens on macOS when the header-size overflow check returns after iov_setup, and when iov_setup for trailers fails after headers were already exported.

Reproducer

import os
import tempfile

header = bytearray(b"header")
with tempfile.TemporaryFile() as src, tempfile.TemporaryFile() as dst:
    try:
        os.sendfile(
            dst.fileno(), src.fileno(), 0, 0,
            headers=[header], trailers=object(),
        )
    except TypeError as exc:
        print(exc)  # sendfile() trailers must be a sequence

header.append(0)
# BufferError: Existing exports of data: object cannot be re-sized

The leak is properly reported by ASan/LSan:

=================================================================
==81993==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 80 byte(s) in 1 object(s) allocated from:
    #0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
    #1 0x000100fbbc00 in iov_setup posixmodule.c:12077
    #2 0x000100fae4d0 in os_sendfile_impl posixmodule.c:12454
    #3 0x000100fae4d0 in os_sendfile posixmodule.c.h:8233
    #4 0x000100a6e23c in _PyObject_VectorcallTstate pycore_call.h:144
    #5 0x000100a6e23c in PyObject_Vectorcall call.c:327
    #6 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
    #7 0x000100ddb36c in _PyEval_EvalFrameDefault generated_cases.c.h:3325
    #8 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
    #9 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
    #10 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
    #11 0x000100f19ea4 in run_mod pythonrun.c:1472
    #12 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
    #13 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573
    #14 0x000100f7aad0 in pymain_run_command main.c:262
    #15 0x000100f7aad0 in pymain_run_python main.c:706
    #16 0x000100f7aad0 in Py_RunMain main.c:796
    #17 0x000100f7b978 in pymain_main main.c:826

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
    #1 0x000100fbbbc0 in iov_setup posixmodule.c:12071
    #2 0x000100fae4d0 in os_sendfile_impl posixmodule.c:12454
    #3 0x000100fae4d0 in os_sendfile posixmodule.c.h:8233
    #4 0x000100a6e23c in _PyObject_VectorcallTstate pycore_call.h:144
    #5 0x000100a6e23c in PyObject_Vectorcall call.c:327
    #6 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
    #7 0x000100ddb36c in _PyEval_EvalFrameDefault generated_cases.c.h:3325
    #8 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
    #9 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
    #10 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
    #11 0x000100f19ea4 in run_mod pythonrun.c:1472
    #12 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
    #13 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573
    #14 0x000100f7aad0 in pymain_run_command main.c:262
    #15 0x000100f7aad0 in pymain_run_python main.c:706
    #16 0x000100f7aad0 in Py_RunMain main.c:796
    #17 0x000100f7b978 in pymain_main main.c:826

Indirect leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
    #1 0x000100bd3684 in _PyObject_MallocWithType pycore_object_alloc.h:46
    #2 0x000100bd3684 in _PyType_AllocNoTrack typeobject.c:2523
    #3 0x000100bd345c in PyType_GenericAlloc typeobject.c:2554
    #4 0x000100bdd884 in type_call typeobject.c:2467
    #5 0x000100a6cd90 in _PyObject_MakeTpCall call.c:242
    #6 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
    #7 0x000100de4dc0 in _PyEval_EvalFrameDefault generated_cases.c.h:1846
    #8 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
    #9 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
    #10 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
    #11 0x000100f19ea4 in run_mod pythonrun.c:1472
    #12 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
    #13 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573
    #14 0x000100f7aad0 in pymain_run_command main.c:262
    #15 0x000100f7aad0 in pymain_run_python main.c:706
    #16 0x000100f7aad0 in Py_RunMain main.c:796

Indirect leak of 39 byte(s) in 1 object(s) allocated from:
    #0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
    #1 0x000100a5c9a4 in _PyBytes_FromSize bytesobject.c:121
    #2 0x000100a5c9a4 in _PyBytes_Resize bytesobject.c:3356
    #3 0x000100a3620c in bytearray_resize_lock_held bytearrayobject.c:280
    #4 0x000100a37b50 in PyByteArray_Resize bytearrayobject.c:299
    #5 0x000100a37b50 in bytearray___init___impl bytearrayobject.c:1021
    #6 0x000100a37b50 in bytearray___init__ bytearrayobject.c.h:102
    #7 0x000100bddab4 in type_call typeobject.c:2479
    #8 0x000100a6cd90 in _PyObject_MakeTpCall call.c:242
    #9 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
    #10 0x000100de4dc0 in _PyEval_EvalFrameDefault generated_cases.c.h:1846
    #11 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
    #12 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
    #13 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
    #14 0x000100f19ea4 in run_mod pythonrun.c:1472
    #15 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
    #16 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573

SUMMARY: AddressSanitizer: 199 byte(s) leaked in 4 allocation(s).

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs

• gh-156287: fix leak in os.sendfile #156288