Repository navigation
Description
Documentation
The pickle docs (and the Pickler/dump docstrings) say:
The file argument must have a write() method that accepts a single bytes argument.
That is not what the picklers actually pass with protocol 4 and 5. write() can get a bytearray, a memoryview or a pickle.PickleBuffer. A custom file object written to the documented contract, for example one that returns len(data) from write(), works for small pickles. With protocol 5 it fails on large buffer-backed objects:
TypeError: object of type 'pickle.PickleBuffer' has no len()
For a non-byte-format buffer, such as array('q') wrapped in PickleBuffer, len(memoryview(data)) silently gives the element count, not the byte count.
Reproducer
import array, io, pickle
class RecordingFile:
def __init__(self):
self.types, self._out = [], io.BytesIO()
def write(self, data):
self.types.append(type(data).__name__)
return self._out.write(data)
SIZE = 1 << 20
values = {
"bytes": b"x" * SIZE,
"bytearray": bytearray(SIZE),
"PickleBuffer(bytearray)": pickle.PickleBuffer(bytearray(SIZE)),
"PickleBuffer(array q)": pickle.PickleBuffer(array.array("q", range(SIZE // 8))),
}
for impl in (pickle.Pickler, pickle._Pickler):
for name, value in values.items():
f = RecordingFile()
impl(f, protocol=5).dump(value)
print(impl.__module__, name, sorted(set(f.types) - {"bytes"}) or "-")
class LenCountingFile:
def write(self, data):
return len(data)
pickle.Pickler(LenCountingFile(), protocol=5).dump(pickle.PickleBuffer(bytearray(SIZE)))
# TypeError: object of type 'pickle.PickleBuffer' has no len()Observed (protocol 5, 1 MiB values; identical on 3.10, 3.11, 3.13, 3.14 and 3.15)
| Value | C _pickle.Pickler passes |
Pure-Python pickle._Pickler passes |
|---|---|---|
bytes |
bytes |
memoryview |
bytearray |
bytearray |
bytearray, memoryview |
PickleBuffer(...) |
PickleBuffer |
memoryview |
Other cases:
- With the pure-Python pickler, even small protocol 4/5 pickles go through
write(memoryview), because_Framer.commit_frame()writesf.getbuffer(). Protocols 2 and 3 pass onlybytes. - In the C pickler, payloads at or above the 64 KiB frame size target bypass the output buffer.
_Pickler_write_bytes()andsave_picklebuffer()callwrite()with the original object.
Prior art
bpo-44439 / gh-88605 hit the same mismatch: LZMAFile.write() / BZ2File.write() used len(data) and failed with protocol 5. Those write() methods were fixed to use memoryview(data).nbytes (GH-26764), but the pickle documentation was not updated, so third-party file-like objects keep hitting it.
Proposal
This is a docs-only change; the behaviour is intentional for zero-copy and efficiency.
- Change "accepts a single bytes argument" to "accepts a single bytes-like object argument" in
Doc/library/pickle.rst, theLib/pickle.pydocstring and the twoModules/_pickle.cclinic docstrings. - Add a short note: depending on the protocol and the object being pickled,
write()may receivebytes,bytearray,memoryvieworPickleBufferinstances, which may not supportlen(). Implementations that need the size of the data should usememoryview(data).nbytes.
I'm happy to submit the PR.
Linked PRs
Metadata
Metadata
Assignees
Labels
Projects
- StatusShow more project fieldsNo status
- StatusShow more project fieldsTodo
Activity
Malavika007 commentedon Nov 11, 2026
helloo! I'd like to work on this issue. I'd be happy to take it up if it's still available. Thanks :D
eachimei commentedon Nov 11, 2026
Hi! It's addressed in #161804
Malavika007 commentedon Nov 11, 2026
Ohh, got it! Thanks for letting me know