Feature/marlin binary file transfer

[mennucc]

Summary

Add an optional service that transfers a file from the ESP3D flash filesystem
or attached ESP3D SD card to Marlin's SD card using Marlin Binary File Transfer.

The service:

• obtains exclusive access to the printer serial link while active;
• temporarily disconnects terminal clients and rejects concurrent polling;
• sends the BFT protocol framing and data;
• uses heatshrink compression when requested;
• exposes HTTP start, status, and cancellation endpoints.

The heatshrink 0.4.1 encoder is vendored with its upstream license and an
encoder-finalization regression test.

Configuration

Enable by defining:

#define MARLIN_BINARY_FILE_TRANSFER_FEATURE

The feature is disabled by default.

Testing

Hardware-tested with ESP32-S3, ESP3D 3.1, and Marlin built with BINARY_FILE_TRANSFER.

A 2MB G-code file was transferred from ESP32 LittleFS to Marlin's SD card, then verified with Marlin M35 against the host POSIX cksum. The transfer took less than 2 minutes. To check the cksum, the new g-code proposed in PR 28257was used.

The transferred file was subsequently selected and printed successfully from Marlin's SD card.

The companion PR for the WEBUI will be posted soon.

[luc-github]

Did you do some reseach why this is not yet implemented ?

[mennucc]

I’m not sure I understand the question. Do you mean that this feature was
not implemented in ESP3D?

I saw that ASCII file transfer was implemented in an earlier branch, but was
later removed because it was too slow. This is different: it is a compressed
binary transfer, so it is much faster and practical to use.

Marlin already supports this through BINARY_FILE_TRANSFER (available in
Marlin 2.1.2.8 and can be enabled at compile time). The ESP3D code in this PR
detects the capability through M115 and uses binary transfer only when the
connected Marlin firmware advertises it.

[luc-github]

my question is did you check why this is not integrated already in ESP3D ?
and for you information it is same M28/M29 protocol but using binary frame

[mennucc]

I checked the current code and the repository history more carefully.

ESP3D 3.1 currently has no existing Marlin SD-upload implementation to extend.

ESP3D 2.0 did have an /upload_serial feature using the ASCII flow M28 <filename>,
numbered/checksummed G-code lines, and M29. It disappeared during the 2.1
code-base reorganisation in 2019; the Git history does not document a specific
technical reason for its removal, but I found a comment saying that it was too slow.

I also confirm that the proposed implementation uses Marlin’s Binary
File Transfer protocol, rather than a different file-transfer protocol:
M28 B1 enters binary mode, followed by SYNC, QUERY, OPEN, WRITE, CLOSE, and
CONTROL CLOSE binary frames. In Binary File Transfer mode the binary CLOSE frame closes the file
and CONTROL CLOSE returns Marlin to ASCII mode; this is the binary equivalent
of the normal M28 / data / M29 flow.

So there is no current ESP3D M28/M29 upload path to reuse that I can see. This PR adds all code that
is needed.

A future generic printer-SD upload abstraction could share a UI and common lifecycle with both
ASCII and BFT uploads, but the BFT protocol handling itself has to be added.

[mennucc]

Let me also add this information about the PR.

The first commit e7aaa7e introduces a small exclusive-printer-link API.
It temporarily detaches the normal terminal bridge and suppresses polling while
a service owns the printer serial connection, then restores both on release.
The commit contains a documentation file.

Although introduced for Marlin Binary File Transfer, this API is intentionally
generic and may also be useful for other protocols that need an exclusive, non-text serial session.

[luc-github]

Ok - I have asked IA to repeat what have been done in several discussions of ESP3D repository:

The PR implements Marlin's "Binary File Transfer Mark II" protocol, which uses M28 B1 to enter binary mode.
Why this is fundamentally risky:
Officially experimental: The protocol is documented as pre-1.0 and experimental upstream. It may evolve without backward compatibility.
Plugin ecosystem abandoned: The primary host implementation (OctoPrint-MarlinBft) is unmaintained since 2020. The maintainer explicitly stated they cannot maintain it due to recurring failures on latest Marlin bugfix branches.
Compilation still broken in 2025: A 2025 bug report shows that enabling BINARY_FILE_TRANSFER on GD32 MFL boards causes compilation failures.
Mutual exclusion with MEATPACK: BFT cannot coexist with MEATPACK, a widely-used serial compression feature in Marlin. Users must disable MEATPACK to use BFT, which degrades normal G-code streaming performance.
Rooted in M28/M29: BFT is triggered by M28 B1, which inherits the same architectural flaw as the text-mode M28/M29 path: all serial traffic between M28 and M29 is written to the file. If any unexpected bytes arrive on the serial bus (from another host, TFT screen, or noise), they are silently written into the destination file. The PR attempts to mitigate this with "exclusive serial access," but this only controls the ESP side. It cannot prevent Marlin from writing unexpected data received from other sources.
Conclusion: Building a feature on a protocol that is experimental, abandoned by its plugin ecosystem, incompatible with other Marlin features, and still causing compile-time failures is not acceptable for a production firmware like ESP3D.

now about the approach you suggest :

The PR embeds the heatshrink 0.4.1 C library directly into the repository.
Why this is a problem:
Marginal compression gain on G-code: G-code is text-based with moderate redundancy. Heatshrink is a streaming entropy encoder optimized for repetitive binary data. On typical G-code files, the compression ratio is low, while the cost is an entire C library (source + regression tests) added to the ESP3D tree.
Maintenance burden: Vendoring an external library means ESP3D inherits its update cycle, bug fixes, and portability issues. The upstream heatshrink project has had compatibility issues across platforms.
Historical precedent of breakage: The OctoPrint ecosystem, which uses the same protocol, had to patch the Python heatshrink dependency because the PyPI version was deprecated and incompatible with Python 3, causing installation failures.
RAM cost on ESP8266/ESP32: Compression requires buffer space for the encoder state. On memory-constrained boards, this is a heavy price for a feature that is optional and disabled by default.
Conclusion: The cost of vendoring and maintaining heatshrink is not justified by the modest bandwidth improvement on G-code text files.

The PR description implies using the ESP internal flash (LittleFS) as the staging area for files "of several megabytes".
Why this is a problem:
Severe size constraints: On a standard 4 MB ESP32 with OTA enabled, the LittleFS partition is approximately 1.5 MB. Without OTA, it reaches ~2 MB. Files larger than this simply cannot be stored.
Flash wear: NOR flash has a limited erase-cycle lifespan (~100,000 cycles per sector). Using internal flash as a temporary buffer for multi-megabyte file transfers accelerates wear leveling exhaustion. LittleFS is designed for small persistent files (config, web assets), not for repeated write/delete cycles of large blobs.
Performance degradation: LittleFS append latency degrades significantly as files grow, with periodic spikes of 250–350 ms observed during sustained writes.
Conclusion: Internal flash is not a viable source for multi-megabyte file transfers. The only realistic source is an SPI-attached SD card on the ESP module, but the PR does not clearly document this distinction or restrict the feature to SD-attached hardware.

The PR requires the user to (1) upload a file to the ESP filesystem, then (2) manually call a separate HTTP endpoint to start the BFT transfer.
Why this is a problem:
No integration with existing upload flow: The standard ESP3D /files upload handler is not modified. The user cannot drag-and-drop a file and have it automatically routed to the printer's SD card. They must understand the internal API and manage the transfer lifecycle themselves.
Polling-based status: The user must repeatedly call GET /?action=status to track progress. There is no WebSocket push or seamless UI integration (the companion WebUI PR is mentioned but not provided).
History repeats: ESP3D 3.0 explicitly removed SD upload via serial (UPLOAD_SERIAL_SD) because the workflow was "too slow, too firmware-dependent, and almost always corrupted when using Serial TFT." This PR reintroduces the same broken workflow pattern, just with a faster underlying protocol.
Conclusion: A file transfer feature that requires manual API orchestration is not a user-facing feature; it is a developer tool. Without integration into the upload flow and the WebUI, it will not be used in practice.

[mennucc]

Thank you for the review. I agree that optional features need clear scope,
documentation, and known limitations. However, several statements in your
review are not technically accurate and conflate Marlin's legacy text upload
with Binary File Transfer.

Protocol status and compatibility

BINARY_FILE_TRANSFER is optional and disabled by default in
Marlin. ESP3D also keeps this support optional and checks
Cap:BINARY_FILE_TRANSFER:1 before using it.

The Marlin protocol documentation
reports version 0.1.0 in its SYNC and QUERY responses. I do not find an
upstream statement that labels the protocol "experimental." A 0.x version
is a reason to document compatibility and negotiate capabilities; it is not,
by itself, a reason to reject an optional client implementation.

This protocol has been in Marlin since 2019 and has seen little modification
since. The protocol negotiates its supported parameters, allowing the ESP3D
client to reject incompatible transfers instead of assuming a fixed
configuration.

An unmaintained OctoPrint plugin is not evidence that the Marlin protocol is
unusable. Likewise, a reported compilation issue on a particular GD32 board
needs to be evaluated as that board's build issue. It does not invalidate a
feature that is implemented in Marlin, compiles on other targets, and has been
tested successfully on real hardware.

The MEATPACK exclusion is a real current Marlin limitation. Marlin documents
BINARY_FILE_TRANSFER
as an optional configuration feature, and its
SanityCheck.h
explicitly rejects a build that enables both BFT and MEATPACK. This known
upstream constraint should be stated in the ESP3D feature documentation. It is
a configuration choice between compressed live G-code streaming and transfer
to the printer SD card. Neither option is enabled by default. BFT does not
degrade a print started from the printer SD card, and MEATPACK does not
accelerate such a print.

MEATPACK makes sense for people printing from an external source, such as
OctoPrint, because it reduces the bandwidth needed while commands are executed
live. It does not accelerate a print started from the printer SD card.

M28 B1 is not the legacy text upload path

The claim that BFT inherits the text-mode M28 / M29 corruption model is
incorrect.

In the normal text path, commands received while saving are written as text to
the open file. In contrast, M28 B1 switches Marlin to binary_mode. Marlin
then processes a BinaryStream protocol with a packet start token, header
checksum, payload length, full-packet checksum, synchronization,
acknowledgments, and retransmission handling. File content is written only
after a valid WRITE packet has been decoded.

Unexpected bytes are therefore not silently appended as G-code to the
destination file. They are rejected, cause resynchronization, a timeout, or a
retransmission request. The exclusive printer-link API (first commit) prevents ESP3D's own
terminal bridge and polling traffic from interfering. It is deliberately
generic and can also be used by other services requiring exclusive raw serial
access.

The relevant Marlin implementation is here:

• https://github.com/MarlinFirmware/Marlin/blob/bugfix-2.1.x/Marlin/src/gcode/sd/M28_M29.cpp
• https://github.com/MarlinFirmware/Marlin/blob/bugfix-2.1.x/Marlin/src/feature/binary_stream.h

It is well documented, and it works.

Compression

Heatshrink is not an “entropy encoder optimized for repetitive binary data.”
It is a small streaming LZSS compressor. G-code is highly repetitive text:
coordinates, G/M commands, feedrates, temperatures, and repeated layer
structures provide exactly the sort of repeated sequences it can encode.

Compression is negotiated with Marlin and is used only when compatible
heatshrink parameters are advertised. The protocol also supports uncompressed
transfer. Only the encoder is needed in ESP3D; Marlin already supplies the
matching decoder.

This has been tested end-to-end on hardware. In my setup, a compressed transfer
of a roughly 2 MB G-code file completes in under two minutes over the printer
serial link. This is a practical, usable transfer time, not a modest or
hypothetical gain. It contrasts sharply with the historically reported
approximately 40-minute time for a 1 MB ASCII serial upload.

The assertion that the encoder state is a "heavy" RAM cost is not supported by
a measurement or by a concrete target limit. Heatshrink was selected precisely
because it is a small streaming compressor with bounded state, and the feature
is optional. A board that does not have sufficient resources need not enable
it. On supported hardware, the measured result shows that the modest additional
RAM cost enables a capability that is otherwise absent from ESP3D 3.1: uploading
a file from the WebUI to the printer's own SD card.

The BFT protocol already provides integrity protection for every transfer
packet: start token, header checksum, payload length, full-packet checksum,
synchronization, acknowledgments, and retransmission handling. Invalid data is
not accepted as file content.

Four files were transferred in hardware tests and then individually checked on
the printer SD card with M35; their reported size and POSIX cksum matched
the original source files in every case. One of the transferred files was then
printed successfully.

M35 is a separately proposed optional Marlin feature:
MarlinFirmware/Marlin#28527. If that feature is
accepted, the WebUI will be extended to invoke it after a transfer as an
independent end-to-end checksum double-check. This supplements the BFT packet
checksums; it does not replace them.

Vendoring has a maintenance cost in principle, but this is not a specific
technical objection to this implementation. ESP3D already vendors third-party
libraries, and this PR imports only the small, pinned heatshrink encoder
subset required by the client, with its license and a documented upstream
finalization fix.

Marlin already contains the matching heatshrink decoder. The encoder version is
pinned to upstream v0.4.1, so ESP3D does not inherit an uncontrolled update
cycle. Any claimed portability problem should be identified with a concrete
affected platform, compiler, and reproducible failure; a general statement
that an upstream project has had “compatibility issues” is not evidence that
this small encoder cannot be maintained in ESP3D.

Storage source

For a user wishing to upload a file from a PC to the printer, the usual
workflow is indeed two-stage:

• The browser first uploads the file to an ESP filesystem.
• BFT then reads it from there and writes it to the printer SD card.

When the intermediate step is ESP internal flash, internal-flash capacity and
write endurance are relevant deployment considerations.

However, BFT itself does not create an additional temporary copy: it reads the
existing source file.

Moreover, the same transfer service can use an ESP-attached SD card
as the source, which is the appropriate option for files that do not fit in the
configured LittleFS partition or for frequent large-file transfers.

The available LittleFS capacity and flash wear depend on the selected partition
layout, flash chip, filesystem wear leveling, and write pattern. A conservative
order-of-magnitude estimate for repeated uploads of roughly 2 MB files is
several thousand to tens of thousands of cycles, not a fixed limit. This is
unlikely to be problematic for ordinary hobbyist use. Users concerned about
internal-flash wear can use an SD card connected to the ESP32 as the transfer
source instead.

The implementation should document this distinction more explicitly, but it is
inaccurate to describe internal flash as the only supported source or to claim
that BFT itself repeatedly writes multi-megabyte temporary files there.

API and WebUI

The HTTP endpoint is the ESP3D service API, not the intended end-user workflow.

The corresponding user-facing integration is already provided by the
separately submitted ESP3D-WEBUI PR:

luc-github/ESP3D-WEBUI#460

The statement that the companion WebUI PR is "mentioned but not provided" is
factually incorrect. The PR is listed in this PR's References section as
luc-github/ESP3D-WEBUI#460.

It presents the action beside local files, shows progress, and reports that
the terminal is unavailable during the exclusive serial session. The split is
intentional because ESP3D and ESP3D-WEBUI are separate repositories.

Automatically routing every browser upload to the printer SD would be a
different workflow and is not appropriate as an implicit default. Users may
want to retain files in ESP storage, inspect them, or select an ESP-attached
SD card as their source.

There is currently no equivalent printer-SD upload implementation in
ESP3D 3.1 to extend. The old ASCII serial-upload code is not present
in the current codebase. Its removal is not evidence against BFT. The cited
problems -- slow text transfer, firmware dependence, and corruption from
uncontrolled serial traffic, particularly with a Serial TFT -- describe the
old ASCII path, where received text could become file content.

BFT is not a reintroduction of that workflow "with a faster underlying
protocol." It has a different transport model: negotiated capability,
structured binary frames, per-frame checksums, acknowledgments and retries,
and validation before a frame can write file content. In ESP3D, the generic
exclusive printer-link API also disconnects its terminal bridge and disables
polling during the transfer. It cannot control an independently connected
physical host, but unsolicited bytes do not become G-code in the destination
file as they could in the old text path. They are rejected, trigger recovery,
or make the transfer fail. The hardware transfers and independent M35
verification reported above demonstrate this in the intended use case.

Please test this PR with a Marlin build that enables
BINARY_FILE_TRANSFER. It provides a real, practical benefit: a WebUI user
can transfer multi-megabyte G-code files to the printer SD card reliably in a
usable time.

Clarification needed

If there is a preferred generic abstraction for printer-SD transfer in ESP3D,
please identify it. I am happy to adapt the implementation to that
architecture, but the feature should be evaluated against what it actually
does rather than against the behavior of the old text M28 / M29 path.

[luc-github]

Ho man this is too long...and you missed/twisted several points... like in other PR

An unmaintained OctoPrint plugin is not evidence that the Marlin protocol is
unusable

This was not what it was written....

Anyway if the BTF is no more experimental, can you share the link of the documetation API and the release message ?

[mennucc]

the protocol is documented here
https://github.com/MarlinFirmware/Marlin/blob/bugfix-2.1.x/docs/BinaryFileTransferProtocol.md

[mennucc]

The first Marlin release to include the binary transfer protocol was Marlin 2.0.0 (Dec 2019), whose release notes explicitly list “Optimized SD binary file transfer option” among the new features.

Notably, the release notes do not describe this feature as experimental

[luc-github]

Man do not overlook things, as I wrote you should check why is not already implemented:
Commit for this realease state it is not stable = experimental because it is Marlin way to introduce features even in releases : MarlinFirmware/Marlin#14817 (you need to read commits comments not only release general descriptons) and the comments mention in 2020 there are issues in transfer
and I am linked in this because of #327 where the dev owner stated himself :

 it was a POC PR that got merged really, I would like to tweak the protocol a bit make error recovery more reliable, and a few other things, before it becomes an official feature.

I did not saw any update for this.

the document is same from MarlinFirmware/Marlin#26570
which also mention is not final because of owner statement, and document do mention any versionning of the protocol because it is a draft.

Again I never saw a commit changing the status, and this discussion has been too long with always fuzzy explainations ,

Before pushing a PR is always better to do some research first then open a ticket to discuss the approach / feasability / acceptability - it would avoid this kind discussion which go in circle.
no mention the target is ESP32 when ESP3D also support ESP8266 no impact analysis, clear speed transfert in several conditions, you mention 2M 2min = 2048/120 = 17KB/s when BTT protocol do 110KB/s and as your wrote need clear scope, documentation, and known limitations.

About your question:
If there is a preferred generic abstraction for printer-SD transfer in ESP3D,
yes it is and you should already noticed it when doing your research: #575

Thank you for your PR but it won't be merged:

1 - This feature is appeliing but still not final : I do not implement experimental feature because I do not have time to support it.
2 - your implementation in 2 steps is not a proper approach - an upload is 1 step : upload
3 - I prefer to work on #575