Skip to content

fix(gzip): stop holding decompressed uploads in memory - #581

Open
CyMule wants to merge 5 commits into
perf/no-copy-mime-detectionfrom
fix/gzip-decompression-memory
Open

CyMule wants to merge 5 commits into
perf/no-copy-mime-detectionfrom
fix/gzip-decompression-memory

Conversation

@CyMule

@CyMule CyMule commented Jul 22, 2026 •

Copy link
Copy Markdown
Contributor

What

Gzip uploads are decompressed in 1 MiB chunks into a temporary file that stays in memory up to 1 MiB and spills to disk above that. Previously the whole expanded payload was held in memory.

Before: .gz upload → gzip.read() → bytes (full size, in RAM) → new UploadFile (never closed by the request)
After:  .gz upload → 1 MiB chunks → SpooledTemporaryFile (disk above 1 MiB) → same request-owned UploadFile
  • The decompressed file sits on the request's own UploadFile, so FastAPI closes it when the request finishes. Partial output is closed if the gzip data is corrupt or truncated.
  • No size limit is introduced, and the partition API is unchanged.

Why

ungz_file held the expanded payload as one bytes object for the whole request, so peak memory grew with the uncompressed size.

An earlier revision passed large outputs downstream behind a proxy object, to stop unstructured from copying them back into memory. That proxy made convert_to_bytes() reject the file: ValueError: Invalid file-like object type for gzipped uploads over 1 MiB on the PDF OCR and text-encoding-detection paths. This revision passes the spool itself. Avoiding the downstream copies now comes from unstructured#4419 (open; expected to ship as 0.27.12).

Merge dependency

Stacked on #579. Released as 0.1.13, after #579's 0.1.12. The decompression-stage savings apply on any unstructured version. The end-to-end savings below need the unstructured#4419 release (expected 0.27.12); this PR does not change the lock. On 0.22.18 the DOCX/PPTX/PDF partitioners still copy the spool into memory.

Impact

Measured on the earlier revision, whose downstream path matches unstructured#4419. Responses were byte-identical:

  • 54.9 MB DOCX: peak RSS 828.6 → 759.9 MiB; latency 1.348s → 1.370s.
  • 70.4 MB PDF: peak RSS 1,029.5 → 951.6 MiB; latency 0.532s → 0.529s.

Risk

A very large expansion now uses temporary disk instead of RAM. Disk capacity is the external bound, and a decompressed-size cap would be a product decision for the upload-limits work.

Validation

  • test_gzip.py + test_filetypes.py pass on unstructured 0.22.18, including an end-to-end gzipped request that asserts the decompressed file is closed after the response (it fails against main's ungz_file). The same tests pass with the unstructured#4419 branch installed over the locked dependencies. That includes a regression test that reads a disk-backed decompressed upload through convert_to_bytes(); it fails with the error above on the earlier proxy revision.
  • Ruff passed.

Decompress gzip uploads in 1 MiB chunks into a spooled temporary file that
spills to disk above 1 MiB, instead of materializing the expanded payload as
bytes plus a BytesIO copy. Reuse the request-owned UploadFile so FastAPI
closes the decompressed file when the request finishes, and close partial
output when decompression fails.
@CyMule
CyMule force-pushed the fix/gzip-decompression-memory branch 2 times, most recently from 9a54345 to 8c00ffd Compare October 1, 2026 16:56
@CyMule
CyMule changed the base branch from main to perf/no-copy-mime-detection October 1, 2026 16:56
@CyMule CyMule changed the title fix(gzip): bound decompression memory fix(gzip): stop holding decompressed uploads in memory Oct 1, 2026
@CyMule
CyMule marked this pull request as ready for review October 1, 2026 17:15

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 3 files

Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic

Comment thread prepline_general/api/general.py

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 3 files (changes from recent commits).

Shadow auto-approve: would not auto-approve. Auto-approval blocked by 1 unresolved issue from previous reviews.

Re-trigger cubic

@CyMule
CyMule marked this pull request as draft October 2, 2026 03:33
@CyMule
CyMule marked this pull request as ready for review October 2, 2026 04:13
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 4 files

Shadow auto-approve: would not auto-approve. This PR does not meet the repository auto-approval settings.

Re-trigger cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant