Skip to content

Skip blank byte lines when reading gzipped JSONL - #125

Merged
honnibal merged 1 commit into
explosion:masterfrom
sueun-dev:fix-gzip-jsonl-blank-lines-20260918
Nov 3, 2026
Merged

honnibal merged 1 commit into
explosion:masterfrom
sueun-dev:fix-gzip-jsonl-blank-lines-20260918

Conversation

@sueun-dev

Copy link
Copy Markdown
Contributor

read_gzip_jsonl passes binary lines to _yield_json_lines, where the empty-string check misses b"" after stripping whitespace. This makes a write_gzip_jsonl(..., append=True) / read_gzip_jsonl(...) round trip raise ValueError on the separator inserted by the writer.

Use an emptiness check that handles both text and bytes. The tests cover the append round trip, blank and whitespace-only lines, UTF-8 content, both skip settings, malformed JSON, and falsy JSON values.

Local validation on Linux arm64:

  • Before the fix: 3 failures and 36 passes in the JSON tests with the new regressions.
  • After the fix: 39 JSON tests pass; the full package suite passes 98 tests on Python 3.12 with NumPy installed.
  • On Python 3.14 without NumPy: 72 tests pass, 2 skip.
  • 1,000 generated gzip/plain append round trips agree with Python's gzip and json readers.
  • Source distribution and wheel builds pass. Other OS/Python configurations in CI were not run locally.

Prepared with Codex assistance.

@sueun-dev

Copy link
Copy Markdown
Contributor Author

A gentle follow-up on this one. The branch is still up to date with master (050951e), and the Test workflow hasn't run any jobs on this PR yet. The same line == "" check is also on v2.x, where the 2.5.4 and 2.5.5 releases were tagged, in case you'd want the fix there too. Happy to adjust anything if you'd prefer a different approach.

@honnibal honnibal closed this Nov 3, 2026
@honnibal honnibal reopened this Nov 3, 2026
@honnibal
honnibal merged commit c9c07bc into explosion:master Nov 3, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants