Skip to content

fix(tokenizer): skip blank segmentation entries with regression CI - #3905

Merged
LauraGPT merged 2 commits into
mainfrom
codex/seg-dict-blank-lines-ci-20261115
Nov 15, 2026
Merged

LauraGPT merged 2 commits into
mainfrom
codex/seg-dict-blank-lines-ci-20261115

Conversation

@LauraGPT

Copy link
Copy Markdown
Collaborator

Summary

Integrate #3903 while preserving the original author commit. Skip blank/whitespace-only segmentation-dictionary lines instead of raising IndexError. Include the new regression file in both existing NumPy matrix event filters and the pytest command.

Verification

  • Private CPU control: baseline 1 expected IndexError failure/3 passes; candidate 4 passes. Original author test bytes retained.
  • Real L20Z native CharTokenizer -> TokenIDConverter -> CTC forward/softmax, FP32, batches 1/2 with clean/LF-blank/CRLF-blank dictionaries: baseline 2 controls pass/4 expected IndexErrors; candidate 6 pass. Identical synthetic weights and hidden states preserve clean token IDs, losses and probabilities. 42 baseline/126 candidate CUDA events. No pretrained ASR, training, throughput or quality claim.
  • Product and test bytes are hash-bound to these results on current main; accepted fix(timestamp): normalize sentence output without punctuation #3904 timestamp changes are preserved. No repeat of unchanged GPU work.
  • CI-selection checks fail before and pass after the three-line workflow update. The original PR green workflow omitted the new test; it is not counted as regression acceptance. New exact-head NumPy matrix and eventual main CI remain gates.
  • This is scoped CPU/GPU validation, not a private full-repository suite. The separate current-main full-repo attempt stopped on missing standalone funasr-onnx imports, not a pass. No new dependencies, weights, production changes or Nano work.
  • Signed+DCO integration with inline review; no self-approval or protection bypass.

MohammadHijjawi97 and others added 2 commits November 14, 2026 19:09
load_seg_dict indexed the first field of every line, so a blank or
whitespace-only line (for example a trailing empty line) raised
IndexError and CharTokenizer(seg_dict=...) could not be built.
Preserve #3903 author ancestry and cover its regression in the existing NumPy matrix.

Signed-off-by: LauraGPT <zhifu.gzf@alibaba-inc.com>
@LauraGPT
LauraGPT merged commit d8a48a8 into main Nov 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants