Skip to content

fix: v32 tokenizer for transformers 5.x#1326

Open
shihaobai wants to merge 3 commits into
mainfrom
transformers_fix
Open

fix: v32 tokenizer for transformers 5.x#1326
shihaobai wants to merge 3 commits into
mainfrom
transformers_fix

Conversation

@shihaobai
Copy link
Copy Markdown
Collaborator

No description provided.

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds support for the deepseek_v32 model type by registering it as an alias of DeepseekV3Config within HuggingFace's AutoConfig, updates the server tokenizer to trigger this registration, and adjusts the evaluation documentation and parameters for DeepSeek-V3.2. Feedback on these changes identifies a critical issue where the regex package was updated to a non-existent version (2026.5.9), which will break dependency installation. Additionally, it is recommended to narrow down the broad exception handling in the registration module to specific exceptions like ImportError and ValueError to prevent masking other unexpected errors.

Comment thread requirements.txt
PyYAML==6.0.1
pyzmq==25.1.1b2
regex==2023.6.3
regex==2026.5.9
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

The version 2026.5.9 for the regex package does not exist on PyPI (as the year 2026 has not yet occurred/concluded). This will cause dependency installation to fail with a Could not find a version that satisfies the requirement error. Please use a valid, existing version of the regex package (e.g., 2024.11.6 or revert to 2023.6.3).

regex==2024.11.6

Comment on lines +22 to +25
except Exception:
# Older transformers without deepseek_v3, or a build that already
# supports deepseek_v32 natively. Nothing to do in either case.
pass
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Catching a broad Exception and silently passing is discouraged as it can mask unexpected errors (such as NameError, TypeError, or syntax issues) during the registration process. It is safer to catch specific exceptions like ImportError (if the DeepSeek-V3 config module is missing) and ValueError (if registration fails), or at least log/warn about the exception to aid in debugging.

Suggested change
except Exception:
# Older transformers without deepseek_v3, or a build that already
# supports deepseek_v32 natively. Nothing to do in either case.
pass
except (ImportError, ValueError):
# Older transformers without deepseek_v3, or a build that already
# supports deepseek_v32 natively. Nothing to do in either case.
pass

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant