fix: v32 tokenizer for transformers 5.x#1326
Conversation
There was a problem hiding this comment.
Code Review
This pull request adds support for the deepseek_v32 model type by registering it as an alias of DeepseekV3Config within HuggingFace's AutoConfig, updates the server tokenizer to trigger this registration, and adjusts the evaluation documentation and parameters for DeepSeek-V3.2. Feedback on these changes identifies a critical issue where the regex package was updated to a non-existent version (2026.5.9), which will break dependency installation. Additionally, it is recommended to narrow down the broad exception handling in the registration module to specific exceptions like ImportError and ValueError to prevent masking other unexpected errors.
| PyYAML==6.0.1 | ||
| pyzmq==25.1.1b2 | ||
| regex==2023.6.3 | ||
| regex==2026.5.9 |
There was a problem hiding this comment.
The version 2026.5.9 for the regex package does not exist on PyPI (as the year 2026 has not yet occurred/concluded). This will cause dependency installation to fail with a Could not find a version that satisfies the requirement error. Please use a valid, existing version of the regex package (e.g., 2024.11.6 or revert to 2023.6.3).
regex==2024.11.6
| except Exception: | ||
| # Older transformers without deepseek_v3, or a build that already | ||
| # supports deepseek_v32 natively. Nothing to do in either case. | ||
| pass |
There was a problem hiding this comment.
Catching a broad Exception and silently passing is discouraged as it can mask unexpected errors (such as NameError, TypeError, or syntax issues) during the registration process. It is safer to catch specific exceptions like ImportError (if the DeepSeek-V3 config module is missing) and ValueError (if registration fails), or at least log/warn about the exception to aid in debugging.
| except Exception: | |
| # Older transformers without deepseek_v3, or a build that already | |
| # supports deepseek_v32 natively. Nothing to do in either case. | |
| pass | |
| except (ImportError, ValueError): | |
| # Older transformers without deepseek_v3, or a build that already | |
| # supports deepseek_v32 natively. Nothing to do in either case. | |
| pass |
No description provided.