Skip to content

Add --min-word-length option to ignore short words - #3971

Open
ekanshul wants to merge 1 commit into
codespell-project:mainfrom
ekanshul:min-word-length-2211
Open

Add --min-word-length option to ignore short words#3971
ekanshul wants to merge 1 commit into
codespell-project:mainfrom
ekanshul:min-word-length-2211

Conversation

@ekanshul

Copy link
Copy Markdown

Closes #2211

Adds a --min-word-length LENGTH option so that words shorter than LENGTH are not checked. This helps with codebases full of 2-3 letter variable names that would otherwise each need an entry in --ignore-words-list.

Implementation: entries shorter than LENGTH are dropped from the misspelling dictionary right after it is built, so short words are ignored consistently everywhere (file contents, stdin and --check-filenames) with no per-word cost during scanning. The default (0) keeps current behavior.

Example:

$ codespell file.txt
file.txt:1: nwe ==> new
file.txt:1: abandonned ==> abandoned
$ codespell --min-word-length 4 file.txt
file.txt:1: abandonned ==> abandoned

Tests included (test_min_word_length) covering the length boundaries, case-insensitivity and --check-filenames.

Words shorter than the given length are dropped from the misspelling
dictionary up front, so they are ignored both in file contents and
with --check-filenames. This avoids having to list every short
variable name in --ignore-words-list.

Closing: codespell-project#2211
@chuenchen309

Copy link
Copy Markdown

Applied this branch and exercised it rather than just reading it — it works, and filtering at the dictionary level is the right layer. Sharing the checks in case they're useful to a maintainer:

$ cat short.txt
The aas value and the abandonned config.

$ codespell short.txt
short.txt:1: aas ==> ass, as
short.txt:1: abandonned ==> abandoned

$ codespell --min-word-length 4 short.txt
short.txt:1: abandonned ==> abandoned          # short key dropped, long one still caught

The one thing I went looking for was whether len(key) could mean something other than "word length" — a multi-word dictionary key would make --min-word-length 4 compare against the whole phrase rather than a word. It can't: no builtin dictionary has a key containing a space (I checked all of codespell_lib/data/dictionary*.txt — zero). So len(key) is unambiguously the word length here. Worth knowing it holds by data rather than by construction, in case custom -D dictionaries are ever expected to carry phrases.

Filtering the dictionary once after build_dict rather than at each match site also means --check-filenames and the -w path get the same treatment for free, which matches what the PR description claims.

For context on the size of what it buys: 162 of the default dictionary's keys are ≤3 characters, so --min-word-length 4 is a meaningful reduction for a codebase full of short identifiers — the use case in #2211.

(Not a maintainer, just a contributor who was recently in this file.)

@ekanshul

Copy link
Copy Markdown
Author

Gentle ping on this one — it's been about four weeks, so I wanted to check it hadn't just slipped off the radar.

One thing that may be worth a maintainer's attention: the GitHub Actions workflows on this PR have never actually run. All five runs on the head commit (Pytest, Build, codespell, autofix.ci) sit at completed/action_required, which is the first-time-contributor approval gate. pre-commit.ci is green, but since the required workflows never started, the PR reports as blocked on missing checks rather than on anything in the diff — so there's currently no way for me to show it passing from my side. Approving the workflow run should clear that.

It isn't specific to this PR: there are currently 42 runs in action_required across the repo, so a few other contributors are likely stuck the same way.

For context, this closes #2211, which is labelled good first issue, and @peternewman's last comment there invited anyone to pick it up. Thanks also to @chuenchen309, who applied the branch and verified the behaviour independently above.

The same gate applies to #3972 and #3973 if they're being triaged together.

Happy to rebase, split this up, or take a different approach if you'd prefer it done another way — just let me know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Whishlist: Optionally ignore short words

3 participants