Repository navigation
Remove pandas inplace and 3.1 deprecations - #1477
Merged
Merged
Conversation
pandas 3.1 deprecates inplace= (Pandas4Warning, a DeprecationWarning) and pandas 4 removes it; under pandas 3.1 every validation with onsets warned and tests/test_deprecations.py failed. - Remove the seven inplace= uses (df_util, key_map, data_util); ruff rule PD002 rejects new ones. - Sort by onset with a stable sort, so rows with equal onsets keep their file order. - Add data_util.remove_rows_by_value, which returns a new dataframe; deprecate delete_rows_by_column (removed in hedtools 2.0.0). - KeyMap.remap fills "n/a" only into the target columns, and merge_dfs adds columns with .loc instead of .at (both pandas 3.1 deprecations). - Tests: stable sort, the deprecation, the new function, and a class that runs each changed function with DeprecationWarning as an error.
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
The version-sensitive pandas behavior still needs compatibility-test confirmation across supported releases.
No open issues 🎉
What changed in this PR
This PR updates dataframe utilities for pandas compatibility while preserving stable event ordering and introducing a non-mutating row-removal API.
Changes:
- Replace most
inplace=Trueoperations and enforce Ruff rulePD002, retaining an exception for the deprecated compatibility wrapper. - Preserve file order for tied onsets and adjust target-column filling and schema-column assignment.
- Add regression coverage for row removal, deprecation warnings, and pandas compatibility.
| File | Description |
|---|---|
| tests/tools/util/test_data_util.py | Test non-mutating row removal and the deprecated API. |
| tests/test_pandas3_compat.py | Cover stable sorting and warning-free dataframe operations. |
| tests/test_deprecations.py | Track removal of the deprecated row-deletion API. |
| pyproject.toml | Enable Ruff's PD002 rule. |
| hed/tools/util/data_util.py | Add non-mutating row removal and deprecate the in-place API. |
| hed/tools/analysis/key_map.py | Replace in-place sorting and restrict missing-value filling to targets. |
| hed/schema/schema_io/hed_id_util.py | Use .loc when copying columns into matching rows. |
| hed/models/df_util.py | Use stable onset sorting and replace in-place operations. |
Review effort: Balanced
Every build job installed the newest pandas its Python allows, so the floor in pyproject.toml (pandas>=2.2.3) was never tested. Add one job, Python 3.10 with pandas 2.2.3, through a pandas matrix key and an include entry. An explicit job name keeps the existing check names.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pandas 3.1 deprecates inplace= (Pandas4Warning, a DeprecationWarning) and pandas 4 removes it; under pandas 3.1 every validation with onsets warned and tests/test_deprecations.py failed.