Skip to content

Remove pandas inplace and 3.1 deprecations - #1477

Merged
VisLab merged 2 commits into
hed-standard:mainfrom
VisLab:pandas_fix
Oct 22, 2026
Merged

VisLab merged 2 commits into
hed-standard:mainfrom
VisLab:pandas_fix

Conversation

@VisLab

@VisLab VisLab commented Oct 22, 2026

Copy link
Copy Markdown
Member

pandas 3.1 deprecates inplace= (Pandas4Warning, a DeprecationWarning) and pandas 4 removes it; under pandas 3.1 every validation with onsets warned and tests/test_deprecations.py failed.

  • Remove the seven inplace= uses (df_util, key_map, data_util); ruff rule PD002 rejects new ones.
  • Sort by onset with a stable sort, so rows with equal onsets keep their file order.
  • Add data_util.remove_rows_by_value, which returns a new dataframe; deprecate delete_rows_by_column (removed in hedtools 2.0.0).
  • KeyMap.remap fills "n/a" only into the target columns, and merge_dfs adds columns with .loc instead of .at (both pandas 3.1 deprecations).
  • Tests: stable sort, the deprecation, the new function, and a class that runs each changed function with DeprecationWarning as an error.

pandas 3.1 deprecates inplace= (Pandas4Warning, a DeprecationWarning)
and pandas 4 removes it; under pandas 3.1 every validation with onsets
warned and tests/test_deprecations.py failed.

- Remove the seven inplace= uses (df_util, key_map, data_util); ruff
  rule PD002 rejects new ones.
- Sort by onset with a stable sort, so rows with equal onsets keep
  their file order.
- Add data_util.remove_rows_by_value, which returns a new dataframe;
  deprecate delete_rows_by_column (removed in hedtools 2.0.0).
- KeyMap.remap fills "n/a" only into the target columns, and merge_dfs
  adds columns with .loc instead of .at (both pandas 3.1 deprecations).
- Tests: stable sort, the deprecation, the new function, and a class
  that runs each changed function with DeprecationWarning as an error.
@VisLab
VisLab requested a balanced review from Copilot October 22, 2026 19:18
@VisLab VisLab added dependencies Pull requests that update a dependency file python Pull requests that update Python code maintenance required-update labels Oct 22, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The version-sensitive pandas behavior still needs compatibility-test confirmation across supported releases.

No open issues 🎉

What changed in this PR

This PR updates dataframe utilities for pandas compatibility while preserving stable event ordering and introducing a non-mutating row-removal API.

Changes:

  • Replace most inplace=True operations and enforce Ruff rule PD002, retaining an exception for the deprecated compatibility wrapper.
  • Preserve file order for tied onsets and adjust target-column filling and schema-column assignment.
  • Add regression coverage for row removal, deprecation warnings, and pandas compatibility.
File Description
tests/​tools/​util/​test_data_util.py Test non-mutating row removal and the deprecated API.
tests/​test_pandas3_compat.py Cover stable sorting and warning-free dataframe operations.
tests/​test_deprecations.py Track removal of the deprecated row-deletion API.
pyproject.toml Enable Ruff's PD002 rule.
hed/​tools/​util/​data_util.py Add non-mutating row removal and deprecate the in-place API.
hed/​tools/​analysis/​key_map.py Replace in-place sorting and restrict missing-value filling to targets.
hed/​schema/​schema_io/​hed_id_util.py Use .loc when copying columns into matching rows.
hed/​models/​df_util.py Use stable onset sorting and replace in-place operations.

Review effort: Balanced

Every build job installed the newest pandas its Python allows, so the
floor in pyproject.toml (pandas>=2.2.3) was never tested. Add one job,
Python 3.10 with pandas 2.2.3, through a pandas matrix key and an
include entry. An explicit job name keeps the existing check names.
@VisLab
VisLab merged commit 9e4d2b9 into hed-standard:main Oct 22, 2026
21 checks passed
@VisLab
VisLab deleted the pandas_fix branch October 22, 2026 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file maintenance python Pull requests that update Python code required-update

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants