Skip to content

Reduce memory of summarize step by dropping unused variables before wide merge - #1125

Open
i-am-sijia wants to merge 3 commits into
ActivitySim:mainfrom
wsp-sag:reduce-summarize-mem
Open

i-am-sijia wants to merge 3 commits into
ActivitySim:mainfrom
wsp-sag:reduce-summarize-mem

Conversation

@i-am-sijia

Copy link
Copy Markdown
Member

This PR addresses #1124.

This pull request introduces a new feature to the summarize model that trims unused columns from tables before merging, improving memory usage and performance. It also adds a corresponding test to ensure this behavior works as intended. The most important changes are grouped below:

Core feature: Trimming unused columns

  • Added a new function trim_unused_columns to summarize.py that drops columns from input tables that are not referenced in the summarize specification or related preprocessor specs, with logic to retain necessary columns for merging and aggregation.
  • Introduced a new settings flag DROP_UNUSED_COLUMNS (default True) in SummarizeSettings to control whether unused columns are dropped, with logic to skip trimming if EXPORT_PIPELINE_TABLES is True.
  • Integrated trim_unused_columns into the main summarize function so that tables are trimmed before merging, based on the new setting.

Refactoring and utility

  • Extracted logic for deleting temporary variables into a new helper function delete_temporary_variables for clarity and reuse.

Testing

  • Added a new test test_summarize_drop_unused_columns to verify that columns are correctly trimmed when the feature is enabled, and that the output remains correct and consistent.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant