Skip to content

[feature] Make pandas optional and align Frequency with polars - #286

Open
ptomecek wants to merge 1 commit into
mainfrom
feature/optional-pandas-frequency
Open

ptomecek wants to merge 1 commit into
mainfrom
feature/optional-pandas-frequency

Conversation

@ptomecek

@ptomecek ptomecek commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Description

This makes pandas an optional dependency. So that ccflow is not coupled to another library, Frequency gets its own implementation, which follows the polars duration language. It is a breaking change and should be released as 0.10.0.

pip install ccflow no longer installs pandas. Frequency, the context classes, normalize_date/normalize_datetime and dates_to_chunks are pure Python and work without pandas; PandasResult, XArrayResult, PandasFilePublisher and Frequency.offset import pandas when used.

Frequency is now a value type. Its canonical string is an integer count and a polars unit (ns, us, ms, s, m, h, d, w, mo, q, y), e.g. 5m, 1h, 1d, 1mo, and the units mean what they mean in polars: ns to h are fixed durations and d, w, mo, q, y are calendar units, so 1d and 24h are distinct. Equal values share one spelling (60m -> 1h, 3mo -> 1q). Frequency parses, validates and converts (n, unit, .timedelta, .offset) but does no date arithmetic: relative dates in normalize_date/normalize_datetime shift by mo/q/y keeping the day of month (polars offset_by), and dates_to_chunks returns polars truncate buckets. IntradayFrequency and CalendarFrequency restrict a field to sub-day or calendar frequencies.

The pandas spellings T, min, H, S, L, U, N and D are still accepted in any case, with a FutureWarning. ISO-8601 durations with integer components (PT5M, P1M), timedeltas and pandas offsets are also accepted and normalized. Other pandas spellings are rejected, as listed below.

Breaking changes

Installation

  • pip install ccflow no longer installs pandas. Code that uses PandasResult, XArrayResult, PandasFilePublisher or Frequency.offset, or that relied on ccflow to bring in pandas, must depend on pandas directly or install ccflow[full].

Strings that still parse but change meaning, with no warning (search configs for lowercase m, q, y and w before upgrading)

  • m was month-end and is now minute: Frequency.validate("5m") was 5ME and is now five minutes, and on 2026-10-15 normalize_datetime("-6m") was 2026-04-30 and is now 2026-10-14 23:54. normalize_date accepts only calendar units, so normalize_date("-6m") now raises.
  • y and q in relative dates moved to the year or quarter end and now keep the day of month: on 2026-10-15, normalize_date("1y") was 2026-12-31 and is now 2027-10-15, and normalize_date("-1q") was 2026-09-30 and is now 2026-07-15. Likewise normalize_date("-1mo") is 2026-09-15, where main's -1ME gave 2026-09-30.
  • w rolled to the next Sunday and is now a seven-day step: on 2026-10-15, normalize_date("1w") was 2026-10-18 and is now 2026-10-22.

Canonical strings and stored values

  • The canonical form changes (1D -> 1d, 5min -> 5m, 60min -> 1h, 1440min -> 24h, 0D -> 0s). Frequency is a str subclass, so this is the value in model dumps, cache keys and path templates; caches keyed on a Frequency must be regenerated.
  • Configs and JSON written by main that hold weekly, monthly, quarterly, yearly or business-day frequencies (1W-SUN, 1ME, 1QE-DEC, 1YE-DEC, 1B) fail validation; rewrite them as 1w, 1mo, 1q, 1y. Pickles written by main load but keep the old string and pandas offset; regenerate them.
  • Field defaults are not validated, so update them to canonical spellings (Frequency("1d")): a legacy default keeps its old string in dumps and cache keys and fails on first use of .n/.unit.
  • Comparisons against old spellings (ctx.freq == "1D") are now false.

Rejected inputs

  • M is ambiguous (use mo for month or m for minute). Canonical units are case-sensitive: US, NS and mixed-case Ms/mS (micro-, nano- and milliseconds on main) are rejected; write us, ns, ms.
  • pandas period-end, anchored, period-start, business-day, business-hour and semi-month spellings have no equivalent and are rejected: ME, QE, YE, A, uppercase Q/Y/W, any anchor suffix (QE-DEC, YE-JUN, W-SUN, W-MON), B, BH, BME, MS, QS, SM. Use mo, q, y and w. The matching offsets (MonthEnd, QuarterEnd, YearEnd, Week(weekday=6), BusinessDay, MonthBegin) are rejected too.
  • Compound and fractional strings (1d12h, 1h30min, 1.5h) are rejected; write 36h or PT36H. A timedelta that is not a whole number of days becomes a fixed duration (timedelta(hours=36) -> 36h).
  • normalize_date accepts only calendar units (d, w, mo, q, y), so sub-day relative dates such as -1h raise. normalize_datetime still accepts every unit.

Frequency.offset

  • ns to h give pandas ticks, d gives Day(n), w gives an unanchored Week(n) (main gave Week(n, weekday=6), so adding it rolled to Sunday), and mo/q/y give DateOffset(n, months=k) instead of MonthEnd/QuarterEnd/YearEnd. Because values are reduced, .offset follows the reduced unit: 60min now gives Hour(1), not Minute(60).
  • .offset is for adding to a timestamp. DateOffset is not a pandas frequency: resample rejects it, and date_range applies it step by step, so month-end dates drift (Jan 31, Feb 29, Mar 29). A Frequency string is not a pandas alias either: pandas 2 reads 5m as five month-ends and pandas 3 rejects 5m and 1mo.
  • 1D and pandas Day() are read as the calendar day 1d, although pandas 2 treats Day as a fixed 24 hours.
  • 1ns has no .timedelta (main returned a zero timedelta).

dates_to_chunks

  • Chunks are unchanged from 1970 on for every size that still parses. It raises for zero, negative and fixed sizes (1h, 5min, 24h; main returned malformed chunks) and for main's spellings M, ME and 2W; the default is now spelled 1mo. Fiscal-year (YE-JUN), W-MON and business-day chunking are gone.
  • It returns [] when start > end, and multi-period sizes (2mo, 2w, 3y) now work for ranges before 1970 (main returned []).

Warnings

  • pandas unit spellings (5min, 5T, 1D, D) emit a FutureWarning naming the canonical form, including in relative dates (date="-1D"); under -W error this is raised from model construction. No removal version is set. Canonical units are coerced silently, including signs, leading zeros, padding and reductions (+01d -> 1d, 60m -> 1h).

PandasFilePublisher

  • data is validated without importing pandas, and data=None is now accepted.

Type of Change

  • Bug fix
  • New feature
  • Documentation update
  • Refactor / code cleanup
  • CI / build configuration
  • Other (describe below)

Breaking change: release as 0.10.0.

Checklist

  • Linting passes (make lint)
  • Tests pass (make test)
  • New tests added for new functionality
  • Documentation updated (if applicable)
  • Changelog / version bump (if applicable)

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Test Results

    1 files  ± 0      1 suites  ±0   1m 56s ⏱️ -14s
1 578 tests +19  1 576 ✅ +19  2 💤 ±0  0 ❌ ±0 
1 644 runs  +19  1 642 ✅ +19  2 💤 ±0  0 ❌ ±0 

Results for commit baf6ec8. ± Comparison against base commit 48b6ea1.

This pull request removes 10 and adds 29 tests. Note that renamed tests count towards both.
ccflow.tests.exttypes.test_frequency.TestFrequency ‑ test_basic
ccflow.tests.exttypes.test_frequency.TestFrequency ‑ test_legacy_and_canonical_aliases_normalize_to_same_frequency
ccflow.tests.exttypes.test_frequency.TestFrequency ‑ test_validate_1D
ccflow.tests.exttypes.test_frequency.TestFrequency ‑ test_validate_1M
ccflow.tests.exttypes.test_frequency.TestFrequency ‑ test_validate_1Y
ccflow.tests.exttypes.test_frequency.TestFrequency ‑ test_validate_5T
ccflow.tests.exttypes.test_frequency.TestFrequency ‑ test_validate_bad
ccflow.tests.test_validators.TestValidators ‑ test_normalize_date_accepts_legacy_and_canonical_frequency_aliases
ccflow.tests.test_validators.TestValidators ‑ test_normalize_datetime_accepts_legacy_and_canonical_frequency_aliases
ccflow.tests.utils.test_chunker.TestChunker ‑ test_dates_to_chunks_accepts_legacy_and_canonical_month_aliases
ccflow.tests.exttypes.test_frequency.TestCalendarFrequency ‑ test_accepts_calendar
ccflow.tests.exttypes.test_frequency.TestCalendarFrequency ‑ test_rejects_non_calendar
ccflow.tests.exttypes.test_frequency.TestCanonicalContract ‑ test_cross_representation_canonical
ccflow.tests.exttypes.test_frequency.TestCanonicalContract ‑ test_offset_roundtrip
ccflow.tests.exttypes.test_frequency.TestCanonicalContract ‑ test_zero_is_canonical
ccflow.tests.exttypes.test_frequency.TestFrequency ‑ test_ambiguous_M_raises
ccflow.tests.exttypes.test_frequency.TestFrequency ‑ test_bad_input
ccflow.tests.exttypes.test_frequency.TestFrequency ‑ test_canonical_roundtrip
ccflow.tests.exttypes.test_frequency.TestFrequency ‑ test_directly_constructed_iso_value
ccflow.tests.exttypes.test_frequency.TestFrequency ‑ test_iso8601_input
…

♻️ This comment has been updated with latest results.

@codecov

codecov Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.96641% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.87%. Comparing base (48b6ea1) to head (baf6ec8).

Files with missing lines Patch % Lines
ccflow/exttypes/frequency.py 98.56% 1 Missing and 1 partial ⚠️
ccflow/tests/test_import.py 60.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #286      +/-   ##
==========================================
+ Coverage   93.83%   93.87%   +0.04%     
==========================================
  Files         203      203              
  Lines       22488    22677     +189     
  Branches     1458     1489      +31     
==========================================
+ Hits        21101    21289     +188     
- Misses       1099     1100       +1     
  Partials      288      288              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ptomecek
ptomecek force-pushed the feature/optional-pandas-frequency branch 6 times, most recently from fca3c80 to cc82137 Compare October 29, 2026 19:48
pip install ccflow no longer installs pandas. Frequency, the context
classes, normalize_date/normalize_datetime and dates_to_chunks are pure
Python; PandasResult, XArrayResult, PandasFilePublisher and
Frequency.offset import pandas when used, and pandas moves to the full,
develop and test extras.

Frequency is a value type whose canonical string is a count and a
polars unit (ns, us, ms, s, m, h, d, w, mo, q, y) with the polars
meaning: ns to h are fixed durations and d, w, mo, q, y are calendar
units, so 1d and 24h are distinct. Equal values share one spelling
(60m -> 1h, 3mo -> 1q). Frequency parses, validates and converts (n,
unit, .timedelta, .offset) and does no date arithmetic:

- normalize_date and normalize_datetime shift relative dates by
  mo/q/y keeping the day of month (polars offset_by); normalize_date
  accepts only calendar units.
- dates_to_chunks returns polars truncate buckets counted from
  1970-01-01 (weeks from Monday 1970-01-05). Chunks from 1970 on are
  unchanged, and multi-period chunks before 1970 now work.

The pandas spellings T, min, H, S, L, U, N and D are accepted with a
FutureWarning; ISO-8601 durations, timedeltas and pandas offsets are
accepted and normalized. M is rejected as ambiguous, and pandas
period-end, anchored, period-start, business-day and semi-month
spellings and offsets are rejected because they have no equivalent.
Frequency.offset returns pandas ticks, Day(n), Week(n) or
DateOffset(n, months=k).

PandasFilePublisher validates data without importing pandas, and the
installation guide notes that pandas is optional.

This is a breaking change; the pull request lists every change in
behaviour.

Signed-off-by: Pascal Tomecek <40371786+ptomecek@users.noreply.github.com>
@ptomecek ptomecek changed the title [feature] Make pandas an optional dependency [feature] Make pandas optional and align Frequency with polars Oct 29, 2026
@ptomecek
ptomecek force-pushed the feature/optional-pandas-frequency branch from cc82137 to baf6ec8 Compare October 29, 2026 22:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant