Skip to content

perf(calendar): pre-filter ICS data before parsing#4168

Merged
khassel merged 1 commit into
MagicMirrorOrg:developfrom
KristjanESPERANTO:test-calendar
Jun 15, 2026
Merged

perf(calendar): pre-filter ICS data before parsing#4168
khassel merged 1 commit into
MagicMirrorOrg:developfrom
KristjanESPERANTO:test-calendar

Conversation

@KristjanESPERANTO

@KristjanESPERANTO KristjanESPERANTO commented May 25, 2026

Copy link
Copy Markdown
Collaborator

This adds a pre-filter that drops out-of-window non-recurring events from the raw ICS data before node-ical parses it. Recurring events and anything we cannot classify cheaply are kept - the existing logic handles those as before.

Some calendars accumulate years of old events. Pre-filtering them reduces the amount of work node-ical has to do and keeps event loop lag lower. My benchmarks on a fast machine showed a small but measurable speedup (roughly 10-20% with generated test data); on a Raspberry Pi it should be significantly more noticeable.

For the implementation I ended up using ics-filter, suggested by @rejas. I had a custom version first, but ICS date string parsing has enough complexity that it makes sense to delegate it to a package built specifically for this. I also contributed a couple of fixes to ics-filter along the way, and the maintainer was responsive and open to improvements, so it seems like a good fit.

Solves #4103.

@rejas

rejas commented May 25, 2026

Copy link
Copy Markdown
Collaborator

Just a thought: would a library like https://github.com/runely/ics-filter offload some complexity from this PR?

@sdetweil

sdetweil commented May 25, 2026

Copy link
Copy Markdown
Collaborator

@rejas that filters out ALL past events (we want SOME, sometimes)

the filter will only filter out past events

@sdetweil

Copy link
Copy Markdown
Collaborator

i also replied in the issue, not in the PR.

@KristjanESPERANTO

Copy link
Copy Markdown
Collaborator Author

Thanks @rejas for the pointer to ics-filter! I had a closer look and did some tests - I think the general approach of that package fits well, though there are a few edges. I just opened a first PR there to address one of them (case-insensitive property names): runely/ics-filter#1

I'm setting this to draft for now and waiting for the author to respond. I'd rather improve an existing package than maintain our own implementation here :)

@KristjanESPERANTO KristjanESPERANTO marked this pull request as draft May 25, 2026 21:24
@KristjanESPERANTO

Copy link
Copy Markdown
Collaborator Author

that filters out ALL past events (we want SOME, sometimes)

We can pass now - maximumNumberOfDays as the min date to icsFilter(), which is the same window filterEvents() (in ./defaultmodules/calendar/calendarfetcherutils.js) already uses for past events.

@KristjanESPERANTO

Copy link
Copy Markdown
Collaborator Author

Status update: The maintainer of ics-filter was open to my pull requests and has merged them :) From my view, ics-filter is now suitable for our use case.

@KristjanESPERANTO KristjanESPERANTO force-pushed the test-calendar branch 3 times, most recently from 18a0bce to 528add5 Compare June 13, 2026 09:00
@KristjanESPERANTO KristjanESPERANTO changed the title perf(calendar): pre-filter ICS text before parsing perf(calendar): pre-filter ICS data before parsing Jun 13, 2026
@KristjanESPERANTO KristjanESPERANTO marked this pull request as ready for review June 13, 2026 10:40
@KristjanESPERANTO

Copy link
Copy Markdown
Collaborator Author

I've switched the implementation to ics-filter and updated the PR text. The PR is ready for review 🙂

@khassel

khassel commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator

@KristjanESPERANTO let me know if you want to wait for others to approve, otherwise I will merge ...

@KristjanESPERANTO

Copy link
Copy Markdown
Collaborator Author

I think one reviewer is enough here, even though I might be biased since this is my own PR 😅

@khassel khassel merged commit ef2c7db into MagicMirrorOrg:develop Jun 15, 2026
12 checks passed
@KristjanESPERANTO KristjanESPERANTO deleted the test-calendar branch June 15, 2026 21:42
@rejas rejas mentioned this pull request Jul 1, 2026
rejas added a commit that referenced this pull request Jul 1, 2026
## Release Notes
Thanks to: @angeldeejay, @egeekial, @khassel, @KristjanESPERANTO,
@MikeBishop, @rejas
> ⚠️ This release needs nodejs version >=22.21.1 <23 || >=24 (no change
to previous release)

[Compare to previous Release
v2.36.0](v2.36.0...develop)


### [core]
- Prepare Release 2.37.0 (#4193)
- fix(electron): map IPv6 :: wildcard to localhost (#4188)
- refactor(main): modernize DOM update flow with async/await (#4186)
- refactor(main): simplify _updateDom with async/await (#4185)
- fix(security): prevent unauthorized secret expansion in socket
payloads (#4184)
- refactor(main): simplify updateDomWithContent async flow (#4182)
- fix: modules losing data after HTTP 304 responses (#4180)
- chore: add missing core defaults (#4181)
- fix(server): enforce ipWhitelist for Socket.IO too (#4169)
- feat(systeminfo): include Git hash and branch in system information
log (#4167)
- feat(electron): support object-based electronSwitches (#4161)
- systeminformation thread not ending: move error handling from utils to
app (#4160)
- fix systeminformation thread not ending (#4155)
- refactor: use ES module imports in browser core (#4158)
- refactor(core): remove old Object.assign polyfill (#4157)
- refactor: rewrite Module as an ES6 class (#4151)
- refactor: rewrite NodeHelper as an ES6 class (#4147)
- update eletron to v42 (#4144)
- refactor(utils): drop ajv dependency (#4142)
- fix(systeminformation): output right 'used node' version (from parent
process) (#4141)
- fix: skip postinstall git clean when not in a git repository (#4139)
- Remove unnecessary conditionals and fix falsy property check in
imperial conversion (#4135)
- update version in package.json

### [dependencies]
- update dependencies (#4191)
- Bump actions/checkout from 6 to 7 (#4190)
- chore: update dependencies and adjust import path for SunCalc (#4189)
- update dependencies incl. electron and revert
yauzl-electron-install-fix (#4183)
- update dependencies, add electron fix in package.json (#4175)
- chore: update dependencies (#4162)
- Bump actions/dependency-review-action from 4 to 5 (#4152)
- Unify linting: replace Stylelint and markdownlint with ESLint (#4148)
- update dependencies and workflows to node v26 (#4140)

### [modules/alert]
- CodeQL cleanup for alerts #18, #19, #20 (#4153)
- fix: resolve CodeQL alerts #24 and #26 (#4145)
- fix(electron): resolve CodeQL alerts #22 and #25 in electron.js
(#4136)

### [modules/calendar]
- perf(calendar): pre-filter ICS data before parsing (#4168)
- perf(calendar): use async ICS parsing to avoid blocking event loop
(#4143)

### [modules/newsfeed]
- [newsfeed] add allowBasicHtmlTags option for basic emphasis (#4176)

### [modules/updatenotification]
- fix(updatenotification): don't spawn a child process when running
under PM2 (#4166)
- fix(updatenotification): use process.argv[0] as restart binary (#4163)
- fix(updatenotification): preserve start mode on restart (#4156)
- fix(updatenotification): fix ref diff parsing for fetch --dry-run
(#4138)
- refactor(updatenotification): replace pm2 usage with node logic
(#4134)

### [modules/weather]
- feat(weather): add Buienradar provider (#4164)

### [testing]
- remove warning in unit tests (for nodejs >= v25) (#4149)
- polish HTTP 304 docs/test/handling (#4129)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Kristjan ESPERANTO <35647502+KristjanESPERANTO@users.noreply.github.com>
Co-authored-by: BugHaver <43462320+bughaver@users.noreply.github.com>
Co-authored-by: BugHaver <43462320+lsaadeh@users.noreply.github.com>
Co-authored-by: Karsten Hassel <hassel@gmx.de>
Co-authored-by: Magnus <34011212+MagMar94@users.noreply.github.com>
Co-authored-by: Koen Konst <koenspero@gmail.com>
Co-authored-by: Koen Konst <c.h.konst@avisi.nl>
Co-authored-by: Bugsounet - Cédric <github@bugsounet.fr>
Co-authored-by: dathbe <github@beffa.us>
Co-authored-by: veeck <gitkraken@veeck.de>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marcel <m-idler@users.noreply.github.com>
Co-authored-by: sam detweiler <sdetweil@gmail.com>
Co-authored-by: Kevin G. <crazylegstoo@gmail.com>
Co-authored-by: Jboucly <33218155+jboucly@users.noreply.github.com>
Co-authored-by: Jboucly <contact@jboucly.fr>
Co-authored-by: Jarno <54169345+jarnoml@users.noreply.github.com>
Co-authored-by: Jordan Welch <JordanHWelch@gmail.com>
Co-authored-by: Blackspirits <blackspirits@gmail.com>
Co-authored-by: Samed Ozdemir <samed@xsor.io>
Co-authored-by: in-voker <58696565+in-voker@users.noreply.github.com>
Co-authored-by: Andrés Vanegas Jiménez <142350+angeldeejay@users.noreply.github.com>
Co-authored-by: cgillinger <christian.gillinger@gmail.com>
Co-authored-by: Sonny B <43247590+sonnyb9@users.noreply.github.com>
Co-authored-by: sonnyb9 <sonnyb9@users.noreply.github.com>
Co-authored-by: Morgan McBee <egeekial@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Mike Bishop <mbishop@evequefou.be>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants