perf(calendar): pre-filter ICS data before parsing#4168
Conversation
|
Just a thought: would a library like https://github.com/runely/ics-filter offload some complexity from this PR? |
|
@rejas that filters out ALL past events (we want SOME, sometimes)
|
|
i also replied in the issue, not in the PR. |
|
Thanks @rejas for the pointer to I'm setting this to draft for now and waiting for the author to respond. I'd rather improve an existing package than maintain our own implementation here :) |
We can pass |
256b500 to
29bbb58
Compare
|
Status update: The maintainer of ics-filter was open to my pull requests and has merged them :) From my view, ics-filter is now suitable for our use case. |
18a0bce to
528add5
Compare
528add5 to
205b32a
Compare
|
I've switched the implementation to |
|
@KristjanESPERANTO let me know if you want to wait for others to approve, otherwise I will merge ... |
|
I think one reviewer is enough here, even though I might be biased since this is my own PR 😅 |
## Release Notes Thanks to: @angeldeejay, @egeekial, @khassel, @KristjanESPERANTO, @MikeBishop, @rejas >⚠️ This release needs nodejs version >=22.21.1 <23 || >=24 (no change to previous release) [Compare to previous Release v2.36.0](v2.36.0...develop) ### [core] - Prepare Release 2.37.0 (#4193) - fix(electron): map IPv6 :: wildcard to localhost (#4188) - refactor(main): modernize DOM update flow with async/await (#4186) - refactor(main): simplify _updateDom with async/await (#4185) - fix(security): prevent unauthorized secret expansion in socket payloads (#4184) - refactor(main): simplify updateDomWithContent async flow (#4182) - fix: modules losing data after HTTP 304 responses (#4180) - chore: add missing core defaults (#4181) - fix(server): enforce ipWhitelist for Socket.IO too (#4169) - feat(systeminfo): include Git hash and branch in system information log (#4167) - feat(electron): support object-based electronSwitches (#4161) - systeminformation thread not ending: move error handling from utils to app (#4160) - fix systeminformation thread not ending (#4155) - refactor: use ES module imports in browser core (#4158) - refactor(core): remove old Object.assign polyfill (#4157) - refactor: rewrite Module as an ES6 class (#4151) - refactor: rewrite NodeHelper as an ES6 class (#4147) - update eletron to v42 (#4144) - refactor(utils): drop ajv dependency (#4142) - fix(systeminformation): output right 'used node' version (from parent process) (#4141) - fix: skip postinstall git clean when not in a git repository (#4139) - Remove unnecessary conditionals and fix falsy property check in imperial conversion (#4135) - update version in package.json ### [dependencies] - update dependencies (#4191) - Bump actions/checkout from 6 to 7 (#4190) - chore: update dependencies and adjust import path for SunCalc (#4189) - update dependencies incl. electron and revert yauzl-electron-install-fix (#4183) - update dependencies, add electron fix in package.json (#4175) - chore: update dependencies (#4162) - Bump actions/dependency-review-action from 4 to 5 (#4152) - Unify linting: replace Stylelint and markdownlint with ESLint (#4148) - update dependencies and workflows to node v26 (#4140) ### [modules/alert] - CodeQL cleanup for alerts #18, #19, #20 (#4153) - fix: resolve CodeQL alerts #24 and #26 (#4145) - fix(electron): resolve CodeQL alerts #22 and #25 in electron.js (#4136) ### [modules/calendar] - perf(calendar): pre-filter ICS data before parsing (#4168) - perf(calendar): use async ICS parsing to avoid blocking event loop (#4143) ### [modules/newsfeed] - [newsfeed] add allowBasicHtmlTags option for basic emphasis (#4176) ### [modules/updatenotification] - fix(updatenotification): don't spawn a child process when running under PM2 (#4166) - fix(updatenotification): use process.argv[0] as restart binary (#4163) - fix(updatenotification): preserve start mode on restart (#4156) - fix(updatenotification): fix ref diff parsing for fetch --dry-run (#4138) - refactor(updatenotification): replace pm2 usage with node logic (#4134) ### [modules/weather] - feat(weather): add Buienradar provider (#4164) ### [testing] - remove warning in unit tests (for nodejs >= v25) (#4149) - polish HTTP 304 docs/test/handling (#4129) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Kristjan ESPERANTO <35647502+KristjanESPERANTO@users.noreply.github.com> Co-authored-by: BugHaver <43462320+bughaver@users.noreply.github.com> Co-authored-by: BugHaver <43462320+lsaadeh@users.noreply.github.com> Co-authored-by: Karsten Hassel <hassel@gmx.de> Co-authored-by: Magnus <34011212+MagMar94@users.noreply.github.com> Co-authored-by: Koen Konst <koenspero@gmail.com> Co-authored-by: Koen Konst <c.h.konst@avisi.nl> Co-authored-by: Bugsounet - Cédric <github@bugsounet.fr> Co-authored-by: dathbe <github@beffa.us> Co-authored-by: veeck <gitkraken@veeck.de> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Marcel <m-idler@users.noreply.github.com> Co-authored-by: sam detweiler <sdetweil@gmail.com> Co-authored-by: Kevin G. <crazylegstoo@gmail.com> Co-authored-by: Jboucly <33218155+jboucly@users.noreply.github.com> Co-authored-by: Jboucly <contact@jboucly.fr> Co-authored-by: Jarno <54169345+jarnoml@users.noreply.github.com> Co-authored-by: Jordan Welch <JordanHWelch@gmail.com> Co-authored-by: Blackspirits <blackspirits@gmail.com> Co-authored-by: Samed Ozdemir <samed@xsor.io> Co-authored-by: in-voker <58696565+in-voker@users.noreply.github.com> Co-authored-by: Andrés Vanegas Jiménez <142350+angeldeejay@users.noreply.github.com> Co-authored-by: cgillinger <christian.gillinger@gmail.com> Co-authored-by: Sonny B <43247590+sonnyb9@users.noreply.github.com> Co-authored-by: sonnyb9 <sonnyb9@users.noreply.github.com> Co-authored-by: Morgan McBee <egeekial@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Mike Bishop <mbishop@evequefou.be>
This adds a pre-filter that drops out-of-window non-recurring events from the raw ICS data before
node-icalparses it. Recurring events and anything we cannot classify cheaply are kept - the existing logic handles those as before.Some calendars accumulate years of old events. Pre-filtering them reduces the amount of work
node-icalhas to do and keeps event loop lag lower. My benchmarks on a fast machine showed a small but measurable speedup (roughly 10-20% with generated test data); on a Raspberry Pi it should be significantly more noticeable.For the implementation I ended up using
ics-filter, suggested by @rejas. I had a custom version first, but ICS date string parsing has enough complexity that it makes sense to delegate it to a package built specifically for this. I also contributed a couple of fixes toics-filteralong the way, and the maintainer was responsive and open to improvements, so it seems like a good fit.Solves #4103.