Resource alerts: threshold alert rules for VMs, volumes, hosts and storage pools

[Damans227]

Description

This PR adds resource alerts. You set a rule on a VM, volume, host or storage pool, like "cpu above 80%", and CloudStack checks it every minute. When it goes over, an alert is fired and sent out.

• rules can be on one resource or on all resources of a type
• alerts go out by webhook, email and the event bus
• a cooldown per rule so the same alert doesnt keep repeating every minute
• alert history on the rule, and an Alerts tab on each resource page
• users, domain admins and root admins each see and manage only what they own. hosts, storage pools and email are root admin only
• a resource can opt out of "all resources" rules with a tag
• a rule on one resource only overrides your own "all resources" rule, so users cant switch off admin rules on their VMs
• volume rules watch used space, in GB or as % of the disk size. disk read/write metrics are on VMs, since CloudStack doesnt collect disk IO per volume by default
• deleting a rule deletes its alert history too
• only one management server runs the checks, so alerts are not doubled when there are more servers
• small change in the webhook plugin so alert deliveries show in the webhook deliveries tab and can be resent

New global settings: resourcealert.evaluation.interval, resourcealert.repeat.interval.default, resourcealert.history.retention.days, resourcealert.per.user.limit

Doc PR: apache/cloudstack-documentation#686

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Screenshots (if appropriate):

Rules list

Settings

New rule form

Rule details

Alert history of a rule

Alerts tab on a resource page

Alert received by a webhook

Alert deliveries on the webhook

Alert email

Alert on the event bus (RabbitMQ)

How Has This Been Tested?

Tested on a KVM lab with 2 management servers, 2 hosts and NFS storage. Went through the UI end to end, and checked everything with cmk and the management server logs.

Main thing tested was delivery, all three ways:

• webhook: a small test receiver got the alerts, signed, with rule, resource, value and severity. filters on the webhook work, so an excluded alert is not sent but still shows in history. deliveries show in the webhook tab and resend works
• email: a test mail server got the alert email only for rules with email on
• event bus: RabbitMQ got the alerts on the cloudstack events exchange next to the normal events

Also tested:

• rules on one resource and on all resources, for VMs, volumes, hosts and storage pools
• cooldown, alerts only repeat after the cooldown
• rules under the threshold stay quiet, and metrics that a resource doesnt report are skipped
• user, domain admin and root admin each only see and edit their own stuff. a user cant open or change another users rule
• opt-out tag
• unit tests for the plugin and the command error codes
• new Marvin smoke test test_resource_alerts.py: rule lifecycle, bad input, user and domain admin limits, alert firing with delivery to a real webhook receiver, all-resources scope and opt-out, history removed with the rule, rule removed when the VM is expunged. The ones that need real VM stats are marked required_hardware=true, the rest run in CI on the simulator

How did you try to break this feature and the system with this change?

• bad input like a percentage over 100 or negative numbers
• trying to create host rules or turn on email as a normal user through the api
• opening, editing and deleting other users rules
• both management servers up at once. checked there are no double alerts
• webhook filters blocking alerts

[Damans227]

@blueorangutan package

[blueorangutan]

@Damans227 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 18352

[Damans227]

@blueorangutan package

[blueorangutan]

@Damans227 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 3.70%. Comparing base (f03cffc) to head (183a671).

Additional details and impacted files

@@            Coverage Diff            @@
##            main   #13478      +/-   ##
=========================================
+ Coverage   3.69%    3.70%   +0.01%     
=========================================
  Files        491      496       +5     
  Lines      42220    42384     +164     
  Branches    7974     8012      +38     
=========================================
+ Hits        1558     1569      +11     
- Misses     40436    40589     +153     
  Partials     226      226              

Flag	Coverage Δ
uitests	3.70% <ø> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18353

[Damans227]

@blueorangutan package

[blueorangutan]

@Damans227 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18355

[Damans227]

@blueorangutan package

[blueorangutan]

@Damans227 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18371

[github-actions]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[Damans227]

@blueorangutan package

[Damans227]

@blueorangutan package

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19312

[blueorangutan]

[SF] Trillian Build Failed (tid-17031)