Resource alerts: threshold alert rules for VMs, volumes, hosts and storage pools

[Damans227]

Description

This PR adds resource alerts. You set a rule on a VM, volume, host or storage pool, like "cpu above 80%", and CloudStack checks it every minute. When it goes over, an alert is fired and sent out.

• rules can be on one resource or on all resources of a type
• alerts go out by webhook, email and the event bus
• a cooldown per rule so the same alert doesnt keep repeating every minute
• alert history on the rule, and an Alerts tab on each resource page
• users, domain admins and root admins each see and manage only what they own. hosts, storage pools and email are root admin only
• a resource can opt out of "all resources" rules with a tag
• a rule on one resource only overrides your own "all resources" rule, so users cant switch off admin rules on their VMs
• volume rules watch used space, in GB or as % of the disk size. disk read/write metrics are on VMs, since CloudStack doesnt collect disk IO per volume by default
• deleting a rule deletes its alert history too
• only one management server runs the checks, so alerts are not doubled when there are more servers
• small change in the webhook plugin so alert deliveries show in the webhook deliveries tab and can be resent

New global settings: resourcealert.evaluation.interval, resourcealert.repeat.interval.default, resourcealert.history.retention.days, resourcealert.per.user.limit

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Screenshots (if appropriate):

Rules list

Settings

New rule form

Rule details

Alert history of a rule

Alerts tab on a resource page

Alert received by a webhook

Alert deliveries on the webhook

Alert email

Alert on the event bus (RabbitMQ)

How Has This Been Tested?

Tested on a KVM lab with 2 management servers, 2 hosts and NFS storage. Went through the UI end to end, and checked everything with cmk and the management server logs.

Main thing tested was delivery, all three ways:

• webhook: a small test receiver got the alerts, signed, with rule, resource, value and severity. filters on the webhook work, so an excluded alert is not sent but still shows in history. deliveries show in the webhook tab and resend works
• email: a test mail server got the alert email only for rules with email on
• event bus: RabbitMQ got the alerts on the cloudstack events exchange next to the normal events

Also tested:

• rules on one resource and on all resources, for VMs, volumes, hosts and storage pools
• cooldown, alerts only repeat after the cooldown
• rules under the threshold stay quiet, and metrics that a resource doesnt report are skipped
• user, domain admin and root admin each only see and edit their own stuff. a user cant open or change another users rule
• opt-out tag
• unit tests for the plugin and the command error codes
• new Marvin smoke test test_resource_alerts.py: rule lifecycle, bad input, user and domain admin limits, alert firing with delivery to a real webhook receiver, all-resources scope and opt-out, history removed with the rule, rule removed when the VM is expunged. The ones that need real VM stats are marked required_hardware=true, the rest run in CI on the simulator

How did you try to break this feature and the system with this change?

• bad input like a percentage over 100 or negative numbers
• trying to create host rules or turn on email as a normal user through the api
• opening, editing and deleting other users rules
• both management servers up at once. checked there are no double alerts
• webhook filters blocking alerts

[Damans227]

@blueorangutan package

[blueorangutan]

@Damans227 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 18352

[Damans227]

@blueorangutan package

[blueorangutan]

@Damans227 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

[codecov]

Codecov Report

❌ Patch coverage is 49.93056% with 721 lines in your changes missing coverage. Please review.
✅ Project coverage is 19.98%. Comparing base (510d0ec) to head (d86bfaa).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...dstack/resourcealert/ResourceAlertManagerImpl.java	55.92%	119 Missing and 82 partials ⚠️
...dstack/resourcealert/ResourceAlertServiceImpl.java	61.80%	102 Missing and 42 partials ⚠️
...ck/resourcealert/dao/ResourceAlertRuleDaoImpl.java	0.00%	90 Missing ⚠️
...dstack/resourcealert/dao/ResourceAlertDaoImpl.java	0.00%	58 Missing ⚠️
...tack/resourcealert/vo/ResourceAlertRuleJoinVO.java	0.00%	31 Missing ⚠️
...urcealert/dao/ResourceAlertRuleWebhookDaoImpl.java	0.00%	29 Missing ⚠️
...t/api/command/user/CreateResourceAlertRuleCmd.java	12.00%	22 Missing ⚠️
...t/api/command/user/UpdateResourceAlertRuleCmd.java	13.63%	19 Missing ⚠️
...oudstack/resourcealert/vo/ResourceAlertRuleVO.java	62.74%	18 Missing and 1 partial ⚠️
.../src/main/java/com/cloud/vm/dao/UserVmDaoImpl.java	0.00%	16 Missing ⚠️
... and 12 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #13478      +/-   ##
============================================
+ Coverage     19.91%   19.98%   +0.07%     
- Complexity    20194    20418     +224     
============================================
  Files          6373     6399      +26     
  Lines        577230   578815    +1585     
  Branches      70696    71004     +308     
============================================
+ Hits         114942   115675     +733     
- Misses       449722   450440     +718     
- Partials      12566    12700     +134     

Flag	Coverage Δ
uitests	3.72% <ø> (+0.01%)	⬆️
unittests	21.26% <49.93%> (+0.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18353

[Damans227]

@blueorangutan package

[blueorangutan]

@Damans227 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18355

[Damans227]

@blueorangutan package

[blueorangutan]

@Damans227 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18371

[github-actions]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[Damans227]

@blueorangutan package

[Damans227]

@blueorangutan package

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19312

[blueorangutan]

[SF] Trillian Build Failed (tid-17031)

[blueorangutan]

[SF] Trillian test result (tid-54)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 56224 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13478-n54-kvm-ol8.zip
Smoke tests completed. 155 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_uservm_host_control_state	Failure	17.10	test_host_control_state.py

[Damans227]

@blueorangutan package

[blueorangutan]

@Damans227 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19360