Fix memory stats for KVM by joseflauzino · Pull Request #6358 · apache/cloudstack

joseflauzino · 2022-05-04T19:04:22Z

Description

Using KVM hypervisor, the free memory stats of the user VMs that are returned by ACS do not correspond to those observed directly in the VMs (when using the free -m command, for example).

It was identified that the problem was in the memory stats collection process performed by Libvirt. This happens because the period that Libvirt would get memory stats updates for each VM was never set. Thus, even if ACS periodically requested the stats for the hosts, the data obtained were always the same.

This PR solves the mentioned problem. A new Agent configuration parameter called vm.memballoon.stats.period was created. This parameter allows operatos to set the time interval in which Libvirt will get memory stats updates. The default value of this parameter is 60 seconds, the same default value as the vm.stats.interval parameter. However, in a cloud containing multiple Management Servers, operators can set the value of vm.memballoon.stats.period lower than the vm.stats.interval, so that each Management Server always gets updated memory stats for all user VMs (since each Management Server can request stats at different instants).

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Bug Severity

How Has This Been Tested?

In a local lab, I tested multiple combinations for all the related Agent properties (vm.memballoon.stats.period, vm.stats.interval, and vm.memballoon.disable). In all cases tested, the memory stats of the user VMs were obtained properly.
Also, I added unit tests.

joseflauzino · 2022-05-06T11:40:03Z

@blueorangutan package

blueorangutan · 2022-05-06T11:41:03Z

@joseflauzino a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-05-06T12:16:57Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 3358

…tead of virsh command to get Libvirt domains

github-actions · 2022-05-21T14:29:58Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

acs-robot · 2022-05-21T14:30:40Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

…memory-stats-for-kvm

joseflauzino · 2022-05-21T14:38:11Z

I moved the code to a more appropriate location. Also, I adjusted the code to get the VM list (Libvirt domains) by using libvirt-java instead of a virsh command.
Thanks for the reviews.

acs-robot · 2022-05-21T14:38:18Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-05-21T14:39:03Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan · 2022-05-21T14:45:12Z

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6358 (SL-JID-1609)

DaanHoogland

@joseflauzino ,
please have a look at the code smells reported:

SonarCloud Quality Gate failed.

0 Bugs 0 Vulnerabilities 0 Security Hotspots 12 Code Smells

71.6% Coverage 0.0% Duplication

some of them make more sense than others, but none arereally senseless as far as I can see.

github-actions · 2022-05-23T13:30:48Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

…ctor

joseflauzino · 2022-06-03T13:38:56Z

@blueorangutan package

blueorangutan · 2022-06-03T13:40:03Z

@joseflauzino a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

sonarqubecloud · 2022-06-03T14:25:08Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
11 Code Smells

70.9% Coverage
0.0% Duplication

blueorangutan · 2022-06-03T14:31:17Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 3525

yadvr · 2022-07-27T09:27:21Z

ping @joseflauzino can you address the review comments?

blueorangutan · 2022-10-22T00:08:50Z

Trillian test result (tid-5184)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 39770 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6358-t5184-kvm-centos7.zip
Smoke tests completed. 99 look OK, 5 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_add_primary_storage_disabled_host	`Error`	0.60	test_primary_storage.py
test_01_primary_storage_nfs	`Error`	0.10	test_primary_storage.py
ContextSuite context=TestStorageTags>:setup	`Error`	0.18	test_primary_storage.py
test_01_secure_vm_migration	`Error`	156.36	test_vm_life_cycle.py
test_02_unsecure_vm_migration	`Error`	270.83	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	`Error`	141.97	test_vm_life_cycle.py
test_08_migrate_vm	`Error`	43.85	test_vm_life_cycle.py
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	33.00	test_kubernetes_clusters.py
test_07_deploy_kubernetes_ha_cluster	`Failure`	60.73	test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster	`Failure`	38.07	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	`Failure`	39.99	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	`Error`	130.33	test_kubernetes_clusters.py
test_02_list_snapshots_with_removed_data_store	`Error`	8.56	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	`Error`	8.56	test_snapshots.py
test_hostha_enable_ha_when_host_in_maintenance	`Error`	304.80	test_hostha_kvm.py

weizhouapache · 2022-10-27T08:52:47Z

@joseflauzino
I use some your changes in fb86898

…y default

…memory-stats-for-kvm

acs-robot · 2022-11-03T00:50:13Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

joseflauzino · 2022-11-03T00:59:02Z

@blueorangutan package

blueorangutan · 2022-11-03T01:00:03Z

@joseflauzino a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-11-03T01:37:13Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. LL-JID 184

GutoVeronezi · 2022-11-07T14:27:58Z

+     * 1st parameter: the VM name;<br>
+     * 2nd parameter: the period (in seconds).
+     */
+    private static final String COMMAND_SET_MEM_BALLOON_STATS_PERIOD = "virsh dommemstat %s --period %s --live";


ping @joseflauzino

…memory-stats-for-kvm

joseflauzino · 2022-11-07T16:42:58Z

@blueorangutan package

blueorangutan · 2022-11-07T16:44:02Z

@joseflauzino a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-11-07T17:19:42Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. LL-JID 212

DaanHoogland · 2022-11-08T13:00:01Z

@blueorangutan test

blueorangutan · 2022-11-08T13:01:17Z

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

DaanHoogland

looks good, two minor sonar issues. will re-run regression tests.
Most of the sonar issues are not your doing @joseflauzino . Could you have a quick look anyway?

sonarqubecloud · 2022-11-08T14:46:52Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
13 Code Smells

75.1% Coverage
0.0% Duplication

blueorangutan · 2022-11-09T01:45:42Z

Trillian test result (tid-46)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 44501 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6358-t46-kvm-centos7.zip
Smoke tests completed. 102 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_upgrade_kubernetes_cluster	`Failure`	264.05	test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster	`Failure`	492.99	test_kubernetes_clusters.py
test_01_redundant_vpc_site2site_vpn	`Failure`	749.85	test_vpc_vpn.py
test_01_vpc_site2site_vpn_multiple_options	`Error`	1275.47	test_vpc_vpn.py
test_01_vpc_site2site_vpn	`Error`	546.30	test_vpc_vpn.py

Fix memory stats for KVM

05339ea

boring-cyborg Bot added component:agent component:kvm labels May 4, 2022

yadvr reviewed May 5, 2022

View reviewed changes

Comment thread agent/src/main/java/com/cloud/agent/Agent.java Outdated

Move the code to a more appropriate location and use libvirt-java ins…

42b83f9

…tead of virsh command to get Libvirt domains

github-actions Bot added the status:has-conflicts label May 21, 2022

Merge branch 'main' of https://github.com/apache/cloudstack into fix-…

2afaac3

…memory-stats-for-kvm

github-actions Bot removed the status:has-conflicts label May 21, 2022

DaanHoogland reviewed May 23, 2022

View reviewed changes

Comment thread agent/src/main/java/com/cloud/agent/properties/AgentProperties.java Outdated

Comment thread agent/src/main/java/com/cloud/agent/properties/AgentProperties.java Outdated

github-actions Bot added the status:has-conflicts label May 23, 2022

joseflauzino added 2 commits June 3, 2022 10:05

Resolve conflicts

17ec5df

Remove the type specification when calling the Property class constru…

c1fcdf3

…ctor

github-actions Bot removed the status:has-conflicts label Jun 3, 2022

nvazquez force-pushed the main branch from c56220f to 0f4c3af Compare June 6, 2022 15:14

weizhouapache reviewed Jul 21, 2022

View reviewed changes

DaanHoogland reviewed Oct 23, 2022

View reviewed changes

Comment thread ...ypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java Outdated

DaanHoogland reviewed Oct 24, 2022

View reviewed changes

Comment thread .../hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtDomainXMLParser.java Outdated

DaanHoogland reviewed Oct 24, 2022

View reviewed changes

Comment thread plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtVMDef.java

weizhouapache reviewed Oct 25, 2022

View reviewed changes

Comment thread agent/src/main/java/com/cloud/agent/properties/AgentProperties.java Outdated

joseflauzino added 2 commits November 2, 2022 21:32

Refactor code and change 'vm.memballoon.stats.period' property to 0 b…

061dd06

…y default

Merge branch 'main' of https://github.com/apache/cloudstack into fix-…

23b1f0b

…memory-stats-for-kvm

weizhouapache reviewed Nov 3, 2022

View reviewed changes

Comment thread agent/conf/agent.properties Outdated

joseflauzino added 2 commits November 4, 2022 14:47

Change the default value of vm.memballoon.stats.period

32cb19e

Fix code smells

b2d18fb

GutoVeronezi reviewed Nov 7, 2022

View reviewed changes

GutoVeronezi self-requested a review November 7, 2022 14:29

joseflauzino added 2 commits November 7, 2022 13:39

Address reviews

e4aded1

Merge branch 'main' of https://github.com/apache/cloudstack into fix-…

30bfcb8

…memory-stats-for-kvm

DaanHoogland approved these changes Nov 8, 2022

View reviewed changes

Comment thread ...ypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java Outdated

Comment thread ...ypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java Outdated

Fix sonar issues

6a7f177

DaanHoogland merged commit 1843632 into apache:main Nov 9, 2022

weizhouapache mentioned this pull request Oct 5, 2023

Cannot start cloudstack-agent if there is vm not managed by cloudstack on the host #8040

Closed

Uh oh!

Conversation

joseflauzino commented May 4, 2022

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Bug Severity

How Has This Been Tested?

Uh oh!

Uh oh!

joseflauzino commented May 6, 2022

Uh oh!

blueorangutan commented May 6, 2022

Uh oh!

blueorangutan commented May 6, 2022

Uh oh!

github-actions Bot commented May 21, 2022

Uh oh!

acs-robot commented May 21, 2022

Uh oh!

joseflauzino commented May 21, 2022

Uh oh!

acs-robot commented May 21, 2022

Uh oh!

blueorangutan commented May 21, 2022

Uh oh!

blueorangutan commented May 21, 2022

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 23, 2022

Uh oh!

joseflauzino commented Jun 3, 2022

Uh oh!

blueorangutan commented Jun 3, 2022

Uh oh!

sonarqubecloud Bot commented Jun 3, 2022

Uh oh!

blueorangutan commented Jun 3, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yadvr commented Jul 27, 2022

Uh oh!

blueorangutan commented Oct 22, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

weizhouapache commented Oct 27, 2022

Uh oh!

acs-robot commented Nov 3, 2022

Uh oh!

joseflauzino commented Nov 3, 2022

Uh oh!

blueorangutan commented Nov 3, 2022

Uh oh!

blueorangutan commented Nov 3, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GutoVeronezi Nov 7, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joseflauzino commented Nov 7, 2022

Uh oh!

blueorangutan commented Nov 7, 2022

Uh oh!

blueorangutan commented Nov 7, 2022

Uh oh!

DaanHoogland commented Nov 8, 2022