fix: correctly handle NFS metalink templates - mount export dir, compute real file size - #13934
Open
waterWang wants to merge 2 commits into
Open
fix: correctly handle NFS metalink templates - mount export dir, compute real file size#13934waterWang wants to merge 2 commits into
waterWang wants to merge 2 commits into
Conversation
Member
|
@blueorangutan package |
|
@kiranchavala a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #13934 +/- ##
============================================
- Coverage 19.65% 19.65% -0.01%
Complexity 19803 19803
============================================
Files 6368 6368
Lines 574935 574966 +31
Branches 70358 70365 +7
============================================
+ Hits 113026 113029 +3
- Misses 449632 449662 +30
+ Partials 12277 12275 -2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18938 |
Member
There was a problem hiding this comment.
@waterWang tested with the packages based on the pr
The issue still exists
Template size
2026-08-21 08:02:39,207 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-4:[]) (logid:13e7fe22) Processing command: org.apache.cloudstack.agent.directdownload.CheckUrlCommand
2026-08-21 08:02:39,207 INFO [resource.wrapper.LibvirtCheckUrlCommand] (AgentRequest-Handler-4:[]) (logid:13e7fe22) Checking URL: http://10.0.35.140:8081/test.metalink, with connect timeout: 5000, connect request timeout: 5000, socket timeout: 5000
2026-08-21 08:02:39,297 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-4:[]) (logid:13e7fe22) Seq 1-159877786771652744: { Ans: , MgmtId: 32986942669855, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.agent.directdownload.CheckUrlAnswer":{"templateSize":"(6580590.5000 TB) 7235435666033759572","result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2026-08-21 08:02:47,393 DEBUG [kvm.storage.MultipathSCSIAdapterBase] (MultipathMapCleanupJob:[]) (logid:) Executing command [/usr/share/cloudstack-common/scripts/storage/multipath/cleanStaleMaps.sh ].
2026-08-21 08:02:47,419 DEBUG [kvm.storage.MultipathSCSIAdapterBase] (MultipathMapCleanupJob:[]) (logid:) Successfully executed process [30640] for command [/usr/share/cloudstack-common/scripts/storage/multipath/cleanStaleMaps.sh ].
2026-08-21 08:02:47,419 DEBUG [kvm.storage.MultipathSCSIAdapterBase] (MultipathMapCleanupJob:[]) (logid:) Multipath Cleanup Job elapsed time (ms): 26; result: 0
deploy vm
2026-08-21 08:06:27,523 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Request:Seq 1-159877786771652772: { Cmd , MgmtId: 32986942669855, via: 1, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.agent.directdownload.MetalinkDirectDownloadCommand":{"url":"http://10.0.35.140:8081/test.metalink","templateId":"202","destData":{"origUrl":"http://10.0.35.140:8081/test.metalink","uuid":"a0dfaa4b-1ef3-47c0-87fe-797802ce001c","id":"202","format":"QCOW2","accountId":"2","hvm":"true","displayText":"test","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1cc93d6e-9fff-3ede-b829-a317cba51b63","name":"ref-trl-12252-k-Mol8-kiran-chavala-kvm-pri1","id":"1","poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-12252-k-Mol8-kiran-chavala/ref-trl-12252-k-Mol8-kiran-chavala-kvm-pri1","port":"2049","url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-12252-k-Mol8-kiran-chavala/ref-trl-12252-k-Mol8-kiran-chavala-kvm-pri1/?ROLE=Primary&STOREUUID=1cc93d6e-9fff-3ede-b829-a317cba51b63","isManaged":"false"}},"name":"202-2-dfcebd98-1a75-36b2-b8ad-847ab4346ba1","size":"(6580590.5000 TB) 7235435666033759572","hypervisorType":"KVM","bootable":"false","uniqueName":"202-2-dfcebd98-1a75-36b2-b8ad-847ab4346ba1","directDownload":"true","deployAsIs":"false","followRedirects":"false"},"destPool":{"uuid":"1cc93d6e-9fff-3ede-b829-a317cba51b63","name":"ref-trl-12252-k-Mol8-kiran-chavala-kvm-pri1","id":"1","poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-12252-k-Mol8-kiran-chavala/ref-trl-12252-k-Mol8-kiran-chavala-kvm-pri1","port":"2049","url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-12252-k-Mol8-kiran-chavala/ref-trl-12252-k-Mol8-kiran-chavala-kvm-pri1/?ROLE=Primary&STOREUUID=1cc93d6e-9fff-3ede-b829-a317cba51b63","isManaged":"false"},"headers":{},"connectTimeout":"5000","soTimeout":"5000","templateSize":"(6580590.5000 TB) 7235435666033759572","format":"QCOW2","followRedirects":"false","wait":"10800","bypassHostMaintenance":"false"}}] }
2026-08-21 08:06:27,523 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Processing command: org.apache.cloudstack.agent.directdownload.MetalinkDirectDownloadCommand
2026-08-21 08:06:27,524 DEBUG [storage.resource.StorageSubsystemCommandHandlerBase] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Executing command MetalinkDirectDownloadCommand: [{"url":"http://10.0.35.140:8081/test.metalink","templateId":202,"destData":{"origUrl":"http://10.0.35.140:8081/test.metalink","uuid":"a0dfaa4b-1ef3-47c0-87fe-797802ce001c","id":202,"format":"QCOW2","accountId":2,"hvm":true,"displayText":"test","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1cc93d6e-9fff-3ede-b829-a317cba51b63","name":"ref-trl-12252-k-Mol8-kiran-chavala-kvm-pri1","id":1,"poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-12252-k-Mol8-kiran-chavala/ref-trl-12252-k-Mol8-kiran-chavala-kvm-pri1","port":2049,"url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-12252-k-Mol8-kiran-chavala/ref-trl-12252-k-Mol8-kiran-chavala-kvm-pri1/?ROLE=Primary&STOREUUID=1cc93d6e-9fff-3ede-b829-a317cba51b63","isManaged":false}},"name":"202-2-dfcebd98-1a75-36b2-b8ad-847ab4346ba1","size":7235435666033759572,"hypervisorType":"KVM","bootable":false,"uniqueName":"202-2-dfcebd98-1a75-36b2-b8ad-847ab4346ba1","directDownload":true,"deployAsIs":false,"followRedirects":false},"destPool":{"uuid":"1cc93d6e-9fff-3ede-b829-a317cba51b63","name":"ref-trl-12252-k-Mol8-kiran-chavala-kvm-pri1","id":1,"poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-12252-k-Mol8-kiran-chavala/ref-trl-12252-k-Mol8-kiran-chavala-kvm-pri1","port":2049,"url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-12252-k-Mol8-kiran-chavala/ref-trl-12252-k-Mol8-kiran-chavala-kvm-pri1/?ROLE=Primary&STOREUUID=1cc93d6e-9fff-3ede-b829-a317cba51b63","isManaged":false},"headers":{},"connectTimeout":5000,"soTimeout":5000,"templateSize":7235435666033759572,"format":"QCOW2","followRedirects":false,"wait":10800,"bypassHostMaintenance":false}].
2026-08-21 08:06:27,524 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Verifying temporary location for downloading the template exists on the host
2026-08-21 08:06:27,526 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Checking for free space on the host for downloading the template with physical size: 412 and virtual size: 7235435666033759572
2026-08-21 08:06:27,527 DEBUG [utils.script.Script] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Executing command [/bin/bash -c df --output=avail /var/lib/libvirt/images -B 1 | tail -1 ].
2026-08-21 08:06:27,535 DEBUG [utils.script.Script] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Successfully executed process [31471] for command [/bin/bash -c df --output=avail /var/lib/libvirt/images -B 1 | tail -1 ].
2026-08-21 08:06:27,535 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Trying to fetch storage pool 1cc93d6e-9fff-3ede-b829-a317cba51b63 from libvirt
2026-08-21 08:06:27,536 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Looking for libvirtd connection at: qemu:///system
2026-08-21 08:06:27,542 DEBUG [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Successfully refreshed pool 1cc93d6e-9fff-3ede-b829-a317cba51b63 Capacity: (3.4170 TB) 3757022642176 Used: (2.6768 TB) 2943200788480 Available: (757.93 GB) 813821853696
2026-08-21 08:06:27,551 ERROR [cloud.utils.UriUtils] (AgentRequest-Handler-2:[]) (logid:9b05f93d) no hash tag in XML response...
2026-08-21 08:06:27,551 INFO [direct.download.MetalinkDirectTemplateDownloader] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Metalink downloader created, metalink URL: http://10.0.35.140:8081/test.metalink parsed - 1 urls and 0 checksums found
2026-08-21 08:06:27,551 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Trying to download template
2026-08-21 08:06:27,551 INFO [direct.download.MetalinkDirectTemplateDownloader] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Trying to download Template from URL: nfs://10.0.35.201/export/primary/ubuntu24-cloud-image-root-password-with-tools.qcow2
2026-08-21 08:06:27,552 DEBUG [utils.script.Script] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Executing command [/bin/bash -c mount -t nfs 10.0.35.201:/export/primary /mnt/9e31ae41-c85a-4827-be72-44e461d23192 ].
2026-08-21 08:06:27,563 WARN [utils.script.Script] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Execution of process [31474] for command [/bin/bash -c mount -t nfs 10.0.35.201:/export/primary /mnt/9e31ae41-c85a-4827-be72-44e461d23192 ] failed.
2026-08-21 08:06:27,564 DEBUG [utils.script.Script] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Exit value of process [31474] for command [/bin/bash -c mount -t nfs 10.0.35.201:/export/primary /mnt/9e31ae41-c85a-4827-be72-44e461d23192 ] is [32].
2026-08-21 08:06:27,564 WARN [utils.script.Script] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Process [31474] for command [/bin/bash -c mount -t nfs 10.0.35.201:/export/primary /mnt/9e31ae41-c85a-4827-be72-44e461d23192 ] encountered the error: [mount.nfs: mount point /mnt/9e31ae41-c85a-4827-be72-44e461d23192 does not exist].
2026-08-21 08:06:27,564 DEBUG [utils.script.Script] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Executing command [/bin/bash -c cp /mnt/9e31ae41-c85a-4827-be72-44e461d23192/ubuntu24-cloud-image-root-password-with-tools.qcow2 null/null/template/n/null/005fe88b-9a82-41ea-91a7-9442ac9e1c81.qcow2 ].
2026-08-21 08:06:27,569 WARN [utils.script.Script] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Execution of process [31476] for command [/bin/bash -c cp /mnt/9e31ae41-c85a-4827-be72-44e461d23192/ubuntu24-cloud-image-root-password-with-tools.qcow2 null/null/template/n/null/005fe88b-9a82-41ea-91a7-9442ac9e1c81.qcow2 ] failed.
2026-08-21 08:06:27,570 DEBUG [utils.script.Script] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Exit value of process [31476] for command [/bin/bash -c cp /mnt/9e31ae41-c85a-4827-be72-44e461d23192/ubuntu24-cloud-image-root-password-with-tools.qcow2 null/null/template/n/null/005fe88b-9a82-41ea-91a7-9442ac9e1c81.qcow2 ] is [1].
2026-08-21 08:06:27,570 WARN [utils.script.Script] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Process [31476] for command [/bin/bash -c cp /mnt/9e31ae41-c85a-4827-be72-44e461d23192/ubuntu24-cloud-image-root-password-with-tools.qcow2 null/null/template/n/null/005fe88b-9a82-41ea-91a7-9442ac9e1c81.qcow2 ] encountered the error: [cp: cannot stat '/mnt/9e31ae41-c85a-4827-be72-44e461d23192/ubuntu24-cloud-image-root-password-with-tools.qcow2': No such file or directory].
2026-08-21 08:06:27,570 DEBUG [utils.script.Script] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Executing command [/bin/bash -c umount /mnt/9e31ae41-c85a-4827-be72-44e461d23192 ].
2026-08-21 08:06:27,576 WARN [utils.script.Script] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Execution of process [31477] for command [/bin/bash -c umount /mnt/9e31ae41-c85a-4827-be72-44e461d23192 ] failed.
2026-08-21 08:06:27,576 DEBUG [utils.script.Script] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Exit value of process [31477] for command [/bin/bash -c umount /mnt/9e31ae41-c85a-4827-be72-44e461d23192 ] is [32].
2026-08-21 08:06:27,576 WARN [utils.script.Script] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Process [31477] for command [/bin/bash -c umount /mnt/9e31ae41-c85a-4827-be72-44e461d23192 ] encountered the error: [umount: /mnt/9e31ae41-c85a-4827-be72-44e461d23192: no mount point specified.].
2026-08-21 08:06:27,576 INFO [direct.download.MetalinkDirectTemplateDownloader] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Successfully downloaded Template from URL: nfs://10.0.35.201/export/primary/ubuntu24-cloud-image-root-password-with-tools.qcow2
2026-08-21 08:06:27,576 INFO [direct.download.MetalinkDirectTemplateDownloader] (AgentRequest-Handler-2:[]) (logid:9b05f93d) No checksum provided, skipping checksum validation
2026-08-21 08:06:27,576 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Trying to fetch storage pool 1cc93d6e-9fff-3ede-b829-a317cba51b63 from libvirt
2026-08-21 08:06:27,576 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Looking for libvirtd connection at: qemu:///system
2026-08-21 08:06:27,581 DEBUG [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Successfully refreshed pool 1cc93d6e-9fff-3ede-b829-a317cba51b63 Capacity: (3.4170 TB) 3757022642176 Used: (2.6768 TB) 2943200788480 Available: (757.93 GB) 813821853696
2026-08-21 08:06:27,582 WARN [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-2:[]) (logid:9b05f93d) Error downloading template TemplateTO {"id":202,"name":"202-2-dfcebd98-1a75-36b2-b8ad-847ab4346ba1","origUrl":"http:\/\/10.0.35.140:8081\/test.metalink","uuid":"a0dfaa4b-1ef3-47c0-87fe-797802ce001c"} with id 202 due to: Direct download template file /var/lib/libvirt/images/template/2/202/d6a2750a-e625-4380-9ead-b404d8b544db.qcow2 does not exist on this host
|
This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #13916
Fixes #13917
Description
Fixes two related bugs in NFS direct download for metalink templates:
Bug 1: Template size is incorrect (#13916)
NfsDirectTemplateDownloader.getRemoteFileSize()returnednullunconditionally, so the template size was never computed for NFS metalink URLs. This caused the template size to be stored as a garbage value (e.g.,7235435666033759572bytes ≈ 6580590 TB).Fix: Implement
getRemoteFileSize()inNfsDirectTemplateDownloader:QCOW2Utils.getVirtualSize()File.length()Bug 2: Unable to launch VM with NFS metalink template (#13917)
Two issues in the NFS downloader:
Mount path was wrong:
NfsDirectTemplateDownloader.parseUrl()used the full NFS path (including the file name) as the mount source.mount -t nfs host:/export/templates/file.qcow2tries to mount a file as a filesystem, which fails with exit code 32. The fix splits the URL path into the export directory (mountable) and the file name.Wrong downloader type in metalink:
MetalinkDirectTemplateDownloader.getRemoteFileSize()used the metalink-level downloader (typicallyHttpDirectTemplateDownloader) to check inner URLs of all types, including NFS URLs. This causedcheckUrl()andgetRemoteFileSize()to be called on the wrong downloader type. The fix creates the correct downloader per URL type usingcreateDownloaderForMetalinks().Changes
NfsDirectTemplateDownloader.java:
fileNamefield to store the file name extracted from the NFS URLparseUrl(): split the NFS path into export directory (srcPath) and file name (fileName)downloadTemplate(): mount the export directory, copy the file from the correct mount point pathgetRemoteFileSize(): NEW implementation — mount NFS, read file size, unmount. Supports qcow2 virtual size viaQCOW2UtilsMetalinkDirectTemplateDownloader.java:
getRemoteFileSize(): usecreateDownloaderForMetalinks()to create the correct downloader type per inner URL, instead of reusing the outer metalink downloader