[Bug]: Fail to set Group Data Repeatedly

[PowellTravis]

Describe the bug

I keep trying to apply a new group configuration to cloud-init, but I keep just getting I/O timeout errors.
failed to set group data in cloud-init error="PutGroups(): failed to PUT group(s) to cloud-init: error making PUT request to cloud-init, failed to execute HTTP request: Put \"https://tempest-ochami-mgmt.rci.montana.edu:8443/cloud-init/admin/groups/rome-gpu\": dial tcp 10.226.104.4:8443: i/o timeout"

Steps to reproduce

1. Try to write a group get I/O timeout.

Relevant log output

[root@tempest-ochami-mgmt tempest-configs]# ochami cloud-init group set -f yaml -d @./rome-gpu.yaml
2025-11-25T15:45:11-05:00 ERR cloud_init-group-set.go:85 > failed to set group data in cloud-init error="PutGroups(): failed to PUT group(s) to cloud-init: error making PUT request to cloud-init, failed to execute HTTP request: Put \"https://tempest-ochami-mgmt.rci.montana.edu:8443/cloud-init/admin/groups/rome-gpu\": dial tcp 10.226.104.4:8443: i/o timeout"
2025-11-25T15:45:11-05:00 WRN cloud_init-group-set.go:91 > cloud-init group data setting completed with errors
2025-11-25T15:45:11-05:00 ERR lib.go:503 > see 'ochami cloud-init group set --help' for long command help
[root@tempest-ochami-mgmt tempest-configs]# ochami cloud-init group set -f yaml -d @./rome-gpu.yaml
2025-11-25T15:45:42-05:00 ERR cloud_init-group-set.go:85 > failed to set group data in cloud-init error="PutGroups(): failed to PUT group(s) to cloud-init: error making PUT request to cloud-init, failed to execute HTTP request: Put \"https://tempest-ochami-mgmt.rci.montana.edu:8443/cloud-init/admin/groups/rome-gpu\": dial tcp 10.226.104.4:8443: i/o timeout"
2025-11-25T15:45:42-05:00 WRN cloud_init-group-set.go:91 > cloud-init group data setting completed with errors
2025-11-25T15:45:42-05:00 ERR lib.go:503 > see 'ochami cloud-init group set --help' for long command help
[root@tempest-ochami-mgmt tempest-configs]# ochami cloud-init group set -f yaml -d @./rome-gpu.yaml
2025-11-25T15:46:17-05:00 ERR cloud_init-group-set.go:85 > failed to set group data in cloud-init error="PutGroups(): failed to PUT group(s) to cloud-init: error making PUT request to cloud-init, failed to execute HTTP request: Put \"https://tempest-ochami-mgmt.rci.montana.edu:8443/cloud-init/admin/groups/rome-gpu\": dial tcp 10.226.104.4:8443: i/o timeout"
2025-11-25T15:46:17-05:00 WRN cloud_init-group-set.go:91 > cloud-init group data setting completed with errors
2025-11-25T15:46:17-05:00 ERR lib.go:503 > see 'ochami cloud-init group set --help' for long command help
[root@tempest-ochami-mgmt tempest-configs]# ochami bss boot params set -d @./test/gpu-fabric-boot-test.yaml -f yaml
2025-11-25T15:46:49-05:00 ERR bss-boot-params-set.go:139 > failed to set boot parameters in BSS error="PutBootParams(): failed to PUT boot parameters to BSS: error making PUT request to BSS, failed to execute HTTP request: Put \"https://tempest-ochami-mgmt.rci.montana.edu:8443/boot/v1/bootparameters\": dial tcp 10.226.104.4:8443: i/o timeout"
2025-11-25T15:46:49-05:00 ERR lib.go:503 > see 'ochami bss boot params set --help' for long command help
[root@tempest-ochami-mgmt tempest-configs]# ochami cloud-init group set -f yaml -d @./rome-gpu.yaml
2025-11-25T15:47:22-05:00 ERR cloud_init-group-set.go:85 > failed to set group data in cloud-init error="PutGroups(): failed to PUT group(s) to cloud-init: error making PUT request to cloud-init, failed to execute HTTP request: Put \"https://tempest-ochami-mgmt.rci.montana.edu:8443/cloud-init/admin/groups/rome-gpu\": dial tcp 10.226.104.4:8443: i/o timeout"
2025-11-25T15:47:22-05:00 WRN cloud_init-group-set.go:91 > cloud-init group data setting completed with errors
2025-11-25T15:47:22-05:00 ERR lib.go:503 > see 'ochami cloud-init group set --help' for long command help
[root@tempest-ochami-mgmt tempest-configs]# ochami cloud-init group set -f yaml -d @./rome-gpu.yaml
2025-11-25T15:47:54-05:00 ERR cloud_init-group-set.go:85 > failed to set group data in cloud-init error="PutGroups(): failed to PUT group(s) to cloud-init: error making PUT request to cloud-init, failed to execute HTTP request: Put \"https://tempest-ochami-mgmt.rci.montana.edu:8443/cloud-init/admin/groups/rome-gpu\": dial tcp 10.226.104.4:8443: i/o timeout"
2025-11-25T15:47:54-05:00 WRN cloud_init-group-set.go:91 > cloud-init group data setting completed with errors
2025-11-25T15:47:54-05:00 ERR lib.go:503 > see 'ochami cloud-init group set --help' for long command help
[root@tempest-ochami-mgmt tempest-configs]# ochami cloud-init group set -f yaml -d @./rome-gpu.yaml
2025-11-25T15:48:26-05:00 ERR cloud_init-group-set.go:85 > failed to set group data in cloud-init error="PutGroups(): failed to PUT group(s) to cloud-init: error making PUT request to cloud-init, failed to execute HTTP request: Put \"https://tempest-ochami-mgmt.rci.montana.edu:8443/cloud-init/admin/groups/rome-gpu\": dial tcp 10.226.104.4:8443: i/o timeout"
2025-11-25T15:48:26-05:00 WRN cloud_init-group-set.go:91 > cloud-init group data setting completed with errors
2025-11-25T15:48:26-05:00 ERR lib.go:503 > see 'ochami cloud-init group set --help' for long command help
[root@tempest-ochami-mgmt tempest-configs]# ochami cloud-init group set -f yaml -d @./rome-gpu.yaml
2025-11-25T15:48:58-05:00 ERR cloud_init-group-set.go:85 > failed to set group data in cloud-init error="PutGroups(): failed to PUT group(s) to cloud-init: error making PUT request to cloud-init, failed to execute HTTP request: Put \"https://tempest-ochami-mgmt.rci.montana.edu:8443/cloud-init/admin/groups/rome-gpu\": dial tcp 10.226.104.4:8443: i/o timeout"
2025-11-25T15:48:58-05:00 WRN cloud_init-group-set.go:91 > cloud-init group data setting completed with errors
2025-11-25T15:48:58-05:00 ERR lib.go:503 > see 'ochami cloud-init group set --help' for long command help
[root@tempest-ochami-mgmt tempest-configs]# ochami cloud-init group set -f yaml -d @./rome-gpu.yaml
2025-11-25T15:49:30-05:00 ERR cloud_init-group-set.go:85 > failed to set group data in cloud-init error="PutGroups(): failed to PUT group(s) to cloud-init: error making PUT request to cloud-init, failed to execute HTTP request: Put \"https://tempest-ochami-mgmt.rci.montana.edu:8443/cloud-init/admin/groups/rome-gpu\": dial tcp 10.226.104.4:8443: i/o timeout"
2025-11-25T15:49:30-05:00 WRN cloud_init-group-set.go:91 > cloud-init group data setting completed with errors
2025-11-25T15:49:30-05:00 ERR lib.go:503 > see 'ochami cloud-init group set --help' for long command help
[root@tempest-ochami-mgmt tempest-configs]# ochami cloud-init group set -f yaml -d @./rome-gpu.yaml
2025-11-25T15:50:02-05:00 ERR cloud_init-group-set.go:85 > failed to set group data in cloud-init error="PutGroups(): failed to PUT group(s) to cloud-init: error making PUT request to cloud-init, failed to execute HTTP request: Put \"https://tempest-ochami-mgmt.rci.montana.edu:8443/cloud-init/admin/groups/rome-gpu\": dial tcp 10.226.104.4:8443: i/o timeout"
2025-11-25T15:50:02-05:00 WRN cloud_init-group-set.go:91 > cloud-init group data setting completed with errors
2025-11-25T15:50:02-05:00 ERR lib.go:503 > see 'ochami cloud-init group set --help' for long command help
[root@tempest-ochami-mgmt tempest-configs]# ochami bss boot params set -d @./test/gpu-fabric-boot-test.yaml -f yaml
2025-11-25T15:50:37-05:00 ERR bss-boot-params-set.go:139 > failed to set boot parameters in BSS error="PutBootParams(): failed to PUT boot parameters to BSS: error making PUT request to BSS, failed to execute HTTP request: Put \"https://tempest-ochami-mgmt.rci.montana.edu:8443/boot/v1/bootparameters\": dial tcp 10.226.104.4:8443: i/o timeout"
2025-11-25T15:50:37-05:00 ERR lib.go:503 > see 'ochami bss boot params set --help' for long command help
[root@tempest-ochami-mgmt tempest-configs]# ochami cloud-init group set -f yaml -d @./rome-gpu.yaml
2025-11-25T15:51:08-05:00 ERR cloud_init-group-set.go:85 > failed to set group data in cloud-init error="PutGroups(): failed to PUT group(s) to cloud-init: error making PUT request to cloud-init, failed to execute HTTP request: Put \"https://tempest-ochami-mgmt.rci.montana.edu:8443/cloud-init/admin/groups/rome-gpu\": dial tcp 10.226.104.4:8443: i/o timeout"
2025-11-25T15:51:08-05:00 WRN cloud_init-group-set.go:91 > cloud-init group data setting completed with errors
2025-11-25T15:51:08-05:00 ERR lib.go:503 > see 'ochami cloud-init group set --help' for long command help
[root@tempest-ochami-mgmt tempest-configs]# ochami cloud-init group set -f yaml -d @./rome-gpu.yaml
2025-11-25T15:51:43-05:00 ERR cloud_init-group-set.go:85 > failed to set group data in cloud-init error="PutGroups(): failed to PUT group(s) to cloud-init: error making PUT request to cloud-init, failed to execute HTTP request: Put \"https://tempest-ochami-mgmt.rci.montana.edu:8443/cloud-init/admin/groups/rome-gpu\": dial tcp 10.226.104.4:8443: i/o timeout"
2025-11-25T15:51:43-05:00 WRN cloud_init-group-set.go:91 > cloud-init group data setting completed with errors
2025-11-25T15:51:43-05:00 ERR lib.go:503 > see 'ochami cloud-init group set --help' for long command help
[root@tempest-ochami-mgmt tempest-configs]# ochami cloud-init group set -f yaml -d @./rome-gpu.yaml
2025-11-25T15:52:26-05:00 ERR cloud_init-group-set.go:85 > failed to set group data in cloud-init error="PutGroups(): failed to PUT group(s) to cloud-init: error making PUT request to cloud-init, failed to execute HTTP request: Put \"https://tempest-ochami-mgmt.rci.montana.edu:8443/cloud-init/admin/groups/rome-gpu\": dial tcp 10.226.104.4:8443: i/o timeout"
2025-11-25T15:52:26-05:00 WRN cloud_init-group-set.go:91 > cloud-init group data setting completed with errors
2025-11-25T15:52:26-05:00 ERR lib.go:503 > see 'ochami cloud-init group set --help' for long command help
ochami cloud-init defaults set -f yaml -d @./cloud-init-defaults.yaml
2025-11-25T15:55:11-05:00 ERR cloud_init-defaults-set.go:72 > failed to set defaults error="PostDefaults(): failed to POST cluster-defaults to cloud-init: error making POST request to cloud-init, failed to execute HTTP request: Post \"https://tempest-ochami-mgmt.rci.montana.edu:8443/cloud-init/admin/cluster-defaults\": dial tcp 10.226.104.4:8443: i/o timeout"

Code of Conduct

• I agree to follow this project's Code of Conduct

[alexlovelltroy]

That tcp error may indicate that the cloud-init server isn't running. Can you confirm with curl? 503 probably means something similar.

[PowellTravis]

The cloudinit server is running. All the service report as healthy when the errors start.On Nov 25, 2025, at 5:10 PM, Alex Lovell-Troy ***@***.***> wrote:alexlovelltroy left a comment (OpenCHAMI/cloud-init#101) That tcp error may indicate that the cloud-init server isn't running. Can you confirm with curl? 503 probably means something similar. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[synackd]

Do other ochami cloud-init commands work? E.g. ochami cloud-init defaults get? If not, you may need to restart cloud-init and HAProxy (assuming you're using quadlets).

sudo systemctl restart cloud-init-server haproxy