Deleting an actor fails when ate-api-server attempts to clean up snapshots from GCS:
rpc error: code = PermissionDenied desc = Caller does not have storage.objects.list access to the Google Cloud Storage bucket. Permission 'storage.objects.list' denied on resource '//storage.googleapis.com/projects/_/buckets/<bucket>'
Because snapshot cleanup fails, the actor remains permanently stuck in ACTOR_STATE_DELETING.
Cause
On GKE, default node service accounts only have read_only storage scopes. The ate-system/ate-api-server Kubernetes service account is not bound to a Google Service Account (GSA) via Workload Identity, leaving it without permissions to list or delete objects in the snapshots bucket.
Suggestions
- Document Workload Identity setup: Specify that
ate-api-server requires roles/storage.objectAdmin on the snapshot bucket via Workload Identity (ate-system/ate-api-server).
- Handle cleanup failures gracefully: Allow actor deletion to complete (or report a warning) instead of trapping the actor in
ACTOR_STATE_DELETING forever when bucket cleanup errors occur.
Deleting an actor fails when
ate-api-serverattempts to clean up snapshots from GCS:Because snapshot cleanup fails, the actor remains permanently stuck in
ACTOR_STATE_DELETING.Cause
On GKE, default node service accounts only have
read_onlystorage scopes. Theate-system/ate-api-serverKubernetes service account is not bound to a Google Service Account (GSA) via Workload Identity, leaving it without permissions to list or delete objects in the snapshots bucket.Suggestions
ate-api-serverrequiresroles/storage.objectAdminon the snapshot bucket via Workload Identity (ate-system/ate-api-server).ACTOR_STATE_DELETINGforever when bucket cleanup errors occur.