Hello everyone,
Background
I have a simple a service in Go that receives a request and makes a read request to the GCS bucket. This service is deployed in GKE in the same project as the GCS bucket. GKE is using Workload Identity to make requests to the storage bucket. This workload identity user is bound to one of the Service Accounts which has the storage admin permissions on the bucket that the service wants to read from.
To summarize the setup:
- Created a Google service account and K8s service account
- Gave the Google service account permissions to fetch data from the bucket. Though redundant, I also added roles/storage.admin role.
gcloud storage buckets add-iam-policy-binding gs://mybucket --member "serviceAccount:google-service-account@$PROJECT.iam.gserviceaccount.com" --role "roles/storage.objectViewer"
- Added Service account impersonation using this binding for GCP
gcloud iam service-accounts add-iam-policy-binding google-service-account@$PROJECT.iam.gserviceaccount.com \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:$PROJECT.svc.id.goog[namespace/k8sserviceaccount]"
- Added Service account impersonation using this binding for Kubernetes
kubectl annotate serviceaccount k8sserviceaccount iam.gke.io/gcp-service-account=google-service-account@$PROJECT.iam.gserviceaccount.com -n namespace
Problem
When the service makes a request to the storage bucket, I see this error
"error":"googleapi: got HTTP response code 403 with body: <?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).</Details></Error>"
I verified that the Workload Identity is working as expected by running the following
kubectl exec -i -t -n namespace gcloud -- /bin/sh
# curl -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/email
google-service-account@myproject.iam.gserviceaccount.com
I also verified that the Workload Identity User is able to make a read request using this Python script. In fact, read and write both are working correctly using this Workload Identity.
Questions
- It seems all permissions are working. Is there any additional setup that is needed to make the workload Identity be able to connect to the storage account?
- Since GKE is connecting to the storage account, does the storage client have to be initialized with any specific option(s) to make it work with workload identity?
Any thoughts/ideas that can help me proceed here are appreciated. Thank you!
LinkedIn
Twitter
