How do I fix ErrImagePull while pulling pod's images from ACR in AKS?


Azure Kubernetes Infrastructure-as-code Azure-Bicep 💪


Table of contents:

As an engineer I want to pull container images that are part of my pods’ deployments to Azure Kubernetes Service (further - AKS) from private container registry Azure Container Registry (further - ACR). I have created Azure Resource Group (further - RG), AKS and ACR. I have submitted my very first manifest with pod using kubectl, but all I can see is that my pod creation is endlessly pending…

By the way, if you don’t want to read this, you can watch this :)

Check this repository for shortcut of AKS and ACR creation via Azure Bicep.

1# Clone the repo
2
3git clone https://github.com/weekendsprints/azure-for-containers
4
5# Review and adjust params in ./src/parameters.json and ./src/provision.sh
6# Set executable permissions on file and run
7
8chmod +x ./src/provision.sh
9./src/provision.sh

provision.sh includes:

 1#!/bin/bash
 2
 3export RESOURCE_GROUP="pardus-rg"
 4export LOCATION="westeurope"
 5
 6echo "Deploying resources to ... "
 7echo "Resource group:   $RESOURCE_GROUP"
 8echo "Location:         $LOCATION"
 9
10az group create -n $RESOURCE_GROUP -l $LOCATION
11
12echo "Resource group $RESOURCE_GROUP has been created/updated"
13
14az deployment group create \
15    --template-file "./main.bicep" \
16    --parameters "./parameters.json" \
17    --resource-group $RESOURCE_GROUP \

Adding image to ACR #

I am going to authenticate with my private registry and then use nginx:latest as the image to store in the private ACR repository:

1az acr login -n ACR_NAME
2docker pull nginx
3docker tag nginx ACR_NAME.azurecr.io/mynginx
4docker push ACR_NAME.azurecr.io/mynginx

We also verify if the image has arrived successfully to our ACR (we can do the same via Azure Portal by going to ACR > Repository).

1az acr repository list -n ACR_NAME
2
3[
4  "mynginx"
5]

Creating pods #

I am going to use az aks get-credentials to authenticate with my AKS and then, with kubectl, submit two pods with images from public docker registry and from private ACR:

 1az aks get-credentials -n AKS_NAME -g RESOURCE_GROUP
 2
 3kubectl run mynginx1 \
 4        --image=nginx \
 5        --image-pull-policy=Always  \
 6        --dry-run=client \
 7        -o yaml  > mynginx1.yaml
 8
 9kubectl run mynginx2 \
10        --image=ACR_NAME.azurecr.io/mynginx \
11        --image-pull-policy=Always  \
12        --dry-run=client \
13        -o yaml  > mynginx2.yaml
14
15kubectl apply -f mynginx1.yaml
16kubectl apply -f mynginx2.yaml
17kubectl get pods 
18
19NAME       READY   STATUS         RESTARTS   AGE
20mynginx1   1/1     Running        0          14s
21mynginx2   0/1     ErrImagePull   0          2s

With kubectl run I generated two yaml files. Let me dump them here: 1) docker’s hub (pub) nginx; 2) private CR’s image. I am also adding --image-pull-policy=Always in order to pull the image always from the registry, so we won’t bump into cached image layers while doing our further experiments. Note that the default behaviour is to pull in case IfNotPresent (check for further details).

Pod with image from public registry (authentication is not required)

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  creationTimestamp: null
 5  labels:
 6    run: mynginx1
 7  name: mynginx1
 8spec:
 9  containers:
10  - image: nginx
11    imagePullPolicy: Always
12    name: mynginx1
13    resources: {}
14  dnsPolicy: ClusterFirst
15  restartPolicy: Always
16status: {}

Pod with image from private registry (authentication is required)

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  creationTimestamp: null
 5  labels:
 6    run: mynginx2
 7  name: mynginx2
 8spec:
 9  containers:
10  - image: ACR_NAME.azurecr.io/mynginx
11    imagePullPolicy: Always
12    name: mynginx2
13    resources: {}
14  dnsPolicy: ClusterFirst
15  restartPolicy: Always
16status: {}

As we can see the image that we use in pod mynginx2 fails with ErrImagePull. We can explore this error further with kubectl describe pod mynginx2 to see what exactly cause the issue:

 1kubectl describe pod mynginx2
 2ame:         mynginx2
 3Namespace:    default
 4Priority:     0
 5Node:         aks-nodepool1-36348178-vmss000000/10.240.0.4
 6Start Time:   Sun, 10 Oct 2021 22:04:51 +0200
 7Labels:       run=mynginx2
 8Annotations:  <none>
 9Status:       Pending
10IP:           10.244.0.10
11IPs:
12  IP:  10.244.0.10
13Containers:
14  mynginx2:
15    Container ID:   
16    Image:          ACR_NAME.azurecr.io/mynginx
17    Image ID:       
18    Port:           <none>
19    Host Port:      <none>
20    State:          Waiting
21      Reason:       ImagePullBackOff
22    Ready:          False
23    Restart Count:  0
24    Environment:    <none>
25    Mounts:
26      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7zgfm (ro)
27Conditions:
28  Type              Status
29  Initialized       True 
30  Ready             False 
31  ContainersReady   False 
32  PodScheduled      True 
33Volumes:
34  default-token-7zgfm:
35    Type:        Secret (a volume populated by a Secret)
36    SecretName:  default-token-7zgfm
37    Optional:    false
38QoS Class:       BestEffort
39Node-Selectors:  <none>
40Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
41                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
42Events:
43  Type     Reason     Age                   From               Message
44  ----     ------     ----                  ----               -------
45  Normal   Scheduled  3m41s                 default-scheduler  Successfully assigned default/mynginx2 to aks-nodepool1-36348178-vmss000000
46  Normal   Pulling    2m6s (x4 over 3m40s)  kubelet            Pulling image "ACR_NAME.azurecr.io/mynginx"
47  Warning  Failed     2m6s (x4 over 3m40s)  kubelet            Failed to pull image "ACR_NAME.azurecr.io/mynginx": rpc error: code = Unknown desc = failed to pull and unpack image "ACR_NAME.azurecr.io/mynginx:latest": failed to resolve reference "ACR_NAME.azurecr.io/mynginx:latest": failed to authorize: failed to fetch anonymous token: unexpected status: 401 Unauthorized
48  Warning  Failed     2m6s (x4 over 3m40s)  kubelet            Error: ErrImagePull
49  Normal   BackOff    114s (x6 over 3m40s)  kubelet            Back-off pulling image "ACR_NAME.azurecr.io/mynginx"
50  Warning  Failed     99s (x7 over 3m40s)   kubelet            Error: ImagePullBackOff

There you go Failed to pull image "ACR_NAME.azurecr.io/mynginx": rpc error: code = Unknown desc = failed to pull and unpack image "ACR_NAME.azurecr.io/mynginx:latest": failed to resolve reference "ACR_NAME.azurecr.io/mynginx:latest": failed to authorize: failed to fetch anonymous token: unexpected status: 401 Unauthorized and this simply means that having AKS and ACR in the same tenant and subscription is not enough to have images pulled from registry to the cluster. Let’s fix the issue.

Enabling anonymous pull access (option #1) #

For development purposes I can add anonymous public pull access to ACR. Unfortunately it is only supported with Standard SKU, so I am going to upgrade my ACR and set it’s SKU first.

Keep in mind, this makes your ACR image available for anyone in the World since it allows anonymous pulls! I’d say consider such configuration if you maintain public registry (like docker hub) or in case you stuck with troubleshooting and want exclude some silly mis-configuration issues.

1az acr update -n ACR_NAME --sku Standard
2az acr update -n ACR_NAME --anonymous-pull-enabled

Since pulls are anonymous now, the control plane will try to pull the image again for pod mynginx2 and it should be successful this time.

Let’s disable anonymous pull access and set SKU back to basic. Let’s also delete pod mynginx2 so we can repeat the experiment with other options.

1az acr update -n ACR_NAME --anonymous-pull-enabled false
2az acr update -n ACR_NAME --sku Basic
3kubectl delete pod mynginx2
4pod "mynginx2" deleted

Azure AD Managed Identity (option #2) #

If we examine the cluster via Portal (probably the easiest way of doing this), we can see that Managed Identity has been also created for us (while creating AKS cluster). Managed identity simplifies the work of cloud resources, so we don’t have to think about authentication and storing credentials, we can define who, to whom and how we can trust.

Managed Identity - AKS

That managed identity doesn’t have any assignment yet, but we can define it with the following command and try to create a pod and see if the ACR image is pullable. Keep in mind the following command requires Owner, Azure account administrator, or Azure co-administrator role on the Azure subscription.

1az aks update -n AKS_NAME -g RESOURCE_GROUP --attach-acr ACR_NAME
2kubectl apply -f mynginx2.yaml

And also if we examine this Managed Identity a bit further by checkin Azure AD > Enterprise Application we can see that managed identity is a special type of service principal and by using above command we assigned AcrPull role to it and binded it to our ACR resource. Check via the portal by going to MC_RESOURCE_GROUP_AKS_NAME_westeurope > AKS_NAME-agentpool > Azure role assignments.

Managed Identity - role assignments

So kubectl get pods should give us successful pull.

1NAME       READY   STATUS    RESTARTS   AGE
2mynginx1   1/1     Running   0          66m
3mynginx2   1/1     Running   0          47s

I am going to delete the image and detach the integration so we can explore some further options:

1az aks update -n AKS_NAME -g RESOURCE_GROUP --detach-acr ACR_NAME
2kubectl delete pod mynginx2

Kubernetes Secret (option #3) #

We create Kubernetes Secret of type docker-registryand make a reference in our pod by using imagePullSecretproperty explicitly! Let’s generate secret out of access keys.

 1ACR_NAME=ACR_NAME.azurecr.io
 2
 3az acr update -n $ACR_NAME --admin-enabled true
 4
 5ACR_USERNAME=$(az acr credential show -n $ACR_NAME --query="username" -o tsv)
 6ACR_PASSWORD=$(az acr credential show -n $ACR_NAME --query="passwords[0].value" -o tsv)
 7
 8kubectl create secret docker-registry mysecret \
 9        --docker-server=$ACR_NAME \
10        --docker-username=$ACR_USERNAME \
11        --docker-password=$ACR_PASSWORD \
12        --docker-email=$EMAIL \
13        --dry-run=client \
14        -o yaml > mysecret.yaml

Now if we check mysecret.yaml we can see that it contains a base64 encoded string in data > .dockerconfigjson. There are some other options to create secrets out of other credentials. Check. I am going to generate a new version of pod manifest with mysecret reference in it. Then create this pod on my cluster.

 1kubectl run mynginx2 \
 2        --image=ACR_NAME.azurecr.io/mynginx \
 3        --image-pull-policy=Always \
 4        --overrides='{ "spec": { "imagePullSecrets": [{"name": "mysecret"}] } }' \
 5        --dry-run=client \
 6        -o yaml > mynginx2.yaml
 7
 8kubectl apply -f mynginx2.yaml
 9kubectl get pods
10NAME       READY   STATUS    RESTARTS   AGE
11mynginx1   1/1     Running   0          120m
12mynginx2   1/1     Running   0          93s

Great, the secret works.

Don’t use this option in production, consider to use Managed Identity and let Azure care about trust between the two services.

Okay, these were 3 different options to authenticate and pull images from ACR to AKS cluster. There are list of available authentication options that I encourage to review in order to learn more about service principal, AKS cluster service principal, various options with managed identity and repository-scoped access token.

comments powered by Disqus