EKS Kubernetes can't pull from ECR
I am using Fargate workers and getting the following error launching pods: ``` Failed to pull image "111122223333.dkr.ecr.us-west-2.amazonaws.com/my-repo:latest": rpc error: code = Unknown desc = faile │ │ d to pull and unpack image "111122223333.dkr.ecr.us-west-2.amazonaws.com/my-repo:latest": failed to resolve reference "111122223333.dkr.ecr.us-west-2.amazonaws.com/my-repo:latest": failed │ │ to do request: Head "https://111122223333.dkr.ecr.us-west-2.amazonaws.com/v2/my-repo/manifests/latest": dial tcp 10.0.166.164:443: i/o timeout ``` Note that we were able to successfully deploy Pods in the past and this only suddenly started happening recently. Note that the repo has the correct associated cross account permissions set.
``` dial tcp 10.0.166.164:443: i/o timeout ``` Indicates that this may be a routing issue. The `10.0.166.164` indicates that it is using [a VPC private link](https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-endpoints.html) to access the ECR endpoint. VPC private links can have their own ACLs, which may be preventing access. Did you recently add a VPC Private Link endpoint for ECR (our Service Catalog module does not deploy a VPC endpoint for ECR by default), and if so, did you add the subnets that Fargate uses to be able to access it? EDIT: This turned out to be the problem and VPC endpoints were added for ECR without the Fargate subnets being added to it. Once the Fargate subnets were added to the ACL, it started to work again.