How do you do blue/green deployments, rolling deployments, and canary deployments with Terraform?
_This is a question we got from a customer. I'll share our recommendation below._ How can I use Terraform to do blue/green deployments, rolling deployments, and canary deployments? How would I integrate this sort of thing into my CI / CD pipelines? --- <ins datetime="2024-01-26T21:28:25Z"> <p><a href="https://support.gruntwork.io/hc/requests/110721">Tracked in ticket #110721</a></p> </ins>
As discussed in [How to use Terraform as a Team](https://blog.gruntwork.io/how-to-use-terraform-as-a-team-251bc1104973), Terraform does _not_ have support built into the _language_ or its standard _workflows_ for deployment strategies such as blue/green deployment, rolling deployment, or canary deployments. However, depending on what you're deploying, there are several ways to accomplish these deployment strategies: # Option 1: deployment strategies built into specific resources/services Some of the services that you manage with Terraform may natively support various deployment strategies if you configure the corresponding Terraform resources correctly. Here are a few examples: - **EKS/K8S**: Kubernetes has built-in support for rolling deployments using the [`Deployment` object](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#updating-a-deployment). Whenever you update a `Deployment`, Kubernetes automatically rolls out the changes by deploying new Pods, waiting for them to come up, and then undeploying the old Pods. So if you're using a Terraform module to manage your Kubernetes `Deployment`, such as the [`kubernetes-service` module](https://docs.gruntwork.io/reference/services/app-orchestration/kubernetes-service/), you get zero-downtime rolling deployment automatically any time you make a change and run `apply`. The `kubernetes-service` module also supports canary deployments by using the [`canary_image`](https://docs.gruntwork.io/reference/services/app-orchestration/kubernetes-service/#canary_image) and [`desired_number_of_canary_pods`](https://docs.gruntwork.io/reference/services/app-orchestration/kubernetes-service/#desired_number_of_canary_pods) parameters. - **ECS**: ECS also has built-in support for rolling deployment. If you manage your ECS services with a Terraform module, such as the [`ecs-service` module](https://docs.gruntwork.io/reference/modules/terraform-aws-ecs/ecs-service), then you can configure the rolling deployment using the [`deployment_maximum_percent`](https://docs.gruntwork.io/reference/modules/terraform-aws-ecs/ecs-service/#deployment_maximum_percent) and [`deployment_minimum_healthy_percent`](https://docs.gruntwork.io/reference/modules/terraform-aws-ecs/ecs-service/#deployment_minimum_healthy_percent) parameters: e.g., if you set `deployment_maximum_percent` to `200`, and you have 3 tasks running, then when you update your service configuration and run `apply`, ECS will spin up 3 new tasks, wait for them to be healthy, and will then tear down the 3 old tasks. The `ecs-service` module also supports canary deployments using the [`desired_number_of_canary_tasks_to_run`](https://docs.gruntwork.io/reference/modules/terraform-aws-ecs/ecs-service/#desired_number_of_canary_tasks_to_run) and [`ecs_task_definition_canary`](https://docs.gruntwork.io/reference/modules/terraform-aws-ecs/ecs-service/#ecs_task_definition_canary) parameters. - **EC2/ASG**: AWS supports rolling deployments for EC2 instances in Auto Scaling Groups (ASG) using [instance refresh](https://docs.aws.amazon.com/autoscaling/ec2/userguide/asg-instance-refresh.html). If you manage your EC2 instances using a Terraform module, such as the [`asg-instance-refresh` module](https://docs.gruntwork.io/reference/modules/terraform-aws-asg/asg-instance-refresh/), every time you make a change and run `apply`, AWS will automatically spin up a new ASG, wait for it to be healthy, and then tear down the old ASG. # Option 2: implement deployment strategies in custom scripts Not all Terraform resources have built-in deployment strategies (in fact, _most_ don't) or if they have them, they might not support the deployment strategy you want. In some cases, there's nothing you can do about it: e.g., you can't really do a blue/green deployment for a VPC route table update; either you update it, or you don't. For some resources, you can implement the deployment strategy yourself using custom scripts. Here are a few examples: - **EKS/K8S**: To do a rolling deployment on the _worker nodes_ of an EKS cluster, we implemented an [`eks deploy` command in our open source CLI tool `kubergrunt`](https://github.com/gruntwork-io/kubergrunt?tab=readme-ov-file#deploy). - **ECS**: To do a rolling deployment on the EC2 instances in an ECS cluster, we implemented a [`roll-out-ecs-cluster-update.py` script](https://docs.gruntwork.io/reference/modules/terraform-aws-ecs/ecs-cluster/#how-do-you-make-changes-to-the-ec2-instances-in-the-cluster). _Note: ECS just recently added support for [managed instance draining](https://aws.amazon.com/about-aws/whats-new/2024/01/amazon-ecs-managed-instance-draining/), so we may be able to replace this script with native resource support soon!_ # Option 3: use tools other than Terraform While Terraform is a great fit for managing your _infrastructure_, for managing the _apps_ that run on top of that infrastructure, dedicated tools can sometimes be a better fit. - **EKS/K8S**: If you install the [Argo Rollouts controller](https://argo-rollouts.readthedocs.io/en/stable/) in your Kubernetes cluster, you can use the [`rollout` object](https://argo-rollouts.readthedocs.io/en/stable/features/specification/) to do a blue/green deployment by configuring the [`BlueGreen` rollout strategy](https://argo-rollouts.readthedocs.io/en/stable/features/bluegreen/). Combined with [ArgoCD](https://argo-cd.readthedocs.io/en/stable/), which you can install using the [`eks-k8s-argocd` module](https://docs.gruntwork.io/reference/modules/terraform-aws-eks/eks-k8s-argocd/), this can be a way to get blue/green deployments for your Kubernetes services.