How should I create the backend (e.g., S3 bucket) for storing Terraform state?
_We get the following question from customers fairly often, so I'll post the question here, and then add an answer below:_ How should I create the backend (e.g., S3 bucket) for storing Terraform state? --- <ins datetime="2023-09-15T09:37:50Z"> <p><a href="https://support.gruntwork.io/hc/requests/110450">Tracked in ticket #110450</a></p> </ins>
There are several different ways to handle this, each with various pros and cons: # Option 1: create the backend manually (ClickOps) ## Overview One option is to create your backend manually. For example, if you're using S3 as a backend, you'd login to the AWS Console, and click around for a while to create an S3 bucket and DynamoDB table. ## Pros - It's an easy way to get started. If you only have a small number of backends to manage (e.g., your whole company just needs 1-3 S3 buckets), this approach works just fine. - You avoid the "chicken & egg" problem described below. - You can configure your backend however you want. ## Cons - It's a manual process, so if you need more backends in the future, and/or you have a lot of backends to manage (e.g., dozens or hundreds of S3 buckets), this approach does not scale well. - You're not managing all your infrastructure as code. - How you set up the backend is not documented, so if you need to do it again the future, it won't be obvious to your team members what to do. - It's easy to make mistakes while doing ClickOps: e.g., you might forget to turn on an important security setting for your S3 bucket. # Option 2: create the backend using Terraform ## Overview Another option is to write a Terraform module to set up your backend. For example, if you're a Gruntwork customer, you can use the [s3-bucket module](https://docs.gruntwork.io/reference/services/data-storage/s-3-bucket/) to create a secure, best-practices S3 bucket. As for the DynamoDB table, it's a single resource ([example](https://github.com/brikis98/terraform-up-and-running-code/blob/3rd-edition/code/terraform/03-terraform-state/file-layout-example/global/s3/main.tf#L55-L64)). ## The chicken & egg problem One catch here is that you have a bit of a **_chicken & egg_** problem: you want to store all your Terraform state in a backend, but here, the Terraform code you're deploying is what creates the backend, which doesn't exist initially, so you can't store state in it. Working around this is pretty simple, but does involve some manual steps: 1. Run `init` and `apply` on your backend module _without_ configuring any sort of `backend` in your code. 2. When `apply` completes, your backend will be set up (e.g., the S3 bucket and DynamoDB table will now exist), and the state file for your backend module will be sitting on your local disk drive (`terraform.tfstate`). 3. At this point, configure a `backend` block for your backend module to store its state in the backend you just created. 4. Run `init` again. Terraform will prompt you to migrate the local state file (`terraform.tfstate`) into your backend. Say, "yes." Note that you have to do these steps _before_ you can run `apply` on any other modules in this environment, as those modules need this backend to be set up first. So this approach involves some ordering dependencies when setting up new environments. ## Pros - If you can automate this process (see "Cons" below), the solution scales reasonably well. - Everything is managed as code. - The code acts as documentation for how the backend is configured, so future team members will be able to figure out how to set up backends for new environments. - You are using code to manage your backend configuration, so you avoid manual errors. - You can configure your backend however you want. ## Cons - It's harder to get started, as you have to write code and set up processes. - You have the chicken and egg problem: a series of manual steps to create backend and then have the backend module use the backend it just created. - You have to enforce ordering, ensuring that the backend module is the first thing deployed in each new environment. # Option 3: let Terragrunt create your backend ## Overview For a few popular backend types (S3 and GCS), if you're using the [remote_state block](https://terragrunt.gruntwork.io/docs/reference/config-blocks-and-attributes/#remote_state), Terragrunt can automatically create the backend for you. That is, when you run `terragrunt init` or `terragrunt apply`, Terragrunt will check if the backend exists, and if it doesn't, it'll make API calls to set up the backend automatically / transparently, following best practices. ## Pros - It's an easy way to get started, as Terragrunt handles the backend set up for you. - It's a fully automated process and it scales well. - Everything is managed as code (the `remote_state` block configures your backend). - The code acts as documentation for how the backend is configured (along with Terragrunt's docs), so future team members will be able to figure out how to set up backends for new environments. - You are using code to manage your backend configuration, so you avoid manual errors. - You avoid the chicken & egg problem described above. Even ordering isn't a problem, as Terragrunt handles locking to deal with concurrency (e.g., if two modules try to use the same backend that doesn't exist at the same time, one will create it, and the other will wait). ## Cons - Not all backend types are supported. - You have some ability to customize the backend configuration, and Terragrunt does try to follow best practices by default, but you don't have as much control as when you set up the backend yourself using ClickOps or a Terraform module. - Terragrunt currently doesn't have a built in way to _update_ existing backends. So creation is handled automatically, but if later on, you need to evolve your backend approach, that's more of a manual process. # Option 4: use a SaaS managed backend ## Overview Some SaaS tools for Terraform, such as Terraform Cloud and Terraform Enterprise, manage backend for you "magically." That is, you don't have to think about the backend at all: the SaaS tool just does it automatically, transparently, behind the scenes. ## Pros - It's an easy way to get started, as the SaaS tool handles the backend set up for you. - It's a fully automated process and it scales well. - You don't really have documentation for how to set up new backends, but you don't entirely need it, as the SaaS does it magically. - The SaaS tool does all the work for you automatically, so you avoid manual errors. - You avoid the chicken & egg problem described above. ## Cons - You aren't really managing things as code. They are just managed magically by the SaaS. - For the most part, you can't customize how your state is stored. You just have to trust the SaaS tool is doing the right thing. Note that state files contain secrets, so you are putting a lot of trust in this SaaS tool. - Lock in. Migrating away from the SaaS can be tricky. - Migrating state is a bit tricky.