What is the recommended folder structure for Terragrunt?
We got a series of questions from a customer: 1. One repo for infra-live and one for infra-modules, or combine them? 1. One Terraform config for an entire environment, or split out by resource? 1. Whether to use one branch per environment, or a single branch with all environments? 1. How to handle global variables? 1. How to set up the backend (e.g., S3 bucket)? 1. How to connect Terraform configuration to the backend (e.g., S3 state files)? 1. How to handle module defaults? 1. How to handle tagging? These questions come up often, so I'll post our recommendations below. --- <ins datetime="2023-08-31T12:07:32Z"> <p><a href="https://support.gruntwork.io/hc/requests/110425">Tracked in ticket #110425</a></p> </ins>
# One repo for infra-live and one for infra-modules, or combine them? We recommend separate repos. The primary reason is so you can version your modules and run different versions of those modules in different environments: e.g., you might run `v1.0.0` of your `eks` module in `prod` while testing out `v2.0.0` of the `eks` module in `stage`. As described in [module versioning](https://blog.gruntwork.io/how-to-create-reusable-infrastructure-with-terraform-modules-25526d65f73d#b9d8): > The easiest way to create a versioned module is to put the code for the module in a separate Git repository and to set the source parameter to that repository’s URL. That means your Terraform code will be spread out across (at least) two repositories: > > - **modules**: This repo defines reusable modules. Think of each module as a “blueprint” that defines a specific part of your infrastructure. > - **live**: This repo defines the live infrastructure you’re running in each environment (stage, prod, mgmt, etc.). Think of this as the “houses” you built from the “blueprints” in the modules repo. [See here](https://github.com/gruntwork-io/terragrunt-infrastructure-live-example#how-is-the-code-in-this-repo-organized) for how to lay out your `infra-live` repo. And [see here](https://terragrunt.gruntwork.io/docs/features/keep-your-terraform-code-dry/) for how, in `infra-live`, to deploy your modules from `infra-modules`. # One Terraform config for an entire environment, or split out by resource? Generally speaking, you want to keep Terraform modules small. Bigger modules are slower (e.g., `plan` can take 10+ minutes), harder to understand (reading 1,000 lines of `plan` output is hard), less secure (you need broad permissions to make any change), more risky (the blast radius for mistakes is huge), and so on (see [large modules considered harmful](https://blog.gruntwork.io/5-lessons-learned-from-writing-over-300-000-lines-of-infrastructure-code-36ba7fadeac1#302b)). Putting all the code for an entire environment in a single module is very likely to result in a large module, so we strongly recommend breaking things down into smaller pieces. That said, those pieces shouldn't be too small; you wouldn't want to do a single module per resource. You need to find a good balance, grouping things that are typically deployed together, have similar deployment cadences, have similar risk/security profiles, have common team ownership, etc. So, for example, you might have one module that handles all your networking; another module that sets up your data stores (e.g., RDS); another module that handles your orchestration tool (e.g., EKS); and perhaps a bunch of individual modules to deploy apps, each owned by a separate team. [See the answer here for more info](https://github.com/orgs/gruntwork-io/discussions/402#discussioncomment-2681334). # Whether to use one branch per environment, or a single branch with all environments? We recommend a single branch for all environments, but with versioned modules everywhere, so different environments can deploy different versions. See [How to manage multiple environments with Terraform](https://blog.gruntwork.io/how-to-manage-multiple-environments-with-terraform-32c7bc5d692) for a comparison of Terraform workspaces, branches, and the Terragrunt approaches (we, of course, recommend the [Terragrunt approach](https://blog.gruntwork.io/how-to-manage-multiple-environments-with-terraform-using-terragrunt-2c3e32fc60a8)). # How to handle global variables? Typically, you have a _hierarchy_ of variables: some that are truly global across all accounts/environments; some that apply across a single account/environment; some that apply across a single region; some that apply across a set of services; etc. There are multiple ways to handle this, depending on the use case, but the most common pattern is: 1. See [here](https://github.com/gruntwork-io/terragrunt-infrastructure-live-example#how-is-the-code-in-this-repo-organized) for how to lay out your folder structure in `infra-live` to capture this hierarchy. 2. Put reusable variables at the right "level" within that hierarchy in a `.hcl` file: e.g., `account.hcl`, `region.hcl`, `networking.hcl`, etc. 3. Use [`read_terragrunt_config`](https://terragrunt.gruntwork.io/docs/reference/built-in-functions/#read_terragrunt_config) and [`find_in_parent_folders`](https://terragrunt.gruntwork.io/docs/reference/built-in-functions/#find_in_parent_folders) to automatically load the data from the file from the appropriate place in the hierarchy. For more details and the other options, see [Keep your Terragrunt Architecture DRY](https://terragrunt.gruntwork.io/docs/features/keep-your-terragrunt-architecture-dry/). # How to set up the backend (e.g., S3 bucket)? See [How should I create the backend (e.g., S3 bucket) for storing Terraform state?](https://github.com/orgs/gruntwork-io/discussions/769). # How to connect Terraform configuration to the backend (e.g., S3 state files)? [See the tutorial here](https://terragrunt.gruntwork.io/docs/features/keep-your-remote-state-configuration-dry/). This will allow you to define your `backend` configuration in a single place so all your state files are stored in S3 in a folder hierarchy that matches the folder hierarchy of the modules themselves: e.g., the state for `prod/us-east-1/networking/vpc/terragrunt.hcl` will automatically be stored at `prod/us-east-1/networking/vpc/terraform.tfstate` in your S3 bucket. # How to handle module defaults? 1. Create Terraform modules in `infra-modules` that set reasonable `default` values for all input variables. 2. When deploying those modules in `infra-live`, you get those defaults automatically, but you can also override them on a per-environment basis if necessary. # How to handle tagging? See [how do I enforce tagging using Terragrunt](https://github.com/orgs/gruntwork-io/discussions/756).