Why limit parallelism for account-baseline?
The production example for the `account-baseline` modules in `terraform-aws-service-catalog` contain the following code snippet (in [this file](https://github.com/gruntwork-io/terraform-aws-service-catalog/blob/master/examples/for-production/infrastructure-live/_envcommon/landingzone/account-baseline-app-base.hcl)): ``` # This module deploys some resources (e.g., AWS Config) across all AWS regions, each of which needs its own provider, # which in Terraform means a separate process. To avoid all these processes thrashing the CPU, which leads to network # connectivity issues, we limit the parallelism here. extra_arguments "parallelism" { commands = get_terraform_commands_that_need_parallelism() arguments = get_env("TG_DISABLE_PARALLELISM_LIMIT", "false") == "true" ? [] : ["-parallelism=2"] } ``` This seems to contradict the release notes for [v0.53.0](https://github.com/gruntwork-io/terraform-aws-service-catalog/releases/tag/v0.53.0) where it saids: > Note that you no longer need to limit parallelism (with the -parallelism flag) or set ulimit to run these modules! Is the parallelism still necessary?
When we released this update initially, we did observe that the parallelism flag was no longer necessary, leading us to mention this fact in the notes. Since the release, we have learned that the `ulimit` setting is indeed no longer necessary, but there are certain deployment modes where it is recommended to continue to use the limited parallelism. Specifically, we have found that `account-baseline` still struggles to deploy without parallelism when either of the following conditions are true: - Any machine that has less than 4 cores, or less than 4 effective cores (e.g., if you are running a lot of things on your machine like Docker that prevents availability of cores to Terraform). - When running many other modules concurrently with `run-all plan` or `run-all apply`. Our recommendation is to try without parallelism first, and then turn it on if you find that the system struggles to manage the `account-baseline` modules. --- Note that the `for-production` example is generated from our Reference Architecture, and our reference architecture deployment includes the parallelism limitation because one of these conditions ends up true for deployment: - In the initial deployment, we deploy all modules at once including `account-baseline`, which triggers the second condition. - For updates, the `ecs-deploy-runner` uses a 2-core 4GB machine due to account limits for Fargate, which triggers the first condition.