How can I increase parallelization in Gruntwork Pipelines jobs?
A customer asked: > We're running several large jobs through Gruntwork Pipelines currently and would like to know if there's a way to increase the number of jobs run in parallel to lower the overall execution time? --- <ins datetime="2023-06-02T13:56:19Z"> <p><a href="https://support.gruntwork.io/hc/requests/110219">Tracked in ticket #110219</a></p> </ins>
Our best recommendation for this currently is to modify your `deploy-infra.sh` script, which comes bundled as part of your Gruntwork Pipelines installation. This script acts as the "glue" code between your version control system and the terragrunt plan and apply logic contained in the ECS Deploy Runner component of Gruntwork Pipelines. [Here's an example](https://github.com/gruntwork-io/terraform-aws-service-catalog/blob/d8190609a072f88750351f78e4ad58f7493ff0bd/examples/for-production/infrastructure-live/_ci/scripts/deploy-infra.sh) of the `deploy-infra.sh` script. Within this script, we use the `xargs` unix utility to help format arguments to complex commands. `xargs` can also accept a `-P` flag, which stands for `max-procs` to increase the number of processes that xargs uses to execute a given command, [as explained here](https://explainshell.com/explain?cmd=xargs+-P+2+). For example, to have `xargs` spin up two processes, you would pass a value of 2 to the -P flag like so: `xargs -P 2 <command>`. ## Find the right call to xargs Note that the deploy.infra script has several different handler functions, each for a different workflow: - [`handle_updated_folders`](https://github.com/gruntwork-io/terraform-aws-service-catalog/blob/d8190609a072f88750351f78e4ad58f7493ff0bd/examples/for-production/infrastructure-live/_ci/scripts/deploy-infra.sh#L96) - [`handle_updated_envcommon`](https://github.com/gruntwork-io/terraform-aws-service-catalog/blob/d8190609a072f88750351f78e4ad58f7493ff0bd/examples/for-production/infrastructure-live/_ci/scripts/deploy-infra.sh#L124) - [`handle_deleted_folders`](https://github.com/gruntwork-io/terraform-aws-service-catalog/blob/d8190609a072f88750351f78e4ad58f7493ff0bd/examples/for-production/infrastructure-live/_ci/scripts/deploy-infra.sh#L213) Each handler ends with a call to `xargs`. Modify the appropriate `xargs` command in one or more handlers to pass the `-P` flag value of your choosing. Initially, we recommend trying a `-P` value of 2 to avoid excessive resource consumption: `xargs -P 2`.