How to handle large plan outputs in Gruntwork Pipeline?
From my understanding Gruntwork Pipelines posts the results/stdout from Terragrunt/Terraform plan/apply into the PR comments. How does this work for very large/long plans and builds? Also, using your example of deploying CloudFront which could take 1 hour, how do we know that the Gruntwork pipeline runner is actually “doing” something and not just dead or hanging? Are the results/stdout streamed into the PR comments as chunks or would we have to wait for a hour after its deployed to actually see the comments in the PR comment section? --- <ins datetime="2022-06-16T14:47:01Z"> <p><a href="https://support.gruntwork.io/hc/requests/108789">Tracked in ticket #108789</a></p> </ins>
First, Gruntwork Pipeline actually doesn’t really post the plan outputs into PR comments. In the default mode, it is assumed to be running in a synchronous call, and streams the results back to the `infrastructure-deployer` CLI that is making the call to it, so it can work just like directly calling terragrunt apply. In this case, you will need to add additional logic in the GitHub Actions to post the results on the PR itself. However, it can also be made to run in the background asynchronously (with the `--no-wait` flag on `infrastructure-deployer`), and post the output in S3. In this mode, the typical way to string it together is to: - Have GitHub Actions create a check suite that tracks the deployment on the commit. - Create a GitHub Actions workflow that completes the check suite from a manual trigger. - Define a Lambda function that calls the "complete" workflow using an [S3 trigger](https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html). Using this setup, the callback from the deploy runner is managed through the S3 trigger lambda function, which will post the plan/apply results onto the commit check as details. Or alternatively, you can have the completion workflow post the results to the PR. In async mode, the disadvantage is that you won't have a way to know the status in real time through GitHub. However, the CLI will post a link to CloudWatch Logs for the job, and you can login to AWS to retrieve the log events, provided you have an IAM user that can observe/retrieve the logs (NOTE: A user with [invoke permissions](https://github.com/gruntwork-io/terraform-aws-ci/tree/master/modules/ecs-deploy-runner-invoke-iam-policy) has this permission).