Skip to main content

How it works

Newer Version Available

This documentation pertains to an old version of Gruntwork Pipelines which used the infrastructure-pipelines repository. Click here to view documentation for the most recent version.

Gruntwork Pipelines ArchitectureGruntwork Pipelines Architecture

External CI Tool​

Gruntwork Pipelines has been validated with CircleCI, GitHub Actions, and GitLab. However, it can be used with any external CI/CD tool. The role of the CI/CD tool is to trigger jobs inside Gruntwork Pipelines. We have example configurations that identify changed terraform modules and call the Gruntwork Pipelines invoker Lambda function.

By default, the invoker Lambda function is run by a CLI tool called infrastructure-deployer from within your CI tool.

ECS Deploy Runner​

The ECS Deploy Runner Module is a flexible framework for running pre-defined, locked-down jobs in an isolated ECS task. It serves as the foundation for Gruntwork Pipelines. The components described below work together to trigger jobs, validate them, run them, and stream the logs back to your CI tool as if they were running locally.

Infrastructure Deployer CLI​

The Infrastructure Deployer CLI tool serves as the interface between your chosen CI tool and Gruntwork Pipelines. It is used to trigger jobs in the deploy-runner. Primarily, it calls instances of the invoker lambda described in the next section.

Usage:

infrastructure-deployer --aws-region AWS_REGION [other options] -- CONTAINER_NAME SCRIPT ARGS...

When launching a task, you may optionally set the following useful flags:

  • max-wait-time (default 2h0m0s) — timeout length for the action, this can be any golang parseable string
  • task-cpu — A custom number of CPU units to allocate to the ECS task
  • task-memory — A custom number of memory units to allocate to the ECS task

To get the list of supported containers and scripts, pass in the --describe-containers option. For example:

infrastructure-deployer --describe-containers --aws-region us-west-2

This will list all the containers and the scripts for each container that can be invoked using the invoker function of the ECS deploy runner stack deployed in us-west-2.

Invoker Lambda​

The Invoker Lambda is an AWS Lambda function written in Python that acts as the AWS entrypoint for your pipeline. It has 3 primary roles:

  1. Serving as a gatekeeper for pipelines runs, determining if a particular command is allowed to be run, and if the arguments are valid
  2. Creating ECS tasks that run terraform, docker, or packer commands
  3. Shipping deployment logs back to your CI/CD tool

Standard Configuration​

The ECS deploy runner is flexible and can be configured for many tasks. The standard configuration is a set of ECS task definitions that we ship with Pipelines by default. Once you have your pipeline deployed you can modify the ECS Deploy Runner configuration as you like. The configuration defines what scripts are accepted by the invoker Lambda and which arguments may be provided. The invoker Lambda will reject any script or argument not defined in the ECS Deploy Runner configuration. The default tasks are defined below.

Docker Image Builder (Kaniko)​

The Docker Image Builder task definition allows CI jobs to build docker images. This ECS task uses an open source library called Kaniko to enable docker builds from within a docker container. We provide a Docker image based on Kaniko for this task.

Packer AMI Builder​

The Packer AMI Builder task definition allows CI jobs to build AMIs using HashiCorp Packer. This task runs in a Docker image we provide.

Terraform Planner and Applier​

The Terraform Planner task definition and Terraform Applier task definition are very similar. They allow CI jobs to plan and apply Terraform and Terragrunt code. These tasks run in the same Docker image as the AMI builder.