Skip to main content

Bootstrap Pipelines in an Existing GitLab Project

This guide provides comprehensive instructions for integrating Gruntwork Pipelines into an existing GitLab project with Infrastructure as Code (IaC). This is designed for Gruntwork customers who want to add Pipelines to their current infrastructure projects for streamlined CI/CD management.

To configure Gruntwork Pipelines in an existing GitLab project, complete the following steps (which are explained in detail below):

  1. (If using a self-hosted GitLab instance) Ensure OIDC configuration and JWKS are publicly accessible.
  2. Plan your Pipelines setup by identifying all environments and cloud accounts/subscriptions you need to manage.
  3. Bootstrap core infrastructure in accounts/subscriptions that don't already have the required OIDC and state management resources.
  4. Configure SCM access using machine users with appropriate Personal Access Tokens (PATs).
  5. Create .gruntwork HCL configurations to tell Pipelines how to authenticate and organize your environments.
  6. Create .gitlab-ci.yml to configure your GitLab CI/CD pipeline.
  7. Commit and push your changes to activate Pipelines.

Ensure OIDC configuration and JWKS are publicly accessible​

This step only applies if you are using a self-hosted GitLab instance that is not accessible from the public internet. If you are using GitLab.com or a self-hosted instance that is publicly accessible, you can skip this step.

  1. Follow GitLab's instructions for hosting your OIDC configuration and JWKS in a public location (e.g. S3 Bucket). This is necessary for both Gruntwork and the AWS OIDC provider to access the GitLab OIDC configuration and JWKS when authenticating JWT's generated by your custom instance.
  2. Note the <ISSUER_URL> (stored as ci_id_tokens_issuer_url in your gitlab.rb file per GitLab's instructions) generated above for reuse in the next steps.
Progress Checklist

Prerequisites​

Before starting, ensure you have:

  • An active Gruntwork subscription with Pipelines access. Verify by checking the Gruntwork Developer Portal and confirming access to "pipelines" repositories in your GitHub team.
  • Cloud provider credentials with permissions to create OIDC providers and IAM roles in accounts where Pipelines will manage infrastructure.
  • Git installed locally for cloning and managing your project.
  • Existing IaC project with Terragrunt configurations you want to manage with Pipelines (if you are using OpenTofu/Terraform, and want to start using Terragrunt, read the Quickstart Guide).

Planning Your Pipelines Setup​

Before implementing Pipelines, it's crucial to plan your setup by identifying all the environments and cloud resources you need to manage.

Identify Your Environments​

Review your existing project structure and identify:

  1. All environments you want to manage with Pipelines (e.g., dev, staging, prod)
  2. Cloud accounts/subscriptions associated with each environment
  3. Directory paths in your project that contain Terragrunt units for each environment
  4. Existing OIDC resources that may already be provisioned in your accounts
Progress Checklist

Determine Required OIDC Roles​

For each AWS Account / Azure Subscription you want to manage, you might already have some or all of the following resources provisioned.

Required AWS Resources:

  • An OIDC provider for GitLab
  • An IAM role for Pipelines to assume when running Terragrunt plan commands
  • An IAM role for Pipelines to assume when running Terragrunt apply commands
Progress Checklist

Configuring SCM Access​

Pipelines needs the ability to interact with GitLab to fetch resources (e.g. IaC code, reusable CI/CD code and the Pipelines binary itself).

To create machine users for GitLab access, follow our machine users guide to set up the appropriate Personal Access Tokens (PATs) with the required permissions.

Progress Checklist

Bootstrapping Cloud Infrastructure​

If your AWS accounts / Azure subscriptions don't already have all the required OIDC and state management resources, you'll need to bootstrap them. This section provides the infrastructure code needed to set up these resources.

tip

If you already have all the resources listed, you can skip this section.

If you have some of them provisioned, but not all, you can decide to either destroy the resources you already have provisioned and recreate them or import them into state. If you are not sure, please contact Gruntwork support.

Prepare Your Project​

Clone your project to your local machine using Git if you haven't already.

tip

If you don't have Git installed, you can install it by following the official guide for Git installation.

For example:

git clone git@gitlab.com:acme/infrastructure-live.git
cd infrastructure-live
Progress Checklist

To bootstrap your project, we'll use Boilerplate to scaffold it with the necessary IaC code to provision the infrastructure necessary for Pipelines to function.

The easiest way to install Boilerplate is to use mise to install it.

tip

If you don't have mise installed, you can install it by following the official guide for mise installation.

mise use -g boilerplate@latest
tip

If you'd rather install a specific version of Boilerplate, you can use the ls-remote command to list the available versions.

mise ls-remote boilerplate
Progress Checklist

If you don't already have Terragrunt and OpenTofu installed locally, you can install them using mise:

mise use -g terragrunt@latest opentofu@latest
Progress Checklist

Cloud-specific bootstrap instructions​

The resources you need provisioned in AWS to start managing resources with Pipelines are:

  1. An OpenID Connect (OIDC) provider
  2. An IAM role for Pipelines to assume when running Terragrunt plan commands
  3. An IAM role for Pipelines to assume when running Terragrunt apply commands

For every account you want Pipelines to manage infrastructure in.

Don't Panic!

This may seem like a lot to set up, but the content you need to add to your project is minimal. The majority of the work will be pulled from a reusable catalog that you'll reference in your project.

If you want to peruse the catalog that's used in the bootstrap process, you can take a look at the terragrunt-scale-catalog repository.

The process that we'll follow to get these resources ready for Pipelines is:

  1. Use Boilerplate to scaffold bootstrap configurations in your project for each AWS account
  2. Use Terragrunt to provision these resources in your AWS accounts
  3. (Optionally) Bootstrap additional AWS accounts until all your AWS accounts are ready for Pipelines

Bootstrap Your Project for AWS

First, confirm that you have a root.hcl file in the root of your project that looks something like this:

root.hcl
locals {
  account_hcl       = read_terragrunt_config(find_in_parent_folders("account.hcl"))
  state_bucket_name = local.account_hcl.locals.state_bucket_name

  region_hcl = read_terragrunt_config(find_in_parent_folders("region.hcl"))
  aws_region = local.region_hcl.locals.aws_region
}

remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite"
  }
  config = {
    bucket       = local.state_bucket_name
    region       = local.aws_region
    key          = "${path_relative_to_include()}/tofu.tfstate"
    encrypt      = true
    use_lockfile = true
  }
}

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
provider "aws" {
  region = "${local.aws_region}"
}
EOF
}

If you don't have a root.hcl file, you might need to customize the bootstrapping process, as the Terragrunt scale catalog expects a root.hcl file in the root of the project. Please contact Gruntwork support for assistance if you need help.

For each AWS account that needs bootstrapping, we'll use Boilerplate to scaffold the necessary content. Run this command from the root of your project for each account:

boilerplate \
  --template-url 'github.com/gruntwork-io/terragrunt-scale-catalog//templates/boilerplate/aws/gitlab/account?ref=v1.1.0' \
  --output-folder .
tip

You'll need to run this boilerplate command once for each AWS account you want to manage with Pipelines. Boilerplate will prompt you for account-specific values each time.

tip

You can reply y to all the prompts to include dependencies, and accept defaults unless you want to customize something.

Alternatively, you could run Boilerplate non-interactively by passing the --non-interactive flag. You'll need to supply the relevant values for required variables in that case.

e.g.

boilerplate \
  --template-url 'github.com/gruntwork-io/terragrunt-scale-catalog//templates/boilerplate/aws/gitlab/account?ref=v1.1.0' \
  --output-folder . \
  --var 'AccountName=dev' \
  --var 'GitLabGroupName=acme' \
  --var 'GitLabRepoName=infrastructure-live' \
  --var 'GitLabInstanceURL=https://gitlab.com' \
  --var 'AWSAccountID=123456789012' \
  --var 'AWSRegion=us-east-1' \
  --var 'StateBucketName=my-state-bucket' \
  --non-interactive

If you're using a self-hosted GitLab instance, you'll want to make sure the issuer is set correctly when calling Boilerplate.

boilerplate \
  --template-url 'github.com/gruntwork-io/terragrunt-scale-catalog//templates/boilerplate/aws/gitlab/account?ref=v1.1.0' \
  --output-folder . \
  --var 'AccountName=dev' \
  --var 'GitLabGroupName=acme' \
  --var 'GitLabRepoName=infrastructure-live' \
  --var 'GitLabInstanceURL=https://gitlab.com' \
  --var 'AWSAccountID=123456789012' \
  --var 'AWSRegion=us-east-1' \
  --var 'StateBucketName=my-state-bucket' \
  --var 'Issuer=<ISSUER_URL>' \
  --non-interactive

You can also choose to store these values in a YAML file and pass it to Boilerplate using the --var-file flag.

vars.yml
AccountName: dev
GitLabGroupName: acme
GitLabRepoName: infrastructure-live
GitLabInstanceURL: https://gitlab.com
AWSAccountID: 123456789012
AWSRegion: us-east-1
StateBucketName: my-state-bucket
boilerplate \
  --template-url 'github.com/gruntwork-io/terragrunt-scale-catalog//templates/boilerplate/aws/gitlab/account?ref=v1.1.0' \
  --output-folder . \
  --var-file vars.yml \
  --non-interactive
Progress Checklist

Provision AWS Bootstrap Resources

Once you've scaffolded out the accounts you want to bootstrap, you can use Terragrunt to provision the resources in each of these accounts.

tip

Make sure that you authenticate to each AWS account you are bootstrapping using AWS credentials for that account before you attempt to provision resources in it.

You can follow the documentation here to authenticate with the AWS provider. You are advised to choose an authentication method that doesn't require any hard-coded credentials, like assuming an IAM role.

For each account you want to bootstrap, you'll need to run the following commands:

First, make sure that everything is set up correctly by running a plan in the bootstrap directory in name-of-account/_global where name-of-account is the name of the AWS account you want to bootstrap.

name-of-account/_global/bootstrap
terragrunt run --all --non-interactive --provider-cache --backend-bootstrap plan
tip

We're using the --provider-cache flag here to ensure that we don't re-download the AWS provider on every run by leveraging the Terragrunt Provider Cache Server.

We're using the --backend-bootstrap flag here to tell Terragrunt to bootstrap the OpenTofu backend automatically for the account.

Next, apply the changes to your account.

name-of-account/_global/bootstrap
terragrunt run --all --non-interactive --provider-cache apply
Progress Checklist

Creating .gruntwork HCL Configurations​

Create HCL configurations in the .gruntwork directory in the root of your project to tell Pipelines how you plan to organize your infrastructure, and how you plan to have Pipelines authenticate with your cloud provider(s).

The repository block​

The core configuration that you'll want to start with is the repository block. This block tells Pipelines which branch has the "live" infrastructure you want provisioned. When you merge IaC to this branch, Pipelines will be triggered to update your infrastructure accordingly.

.gruntwork/repository.hcl
repository {
  deploy_branch_name = "main"
}
Progress Checklist

The environment block​

Next, you'll want to define the environments you want to manage with Pipelines using the environment block.

For each environment, you'll want to define a filter block that tells Pipelines which units are part of that environment. You'll also want to define an authentication block that tells Pipelines how to authenticate with your cloud provider(s) for that environment.

.gruntwork/environment-production.hcl
environment "production" {
  filter {
    paths = ["prod/*"]
  }

  authentication {
    aws_oidc {
      account_id         = "123456789012"
      plan_iam_role_arn  = "arn:aws:iam::123456789012:role/pipelines-plan"
      apply_iam_role_arn = "arn:aws:iam::123456789012:role/pipelines-apply"
    }
  }
}
tip

Learn more about how Pipelines authenticates to AWS in the Authenticating to AWS page.

tip

Check out the aws block for more information on how to configure Pipelines to reuse common AWS configurations.

Progress Checklist

Creating .gitlab-ci.yml​

Create a .gitlab-ci.yml file in the root of your project with the following content:

.gitlab-ci.yml
include:
  - project: 'gruntwork-io/gitlab-pipelines-workflows'
    file: '/workflows/pipelines.yml'
    ref: 'v1'
tip

You can read the Pipelines GitLab CI Workflow to learn how this GitLab CI pipeline calls the Pipelines CLI to run your pipelines.

Progress Checklist

Commit and Push Your Changes​

Commit and push your changes to your project.

note

You should include [skip ci] in your commit message here to prevent triggering the Pipelines workflow before everything is properly configured.

git add .
git commit -m "Add Pipelines configurations and GitLab CI workflow [skip ci]"
git push
Progress Checklist

🚀 You've successfully added Gruntwork Pipelines to your existing GitLab project!

Next Steps​

You have successfully completed the installation of Gruntwork Pipelines in an existing GitLab project. Proceed to Deploying your first infrastructure change to begin deploying changes.

Troubleshooting Tips​

If you encounter issues during the setup process, here are some common troubleshooting steps:

Bootstrap Resources Failure​

If your bootstrap resource provisioning fails:

HCL Configuration Issues​

If your HCL configurations aren't working as expected:

GitLab CI Pipeline Issues​

If your GitLab CI pipeline isn't working as expected: