Skip to main content

Adding Pipelines to a GitLab Project

This guide walks you through the process of adding Gruntwork Pipelines to a GitLab project. By the end, you'll have a fully configured GitLab CI/CD pipeline that can deploy infrastructure changes automatically.

Prerequisites​

Before you begin, make sure you have:

  • Basic familiarity with Git, GitLab, and infrastructure as code concepts
  • Access to one (or many) AWS account(s) where you have permission to create IAM roles and OIDC providers
  • Completed the Pipelines Auth setup for GitLab and setup a machine user with appropriate PAT tokens
  • Local access to Gruntwork's GitHub repositories, specifically boilerplate and the architecture catalog
info

For custom GitLab instances only: You must fork Gruntwork's public Pipelines workflow project into your own GitLab instance.

This is necessary because Gruntwork Pipelines uses GitLab CI/CD components, and GitLab requires components to reside within the same GitLab instance as the project referencing them.

When creating the fork, we recommend configuring it as a public mirror of the original Gruntwork project and ensuring that tags are included.

Setup Process Overview​

Setting up Gruntwork Pipelines for GitLab involves these main steps:

(prerequisite) Complete the Pipelines Auth setup for GitLab

  1. Authorize Your GitLab Group with Gruntwork
  2. Install required tools (mise, boilerplate)
  3. Install Gruntwork Pipelines in Your Repository
  4. Install AWS OIDC Provider and IAM Roles for Pipelines
  5. Complete the setup

Detailed Setup Instructions​

Step 0: Ensure OIDC configuration and JWKS are publicly accessible​

This step only applies if you are using a self-hosted GitLab instance that is not accessible from the public internet. If you are using GitLab.com or a self-hosted instance that is publicly accessible, you can skip this step.

  1. Follow GitLab's instructions for hosting your OIDC configuration and JWKS in a public location (e.g. S3 Bucket). This is necessary for both Gruntwork and the AWS OIDC provider to access the GitLab OIDC configuration and JWKS when authenticating JWT's generated by your custom instance.
  2. Note the <ISSUER_URL> (stored as ci_id_tokens_issuer_url in your gitlab.rb file per GitLab's instructions) generated above for reuse in the next steps.

Step 1: Authorize Your GitLab Group with Gruntwork​

To use Gruntwork Pipelines with GitLab, your group needs authorization from Gruntwork:

  1. Email your Gruntwork account manager or support@gruntwork.io with:

    GitLab group name(s): <GITLAB_GROUP_NAME> (e.g. acme-io)
    GitLab Issuer URL: <ISSUER_URL> (For most users this is the URL of your GitLab instance e.g. https://gitlab.acme.io. If your instance is not publicly accessible, this should be a separate URL that is publicly accessible per step 0, e.g. https://s3.amazonaws.com/YOUR_BUCKET_NAME/)
    Organization name: <ORGANIZATION_NAME> (e.g. Acme, Inc.)

  2. Wait for confirmation that your group has been authorized.

Step 2: Install Required Tools​

First, you'll need to install mise, a powerful environment manager that will help set up the required tools:

  1. Install mise by following the getting started guide

  2. Activate mise in your shell:

    # For Bash
    echo 'eval "$(~/.local/bin/mise activate bash)"' >> ~/.bashrc

    # For Zsh
    echo 'eval "$(~/.local/bin/mise activate zsh)"' >> ~/.zshrc

    # For Fish
    echo 'mise activate fish | source' >> ~/.config/fish/config.fish

  3. Install the boilerplate tool, which will generate the project structure:

    # For mise version BEFORE 2025.2.10
    mise plugin add boilerplate https://github.com/gruntwork-io/asdf-boilerplate.git

    # For mise version 2025.2.10+
    mise plugin add boilerplate

    mise use boilerplate@0.6.0

  4. Verify the installation:

    boilerplate --version

    # If that doesn't work, try:
    mise x -- boilerplate --version

    # If that still doesn't work, check where boilerplate is installed:
    mise which boilerplate

Step 3: Install Gruntwork Pipelines in Your Repository​

  1. Identify where you want to install Gruntwork Pipelines, for example create a new project/repository in your GitLab group (or use an existing one) named <REPOSITORY_NAME>

  2. Clone the repository to your local machine if it's not already cloned:

    git clone git@gitlab.com:<GITLAB_GROUP_NAME>/<REPOSITORY_NAME>.git
    cd <REPOSITORY_NAME>

  3. Create a new branch for your changes:

    git checkout -b gruntwork-pipelines

  4. Download the sample vars.yaml file to the root of <REPOSITORY_NAME>

  5. Edit the vars.yaml file to customize it for your environment. If using a custom GitLab instance, update any custom instance variables.

  6. cd to the root of <REPOSITORY_NAME> where you wish to install Gruntwork Pipelines. Run the boilerplate tool to generate your repository structure:

    boilerplate --template-url "git@github.com:gruntwork-io/terraform-aws-architecture-catalog.git//templates/gitlab-pipelines-infrastructure-live-root/?ref=v2.13.0" --output-folder . --var-file vars.yaml --non-interactive

    If you encounter SSH issues, verify your SSH access to GitHub:

    ssh -T git@github.com
    # or try cloning manually
    git clone git@github.com:gruntwork-io/terraform-aws-architecture-catalog.git

  7. Commit the changes:

    git add .
    git commit -m "[skip ci] Add Gruntwork Pipelines"
    git push origin gruntwork-pipelines

  8. Create a merge request in GitLab and review the changes.

Step 4: Install AWS OIDC Provider and IAM Roles for Pipelines​

  1. Navigate to the _global folder under each account in your repository and review the Terragrunt files that were created:

    • The GitLab OIDC identity provider in AWS.

      note

      If using a custom GitLab instance, ensure the URL and audiences inputs in this configuration are correct.

    • IAM roles for your the account (root-pipelines-plan and root-pipelines-apply)

  2. Apply these configurations to create the required AWS resources:

    cd <ACCOUNT_NAME>/_global/
    terragrunt run-all plan
    terragrunt run-all apply
    note

    In the event you already have an OIDC provider for your SCM in the AWS account you can import the existing one:

    cd _global/<ACCOUNT_NAME>/gitlab-pipelines-openid-connect-provider/
    terragrunt import "aws_iam_openid_connect_provider.gitlab" "ARN_OF_EXISTING_OIDC_PROVIDER"

Step 5: Complete the Setup​

  1. Return to GitLab and merge the merge request with your changes.
  2. Ensure that PIPELINES_GITLAB_TOKEN and PIPELINES_GITLAB_READ_TOKEN are set as a CI/CD variables in your group or project if you haven't already (see the Machine Users setup guide for details).
  3. Test your setup by creating a new branch with some sample infrastructure code and creating a merge request.

Next Steps​

After setting up Pipelines, you can:

Troubleshooting​

If you encounter issues during setup:

  • Ensure your GitLab CI user has the correct permissions to your group and projects
  • Verify that both PIPELINES_GITLAB_TOKEN and PIPELINES_GITLAB_READ_TOKEN are set correctly as CI/CD variables and are NOT marked as protected
  • Confirm your GitLab group has been authorized by Gruntwork for Pipelines usage

For further assistance, contact support@gruntwork.io.