Skip to main content

Update Grouping & Pull Request Strategy

Overview​

When working with enterprise IaC repositories you'll often find that you tend to reuse modules many times across a single repository. This creates a scenario where when a dependent module releases a new version, you are prompted with updating your reference to that dependency many times. This is in contrast to the traditional dependency update scenario for application codebases in e.g. a Java or NodeJS codebase, where a single dependency update may result in a change in a single change in a gradle.build or package.json file. In those environments, a pull request to bump a version would likely be only a few lines of code, and the build with those updates would progress through CI to reach higher environments in sequence. In IaC, you often have those environments repeated within a codebase, and references to those dependencies in many places. Patcher thus has a choice - when a dependency releases a new version, how to divide the diff of applying those updates into pull requests.

The naive option is to simply make one pull request per dependency update found within a repository. At scale, this quickly becomes unmanageable. Patcher does support this behavior, but it also includes the ability to create PRs to address all of those changes in a single PR, to create a single PR per environment (e.g. update everything in dev), to create a PR per dependency update (e.g. update every instance of dependency XYZ), or to make a PR per dependency update per environment (update dependency XYZ in dev).

In summary, there are five ways to group pull requests generated by Patcher:

  1. Full-Consolidation: One PR to update everything.
  2. No-Consolidation: One PR for each update.
  3. Dependency-Only Consolidation: One PR per dependency. This allows you to merge one dependency at a time to all environments where they are used.
  4. Environment-Only Consolidation: One PR per environment. This allows you to promote all dependencies as a group through environments one at a time.
  5. (Environment x Dependency) Consolidation: One per dependency per environment. This allows you to promote a single dependency at a time through environments

Grouping Examples​

To clarify via example, let's imagine we have the following repository

/dev/unit1/terragrunt.hcl -> dependency1@1.0.0
/dev/unit2/terragrunt.hcl -> dependency1@1.0.0
/dev/unit3/terragrunt.hcl -> dependency2@1.0.0
/prod/unit1/terragrunt.hcl -> dependency1@1.0.0
/prod/unit2/terragrunt.hcl -> dependency1@1.0.0
/prod/unit3/terragrunt.hcl -> dependency2@1.0.0
/prod/unit4/terragrunt.hcl -> dependency3@1.0.0

In this example, assuming all 3 dependencies have newer versions available, the 5 strategies would result in:

  1. Full-Consolidation: One pull request with an update to all 7 units
  2. No-Consolidation: Seven pull requests, one for each unit
  3. Dependency-Only Consolidation: Three pull requests, one for each dependency, with the PRs for dependency1 and dependency2 updating both dev and prod.
  4. Environment-Only Consolidation: Two pull requests, one for dev and one for prod.
  5. (Environment x Dependency) Consolidation: Five pull requests, two for dev and three for prod.

Terminology​

  • unit A unit refers to a folder containing a terragrunt.hcl file, and thus a single corresponding OpenTofu state file. A unit may specify one or multiple modules as dependencies.
  • dependency (also referred to as target) A dependency is an OpenTofu module that is referenced by ref (usually a full source code path AND a version number) inside your unit. Patcher understands the semantics of semantic versioning on dependency refs.
  • environment is a logical grouping of infrastructure to represent your application environments, such as dev or prod. An environment usually contains multiple units and thus many dependencies. Generally IaC environments are similar to each other, and represented as a folder structure in your repository.
  • update is the the act of changing a single instance of a dependency to reference a newer version and to accommodate any breaking changes.
    info

    As of November 2024 Patcher's understanding of environments is limited to groupings of folders matched with glob patterns. E.g. dev is all folders matching dev-*, prod is all folders matching prod-*. Pipelines has a sophisticated HCL configuration syntax that allows for much more powerful definitions of environments. It is planned that Patcher will be able to leverage this method of defining environments in the future. Let us know if this expanded definitional capability is important to your use case.

Implementation Discussion​

When generating pull requests in CI, the workflow generally involves first invoking patcher report to identify what updates are available, then patcher update to create PRs applying those updates. Patcher does not have a built-in way to specify the grouping strategy by name, instead the grouping strategies are implemented with flags to the report and update commands.

Patcher Report​

Patcher report accepts a --include-dirs argument. When passed, this will cause patcher report to only output updates matching the glob pattern specified. This allows the developer to in essence limit the resulting PR to only a single environment. By invoking patcher report multiple times with different --include-dirs arguments, the developer can in effect create different PR workflows for each environment.

Patcher report outputs in JSON which can be inspected or iterated over to achieve desired behaviors.

Patcher Update​

Patcher update accepts a --target argument. This argument can be specified one or many times, and limits which dependencies patcher will include in the generated pull request. By invoking patcher update multiple times with different --target arguments, the developer can in effect create different PR workflows for each dependency.

Implementation Walkthrough​

Full-Consolidation​

To implement a full-consolidation workflow, do not pass either --include-dirs or --target arguments to patcher report and update. The output of patcher report will include all updates that are available, and the effect of patcher update will be to create a single PR that includes all targets (dependencies) across all folders.

Pseudocode:

run patcher report
run patcher update

No-Consolidation​

Patcher report has two types of outputs, a plan and a spec. Generally we use the spec file as its simpler and contains the information needed for most workflows. To achieve a no-consolidation workflow, however, we need the full plan.

Pseudocode:

run patcher report --output-plan plan.json
for each dependency in plan.json
    for each usage of the dependency:
        cd to the directory of the usage, e.g. (dirname usage.source.file)
        run patcher update --target $dependency.org/$dependency.repo/$dependency.module

Dependency-Only Consolidation​

To implement a dependency-only consolidation workflow, run patcher update without any --include-dirs arguments. This will create a single report output JSON file for all updates. Iterate over the targets in that JSON file and run patcher update once per identified target.

Pseudocode:

run patcher report without an include-dirs argument
for each $target in the output of patcher report
    run patcher update --target=$target

Environment-Only Consolidation​

To implement a environment-only consolidation workflow you need to iterate over each of your environments, passing in the environment to --include-dirs to report and then pass the complete output from report to a single update command.

Pseudocode:

for each $environment
    run patcher report -include-dirs=$environment
    run patcher update without any target argument

(Environment x Dependency) Consolidation​

To implement a environment x dependency consolidation workflow you will need to iterate over each of your environments, passing in the environment to --include-dirs to report and then iterating over each target in the report output and passing --target to update.

Pseudocode:

for each $environment (identified as a glob pattern of folders)
  run patcher report --include-dirs=$environment
  for each $target in the output of patcher report
    run patcher update --target=$target

Pseudocode: