Summary of mitigations
With this production design in mind, let’s take a look at how each of the design decisions addresses the concerns of the threat model:
Minimal access to target environments
All the infrastructure is deployed from within the accounts using a serverless platform. This means that attackers
that gain access to the underlying AWS secrets used by the CI environments will at most have the ability to run
deployments against a predefined set of code. This means that external attackers who do not have access to the source
code will at most be able to: (a) deploy code that has already been deployed before, (b) see the plan of the
infrastructure between two points of time. They will not be able to write arbitrary infrastructure code to read DB
secrets, for example. The IAM policies are set up such that the IAM user for CI only has
access to trigger predefined events. They do not have access to arbitrarily invoke the ECS task, as that could
potentially expose arbitrary deployments by modifying the command property (e.g., use a command to
infrastructure code and run
- Note that there is still a risk of rolling back the existing infrastructure by attempting to deploy a previous
version. See below for potential ways to mitigate this type of attack.
- Similarly, this alone does not mitigate threats from internal attackers who have access to the source code, as a
potential attacker with access to the source code can write arbitrary code to destroy or lookup arbitrary
infrastructure in the target environment. See below for potential ways to mitigate this type of attack.
Minimal options for deployment
The Lambda function exposes a minimal interface for triggering deployments. Attackers will only be able to trigger a deployment against a known repo and known git refs (branches, tags, etc). To further limit the scope, the lambda function can be restricted to only allow references to repositories that matches a predefined regular expression. Terraform Enterprise exposes similar configuration parameters to restrict what deployments can be triggered. This prevents attackers from creating an open source repo with malicious code that they subsequently deploy by pointing the deploy runner to it.
Restricted refs for
Since many CI systems depend on the pipeline being managed as code in the same repository, internal attackers can
easily circumvent approval flows by modifying the CI configuration on a test branch. This means that potential
attackers can run an
apply to destroy the environment or open backdoors by running infrastructure code from test
branches without having the code approved. To mitigate this, the Lambda function allows specifying a list of git refs
(branches, tags, etc) as the source of
apply-all. If you limit the source of
apply to only protected
branches (see below), it prevents attackers from having the ability to run
apply unless it has been reviewed.
CI server does not need access to the source code
Since the deployments are being done remotely in separate infrastructure, the actual CI server does not need to make any modifications to the code for the deployment. You can limit the CI server to read only access to the underlying repository, limiting the damage from a potential breach of the CI server.
These mitigations alone will not prevent all attacks defined in the threat model. For example, an internal
attacker with access to the source code can still do damage to the target environments by merging in code that removes
all the infrastructure resources, thereby destroying all infrastructure when the
apply command is run. Or, they could
expose secrets by writing infrastructure code that will leak the secrets in the logs via a
However, the reality is that any CI/CD solution can likely be compromised if an attacker has full access to your source code.
For these types of threats, your best bet is to implement various policies and controls on the source control repository and build configurations:
Use approval flows
In addition to providing a moment to pause and inspect the exact infrastructure changes that are about to be deployed, approval workflows in the CI server can mitigate attacks such that attackers will need enough privileges on the CI server to approve builds in order to actually modify infrastructure. This can mitigate potential attacks where the attacker has access to the CI server to trigger arbitrary builds manually (e.g., to run a previous job that is deploying an older version to roll back the infrastructure), but not enough access to approve the job. Note that this will not mitigate potential threats from internal attackers who have enough permissions to approve builds.
Lock down VCS systems
As mentioned in the previous section, it is important that you implement various controls on the VCS repositories. Once you implement a CI/CD pipeline, access to source code translates to access to your infrastructure environments, so you will want to reflect the same kind of security controls you implement on your environments in your VCS repositories.
Avoid logging secrets
Our threat model assumes that attackers can get access to the CI servers, which means they will have access to the
deployment logs. This will include detailed outputs from a
terraform plan or
apply. While it is impossible to
prevent terraform from leaking secrets into the state, it is possible to avoid terraform from logging sensitive
information. Make use of PGP encryption functions or encrypted environment variables / config files (in the case of
service deployments) to ensure sensitive data does not show up in the plan output. Additionally, tag sensitive outputs
sensitive keyword so that terraform will mask the outputs.