Amazon EKS 0.67.1Last updated in version 0.66.0

EKS K8s Karpenter Module

This Module can be used to deploy Karpenter as an alternative to the Cluster Autoscaler for autoscaling capabilities of an EKS cluster. This module will create all of the necessary resources for a functional installation of Karpenter as well as the installation of Karpenter. This module does not create Karpenter Provisioners or Node Templates, only the installation of the Karpenter Controller. See the Karpenter Example for an example of how to deploy the additional CRDs (Provisioners, Node Templates, etc) to the EKS cluster.

Note: For EKS cluster autoscaling capabilities, either Karpenter OR the cluster-autoscaler should be used; not both. To migrate to using karpenter instead of the cluster-autoscaler see Migrating to Karpenter from the Cluster Autoscaler

To leverage the full power and potential of Karpenter, one must understand the Karpenter Core Concepts. Deploying this module without additional configuration (ie deploying Karpenter CRDs) will not enable EKS cluster autoscaling. As use-cases are presented, we will do our best effort to continue to add meaningful examples to the examples folder to help ease the complexities of configuring Karpenter. At minimum, one should configure and deploy a default Provisioner and Node Template for just in time node provisioning via Karpenter.

Resources Created

This module will create the following core resources, some of which are optional which are noted in the input variables:

AWS Resource	Description
Karpenter Node IAM Role	IAM Role used by Karpenter Nodes provisioned by the Karpenter Controller
Karpenter Node Instance Profile	IAM Instance Profile attached to EC2 instances launched by the Karpenter Controller
Karpenter Controller IRSA	IAM Role for Service Account (IRSA) to be used by the Karpenter Controller
Karpenter Helm Release	Karpenter deployment to EKS via Helm
Karpenter SQS Queue	SQS Queue used to listen to EC2 events

Additional supporting resources do exist in addition to the table above, such as IAM Policy Documents, Attachments and CloudWatch rules.

What is Karpenter?

From the Karpenter official docs:

Karpenter is an open-source node provisioning project built for Kubernetes. Adding Karpenter to a Kubernetes cluster can dramatically improve the efficiency and cost of running workloads on that cluster.
Karpenter works by:
Watching for pods that the Kubernetes scheduler has marked as unschedulable
Evaluating scheduling constraints (resource requests, nodeselectors, affinities, tolerations, and topology spread constraints) requested by the pods
Provisioning nodes that meet the requirements of the pods
Removing the nodes when the nodes are no longer needed

For additional details and in-depth information on Karpenter, please see the Karpenter Docs Site.

How to enable de Deprovisioning based on EC2 events?

This is used to inform Karpenter of EC2 events that affect the cluster capacity ( Spot Interruption Warnings, Scheduled Change Health Events (Maintenance Events) , Instance Terminating Events, Instance Stopping Events).

This is particularly useful to users that rely on Spot Instances that can be terminated at will.

For more information read the Karpenter Intrerruption section

From variables.tf enable create_karpenter_deprovisioning_queuet

Sample Usage

Terraform
Terragrunt

main.tf

# ------------------------------------------------------------------------------------------------------
# DEPLOY GRUNTWORK'S EKS-K8S-KARPENTER MODULE
# ------------------------------------------------------------------------------------------------------

module "eks_k_8_s_karpenter" {

  source = "git::git@github.com:gruntwork-io/terraform-aws-eks.git//modules/eks-k8s-karpenter?ref=v0.67.1"

  # ----------------------------------------------------------------------------------------------------
  # REQUIRED VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # The AWS Region to create resources in.
  aws_region = <string>

  # URL endpoint of the Kubernetes control plane provided by EKS.
  eks_cluster_endpoint = <string>

  # The name of the EKS Cluster that Karpenter will be deployed to.
  eks_cluster_name = <string>

  # The ARN of the EKS OIDC provider. This is required if creating IRSA for the
  # Karpenter Controller.
  eks_openid_connect_provider_arn = <string>

  # The URL of the EKS OIDC provider. This is required if creating IRSA for the
  # Karpenter Controller.
  eks_openid_connect_provider_url = <string>

  # ----------------------------------------------------------------------------------------------------
  # OPTIONAL VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # Optionally create an IAM Role for Service Account (IRSA) for the Karpenter
  # Controller.
  create_karpenter_controller_irsa = true

  # Conditional flag to optionally create the Karpenter Deprovisioning Queue.
  create_karpenter_deprovisioning_queue = true

  # Conditional flag to create the Karpenter Node IAM Role. If this is set to
  # false, then an existing IAM Role must be provided with the
  # `karpenter_node_iam_role_arn` variable
  create_karpenter_node_iam_role = true

  # Conditional flag to optionally create resources in this module.
  create_resources = true

  # Additional Helm chart values to pass to the Karpenter Helm chart. See the
  # official Karpenter Helm chart values file and documentation for available
  # configuration options.
  karpenter_chart_additional_values = {}

  # The Helm chart name for the Karpenter chart.
  karpenter_chart_name = "karpenter"

  # The k8s namespace that the Karpenter Helm chart will be deployed to.
  karpenter_chart_namespace = "karpenter"

  # The Helm release name for the Karpenter chart.
  karpenter_chart_release_name = "karpenter"

  # The Helm repository to obtain the Karpenter chart from.
  karpenter_chart_repository = "oci://public.ecr.aws/karpenter"

  # Whether or not to install CRDs with the Karpenter Helm Chart. This should be
  # set to true if using the karpenter-crd Helm Chart
  # (karpenter_chart_additional_values = true).
  karpenter_chart_skip_crds = false

  # The version of the Karpenter Helm chart.
  karpenter_chart_version = "v0.32.7"

  # Provide an existing IAM Role ARN to be used with the Karpenter Controller
  # Service Account. This is required if `create_karpenter_controller_irsa` is
  # set to false.
  karpenter_controller_existing_role_arn = null

  # The Helm chart name for the Karpenter CRD chart.
  karpenter_crd_chart_name = "karpenter-crd"

  # The k8s namespace that the Karpenter CRD Helm chart will be deployed to.
  karpenter_crd_chart_namespace = "karpenter"

  # The Helm release name for the Karpenter CRD chart.
  karpenter_crd_chart_release_name = "karpenter-crd"

  # The Helm repository to obtain the Karpenter CRD chart from.
  karpenter_crd_chart_repository = "oci://public.ecr.aws/karpenter"

  # The version of the Karpenter CRD Helm chart. This should typically be the
  # same version as karpenter_chart_version.
  karpenter_crd_chart_version = "v0.32.7"

  # Whether or not to create the Karpneter CRDs via the karpenter-crd Helm
  # chart. It is suggested to manage the Karpenter CRDs via this Helm chart.
  karpenter_crd_helm_create = true

  # Additional tags to add to the Karpenter Deprovisioning Queue.
  karpenter_deprovisioning_queue_tags = {}

  # A tag that is used by Karpenter to discover resources.
  karpenter_discovery_tag = "karpenter.sh/discovery"

  # ARN of the policy that is used to set the permissions boundary for the role.
  karpenter_irsa_permissions_boundary = null

  # Use an existing IAM Role to be used for the Karpenter Node Instance Profile.
  # This is required if `create_karpenter_node_iam_role` is set to false. This
  # should be the ARN of the IAM Role.
  karpenter_node_existing_iam_role_arn = null

  # Use an existing IAM Role to be used for the Karpenter Node Instance Profile.
  # This is required if `create_karpenter_node_iam_role` is set to false. This
  # should be the Name of the IAM Role.
  karpenter_node_existing_iam_role_name = null

  # A description of the Karpenter Node IAM Role.
  karpenter_node_iam_role_description = "IAM Role attached to nodes launched by Karpenter."

  # Maximum session duration (in seconds) that you want to set for the Karpenter
  # Node role. Value can be between 3600 and 43200.
  karpenter_node_iam_role_max_session_duration = 3600

  # Optionally provide a name for the Karpenter Node IAM Role. If unset, a name
  # will be generated.
  karpenter_node_iam_role_name = null

  # Optionally provide a path to the Karpenter Node IAM Role.
  karpenter_node_iam_role_path = null

  # ARN of the policy that is used to set the permissions boundary for the role.
  karpenter_node_iam_role_permissions_boundary = null

  # Additional tags to add to the Karpenter Node IAM Role.
  karpenter_node_iam_role_tags = {}

  # Optionally use an IAM name prefix for the Karpenter IAM Role.
  karpenter_node_iam_role_use_name_prefix = false

  # Optionally provide a name for the Karpenter service account that will be
  # associated with IRSA.
  karpenter_service_account_name = "karpenter"

}

terragrunt.hcl

# ------------------------------------------------------------------------------------------------------
# DEPLOY GRUNTWORK'S EKS-K8S-KARPENTER MODULE
# ------------------------------------------------------------------------------------------------------

terraform {
  source = "git::git@github.com:gruntwork-io/terraform-aws-eks.git//modules/eks-k8s-karpenter?ref=v0.67.1"
}

inputs = {

  # ----------------------------------------------------------------------------------------------------
  # REQUIRED VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # The AWS Region to create resources in.
  aws_region = <string>

  # URL endpoint of the Kubernetes control plane provided by EKS.
  eks_cluster_endpoint = <string>

  # The name of the EKS Cluster that Karpenter will be deployed to.
  eks_cluster_name = <string>

  # The ARN of the EKS OIDC provider. This is required if creating IRSA for the
  # Karpenter Controller.
  eks_openid_connect_provider_arn = <string>

  # The URL of the EKS OIDC provider. This is required if creating IRSA for the
  # Karpenter Controller.
  eks_openid_connect_provider_url = <string>

  # ----------------------------------------------------------------------------------------------------
  # OPTIONAL VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # Optionally create an IAM Role for Service Account (IRSA) for the Karpenter
  # Controller.
  create_karpenter_controller_irsa = true

  # Conditional flag to optionally create the Karpenter Deprovisioning Queue.
  create_karpenter_deprovisioning_queue = true

  # Conditional flag to create the Karpenter Node IAM Role. If this is set to
  # false, then an existing IAM Role must be provided with the
  # `karpenter_node_iam_role_arn` variable
  create_karpenter_node_iam_role = true

  # Conditional flag to optionally create resources in this module.
  create_resources = true

  # Additional Helm chart values to pass to the Karpenter Helm chart. See the
  # official Karpenter Helm chart values file and documentation for available
  # configuration options.
  karpenter_chart_additional_values = {}

  # The Helm chart name for the Karpenter chart.
  karpenter_chart_name = "karpenter"

  # The k8s namespace that the Karpenter Helm chart will be deployed to.
  karpenter_chart_namespace = "karpenter"

  # The Helm release name for the Karpenter chart.
  karpenter_chart_release_name = "karpenter"

  # The Helm repository to obtain the Karpenter chart from.
  karpenter_chart_repository = "oci://public.ecr.aws/karpenter"

  # Whether or not to install CRDs with the Karpenter Helm Chart. This should be
  # set to true if using the karpenter-crd Helm Chart
  # (karpenter_chart_additional_values = true).
  karpenter_chart_skip_crds = false

  # The version of the Karpenter Helm chart.
  karpenter_chart_version = "v0.32.7"

  # Provide an existing IAM Role ARN to be used with the Karpenter Controller
  # Service Account. This is required if `create_karpenter_controller_irsa` is
  # set to false.
  karpenter_controller_existing_role_arn = null

  # The Helm chart name for the Karpenter CRD chart.
  karpenter_crd_chart_name = "karpenter-crd"

  # The k8s namespace that the Karpenter CRD Helm chart will be deployed to.
  karpenter_crd_chart_namespace = "karpenter"

  # The Helm release name for the Karpenter CRD chart.
  karpenter_crd_chart_release_name = "karpenter-crd"

  # The Helm repository to obtain the Karpenter CRD chart from.
  karpenter_crd_chart_repository = "oci://public.ecr.aws/karpenter"

  # The version of the Karpenter CRD Helm chart. This should typically be the
  # same version as karpenter_chart_version.
  karpenter_crd_chart_version = "v0.32.7"

  # Whether or not to create the Karpneter CRDs via the karpenter-crd Helm
  # chart. It is suggested to manage the Karpenter CRDs via this Helm chart.
  karpenter_crd_helm_create = true

  # Additional tags to add to the Karpenter Deprovisioning Queue.
  karpenter_deprovisioning_queue_tags = {}

  # A tag that is used by Karpenter to discover resources.
  karpenter_discovery_tag = "karpenter.sh/discovery"

  # ARN of the policy that is used to set the permissions boundary for the role.
  karpenter_irsa_permissions_boundary = null

  # Use an existing IAM Role to be used for the Karpenter Node Instance Profile.
  # This is required if `create_karpenter_node_iam_role` is set to false. This
  # should be the ARN of the IAM Role.
  karpenter_node_existing_iam_role_arn = null

  # Use an existing IAM Role to be used for the Karpenter Node Instance Profile.
  # This is required if `create_karpenter_node_iam_role` is set to false. This
  # should be the Name of the IAM Role.
  karpenter_node_existing_iam_role_name = null

  # A description of the Karpenter Node IAM Role.
  karpenter_node_iam_role_description = "IAM Role attached to nodes launched by Karpenter."

  # Maximum session duration (in seconds) that you want to set for the Karpenter
  # Node role. Value can be between 3600 and 43200.
  karpenter_node_iam_role_max_session_duration = 3600

  # Optionally provide a name for the Karpenter Node IAM Role. If unset, a name
  # will be generated.
  karpenter_node_iam_role_name = null

  # Optionally provide a path to the Karpenter Node IAM Role.
  karpenter_node_iam_role_path = null

  # ARN of the policy that is used to set the permissions boundary for the role.
  karpenter_node_iam_role_permissions_boundary = null

  # Additional tags to add to the Karpenter Node IAM Role.
  karpenter_node_iam_role_tags = {}

  # Optionally use an IAM name prefix for the Karpenter IAM Role.
  karpenter_node_iam_role_use_name_prefix = false

  # Optionally provide a name for the Karpenter service account that will be
  # associated with IRSA.
  karpenter_service_account_name = "karpenter"

}

Reference

Inputs
Outputs

Required

aws_regionstringrequired

The AWS Region to create resources in.

eks_cluster_endpointstringrequired

URL endpoint of the Kubernetes control plane provided by EKS.

eks_cluster_namestringrequired

The name of the EKS Cluster that Karpenter will be deployed to.

eks_openid_connect_provider_arnstringrequired

The ARN of the EKS OIDC provider. This is required if creating IRSA for the Karpenter Controller.

eks_openid_connect_provider_urlstringrequired

The URL of the EKS OIDC provider. This is required if creating IRSA for the Karpenter Controller.

Optional

create_karpenter_controller_irsabooloptional

Optionally create an IAM Role for Service Account (IRSA) for the Karpenter Controller.

Default:true

create_karpenter_deprovisioning_queuebooloptional

Conditional flag to optionally create the Karpenter Deprovisioning Queue.

Default:true

create_karpenter_node_iam_rolebooloptional

Conditional flag to create the Karpenter Node IAM Role. If this is set to false, then an existing IAM Role must be provided with the karpenter_node_iam_role_arn variable

Default:true

create_resourcesbooloptional

Conditional flag to optionally create resources in this module.

Default:true

karpenter_chart_additional_valuesanyoptional

Additional Helm chart values to pass to the Karpenter Helm chart. See the official Karpenter Helm chart values file and documentation for available configuration options.

Type Details

Any types represent complex values of variable type. For details, please consult `variables.tf` in the source repo.

Default:{}

karpenter_chart_namestringoptional

The Helm chart name for the Karpenter chart.

Default:"karpenter"

karpenter_chart_namespacestringoptional

The k8s namespace that the Karpenter Helm chart will be deployed to.

Default:"karpenter"

karpenter_chart_release_namestringoptional

The Helm release name for the Karpenter chart.

Default:"karpenter"

karpenter_chart_repositorystringoptional

The Helm repository to obtain the Karpenter chart from.

Default:"oci://public.ecr.aws/karpenter"

karpenter_chart_skip_crdsbooloptional

Whether or not to install CRDs with the Karpenter Helm Chart. This should be set to true if using the karpenter-crd Helm Chart (karpenter_chart_additional_values = true).

Default:false

karpenter_chart_versionstringoptional

The version of the Karpenter Helm chart.

Default:"v0.32.7"

karpenter_controller_existing_role_arnstringoptional

Provide an existing IAM Role ARN to be used with the Karpenter Controller Service Account. This is required if create_karpenter_controller_irsa is set to false.

Default:null

karpenter_crd_chart_namestringoptional

The Helm chart name for the Karpenter CRD chart.

Default:"karpenter-crd"

karpenter_crd_chart_namespacestringoptional

The k8s namespace that the Karpenter CRD Helm chart will be deployed to.

Default:"karpenter"

karpenter_crd_chart_release_namestringoptional

The Helm release name for the Karpenter CRD chart.

Default:"karpenter-crd"

karpenter_crd_chart_repositorystringoptional

The Helm repository to obtain the Karpenter CRD chart from.

Default:"oci://public.ecr.aws/karpenter"

karpenter_crd_chart_versionstringoptional

The version of the Karpenter CRD Helm chart. This should typically be the same version as karpenter_chart_version.

Default:"v0.32.7"

karpenter_crd_helm_createbooloptional

Whether or not to create the Karpneter CRDs via the karpenter-crd Helm chart. It is suggested to manage the Karpenter CRDs via this Helm chart.

Default:true

karpenter_deprovisioning_queue_tagsmap(string)optional

Additional tags to add to the Karpenter Deprovisioning Queue.

Default:{}

karpenter_discovery_tagstringoptional

A tag that is used by Karpenter to discover resources.

Default:"karpenter.sh/discovery"

karpenter_irsa_permissions_boundarystringoptional

ARN of the policy that is used to set the permissions boundary for the role.

Default:null

karpenter_node_existing_iam_role_arnstringoptional

Use an existing IAM Role to be used for the Karpenter Node Instance Profile. This is required if create_karpenter_node_iam_role is set to false. This should be the ARN of the IAM Role.

Default:null

karpenter_node_existing_iam_role_namestringoptional

Use an existing IAM Role to be used for the Karpenter Node Instance Profile. This is required if create_karpenter_node_iam_role is set to false. This should be the Name of the IAM Role.

Default:null

karpenter_node_iam_role_descriptionstringoptional

A description of the Karpenter Node IAM Role.

Default:"IAM Role attached to nodes launched by Karpenter."

karpenter_node_iam_role_max_session_durationnumberoptional

Maximum session duration (in seconds) that you want to set for the Karpenter Node role. Value can be between 3600 and 43200.

Default:3600

karpenter_node_iam_role_namestringoptional

Optionally provide a name for the Karpenter Node IAM Role. If unset, a name will be generated.

Default:null

karpenter_node_iam_role_pathstringoptional

Optionally provide a path to the Karpenter Node IAM Role.