Skip to main content
Service Catalog Version 0.116.1Last updated in version 0.114.0

Amazon EKS Core Services

View Source Release Notes

Overview​

This service contains Terraform and Helm code to deploy core administrative services, such as FluentD and the ALB Ingress Controller, onto Elastic Kubernetes Service(EKS).

EKS Core Services architectureEKS Core Services architecture

Features​

  • Deploy FluentD DaemonSet to ship container logs to CloudWatch Logs
  • Deploy ALB Ingress Controller to configure ALBs from within Kubernetes
  • Deploy external-dns to manage Route 53 DNS records from within Kubernetes
  • Deploy Kubernetes cluster-autoscaler to configure auto scaling of ASGs based on Pod demand
  • Deploy AWS CloudWatch Agent to configure container and node level metrics from worker nodes

Learn​

note

This repo is a part of the Gruntwork Service Catalog, a collection of reusable, battle-tested, production ready infrastructure code. If you’ve never used the Service Catalog before, make sure to read How to use the Gruntwork Service Catalog!

Under the hood, this is all implemented using Terraform modules from the Gruntwork terraform-aws-eks repo. If you are a subscriber and don’t have access to this repo, email support@gruntwork.io.

Core concepts​

For information on each of the core services deployed by this service, see the documentation in the terraform-aws-eks repo.

Repo organization​

  • modules: the main implementation code for this repo, broken down into multiple standalone, orthogonal submodules.
  • examples: This folder contains working examples of how to use the submodules.
  • test: Automated tests for the modules and examples.

Deploy​

Non-production deployment (quick start for learning)​

If you just want to try this repo out for experimenting and learning, check out the following resources:

  • examples/for-learning-and-testing folder: The examples/for-learning-and-testing folder contains standalone sample code optimized for learning, experimenting, and testing (but not direct production usage).

Production deployment​

If you want to deploy this repo in production, check out the following resources:

Sample Usage​

main.tf

# ------------------------------------------------------------------------------------------------------
# DEPLOY GRUNTWORK'S EKS-CORE-SERVICES MODULE
# ------------------------------------------------------------------------------------------------------

module "eks_core_services" {

  source = "git::git@github.com:gruntwork-io/terraform-aws-service-catalog.git//modules/services/eks-core-services?ref=v0.116.1"

  # ----------------------------------------------------------------------------------------------------
  # REQUIRED VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # The AWS region in which all resources will be created
  aws_region = <string>

  # The name of the EKS cluster where the core services will be deployed into.
  eks_cluster_name = <string>

  # Configuration for using the IAM role with Service Accounts feature to
  # provide permissions to the applications. This expects a map with two
  # properties: `openid_connect_provider_arn` and `openid_connect_provider_url`.
  # The `openid_connect_provider_arn` is the ARN of the OpenID Connect Provider
  # for EKS to retrieve IAM credentials, while `openid_connect_provider_url` is
  # the URL. Set to null if you do not wish to use IAM role with Service
  # Accounts.
  eks_iam_role_for_service_accounts_config = <object(
    openid_connect_provider_arn = string
    openid_connect_provider_url = string
  )>

  # ARN of IAM Role to use as the Pod execution role for Fargate. Required if
  # any of the services are being scheduled on Fargate. Set to null if none of
  # the Pods are being scheduled on Fargate.
  pod_execution_iam_role_arn = <string>

  # The ID of the VPC where the EKS cluster is deployed.
  vpc_id = <string>

  # The subnet IDs to use for EKS worker nodes. Used when provisioning Pods on
  # to Fargate. Required if any of the services are being scheduled on Fargate.
  # Set to empty list if none of the Pods are being scheduled on Fargate.
  worker_vpc_subnet_ids = <list(string)>

  # ----------------------------------------------------------------------------------------------------
  # OPTIONAL VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # ARN of IAM Role to assume to create and control ALB's. This is useful if
  # your VPC is shared from another account and needs to be created somewhere
  # else.
  alb_ingress_controller_alb_iam_role_arn = null

  # The version of the aws-load-balancer-controller helmchart to use.
  alb_ingress_controller_chart_version = "1.4.1"

  # Tags to apply to all AWS resources managed by this controller
  alb_ingress_controller_default_tags = {}

  # The repository of the aws-load-balancer-controller docker image that should
  # be deployed.
  alb_ingress_controller_docker_image_repo = "602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-load-balancer-controller"

  # The tag of the aws-load-balancer-controller docker image that should be
  # deployed.
  alb_ingress_controller_docker_image_tag = "v2.4.1"

  # Configure affinity rules for the ALB Ingress Controller Pod to control which
  # nodes to schedule on. Each item in the list should be a map with the keys
  # `key`, `values`, and `operator`, corresponding to the 3 properties of
  # matchExpressions. Note that all expressions must be satisfied to schedule on
  # the node.
  alb_ingress_controller_pod_node_affinity = []

  # Configure tolerations rules to allow the ALB Ingress Controller Pod to
  # schedule on nodes that have been tainted. Each item in the list specifies a
  # toleration rule.
  alb_ingress_controller_pod_tolerations = []

  # Minimum time to wait after a scale up event before any node is considered
  # for scale down.
  autoscaler_down_delay_after_add = "10m"

  # ARN of permissions boundary to apply to the autoscaler IAM role - the IAM
  # role created for the Autoscaler
  autoscaler_iam_role_permissions_boundary = null

  # Number for the log level verbosity. Lower numbers are less verbose, higher
  # numbers are more verbose. (Default: 4)
  autoscaler_log_level_verbosity = 4

  # Minimum time to wait since the node became unused before the node is
  # considered for scale down by the autoscaler.
  autoscaler_scale_down_unneeded_time = "10m"

  # If true cluster autoscaler will never delete nodes with pods with local
  # storage, e.g. EmptyDir or HostPath
  autoscaler_skip_nodes_with_local_storage = true

  # The Container repository to use for looking up the cloudwatch-agent
  # Container image when deploying the pods. When null, uses the default
  # repository set in the chart. Only applies to non-fargate workers.
  aws_cloudwatch_agent_image_repository = null

  # Configure affinity rules for the AWS CloudWatch Agent Pod to control which
  # nodes to schedule on. Each item in the list should be a map with the keys
  # `key`, `values`, and `operator`, corresponding to the 3 properties of
  # matchExpressions. Note that all expressions must be satisfied to schedule on
  # the node.
  aws_cloudwatch_agent_pod_node_affinity = []

  # Pod resource requests and limits to use. Refer to
  # https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
  # for more information.
  aws_cloudwatch_agent_pod_resources = null

  # Configure tolerations rules to allow the AWS CloudWatch Agent Pods to
  # schedule on nodes that have been tainted. Each item in the list specifies a
  # toleration rule.
  aws_cloudwatch_agent_pod_tolerations = []

  # Which version of amazon/cloudwatch-agent to install. When null, uses the
  # default version set in the chart. Only applies to non-fargate workers.
  aws_cloudwatch_agent_version = null

  # Restrict the cluster autoscaler to a list of absolute ASG ARNs upon initial
  # apply to ensure no new ASGs can be managed by the autoscaler without
  # explicitly running another apply. Setting this to false will ensure that the
  # cluster autoscaler is automatically given access to manage any new ASGs with
  # the k8s.io/cluster-autoscaler/CLUSTER_NAME tag applied.
  cluster_autoscaler_absolute_arns = true

  # The version of the cluster-autoscaler helm chart to deploy. Note that this
  # is different from the app/container version, which is sepecified with
  # var.cluster_autoscaler_version.
  cluster_autoscaler_chart_version = "9.21.0"

  # Annotations to apply to the cluster autoscaler pod(s), as key value pairs.
  cluster_autoscaler_pod_annotations = {}

  # Labels to apply to the cluster autoscaler pod(s), as key value pairs.
  cluster_autoscaler_pod_labels = {}

  # Configure affinity rules for the cluster-autoscaler Pod to control which
  # nodes to schedule on. Each item in the list should be a map with the keys
  # `key`, `values`, and `operator`, corresponding to the 3 properties of
  # matchExpressions. Note that all expressions must be satisfied to schedule on
  # the node.
  cluster_autoscaler_pod_node_affinity = []

  # Pod resource requests and limits to use. Refer to
  # https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
  # for more information. This is most useful for configuring CPU+Memory
  # availability for Fargate, which defaults to 0.25 vCPU and 256MB RAM.
  cluster_autoscaler_pod_resources = {"limits":{"cpu":"250m","memory":"1024Mi"},"requests":{"cpu":"250m","memory":"1024Mi"}}

  # Configure tolerations rules to allow the cluster-autoscaler Pod to schedule
  # on nodes that have been tainted. Each item in the list specifies a
  # toleration rule.
  cluster_autoscaler_pod_tolerations = []

  # The name to use for the helm release for cluster-autoscaler. This is useful
  # to force a redeployment of the cluster-autoscaler component.
  cluster_autoscaler_release_name = "cluster-autoscaler"

  # Which docker repository to use to install the cluster autoscaler. Check the
  # following link for valid repositories to use
  # https://github.com/kubernetes/autoscaler/releases
  cluster_autoscaler_repository = "registry.k8s.io/autoscaling/cluster-autoscaler"

  # ARN of IAM Role to use for the Cluster Autoscaler. Only used when
  # var.create_cluster_autoscaler_role is false.
  cluster_autoscaler_role_arn = null

  # Specifies an 'expander' for the cluster autoscaler. This helps determine
  # which ASG to scale when additional resource capacity is needed.
  cluster_autoscaler_scaling_strategy = "least-waste"

  # The name of the service account to create for the cluster autoscaler.
  cluster_autoscaler_service_account_name = "cluster-autoscaler-aws-cluster-autoscaler"

  # Which version of the cluster autoscaler to install. This should match the
  # major/minor version (e.g., v1.20) of your Kubernetes Installation. See
  # https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#releases
  # for a list of versions.
  cluster_autoscaler_version = "v1.30.0"

  # When set to true, create a new dedicated IAM Role for the cluster
  # autoscaler. When set to true, var.iam_role_for_service_accounts_config is
  # required.
  create_cluster_autoscaler_role = true

  # Whether or not to enable the AWS LB Ingress controller.
  enable_alb_ingress_controller = true

  # Whether to enable the AWS CloudWatch Agent DaemonSet for collecting
  # container and node metrics from worker nodes (self-managed ASG or managed
  # node groups).
  enable_aws_cloudwatch_agent = true

  # Whether or not to enable cluster-autoscaler for Autoscaling EKS worker
  # nodes.
  enable_cluster_autoscaler = true

  # Whether or not to enable external-dns for DNS entry syncing with Route 53
  # for Services and Ingresses.
  enable_external_dns = true

  # Whether or not to enable fluent-bit on EKS Fargate workers for log
  # aggregation.
  enable_fargate_fluent_bit = true

  # Whether or not to enable fluent-bit for log aggregation.
  enable_fluent_bit = true

  # Duration string (e.g. 1m) indicating the interval between making changes to
  # Route 53 by external-dns. When null, use the default defined in the chart
  # (1s).
  external_dns_batch_change_interval = null

  # The maximum number of changes that should be applied in a batch by
  # external-dns. When null, use the default defined in the chart (1000).
  external_dns_batch_change_size = null

  # Name of the Helm chart for external-dns. This should usually be
  # 'external-dns' but may differ in the case of overriding the repository URL.
  external_dns_chart_name = "external-dns"

  # Helm chart repository URL to obtain the external-dns chart from. Useful when
  # using Bitnami charts that are older than 6 months due to Bitnami's lifecycle
  # policy which removes older chart from the main index.
  external_dns_chart_repository_url = "https://charts.bitnami.com/bitnami"

  # The version of the helm chart to use. Note that this is different from the
  # app/container version.
  external_dns_chart_version = "6.12.2"

  # The registry to use for the external-dns image.
  external_dns_image_registry = null

  # Configure affinity rules for the external-dns Pod to control which nodes to
  # schedule on. Each item in the list should be a map with the keys `key`,
  # `values`, and `operator`, corresponding to the 3 properties of
  # matchExpressions. Note that all expressions must be satisfied to schedule on
  # the node.
  external_dns_pod_node_affinity = []

  # Configure tolerations rules to allow the external-dns Pod to schedule on
  # nodes that have been tainted. Each item in the list specifies a toleration
  # rule.
  external_dns_pod_tolerations = []

  # Duration string (e.g. 1m) indicating the polling interval for syncing the
  # domains by external-dns. When null, use the default defined in the chart
  # (1m).
  external_dns_poll_interval = null

  # Only create records in hosted zones that match the provided domain names.
  # Empty list (default) means match all zones. Zones must satisfy all three
  # constraints (var.external_dns_route53_hosted_zone_tag_filters,
  # var.external_dns_route53_hosted_zone_id_filters, and
  # var.external_dns_route53_hosted_zone_domain_filters).
  external_dns_route53_hosted_zone_domain_filters = []

  # Only create records in hosted zones that match the provided IDs. Empty list
  # (default) means match all zones. Zones must satisfy all three constraints
  # (var.external_dns_route53_hosted_zone_tag_filters,
  # var.external_dns_route53_hosted_zone_id_filters, and
  # var.external_dns_route53_hosted_zone_domain_filters).
  external_dns_route53_hosted_zone_id_filters = []

  # Only create records in hosted zones that match the provided tags. Each item
  # in the list should specify tag key and tag value as a map. Empty list
  # (default) means match all zones. Zones must satisfy all three constraints
  # (var.external_dns_route53_hosted_zone_tag_filters,
  # var.external_dns_route53_hosted_zone_id_filters, and
  # var.external_dns_route53_hosted_zone_domain_filters).
  external_dns_route53_hosted_zone_tag_filters = []

  # Duration string (e.g. 1m) indicating the amount of time the Hosted Zones are
  # cached in external-dns. When null, use the default defined in the chart (0 -
  # no caching).
  external_dns_route53_zones_cache_duration = null

  # K8s resources type to be observed for new DNS entries by ExternalDNS.
  external_dns_sources = ["ingress","service"]

  # When enabled, triggers external-dns run loop on create/update/delete events
  # (optional, in addition of regular interval)
  external_dns_trigger_loop_on_event = false

  # List of ARNs of Fargate execution IAM Roles that should get permissions to
  # ship logs using fluent-bit. This must be provided if
  # enable_fargate_fluent_bit is true.
  fargate_fluent_bit_execution_iam_role_arns = []

  # Additional filters that fluent-bit should apply to log output. This string
  # should be formatted according to the Fluent-bit docs
  # (https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/configuration-file#config_filter).
  fargate_fluent_bit_extra_filters = ""

  # Additional parsers that fluent-bit should export logs to. This string should
  # be formatted according to the Fluent-bit docs
  # (https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/configuration-file#config_output).
  fargate_fluent_bit_extra_parsers = ""

  # Whether or not Kubernetes metadata is added to the log files
  fargate_fluent_bit_include_kubernetes_metadata = true

  # Prefix string to use for the CloudWatch Log Stream that gets created for
  # each Fargate pod.
  fargate_fluent_bit_log_stream_prefix = "fargate"

  # A list of availability zones in the region that we CANNOT use to deploy the
  # EKS Fargate workers. You can use this to avoid availability zones that may
  # not be able to provision the resources (e.g ran out of capacity). If empty,
  # will allow all availability zones.
  fargate_worker_disallowed_availability_zones = ["us-east-1d","us-east-1e","ca-central-1d"]

  # Can be used to add additional filter configuration blocks. This string
  # should be formatted according to Fluent Bit docs, as it will be injected
  # directly into the fluent-bit.conf file.
  fluent_bit_additional_filters = ""

  # Can be used to add more inputs. This string should be formatted according to
  # Fluent Bit docs
  # (https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file#config_input).
  fluent_bit_additional_inputs = ""

  # Configurations for adjusting the default input settings. Set to null if you
  # do not wish to use the default filter.
  fluent_bit_default_input_configuration = {"db":"/var/log/flb_kube.db","dockerMode":"On","enabled":true,"memBufLimit":"5MB","parser":"docker","path":"/var/log/containers/*.log","refreshInterval":"10","skipLongLines":"On","tag":"kube.*"}

  # Can be used to provide additional kubernetes plugin configuration parameters
  # for the default kubernetes filter that is pre-configured in the
  # aws-for-fluent-bit Helm chart. This string should be formatted according to
  # Fluent Bit docs, as it will append to the default kubernetes filter
  # configuration.
  fluent_bit_extra_filters = ""

  # Can be used to append to existing input. This string should be formatted
  # according to Fluent Bit docs, as it will be injected directly into the
  # fluent-bit.conf file.
  fluent_bit_extra_inputs = ""

  # Additional output streams that fluent-bit should export logs to. This string
  # should be formatted according to the Fluent-bit docs
  # (https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/configuration-file#config_output).
  fluent_bit_extra_outputs = ""

  # Can be used to add additional log parsers. This string should be formatted
  # according to Fluent Bit docs, as it will be injected directly into the
  # fluent-bit.conf file.
  fluent_bit_extra_parsers = ""

  # The Container repository to use for looking up the aws-for-fluent-bit
  # Container image when deploying the pods. When null, uses the default
  # repository set in the chart. Only applies to non-fargate workers.
  fluent_bit_image_repository = null

  # If set to true, that means that the CloudWatch Log Group fluent-bit should
  # use for streaming logs already exists and does not need to be created.
  fluent_bit_log_group_already_exists = false

  # The ARN of the KMS key to use to encrypt the logs in the CloudWatch Log
  # Group used for storing container logs streamed with FluentBit. Set to null
  # to disable encryption.
  fluent_bit_log_group_kms_key_id = null

  # Name of the CloudWatch Log Group fluent-bit should use to stream logs to.
  # When null (default), uses the eks_cluster_name as the Log Group name.
  fluent_bit_log_group_name = null

  # number of days to retain log events. Possible values are: 1, 3, 5, 7, 14,
  # 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 3653, and 0. Select 0
  # to never expire.
  fluent_bit_log_group_retention = 0

  # ARN of the lambda function to trigger when events arrive at the fluent bit
  # log group.
  fluent_bit_log_group_subscription_arn = null

  # Filter pattern for the CloudWatch subscription. Only used if
  # var.fluent_bit_log_group_subscription_arn is set.
  fluent_bit_log_group_subscription_filter = ""

  # Prefix string to use for the CloudWatch Log Stream that gets created for
  # each pod. When null (default), the prefix is set to 'fluentbit'.
  fluent_bit_log_stream_prefix = null

  # Configure affinity rules for the fluent-bit Pods to control which nodes to
  # schedule on. Each item in the list should be a map with the keys `key`,
  # `values`, and `operator`, corresponding to the 3 properties of
  # matchExpressions. Note that all expressions must be satisfied to schedule on
  # the node.
  fluent_bit_pod_node_affinity = []

  # Configure tolerations rules to allow the fluent-bit Pods to schedule on
  # nodes that have been tainted. Each item in the list specifies a toleration
  # rule.
  fluent_bit_pod_tolerations = []

  # Merge and mask sensitive values like apikeys or passwords that are part of
  # the helm charts `values.yaml`. These sensitive values will show up in the
  # final metadata as clear text unless passed in as K:V pairs that are injected
  # into the `values.yaml`. Key should be the paramater path and value should be
  # the value.
  fluent_bit_sensitive_values = {}

  # Optionally use a cri parser instead of the default Docker parser. This
  # should be used for EKS v1.24 and later.
  fluent_bit_use_cri_parser_conf = true

  # Which version of aws-for-fluent-bit to install. When null, uses the default
  # version set in the chart. Only applies to non-fargate workers.
  fluent_bit_version = null

  # A map of PriorityClass configurations, with the key as the PriorityClass
  # name.
  # https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass
  kubernetes_priority_classes = {}

  # Policy for how DNS records are sychronized between sources and providers
  # (options: sync, upsert-only).
  route53_record_update_policy = "sync"

  # When true, the ALB ingress controller pods will be scheduled on Fargate.
  schedule_alb_ingress_controller_on_fargate = false

  # When true, the cluster autoscaler pods will be scheduled on Fargate. It is
  # recommended to run the cluster autoscaler on Fargate to avoid the autoscaler
  # scaling down a node where it is running (and thus shutting itself down
  # during a scale down event). However, since Fargate is only supported on a
  # handful of regions, we don't default to true here.
  schedule_cluster_autoscaler_on_fargate = false

  # When true, the external-dns pods will be scheduled on Fargate.
  schedule_external_dns_on_fargate = false

  # Configure Kubernetes Services to lookup external DNS records. This can be
  # useful to bind friendly internal service names to domains (e.g. the RDS
  # database endpoint).
  service_dns_mappings = {}

  # If this variable is set to true, then use an exec-based plugin to
  # authenticate and fetch tokens for EKS. This is useful because EKS clusters
  # use short-lived authentication tokens that can expire in the middle of an
  # 'apply' or 'destroy', and since the native Kubernetes provider in Terraform
  # doesn't have a way to fetch up-to-date tokens, we recommend using an
  # exec-based provider as a workaround. Use the use_kubergrunt_to_fetch_token
  # input variable to control whether kubergrunt or aws is used to fetch tokens.
  use_exec_plugin_for_auth = true

  # EKS clusters use short-lived authentication tokens that can expire in the
  # middle of an 'apply' or 'destroy'. To avoid this issue, we use an exec-based
  # plugin to fetch an up-to-date token. If this variable is set to true, we'll
  # use kubergrunt to fetch the token (in which case, kubergrunt must be
  # installed and on PATH); if this variable is set to false, we'll use the aws
  # CLI to fetch the token (in which case, aws must be installed and on PATH).
  # Note this functionality is only enabled if use_exec_plugin_for_auth is set
  # to true.
  use_kubergrunt_to_fetch_token = true

  # When true, all IAM policies will be managed as dedicated policies rather
  # than inline policies attached to the IAM roles. Dedicated managed policies
  # are friendlier to automated policy checkers, which may scan a single
  # resource for findings. As such, it is important to avoid inline policies
  # when targeting compliance with various security standards.
  use_managed_iam_policies = true

}


Reference​

Required​

aws_regionstringrequired

The AWS region in which all resources will be created

eks_cluster_namestringrequired

The name of the EKS cluster where the core services will be deployed into.

Configuration for using the IAM role with Service Accounts feature to provide permissions to the applications. This expects a map with two properties: openid_connect_provider_arn and openid_connect_provider_url. The openid_connect_provider_arn is the ARN of the OpenID Connect Provider for EKS to retrieve IAM credentials, while openid_connect_provider_url is the URL. Set to null if you do not wish to use IAM role with Service Accounts.

object({
    openid_connect_provider_arn = string
    openid_connect_provider_url = string
  })

ARN of IAM Role to use as the Pod execution role for Fargate. Required if any of the services are being scheduled on Fargate. Set to null if none of the Pods are being scheduled on Fargate.

vpc_idstringrequired

The ID of the VPC where the EKS cluster is deployed.

worker_vpc_subnet_idslist(string)required

The subnet IDs to use for EKS worker nodes. Used when provisioning Pods on to Fargate. Required if any of the services are being scheduled on Fargate. Set to empty list if none of the Pods are being scheduled on Fargate.

Optional​

ARN of IAM Role to assume to create and control ALB's. This is useful if your VPC is shared from another account and needs to be created somewhere else.

null

The version of the aws-load-balancer-controller helmchart to use.

"1.4.1"

Tags to apply to all AWS resources managed by this controller

{}

The repository of the aws-load-balancer-controller docker image that should be deployed.

"602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-load-balancer-controller"

The tag of the aws-load-balancer-controller docker image that should be deployed.

"v2.4.1"

Configure affinity rules for the ALB Ingress Controller Pod to control which nodes to schedule on. Each item in the list should be a map with the keys key, values, and operator, corresponding to the 3 properties of matchExpressions. Note that all expressions must be satisfied to schedule on the node.

list(object({
    key      = string
    values   = list(string)
    operator = string
  }))
[]
Details

   Each item in the list represents a matchExpression for requiredDuringSchedulingIgnoredDuringExecution.
   https://kubernetes.io/docs/concepts/configuration/assign-pod-node/affinity-and-anti-affinity for the various
   configuration option.

   Example:

   [
     {
       "key" = "node-label-key"
       "values" = ["node-label-value", "another-node-label-value"]
       "operator" = "In"
     }
   ]

   Translates to:

   nodeAffinity:
     requiredDuringSchedulingIgnoredDuringExecution:
       nodeSelectorTerms:
       - matchExpressions:
         - key: node-label-key
           operator: In
           values:
           - node-label-value
           - another-node-label-value

Configure tolerations rules to allow the ALB Ingress Controller Pod to schedule on nodes that have been tainted. Each item in the list specifies a toleration rule.

list(map(any))
[]
Details

   Each item in the list represents a particular toleration. See
   https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/ for the various rules you can specify.

   Example:

   [
     {
       key = "node.kubernetes.io/unreachable"
       operator = "Exists"
       effect = "NoExecute"
       tolerationSeconds = 6000
     }
   ]

Minimum time to wait after a scale up event before any node is considered for scale down.

"10m"

ARN of permissions boundary to apply to the autoscaler IAM role - the IAM role created for the Autoscaler

null

Number for the log level verbosity. Lower numbers are less verbose, higher numbers are more verbose. (Default: 4)

4

Minimum time to wait since the node became unused before the node is considered for scale down by the autoscaler.

"10m"

If true cluster autoscaler will never delete nodes with pods with local storage, e.g. EmptyDir or HostPath

true

The Container repository to use for looking up the cloudwatch-agent Container image when deploying the pods. When null, uses the default repository set in the chart. Only applies to non-fargate workers.

null

Configure affinity rules for the AWS CloudWatch Agent Pod to control which nodes to schedule on. Each item in the list should be a map with the keys key, values, and operator, corresponding to the 3 properties of matchExpressions. Note that all expressions must be satisfied to schedule on the node.

list(object({
    key      = string
    values   = list(string)
    operator = string
  }))
[]
Details

   Each item in the list represents a matchExpression for requiredDuringSchedulingIgnoredDuringExecution.
   https://kubernetes.io/docs/concepts/configuration/assign-pod-node/affinity-and-anti-affinity for the various
   configuration option.

   Example:

   [
     {
       "key" = "node-label-key"
       "values" = ["node-label-value", "another-node-label-value"]
       "operator" = "In"
     }
   ]

   Translates to:

   nodeAffinity:
     requiredDuringSchedulingIgnoredDuringExecution:
       nodeSelectorTerms:
       - matchExpressions:
         - key: node-label-key
           operator: In
           values:
           - node-label-value
           - another-node-label-value

Pod resource requests and limits to use. Refer to https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ for more information.

Any types represent complex values of variable type. For details, please consult `variables.tf` in the source repo.
null
Details

   We use any type here to avoid maintaining the kubernetes defined type spec for the resources here. That way, we can
   support wide range of kubernetes versions.

Details

   Example value:
   {
     requests = {
       memory = "1024Mi"
       cpu    = "250m"
     }
     limits = {
       memory = "1024Mi"
       cpu    = "250m"
     }
   }

Configure tolerations rules to allow the AWS CloudWatch Agent Pods to schedule on nodes that have been tainted. Each item in the list specifies a toleration rule.

list(map(any))
[]
Details

   Each item in the list represents a particular toleration. See
   https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/ for the various rules you can specify.

   Example:

   [
     {
       key = "node.kubernetes.io/unreachable"
       operator = "Exists"
       effect = "NoExecute"
       tolerationSeconds = 6000
     }
   ]

Which version of amazon/cloudwatch-agent to install. When null, uses the default version set in the chart. Only applies to non-fargate workers.

null

Restrict the cluster autoscaler to a list of absolute ASG ARNs upon initial apply to ensure no new ASGs can be managed by the autoscaler without explicitly running another apply. Setting this to false will ensure that the cluster autoscaler is automatically given access to manage any new ASGs with the k8s.io/cluster-autoscaler/CLUSTER_NAME tag applied.

true

The version of the cluster-autoscaler helm chart to deploy. Note that this is different from the app/container version, which is sepecified with cluster_autoscaler_version.

"9.21.0"

Annotations to apply to the cluster autoscaler pod(s), as key value pairs.

{}

Labels to apply to the cluster autoscaler pod(s), as key value pairs.

{}
cluster_autoscaler_pod_node_affinitylist(object(…))optional

Configure affinity rules for the cluster-autoscaler Pod to control which nodes to schedule on. Each item in the list should be a map with the keys key, values, and operator, corresponding to the 3 properties of matchExpressions. Note that all expressions must be satisfied to schedule on the node.

list(object({
    key      = string
    values   = list(string)
    operator = string
  }))
[]
Details

   Each item in the list represents a matchExpression for requiredDuringSchedulingIgnoredDuringExecution.
   https://kubernetes.io/docs/concepts/configuration/assign-pod-node/affinity-and-anti-affinity for the various
   configuration option.

   Example:

   [
     {
       "key" = "node-label-key"
       "values" = ["node-label-value", "another-node-label-value"]
       "operator" = "In"
     }
   ]

   Translates to:

   nodeAffinity:
     requiredDuringSchedulingIgnoredDuringExecution:
       nodeSelectorTerms:
       - matchExpressions:
         - key: node-label-key
           operator: In
           values:
           - node-label-value
           - another-node-label-value

Pod resource requests and limits to use. Refer to https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ for more information. This is most useful for configuring CPU+Memory availability for Fargate, which defaults to 0.25 vCPU and 256MB RAM.

Any types represent complex values of variable type. For details, please consult `variables.tf` in the source repo.
{
  limits = {
    cpu = "250m",
    memory = "1024Mi"
  },
  requests = {
    cpu = "250m",
    memory = "1024Mi"
  }
}
Details

   We use any type here to avoid maintaining the kubernetes defined type spec for the resources here. That way, we can
   support wide range of kubernetes versions.

Details

   cluster-autoscaler is known to fail on Fargate when the default resource limits are used, so we set a saner default
   here.

Configure tolerations rules to allow the cluster-autoscaler Pod to schedule on nodes that have been tainted. Each item in the list specifies a toleration rule.

list(map(any))
[]
Details

   Each item in the list represents a particular toleration. See
   https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/ for the various rules you can specify.

   Example:

   [
     {
       key = "node.kubernetes.io/unreachable"
       operator = "Exists"
       effect = "NoExecute"
       tolerationSeconds = 6000
     }
   ]

The name to use for the helm release for cluster-autoscaler. This is useful to force a redeployment of the cluster-autoscaler component.

"cluster-autoscaler"

Which docker repository to use to install the cluster autoscaler. Check the following link for valid repositories to use https://github.com/kubernetes/autoscaler/releases

"registry.k8s.io/autoscaling/cluster-autoscaler"

ARN of IAM Role to use for the Cluster Autoscaler. Only used when create_cluster_autoscaler_role is false.

null

Specifies an 'expander' for the cluster autoscaler. This helps determine which ASG to scale when additional resource capacity is needed.

"least-waste"

The name of the service account to create for the cluster autoscaler.

"cluster-autoscaler-aws-cluster-autoscaler"

Which version of the cluster autoscaler to install. This should match the major/minor version (e.g., v1.20) of your Kubernetes Installation. See https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#releases for a list of versions.

"v1.30.0"

When set to true, create a new dedicated IAM Role for the cluster autoscaler. When set to true, iam_role_for_service_accounts_config is required.

true

Whether or not to enable the AWS LB Ingress controller.

true

Whether to enable the AWS CloudWatch Agent DaemonSet for collecting container and node metrics from worker nodes (self-managed ASG or managed node groups).

true

Whether or not to enable cluster-autoscaler for Autoscaling EKS worker nodes.

true

Whether or not to enable external-dns for DNS entry syncing with Route 53 for Services and Ingresses.

true

Whether or not to enable fluent-bit on EKS Fargate workers for log aggregation.

true
enable_fluent_bitbooloptional

Whether or not to enable fluent-bit for log aggregation.

true

Duration string (e.g. 1m) indicating the interval between making changes to Route 53 by external-dns. When null, use the default defined in the chart (1s).

null

The maximum number of changes that should be applied in a batch by external-dns. When null, use the default defined in the chart (1000).

null

Name of the Helm chart for external-dns. This should usually be 'external-dns' but may differ in the case of overriding the repository URL.

"external-dns"

Helm chart repository URL to obtain the external-dns chart from. Useful when using Bitnami charts that are older than 6 months due to Bitnami's lifecycle policy which removes older chart from the main index.

"https://charts.bitnami.com/bitnami"

The version of the helm chart to use. Note that this is different from the app/container version.

"6.12.2"

The registry to use for the external-dns image.

null
external_dns_pod_node_affinitylist(object(…))optional

Configure affinity rules for the external-dns Pod to control which nodes to schedule on. Each item in the list should be a map with the keys key, values, and operator, corresponding to the 3 properties of matchExpressions. Note that all expressions must be satisfied to schedule on the node.

list(object({
    key      = string
    values   = list(string)
    operator = string
  }))
[]
Details

   Each item in the list represents a matchExpression for requiredDuringSchedulingIgnoredDuringExecution.
   https://kubernetes.io/docs/concepts/configuration/assign-pod-node/affinity-and-anti-affinity for the various
   configuration option.

   Example:

   [
     {
       "key" = "node-label-key"
       "values" = ["node-label-value", "another-node-label-value"]
       "operator" = "In"
     }
   ]

   Translates to:

   nodeAffinity:
     requiredDuringSchedulingIgnoredDuringExecution:
       nodeSelectorTerms:
       - matchExpressions:
         - key: node-label-key
           operator: In
           values:
           - node-label-value
           - another-node-label-value

external_dns_pod_tolerationslist(map(…))optional

Configure tolerations rules to allow the external-dns Pod to schedule on nodes that have been tainted. Each item in the list specifies a toleration rule.

list(map(any))
[]
Details

   Each item in the list represents a particular toleration. See
   https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/ for the various rules you can specify.

   Example:

   [
     {
       key = "node.kubernetes.io/unreachable"
       operator = "Exists"
       effect = "NoExecute"
       tolerationSeconds = 6000
     }
   ]

Duration string (e.g. 1m) indicating the polling interval for syncing the domains by external-dns. When null, use the default defined in the chart (1m).

null

Only create records in hosted zones that match the provided domain names. Empty list (default) means match all zones. Zones must satisfy all three constraints (external_dns_route53_hosted_zone_tag_filters, external_dns_route53_hosted_zone_id_filters, and external_dns_route53_hosted_zone_domain_filters).

[]

Only create records in hosted zones that match the provided IDs. Empty list (default) means match all zones. Zones must satisfy all three constraints (external_dns_route53_hosted_zone_tag_filters, external_dns_route53_hosted_zone_id_filters, and external_dns_route53_hosted_zone_domain_filters).

[]

Only create records in hosted zones that match the provided tags. Each item in the list should specify tag key and tag value as a map. Empty list (default) means match all zones. Zones must satisfy all three constraints (external_dns_route53_hosted_zone_tag_filters, external_dns_route53_hosted_zone_id_filters, and external_dns_route53_hosted_zone_domain_filters).

list(object({
    key   = string
    value = string
  }))
[]
Example
   [
     {
       key = "Name"
       value = "current"
     }
   ]

Duration string (e.g. 1m) indicating the amount of time the Hosted Zones are cached in external-dns. When null, use the default defined in the chart (0 - no caching).

null
external_dns_sourceslist(string)optional

K8s resources type to be observed for new DNS entries by ExternalDNS.

[
  "ingress",
  "service"
]
Details

   NOTE ON ISTIO: By default, external-dns will listen for "ingress" and "service" events. To use it with Istio, make
   sure to include the "istio-gateway" events here. See the docs for more details:
   https://github.com/kubernetes-incubator/external-dns/blob/master/docs/tutorials/istio.md

When enabled, triggers external-dns run loop on create/update/delete events (optional, in addition of regular interval)

false

List of ARNs of Fargate execution IAM Roles that should get permissions to ship logs using fluent-bit. This must be provided if enable_fargate_fluent_bit is true.

[]

Additional filters that fluent-bit should apply to log output. This string should be formatted according to the Fluent-bit docs (https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/configuration-file#config_filter).

""

Additional parsers that fluent-bit should export logs to. This string should be formatted according to the Fluent-bit docs (https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/configuration-file#config_output).

""

Whether or not Kubernetes metadata is added to the log files

true

Prefix string to use for the CloudWatch Log Stream that gets created for each Fargate pod.

"fargate"

A list of availability zones in the region that we CANNOT use to deploy the EKS Fargate workers. You can use this to avoid availability zones that may not be able to provision the resources (e.g ran out of capacity). If empty, will allow all availability zones.

[
  "us-east-1d",
  "us-east-1e",
  "ca-central-1d"
]

Can be used to add additional filter configuration blocks. This string should be formatted according to Fluent Bit docs, as it will be injected directly into the fluent-bit.conf file.

""

Can be used to add more inputs. This string should be formatted according to Fluent Bit docs (https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file#config_input).

""

Configurations for adjusting the default input settings. Set to null if you do not wish to use the default filter.

object({
    # This assumes the filter is being created (ie, not null), and provides a
    # means to disable it.
    enabled = bool

    # This option allows a tag name associated to all records coming from this plugin.
    # logs, defaults to "kube.*"
    tag = string

    # This option allows to change the default path where the plugin will look for
    # Docker containers logs, defaults to "/var/log/containers/*.log"
    path = string

    # This option allows to change the default database file where the plugin will
    # store the state of the logs, defaults to "/var/log/flb_kube.db"
    db = string

    # This option allows to change the default parser used to read the Docker
    # containers logs, defaults to "docker"
    parser = string

    # This option enabled or disables the Docker Mode, defaults to "On"
    dockerMode = string

    # This option allows to change the default memory limit used, defaults to "5MB"
    memBufLimit = string

    # This option allows to change the default number of lines to skip if a line
    # is bigger than the buffer size, defaults to "On"
    skipLongLines = string

    # This option allows to change the default refresh interval to check the
    # status of the monitored files, defaults to "10"
    refreshInterval = string
  })
{
  db = "/var/log/flb_kube.db",
  dockerMode = "On",
  enabled = true,
  memBufLimit = "5MB",
  parser = "docker",
  path = "/var/log/containers/*.log",
  refreshInterval = "10",
  skipLongLines = "On",
  tag = "kube.*"
}
Details

     This option allows a tag name associated to all records coming from this plugin.
     logs, defaults to "kube.*"

Details

     This option allows to change the default path where the plugin will look for
     Docker containers logs, defaults to "/var/log/containers/*.log"

Details

     This option allows to change the default database file where the plugin will
     store the state of the logs, defaults to "/var/log/flb_kube.db"

Details

     This option allows to change the default parser used to read the Docker
     containers logs, defaults to "docker"

Details

     This option enabled or disables the Docker Mode, defaults to "On"

Details

     This option allows to change the default memory limit used, defaults to "5MB"

Details

     This option allows to change the default number of lines to skip if a line
     is bigger than the buffer size, defaults to "On"

Details

     This option allows to change the default refresh interval to check the
     status of the monitored files, defaults to "10"

Details

     Default settings for input

Can be used to provide additional kubernetes plugin configuration parameters for the default kubernetes filter that is pre-configured in the aws-for-fluent-bit Helm chart. This string should be formatted according to Fluent Bit docs, as it will append to the default kubernetes filter configuration.

""

Can be used to append to existing input. This string should be formatted according to Fluent Bit docs, as it will be injected directly into the fluent-bit.conf file.

""

Additional output streams that fluent-bit should export logs to. This string should be formatted according to the Fluent-bit docs (https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/configuration-file#config_output).

""

Can be used to add additional log parsers. This string should be formatted according to Fluent Bit docs, as it will be injected directly into the fluent-bit.conf file.

""

The Container repository to use for looking up the aws-for-fluent-bit Container image when deploying the pods. When null, uses the default repository set in the chart. Only applies to non-fargate workers.

null

If set to true, that means that the CloudWatch Log Group fluent-bit should use for streaming logs already exists and does not need to be created.

false

The ARN of the KMS key to use to encrypt the logs in the CloudWatch Log Group used for storing container logs streamed with FluentBit. Set to null to disable encryption.

null

Name of the CloudWatch Log Group fluent-bit should use to stream logs to. When null (default), uses the eks_cluster_name as the Log Group name.

null

number of days to retain log events. Possible values are: 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 3653, and 0. Select 0 to never expire.

0

ARN of the lambda function to trigger when events arrive at the fluent bit log group.

null

Filter pattern for the CloudWatch subscription. Only used if fluent_bit_log_group_subscription_arn is set.

""

Prefix string to use for the CloudWatch Log Stream that gets created for each pod. When null (default), the prefix is set to 'fluentbit'.

null
fluent_bit_pod_node_affinitylist(object(…))optional

Configure affinity rules for the fluent-bit Pods to control which nodes to schedule on. Each item in the list should be a map with the keys key, values, and operator, corresponding to the 3 properties of matchExpressions. Note that all expressions must be satisfied to schedule on the node.

list(object({
    key      = string
    values   = list(string)
    operator = string
  }))
[]
Details

   Each item in the list represents a matchExpression for requiredDuringSchedulingIgnoredDuringExecution.
   https://kubernetes.io/docs/concepts/configuration/assign-pod-node/affinity-and-anti-affinity for the various
   configuration option.

   Example:

   [
     {
       "key" = "node-label-key"
       "values" = ["node-label-value", "another-node-label-value"]
       "operator" = "In"
     }
   ]

   Translates to:

   nodeAffinity:
     requiredDuringSchedulingIgnoredDuringExecution:
       nodeSelectorTerms:
       - matchExpressions:
         - key: node-label-key
           operator: In
           values:
           - node-label-value
           - another-node-label-value

fluent_bit_pod_tolerationslist(map(…))optional

Configure tolerations rules to allow the fluent-bit Pods to schedule on nodes that have been tainted. Each item in the list specifies a toleration rule.

list(map(any))
[]
Details

   Each item in the list represents a particular toleration. See
   https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/ for the various rules you can specify.

   Example:

   [
     {
       key = "node.kubernetes.io/unreachable"
       operator = "Exists"
       effect = "NoExecute"
       tolerationSeconds = 6000
     }
   ]

fluent_bit_sensitive_valuesmap(string)optional

Merge and mask sensitive values like apikeys or passwords that are part of the helm charts values.yaml. These sensitive values will show up in the final metadata as clear text unless passed in as K:V pairs that are injected into the values.yaml. Key should be the paramater path and value should be the value.

{}
Details

   EXAMPLE
   {
    "additionalOutputs" = var.extraOutputs
   }

Optionally use a cri parser instead of the default Docker parser. This should be used for EKS v1.24 and later.

true
fluent_bit_versionstringoptional

Which version of aws-for-fluent-bit to install. When null, uses the default version set in the chart. Only applies to non-fargate workers.

null
kubernetes_priority_classesmap(object(…))optional

A map of PriorityClass configurations, with the key as the PriorityClass name. https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass

map(object({
    description    = string
    global_default = bool
    value          = number
  }))
{}

Policy for how DNS records are sychronized between sources and providers (options: sync, upsert-only).

"sync"

When true, the ALB ingress controller pods will be scheduled on Fargate.

false

When true, the cluster autoscaler pods will be scheduled on Fargate. It is recommended to run the cluster autoscaler on Fargate to avoid the autoscaler scaling down a node where it is running (and thus shutting itself down during a scale down event). However, since Fargate is only supported on a handful of regions, we don't default to true here.

false

When true, the external-dns pods will be scheduled on Fargate.

false
service_dns_mappingsmap(object(…))optional

Configure Kubernetes Services to lookup external DNS records. This can be useful to bind friendly internal service names to domains (e.g. the RDS database endpoint).

map(object({
    # DNS record to route requests to the Kubernetes Service to.
    target_dns = string

    # Port to route requests
    target_port = number

    # Namespace to create the underlying Kubernetes Service in.
    namespace = string
  }))
{}
Details

     Port to route requests

Details

     Namespace to create the underlying Kubernetes Service in.

If this variable is set to true, then use an exec-based plugin to authenticate and fetch tokens for EKS. This is useful because EKS clusters use short-lived authentication tokens that can expire in the middle of an 'apply' or 'destroy', and since the native Kubernetes provider in Terraform doesn't have a way to fetch up-to-date tokens, we recommend using an exec-based provider as a workaround. Use the use_kubergrunt_to_fetch_token input variable to control whether kubergrunt or aws is used to fetch tokens.

true

EKS clusters use short-lived authentication tokens that can expire in the middle of an 'apply' or 'destroy'. To avoid this issue, we use an exec-based plugin to fetch an up-to-date token. If this variable is set to true, we'll use kubergrunt to fetch the token (in which case, kubergrunt must be installed and on PATH); if this variable is set to false, we'll use the aws CLI to fetch the token (in which case, aws must be installed and on PATH). Note this functionality is only enabled if use_exec_plugin_for_auth is set to true.

true

When true, all IAM policies will be managed as dedicated policies rather than inline policies attached to the IAM roles. Dedicated managed policies are friendlier to automated policy checkers, which may scan a single resource for findings. As such, it is important to avoid inline policies when targeting compliance with various security standards.

true