Skip to main content
AWS Messaging 0.12.5Last updated in version 0.12.4

Amazon Managed Streaming for Apache Kafka (Amazon MSK) Module

View Source Release Notes

This Terraform module configures and launches an Amazon MSK cluster.

Cluster Configuration​

Amazon Managed Streaming for Apache Kafka provides a default configuration for brokers, topics, and Apache ZooKeeper nodes. You can also create custom configurations and use them to create new MSK clusters or to update existing clusters. An MSK configuration consists of a set of properties and their corresponding values:

  • default configuration: when you create an MSK cluster and don't specify a custom MSK configuration, Amazon MSK creates and uses a default configuration with the values shown in this page. For properties that aren't in this table, Amazon MSK uses the defaults associated with your version of Apache Kafka.
  • custom configuration: you can create custom configuration using var.server_properties. Refer to the Custom MSK configurations for more information.

Configuration Update​

You can use the var.server_properties to update the cluster configuration. When updating the configuration of an Amazon Managed Streaming for Apache Kafka (MSK) cluster, the update happens in place. MSK applies the configuration update to the running brokers in a rolling update fashion, meaning that each broker is updated one at a time while the others remain operational.

This approach ensures that the Kafka cluster remains available during the configuration update process. The update process does not require destroying and recreating the cluster, which would cause significant downtime and disruption to the running applications.

Capacity Planning​

Capacity planning involves ensuring that your cluster has enough resources to meet the current and future demands of your use case. There are many aspects and requirements to consider, including data volume, performance, availability, workload, future growth, cost, etc. Refer to the following references for best practices, recommendations, and further details on planning capacity:

Based on the result of the capacity planning, you will be tweaking the following configuration to provision your cluster:

  • number of partitions per broker: the number of partitions per broker in Apache Kafka is determined by the total number of partitions across all topics and the number of brokers in the cluster. By default, Kafka evenly distributes the partitions across all available brokers, so the number of partitions per broker will depend on the number of brokers and the total number of partitions.
  • number of brokers per cluster: can be configured to any value between 1 and 15.
  • replication factor: controls the # of other brokers to replicate the data for fault tolerance. It is recommended to have a topic replication factor > 1.
  • broker_type: depending on the broker type, recommended number of partitions per broker and maximum storage throughput changes.

Broker Storage Scaling​

Amount of required EBS storage depends on multiple factors, for example number of topics, amount and size of your data, data retention, and replication factor. As such it is not possible to give an exact recommendation, instead the storage requirements should be calculated based on your use case. It is important to monitor disk usage and increase disk size when needed.

You can specify the initial EBS volume size with the input variable var.initial_ebs_volume_size and use the following variables to configure auto-scaling broker storage:

  • var.broker_storage_autoscaling_max_capacity: the max capacity of broker node EBS storage
  • var.broker_storage_autoscaling_target_percentage: triggering point of the auto-scaling
  • var.disable_broker_storage_scale_in: flag to disable scaling-in broker storage after auto-scale.

Note: if you do not want to use the auto-scaling feature for whatever reason, you can set the var.broker_storage_autoscaling_max_capacity the same as the var.initial_ebs_volume_size.

Monitoring​

Amazon Managed Streaming for Apache Kafka gathers Apache Kafka metrics and sends them to Amazon CloudWatch where you can view them. For more information about Apache Kafka metrics, including the ones that Amazon MSK surfaces, see Monitoring in the Apache Kafka documentation.

You can also monitor your MSK cluster with Prometheus, an open-source monitoring application. For information about Prometheus, see Overview in the Prometheus documentation. To learn how to monitor your cluster with Prometheus, see Open monitoring with Prometheus.

Monitoring With CloudWatch​

Amazon MSK integrates with Amazon CloudWatch so that you can collect, view, and analyze CloudWatch metrics for your Amazon MSK cluster. The metrics that you configure for your MSK cluster are automatically collected and pushed to CloudWatch. You can set the monitoring level for an MSK cluster to one of the following: DEFAULT, PER_BROKER, PER_TOPIC_PER_BROKER, or PER_TOPIC_PER_PARTITION. Only the DEFAULT-level metrics are free.

Note: currently, we do not support setting different monitoring level for CloudWatch Monitoring. You would have to do this via the Amazon console. You can read more about metrics and monitoring here: https://docs.aws.amazon.com/msk/latest/developerguide/metrics-details.html

Monitoring with Prometheus​

You can monitor your MSK cluster with Prometheus, an open-source monitoring system for time-series metric data. You can publish this data to Amazon Managed Service for Prometheus using Prometheus's remote write feature. Please use the following variables to setup monitoring;

  • var.open_monitoring_enable_jmx_exporter
  • var.open_monitoring_enable_node_exporter

Security​

Data Protection w/ Encryption​

Amazon MSK provides data encryption options that you can use to meet strict data management requirements. Refer to the Amazon MSK encryption page for more information.

Encryption at Rest​

Amazon MSK integrates with AWS Key Management Service (KMS) to offer transparent server-side encryption. Amazon MSK always encrypts your data at rest. When you create an MSK cluster, you can specify the AWS KMS customer master key (CMK) with var.encryption_at_rest_kms_key_arn that you want Amazon MSK to use to encrypt your data at rest. If no key is specified, an AWS managed KMS (aws/msk managed service) key will be used for encrypting the data at rest.

Encryption in Transit​

Amazon MSK uses TLS 1.2. By default, it encrypts data in transit between the brokers of your MSK cluster. You can override this default using var.encryption_in_transit_in_cluster input variable at the time you create the cluster. You can also control client-to-broker encryption using var.encryption_in_transit_client_broker input variable.

Authentication and Authorization​

The MSK module supports the following authentication and authorization methods:

You can read more about available authentication and authorization options from the Authentication and authorization for Apache Kafka APIs page

Logging​

Broker logs enable you to troubleshoot your Apache Kafka applications and to analyze their communications with your MSK cluster. You can deliver Apache Kafka broker logs to one or more of the following destination types with variables:

  • Amazon CloudWatch Logs: var.enable_cloudwatch_logs, var.cloudwatch_log_group
  • Amazon S3: var.enable_s3_logs, var.s3_logs_bucket, var.s3_logs_prefix
  • Amazon Kinesis Data Firehose: var.enable_firehose_logs, var.firehose_delivery_stream).

You can read more about MSK logging here: https://docs.aws.amazon.com/msk/latest/developerguide/msk-logging.html

Connecting to Kafka brokers​

Once you've used this module to deploy the Kafka brokers, you'll want to connect to them from Kafka clients (e.g., Kafka consumers and producers in your apps) to read and write data. To do this, you typically need to configure the bootstrap.servers property for your Kafka client with the IP addresses of a few of your Kafka brokers (you don't need all the IPs, as the rest will be discovered automatically via ZooKeeper):

--bootstrap.servers=10.0.0.4:9092,10.0.0.5:9092,10.0.0.6:9092

Depending on which client authentication method you configured, there are a number of output variables (bootstrap_brokers_*) that provide you with a list of bootstrap servers. You can also get the list of bootstrap servers using the AWS Cli:

$ aws kafka get-bootstrap-brokers --cluster-arn ClusterArn

{
    "BootstrapBrokerStringSaslIam": "b-1.myTestCluster.123z8u.c2.kafka.us-west-1.amazonaws.com:9098,b-2.myTestCluster.123z8u.c2.kafka.us-west-1.amazonaws.com:9098"
}

Other Resources for MSK​

MSK Serverless​

MSK Serverless is a cluster type for Amazon MSK that makes it possible for you to run Apache Kafka without having to manage and scale cluster capacity. Refer to the MSK Serverless page for more information. You can enable it by setting var.enable_serverless = true.

MSK Serverless provides limited configuration options. Only the following list of variables would work with serverless:

  • var.enable_client_sasl_iam: only allows SASL IAM based authentication.
  • var.allow_connections_from_cidr_blocks and var.allow_connections_from_security_groups: controls the traffic that is allowed to reach and leave the mks cluster.
  • var.custom_tags and var.custom_tags_security_group: specify custom tags on msk cluster & security group.

Other variables are only applicable to non-serverless MSK deployment.

Connecting to MSK Serverless​

It's not possible to output endpoint for serverless cluster type deployment - related issue. You would have to use the Amazon console UI or use the following command to retrieve the endpoint:

aws kafka  get-bootstrap-brokers --cluster-arn "<CLUSTER_ARN>"

Tiered Storage​

Tiered storage is a low-cost storage tier for Amazon MSK that scales to virtually unlimited storage, making it cost-effective to build streaming data applications. Refer to the Tiered storage for requirements, constraints, limitations, and more information.

You can enable the tiered storage by setting the following variables:

  • var.storage_info = "TIERED"
  • var.kafka_version = "2.8.2.tiered" (Note: this is the only supported kafka version for tiered storage)
  • var.instance_type: set to other than kafka.t3.small.

It's only supported for the provisioned cluster type (non-serverless mode).

Sample Usage​

main.tf

# ------------------------------------------------------------------------------------------------------
# DEPLOY GRUNTWORK'S MSK MODULE
# ------------------------------------------------------------------------------------------------------

module "msk" {

  source = "git::git@github.com:gruntwork-io/terraform-aws-messaging.git//modules/msk?ref=v0.12.5"

  # ----------------------------------------------------------------------------------------------------
  # REQUIRED VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # The name of the Kafka cluster (e.g. kafka-stage). This variable is used to
  # namespace all resources created by this module.
  cluster_name = <string>

  # The subnet IDs into which the broker instances should be deployed. You
  # should typically pass in one subnet ID per node in the cluster_size
  # variable. The number of broker nodes must be a multiple of subnets. We
  # strongly recommend that you run Kafka in private subnets.
  subnet_ids = <list(string)>

  # The ID of the VPC in which to deploy the cluster.
  vpc_id = <string>

  # ----------------------------------------------------------------------------------------------------
  # OPTIONAL VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # A list of Security Group IDs that should be added to the MSK cluster broker
  # instances.
  additional_security_group_ids = []

  # A list of CIDR-formatted IP address ranges that will be allowed to connect
  # to the Kafka brokers.
  allow_connections_from_cidr_blocks = []

  # A list of security group IDs that will be allowed to connect to the Kafka
  # brokers.
  allow_connections_from_security_groups = []

  # Max capacity of broker node EBS storage (in GiB)
  broker_storage_autoscaling_max_capacity = 100

  # Broker storage utilization percentage at which scaling is triggered.
  broker_storage_autoscaling_target_percentage = 70

  # List of ARNs for SCRAM secrets stored in the Secrets Manager service.
  client_sasl_scram_secret_arns = []

  # Optional list of ACM Certificate Authority Amazon Resource Names (ARNs).
  client_tls_certificate_authority_arns = []

  # Name of the Cloudwatch Log Group to deliver logs to.
  cloudwatch_log_group = null

  # The number of brokers to have in the cluster. This field is required for
  # non-serverless cluster type.
  cluster_size = null

  # Custom tags to apply to the Kafka broker nodes and all related resources.
  custom_tags = {}

  # A map of custom tags to apply to the Security Group for this MSK Cluster.
  # The key is the tag name and the value is the tag value.
  custom_tags_security_group = {}

  # Flag indicating whether broker storage should never be scaled in.
  disable_broker_storage_scale_in = false

  # Whether SASL IAM client authentication is enabled.
  enable_client_sasl_iam = false

  # Whether SASL SCRAM client authentication is enabled.
  enable_client_sasl_scram = false

  # Whether TLS client authentication is enabled.
  enable_client_tls = false

  # Indicates whether you want to enable or disable streaming broker logs to
  # Cloudwatch Logs.
  enable_cloudwatch_logs = false

  # Indicates whether you want to enable or disable streaming broker logs to
  # Kinesis Data Firehose.
  enable_firehose_logs = false

  # Indicates whether you want to enable or disable streaming broker logs to S3.
  enable_s3_logs = false

  # Indicates whether you want to enable msk serverless or not
  enable_serverless = false

  # You may specify a KMS key short ID or ARN (it will always output an ARN) to
  # use for encrypting your data at rest. If no key is specified, an AWS managed
  # KMS ('aws/msk' managed service) key will be used for encrypting the data at
  # rest.
  encryption_at_rest_kms_key_arn = null

  # Encryption setting for data in transit between clients and brokers. Valid
  # values: TLS, TLS_PLAINTEXT, and PLAINTEXT. Default value is `TLS`.
  encryption_in_transit_client_broker = "TLS"

  # Whether data communication among broker nodes is encrypted. Default value:
  # true.
  encryption_in_transit_in_cluster = true

  # Specify the desired enhanced MSK CloudWatch monitoring level. See
  # https://docs.aws.amazon.com/msk/latest/developerguide/metrics-details.html
  # for valid values.
  enhanced_monitoring = "DEFAULT"

  # Name of the Kinesis Data Firehose delivery stream to deliver logs to.
  firehose_delivery_stream = null

  # The initial size of the EBS volume (in GiB) for the data drive on each
  # broker node. 
  initial_ebs_volume_size = 50

  # Specify the instance type to use for the kafka brokers (e.g.
  # `kafka.m5.large`). See
  # https://docs.aws.amazon.com/msk/latest/developerguide/msk-create-cluster.html#broker-instance-types
  # for available instance types. This field is required for non-serverless
  # cluster type.
  instance_type = null

  # Kafka version to install. See
  # https://docs.aws.amazon.com/msk/latest/developerguide/supported-kafka-versions.html
  # for a list of supported versions. This field is required for non-serverless
  # cluster type.
  kafka_version = null

  # Indicates whether you want to enable or disable the Prometheus JMX Exporter.
  open_monitoring_enable_jmx_exporter = false

  # Indicates whether you want to enable or disable the Prometheus Node
  # Exporter.
  open_monitoring_enable_node_exporter = false

  # Override automatically created Service-linked role for storage autoscaling.
  # If not provided, Application Auto Scaling creates the appropriate
  # service-linked role for you.
  override_broker_storage_autoscaling_role_arn = null

  # Name of the S3 bucket to deliver logs to.
  s3_logs_bucket = null

  # Prefix to append to the folder name.
  s3_logs_prefix = null

  # Contents of the server.properties file. Supported properties are documented
  # in the MSK Developer Guide
  # (https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-properties.html).
  server_properties = {}

  # Controls storage mode for supported storage tiers. Valid values are: LOCAL
  # or TIERED.
  storage_mode = null

}


Reference​

Required​

cluster_namestringrequired

The name of the Kafka cluster (e.g. kafka-stage). This variable is used to namespace all resources created by this module.

subnet_idslist(string)required

The subnet IDs into which the broker instances should be deployed. You should typically pass in one subnet ID per node in the cluster_size variable. The number of broker nodes must be a multiple of subnets. We strongly recommend that you run Kafka in private subnets.

vpc_idstringrequired

The ID of the VPC in which to deploy the cluster.

Optional​

additional_security_group_idslist(string)optional

A list of Security Group IDs that should be added to the MSK cluster broker instances.

[]

A list of CIDR-formatted IP address ranges that will be allowed to connect to the Kafka brokers.

[]

A list of security group IDs that will be allowed to connect to the Kafka brokers.

[]

Max capacity of broker node EBS storage (in GiB)

100

Broker storage utilization percentage at which scaling is triggered.

70
client_sasl_scram_secret_arnslist(string)optional

List of ARNs for SCRAM secrets stored in the Secrets Manager service.

[]

Optional list of ACM Certificate Authority Amazon Resource Names (ARNs).

[]
cloudwatch_log_groupstringoptional

Name of the Cloudwatch Log Group to deliver logs to.

null
cluster_sizenumberoptional

The number of brokers to have in the cluster. This field is required for non-serverless cluster type.

null
custom_tagsmap(string)optional

Custom tags to apply to the Kafka broker nodes and all related resources.

{}
Example
     {
       key1 = "value1"
       key2 = "value2"
     }

custom_tags_security_groupmap(string)optional

A map of custom tags to apply to the Security Group for this MSK Cluster. The key is the tag name and the value is the tag value.

{}
Example
     {
       key1 = "value1"
       key2 = "value2"
     }

Flag indicating whether broker storage should never be scaled in.

false

Whether SASL IAM client authentication is enabled.

false

Whether SASL SCRAM client authentication is enabled.

false
enable_client_tlsbooloptional

Whether TLS client authentication is enabled.

false

Indicates whether you want to enable or disable streaming broker logs to Cloudwatch Logs.

false

Indicates whether you want to enable or disable streaming broker logs to Kinesis Data Firehose.

false
enable_s3_logsbooloptional

Indicates whether you want to enable or disable streaming broker logs to S3.

false
enable_serverlessbooloptional

Indicates whether you want to enable msk serverless or not

false

You may specify a KMS key short ID or ARN (it will always output an ARN) to use for encrypting your data at rest. If no key is specified, an AWS managed KMS ('aws/msk' managed service) key will be used for encrypting the data at rest.

null

Encryption setting for data in transit between clients and brokers. Valid values: TLS, TLS_PLAINTEXT, and PLAINTEXT. Default value is TLS.

"TLS"

Whether data communication among broker nodes is encrypted. Default value: true.

true
enhanced_monitoringstringoptional

Specify the desired enhanced MSK CloudWatch monitoring level. See https://docs.aws.amazon.com/msk/latest/developerguide/metrics-details.html for valid values.

"DEFAULT"

Name of the Kinesis Data Firehose delivery stream to deliver logs to.

null

The initial size of the EBS volume (in GiB) for the data drive on each broker node.

50
instance_typestringoptional

Specify the instance type to use for the kafka brokers (e.g. kafka.m5.large). See https://docs.aws.amazon.com/msk/latest/developerguide/msk-create-cluster.html#broker-instance-types for available instance types. This field is required for non-serverless cluster type.

null
kafka_versionstringoptional

Kafka version to install. See https://docs.aws.amazon.com/msk/latest/developerguide/supported-kafka-versions.html for a list of supported versions. This field is required for non-serverless cluster type.

null

Indicates whether you want to enable or disable the Prometheus JMX Exporter.

false

Indicates whether you want to enable or disable the Prometheus Node Exporter.

false

Override automatically created Service-linked role for storage autoscaling. If not provided, Application Auto Scaling creates the appropriate service-linked role for you.

null
s3_logs_bucketstringoptional

Name of the S3 bucket to deliver logs to.

null
s3_logs_prefixstringoptional

Prefix to append to the folder name.

null
server_propertiesmap(string)optional

Contents of the server.properties file. Supported properties are documented in the MSK Developer Guide (https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-properties.html).

{}
Example
     {
       "auto.create.topics.enable" = "true"
       "default.replication.factor" = "2"
     }

storage_modestringoptional

Controls storage mode for supported storage tiers. Valid values are: LOCAL or TIERED.

null