Skip to main content
Amazon EKS 0.72.0Last updated in version 0.64.3

EKS Fargate Container Logs Module

View Source Release Notes

This module supports collecting logs from Fargate Pods and shipping them to CloudWatch Logs, Elasticsearch, Kinesis Streams, or Kinesis Firehose.

This Terraform Module sets up the required Kubernetes Namespace and ConfigMap for configuring the Fluent Bit instance that runs on Fargate worker nodes. This allows you to instrument container log aggregation on Fargate Pods in EKS without setting up a side car container.

This module is for setting up log aggregation for EKS Fargate Pods. For other pods, take a look at the eks-container-logs module.

How does this work?​

This module solves the problem of unifying the log streams from EKS Fargate Pods in your Kubernetes cluster to be shipped to an aggregation service on AWS (CloudWatch Logs, Kinesis, or Firehose) so that you have a single interface to search and monitor your logs. Since Fargate doesn't support DaemonSets, traditionally you had to rely on side car containers to implement the log aggregation. This required writing logs to a location that was shared with the side cars, requiring instrumentation to both the application and infrastructure.

This module leverages the built in fluent-bit service on Fargate worker nodes that run the EKS Pods. EKS supports configuring fluent-bit on the Fargate workers to ship to arbitrary targets if it sees a special ConfigMap that contains the fluent-bit configuration. The Fargate fluent-bit service expects to see the fluent-bit configuration in a ConfigMap named aws-logging in the aws-observability Namespace. This module can be used to manage the Namespace and the ConfigMap.

You can read more about fluent-bit in their official home page. You can also learn more about Fargate Pod Logging in the official AWS documentation.

What is the difference with fluentd?​

fluent-bit is an optimized version of fluentd that focuses on streaming and aggregating log files. fluentd has a larger ecosystem of plugins that enable various processing capabilities on top of the logs prior to aggregating in the data store.

For most EKS deployments, it is recommended to use this fluent-bit module for container log aggregation. Unless you have a specific need for a plugin only supported by fluentd, the superior performance and memory footprint of fluent-bit will ensure resources are available on your EKS workers for your Pods.

Log format​

This module leverages native plugins for Kubernetes built into fluent-bit that extract additional metadata for each Pod that is reporting. Each log is shipped to the respective outputs in the following structure:

{
    "kubernetes": {
        "namespace_name": "NAMESPACE_WHERE_POD_LOCATED",
        "pod_name": "NAME_OF_POD_EMITTING_LOG",
        "pod_id": "ID_IN_KUBERNETES_OF_POD",
        "container_hash": "KUBERNETES_GENERATED_HASH_OF_CONTAINER_EMITTING_LOG",
        "container_name": "NAME_OF_CONTAINER_IN_POD_EMITTING_LOG",
        "docker_id": "ID_IN_DOCKER_OF_CONTAINER",
        "host": "NODE_NAME_OF_HOST_EMITTING_LOG",
        "labels": {
            "KEY": "VALUE",
        },
        "annotations": {
            "KEY": "VALUE"
        }
    },
    "log": "CONTENTS_OF_LOG_MESSAGE",
    "stream": "STDERR_OR_STDOUT",
    "time": "TIMESTAMP_OF_LOG"
}

This allows you to filter and search the logs by the respective attributes. For example, the following CloudWatch Insights Query can be used to search for all logs from Pods in the kube-system Namespace:

fields @timestamp, @message
| filter kubernetes.namespace_name = "kube-system"
| sort @timestamp desc
| limit 20

Sample Usage​

main.tf

# ------------------------------------------------------------------------------------------------------
# DEPLOY GRUNTWORK'S EKS-FARGATE-CONTAINER-LOGS MODULE
# ------------------------------------------------------------------------------------------------------

module "eks_fargate_container_logs" {

  source = "git::git@github.com:gruntwork-io/terraform-aws-eks.git//modules/eks-fargate-container-logs?ref=v0.72.0"

  # ----------------------------------------------------------------------------------------------------
  # REQUIRED VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # List of ARNs of Fargate execution IAM roles that should have permission to
  # talk to each output target. Policies that grant permissions to each output
  # service will be attached to these IAM roles.
  fargate_execution_iam_role_arns = <list(string)>

  # ----------------------------------------------------------------------------------------------------
  # OPTIONAL VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # Configurations for forwarding logs to AWS managed Elasticsearch. Set to null
  # if you do not wish to forward the logs to ES.
  aws_elasticsearch_configuration = null

  # The AWS partition used for default AWS Resources.
  aws_partition = "aws"

  # Configurations for forwarding logs to CloudWatch Logs. Set to null if you do
  # not wish to forward the logs to CloudWatch Logs.
  cloudwatch_configuration = null

  # Annotations to associate with the aws-logging ConfigMap
  configmap_annotations = {}

  # Labels to associate with the aws-logging ConfigMap
  configmap_labels = {}

  # Can be used to provide custom filtering of the log output. This string
  # should be formatted according to Fluent Bit docs, as it will be injected
  # directly into the fluent-bit.conf file.
  extra_filters = ""

  # Can be used to provide custom parsers of the log output. This string should
  # be formatted according to Fluent Bit docs, as it will be injected directly
  # into the fluent-bit.conf file.
  extra_parsers = ""

  # Configurations for forwarding logs to Kinesis Firehose. Set to null if you
  # do not wish to forward the logs to Firehose.
  firehose_configuration = null

  # Whether or not Kubernetes metadata is added to the log files
  include_kubernetes_metadata = true

  # Configurations for forwarding logs to Kinesis stream. Set to null if you do
  # not wish to forward the logs to Kinesis.
  kinesis_configuration = null

  # The time Fluent Bit waits until it communicates with the API server for the
  # latest metadata. The smaller the TTL, the more load is generated on the API
  # server. This setting will only have effect, when
  # 'include_kubernetes_metadata' is 'true'.
  kubernetes_metadata_cache_ttl = "300s"

  # When enabled, it checks if the log field content is a JSON string map, if
  # so, it append the map fields as part of the log structure. This setting will
  # only have effect, when 'include_kubernetes_metadata' is 'true'.
  kubernetes_metadata_merge_log = false

  # If Merge_Log_Key is set, all the new structured fields taken from the
  # original log content are inserted under the new key. This setting will only
  # have effect, when 'include_kubernetes_metadata' is 'true'.
  kubernetes_metadata_merge_log_key = null

  # Annotations to associate with the aws-observability Namespace
  namespace_annotations = {}

  # Labels to associate with the aws-observability Namespace
  namespace_labels = {}

  # When true, all IAM policies will be managed as dedicated policies rather
  # than inline policies attached to the IAM roles. Dedicated managed policies
  # are friendlier to automated policy checkers, which may scan a single
  # resource for findings. As such, it is important to avoid inline policies
  # when targeting compliance with various security standards.
  use_managed_iam_policies = true

}


Reference​

Required​

List of ARNs of Fargate execution IAM roles that should have permission to talk to each output target. Policies that grant permissions to each output service will be attached to these IAM roles.

Optional​

Configurations for forwarding logs to AWS managed Elasticsearch. Set to null if you do not wish to forward the logs to ES.

object({
    # The AWS region where the Elasticsearch instance is deployed.
    region = string

    # Elasticsearch endpoint to ship logs to.
    endpoint = object({
      host = string
      port = number
    })

    # Whether or not AWS based authentication and authorization is enabled on the Elasticsearch instance.
    use_aws_auth = bool

    # Whether or not TLS is enabled on the Elasticsearch endpoint.
    use_tls = bool

    # Match string for logs to send to Elasticsearch.
    match = string
  })
null
Details

     Elasticsearch endpoint to ship logs to.

Details

     Whether or not AWS based authentication and authorization is enabled on the Elasticsearch instance.

Details

     Whether or not TLS is enabled on the Elasticsearch endpoint.

Details

     Match string for logs to send to Elasticsearch.

aws_partitionstringoptional

The AWS partition used for default AWS Resources.

"aws"
cloudwatch_configurationobject(…)optional

Configurations for forwarding logs to CloudWatch Logs. Set to null if you do not wish to forward the logs to CloudWatch Logs.

object({
    # The AWS region that holds the CloudWatch Log Group where the logs will be streamed to.
    region = string

    # The name of the AWS CloudWatch Log Group to use for all the logs shipped by the cluster.
    log_group_name = string

    # Prefix to append to all CloudWatch Log Streams in the group shipped by fluentbit.
    log_stream_prefix = string
  })
null
Details

     The name of the AWS CloudWatch Log Group to use for all the logs shipped by the cluster.

Details

     Prefix to append to all CloudWatch Log Streams in the group shipped by fluentbit.

configmap_annotationsmap(string)optional

Annotations to associate with the aws-logging ConfigMap

{}
configmap_labelsmap(string)optional

Labels to associate with the aws-logging ConfigMap

{}
extra_filtersstringoptional

Can be used to provide custom filtering of the log output. This string should be formatted according to Fluent Bit docs, as it will be injected directly into the fluent-bit.conf file.

""
extra_parsersstringoptional

Can be used to provide custom parsers of the log output. This string should be formatted according to Fluent Bit docs, as it will be injected directly into the fluent-bit.conf file.

""
firehose_configurationobject(…)optional

Configurations for forwarding logs to Kinesis Firehose. Set to null if you do not wish to forward the logs to Firehose.

object({
    # The AWS region that holds the Firehose delivery stream.
    region = string

    # The name of the delivery stream you want log records sent to. This must already exist.
    delivery_stream_name = string
  })
null
Details

     The name of the delivery stream you want log records sent to. This must already exist.

Whether or not Kubernetes metadata is added to the log files

true
kinesis_configurationobject(…)optional

Configurations for forwarding logs to Kinesis stream. Set to null if you do not wish to forward the logs to Kinesis.

object({
    # The AWS region that holds the Kinesis stream.
    region = string

    # The name of the stream you want log records sent to. This must already exist.
    stream_name = string
  })
null
Details

     The name of the stream you want log records sent to. This must already exist.

The time Fluent Bit waits until it communicates with the API server for the latest metadata. The smaller the TTL, the more load is generated on the API server. This setting will only have effect, when 'include_kubernetes_metadata' is 'true'.

"300s"

When enabled, it checks if the log field content is a JSON string map, if so, it append the map fields as part of the log structure. This setting will only have effect, when 'include_kubernetes_metadata' is 'true'.

false

If Merge_Log_Key is set, all the new structured fields taken from the original log content are inserted under the new key. This setting will only have effect, when 'include_kubernetes_metadata' is 'true'.

null
namespace_annotationsmap(string)optional

Annotations to associate with the aws-observability Namespace

{}
namespace_labelsmap(string)optional

Labels to associate with the aws-observability Namespace

{}

When true, all IAM policies will be managed as dedicated policies rather than inline policies attached to the IAM roles. Dedicated managed policies are friendlier to automated policy checkers, which may scan a single resource for findings. As such, it is important to avoid inline policies when targeting compliance with various security standards.

true