How do I consolidate application logs across accounts?
I am trying to consolidate our app logs in to the `logs` account and could use some guidance on how Gruntwork proposes I do that. We have EKS clusters in various accounts and have `fluent-bit` installed in the cluster which ships the logs to a CloudWatch Log Group in the account itself. We would want these logs be available in the centralized logs account, so we can look at exposing the logs to developers. --- <ins datetime="2022-08-09T19:08:27Z"> <p><a href="https://support.gruntwork.io/hc/requests/109119">Tracked in ticket #109119</a></p> </ins>
This is actually a bit tricky to do since CloudWatch Logs doesn’t natively support cross account access, making it a bit of a headache to consolidate the logs. There are a couple of workarounds, but none of them are particularly easy to implement, and we don't really have modules that make the process any easier. - **Use Lambda and CloudWatch Log Subscription Filters**. In this approach, you can create a Lambda function that does the cross account migration process for you. You will want to define a new Lambda function that is able to assume a role in the `logs` account to push the logs from the app account to a CloudWatch Log Group in the `logs` account. You can then hook this up to the source Log Group in the app account to stream the logs every time a new entry is pushed to the app account log group. - **Use Kinesis to ship log data to the logs account**. In this approach, you can configure `fluent-bit` to ship the logs to an AWS Kinesis stream instead of CloudWatch Logs. This works because Kinesis natively supports [cross account streaming](https://aws.amazon.com/blogs/architecture/field-notes-how-to-enable-cross-account-access-for-amazon-kinesis-data-streams-using-kinesis-client-library-2-x/), unlike CloudWatch Log Groups, so you can define the Kinesis stream in the `logs` account without deploying any additional infrastructure in the app accounts. You can then stream the log entries from Kinesis out to a CloudWatch Log Group in the `logs` account for searching and indexing. - **Use Elasticsearch as your log monitoring solution instead of CloudWatch Logs**. In this approach, you would deploy a managed OpenSearch instance in the logs account that is available to each of the app environments (either using a public OpenSearch instance, or via VPC peering across accounts). You can then configure `fluent-bit` to ship the logs to the single OpenSearch instance in the `logs` account from each of the app accounts. From there, your developers can access the Kibana interface deployed with the OpenSearch interface to view the logs. You can stand up an OpenSearch instance using our [service module](https://github.com/gruntwork-io/terraform-aws-service-catalog/tree/master/modules/data-stores/elasticsearch).