CloudWatch is good for getting the data logs from your stack. But for analysis and visualization, Elasticsearch and Kibana may be a better option
When using cloud computing services for running your business processes, it’s important to constantly monitor them. By keeping track of your application and the services supporting it, you can ensure that your users are getting the required performance.
Cloud monitoring is also essential to keep your applications healthy and to keep your costs from going out of control. AWS CloudWatch is a solution that helps you monitor your entire stack and gets you detailed operational and performance data from them.
But for many businesses, this solution has many drawbacks. With the help of Elasticsearch and Kibana, businesses can better monitor their cloud infrastructure.
In this article, we explore the three services and how businesses can benefit from connecting CloudWatch to Elasticsearch. We’ll also discuss the steps to do that.
AWS CloudWatch is a cloud service that lets you manage and monitor your infrastructure easily. Once set up, the solution provides detailed data logs and metrics for your AWS, hybrid, or on-premise infrastructure. CloudWatch offers logs and metrics showcasing both the performance and status of your infrastructure.
One of the main benefits of CloudWatch is that it offers a single solution to monitor your entire stack. It can monitor your application, the underlying infrastructure, the network, and the database data through one platform. With CloudWatch you can view the metrics for every one of the AWS services that you use and can create custom dashboards to monitor your apps.
Besides monitoring the AWS services, CloudWatch also offers you various automation to help you manage your applications efficiently. For instance, you can monitor CPU usage, memory disk, and data and set up alarms in case they go beyond your thresholds. You can also set up automation to shut down services in case they go beyond the limits you’ve set.
The goal of Amazon CloudWatch is to simplify monitoring applications and associated services and help you focus your resources better. The service offers detailed insights into your infrastructure, allows you to analyze the metrics, and optimize the costs and performance associated with your applications.
AWS Elasticsearch is an open-source database that’s primarily used for data analytics and search. Elastic NV initially offered the solution under the Apache license in 2010 and Elasticsearch soon became a popular tool for cloud analytics, business intelligence, and other use cases.
But when Elastic announced it will stop offering new versions under the open-source version, AWS created an open-source project to continue offering the solution with the same level of freedom and flexibility. While both names are used, officially, the AWS solution is known as OpenSearch.
AWS also offers a fully managed OpenSearch service or AWS-managed Elasticsearch in which Amazon will take care of provisioning the hardware, updating the software, and ensuring recovery and backups.
Amazon Elasticsearch works by storing the documents you upload into its database and making it more searchable. After this, you can easily search through the data and quickly retrieve what you want using the Elasticsearch API.
The solution offers high-speed searches even when you’re working with large amounts of data. It also offers high read and write speeds making it ideal for monitoring your applications and detecting anomalies in real time.
OpenSearch also comes with a lot of plugins and tools that add more functionality to it. For instance, you can use Logstash, Beats, and Amazon Kinesis Firehose to add data to the solution. You can also use Kibana to visualize your data and create custom dashboards easily.
Kibana is an open-source data visualization tool popularly used to visualize and analyze logs and large volumes of data. The solution is part of the ELK stack or the Elastic stack, made of Elasticsearch, Logstash, and Kibana. In this approach, Logstash gets the data into the Elasticsearch database which then Kibana will use to create the visualizations.
Kibana lets you visualize the data in the form of line charts, bar graphs, pie charts, heat maps, and others. You can also create dashboards using these visualizations and from those, you can make CSV reports.
If you want more out of Kibana — more visualizations or other changes — plenty of third-party plugins can help you out. If you want to visualize time series data, Kibana offers Timelion.
Kibana also offers Canvas with which you can showcase your data in a way that reflects your brand. The tool lets you create reports with your brand colors, logos, and other elements. Canvas makes it easy to create effective presentations to tell your story with data.
While Elasticsearch is no longer open, Kibana is still available for free and the code is still open.
AWS CloudWatch is a powerful tool to monitor your AWS services and applications. It allows users to create dashboards, reports, and visualizations and offers a single platform to monitor and manage your entire stack. But it has some drawbacks or limitations.
While CloudWatch can collect detailed logs for your applications, it’s not easy to analyze or visualize the collected data with it. The service does offer visualizations and dashboards, but it is very limited. They offer limited functionality compared to Kibana and are not easy to create in the first place. It takes a lot of effort to create dashboards, alarms, and alerts (not impossible, just challenging, compared to Elasticsearch).
On the other hand, Elasticsearch and Kibana offer a diverse range of visualizations, and it’s easier to create dashboards here. It provides a suite of editors with which you can create different types of visualizations and panels of data. For instance, with the Lens editor, you can drag in the data fields to your workspace and it will automatically choose the visualizations that best display or present it. You can of course choose your own visualizations.
Kibana also has built-in interactive tools with which your users can go in-depth into the data. It also lets you decide what happens when users click on the different data points and visualization using drilldowns. Using these drilldowns, you can send the user to another dashboard or a different URL.
It’s also not easy to manage identity and access control with CloudWatch. While you can grant access using IAM policies, it may be too complicated.
With Elasticsearch, you get role-based access control. You can assign different privileges to different roles and in turn assign these roles to users or groups. After a user is identified and authenticated, Elasticsearch will also check if they’re allowed to execute the request they sent.
Elasticsearch comes with a set of built-in roles which you can just straight up apply to users or groups. It also lets you define roles according to your preferences.
While CloudWatch lets you collect data logs from your AWS services, it’s not easy to search through them. But with Elasticsearch, it’s easy to search through the logs with reference to time, keywords, and even errors.
Kibana lets users explore logs in Elasticsearch and filter and search through them. With its Logs UI, users can go through Elasticsearch data visually, and with its Discover tool, you can find go through it with a query language.
CloudWatch is not generally easy to work with; AWS lets you forward logs only to Elasticsearch, Lambda, and S3. The platform also has limitations on batch size and data archives which cannot be changed. CloudWatch also limits the number of log event fields in a group to 1000.
We’ll be starting by creating an Elasticsearch domain. If you already have an Elasticsearch cluster, you can skip those steps.
You can use different formats to send logs to Elasticsearch. In this example, we’ll be using JSON.
There are other methods to stream CloudWatch logs to AWS Elasticsearch. Here we’ll be discussing how to do so with a Lambda function.
1.1 Open the AWS console
1.2 Open Elasticsearch Service management console
1.3 Click ‘Create Domain.’
1.4 Select the deployment type as ‘Development and Testing.’
1.5 Choose the Elasticsearch version, give a name, configure your domain, configure your networks, and after reviewing, click on the ‘Confirm’ button.
1.6 Ensure that the domain status changes to ‘Active’.
Since we are using an AWS Lambda function to stream, we need to set up the necessary permissions for it.
2.1 Open the AWS IAM console
2.2 Click on ‘Policies’ – click ‘Create Policy.’
2.3 Open the JSON tab in the window and paste the following commands. Change the arn in the below command to that of your Elasticsearch cluster:
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Action”: [
“es:*”
],
“Effect”: “Allow”,
“Resource”: “arn:aws:es:eu-west-1:****************:domain/test-es-cluster/*”
}
]
}
2.4 Click review and add a name and description to the policy, click ‘Create Policy.’
2.5 Click on the roles tab on the IAM and click ‘Create Role’
2.6 In the next Window, you’ll be asked to choose the trusted entity. Click ‘AWS Service’. And choose AWS Lambda as the use case.
2.7 Click ‘Next: Permissions.’
2.8 In this window, choose the policy that we created earlier.
2.9 Click ‘Next: Tags’.
2.10 Choose the relevant tags.
2.11 Click on ‘Review,’ add the name and description, and click ‘Create Role’.
2.12 Head back to the ‘Roles’ tab and select the one we just created.
2.13 In the Trust Relationship tab, click on the ‘Edit Relationship’ button.
2.14 Copy and paste the below code after removing everything that’s already there:
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Principal”: {
“Service”: “lambda.amazonaws.com”
},
“Action”: “sts:AssumeRole”
}
]
}
3.1 Open the AWS console and get CloudWatch.
3.2 Open the Logs tab from the menu on the left – choose the log group we want to stream to Elasticseach and click ‘Actions’.
3.3 Click on ‘Stream to Amazon Elasticsearch service.’
3.4 In the next window, choose the Elasticsearch cluster you want to stream to.
3.5 Choose the log format — JSON in our case — in the next window.
3.6 Choose the filter pattern if necessary. If not, CloudWatch will stream all logs to the Elasticsearch cluster.
3.7 Review your configurations in the next Window and hit ‘Start Streaming.’
While AWS CloudWatch is a powerful tool for monitoring your infrastructure, it has limitations when it comes to data analysis and visualization. By connecting CloudWatch to AWS Elasticsearch or OpenSearch, businesses can gain access to a broader range of visualization options and analyze their data more easily. Kibana, a free and open-source data visualization tool, allows for customizable dashboards and reports that can help businesses optimize their applications’ performance and minimize costs. Overall, the integration of CloudWatch with Elasticsearch and Kibana can provide businesses with more comprehensive monitoring capabilities and help them make more informed decisions.
We hope this guide provided enough insight to give you the understanding you need to connect CloudWatch to AWS Elasticsearch — but if not, feel free to reach out to the AWS experts at MA Technologies for further assistance.
