Setting-up Apache Kafka Monitoring

• Apache Kafka - An Overview
• Monitoring Apache Kafka - What we do
• Creating a new Apache Kafka monitor
• Monitored Parameters

Apache Kafka - An Overview

Apache Kafka is an open-sourced, fault-tolerant publish-subscribe-based messaging system developed by LinkedIn. A distributed log-service, Kafka is often used in place of traditional message brokers because of its higher throughput, scalability, reliability and replication.

Monitoring Apache Kafka - What we do

An attractive option for data integration, Apache Kafka is fast and highly scalable. Kafka nodes are created and taken down in an elastic manner; with a single node handling hundreds of read/writes from thousands of clients in real-time. Data streams are split into partitions and spread over different brokers. Although very simple at a high level, Kafka has an incredible depth of technical detail, for which, having a robust Kafka monitoring software is essential to troubleshoot issues and optimize performance.

Applications Manager's Kafka monitoring aims to help administrators collect Kafka metrics, manage clusters and be alerted automatically on potential issues. Let us take a look at what you need to see to monitor Kafka and the performance metrics to gather with Applications Manager Kafka monitor:

• Resource utilization details - Automatically discover Kafka servers, monitor memory and CPU and get alerts of changes in resource consumption.
• Thread and JVM usage - Track thread usage with metrics like Daemon, Peak and Live Thread Count. Ensure that started threads don’t overload the server's memory.
• Broker, Controller and Replication Statistics - Gauge active controllers and see if brokers are up with the number of unavailable partitions. Monitor broker stats like log flush latency (to make sure longer flushes don’t back up the pipeline) and under-replicated partitions (indicating replication is not going as fast as configured).
• Network and Topic Details - Pinpoint the requests segment causing a slowdown. Keep an eye on network usage on your host so degraded performance is not network-related. Ensure disk throughput does not cause performance bottlenecks, with Broker Topic byte rates metrics.
• Fix Performance Problems Faster - Get instant notifications when there are performance issues with the components of Apache Kafka. Become aware of performance bottlenecks and take quick remedial actions before your end users experience issues.

Creating a new Kafka monitor

Supported versions: Versions 0.7.0 to 3.4.1

Prerequisites for monitoring Apache Kafka: JMX support must be enabled in order to monitor Apache Kafka in Applications Manager. To learn more about enabling JMX in Kafka, click here.

Using the REST API to create a new Kafka monitor: Click here

To create an Apache Kafka Monitor, follow the steps given below: 

1. Click on New Monitor link. Choose Apache Kafka.
2. Enter Display Name of the monitor.
3. Enter the IP Address or hostname of the host in which Kafka is running.
4. Enter the JMX Port in the JMX Port field.
5. Enter the credential details like user name, password and JNDIPath or select credentials from a Credential Manager list.
6. Enter the polling interval time in minutes.
7. Click Test Credentials button, if you want to test the access to Apache Kafka Server.
8. Choose the Monitor Group from the combo box with which you want to associate Apache Kafka Monitor (optional). You can choose multiple groups to associate your monitor.
9. Click Add Monitor(s). This discovers Apache Kafka from the network and starts monitoring.

Note:
In case you are unable to add the monitor even after enabling JMX, try providing the below argument:
  -Djava.rmi.server.hostname=[YOUR_IP]

Monitored Parameters

Go to the Monitors Category View by clicking the Monitors tab. Click on Apache Kafka under the Middleware/Portal Table. Displayed is the Apache Kafka bulk configuration view distributed into three tabs:

• Availability tab gives the Availability history for the past 24 hours or 30 days.
• Performance tab gives the Health Status and events for the past 24 hours or 30 days.
• List view enables you to perform bulk admin configurations.

Applications Manager's Kafka performance monitoring provides complete visibility into your Kafka servers based on the metrics listed in the following tabs:

• Overview
• Controller Details
• Broker Details
• Network Details
• Topics Details
• Configurations

Overview

Controller Details

In a Kafka cluster, one of the brokers serves as the controller, which is responsible for managing the states of partitions and replicas and for performing administrative tasks like reassigning partitions.

Broker Details

Network Details

Topics Details

Configurations