Help Docs

How to monitor Kafka in Huawei Cloud

Site24x7 offers monitoring support for Huawei Cloud Kafka. By monitoring the performance and health, you can manage distributed message streaming services based on Apache Kafka. Monitoring Kafka metrics ensures healthy producer and consumer throughput and prevents message accumulation that could cause processing delays.

Use cases

Prevent data loss: When message backlog keeps increasing over time, Site24x7 alerts teams early so they can fix slow consumers or broker issues before messages expire.

Avoid storage issues: When broker disk usage crosses safe limits, Site24x7 sends a warning, giving teams time to adjust retention settings or add storage before new messages are rejected.

Maintain pipeline performance: Alerts on rising processing times helps the team to rebalance workloads or scale brokers, keeping message flow smooth and preventing delays for downstream consumers.

Setup and configuration

Kafka resources are auto-discovered and monitored during the Huawei Cloud integration. To enable monitoring, follow the steps below:

  • Navigate to Cloud > Huawei > Add Huawei Monitor. Learn how to add a Huawei Cloud monitor.
  • While adding or editing a Huawei Cloud monitor, select Kafka from the Service/Resource Types drop-down menu and click Save.
  • Navigate to Cloud > Huawei, select the created Huawei monitor, and then click Kafka.

Supported metrics

Instance

Metric name

Description

Units

Partitions The total number of topic partitions across the Kafka instance. Count
Topics The total number of topics created in the Kafka instance. Count
Accumulated Messages The total number of messages waiting to be consumed across all consumer groups. Count
Message Production The rate at which messages are being produced to the Kafka instance across all topics. Bytes/second
Message Consumption The rate at which messages are being consumed from the Kafka instance across all topics. Bytes/second
Partitions Usage The percentage of the maximum allowed partitions currently in use. Percentage

Broker

Metric name

Description

Units

Message Creation Rate The rate at which new messages are being written to the broker. Bytes/second
Message Creation The byte rate at which new message data is being ingested by the broker. Bytes/second
Message Retrieval The byte rate at which message data is being read from the broker by consumers. Bytes/second
Disk Capacity Usage The percentage of the total broker disk capacity currently consumed by message data. Percentage
Memory Usage The percentage of memory currently in use by the Kafka broker process. Percentage
CPU Usage The percentage of CPU resources currently consumed by the Kafka broker. Percentage
Connections The number of active client connections to the Kafka broker. Count
JVM Heap Memory Usage The percentage of Java Virtual Machine (JVM) heap memory currently in use by the Kafka broker. Percentage
Message Size The total size of messages currently stored on the broker disk. Bytes

Latency

Metric name

Description

Units

Average Message Creation Processing Duration The average time taken by the broker to process and acknowledge a produce request from producers. Milliseconds
Average Message Retrieval Processing Duration The average time taken by the broker to process and fulfill a fetch request from consumers. Milliseconds

Dependency View

The Dependency View  tab maps Kafka groups based on your customer requirements. Contact support and share your requirements to help us prioritize dependency mapping for the Kafka groups in your account.

Common monitored data

Outages

The Outages tab displays outage details, including start time, end time, duration, and related comments, to help track and review downtime events.

Notes

The Notes tab enables you to record descriptions, operational context, runbook steps, recent changes, and reference links to keep your team informed. Format the content using markdown syntax.

Log Report

The Log Report tab provides a unified view of the respective monitor's log status and supports CSV downloads for analysis and auditing.  

Alert Logs

The Alert Logs tab provides a comprehensive list of all alerts generated by the monitor. Easily track key details such as the monitored resource name, alert trigger time, root cause, and notification status. Learn more.

Threshold configuration

You can configure thresholds and alerts for all Kafka metrics to detect performance degradation proactively or connection issues.

  1. Go to Admin > Configuration Profiles > Threshold and Availability.
  2. Create or edit your Threshold Profile for Kafka.
  3. Assign the profile to the respective monitors to trigger alerts.

IT Automation

Use Site24x7's IT Automation to resolve common issues with Kafka performance:

  1. Go to Admin > IT Automation Templates. Then, click Add Automation Templates.
  2. Create an automation rule by selecting the automation Type (e.g., Server reboot, clear queue).
  3. Map the created rules to the Kafka, for automatic execution during alerts.

Configuration rules

Use Configuration Rules to simplify bulk setup across Kafka instances. Automatically assign Threshold Profiles, Notification Profiles, Tags, and Monitor Groups when new monitors are discovered.

Related article

Was this document helpful?

Would you like to help us improve our documents? Tell us what you think we could do better.


We're sorry to hear that you're not satisfied with the document. We'd love to learn what we could do to improve the experience.


Thanks for taking the time to share your feedback. We'll use your feedback to improve our online help resources.

Shortlink has been copied!