Genpact Cora Knowledge Center

Support

Kafka Cluster Minimum Requirements

V9.2

Overview

You can use Apache Kafka® to send messages within Cora SeQuence or between Cora SeQuence and other applications. Apache Kafka is a distributed streaming platform that:

  • Publishes and subscribes to streams of records, similar to a message queue or enterprise messaging system.
  • Stores streams of records in a fault-tolerant durable way.
  • Processes streams of records as they occur.

To integrate with Cora SeQuence and maintain a robust, fault-tolerant and high-availability Kafka cluster, follow the recommendations below.   

ComponentRecommendationDetails
Kafka brokersThree brokers
 
To maintain high-availability of the Kafka cluster (broker leader re-election),  and to provide partition replication and better partitioning of  topics.
Apache ZooKeeper™ One nodeZookeeper coordinates between the brokers (controller election), keeps topic configurations, and access control lists (ACL).
Security protocolSASL_SSL protocolTo enable Kafka authentication and encryption.

Terminology

Make sure that you are aware of the terminology used in Cora SeQuence. 

Cora SeQuence activityKafka component
Kafka ProducerProducer 
Kafka SubscriberConsumer

For more details on Kafka components, see this page.

Minimum system requirements to run Kafka in a production environment

ItemRequirementDetails
Memory8 GB RAMKafka relies heavily on the file system for storing and caching messages.
Kafka uses heap space very carefully and does not require setting heap sizes more than 6 GB.
CPU4 cores



Note that if SSL is enabled, the CPU requirements can be significantly higher. The exact requirements depend on the CPU type and JVM implementation.

If you need to choose between faster CPUs or more cores, choose more cores. The extra concurrency that multiple cores offers will far outweigh a slightly faster clock speed.

Disk storage size500 GB Use multiple drives to maximize throughput.
Avoid network-attached storage (NAS). 
Network1 GbE-10 GbEAvoid clusters that span multiple data centers. 

Minimum software version 

SoftwareVersion
Apache Kafka2.3.1-0
Apache Zookeeper3.4.14
Java1.8 and later

For more information on Kafka system requirements, see this page.