Apache Kafka vs RabbitMQ

Kafka, an open-source distributed event streaming platform developed by the Apache Software Foundation. It is written in Java and Scala. Kafka uses a data stream for the delivery of messages and is suitable for both online and offline message consumption. The aim of the project is to provide high throughput, low latency platform for real-time data feeds handling, streaming analytics, data integration, and mission-critical applications. It employs an approach that is pull-based and lets users request message batches from the specific offsets. Although only being shipped with a Java client, it offers an adapter SDK that allows programmers to build their unique system integration.

Features of Kafka

1. It has high throughput and is built to process tons of messages per second and to handle huge amounts of data.
2. Zero downtime and data loss can be achieved using replication-factor > 1 for brokers.
3. Using ingest pipelines, Kafka can replicate the events in a broker.
4. The reliability of Kafka is because of its distributed, replicated, partitioned, and fault-tolerant nature.

RabbitMQ

RabbitMQ is an open-source distributed general-purpose message-broker software facilitating efficient message delivery in complex routing scenarios. Distributed because it runs as a cluster of nodes where the queues are distributed across the nodes.

It initially implemented the Advanced Message Queuing Protocol. It employs a push model and prevents overwhelming users via the consumer-configured prefetch limit. It is commonly used for handling background jobs or as a message broker between the microservices.

Features of RabbitMQ

1. It has built-in clustering, a single broker can be formed from multiple RabbitMQ servers on a local network by clustering them together.
2. It supports messaging over a wide variety of messaging protocols like AMQP, STOMP, and MQTT.
3. Trace support is provided by RabbitMQ which allows debugging and discovering what happens in case the messaging system misbehaves.
4.  It has clients for any language that may be used.

Note: 

The purpose of Kafka and RabbitMQ is the same but have different capacities. They are event-handling systems that are open-source and readily adopted by enterprises. They are commercially supported pub/sub systems.

Following is a table of differences between Kafka and RabbitMQ:

S.No.	Parameter	Kafka	RabbitMQ
1.	Performance	Around 1 million messages per second	Around 4k-10k messages per second
2.	Payload Size	It has a default 1MB limit	There are no constraints
3.	Messages Synchronicity	Durable message store which replays messages	Can be asynchronous/synchronous
4.	Data Unit	Continuous stream	Message
5.	Data Type	Operational	Transactional
6.	Topology	Subscribe/Publish based	Exchange type – Fan out, Topic, Direct, Header-based
7.	Data flow	Continuous unbounded data in form of key-value pairs	Bounded distinct data packets in the form of messages
8.	Structure of Event Storage	Logs	Queue
9.	Message delivery system	Consumer pulls messages as required (Pull based model)	Message pushed to specific queues
10.	Consumer Mode	Smart consumer/Dumb broker	Dumb consumer/Smart broker

What to use Kafka or RabbitMQ?

Kafka is to be used when:

• Messages need to be consumed very quickly, it should be our go-to event handling platform.
• There is a need for replaying messages.
• The application has a High Throughput around 100K/sec events or more than that.

RabbitMQ is to be used when:

• Addition of consumers dynamically as it does not require a change of publisher.
• Feature of replaying messages on a topic is not needed as it can’t replay events.
• Then the application has to support legacy protocols such as AMQP 0-9-1, STOMP, AMQP 1.0, and MQTT.

Use Cases of Kafka and RabbitMQ

Kafka is used for tracking high-throughput activity, event sourcing, stream processing, log aggregation, etc.

RabbitMQ is used for Legacy Applications, Complex Routing, etc.