Messaging Communication Among Services With Kafka

Knowing about Publish and Subscribe

Communication methods among services can be achieved directly and indirectly.

• Direct communication commonly using RPC (Remote Procedure Call) with RESTful API as the most popular choice nowadays. Take a look about RESTful API here.
• On the other hand, we can do indirect communication using messaging or publisher/subscriber method. This method needs an application called “Message Broker” to act as a mediator to hold informations about what to be processed by its subscribers (publisher is the sender of the data, and subscriber is the consumer of the data).

Using Pub/Sub has an advantage below:

The publisher services don’t need to have any complexity about how the informations they sent used, they just have to know that the data is successfully stored in the Message Broker.

If we’re using direct communication, when we got new applications or services that have to work with the related data, the producer of the resource has to adjust about sending the data, it has to think who has gonna receive the information, and this process of adjustment will happen every any brand new service that needs the data.

Otherwise, with messaging communication method we can make sure when there’s any new brand service comes to consume the related data, the producer of the data doesn’t need take extra action, it will be as it is, just let the data sent into the Message Broker.

Therefore, besides the advantage of reducing complexity above, we have to consider about the Pub/Sub disadvantages:

• Delay. Because of doing indirect communication and using a mediator, there’s a chance our informations that come into the Message Broker will be postponed and delayed to be processed in a duration we probably difficult to predict, it can be happened when the subscriber is quite slow on processing the data it consumes, or the Message Broker server is down or has low specs.
• Consistency of status. If somehow any subscriber fail to process the data, our publisher will not know about it in default, so the subscriber should has a capability of retrying on getting and processing the data.

Introduction to Kafka

Kafka is one of the popular Message Broker applications out there, it has a dozen of users. With its Apache 2.0 licence we can use Kafka for personal or even commercial purposes for free. There are few reasons why Kafka is the best choice:

• Scalable. Kafka is known reliable Message Broker to handle a lot of data, even the incoming data is increasing over time.
• Good performance. Compares to similiar applications Kafka is well tested having the ability to process data pretty good and quick.
• Persistence. Kafka store our data on disk to make sure we can avoid of losing it, so when failure happens on processing data in subsciber sides, the data will stay and standby to be reconsumed.
• Reliable ecosystem. The ecosystem of Kafka is suprisingly good, because this Message Broker is so popular and has a lot of users, we don’t need to worry of it. Kafka can be integrated with several programming languages such Java, Python, Go, JavaScript (NodeJS), C++ and so on, we also can use Kafka combined with data ecosystem software like Elasticsearch, Apache Hadoop etc.

...