Message Queues & Kafka : An Overview
Overview in layman's term: In a typical Distributed system , Message Queues play a critical role. How ? Let us start from scratch . We have a monolithic application let's suppose. The server calls are handled by endpoints and things are going smooth. With time , the number of request grow. Now we want to scale up. We try to add more server nodes , where each node can now handle number of requests in parallel. The server nodes can be distributed. We still get latency. To improve upon that , we try to scale up further by breaking the services deployed on server with microservices . These microservices may reside on different nodes . This now becomes a distributed system. Within a distributed system , the major challenge is to communicate between nodes . We could use plain https , but then within a large systems , it would be very difficult to maintain data flow and offsets at the receiving end. This bottleneck is resolved by use of message queues, where the producer put
