Messaging Architecture for Distributed Systems
A distributed system can be simply viewed as a collection of interconnected devices (or nodes) that work together to perform certain tasks or provide a set of services. A distributed system can host a wide range of devices that typically don’t share memory. These devices can be homogenous or heterogeneous in terms of their capabilities, characteristics, and operating systems.
Enabling communication between such devices to coordinate certain tasks, synchronize their work, and exchange data is considered a core topic to build a reliable, scalable, and functional distributed system. Communication style can follow a centralized architecture where a central device facilitates communication between other devices (as in client-server model) or a decentralized architecture where all devices play the same role (as in peer-to-peer), or a hybrid architecture. Communication channels can also be classified by the messaging patterns that include request-response, push-pull, and publish-subscribe. Under the umbrella of enabling communication comes a wide range of sub-topics that include encoding and serialization, message routing and ordering, security and privacy, and handling network failures and delays. However, we would like to discuss another important sub-topic — distributed messaging systems.
A communication system -from a high-level architectural point of view- is composed of four main entities: 1) the message (e.g., instruction, data, document), 2) the producer or sender process running on a device and composing the message, 3) the consumers or recipient process(es) on the same or different device(s) and react according to the received message, and 4) the messaging system which is software and hardware infrastructure, protocols, tools, and communication links utilized to deliver the message from the sender process to the recipient processes.
Taking one step back, we can see that interactions in general (and message-passing in particular) between devices through the different communication channels can be classified into synchronous, asynchronous, and hybrid. In a synchronous message-passing model, the sender process is blocked after sending the message, waiting for the receiver to accept, process the message, and then respond back to the sender process. This model targets more sequential and predictable execution. However, some scenarios may not accept the possible delays, thus an asynchronous message-passing model targets a more loose coupling between the sender and receiver. In this case, the sender process can continue to work after sending the message and be notified (e.g., via callback function) later when a response is available. In other scenarios, a hybrid message-passing model is required, where the processes are given the flexibility to choose between block and wait or continue and be notified. The choice depends on the type and priority of the communication, the target application, intended reliability, and other performance metrics.
To enable synchronous and asynchronous message-passing, the communicating devices need a storage area or a buffer that can accept these messages temporarily — and this is referred to as a Message Queue (MQ). MQ -an essential and critical component of any messaging system- is simply a queue that accepts messages from the sender processes, stores them in the order of their arrival, and waits for the consumers to request them. Such MQ, like a mailbox, can be locally hosted on individual devices to accept the received messages and enable the recipient process to access and process the messages at their own speed. MQ can also, like the post office, be located on centralized device(s) that either notify the intended recipients or are periodically checked by the connected devices for messages.
It is worth mentioning that there are different forms and styles for messaging systems to enable unicast, multicast, and broadcast communication paths efficiently. Aside from the traditional message delivery method, where the senders list the intended recipients in the header of their messages, the publish-subscribe message delivery method works in a different way. Let’s say that some devices are interested in data about “Weather” in “Chicago”, such devices -under this model- can subscribe to such topics. Message producers (referred to as publishers) can tag their messages with the topic(s) that describe the content, and the message queue (also referred to as message broker) automatically delivers the received messages to the corresponding subscribers. In the publish-subscribe (or Pub/Sub) model, MQ can enable asynchronous message-passing along with complete communication decoupling, where the senders and receivers may not be aware of each other.
There is a number of messaging system protocols that define the formal rules, format, sequence, and semantics between communicating devices. Such protocols include Message Queuing Telemetry Transport (MQTT), Extensible Messaging and Presence Protocol (XMPP), Simple Text Oriented Messaging Protocol (STOMP), and Advanced Message Queuing Protocol (AMQP)
There is also a large set of cloud-hosted and open-source messaging system platforms available to enable different models of communication in your distributed systems: Apache Kafka, Apache ActiveMQ, RabbitMQ, Apache Pulsar, Microsoft Azure Service Bus, and Amazon Simple Queue Service (SQS).
