Causal Ordering of Messages in Distributed System

Causal ordering of messages is one of the four semantics of multicast communication namely unordered, totally ordered, causal, and sync-ordered communication. Multicast communication methods vary according to the message’s reliability guarantee and ordering guarantee. The causal ordering of messages describes the causal relationship between a  message send event and a message receive event. 

For example, if send(M1) -> send(M2) then every recipient of both the messages M1 and M2 must receive the message M1 before receiving the message M2. In Distributed Systems the causal ordering of messages is not automatically guaranteed. 

Reasons that may lead to violation of causal ordering of messages

1. It may happen due to a transmission delay.
2. Congestion in the network.
3. Failure of a system.

Protocols that are used to provide causal ordering of messages

1. Birman Schipher Stephenson Protocol
2. Schipher Eggli Sandoz Protocol

Both protocol’s algorithm requires that the messages be delivered reliably and both prefer that there is no network partitioning between the systems. The general idea of both protocols is to deliver a message to a process only if the message immediately preceding it has been delivered to the process. Otherwise, the message is not delivered immediately instead it is stored in a buffer memory until the message preceding it has been delivered.

The ISIS System

The ISIS system was developed by Ken Birman and Joseph in 1987 and 1993. It is a framework for reliable distributed communication which is achieved through the help of process groups. It is a programming toolkit whose basic features consist of process group management calls and ordered multicast primitives for communicating with the process group members. ISIS provides multicast facilities such as unordered multicast (FBCAST), casually ordered multicast (CBCAST), totally ordered multicast (ABCAST), and sync-ordered multicast (GBCAST).

The ISIS CBCAST Protocol

ISIS uses vector timestamps to implement causally ordered multicast between the members of a process group. It is assumed that all the messages are multicast to all the members of the group including the sender. ISIS uses UDP/IP protocol as its basic transport facility and sends acknowledgments and retransmits packets as necessary to achieve reliability. Messages from a given member are sequenced and delivered in order. There is no assumption that hardware support for broadcast or multicast exists. If IP multicast is implemented, then ISIS can exploit it to send a single UDP packet to the appropriate multicast address. IP multicast takes advantage of hardware like ethernet, for multicast facilities. Otherwise, packets are sent point-to-point to the individual group members.

Let the members of the group be p1,p2,p3,…pn. Once more we shall define, for each pi, a vector timestamp denoted by VTi which is used to order multicast delivery. It will turn out that VT[i] is the count of multicast messages sent by pi that causally lead up to the latest message delivered to pj. The following is the vector timestamp update algorithm:

1. All the processes pi initialize VTi to zeroes.
2. When pi multicasts a new message, it first increments VT[i] by 1. Then it piggybacks the value vt = VTi on the message.
3. When a message bearing a timestamp vt is delivered to pj, pj‘s timestamp is updated as VTj = merge(VTj, vt).

Every multicast message can be delivered to its sender immediately since it is by definition in causal order with respect to the messages already delivered to it. A multicast message arriving at pj‘s site from pi (pj != pj), however, has to be placed on the hold-back queue until it can be delivered in causal order. The incoming message’s timestamp vt is examined and the following criteria are used for transferring the message to pj‘s delivery queue:

• The message must be the next in sequence expected from pi, that is, .
• All causally prior messages that have been delivered to pi when it sent the message, should have been delivered to pj, that is, VTj[k] >= vt[k] for k != i.

It is straightforward to show that these criteria are necessary and sufficient to satisfy the safety property of causal ordering being assured. Liveness can be established if:

• All members of the group are destinations for every message.
• Multicast delivery is atomic.

Otherwise, a message delivered to pj might never be delivered to pj and so some messages might indefinitely fail the second criteria mentioned above.