0x370 Communication

• 1. Communication
  • 1.1. REST
  • 1.2. SOAP
  • 1.3. RMI (Remote Method Invocation)
  • 1.4. RPC (Remote Procedure Call)
  • 1.5. Message Passing
• 2. Naming
• 3. Coordination
  • 3.1. Paxos
  • 3.2. Raft
  • 3.3. Consistency Models
    • 3.3.1. strong consistency model
    • 3.3.2. weak consistency model
  • 3.4. CAP Theorem
• 4. MPI
  • 4.1. Communicator
  • 4.2. Point-to-Point Communication
    • 4.2.1. Send and Recv
    • 4.2.2. All-to-All
  • 4.3. Collectives Operations
  • 4.4. Python API

1. Communication

1.1. REST

REST's principle is to defined named resources that can be manipulated using a small number of methods.

• HTTP URLs are used to identify resources (i.e. nouns)
• HTTP methods (e.g. POST, GET) are known as verbs of API, HTTP features for cache control, authentication and content type negotiation

API of RESTful can be defined using OpenAPI, which is a yaml/json file defining

REST is popular in the context of cross-organizational service and often associated with microservices

1.2. SOAP

• XML-based protocol
• it can be independent from HTTP and avoid using most HTTP features

API of SOAP is described using a language called WSDL (WebServices Description Language)

1.3. RMI (Remote Method Invocation)

using distributed object

CORBA

Protocol (DCOM, distributed COM) It is an extension of the COM

1.4. RPC (Remote Procedure Call)

using procedure (RPC) instead of objects (RMI), check this article

Check this article why distributed objects (in RMI) should be avoided in microservice

Protocol (gRPC, stubby) based on Google's internal stubby framework

Microsoft's doc has a good comparison between REST-based API and RPC

1.5. Message Passing

Message Passing dataflow or Pub/Sub is a form of asynchronous service-to-service communication used in serverless and microservices architectures.

Work flow:

• A message (just a sequence of bytes) is delivered to an intermediary called a message broker (or message queue) which stores the message temporarily.
• The message is then delivered to subscriber.

It has several advantages over RPC:

• act as a buffer when recipient is unavailable
• will redeliver when failed
• sender does not need to know the IP/port of receiver
• decouples sender from receiver

2. Naming

3. Coordination

3.1. Paxos

Paxos Made Simple

3.2. Raft

3.3. Consistency Models

Consistency model is a contract between process and a data store. It says that if processes agree to obey certain ruless, the store promises to work correctly

3.3.1. strong consistency model

Model (strict consistency) the strongest consistency model

it returns a value corresponding to the latest write (due to message latency, not possible in a distributed system)

• Unix file system guarantees this.
• NFS/AFS approximates this
• S3 makes no real attempt to approximate it

Model (sequential consistency)

The result execution is the sequential consistent if we can find a sequential order of execution by all processes subject to the execution order in each process.

3.3.2. weak consistency model

Model (eventual consistency)

If no new updates are made to a given data item, eventually, all accesses to that item will return the last updated value

Example

• DNS

3.4. CAP Theorem

Two out of three C/A/P can be satisfied

• consistency: every read receives most recent write
• availability: every request receives a (non-error) response, without the guarantee it contains the latest write
• partition tolerance: the system continues to operate despite the network failure

4. MPI

MPI defines the syntax and semantics of portable message-passing libraries routines.

Check this tutorial

4.1. Communicator

Communicator objects connect groups of processes in the MPI session. the default one is MPI_COMM_WORLD

4.2. Point-to-Point Communication

4.2.1. Send and Recv

MPI_send, MPI_Recv are both blocking API sending number of count of type datatype in data stream with following syntax

MPI_Send(
    void* data,
    int count,
    MPI_Datatype datatype,
    int destination,
    int tag,
    MPI_Comm communicator)

MPI_Recv(
    void* data,
    int count,
    MPI_Datatype datatype,
    int source,
    int tag,
    MPI_Comm communicator,
    MPI_Status* status)

4.2.2. All-to-All

4.3. Collectives Operations

4.4. Python API

mpi4py provides a python API, see the following example:

It can be launched with something like mpirun -n 8 /python hello_mpi.py

from mpi4py import MPI

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
world = comm.Get_size()

import os

os.environ["CUDA_VISIBLE_DEVICES"] = str(rank)
print("rank ", rank, " world ", world)

Reference: https://www.quora.com/What-Is-CAP-Theorem-1