Advanced Communication
An advantage of the C language is the possibility it offers to create structures corresponding to the physical quantities. This section deals with advanced MPI routines to send/receive C structures from a processor to another.
Classical N-body Systems
A classical $N$-body system consists of $N$ particles following the laws of classical physics and interacting via a given potential functions. A particle can be characterized by its position and velocity vectors, its mass, and possibly its charge. The following C structure could therefore correspond to such a classical particle.:
typedef struct { double x, y, z; double vx, vy, vz; double mass; int charge; } body;
The charge could be measured in terms of the electron charge, and the position and velocity vectors in terms of scaled units.
A need arises to transfer particles, i.e., structures, from processor to processor. For instance, a model of a gas flow in a tube
could consist of a large number of particles following classical trajectories as they move through the tube, with cyclic boundary conditions (i.e., particles escaping to the right re-enter to the left of the tube), colliding on each other at random, and bouncing off the walls elastically. The parallelization of such a system can be achieved via a domain decomposition of the volume inside the tube; the trajectories of each particle in a sub-domain are calculated by a given processor. Each time a particle crosses from a sub-domain to an adjacent one, a structure must be exchanged between two processors.
This section gives snippets of an $N$-body code illustrating how to exchange structures between different processors. This is done via two groups of MPI routines, the first one to build a user defined MPI data type, and the second to pack/unpack data in buffers.
User Defined MPI Data Structures
The concept is to build MPI data structures that correspond to arbitrary groupings of MPI pre-defined or user defined data types. The MPI data structures corresponding to C structures is therefore built by simply assembling the members of the MPI data structure based on the C structure members. This is somewhat a complex process which becomes worthwhile only if the C structure need to be exchanged between processors many times during the execution of a code.
The idea is to define each member of the MPI structure via its MPI
type, address, and the number of items in the member. The MPI routines
MPI_Type_struct
and MPI_Type_commit
respectively build and commit the user defined MPI
structure. MPI_Address allows to find the MPI address of any C
variable. The MPI address of the members of a C structure are best
calculated in relative form.
The basic MPI routines that allow this process are:
- MPI_Address
- Calculate the MPI absolute address of a C variable, including of
the members of a C structure. The address is with respect to
MPI_BOTTOM
. - MPI_Type_struct
- Build the user defined structure.
- MPI_Type_commit
- Commit the user defined structure.
These routines are communicator independent.
Packing/Unpacking Data
Data must be packed to be sent from node to node in a buffer, and later unpacked from the received buffer into its components. This packing/unpacking avoids latency time in the communication process. It might be considered an alternate way to building a user defined MPI structure. It is particularly useful when the number of items to send varies from transmission to transmission during the execution of a code.
The basic MPI routines that allow this process are:
- MPI_Pack_size
- Calculate an upper estimate of the buffer size.
- MPI_Pack
- Pack data in a buffer.
- MPI_Unpack
- Unpack data from the buffer (must be done in the same order as when packed).
The MPI_Ssend
, MPI_Recv
,
and MPI_Bcast
routines must be called with
the MPI_PACKED
argument in this case. Note that the
packing/unpacking routines are communicator dependent.
N-body Code Snippet
The code structure.c illustrates both
the user defined MPI data types and the packing/unpacking of data in
buffers. It is based on an $N$-body code. Node 0 first initializes few
(N_bodies
) particles to random positions, velocities, and
masses. It initializes the charges to $+1$ and $-1$ (think of a plasma
of H+ and Cl- with +/- the charge of an
electron). The code then builds an MPI structure corresponding to
this particle definition and uses it to exchange a single particle
with node 1. It then packs all the initial values of the bodies in a
buffer and send these to all the nodes. The code has extra comments
to facilitate reading.