Download Platform MPI User's Guide - Platform Cluster Manager

Understanding Platform MPI
Scalability
Interconnect support of MPI-2 functionality
Platform MPI has been tested on InfiniBand clusters with as many as 2048 ranks using the VAPI protocol.
Most Platform MPI features function in a scalable manner. However, the following are still subject to
significant resource growth as the job size grows.
Table 17: Scalability
Feature
Affected Interconnect/
Protocol
Scalability Impact
spawn
All
Forces use of pairwise socket connections between all
mpid's (typically one mpid per machine).
one-sided shared lock/
unlock
All except VAPI and IBV
Only VAPI and IBV provide low-level calls to efficiently
implement shared lock/unlock. All other interconnects
require mpid's to satisfy this feature.
one-sided exclusive
lock/unlock
All except VAPI, IBV, and
Elan
VAPI, IBV, and Elan provide low-level calls that allow
Platform MPI to efficiently implement exclusive lock/
unlock. All other interconnects require mpid's to satisfy this
feature.
one-sided other
TCP/IP
All interconnects other than TCP/IP allow Platform MPI to
efficiently implement the remainder of the one-sided
functionality. Only when using TCP/IP are mpid's required
to satisfy this feature.
Resource usage of TCP/IP communication
Platform MPI has been tested on large Linux TCP/IP clusters with as many as 2048 ranks. Because each
Platform MPI rank creates a socket connection to each other remote rank, the number of socket
descriptors required increases with the number of ranks. On many Linux systems, this requires increasing
the operating system limit on per-process and system-wide file descriptors.
The number of sockets used by Platform MPI can be reduced on some systems at the cost of performance
by using daemon communication. In this case, the processes on a host use shared memory to send
messages to and receive messages from the daemon. The daemon, in turn, uses a socket connection to
communicate with daemons on other hosts. Using this option, the maximum number of sockets opened
by any Platform MPI process grows with the number of hosts used by the MPI job rather than the number
of total ranks.
Platform MPI User's Guide 145