Download Scali System Guide

7.4 Routing
7.4.3 3D torus topology
Three dimensional torus topology is detected by the topology TOP_TORUS_3D line in the
configuration file /opt/scali/etc/ScaConf.conf. In a 3D torus each node is connected
to three SCI ringlets, one in each dimension of the torus. The following routing types
are available in a 3D torus:
•
•
Dimensional routing
C3 (Close Commutative Connection) routing
7.4.3.1 Dimensional routing in a 3D torus
Dimensional routing is available for a 3D torus, but it is less fault tolerant than C3
routing. When using dimensional routing, packets are routed along link 0 first, then
along link 1 and then along link 2. You may specify another order of dimensions as a
parameter to the reroute command. The command
ConfTool> reroute dimensional order=210
will route link 2 first, then link 1 and link 0 last. If a node or a ring fails, the number
of nodes in the main routing partition (partition number = 1, see 7.4.4) depends on the
order of the dimensions in the routing algorithm. Other nodes, not part of the main
partition will not have SCI connectivity to nodes in the main partition. These node may
route other SCI-traffic, though.
ClusterEdge 7.4.3.2 C3 routing in a 3D torus
The C3 algorithm is a fault tolerant routing algorithm available for systems with a C3
routing license. The C3 algorithm is capable of maintaining full connectivity between
the remaining nodes even if more than one node have failed.
7.4.4 Routing partitions
A routing partition (partition from here) is a collection of nodes which has a route to
all other nodes in the same partition via the SCI network. Hence all nodes in the same
partition can communicate via the SCI interconnect. Nodes not part of the same
partition does not have a route between them, and can not communicate via the SCInetwork. Partitions are reported as a positive integer called partition number for each
node. If a negative number or 0 is reported, this has a special meaning:
0
The node is unreachable.(Power off or considered dead in some sense). No MPI
programs or anything else may run on nodes with partition number 0.
-1
Partition number for this node is not known (initial state). Before routing is
set, all nodes will have this partition number.
Scali System Guide Version 3.0
109