Download CuPit-2 | A Parallel Language for Neural Algorithms: Language

6.2 Properties of a problem-adequate solution
15
6.2.2 Description of the algorithm
For the program description of the actual algorithm to reect the distinction between network object
operations and global operations (gure 2), it is best to attach the network object operations to the object
types they work on. For example the weight multiply operation that computes a connection's output from
its input is best given as a procedure or function that is declared in the context of the connection type
denition. This approach is in the spirit of object oriented languages, where each method declaration
is attached to a class declaration. Using the same approach, the unit activation computation operation
that computes a unit's output by applying the activation function to the unit's input will be declared in
the context of the type declaration of the unit type and so on.
Connection procedures are only allowed to access the local data elements of the connection they are
applied to (reading and writing). Unit procedures can access the local data elements of the unit they are
applied to and can also call the connection procedures of the connections that connect into or out of the
unit.
When the input of a node must be computed, though, these procedures do not suce: A collection of
values (one from each connection that goes into the node) has to be combined into a single value; this
single value must then be delivered to the unit as an input value. This problem is most elegantly solved by
declaring a reduction function . A reduction function combines two values into one by applying a binary
operator to them. In the case of backpropagation, this operator is simply addition: Reduce(a; b) =
a + b. To reduce more than two values, this operator can be applied recursively if it is associative, e.g.
Reduce(a; b; c) =Reduce(Reduce(a; b); c).
To compute the input of a unit, we can dene that it is allowed to call a reduction function for values
residing in the whole set of connections ending at that unit and that such a call shall mean the recursive
application of the reduction function to all the connection's values resulting in the unit input value.
Global operations are simply global procedures just like in any other programming language that has a
procedure concept.
6.2.3 Description of parallelism
The kinds of parallelism in neural network learning algorithms can best be divided according to the
second level of gure 1 into
1. parallelism of operations on connections,
2. parallelism of operations on units or concerned with units, and
3. parallelism of examples.
Parallelism of examples (\example independence" in gure 1) can be described transparently: If the
network is modeled as an explicit entity, we can couple operations to this network object. Such operations
can then be applied to a single instance of this network or to several instances at once without any change.
We might then have a statement that says \Now take this network and make several copies of it. Whenever
I call a network operation, apply it to all of the copies in parallel". All that is left to do then, is to fetch
several examples at once whenever a new example is needed. After each epoch, all the copies of the
network have to be united to a single network again, in order to make the proper weight change. This
recombination can be described by a call to a procedure that merges two networks into one (another kind
of reduction operation). This call has to apply the merge function recursively to all the copies of the
network in order to reduce the whole set of copies to a single copy. In the case of backpropagation, all
this merge procedure has to do is to sum the deltas at each weight. Since the merging process is a local
process at each unit and at each weight, we can declare the merge procedure for the network as a whole
implicitly by declaring a merge procedure for each connection type and for each unit type.
Parallelism of operations on units (e.g. \activation" in gure 1) is best described by coupling procedures
to each unit type as described in the previous section. Given such unit procedures, we can operate on
units in parallel by just calling such a unit procedure not only for a single unit, but for a whole group of
units at once. Thus, the same feature of the language serves two important abstraction purposes at once.