MP net as Abstract Model of Communication for Message-passing Applications

1 Introduction

MP net (Message Passing net) is a formal model specifically designed to capture the communication within MPI¹¹1https://www.mpi-forum.org (Message Passing Interface) applications. MPI is widely used in the field of HPC (high-performance computing) as a means of communication between nodes of a computer cluster. Generally speaking, programming parallel application is significantly harder in comparison to the sequential ones, and the new element which makes it more difficult is the involved communication. Therefore, a good understanding of communication is important.

One could ask why care about the communication, the authors know exactly how they have written the application, don’t they? Everyone who has ever written an application knows how easy it is to make a mistake. The code of MPI applications actually consists of several parts. Such a program is then distributed among the nodes of a computer cluster where each node can perform a different part of the program. In other words, MPI application describes a set of programs that are spatially distributed, but recorded in a linear form (linear stream of code). This is one of the aspects which makes programming of distributed applications more difficult. Moreover, communication involved in such kind of programs gives rise to a new sort of problems and errors, and programmers have to bear them in mind. Therefore, it can be very useful to have a tool that provides a different view of the MPI applications, the view that stresses their spatial character.

The aim is to separate the communication and computational layers of the MPI code with a focus on the communication part and its presentation via the MP net. In this way, to provide the users²²2By users we mean the application developers. with a different kind of view of their program that can help them to confront the actual model of communication with their mental image of it.

As the MP net is going to be presented back to users, one of the main requirements is that the model has to have a “nice” visualization. However, once such a model is available it can be used for other supportive activities than just the visualization (see Figure 1). Nevertheless, the first step is to define the abstract model of communication (MP net) and how it is extracted from the code of MPI applications.

Figure 1: “Big picture” with the highlighted part considering this paper.

This paper describes and formalizes MP nets. The need of a different view of the communication is demonstrated on a very simple motivational example. The tool which will implement the presented ideas is currently under development.

The text is organized as follows. At the beginning, the state of the art is summarized. Then a motivational example is presented. It shows that a different kind of view of the communication can be useful even for a very simple program. The following section introduces and formalizes MP nets. After that we return to the motivational example and generate MP nets for it. The conclusion sums up the article and discusses the possible utilization of MP nets.

2 Related work

The motivation and original idea for the work presented here comes from the work on the tool Kaira [3, 4, 2]. Kaira is a development environment for applications using MPI. It strictly separates the communication and computational layers of the application. The former one is specified via a visual programming language while the latter one is written in C++. From the combination of these two inputs the MPI code of the resulting application is eventually generated. Despite many nice features the Kaira is provided with, it has not been used by people other than those from the developer’s group. Yet the idea of using a visual form for describing communication seems to be right because it emphasizes the spatial character of the communication. However, it turned out that using this visual form as an input which is provided and created by programmers, is similarly demanding as writing the application using MPI.

The current work aims to reverse the process and instead of modeling the communication via a visual language, the goal is to provide an abstract model of communication for an existing code of an MPI application. In a fully automatic way without need of any modification in the analyzed program. Hence, it shifts the paradigm from the field of visual programming to program visualization. In this way, the programmer is a reader of such model and not its creator.

The genre of visual programming has been defined by Grafton and Ichikawa [7] and includes three distinct areas: graphics techniques that provide both static and dynamic multidimensional views of software, graphics-based high-level programming languages, and animation of algorithms and software [16]. Their work was later refined by Myers [14]. He strongly distinguished visual programming, program visualization, and other high-level programming paradigms. Moreover, he also provided two taxonomies for the classification of programming systems, one for visual programming and the other for program visualization. A nice overview of possible program visualization is summarized in [12]. A brief overview of these categories can be seen in Figure 2. Each category is filled with examples. As can be seen, the program visualization is divided into four quadrants according to whether the analysis is done on code or data and whether a static or dynamic aspect is investigated. MP nets fall into the static-code analysis.

Figure 2: Relationship between visual programming and program visualization with taxonomy within program visualization.

Since this work mainly considers HPC applications, let us look at the supportive tools in this field. There is a mature category of profiler and trace analyzer tools. As the representatives of these can be named: TAU [15], Scalasca [6], Vampire ³³3https://vampir.eu/ and the environment Score-P [9]. These performance measurement tools can however be identified as data-static in Myers taxonomy as the analyses are done during the run time of the application and the result is usually a kind of graph. As an example of a parallel debugger ARM DDT⁴⁴4https://www.arm.com/products/development-tools/server-and-hpc/forge/ddt can be named.

As for the static-code analyzers whose results are a kind of visualization presented back to the users, tools such as Sourcetrail⁵⁵5https://www.sourcetrail.com/ or Code Map⁶⁶6https://docs.microsoft.com/en-us/visualstudio/modeling/map-dependencies-across-your-solutions?view=vs-2017 can be named. These sort of tools are called code explorers and they help users with orientation in large projects by inspecting the complicated relationships in the application. The tool that also belongs to this category and concerns MPI is MPI-CHECK[13]. MPI-CHECK, however, checks the correct usage of MPI calls.

Most of the currently available tools that provide a view on the communication are based on space-time graphs [11]. These tools collect the data during the run of an analyzed program and provide a visualization based on it. As far as the author is aware, a tool that would transform the code of an MPI application into a more comprehensible (visual) form does not exists, yet.

2.1 Petri nets

A Petri net is a well-established mathematical modeling language for the descriptions of distributed systems. A basic form of Petri nets is called Place/Transition net (P/T net or PTN). A PTN is a 6-tuple $(P, T, I, O, w, m_{0})$ , where $P$ is a finite set of places, $T$ is a finite set of transitions, $P \cap T = \emptyset$ , $I$ is a set of input arcs $I \subseteq P \times T$ , $O$ is a set of output arcs $O \subseteq T \times P$ , $w$ is a weight function $w : (I \cup O) \to N$ , and $m_{0}$ is the initial marking; marking is a function $m : P \to N$ .

The behavior of such a system is defined through the changes of its state (marking). Any change of a state in a Petri net is directed by transition firing rule;

A transition $t$ is said to be enabled at marking $m$ if $\forall p \in P : m (p) \geq w (p, t)$ .
An enabled transition may fire/occur.
Firing of an enabled transition $t$ changes a marking $m$ , written $m t \to m^{'}$ , where $m^{'} = m (p) - w (p, t) + w (t, p)$ for all $p \in P$ .

One of the advantages of Petri nets is that they have established and comprehensible visualization. Figure 3 shows an example of a simple PTN with an initial marking LABEL:sub@fig:ptn-example-a, and its change LABEL:sub@fig:ptn-example-b after firing transition $t_{1}$ .

In a standard way, places are drawn as circular objects while transitions as rectangular ones. Arcs between places and transitions represent a flow relation, and numbers at arcs the weight. The weight of an arc without a number is $1$ . The current marking is determined by a distribution of tokens, black dots, among places.

An extension, called Coloured Petri nets [8] (CPNs), enriches the P/T nets with tokens with an inner structure. The places do not hold just “black” opaque tokens, but rather tokens carrying a value. The places in CPNs then do not contain only a number of tokens but multisets of values. This involves more complex conditions under which a transition is enabled and what will happen after its occurrence. Nevertheless, the core idea remains the same; input places of a transition have to contain an appropriate number of “right” tokens. One of the best-known tools based on CPNs is CPN Tools⁷⁷7http://cpntools.org.

The QPN [1, 10] (Queuing Petri nets) enriches the concept with queuing places. In comparison to CPN the data in a place are arranged in a queue instead of a multiset, hence not just presence of a token in a place is important but also its position in the queue.

3 Motivational example

The main idea and motivation of this work is presented in this section. We will look at three different implementations of one particular example. For each, we will look closely at the communication and discuss the differences. It will show that the first idea we implement might be different from the one we want. Therefore, a good understanding of communication within a parallel application is important.

Before we start with an example, let us briefly look at Figure 4. Its gray part (right-hand side) shows a “standard” workflow when developing a program. Every programmer has to think through a problem at the beginning, come up with an idea of what the main components are, and how they interact with each other or with an external environment. They have to create a mental image of the program that is going to be developed. The same holds for any programs including parallel ones.

Figure 4: Workflow with extracted abstract models.

When developing parallel programs programmers have to consider the communication done during computation. MPI often used to deal with that communication, is quite low-level. Then it is not surprising that the actual communication model of the resulting program may differ from the intended one. The difference between the intended communication model and the actual one – written in an application’s code – is the main motivation of this work. As can be seen in Figure 4

, the standard workflow is enriched by a black part – work with an actual model of communication. At the moment, the programmer can use that model and confront it with their mental image.

3.1 All-send-one example

Let us demonstrate this workflow on a very simple example. Imagine we have a task to create a parallel program that works on $n$ processes, $p_{0}, p_{1}, \dots, p_{n - 1}$ . Each process except the $0^{t h}$ one sends a message (e.g. its process ID) to $p_{0}$ and $p_{0}$ receives all these messages. Our mental image may look like the one in Figure 5.

Figure 5: Mental model of the simple task.

With this image in our mind we can make a program (Listing 1) called all-send-one, that may look as follows. We can see, there is a loop on the $0^{t h}$ process ( $r a n k = 0$ ) that receives all the messages sent by other processes. But, does the program behave (communicate) the way we expected? To answer this question, we can examine the code and state whether it does or does not. Nevertheless, it would be nice to have a tool that presents communication in the program in a more comprehensive form than just looking at the source code.

⬇

1#include "mpi.h"

3int main (int argc, char *argv[]) {

4 enum { TAG_A };

6 MPI_Init(&argc, &argv);

8 int rank, size;

9 MPI_Comm_rank(MPI_COMM_WORLD, &rank);

10 MPI_Comm_size(MPI_COMM_WORLD, &size);

12 if (rank == 0) {

13 int ns[size-1];

14 for (int i=0, src=1; src < size; i++, src++) {

15 MPI_Recv(&ns[i], 1, MPI_INT, src, TAG_A, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

16 }

17 } else {

18 MPI_Send(&rank, 1, MPI_INT, 0, TAG_A, MPI_COMM_WORLD);

19 }

21 MPI_Finalize();

22 return 0;

23}

Listing 1: First version of all-send-one program.

Let us imagine we have an application that shows us a model of communication as it really is. By applying it on the code of our all-to-one program we could get a model similar to the one in Figure 6. It is visible that there is a difference between the extracted model and our mental image; causal dependencies between individual sends. This is caused by the receiving loop (lines 14-16), because all the messages are received gradually in one specific order. At this moment, the programmer can see the difference and it is their decision whether to make some changes in the original program or not.

Figure 6: “Extracted” model of program 1.

Suppose we want to get rid of these causal dependencies. The very first thing we can notice is that the receiving operation waits for a specific source (argument src on line 15). Actually, we don’t need to wait for a specific source. Instead, we can use the MPI_ANY_SOURCE wild card; the process which sends the data as the first one, is the one which the data is received from. The modification can be seen in the following program (Listing 2).

⬇

13#include "mpi.h"

15int main (int argc, char *argv[]) {

16 enum { TAG_A };

Listing 2: all-send-one using MPI_ANY_SOURCE wild card.

The question is whether the change we have made solved the problem with the causal dependency. The answer is “yes and no”. Look at Figure 7. Although the specific order of receiving messages was removed, each receive is completed within the cycle, hence before the next receive can start, the previous receive request has to complete. Therefore, the data is still received gradually (one by one). The only change is that the data is received in a non-deterministic order.

Figure 7: “Extracted” model of program 2.

Another change we can do is to send the receive requests in a non-blocking way and then wait for all of them (Listing 3).In this way we still have to wait for all the messages, but the way how the messages are received is left on the used implementation of MPI. So, we can say, that the last version is the closest one to the original idea (Figure 5).

⬇

13#include "mpi.h"

15int main (int argc, char *argv[]) {

16 enum { TAG_A };

18 MPI_Init(&argc, &argv);

Listing 3: all-send-one using non-blocking communication to break causal dependency.

As we have seen, even in a very simple example the actual model of communication may differ from the intended one. In this particular example, the first naive implementation can be considered as a mistake. Nevertheless, it would not cause any failure. On the other hand, such mistakes can be seen as even worse in comparison to those that cause an application crash, because it can take a significant amount of time when they begin to show up. The main goal of this work is to provide a tool that helps programmers keep track of what the actual communication looks like in their programs and not to fall into a state where they think “the communication is alright”.

4 MP net

MP net, proposed in this text, is a formal model specifically designed for the field of parallel applications that use MPI. The purpose of MP net is to provide a different view of MPI applications that is focused rather on their communication part than the computational one. In other words, it exposes the spatial character of the communication compared to the linear record of standard programming languages like C/C++.

MP net is built on Coloured Petri nets [8] (CPNs) and a simplified version⁸⁸8The stochastic and timed aspect of QPN is discarded. From the perspective of MP net, only scheduling of tokens in queuing places is important. of Queuing Petri nets [1, 10] (QPNs). The goal is to propose a model that is, on one hand, powerful to fully capture the semantics⁹⁹9In the current version the semantics of point-to-point communication is mainly considered. of MPI [5], and, on the other hand, simple enough to be understood relatively easily by people. The balance between these two main requirements is the reason why a new model is proposed. However, there is no reason “to re-invent the wheel”; therefore, MP net is defined through the specification of its structure (syntax) and the behavior (semantics) is defined by a set of transformations into the simplified version of QPN.

4.1 Basic definitions

For any set $A$ , $A^{*}$ is a free monoid over this set with the operation of sequence concatenation: $\cdot : A^{*} \times A^{*} \to A^{*}$ defined as $\forall s, s^{'} \in A^{*} : s \cdot s^{'} = s^{''}$ where $s^{''} \in A^{*}$ , and zero element empty sequence: $ϵ \in A^{*}$ . For any sequence $s = [s_{1}, s_{2}, \dots, s_{n}] \in A^{*}$ , $s_{i}$ is the $i^{t h}$ element of the sequence where $i \in {x \in N | 1 \leq x \leq n} = N_{n}$ , and $| s |$ is the length of the sequence. For any $o p : A - > B$ and a sequence $s = [s_{1}, s_{2}, \dots, s_{n}] \in A^{*}$ , let us define $o p (s) := [o p (s_{1}), o p (s_{2}), \dots, o p (s_{n})] \in B^{*}$ . An application of a function, $f : A \times \prod_{P \in P} P \to B$ , on a sequences $s = [s_{1}, s_{2}, \dots, s_{n}] \in A^{*}$ is defined as $f(s):=[f(s1,p1,…,pm),f(s2,p1,…,pm),…,\allowbreakf(sn,p1,…,pm)]∈B∗$ where $m = | P |$ .

4.2 Structure of MP net

Definition 4.1

MP net (MPN) is a finite set of addressable areas, where each area is a pair of an annotated sequential program fragment and communication net:

An annotated sequential program fragment arises from an input program code. The program is split into the fragments according to conditions directed by rank value¹⁰¹⁰10The value provided by MPI_Comm_rank call.. In fact, the fragments represent subprograms performed on particular computational units (ranks). The segments transformed into a CN are highlighted. There are one or more elements within the CN for boldfaced expressions. All such expressions are followed by an annotation. The annotation comprises a label and sequence of directives. Labels help with orientation in CN. All the elements that appear in a net because of a particular annotated expression use the label as a prefix for their name. A sequence of directives set up data exchange between the program and communication net. There are three allowed directives with the following syntax:

put(place=expression)
wait(place)
get(place $\to$ variable)

Communication net is built upon a simplified version of Queuing Petri net [1, 10], defined in Definition 4.4. Transformation into the simplified version of QPN is presented in Section 4.4

Definition 4.2

A Communication Net (CN) is a triple $C N = (C P N, P_{Q}, I_{Q}, A_{C F})$ , where

$C P N = (P, T, C, I, O, M_{0})$ is the underlying Coloured Petri net.
$P_{Q} \subseteq P$ is a set of queuing places.
$I_{Q} \subseteq P_{Q} \times T$ is a set of queuing-input arcs. The set is composed of four distinguished types: $I_{Q} = I_{Q}^{▶} \cup I_{Q}^{▶ ▶} \cup I_{Q}^{▹} \cup I_{Q}^{▹ ▹}$ where $I_{Q}^{▶} \cap I_{Q}^{▶ ▶} \cap I_{Q}^{▹} \cap I_{Q}^{▹ ▹} = \emptyset$ .
$A_{C F} \subseteq X \times Y$ a set of control flow arcs where $X, Y \in {P, T}$ .

The graphical representation is similar to the one of CPNs. Unlike QPNs¹¹¹¹11QPNs depict queuing places as circles with a horizontal line splitting it into two parts which reminds us that the queuing place is composed of two parts: the queue and depository., queuing places are depicted in CNs as rounded objects with a white background, i.e., as standard places in CPNs. The multiset places are then distinguished by gray color.

There are several types of arcs. The first category forms input arcs leading from queuing places, $I_{Q}$ . This category is formed by four different types. The types are distinguished by a used arrow head which corresponds to upper indexes of particular disjoint sets. A combination of a queuing place and a particular type of input arc determines the strategy used to access data in the queue. Conditional arcs, i.e., arcs with an expression preceded by a conditional one are emphasized using dashed style. The last type of arcs represents control flow arcs, $A_{C F}$ . These help a user to orient in the communication net while the control is given back to the program. In fact, the control flow arcs just have an informative character without any effect and are removed during the transformation¹²¹²12Therefore, control flow arcs can connect the same net elements.. This type of arcs is depicted as tiny gray arrows.

4.3 Arc expressions

Let us define a set of values $V$ and a set of variables $X$ . There are three types of values that are always part of the value set; unit for a Unit value, true and false for Boolean values, and natural numbers $N$ , i.e., ${unit, true, false} \cup N \subseteq V$ . These values and objects composed of them, e.g., tuples or records are considered to be transparent. The structure of transparent object can be inspected. There is another category of values, let us call them opaque. Opaque values arise from the evaluation of expressions used in a source language, e.g., C++. They are viewed as binary objects without any inner structure.

A set of all expressions used within MP net as $E$ . Values and variables are considered to be expressions too, $V, X \subseteq E$ . To evaluate an expression to the value a function $eval : (E \times X - > V) - > V$ is defined. It takes an expression $e \in E$ and binding $b : X - > V$ that specifies a binding function assigning values to free variables of the expression. A set of free variables of an expression is obtained by the function: $F V : E - > X$ . To evaluate the entire arc expression, the function $Eval : (2^{E} \times E^{*}) \times (X - > V) - > V^{*}$ is defined.

Eval ((C, τ), b) = {\begin{matrix} ϵ & if \exists c \in C : eval (c, b) = false eval (τ, b) & otherwise \end{matrix}

Input arc expressions

are assigned via the function $i e$ , $i e : I - > (2^{E} \times (E^{*} \times (X \cup N) \times X))$ . It composes of two parts: a set of conditions and main expression. The used syntax is the following:

[conditions] (pattern, size, values)

conditions – a set of conditions $C \in 2^{E}$ .
pattern – a sequence of expressions $[e_{1}, e_{2}, \dots, e_{n}] \in E^{*}$ representing value(s) of token(s) that is taken from the corresponding place.
size – an expression specifying of how many tokens is desired. It can be a natural number or free variable.
values – a free variable for actual values.

Let us define two derived forms of the input-arc’s sub-expression, so that: $(p a t t e r n, s i z e) := (p a t t e r n, s i z e,_)$ and $pattern:=(pattern,\allowbreak|pattern|,_)$ . The former one allows the values variable to be omitted while the latter one omits also the size, in this case the size of the pattern is used.

Output arc expressions

are assigned via the function $o e$ , $o e : O - > (2^{E} \times E^{*} \times A)$ . It composes of three parts: a set of conditions, main expression, and location (addressable area). The syntax of the arc expression is:

[conditions] pattern@location

conditions – a set of conditions $C \in 2^{E}$ .
pattern – a sequence of expressions $[e_{1}, e_{2}, \dots, e_{n}] \in E^{*}$ representing value(s) of token(s) that are put into the connected place.
@location – addressable are $a \in M P N$ . It may be omitted, but in this case a default addressable is used, i.e., addressable area of the arc’s corresponding place.

4.4 Behavior of MP net

Semantics of MP net is given by a set of transformations into a simplified version of Queuing Petri net. There are three parts: a program, a communication net, and addressable areas whose transformation will be solved separately.

4.4.1 Transformation of annotated sequential program

Any program can be represented in a form of state machine where each node represents the current state of the program and directed edges the statements that change the state. The state is composed of a position in such a graph and the current state of the memory. Such a graph can be transformed into a Coloured Petri net straightforwardly. Each node in the original graph is replaced with a unit place and all directed edges with transitions connected to corresponding input/output places. For further reference, let us call these arcs control arcs. To capture the memory state, one colored place is added. Each transition is then connected with this “memory place” via an input and output arc. An input arc takes a current state of the memory and returns a modified version according to a program statement. Let us call such a resulting CPN, program net.

The directives that extend the original program are transformed in the following way. For each directive a new series of unit place and transition is always added. With this series the output control arc of the transition representing the annotated statement is replaced. If there are more directives in an annotation, the output control arc of the transition representing the preceding directive is replaced. The newly added transition is connected to the rest of the MP net as follows:

put(place=expression) – connects the transition with the memory place so that it takes the actual value and without any modification is returned back. Moreover, from the transition a new arc leads to the place specified by the put directive within the communication net, using the specified expression.
wait(place) – adds a new read-only input arc to the transition leading from the specified place. The transition then synchronizes the program net with the communication one but does not modify the content of the place.
get(place $\to$ variable) – connects the transition with the memory place in the similar way as with the put directive, but this time a new input arc from the specified place is added. This arc is a bulk arc which takes the entire content (possibly empty) of the place whenever the transition occurs. Therefore, the value of specified variable is set by the content of a specified place.

4.4.2 Transformation of the communication net

A communication net is transformed into a simplified version of Queuing Petri net. Therefore, before we start, let us rephrase the definition of Queuing Petri net and queue from [1]. The definition is simplified in comparison to the original one as it is considered neither a timed nor stochastic aspect of QPN. For the purpose of MP net, queuing places and a way of scheduling tokens in the queue need to be established.

Definition 4.3

$Q U (p)$ defines a queue as a 4-tuple $QU(p)=(ST(p),TR(p),\allowbreakAC(p),TN(p))$ where

$S T (p)$ specifies the feasible states of a queue.
$T R (p) : (S T (p) \times D M (p)) \times (S T (p) \times D M (p)) \times C (p) \to 2$ . This two-state value can be interpreted as decision procedure whether or not the token is serviced.
$A C (p) : S T (p) \times C (p) \to S T (p)$ specifying the next state of the queue after arrival of a token of color $\in C (p)$ .
$T N (p) : S T (p) \times C (p) \to N_{0}$ specifying the number of tokens in the queue.

Definition 4.4

Simplified Queuing Petri net is defined as a pair $S Q P N = (C P N, Q)$ where

$C P N$ is the underlying Coloured Petri net,
$Q = (¯ Q, (q_{1}, \dots, q_{| P |}))$ where
- $¯ Q \subseteq P$ is the set queuing places,
- $q_{i} = {\begin{matrix} Q U (p_{i}) & if p_{i} \in ¯ Q ⊥ & if p_{i} \in P ∖ ¯ Q \end{matrix}$

By defining $q_{i} = ⊥$ for $p_{i} \in P ∖ ¯ Q$ is meant that there is no scheduling strategy for this kind of places. In other words, arriving tokens are put directly into the depository. In QPN each place is composed of a queue and depository.

Tokens located in a queue are not accessible for the net. In order to make them accessible, tokens need to get into the depository from the queue. $D M (p)$ specifies the current marking of the depository. $D M (p) := [C (p) \to N_{0}]$ is the set of all functions $f : C (p) \to N_{0}$ . The action of moving a token from a queue to the depository is called service of a token and it is specified via the $T R (p)$ function.

Besides the transition enableness and occurrence which determine the dynamic aspect of petri-net like systems, QPN defines a phase of scheduling in queuing places. During this phase, the tokens that are ready to serve are moved from the queue to the depository. The check whether or not a specific token is ready is done via $T R (p)$ function, $p \in P$ . All tokens for which $T R (p) ((s, d), (s^{'}, d^{'}), v) > 0$ can be served, where $(s, d)$ is the state of the place’s queue and depository and $(s^{'}, d^{'})$ is the state of the place’s queue and depository after serving token $v$ .

Communication net to SQPN

– Queuing places of CN corresponds to the queuing places of SQPN, $¯ Q = P_{Q}$ . The definition of queue types ( $q_{i}$ ), i.e., definitions of feasible states ( $S T$ ), value arriving function ( $A C$ ), and token number function ( $T N$ ) for queue places look as follows: $\forall q \in P_{Q} :$

$S T (q) = C (q)^{*}$
$A C (q) (s, v) = s \cdot v$ (concatenation)
$T N (q) (s, v) = ⎧ ⎪ ⎨ ⎪ ⎩ \begin{matrix} 0 & i f | s | = 0 1 + T N (q) (s_{1}, v) & i f s = v \cdot s_{1} T N (q) (s_{1}, v) & i f s = v^{'} \cdot s_{1}, v \neq v^{'} \end{matrix}$

The definition of $T R$ depends on the type of queuing input arc. There are four distinguished types: $I_{Q}^{▶}$ , $I_{Q}^{▶ ▶}$ , $I_{Q}^{▹}$ , and $I_{Q}^{▹ ▹}$ . These can be categorized into two groups: single-headed ( $I_{Q}^{▶} \cup I_{Q}^{▹}$ ) and double-headed ( $I_{Q}^{▶ ▶} \cup I_{Q}^{▹ ▹}$ ) arcs.

$∀(q,t)∈I▶▶Q∪I▹▹Q:TR(q)((s,d),(s′,d′),v)= {1if \parbox[t]346.896pt$d′=d+δv,d(v)=0$and\lx@parboxnewline$∃v∈C(q),s1,s2,∈C(q)∗:s=s1⋅v⋅s2$and$s′=s1⋅s2$,\lx@parboxnewlines.t.$∄s11,s12∈C(p)∗:s1=s11⋅v⋅s12$0otherwise$

where $δ_{v}$ denotes the Kronecker function. Figure 8 shows a simple example demonstrating the queue behaviour using single-headed and doubles-headed arcs.

(a)

(b)

Figure 8: LABEL:sub@fig:single-headed-case Single-headed arcs match only values on the head of queue (marked by angle bracket); LABEL:sub@fig:double-headed-case double-headed arcs can “look further” in the queue (the transition is enabled with binding:

x \mapsto 3

The empty-headed arcs ( $I_{Q}^{▹}$ and $I_{Q}^{▹ ▹}$ ) represent read-only arcs. These arcs do not modify the content of the depository place. In other words, there is a back arc returning the same value to the depository.

4.4.3 Transformation of the MP net

MP net is defined in Definition 4.1 as a finite set of addressable areas. At the first sight, these areas represent separate nets. Nevertheless, addressable areas are interconnected via either the output arcs with a location¹³¹³13A different location or compound places. Compound places represent shared memory in the entire system; therefore, places with the same compound place label refer to the one and the same place no matter of its location. This situation is illustrated in Figure 9. The arcs with a specified location do not put the token(s) into the directly connected output place, but into the corresponding place into the particular addressable area instead.

Generally, the arcs with a specified location are converted as follows. For each arc using ‘@’-sign there is a conditional arc leading from the transition to all corresponding places in different addressable areas. The expression behind the ‘@’ is moved to the condition and checked against the address of a particular output place. One big SQPN capturing the behavior of the MP net is composed in this way. This situation is depicted in Figure 10. The used function pic_address pick an address from the specified range. The upper indexes of element names (Figure (b)b) refer to the original addresses.

5 Motivational example in MP nets

Figure 11: MP net for all-to-one program – $r a n k \neq 0$ .

Let us look at the MP net for all-to-one program. It consists of two parts. The first one (see Figure 11) represents all the processes that send its rank to process zero. It can be seen, that the MPI_Send call actually consists of two parts. Sending the request and waiting for its completion (green and blue part of the picture). There are two special places: Active Receive Requests and Completed Receive Requests. The former one represents a queue of requests that is visible for the MPI message broker. It is labeled by compound place label ASR, i.e., all such places form different views of the same place. The latter place then represents a local place where the message broker sends the information about completion of the request.

The second part of MP net (see Figure (a)a) represents the 0 process that receives all the messages. It is visible that both sending the receive request and waiting for its completion are done within the cycle. For the 2 version (see Listing 2) the block of MP net look almost the same, but the source entry in the request would be unset, i.e. the request is less restricted.

The second part of MP net (see Figure (b)b) for the 3 version (cf. Listing 3) of the program shows that all the receive requests are sent first (green part) while the waiting for the completion is done out of the cycle.

The message broker mentioned at the beginning is also a part of MP net and can be seen in Figure 13. Nevertheless, it is the same for all the applications. There is one transition which controls the Active Send/Receive Requests places and pairs the requests according to the source, destination, and tag entries. Once these match, completed requests are generated and sent back (the address specified by the expression behind ‘@’ sign) to appropriate blocks, into their Completed Send/Receive Requests places.

Figure 13: Generic message broker expressed as a part of MP net.

6 Conclusion

The main goal of this work is to provide programmers of MPI applications with a kind of feedback that can help them to quickly check whether or not the actual communication within their application corresponds to the intended one. For this purpose MP net is established as a model that should be relatively easily understood by people and which is capable of capturing the general semantics of MPI. At the current state, only point-to-point communication is covered. Nevertheless, the proposed approach does not disable to cover other, more advanced, routines from MPI. This may be one of a possible future work. However, the near future is devoted to the implementation of the presented approach into a working tool.

One could object that the model will increase with the size of the input program. That is not necessarily true, because the size is directly related to the number of MPI calls used within the program. Moreover, no matter of the program complexity MP nets can be still used locally to identify local issues.

Another objection could be that the model is still difficult to read. But the author is convinced that with a little practice it will be manageable. There is definitely a learning curve, but in comparison to visual programming tools, users just have to learn “to read” and can benefit from it. Moreover, it may be even more interesting to use this model further. For example, once it is available, performance data can be projected into the model, like coloring the most used transitions or stress the paths that are used most often during the real run. It can also be used for simulation of the application run and, for example, stop the program in a situation when two transitions are enabled to see the source of non-determinism.

As said, the main idea is to provide a model that present the communication in MPI application in a more comprehensive way. Nevertheless, once the model is available its potential is much wider than being used only as a static picture (cf. Figure 1).

References

[1] Bause, F. Queueing petri nets-a formalism for the combined qualitative and quantitative analysis of systems. In Proceedings of 5th International Workshop on Petri Nets and Performance Models (Oct 1993), pp. 14–23.
[2] Böhm, S. Unifying Framework For Development of Message-Passing Applications. PhD thesis, FEI VŠB-TUO Ostrava, 17. listopadu 15, Ostrava, 11 2013. http://verif.cs.vsb.cz/sb/thesis.pdf.
[3] Böhm, S., Běhálek, M., Meca, O., and Šurkovský, M. Application and Theory of Petri Nets and Concurrency: 35th International Conference, PETRI NETS 2014, Tunis, Tunisia, June 23-27, 2014. Springer International Publishing, Cham, 2014, ch. Kaira: Development Environment for MPI Applications, pp. 385–394.
[4] Böhm, S., Běhálek, M., Meca, O., and Šurkovský, M. Visual programming of mpi applications: Debugging, performance analysis, and performance prediction. Computer Science and Information Systems 11 (2014), 1315–1336. IF: 0.477 (2014).
[5] Forum, M. P. Mpi: A message-passing interface standard. Tech. rep., Knoxville, TN, USA, 2015. Version 3.1.
[6] Geimer, M., and et al. Scalable collation and presentation of call-path profile data with CUBE. In Proc. of the Conference on Parallel Computing (ParCo), Aachen/Jülich, Germany (September 2007), pp. 645–652. Minisymposium Scalability and Usability of HPC Programming Tools.
[7] Grafton, R., and Ichikawa, T. Guest Editor’s introduction to the Special Issue on Visual Programming 18, 8 (1985), 6–94.
[8] Jensen, K., and Kristensen, L. M. Coloured Petri Nets - Modelling and Validation of Concurrent Systems. Springer, 2009.
[9] Knüpfer, A., and et al. Score-p: A joint performance measurement run-time infrastructure for periscope,scalasca, tau, and vampir. In Tools for High Performance Computing 2011 (Berlin, Heidelberg, 2012), Springer Berlin Heidelberg, pp. 79–91.
[10] Kounev, S., Spinner, S., and Meier, P. Introduction to queueing petri nets: Modeling formalism, tool support and case studies. In Proceedings of the 3rd ACM/SPEC International Conference on Performance Engineering (New York, NY, USA, 2012), ICPE ’12, ACM, pp. 9–18.
[11] Lamport, L. Time, clocks, and the ordering of events in a distributed system. Commun. ACM 21, 7 (July 1978), 558–565.
[12] Lemieux, F., and Salois, M. Visualization techniques for program comprehensiona literature review. In Proceedings of the 2006 Conference on New Trends in Software Methodologies, Tools and Techniques: Proceedings of the Fifth SoMeT_06 (Amsterdam, The Netherlands, The Netherlands, 2006), IOS Press, pp. 22–47.
[13] Luecke, G., Chen, H., Coyle, J., Hoekstra, J., Kraeva, M., and Zou, Y. Mpi-check: a tool for checking fortran 90 mpi programs. Concurrency and Computation: Practice and Experience 15 (02 2003), 93–100.
[14] Myers, B. A. Visual programming, programming by example, and program visualization: A taxonomy. SIGCHI Bull. 17, 4 (Apr. 1986), 59–66.
[15] Shende, S. S., and Malony, A. D. The tau parallel performance system. Int. J. High Perform. Comput. Appl. 20, 2 (May 2006), 287–311.
[16] Taylor, R. P., Cunniff, N., and Uchiyama, M. Learning, research, and the graphical representation of programming. In Proceedings of 1986 ACM Fall Joint Computer Conference (Los Alamitos, CA, USA, 1986), ACM ’86, IEEE Computer Society Press, pp. 56–63.

MP net as Abstract Model of Communication for Message-passing Applications

Authors

Poster on MPI application in Computational Fluid Dynamics

Communication Dualism in Distributed Systems with Petri Net Interpretation

Textual Paralanguage and its Implications for Marketing Communications

Towards deductive verification of MPI programs against session types

Parallel Multiphysics Simulation for the Stabilized Optimal Transportation Meshfree (OTM)

Efficient Generation of Mandelbrot Set using Message Passing Interface

Optimised allgatherv, reduce_scatter and allreduce communication in message-passing systems

1 Introduction