Makespan minimization of Time-Triggered traffic on a TTEthernet network

06/20/2020 ∙ by Jan Dvořák, et al. ∙ Czech Technical University in Prague 0

The reliability of the increasing number of modern applications and systems strongly depends on interconnecting technology. Complex systems which usually need to exchange, among other things, multimedia data together with safety-related information, as in the automotive or avionic industry, for example, make demands on both the high bandwidth and the deterministic behavior of the communication. TTEthernet is a protocol that has been developed to face these requirements while providing the generous bandwidth of Ethernet up to 1 Gbit/s and enhancing its determinism by the Time-Triggered message transmission which follows the predetermined schedule. Therefore, synthesizing a good schedule which meets all the real-time requirements is essential for the performance of the whole system. In this paper, we study the concept of creating the communication schedules for the Time-Triggered traffic while minimizing its makespan. The aim is to maximize the uninterrupted gap for remaining traffic classes in each integration cycle. The provided scheduling algorithm, based on the Resource-Constrained Project Scheduling Problem formulation and the load balancing heuristic, obtains near-optimal (within 15% of non-tight lower bound) solutions in 5 minutes even for industrial sized instances. The universality of the provided method allows easily modify or extend the problem statement according to particular industrial demands. Finally, the studied concept of makespan minimization is justified through the concept of scheduling with porosity according to the worst-case delay analysis of Event-Triggered traffic.



There are no comments yet.


This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

This paper focuses on the problem of creating schedules for Time-Triggered traffic on the TTEthernet network. The objective is to minimize the makespan of the Time-Triggered traffic and, thus, maximize the continuous gap for the Event-Triggered traffic. The aim of the paper is to develop the scheduling algorithm and, consequently, to compare the concept of the makespan minimization with the concept of the porosity optimization proposed by Steiner in [28]. The TTEthernet network is a network protocol which offers Time-Triggered communication and preserves the backward compatibility with Ethernet.

Ethernet (IEEE 802.3) has been the prevalent technology for home and office networks over the last few decades. Thanks to its widespread adoption, it has developed into a mature technology offering high bandwidth with cheap and readily available hardware.

Figure 1: Example of the TTEthernet network infrastructure with routing and scheduling of message from node l to node q

Conventional computer networks are mainly used for on-demand data transfer without an immediate physical impact on the real world. In case of failure, the transmission can usually be repeated without causing major difficulties. Therefore, the focus is on the bandwidth and efficiency with only moderate demands on reliability.

In contrast, in industrial applications various control loops of physical devices are realized by the network, e.g. data from sensors are transferred to a processing unit which then sends commands to actuators. Any disturbance can, thus, have immediate effects on the real world with possible severe consequences. As jitter (variance in transmission times) is detrimental to the function of the control loops, determinism is often required. In addition, individual devices in the network are often limited in hardware and need to operate in demanding environments.

Due to these differences, different technologies and protocols were traditionally used for conventional computer networks and industrial networks. Ethernet has been used for home and office networks while various Fieldbus networks (such as CAN [10] or Profinet [14]) have been used for industrial applications. However, with the increasing integration of industrial systems, increasing demands on the volume of data transferred and also the maturing and development of the Ethernet, there has been a trend of using Ethernet-based networks for industrial applications as well. This trend has become even more pronounced with the ever increasing amounts of data transfers necessary to facilitate features like real-time image processing and recognition or communication among individual units in a smart system. Therefore, extensions of Ethernet are being developed to meet the demands of industrial applications. An overview of the development of industrial networks is given by [13].

TTEthernet is a promising extension of Ethernet, which provides determinism and fault-tolerance while being compatible with standard Ethernet. Besides the TTEthernet standard, there has been ongoing effort on the standardization of extensions to Ethernet for scheduled traffic by the IEEE 802.1 Time-Sensitive Networking Task Group.

Determinism with the strictest guarantees is achieved through a fixed schedule for the traffic. Therefore, synthesizing a good (exactly what this means will be discussed later) schedule which meets all the requirements and deadlines is essential for the performance of the network. Because TTEthernet allows for complex topologies, the scheduling involves additional complexity compared to bus or passive star topologies of networks like FlexRay [17] or CAN.

1.1 TTEthernet Overview

TTEthernet (TT stands for Time-Triggered) is an extension of Ethernet for deterministic communication developed as joint project among the Vienna University of Technology [19], TTTech and Honeywell, and standardized as SAE AS 6802 [22] in 2011.

TTEthernet operates at Level 2 of the ISO/OSI model, above the physical layer of Ethernet. It requires a switched network with fully duplex physical links, such as Fast Ethernet physical link 100BASE-TX or Automotive Ethernet standard 100BASE-T1, so that unpredictable conflicts, while accessing a shared medium, are avoided. An example of the TTEthernet infrastructure is depicted in Fig. 1.

TTEthernet specifies a protocol for clock synchronization and the rules for managing the traffic on the network. After an initial startup phase when the clocks of the devices in the network are synchronized for the first time, the operation of TTEthernet is periodic. The clocks are being periodically synchronized to counter any possible clock drift when in steady operation. This period is called the integration cycle. The messages which follow a deterministic schedule are also periodic. The least common multiple of their period is called the cluster cycle.

TTEthernet integrates traffic of different time-criticality levels into one physical network. There are three traffic classes in TTEthernet. These classes, ordered by decreasing priority, are Time-Triggered (TT), Rate-Constrained (RC) and Best-Effort (BE) traffic.

Figure 2: Example of communication on a link in one cluster cycle

The TT traffic class has the highest priority, and sub-s jitter can be achieved (depending on the network devices). The TT messages are periodic. We assume that they are strictly periodic (i.e. no jitter is allowed) in agreement with [29]. Their schedule is calculated offline and then loaded into the individual devices. The schedule is repetitive with a hyperperiod of the cluster cycle.

A simple example of the TT traffic together with the RC traffic on one direction of a physical link is presented in Fig. 2. In the figure, the integration cycles are situated in rows and x-axis represents the time instants in the particular integration cycle. The figure shows that messages , and have the same period (twice the integration cycle) and has another period (four times the integration cycle). The length of the cluster cycle is equal to four times the length of the integration cycle. The dark message at the beginning of each integration cycle is the synchronization message. The synchronization message is used for periodical clock synchronization among nodes. This message represents the time allocated for the synchronization mechanism of the TTEthernet and it does not participate in the optimization process.

The schedule also provides temporal isolation and enables fault tolerance. This is due to the fact that not only the sending of a frame is scheduled, but its reception is scheduled as well. If a TT frame arrives outside the acceptance window (i.e. the time the frame is supposed to arrive considering the synchronization inaccuracy), it is discarded by the receiver. This mechanism is called the temporal firewall.

For traffic with less strict precision requirements, the RC traffic class can be used. This traffic class conforms to the ARINC 664p7 specification [2] (also called AFDX). It offers greater flexibility because only the frame routing needs to be determined offline. The messages themselves are event-driven within some limitations.

The RC traffic represents the event-triggered communication. Thus, it does not follow any schedule known in advance. It is organized in so-called virtual links. As stated in [6], a virtual link is an analogy to the ARINC 429 [3] single-source multi-drop bus. The virtual link determines the routing of the messages associated with it. Furthermore, there are two parameters: the maximum allowed frame size and the bandwidth allocation gap, associated with each virtual link. The bandwidth allocation gap represents the minimum allowed length of an interval between consecutive frames on the virtual link. This effectively limits the bandwidth of the virtual link. In return for this limitation, the maximum possible delay of any RC message can be calculated offline.

Standard Ethernet traffic can also be transmitted through the network. Even standard Ethernet devices, unaware of TTEthernet, can communicate through the network. Such traffic is called the Best-Effort (BE) traffic and has the lowest priority.

When the TT traffic is used together with other traffic classes, a TT frame could be delayed by another RC or BE frame. This happens when a TT frame arrives while an RC or BE frame is in transmission. Then, unless the transmission of the RC or BE frame can be interrupted, the TT frame has to wait until the transmission is finished. Therefore, a method of handling such situations, called the traffic integration policy, is needed. There are three policies for the integration of different traffic classes as mentioned by [30]: Pre-emption, Shuffling and Timely block. The Timely block integration policy, which causes no extra delay of the TT traffic, is used in this paper. In this case, an RC frame can only be transmitted if there is enough time for the transmission of the entire frame before the next TT frame is scheduled. If there is insufficient time, the transmission of the RC frame is postponed until after the TT frame is transmitted. It additionally means, that the TT traffic follows the schedule without any delays.

1.2 Related works

Many recent publications focus on the scheduling of Time-Triggered communication on the Ethernet network. The most significant research in this area was done by Steiner et al. Their first paper [27] described the basic constraints for scheduling the TT traffic. The scheduling problem was solved by the reformulation of the constraints to the SMT (Satisfiability modulo theories) model. The native SMT formulation was used just for small cases with up to 100 message instances. The second approach presented allows one to find a schedule for bigger instances. It uses the SMT solver as a backend for scheduling smaller groups of frames, into which the frames are divided. After a group of frames is scheduled, the positions of the corresponding frame instances are fixed, and the next group is scheduled iteratively. The paper does not consider other traffic classes, and the only criterion is to find any feasible schedule. The SMT formulation of the Time-Trigged message scheduling problem has been adapted to 802.1Qbv [8] in 2016. In [28], which builds on and extends [27], traffic from other classes than just the TT class is considered. To reserve capacity for rate-constrained traffic, the concept of schedule porosity, i.e. inserting blank slots reserved for the RC traffic into the schedule, is introduced. The pessimistic time analysis for the RC traffic is proposed to evaluate the concept. Tamas-Selicean et al. published a new method for calculating the worst-case delay [29]

which promises much tighter estimates compared to the previous work. The improved precision comes, however, at the cost of much more expensive computation. As noted by 

[26], porosity scheduling has the disadvantage that gaps introduced at the beginning of the scheduling process do not consider the profile of the RC traffic.

The idea of using the metaheuristic approach for scheduling the TTEthernet communication by TabuSearch was proposed in [30] and further extended and more intensely evaluated in [31]. The operators of the TabuSearch [4] algorithm take the RC worst-case delay calculation into account and try to change the current schedule in such a way that all the real-time constraints imposed on the RC traffic are satisfied.

Craciunas et al. took the given TTEthernet communication schedule into account while scheduling the tasks on the communication endpoints in [7]. They extended the work and introduced the combined network-level and task-level scheduling in [9]. The CPU tasks are modeled in a very similar way to the network transmission tasks. A CPU is modeled as another physical link in the network. A task running on this CPU is modeled as a frame which needs to be transmitted over this physical link in one direction, and its transmission time is equal to the required CPU processing time.

We have formulated our problem as the Resource Constrained Project Scheduling Problem (RCPSP [20]), as will be described later, and its Multi-Mode version (MMRCPSP) [16]. RCPSP is already a well-studied optimization problem. The survey for the problem and its related variants can be found in [15]. The Multi-Mode version, where activities have several alternative modes with different parameters, was studied by Schnell et al. in [23]. The authors developed the exact algorithm by extending the SCIP Solver [1]. Added constraint handlers (i.e. functions for constraints propagation) allowed to directly cooperate between a low-level constraint integer program solver and high-level MMRCPSP constraints and objective. Another exact approach is described in [24] where the constraint-based modeling tool CP Optimizer was used to solve the RCPSP problem. The expressive power and universality of the CP Optimizer allows one to easily extend or modify the RCPSP model, which is used for justification of the makespan and porosity optimization in the paper.

1.3 Paper outline and contribution

The paper is organized as follows: Section 2 describes the studied problem of the TT message scheduling in the TTEthernet network. In Section 3, the proposed method is described consisting of a message routing algorithm, a load-balancing heuristic and an RCPSP based formulation of the scheduling problem. The method is evaluated from the resulting schedule length point of view and also the RC traffic worst-case delay point of view in Section 4. Section 5 concludes the paper.

The main contribution of the paper is the investigation of a new concept for creating schedules of the TT traffic in the TTEthernet network so that, in opposite to the preceding studies, it focuses on shortening the makespan (the latest completion of any message transmission among all integration cycles and links - see Fig. 2) of the TT traffic instead of introducing porosity (blank slots in the TT traffic schedules that are reserved for RC and BE traffic). The makespan serves as a good measurement for the schedule quality evaluation. The proposed idea has been inspired by the FlexRay communication scheme, where the Time-Triggered and Event-Triggered segments are separated. A further contribution is the novel formulation of the TTEthernet scheduling problem as an RCPSP problem. Both contributions are evaluated and discussed.

2 Problem statement

This paper aims to design a method for finding feasible periodic schedules for Time-Triggered communication on the TTEthernet network such that the maximal part of the remaining bandwidth can be preserved for RC and BE messages.

Each message from a set of the TT messages has assigned:

  • - period

  • - message length in the number of bits considering headers and interframe gap

  • - deadline

  • - release date

  • - identifier of the transmitting node

  • - set of the receiving nodes

The length of the resulting schedule is determined by the cluster cycle (40 ms in Fig. 2). The length of the integration cycle (10 ms in Fig. 2) is assumed to be the greatest common divisor of message periods (i.e. ). In other words, all the periods has to be an integer multiple of the length of integration cycle . The schedule is so-called strictly periodic, which means that the next occurrence of message in a particular link (further called message occurrence - see Fig. 3) appears in the schedule exactly time units after the current one. The positions of all message occurrences of message in the strictly periodic schedule can be deduced from the position of the first message occurrence.

The transmission time of message has to be smaller than or equal to the integration cycle and its length does not exceed the maximal Ethernet frame length of 1530 bytes. It would not be possible to send a synchronization message otherwise. Deadline and release date are assumed to have the value in the range .

Each node in the network has its identifier assigned. The nodes are divided into two classes: redistribution nodes and communication endpoints. The communication endpoints are nodes that generate or process the data (e.g. sensors, actuators, control units and other ECUs). Thus, the identifier of any communication endpoint can be assigned to message as transmitter or one of the receivers from set . The redistribution nodes, on the other side, are switches without any own data to transmit and serve as intermediary nodes for communication.

The TTEthernet infrastructure consists of nodes and links which interconnect them. Each link from a set of links connects two nodes and . This connection covers just one direction of the full-duplex communication. Therefore, two links and model one physical link between nodes and . These two links are independent from the scheduling point of view.

Figure 3: Visualization of the difference between message occurrence and message instance

A feasible schedule has to fulfill the following hard constraints:
Completeness constraint: Each message has to be scheduled.
Contention-free constraint: Any link is capable of transferring at most one message at a time.
Precedence constraint: A sequence of the links represents the path that message has to go from transmitter to receiver through redistribution nodes . An example of such a message transmission is presented in Fig. 1 where communication endpoints are titled by ”ECU” and the redistribution nodes has arrows painted on the top side. The front side of all nodes is labeled by its name. Only one direction of each physical link is labeled in Fig. 1 for the sake of simplicity.

The instance of message in link is denoted as message instance . Thus, all the transmissions of some message in one particular link represents the same message instance. The message occurrence, on the other hand, represents all the transmissions of some message in one particular integration cycle. The difference between message instance and message occurrence is graphically explained in Fig. 3. The figure shows the detail view on the sub-segment of the network infrastructure from Fig. 1 with node , and only. The both links of any physical link are labeled here already.

The path is not known in advance if the infrastructure does not have a tree topology. Therefore, part of the optimization process is to find an appropriate path. The message transition from transmitting node to all the receivers from has to be accomplished in one integration cycle. In the real multi-hop networks, each hop introduces a technical delay caused by queuing in the ingress and egress port, etc. Such a delay in a switch is represented by parameter for TT messages. The value of can be in range from to according to the network configuration [25]. Sequence entails the generalized precedence constraints because message has to be scheduled in link time units after it is scheduled in if precedes in .

The coherent TT traffic segment should be compressed as much as possible to preserve the maximum part of the remaining bandwidth for the RC and BE traffic. This idea follows the practice from the FlexRay bus or Profinet where the TT traffic has its communication segment. The TT traffic can be scheduled at the beginning of the integration cycle, and the remaining TT-free gap (coherent gap in integration cycle without the TT traffic) is preserved for the RC and BE traffic. The shortest TT-free gap among all links is denoted as a critical gap (see Fig. 2). Considering the constraints above, the goal of the scheduling is to find a schedule for TT messages which maximizes the critical gap and, thus, minimizes its makespan.

3 Algorithm

The algorithm proposed in the paper is divided into three stages. In the first stage (Sec. 3.1), the routing of the messages is established. In the second stage (Sec. 3.2), the algorithm finds the assignment of the messages to the particular integration cycles. The transmission times for each message occurrence in each link are scheduled in the last stage (Sec. 3.3).

3.1 Determination of Time-Triggered messages routing

The network topology is often a tree in industrial networks. It means that there are no cycles and, therefore, there exists only one possible path from a communication endpoint to any other endpoint. Thus, the routing determination is trivial in such a case. However, the TTEthernet does not restrict the network topology, according to the specification, to the tree. The cycles introduce new redundant paths for messages. The redundant paths can relieve busy links and serve as a backup during a partial network malfunction. However, the TT messages have to know which path they are routed through in advance. Therefore, the first stage of the algorithm finds the routing. Each hop the message has to pass implies the increase in overall traffic of the network. The long path also causes the prolongation of the end-to-end delay because the message is delayed in the ingress and egress buffer of each redistribution node in the path. This means that utilizing the shortest path algorithm provides an efficient routing with the minimal number of hops. The network topology is transformed to graph where edges represent the physical links and its weight is set to one. The Floyd-Warshall algorithm [12] is used, consequently, to find a routing among all nodes in the network. All the messages follow the routing consequently.

Considering that the aim of this routing is to minimize delay caused by the switch hoping, the routing over the shortest path can cause unbalanced load among links. Thus, there are cases where it is better to choose another routing strategy. However, this default routing strategy can be substituted by any other routing that follows demands of the particular application. Other routing strategies can be found for example in [11] or [18].

The routing defines the links in which the messages are to be scheduled and specify the precedence relations among message instances of each message.

3.2 Integration cycle assignment problem

The used idea how to distribute messages among the integration cycles comes from the multiprocessor scheduling area. In the multiprocessor scheduling area, if all the workload of tasks is distributed among the processors evenly, then the schedule makespan has a good chance to be minimal. Following that, the algorithm tries to distribute the messages among integration cycles evenly. All the precedence constraints, time lags imposed by the switch delay and real-time constraints are relaxed here. The integration cycle assignment problem is formulated as the following ILP model:

The binary variable

if message is scheduled to the integration cycle and zero otherwise. The parameter represents the transmission time of message on line . Note, that if the links are configured to have a different bandwidth then the transmission time of the same message varies among the links. The first constraint assures that the first message occurrence appears in exactly one of the possible integration cycles. Thus, it satisfies the completeness constraint. The second constraint makes the variable to have the value equal to or greater than the time needed to exchange all messages in any integration cycle of any link in the network. The constraint is evaluated for each resource (each link and each integration cycle in the cluster cycle) such that the transmission time of all message occurrences assigned to the particular resource is summed up. The resulting total must be less than or equal to variable . The aim of the ILP model is to find such an assignment that minimizes . Thus, the maximal time needed for message exchange among all resources is minimized. The last two constraints force messages to be assigned to the integration cycle which can satisfy the release date and deadline constraints.

The resulting assignment balances the load among the resources, follows the routing of the messages and satisfies the release date and deadline constraints.

3.3 Link schedule creation problem

The RCPSP formulation, inspired by the RCPSP formulation of the traffic scheduling on ProfinetIRT [14], is used for the message scheduling problem. Thus, let us provide a brief overview for RCPSP first.

3.3.1 RCPSP Overview

The RCPSP problem used in the paper is classified as PSm,1,1|temp|

by Graham’s notation. Translated to common language, the problem is a Project Scheduling problem with m resources, one unit of each resource available, and each activity demands at most one unit of the resource. The problem is constrained by temporal constraints (time lags among activities), and the objective is to minimize the makespan .

The problem is defined as a sextuplet . is a set of non-preemptive activities that should be scheduled. and are special dummy activities where represents the start of the schedule and the end of the schedule, respectively.

is the vector of the activities duration. The dummy activities have zero duration. The resulting schedule would be shifted otherwise.

is a set of pairs representing time constraints. If activity has a temporal relation to activity then . Each pair in is valued by the start-start time lag . Note that the time lag between any activity and dummy activity is equal to the activity duration . The temporal constraints are often visualized by the so-called activity-on-node graph . The set is a set of resources considered in the problem. , consequently, represents the number of units available in the resources. All the resources are unary () in our case. Parameter is a set of activity demands where represents the amount of resource demanded by the execution of activity . The value of is binary () in the PSm,1,1|temp| problem.

The goal is to assign the start time to the activities such that resource demands and time constraints are satisfied and the start time of activity is minimal. An interested reader is referred to [5] for more detailed information on the RCPSP problem.

3.3.2 Formulation of the link schedule creation problem to RCPSP

The stated TTEthernet scheduling problem can be formulated as the PSm,1,1|temp|Cmax problem, considering the following conditions are met:

  1. Routing: It is known in which links each message appears

  2. Integration cycle assignment: It is known in which integration cycles each message appears

The first condition is satisfied by the first stage of the algorithm where the messages are assigned to links according to the routing. The second condition is also met and the assignment of each message to the integration cycles is obtained from the second stage of the algorithm.

In the RCPSP model [20], the message instances are represented by the activities. Thus, the set of message instances is translated to the set of activities where activity corresponds to message instance . The dummy activities and are artificially added to . The duration of activity expresses the transmission time of message on link . However, we assume that the bandwidth of all physical links is the same for the sake of simplicity in this paper. The resulting start time of activity will, consequently, represent the offset of message instance in the particular integration cycles in link . Recall that the integration cycles, in which the message appears, are determined by the first message occurrence.

The set of resources is obtained from - the set of integration cycles in the cluster cycle and - the set of links in the network. The number of resources used in the model is then . Each resource represents the usage of link in the integration cycle . The resources have unary availability . Correspondingly, the message demands are unary too, which ensures that the contention-free constraint is satisfied. Considering that the given conditions are met, the activities demand can be directly derived from the routing and the message to the integration cycles assignment. The notation , which represents the amount of resource demanded by activity , is used from this point further. If message is to be routed through link , among others, and it is known that the message appears, for example, in , and then , , . All other demands for the activity representing message instance will be equal to zero.

Figure 4: Example of activity-on-node graph for outlined message from Fig. 1

All the real-time constraints are modeled by set and time lags. In order to model release date and deadline , it is necessary to know their relative values and with respect to the integration cycle in which the message is transmitted. The values are obtained by subtracting the integration cycle start time from or respectively. The release date of the message is transformed to the positive time lag led from the dummy activity to activity where is the first link in the path the message traverses. The time lag is equal to the relative release date . The deadline, on the other hand, is transformed to a negative time lag led from activity , where is the last link before receiving the communication endpoint, to the dummy activity . The value of the deadline time lag is equal to the negative value of the relative deadline extended by duration (i.e. ). Note that if there is more receivers in set then it is necessary to add the deadline time lag for each receiver separately. Technical delays and precedence constraints are represented by the positive precedence time lags between two consecutive message instances and . The precedence time lag consists of duration and redistribution nodes delay parameter in store-and-forward networks.

In Fig. 4, the example of the activity-on-node graph for message is presented. Message can be also observed in Fig. 1 or in its crop in Fig. 3 - the one with a thick outline. The links assumed here have a bandwidth of 10 Mbit/s and the message consists of 672 bits (64 octets of Ethernet frame + 12 octets interframe gap + 8 octets preamble and start of frame delimiter). The message is transmitted by node and received by nodes and . It traverses the redistribution nodes , and and the redistribution node delay is . The relative release date , expressed by the edge from to , is . The relative deadline , represented by the edge from and to , is .

The time lags allow one to limit the latency between the message transmission and reception or the precedence constraints between the messages (e.g. derived from the application level constraints). However, these constraints are out of the scope of our problem statement and are not used in the paper.

With a given configuration of the RCPSP model, the objective of the RCPSP (i.e. to minimize the maximal makespan) corresponds to the objective of our problem statement. In other words, the minimal RCPSP makespan assures the maximal critical gap.

The scheduling problem can be formulated as MMRCPSP as well. In this case, the integration cycle assignment is not necessary to be solved in advance, because it can be formulated as part of the MMRCPSP [16]. However, the complexity of the resulting MMRCPSP model is significantly higher and, thus, the solver needs more time to find a good or even any solution (see Sec. 4.2).

4 Experimental results

The proposed scheduling method was tested on a PC with Intel ® Core™ i7-4610M CPU (two cores with 3 GHz and hyper-threading) and 8 GB RAM. The algorithm uses the CPLEX ILP Solver for solving the Integration cycle assignment problem. The RCPSP problem was solved by both the CP Optimizer and SCIP Solver. However, the results obtained by the CP Optimizer were significantly better in all the cases. Thus, the results of the SCIP Solver are omitted here.

4.1 Benchmark instance sets

Eight different benchmark sets were used for testing purposes. Seven of them were generated by our instance generator. They represent the artificial instances. The last instance is the real problem instance obtained from our industrial partner.

The synthetic instances are divided into seven groups according to the number of the used TT messages. The sets are called according to that - from Set_20TT (set with twenty TT messages) to Set_2000TT. The length of the integration cycle equals thousand times the number of the TT messages, which approximately caused the utilization of half of the bandwidth. Each such set contains thirty independently generated benchmark instances. During the instance creation, the algorithm for generating the network topology is selected randomly. The generator provides four possible algorithms - the star topology generator, the snowflake topology generator, the Barabási-Albert algorithm for random tree generation and a random topology generator (the Barabási-Albert algorithm with additional redundant links). The Barabási-Albert algorithm is slightly modified. The nodes of the network with degree 2 are eliminated to remove long linear network segments which are uninteresting from the scheduling point of view. The network consists of twenty communication endpoints for all the topologies. Each link has the bandwidth of 1 Gbit/s.

The messages have a randomly chosen transmitter and set of receivers from the set of communication endpoints. Thus, any communication endpoint can send a message to an arbitrary subset of other communication endpoints. The payload of the messages is taken randomly from interval of 46 to 256 bytes. The message periods should be harmonic or close to harmonic to keep the length of the cluster cycle reasonable. Thus, the periods are from the domain where and . The release times and deadlines are also generated randomly such that .

The last benchmark set Set_Indust consists of one benchmark instance given to us by our industrial partner. The instance evaluated the behavior of the proposed method on instances from a natural environment which often does not directly correspond to the artificial ones. The network consists of 6 redistribution nodes and 59 communication endpoints and contains alternative paths. The instance has 1018 messages to exchange with periods from domain and efficient payloads (payload that contain useful information) from domain bits. There are no hard real-time constraints imposed on the instance. Each message has from one up to forty-two receivers.

4.2 Quality and Performance evaluation of the solver

The provided method for the TT message scheduling has been evaluated on the described benchmark set and compared with the Multi-mode method. The Multi-mode method developed by us is similar to the described algorithm, but it uses the Multi-mode RCPSP instead of the general RCPSP model. In the Multi-mode RCPSP, the alternative modes can be assigned to the activities, and the resulting schedule uses just one of the provided activities. The TTEthernet problem can be formulated as a Multi-mode RCPSP without the necessity to decide in which integration cycles each message appears in advance. Thus, it is not necessary to solve the integration cycle assignment problem heuristically by ILP. The Multi-mode RCPSP can return better solution than our described algorithm denoted as ICAP in Table 1 that uses heuristic integration cycle assignment, but the computational complexity increases significantly.

ICAP Multi-mode
N [-] LB [ns]  [ns]  [ns]
Set_20TT 124 9626 13430 13255
Set_50TT 312 19468 22908 23678
Set_100TT 624 35603 40450 42291
Set_200TT 1243 66594 74663 79733
Set_500TT 3107 157317 182185 181913
Set_1000TT 6244 318017 354577 354950
Set_2000TT 12460 616446 663532 -
Set_Industrial 5844 59888 63568 -
Average 177429 194708 -
Table 1: Quality comparison of different approaches

Table 1 represents the results obtained by the methods. The captions of the benchmark sets are situated in the first column. The second column presents the average number of message instances/activities in the set (N). Recall that each message has one message instance for each link the message passes. Therefore, the number of message instances is often much higher than the number of messages. All the remaining columns shows the results for the particular algorithm. The values are averaged over thirty independent instances in each set. The third column shows the lower bound (LB) for the problem. The lower bound is obtained from the ILP solution of the integration cycle assignment problem. The makespan of our problem can never be shorter than the sum of the messages transmission time exchanged in any link if the message to the integration cycle assignment is balanced. That is why the ILP solution can be used as a lower bound for the whole problem too. However, it is often not possible to obtain a makespan equal to the lower bound because time lags are not considered in the ILP model. Therefore, it is not-tight lower bound. The fourth column contains the average makespan value of the instances in the given set for our provided algorithm (ICAP) in nanoseconds. In comparison, the fifth column contains the makespan for the Multi-mode method. The time limit for the computation of each instance was set to 300 seconds for both methods.

The small theoretical instances with 20 TT messages were solved to an optimum by RCPSP solver by both methods in time. The final results for these small instances are better for the Multi-mode version because it is not heuristically guided by the ILP solution and, therefore, no states of the search space are pruned. One can observe the gap between the optimal solution of the Multi-mode method and the non-tight lower bound. The Multi-mode RCPSP solver was not able to solve bigger instances to an optimum in time. This leads one to the fact that the guided ICAP method obtains better solutions (except the set with 500 TT messages where relaxing the time-lags in the ICAP formulation led to the weaker integration cycle assignment) from this point further. Furthermore, no feasible solution was found by the Multi-mode RCPSP solver for instances with two thousand messages in 300 seconds. Note that the solutions obtained by the ICAP method are not more than 15% distant from the lower bound for instances with more than 50 TT messages.

4.3 Worst-case RC traffic delay evaluation

To justify the used scheduling method, the solver based on the porosity idea from [28] was implemented to compare the makespan optimization method with the porosity optimization method with respect to the RC traffic. The porosity is represented by the set of gaps introduced into the schedule. Consequently, the optimization objective is to maximize the length of these gaps. Compared to the makespan optimization, which aims to introduce just one big gap at the end of the schedule, the porosity optimization distributes the free bandwidth through the schedule.

Note that unlike in other protocols that has strict separations of time-triggered and event-triggered segments, the introduced porosity gaps as well as one critical gap introduced by makespan optimization does not separate the TT traffic and RC traffic so strictly. It means that even if the makespan is e.g. 6 ms (just like in the Fig. 2) the RC message can be transmitted sooner if there is no TT traffic scheduled in the particular integration cycle (e.g. message in Fig. 2). Moreover, the RC traffic can be transmitted even in the gaps of the schedule where they can fit (e.g. message in Fig. 2).

Figure 5: Worst-Case delays for the RC traffic

To evaluate the difference between the makespan optimization and the porosity optimization in an objective way, the Worst-Case delay calculation for the RC messages taken from [29] was employed.

In Fig. 5

, the average delays among the instances in the particular sets and its standard deviations are presented. The evaluation was made for all the artificial sets. However, Set_2000TT has been omitted from the graph because it caused graph scaling which worsens its clarity. The worst-case RC traffic delay of Set_2000TT was

for makespan optimization and it was for porosity optimization.

According to the results, the makespan optimization behaves comparable and even slightly better than the porosity optimization with respect to the RC traffic. The standard deviation is higher in the case of Set_500TT because there were a few significant outliers. Overall, the obtained Worst-Case delays are more stable in the case of the makespan optimization.

5 Conclusion

The paper in hand focuses on the problem of the Time-Triggered communication scheduling on the TTEthernet network. To incorporate the Event-Triggered Rate-Constrained traffic with TT traffic is very important for mixed-criticality applications [21]. The majority of communication is Event-Triggered in industrial applications nowadays. However, the necessity to incorporate safety-related communication, e.g. in automotive or avionics industry, pushes system developers to use the TT traffic because it can be more easily verified and certified.

We have investigated the idea of separating the Time-Triggered traffic and Event-Triggered traffic, which was inspired by the scheme of the FlexRay bus communication cycle. The aim has been to minimize the length of the Time-Triggered schedule and leave the rest of the communication bandwidth free for the Rate-Constrained traffic. For such a problem statement, we have designed the algorithm based on the ILP formulation of the integration cycle assignment problem (load balancing among integration cycles) and the RCPSP formulation of the message scheduling problem.

The experiments performed on several artificial benchmark sets and one real-case instance show that the designed method is able to obtain near-optimal results (for bigger instances in 15% of the non-tight lower bound) concerning the proposed criterion. Moreover, the RC traffic worst-case delay calculations suggest that the proposed concept (minimization of the schedule makespan) can provide slightly better delays in average than the porosity imposed to the schedule by gaps.

For the future work, we would like to study the behavior of the provided method in the environment of the 802.1Qbv [32] network. The Ethernet technology is used in the automotive, avionics, etc. industry nowadays, where the development cycle of the products is specific - the new product models are created taking a backward compatibility with the previous ones into account. Thus, we are also going to study the incremental scheduling case where backward compatibility needs to be preserved.


This work was supported by the Grant Agency of the Czech Republic under the Project FOREST GACR P103-16-23509S.


  • [1] T. Achterberg (2009-07) SCIP: Solving constraint integer programs. Math. Programming Computation 1 (1), pp. 1–41. Cited by: §1.2.
  • [2] ARINC (Aeronautical Radio, Inc.) (2009) ARINC 664P7: Aircraft Data Network, Part 7, Avionics Full-Duplex Switched Ethernet Network. Technical report ARINC (Aeronautical Radio, Inc.). Cited by: §1.1.
  • [3] ARINC Industry Activities (2017)(Website) External Links: Link Cited by: §1.1.
  • [4] M. Baghel, S. Agrawal, and S. Silakari (2012-11)

    Survey of metaheuristic algorithms for combinatorial optimization

    Int. J. Comput. Applicat. 58 (19), pp. 21–31. Cited by: §1.2.
  • [5] P. Brucker, A. Drexl, R. Möhring, K. Neumann, and E. Pesch (1999-01) Resource-constrained project scheduling: notation, classification, models, and methods. European J. of Operational Research 112 (1), pp. 3 – 41. Cited by: §3.3.1.
  • [6] Condor Engineering, Inc. (2005) AFDX / ARINC 664 Tutorial (1500-049). Technical report Condor Engineering, Inc.. Cited by: §1.1.
  • [7] S. S. Craciunas, R. S. Oliver, and V. Ecker (2014) Optimal static scheduling of real-time tasks on distributed time-triggered networked systems. In Proc. IEEE Emerging Technology and Factory Automation (ETFA), Barcelona, Spain, pp. 1–8. Cited by: §1.2.
  • [8] S. S. Craciunas, R. S. Oliver, M. Chmelík, and W. Steiner (2016) Scheduling real-time communication in IEEE 802.1Qbv time sensitive networks. In Proc. 24th Int. Conf. on Real-Time Networks and Syst. (RTNS), Brest, France, pp. 183–192. Cited by: §1.2.
  • [9] S. S. Craciunas and R. S. Oliver (2016-03) Combined task- and network-level scheduling for distributed time-triggered systems. Real-Time Syst. 52 (2), pp. 161–200. Cited by: §1.2.
  • [10] R. I. Davis, S. Kollmann, V. Pollex, and F. Slomka (2013) Schedulability analysis for Controller Area Network (CAN) with FIFO queues priority queues and gateways. Real-Time Syst. 49 (1), pp. 73–116. Cited by: §1.
  • [11] G. E. de Andrade, L. A. de Paula Lima, A. Calsavara, J. A. de Oliveira, and G. Michelon (2016) Message routing in vehicular delay-tolerant networks based on human behavior. In 10th Int. Symp. Commun. Syst., Networks and Digital Signal Process. (CSNDSP), Prague, Czech Republic, pp. 1–6. Cited by: §3.1.
  • [12] R. W. Floyd (1962-06) Algorithm 97: shortest path. Commun. ACM 5 (6), pp. 345. Cited by: §3.1.
  • [13] B. Galloway and G. P. Hancke (2013-07) Introduction to Industrial Control Networks. IEEE Communications Surveys Tutorials 15 (2), pp. 860–880. Cited by: §1.
  • [14] Z. Hanzálek, P. Burget, and P. Šůcha (2010-08) Profinet IO IRT Message Scheduling With Temporal Constraints. IEEE Trans. Ind. Informat. 6 (3), pp. 369–380. Cited by: §1, §3.3.
  • [15] S. Hartmann and D. Briskorn (2010-11) A survey of variants and extensions of the resource-constrained project scheduling problem. European J. of Operational Research 207 (1), pp. 1 – 14. Cited by: §1.2.
  • [16] M. Heller (2016) Scheduling of the TTEthernet communication. Master’s Thesis, Czech Technical University in Prague, Czech Republic. Cited by: §1.2, §3.3.2.
  • [17] International Organization for Standardization (2015)(Website) External Links: Link Cited by: §1.
  • [18] A. Jouy, J. Yao, and G. Zhu (2014) Optimal bandwidth allocation with dynamic multi-path routing for non-critical traffic in afdx networks. In 20th IEEE Int. Conf. on Parallel and Distributed Syst. (ICPADS), Hsinchu, Taiwan, pp. 600–607. Cited by: §3.1.
  • [19] H. Kopetz, A. Ademaj, P. Grillinger, and K. Steinhammer (2005) The time-triggered ethernet (TTE) design. In 8h IEEE Int. Symp. Object-Oriented Real-Time Distributed Computing (ISORC), pp. 22–33. Cited by: §1.1.
  • [20] K. Neumann, C. Schwindt, and J. Zimmermann (2012) Project scheduling with time windows and scarce resources. Springer US, Boston, MA. Cited by: §1.2, §3.3.2.
  • [21] A. Novak, P. Sucha, and Z. Hanzalek (2016) Efficient algorithm for jitter minimization in time-triggered periodic mixed-criticality message scheduling problem. In Proc. 24th Int. Conf. on Real-Time Networks and Syst. (RTNS), Brest, France, pp. 23–31. Cited by: §5.
  • [22] SAE International (2011) AS6802: Time-Triggered Ethernet. Technical report SAE International. Cited by: §1.1.
  • [23] A. Schnell and R. F. Hartl (2016-03) On the efficient modeling and solution of the multi-mode resource-constrained project scheduling problem with generalized precedence relations. OR Spectrum 38 (2), pp. 283–303. Cited by: §1.2.
  • [24] P. Sitek and J. Wikarek (2016) A constraint-based approach to modeling and solving resource-constrained scheduling problems. In Proc. 8th Int. Con. Computational Collective Intell. (ICCCI), Halkidiki, Greece, pp. 423–433. Cited by: §1.2.
  • [25] T. Steinbach, F. Korf, and T. C. Schmidt (2010) Comparing time-triggered Ethernet with FlexRay: An evaluation of competing approaches to real-time for in-vehicle networks. In Proc. 8th IEEE Int. Workshop on Factory Commun. Syst., Nancy, France, pp. 199–202. Cited by: §2.
  • [26] W. Steiner, M. Gutiérrez, Z. Matyas, F. Pozo, and G. Rodriguez-Navas (2015) Current techniques, trends and new horizons in avionics networks configuration. In 34th IEEE/AIAA Digital Avionics Syst. Conf. (DASC), Prague, Czech Republic, pp. 1–26. Cited by: §1.2.
  • [27] W. Steiner (2010) An Evaluation of SMT-Based Schedule Synthesis for Time-Triggered Multi-hop Networks. In 31st IEEE Real-Time Syst. Symp. (RTSS), San Diego, CA, USA, pp. 375–384. Cited by: §1.2.
  • [28] W. Steiner (2011) Synthesis of static communication schedules for mixed-criticality systems. In 14th IEEE Int. Symp. Object/Component/Service-Oriented Real-Time Distributed Computing Workshops (ISORCW), Newport Beach, CA, USA, pp. 11–18. Cited by: §1.2, §1, §4.3.
  • [29] D. Tamas-Selicean, P. Pop, and W. Steiner (2015) Timing Analysis of Rate Constrained Traffic for the TTEthernet Communication Protocol. In 18th IEEE Int. Symp. on Real-Time Distributed Computing, Auckland, New Zealand, pp. 119–126. Cited by: §1.1, §1.2, §4.3.
  • [30] D. Tamas-Selicean, P. Pop, and W. Steiner (2012) Synthesis of Communication Schedules for TTEthernet-based Mixed-criticality Systems. In Proc. 8th IEEE/ACM/IFIP Int. Conf. Hardware/Software Codesign and Syst. Synthesis (CODES+ISSS), Tampere, Finland, pp. 473–482. Cited by: §1.1, §1.2.
  • [31] D. Tamas-Selicean, P. Pop, and W. Steiner (2015-01) Design optimization of TTEthernet-based distributed real-time systems. Real-Time Syst. 51 (1), pp. 1–35. Cited by: §1.2.
  • [32] WG802.1 - Higher Layer LAN Protocols Working Group (2016) 802.1Qbv-2015 - IEEE Standard for Local and metropolitan area networks – Bridges and Bridged Networks - Amendment 25: Enhancements for Scheduled Traffic. Technical report IEEE Standards Association. Cited by: §5.