1 Introduction
Deep learning (DL), which refers to a class of neural networks (NNs) with deep architectures, is now a workhorse powering stateoftheart results on a wide spectrum of tasks Yan:2015:HDCNN ; yan2016automatic ; mikolov2013efficient
. One reason for its widespread adoption is the variety and quality of software toolkits, such as Caffe
jia2014caffe , TensorFlow abadi2016tensorflow and DyNet neubig2017dynet ; neubig2017fly , which ease programming of DL models, and speed computation by harnessing modern computing hardware (e.g. GPUs), software libraries (e.g. CUDA, cuDNN chetlur2014cudnn ), and compute clusters zhang2015poseidon ; zhang2017poseidon ; cui2016geeps .One dominant paradigm in the training of DL models, adopted by toolkits such as Caffe and TensorFlow, uses static dataflow graphs abadi2016tensorflow ; murray2013naiad . These graphs represent the flow of data through computational functions, and are defined using symbolic programming Bergstra:2011:NIPSW ; abadi2016tensorflow , once before beginning training or testing of the model. The training of these models is performed through autodifferentiation, in which users are only required to assemble their model architectures by connecting operators using highlevel language interface (e.g. Python), after which the framework will automatically derive the correct algorithm for training bartholomew2000automatic . With proper optimization, the execution of these static dataflow graphs can be highly efficient. Specifically, by separating model declaration and execution, it makes it possible for the graph to be further processed and optimized before runtime abadi2016tensorflow . In addition, the evaluation of multiple data samples in a dataflow graph can be naturally batched to leverage the improved computational capability of modern hardware (e.g. GPUs), which is extremely advantageous for DL workloads Krizhevsky:2012:NIPS .
While these static dataflow graph have major efficiency advantages, their applicability highly relies on a key assumption – the dataflow graph (i.e. NN architecture) fixed throughout the runtime. With the increasing complexity of the problems to be addressed, DL has been extended and applied on data with more complicated structures, such as sequences hochreiter1997long ; sutskever2014sequence , trees tai2015improved and graphs liang2016semantic , over which the NN may conditionally choose its own computation order for specific modeling needs, i.e. the structure of its dataflow graph changes over training. To better support these dynamic models, some recent frameworks tokui2015chainer ; neubig2017dynet propose to declare a dataflow graph per sample (a.k.a. dynamic declaration). While dynamic declaration is convenient to developers as code can basically be written in the same way as it usually is in the native programming language (e.g. Python, C++), it exhibits a few limitations. First, programmers still have to write code to explicitly assemble the dataflow graph for each input sample, which might be nontrivial for graphs with sophisticated structures. Second, as the graph construction needs to be performed repeatedly, its overhead grows linearly with the number of training instances, preventing the application of complex static graph optimization techniques (in fact, graph construction takes longer time than the computation in some frameworks looks2017deep , see §5.2). Finally, since each sample owns a dataflow graph specifying its unique computational pattern, batching together similarly shaped computations across instances is nontrivial. Without batching operations, the computation is inefficient due to its lack of ability to exploit modern computational hardware, and while some progress has been made in recent research neubig2017fly ; looks2017deep , how to automatically batch the computational operations from different graphs remains a difficult problem.
To address these challenges, we present Cavs, a new programming interface for dynamic NNs, and a system implementation with optimization strategies tailored to it. Cavs leverages the recurrent and recursive nature of dynamic NNs. Instead of declaring a dataflow graph per sample, it alternatively decomposes a dynamic dataflow graph as two components: one static vertex function that is only declared (by the user) and optimized once, and an input graph that is instancespecific and not used until runtime. Thereby, the workflow of training a dynamic NN can be represented as scheduling the execution of following the structure of the input graph . Cavs combines the best of symbolic construction of dataflow graphs for DL abadi2016tensorflow ; Bergstra:2011:NIPSW with the vertexcentric model gonzalez2012powergraph in graph computing: it only requires users to define symbolically by “thinking locally like a vertex” tian2013think . Cavs will perform autodifferentiation, schedule the function execution following the dependency reflected by , and guarantee efficiency and correctness. It also inherits the flexibility of symbolic programming, i.e. users are allowed to declare multiple vertex functions to express more dynamics, or connect static dataflow graphs with dynamic ones to construct more complex NN architectures.
Cavs demonstrates a few advantages over other programming models. It simplifies user programs and avoids the overhead of repeated dataflow graph construction. Moreover, this vertexcentric model naturally exposes opportunities for batched computation: we introduce a simple batching policy in Cavs’ scheduler to parallelize the execution of on multiple vertices during the evaluation of a batch of samples with different input graphs (§3.2), and a novel memory management mechanism to guarantee the memory coalescing (§3.3). Together they yield significant performance improvements. Compared to dynamic declaration, as the dataflow graph encoded by the vertex function is static throughout the runtime, it can benefit from various static graph optimizations abadi2016tensorflow ; chen2015mxnet ; caffe2 ; xla , such as lazy batching, streaming, and kernel fusion (§3.5), which would otherwise be less effective on the scenario of dynamic declaration because of the repeated preprocessing/optimization cost (see §6).
We implement Cavs as an additional layer pluggable to most existing DL frameworks to enhance their support for dynamic NNs. To evaluate its performance, we compare Cavs to TensorFlow Fold looks2017deep and DyNet neubig2017dynet ; neubig2017fly , two stateoftheart systems supporting dynamic NNs and dynamic batching. We focus our experiments on GPU training, and verify that both Fold and DyNet suffer from substantial overhead caused by repeated graph preprocessing or construction, which is bypassed by Cavs (§5.2). In terms of overall performance, on static NNs, Cavs demonstrates equivalent or slightly better performance than Fold and DyNet, while on several dynamic NNs with notably difficulttobatch workloads (e.g. TreeLSTM tai2015improved and TreeFC looks2017deep ), Cavs demonstrates near one order of magnitude speedups across various dataset and hyperparameter settings (§5.1). We further investigate the key contributing factors to the performance: Cavs benefits from not only a better memory management strategy, but also graph execution optimizations which are originally designed for static dataflow graphs and perhaps less useful in dynamic declaration.

2 Background
DL is distinguished from other ML algorithms mainly by its use of deep neural networks, a family of ML models with many interconnected layers, each composed of various mathematical operations (e.g.
). Before a DL model can give predictions, it is usually trained by stochastic gradient descent, an iterative process in which gradients are calculated through backpropagation
rumelhart1988learning . There is a natural connection between directed graphs and NNs: we can map the graph nodes to the computational operations or parameters in NNs, and let the edges indicate the direction of the data being passed between the nodes. In this case, we can represent the process of training NNs as batches of data flowing through computational graphs, i.e. dataflow graphs Bergstra:2011:NIPSW ; abadi2016tensorflow ; neubig2017dynet .Figure 1(a) summarizes the programming model derived from these dataflow graphs, which is named as static declaration and has been adopted in many DL frameworks Bergstra:2011:NIPSW ; abadi2016tensorflow ; chen2015mxnet . Without ambiguity, we use to denote both the dataflow graph itself and the computational function implied by . On one hand, we note its execution is highly efficient as the computation over multiple samples is batched – at each iteration
, a batched tensor of
samples is fed to , and the computation is executed in a single pass, allowing for efficient use of memory caches or parallelized computation. On the other hand, this paradigm relies on a key assumption: the dataflow graph is static for all samples and fixed throughout the computation. Hence, will only be declared once with a constant graph construction/optimization overhead; all samples share the same computational pattern specified in , so the computation of different samples can be by naturebatched by simply expanding the input with a batch dimension. Though static declaration is effective on a wide range of NN models, such as convolutional neural networks (CNNs) over fixedsize images, it is much more difficult to apply to graphs with dynamically changing structures, some examples of which are shown in the next section.
2.1 Dynamically Structured Computational Graphs
Modern DL has been developed and applied extensively over data with more complicated structures, e.g. data structured as sequences, trees and graphs, which are required to tackle practical problems such as machine translation sutskever2014sequence ; tai2015improved , questionanswering tan2015lstm , and semantic image segmentation yan2016combining ; liang2016semantic . As a concrete example of dynamic NNs, we will use recurrent and recursive neural networks (RNNs) elman1990finding ; hochreiter1997long ; socher2013recursive . RNNs are a class of NNs generally applied on modeling structured inputs or outputs, e.g. sequences or graphs. At the core of an RNN is a cell function with trainable parameters. It will be dynamically applied at different places of the input structure, and optionally produce an output if needed. Figure 2(a) illustrates such a cell function: it takes an input element , forwards it through a few mathematical transformations, and generates some intermediate state and an output
. Depending on what transformations are applied, different variants of RNNs have been derived, such as longshort term memory units (LSTM)
hochreiter1997longand gated recurrent units (GRU)
chung2014empirical . However, the internals of the cells themselves are secondary; the dynamics of the net as a whole are mainly reflected by the structures that the NN works on.Sequence RNNs. When the input to the RNN are sequences (e.g. sentences) as in Figure 2b, the cell function is applied across all elements of the sequence. At each step , it takes the element (e.g. a word) from the input sequence, and the state variable maintained by the model at step . It computes an output , and a new state that will be used by the next step . Hence, This sequence RNN encodes not only the data values, but also the dependencies present in the sequence. If represented as a dataflow graph, the graph exhibits a chain structure. As the input or output sequences usually have variable length (e.g. translating an arbitrarylength English sentence into Chinese), the dataflow graph needs to be dynamically changed, i.e. the steps of the chain must be adapted to fit the length of the input or output.
Treestructured RNNs. Further, RNNs can be enhanced to model data with more complex structures suited for downstream tasks. For example, treestructured RNNs (TreeRNNs, Figure 2
c), have been used to classify the sentiment of sentences
pang2002thumbs given an associated binary tree representing the sentence structure tai2015improved ; socher2011parsing . In this case, a leaf of the tree maps to a word of the sentence, an internal node corresponds to a multiword phrase. To process this structure, the cell function scans the tree recursively, starting from leaf nodes until reaching the root. At the node , it computes the state , where is the input to the node, and are the states of its left and right children, respectively. As the tree structure vary from example to example, the dataflow graph of a TreeRNN is highly dynamic.Graphstructured RNNs. Similarly, RNNs can be extended to compute over more general graphs, such as Nary trees or graphs (Figure 2d), as long as their parameters are learnable. In fact, various NNs have been developed toward having more dynamic workflows liang2016semantic ; tai2015improved , and proven quite effective because of their ability to encode structured information. While we take RNNs as examples for explanation, we note there are many other dynamic NNs in the literature or production with their dynamics reflected in various perspectives: variably sized inputs/outputs sutskever2014sequence ; bahdanau2014neural ; elman1990finding ; hochreiter1997long ; dyer2015transition ; buckman2016transition , variably structured inputs/outputs socher2011parsing ; tai2015improved ; liang2016semantic , or with nontrivial inference algorithms graves2006connectionist ; zheng2015conditional ; gormley2015approximation ; kong2015segmental .
2.2 Programming Dynamic Dataflow Graphs
As the assumption in §2 no longer holds for dynamic structures, static dataflow graphs in their original form cannot be used. There are currently two remedies to this problem: expanding the graph programming language to allow it to explicitly include controls structure necessary to implement these applications, or forgo the efficiency gains afforded by static dataflow graphs and instead use a dynamic declaration framework that reconstructs the graph for every training example. We explain below and summarize them in Figure 2.
Static declaration. Static unrolling abadi2016tensorflow is a standard way to express sequence RNNs with fixed steps. To handle variablelength data, it declares an RNN that has number of steps equal with the length of the longest sequence in the dataset. It then appends zeros at the end of other sequences to have equal length, and feeds them in batches to the dataflow graph for computation. Static unrolling enables batched computation of multiple sequences, but obviously results in substantial unnecessary computation.^{1}^{1}1It is also possible to split sentences into several buckets of different lengths, which alleviates this problem somewhat but adds some degree of code complexity and is not a fundamental solution. Dynamic unrolling implements basic control flow functionality within static graphs, allowing for the declaration of graph operators similar to while
loops. At each iteration of the training, the cell function of the RNN will be executed a conditional number of times determined at runtime by the length of the longest sentence in the batch. It then pads the sequences in the batch and perform batched computation, it waste computational resources. Notably, both of these methods essentially cannot support more complex structures than sequences.
Dynamic declaration. Dynamic declaration is able to express dynamically varying dataflow graphs, by creating a unique dataflow graph for each sample according to its associated structure. It however requires users to explicitly write (more) code to build a dataflow graph for each input graph, which is nontrivial for graphs with sophisticated structures. As the dataflow graphs are always changing, it can hardly benefit from wellestablished dataflow graph optimization techniques (§3.5) – we will have to perform graph processing/optimization for each dataflow only for a single sample, but incorporating the optimization itself has an overhead. More importantly, as we are unable to naturally batch the computation of different sample, singleinstance training would be very inefficient in the absence of batched computation. At the backend, since a dataflow graph needs to be constructed per sample, the overhead is linearly increasing with the number of samples, and sometimes yields downgraded performance looks2017deep (§5.2), even for frameworks with optimized graph construction implementations neubig2017dynet .
Tensorflow Fold looks2017deep and DyNet neubig2017fly go one step further and perform dynamic batching for dynamic dataflow graphs. Fold turns dynamic dataflow graphs into a static control flow graph to enable batched execution, but introduces a complicated functional programminglike languages and a large graph preprocessing overhead. DyNet proposes an autobatching strategy that searches for batching opportunities by profiling every finegrained operator, while this step itself has nonnegligible overhead, and loses the opportunities of graphlevel optimizations. There are also some “imperative” frameworks, such as PyTorch pytorch and Chainer chainer that allow users to construct dynamic NNs but performs instant evaluation of each user expression. As model construction and execution are coupled, they are usually difficult to perform dynamic batching or graph optimization. Overall, they are still far from efficient when handling dynamic NNs. We next describe our proposed vertexcentric programming model to overcome the aforementioned limitations.
3 Cavs Design and Optimization
Our motivation comes from several key principles ML developers usually comply with to ensure the feasibility and learnability of the model during their design of dynamic NNs. We note most dynamic NNs are designed to exhibit a recursive structure (e.g. sequence RNN, TreeRNN), or a combination of static and recursive structures (e.g. LRCN donahue2015long ; andreas2016neural , attention xu2015show ), or even a combination of different recursive structures (e.g. encoderdecoder RNNs sutskever2014sequence ). Within one such structure, a function is dynamically applied over instancespecific graphs, and every vertex of the graph usually interacts in a same way with it neighboring vertices following the function. The computational function itself, however, is usually static and parameterized by fixed learnable parameters.
This observation motivates us to design a new programming model, called Cavs, that combines the best of dataflow graphs with the vertexcentric model in graph computing. As a comparison, we present the workflow of Cavs in Figure 1c. For clarity, we will use the following terminology and notation in the rest of the paper: we call the instancespecific structure associated with the input sample as an input graph, and notate it as , and a node in that graph as a vertex, to be distinguished from a dataflow graph and the nodes (which are usually operators or variables) therein. Figure 3 illustrates the concept of this vertexcentric programming model. To describe an aforementioned dynamic structure, different from dynamic declaration, which requires users to manually declare dataflow graphs for each sample according to its associated graph, Cavs instead directly takes it as an input argument. To be aware of what computation shall be performed, Cavs requires users to implement a simple vertex function by “thinking like a vertex”, informing the framework how one vertex in a dynamic NN will interact with its connected vertices (if these is any). In , users can utilize conventional DL operators to assemble a symbolic construct that will be evaluated dynamically following the structure of , while Cavs will ensure the correctness and efficiency. Therefore, a vertex function , together with an input graph , implicitly encodes a recurrent dataflow graph, which maps to a subgraph of the implicit full dataflow graph of the model that may needs to be explicitly declared in traditional programming models. For convenience of notations, we will call any part of the structure that cannot be encoded by and as external to , and vice versa. Cavs allows users to connect any external static dataflow graph to a dynamic structure encoded by to express various model architectures (e.g. connecting a CNN to an RNN), or declare multiple vertex functions for different structures, and connect them appropriately to express more complex models (e.g. an encoderdecoder LSTM network).
While it is still necessary to create an I/O function to read input graphs for each sample, this must be done in any models, and only once before training commences, which means that it can be shared across epochs or even training runs. Cavs no longer requires users to construct the full dataflow graphs for each sample by themselves. As repeated graph construction is bypassed, the overhead will also be avoided. With this vertexcentric model, Cavs transforms the problem of evaluating multiple dataflow graphs with different structures
looks2017deep ; neubig2017fly into a simpler form – scheduling the execution of the vertex functions following the input graphs. For the later problem, we can easily batch the execution of over multiple vertices at runtime (§3.2), leveraging the batching computational capability of modern hardware. Moreover, as the vertex function itself maps to a static symbolic dataflow graph, it is open and can benefit from various graph optimization techniques originally developed for static declaration, such as kernel fusion, streaming, and our proposed lazy batching, which might not be effective in the scenario of dynamic declaration. We next describe Cavs’ APIs.3.1 Programming Interface
Besides the generic math operators used to declare the computation, Cavs exposes four symbolic APIs for users to specify how the messages shall be passed between vertices in their vertex functions: gather, scatter, pull, push.

gather(child_idx): gather accepts an index of the child vertices, gets the child content from gather/scatter buffer and returns a list of symbols that represent the output of these vertices.

scatter(op): scatter is a reverse API of gather, and has a symbol op as its input argument. Scatter will set the output of current vertex to gather/scatter buffer.
gather and scatter resemble the GAS model in graph computing gonzalez2012powergraph – both are vertexcentric APIs that help users express the overall computational patterns by thinking locally like a vertex: gather receives messages from dependent vertices, while scatter updates information to parent vertices. But note several key differences: (1) gather and scatter here are fully symbolic – gather allows backpropagation through it; (2) In graph computing, all nodes interact with their connected nodes in the same way following a userspecified apply function, while in dynamic NNs, a vertex usually interacts differently with its different child vertices, specified by the symbolic programs (between the call of gather and scatter) in the vertex function; (3) In graph computing, a vertex of a graph always interacts with other vertices of this graph, while in DL, the vertex of a dynamic NN usually takes input from not only the internal of the structure expressed by and (internal data path in Figure 3), but also from the external of it, e.g. a step in an RNN can take inputs from a CNN feature extractor or some external I/O (external data path in Figure 3). In this case, gather and scatter are insufficient to express such semantics. Cavs therefore provides another two APIs:

pull(): pull grabs inputs from the external of the current dynamic structure, e.g. another NN, or some I/O.

push(op): push is thus the reverse of pull that sets the output of the current vertex as op. If this vertex is pulled by others, the content of op will be returned.
With appropriate indexing, push and pull connect a vertex inside a dynamic structure expressed by to other connectors external to , such as another dynamic structure, or another static dataflow graph. With these four APIs, we present in Figure 4 an example user program how the ary childsum TreeLSTM tai2015improved can be simply expressed by using them and other mathematical operators.
Expressiveness. With these four APIs, Cavs can be seen as a middle ground between static and dynamic declaration: In the best case, the model can be easily represented by a single vertex function plus input graphs. While in the worse case scenario, that every sample has a unique input graph while every vertex in the graph has a unique way to interact with its neighboring vertices, Cavs reduces to dynamic declaration that one has to define a vertex function for each vertex of input graphs. However, dynamic NNs in this scenario are very rare and usually not preferred because of the difficulty of design, programming and learning.
3.2 Scheduling
Once users define the vertex function and launch the execution, the Cavs scheduler arranges the evaluation of over the input graphs. Given , Cavs’s scheduler follows designed policies to efficiently perform backpropagation for all samples and their associated graphs .
Backpropagation. Cavs performs backpropagation hinton2006reducing as follows. For a sample with its input graph , the scheduler starts the forward pass from the input vertices of , and proceeds following the direction indicated by the edges in : at each substep, the scheduler figures out the next activated vertex in , and evaluates at this vertex following the symbolic programs in . It then marks this vertex as evaluated
, and proceeds with the next activated vertex until reaching a terminal vertex (e.g. the loss function). A vertex of
is activated if and only if all its dependent vertices have been evaluated. The backward pass is continued right after the forward. The scheduler first resets the status of all vertices as not evaluated, then scans the graph in a reverse direction, starting from the ending point of the forward pass. It similarly figures out the next activated vertex, but applies another function , which is the backward function of and automatically derived by Cavs via autodifferentiation (§3.4), until all vertices have been evaluated in backward.To train a NN to convergence, the above process has to be iterated by the scheduler over all samples and their associated graphs , for many epochs. Instead a sequential execution, Cavs designs a batching policy to perform batched computation, considering the fact that evaluating a set of same arithmetic operations together is significantly faster than the sequential evaluation of each of them.
Batching policy. Given a data batch and associated graphs , this policy groups multiple vertices and then performs batched evaluation of in order to reduce kernel launches and exploit parallelism. Algorithm 1 details this policy. Specifically, the scheduler divides the forward evaluation of (a batch of) graphs into multiple steps. At each step , it analyzes at runtime and determines a set contains all activated vertices in . It then evaluates over these vertices by creating a batching execution task, with the task ID set to ^{2}^{2}2Without ambiguity, we use to denote both the set of vertices to be batched together, as well as the batching task itself.. The task is passed to a graph execution engine, which will further optimize the execution and conduct the actual computation (§3.5). Meanwhile, the scheduler records the information of this task by pushing into a stack . At each step of the backward, the scheduler pops out an element from , creates a corresponded backward batching task – the execution engine will evaluate the derivative function over vertices in , until all vertices of are evaluated.
We note the batching policy plays a similar role as the dynamic batching in Fold looks2017deep and DyNet neubig2017dynet in the scenario of dynamic declaration. However, Cavs determines how to batch fully dynamically during runtime using simple breadthfirst search with negligible cost (instead of analyzing graphs before every iteration of the execution). We next describe an improved management management strategy based on this batching policy.
3.3 Memory Management
In static declaration abadi2016tensorflow ; neubig2017dynet , a symbol in the user program usually corresponds to a tensor object, with its shape inferred from the program and batch size specified in advance. The framework usually preallocates a continuous storage on memory for each tensor and fixes it throughout runtime. However, in Cavs, as each batching task is determined at runtime and not visible before execution, its memory management exhibits more complexities. For the batched computation to be efficient, Cavs must guarantee the inputs and intermediate states during the evaluation of over a group of runtimedetermined vertices coalescing in memory. If we adopt the aforementioned strategy, for each operation in , Cavs has to index each slice of its input tensor (which may be scattered in different places) and rearrange them as a continuous memory block, which might cause nontrivial overhead.
Cavs proposes a novel data structure dynamic tensor to address this challenge (Figure 6). A dynamic tensor is a wrapper of a multidimensional array abadi2016tensorflow ; walt2011numpy that contains four main members: shape, bs, a pointer p to a chunk of memory, and offset. shape is an array of integers representing the specific shape of the tensor excluding the batch dimension. It can be inferred from the user program and set before execution. The batch size bs is implemented as a placeholder in Cavs, with its value dynamically set by the scheduler at runtime at the beginning of a batching task. For each dynamic tensor, Cavs will preallocate a chunk of continuous memory, and point p to its starting address. This memory block is often very large and not fixedsized – it can be further extended if needed. To access a dynamic tensor, the execution engine moves p forward with the value specified in offset, and reads/writes number of elements equal to . Therefore, bs together with offset provide a view of the tensor, and the state of the tensor will vary based on their values. Given a vertex function , Cavs creates dynamic tensors for each nonparameter symbol in , and also as their gradients, while it creates static tensors for model parameters.
Figure 5 illustrates how the memory is assigned during the forward pass by simply manipulating dynamic tensors. In particular, in a training iteration, for a batching task , the scheduler sets bs of all to (the number of vertices in ). The execution engine then performs batched evaluation of each expression in one by one. For an expression ^{3}^{3}3Without losing generality the userdefined expressions can be arbitrary, e.g. with more than one argument or return values., the engine first reads (the dynamic tensor of the RHS symbol ) by offsetting by then reading a block of elements, and presents it as a tensor with batch size and other dimensions specified in . It then applies batched computational kernels of the operator op over this continuous block, and writes the results to (the dynamic tensor of the LHS symbol ) on the continuous block in between . Upon the completion of , the scheduler increases the offset of all by , respectively. It then starts the next batching task until has been evaluated at all vertices of . Hence, intermediate results generated in each batching task at forward pass are stored continuously in the dynamic tensors, and their offsets are recorded.
The scheduler then starts the backward pass. It initializes for each as . Since the backward execution follows an exactly reverse order of the forward pass (Algorithm 1), the intermediate results generated during forward can be easily accessed by decreasing the offset of . Specifically, for each batching task popped out from , the execution engine sets bs of all dynamic tensors to , and for each and , decreases their offset by . For an expression in that corresponds to in (see §3.4), the engine reads the current states of (which are continuous on memory) and performs batched execution of grad_op. Different from the forward pass, the gradient result will be added to the current state of instead of overwriting.
At the entrance of , the vertices in need to interact with its dependent vertices in previous to gather their outputs as inputs (L3 of Figure 4), or pull inputs from the external (L5 of Figure 4). Cavs maintains memory buffers to enable this (Figure 5). The memory buffers are keyvalue stores where the key is the vertex ID and the value is a tensor slice corresponding to the results of the scattered symbol at vertex (batch size is 1). Cavs provides a query function IndexBuffer(op, key) that returns the value in op’s corresponded buffer given a key. During the execution, a gather expression will trigger memory movements: for each , the scheduler figures out the vertex ID of its child vertex in the input graph with index child_idx; it then indexes the content in gatherBuffer with the vertex ID as key, and copies and writes it into as a slice. Similarly, at the exit of , a scatter expression will split the current state of into a few slices, and move them to the gatherBuffer for its parent nodes to gather. The push/pull work similarly with gather/scatter, but over the pushBuffer and pullBuffer, respectively, to communicate messages with the external.
Algorithm 2 summarizes the memory management process during forward pass. With this strategy, Cavs guarantees memory continuity for any batched computation in . Compared to dynamic batching in DyNet, Cavs performs memory movement only at the entrance and exit of , instead of for each expression (operator), it therefore significantly reduces overhead by memory operations (§5.3).
3.4 Autodifferentiation
Cavs by nature supports autodifferentiation. Given a vertex function it derives following the autodifferentiation rules: for each math expression such as in , Cavs generates a corresponded backward expression in as . For the four proposed operators, with the memory management strategy described above, we note scatter is the backward operator of gather in the sense that if gather collects inputs from gatherBuffer previously written by scatter at the forward pass, a scatter needs to be performed to write the gradients to the gatherBuffer for its dependent vertices to gather at the backward pass. Hence, for an expression like in , Cavs will generate a backward expression in . Similarly, the gradient operator of scatter is gather. The same autodifferentiation rule applies for push and pull as well.
3.5 Graph Execution Engine
Benefiting from the vertexcentric representation, the vertex function essentially defines a (small) static dataflow graph that is open to various graph execution optimization techniques (which might not be the case in dynamic declaration). We next discuss three proposed optimizations on Cavs’ execution engine for improved performance.
Lazy batching and streaming^{4}^{4}4Streaming is a borrowed terminology from CUDA programming which means executing different commands concurrently or out of order with respect to each other on different GPU streams. As Cavs’ optimization strategies are agnostic to the lowlevel hardware, we use streaming interchangeably with multithreading if the underlying computing hardware is CPU.. In addition to batched execution of , the lazy batching and streaming explore potential parallelism for a certain group of finergrained operators in or called lazy and eager operators.
Definition.
An operator in () is a lazy operator if at the forward (backward) pass, for , the evaluation of () at any parent (dependent) vertex of does not rely on the evaluation of at . It is an eager operator if the evaluation at does not rely on the evaluation of () at any dependent (parent) vertices of .
In Cavs, figuring out eager and lazy operators in and is straightforward given the following proposition:
Proposition.
Denote () as the dataflow graph encoded by (), and () as corresponded nodes of the gather and scatter operator, respectively. A node that has as its dependent and is not on any path from to is a lazy operator. A node that has as its ancestor and is not on any path from to is an eager operator.
Figure 7 illustrates a dataflow graph of the vertex function of TreeLSTM, with eager and lazy operators colored. A property of them is that their evaluation is not fully subject to the dependency reflected by the input graph . For instance, the pull operator in Figure 7 is eager and can be executed in prior – even before has been evaluated at the vertices gather tries to interact with; the push operator is lazy, so we can defer its execution without impacting the evaluation of at parent vertices. Similarly, in , the gradient derivation for model parameters are mostly lazy – their execution can be deferred as long as the gradients of hidden states are derived and propagated in time. Cavs leverages this property and proposes the lazy batching strategy. It defers the execution of all lazy operators in and until all batching tasks has finished. It then performs a batched execution of these lazy operators over all vertices of . These operators includes, but is not limited to, the push operator that are doing memory copy, and the math operators for computing gradients of the model parameters. Lazy batching helps exploit more parallelism for the execution of lazy operators and significantly reduces kernel launches. Empirically lazy batching brings overall improvement (see §5.3).
On the other hand, we are unable to “eagerly” batch eager operators, as their execution over some vertices relies on knowing the detailed memory location of all intermediate results in advance, a condition which is not satisfied in Cavs where memory is dynamically assigned. To leverage the exhibited parallelization opportunity between eager operators and the operators on the path from gather to scatter (Figure 7), Cavs proposes a streaming strategy that pipelines the execution of these two groups of operators. It allocates two streams, and puts the eager operators on one stream, and the rest (excluding lazy operators) on the other. Hence, independent operators in two streams run in parallel, while for those operators that depend on an eager operator, this dependency is respected by synchronization barriers (see Figure 7). It is also possible to parallelize independent paths from to on the graph, but we find it does not yield further improvement.
Automatic kernel fusion. Since Cavs abstracts out a static dataflow graph encoded by that will be dynamically evaluated elsewhere, we can replace the original with an optimized one that runs more efficiently, as long as it accepts the same input and produces the same output.
Particularly, given , before execution, Cavs will run a fusion detector gysi2015stella to scan its corresponded dataflow graph and report all fuseable subgraphs therein, i.e. all nodes in a fuseable subgraph can be fused as a single operator that behaves equivalently but takes less execution time (e.g. with fewer kernel launches and I/O, or faster computation). Currently, we only detect groups of directly linked elementwise operators, such as , as shown in Figure 7, and we use a simple unionfind algorithm to detect the largest possible fuseable subgraphs. Given a fuseable subgraph, Cavs adopts de facto automatic code generation techniques quinlan2000rose ; dave2009cetus ; ragan2013halide ; nvrtc to generate lowerlevel kernel codes as an implementation for it. Replacing the original fuseable subgraphs with fused operators during execution is beneficial in many aspects: (1) it reduces the number of kernel launches; (2) on some devices such as GPUs, kernel fusion transform device memory access into faster device registers access. We empirically report another improvement with automatic kernel fusion (§5.3).
4 Implementation
We implement Cavs as a pluggable C++ library that can be integrated with existing DL frameworks to provide or enhance their support for dynamic NNs. We next briefly discuss implementation details. For clarity, we assume the host framework is composed of three major layers (which is the case for most popular frameworks Bergstra:2011:NIPSW ; abadi2016tensorflow ; neubig2017dynet ): (1) a frontend that provides deviceagnostic symbolic programming interface; (2) an intermediate layer that implements the core execution logic; (3) a backend with devicespecific kernels for all provided operators.
Frontend. Cavs provides a base class GraphSupport in addition to conventional operators and the four proposed APIs (§3.1). Users are required to instantiate it by providing a symbolic vertex function – therefore an instantiation of GraphSupport represents a single dynamic structure. To construct more complex structures (e.g. encoderdecoder LSTM sutskever2014sequence , LRCN donahue2015long )), users employ push and pull to connect this dynamic structure to external structures.
Intermediate Layer. At the intermediate layer, Cavs will create a unique scope abadi2016tensorflow , and generates a small dataflow graph for each instantiation of GraphSupport, connecting them appropriately with other parts of the model according to user programs. Cavs implements its core runtime logic at this layer, i.e. the scheduler, the memory management, and the graph execution engine, etc. During execution, the execution engine first analyzes the received dataflow graphs and incorporates described optimization in §3.5. The scheduler then instructs the system to read training samples and their associates graphs (e.g. adjacency matrices). It starts training by submitting batching tasks to the execution engine and assigning memory accordingly.
Backend. Following common practice abadi2016tensorflow ; neubig2017dynet ; caffe2 , Cavs puts devicespecific kernel implementations for each supported operator at this layer. Each operator implementation is a function that takes as inputs static tensors and produces static tensors as outputs – therefore the higherlayer logic, i.e. how the computation is scheduled or how the memory is assigned are invincible to this layer. Cavs will reuse the native operator implementations from the host framework, while it provides optimized implementations for the four proposed primitives (gather, scatter, pull, push). Specifically, gather and pull index different slices of a tensor and puts them together continuously on memory; scatter and push by contrast splits a tensor along its batch dimension, and copy different slices to different places. Cavs implements a customized memcpy kernel for there four operators so that copying multiple slices from (or to) different places is performed within one kernel, to further reduce kernel launches.
5 Evaluation
In this section, we evaluate Cavs on training different NNs across multiple datasets, obtaining the following major findings: (1) Cavs has little overhead: when training static NNs that can be by nature batched, Cavs demonstrates equal performance with other DL systems. On several NNs with notably difficulttobatch structures, Cavs outperforms all existing frameworks by a large margin; (2) We confirm the graph construction overhead is substantial in both Fold looks2017deep and dynamic declaration neubig2017dynet , while Cavs bypasses it by loading input graphs through I/O; (3) We verify the effectiveness of our proposed design and optimization via ablation studies, and discuss Cavs’ advantages over other stateoftheart DL systems for dynamic dataflow graphs.
Environment. We perform all experiments in this paper on a single machine with an NVIDIA Titan X (GM200) GPU, a 16core (32 threads) CPU, and CUDA toolkit 8.0 and cuDNN v6 installed. As modern DL models are mostly trained using GPUs, we focus our evaluation on GPUs, but note Cavs’ design and implementation do not reply on a specific type of device. We borrow the implementations of most mathematical operators from TensorFlow v1.2, while we implement the four proposed operators and other system modules by ourselves. We mainly compare Cavs to TensorFlow v1.2 abadi2016tensorflow with XLA xla and its variant Fold looks2017deep , as well as DyNet v2.0 neubig2017dynet with autobatching neubig2017fly , as they have reported better performance than other frameworks pytorch ; chainer on dynamic NNs. We focus on metrics for system performance, e.g. the average time to scan one epoch of data. Cavs produces exactly the same numerical results with other frameworks, hence the same perepoch convergence^{5}^{5}5The code of Cavs will be released along with a the next major release of the DyNet project: http://dynet.io/..
Models and dataset. We experiment on the following models with increasing difficulty to batch: (a) FixedLSTM language model (LM): a static sequence LSTM with fixed steps for language modeling sundermeyer2012lstm ; sutskever2014sequence ; zaremba2014recurrent . We train it using the PTB dataset ptb that contains over 10K different words. We set the number of steps as 64, i.e. at each iteration of training, the model takes a 64word sentence from the training corpus, and predicts the next word of each word therein. Obviously, the computation can be by nature batched easily, as each sentence has exactly the same size. (b) VarLSTM LM: that accepts variablelength inputs. At each iteration the model takes a batch of natural sentences with different length from PTB, and predicts the next words; (c) TreeFC: the benchmarking model used in looks2017deep with a single fullyconnected layer as its cell function. Following the same setting in looks2017deep , we train it over synthetic samples generated by their code treefccode – each sample is associated with a complete binary tree with 256 leaves (therefore 511 vertices per graph); (d) TreeLSTM: a family of dynamic NNs widely adopted for text analysis liang2016semantic ; vinyals2015grammar . We implement the binary childsum TreeLSTM model in tai2015improved
, and train it as a sentiment classifier using Stanford sentiment treebank (SST) dataset
socher2013recursive , which contains 8544 training sentences in which the longest sentence has 54 words. Each sentence is associated with a human annotated grammar tree.5.1 Overall Performance
We first verify the viability of our design on the easiesttobatch case: FixedLSTM language model. We compare Cavs to the following three strong baselines: (1) CuDNN chetlur2014cudnn : a CuDNNbased fixedstep sequence LSTM, which is highly optimized by NVIDIA using handcrafted kernels and stands as the best performed implementation on NVIDIA GPUs; (2) TF: the official implementation of FixedLSTM LM in TensorFlow repository TFfixedLSTM based on static declaration; (3) DyNet: we implement a 64step LSTM in DyNet based on dynamic declaration – we declare a dataflow graph per sample, and train with the autobatching neubig2017fly enabled; (4) Cavs with batching policy, and all input samples have a same input graph – a 64node chain. We train the model to converge, and report the average time per epoch in Figure 8(a)(e), where in (a) we fix the hidden size of the LSTM unit as 512 and vary the batch size , and in (e) we fix and vary . Empirically, CuDNN performs best in all cases, but note it is highly inflexible. Cavs performs slightly better than TF in various settings, verifying that our system has little overhead dealing with fully static graphs, though it is specialized for dynamic ones. We also conclude from Figure 8 that batching is essential for GPUbased DL: is nearly one order of magnitude faster than regardless of used frameworks. For Cavs, the batching policy is 1.7x, 3.8x, 7.0x, 12x, 15x, 25x, 36x faster than the serial policy at , respectively.
Next, we experiment with VarLSTM, the most commonly used RNN for variablelength sequences. We compare the following three implementations (CuDNNbased LSTM cannot handle variablelength inputs): (1) TF: an official TensorFlow implementation based on the dynamic unroll approach described in §2.2; (2) DyNet: an official implementation from DyNet benchmark repository based on dynamic declaration DyNetVarLSTM ; (3) Cavs: where each input sentence is associated with a chain graph that has number of vertices equal to the number of words. We vary and , and report the results in Figure 8(b)(f), respectively. Although all three systems perform batched computation in different ways, Cavs is constantly 23 times faster than TF, and outperforms DyNet by a large margin. Compared to TF, Cavs saves computational resources. TF dynamically unrolls the LSTM unit according to the longest sentence in the current batch, but it cannot prevent unnecessary computation for those sentences that are shorter than the longest one.
We then turn to TreeFC, a dynamic model for benchmarking. Since vanilla TensorFlow is unable to batch its computation, we compare Cavs to (1) DyNet and (2) Fold, a specialized library built upon TensorFlow for dynamic NNs, with a depthbased dynamic batching strategy. To enable the batching, it however needs to preprocess the input graphs, translate them into intermediate representations and pass them to lowerlevel TensorFlow control flow engine for execution. We report the results in Figure 8(c)(g) with varying and , respectively. For all systems, we allocate a single CPU thread for graph preprocessing or construction. Cavs shows at least an order of magnitude speedups than Fold and DyNet at (). Because the size of the synthetic trees is large, one major advantage of Cavs over them is the alleviation of substantial graph preprocessing/construction overhead. With a single CPU thread, Fold takes even more time on graph preprocessing than computation (§5.3).
Finally, we compare three frameworks on TreeLSTM in Figure 8(d)(h): Cavs is 810x faster than Fold, and consistently outperforms DyNet. One difference in this experiment is that we allocate as many CPU threads as possible (32 on our machine) to accelerate graph preprocessing for Fold, otherwise it will take much longer time. Further, we note DyNet performs much better here than on TreeFC, as the size of the input graphs in SST (maximally 52 leaves) is much smaller than the synthetic ones (256 leaves each) in TreeFC experiments. We observe DyNet needs more time on graph construction for large input graphs, and DyNet’s dynamic batching is less effective on larger input graphs, as it has to perform frequent memory checks to support its dynamic batching, which we will discuss in §5.3.
5.2 Graph Construction and Computation
In this section, we investigate the graph construction overhead in Fold and DyNet. To batch computation of different graphs, Fold analyzes the input graphs to recognize batchable dynamic operations, then translates them into intermediate instructions, with which, TensorFlow generates appropriate control flow graphs for evaluation – we will treat the overhead caused in both steps as Fold’s graph construction overhead. DyNet, as a typical dynamic declaration framework, has to construct as many dataflow graphs as the number of samples. Though DyNet has optimized its graph construction to make it lightweight, the overhead still grows with the training set and the size of input graphs. By contrast, Cavs takes constant time to construct a small dataflow graph encoded by , then reads input graphs through I/O. To quantify the overhead, we separate the graph construction from computation, and visualize in Figure 9(a) how it changes with the average number of leaves (graph size) of input graphs on training TreeFC, with fixed . We compare (1) Cavs (2) Fold1 which is Fold with one graph processing thread and (3) DyNet. We plot for one epoch, both the (averaged) absolute time for graph construction and it percentage of the overall time. Clearly we find that all three systems take increasingly more time when the size of the input graphs grows, but Cavs, which loads graphs through I/O, causes the least overhead at all settings. In terms of the relative time, Fold unfortunately wastes 50% at 32 leaves, and 80% when the tree has 1024 leaves, while DyNet and Cavs take only and , respectively.
We also wonder how the overhead is related with batch size when there is fixed computational workload. We report in Figure 9(b) the same metrics when training TreeLSTM with varying . We add another baseline Fold32 with 32 threads for Fold’s graph preprocessing. As Fold1 takes much longer time than others, we report its time at here (instead of showing in Figure 9): 1.1, 7.14, 31.35, 40.1, 46.13, 48.77. Except , all three systems (except Fold1) take almost constant time for graph construction in one epoch, regardless of , while Fold32 and DyNet take similar time, but Cavs takes 20x less. Nevertheless, at the percentage scale, increasing makes this overhead more prominent, because larger batch size yields improved computational efficiency, therefore less time to finish one epoch. This, from one perspective, reflects that the graph construction is a main obstacle that grows with the number of training samples and prevents the efficient training of dynamic NNs in existing frameworks, while Cavs successfully overcomes this barrier through its design.
Apart from the graph construction we report in Table 1 the computationonly time – Cavs demonstrates maximally 5.4x/9.7x and 7.2x/2.4x speedups over Fold/DyNet on TreeFC and TreeLSTM, respectively. Besides less system overhead, the advantages stem from two main sources: an optimized graph execution engine, and a bettersuited memory management strategy, which we investigate next.
# leaves  Comp. time (s)  Speedup  Comp. time (s)  Speedup  
32  0.58 / 3.1 / 4.1  5.4x / 7.1x  1  76.2 / 550 / 61.6  7.2x / 0.8x 
64  1.1 / 3.9 / 8.0  3.7x / 7.5x  16  9.80 / 69 / 12  7.0x / 1.2x 
128  2.0 / 6.2 / 16.0  3.0x / 7.9x  32  6.15 / 43 / 9.9  7.0x / 1.6x 
256  3.9 / 10.6 / 33.7  2.7x / 8.7x  64  4.1 / 29 / 7.4  7.2x / 1.8x 
512  8.0 / 18.5 / 70.6  2.3x / 8.9x  128  2.9 / 20.5 / 5.9  7.1x / 2.0x 
1024  15.8 / 32.4 / 153  2.1x / 9.7x  256  2.3 / 15.8 / 5.4  7.0x / 2.4x 
5.3 Ablation Studies
Graph Execution Engine. To reveal how much each optimization in §3.5 contributes to the final performance, we disable lazy batching, fusion and streaming in Cavs and set this configuration as a baseline (speedup = 1). We then turn on one optimization at a time and record how much speedup it brings. We train FixedLSTM and TreeLSTM, and report the averaged speedups one computationonly time in one epoch over the baseline configuration in Figure 10, with but varying . Lazy batching and fusion consistently deliver nontrivial improvement – lazy batching is more beneficial with a larger while fusion is more effective at smaller , which are expected: lazy batching mainly parallelizes matrixwise operations (e.g. matmul) commonly with our higher complexity, while fusion mostly works on elementwise operations with a linear complexity with gustafson1988reevaluating .
Streaming, compared to the other strategies, is less effective on TreeLSTM than on FixedLSTM
, as we have found the depth of the input trees in SST exhibit high variance, i.e. some trees are much deeper than others. In this case, many batching tasks only have one vertex to be evaluated. The computation is highly fragmented and the efficiency is bounded by kernel launching latency. Lazy batching and fusion still help as they both reduce kernel launches (§
3.5). Streaming, which tries to pipeline multiple kernels, can hardly yield obvious improvement.Memory Management. Cavs’ performance advantage also credits to its better memory management that reduces memory movements while guarantees memory continuity.
Quantitatively, it is difficult to compare Cavs to Fold, as Fold relies on TensorFlow where memory management is highly coupled with other system aspects. Qualitatively, we find Cavs requires less memory movement (e.g. memcpy) during dynamic batching. Built upon the tf_while operator, whenever Fold performs depthbased batching at , it has to move all the contents of nodes in the dataflow graphs at depth
to a desired location, as the control flow does not support crossdepth memory indexing. This results in redundant memcpy, especially when the graphs are highly skewed. By contrast, Cavs only copies contents that are necessary to the batching task. DyNet has a specialized memory management strategy for dynamic NNs. Compared to Cavs, it however suffers substantial overhead caused by repeated checks of the memory continuity – whenever DyNet wants to batch operators with same signatures, it checks whether their inputs are continuous on memory
neubig2017fly . The checking overhead increases with and is more prominent on GPUs. Thanks to the simplicity of both systems, we are able to profile the memoryrelated overhead during both training and inference, and separate it from computation. We compare them on TreeLSTM, and report the breakdown time per epoch in Table 2 under different . We observe the improvement is significant (2x  3x) at larger , especially during inference where DyNet has its continuity checks concentrated.Memory operations (s) (Cavs / DyNet)  Computation (s) (Cavs / DyNet)  
Train  Inference  Train  Inference  
16  1.14 / 1.33  0.6 / 1.33  9.8 / 12  2.9 / 8.53 
32  0.67 / 0.87  0.35 / 0.87  6.1 / 9.8  1.9 / 5.35 
64  0.39 / 0.6  0.21 / 0.6  4.0 / 7.4  1.3 / 3.48 
128  0.25 / 0.44  0.13 / 0.44  2.9 / 5.9  0.97 / 2.52 
256  0.17 / 0.44  0.09 / 0.44  2.3 / 5.4  0.77 / 2.58 
Others. Despite system advantages, we also try to investigate whether Cavs, as an interface, simplifies user programs (though we do not claim as a contribution). We compare Cavs to Fold and DyNet in terms of the lines of code (LoC) needed to create a few notable dynamic NNs, including VarLSTM, TreeLSTM, and multilayer sequence LSTM, with Python as the host language. If only for model declaration, Fold in general has 3.5x more LoC than Cavs, while DyNet has slightly more LoC than Cavs because of the function to repeatedly declare graphs.
6 Related Work
In addition to §2, we discuss some other related works.
Graph execution optimization. Optimizing the execution of DL dataflow graphs comes in mainly two ways: better operator implementations or optimizing the execution of (sub)graphs. As Cavs is implemented as a plugin to enhance existing frameworks, it benefits from any improved implementations of specific operators (e.g. cuDNN) grave2016efficient ; MKLDNN ; chetlur2014cudnn ; eigenweb . In addition, Cavs has optimized implementations for its proposed four primitives (gather/scatter/pull/push). At the graph level, a variety of welldeveloped techniques from other areas, such as kernel fusion, common subexpression elimination, and constant folding, have been adapted and applied on speeding the computation of DL dataflow graphs abadi2016tensorflow ; chen2015mxnet ; caffe2 ; xla . They are usually incorporated after the graph declaration, but before the execution, so that the actual computation is conducted on an optimized graph other than the original one. However, these graph optimizations are less beneficial in dynamic declaration, in which the graph changes with the sample, and needs to be reprocessed and reoptimized every iteration, and may cause substantial overhead. On the contrary, Cavs separates the static vertex function from the dynamicvarying input graph, so it benefits from most of the aforementioned optimizations, as we have shown in §5.3. We draw insights from these strategies and reflect them in Cavs’ execution engine. We further propose lazy batching and streaming to exploit more parallelism exposed by our programming model.
Vertexcentric models. The vertexcentric programming model has been extensively developed in the area of graph computing malewicz2010pregel ; gonzalez2012powergraph ; chen2015powerlyra ; sundaram2015graphmat . Cavs draws insights from the GAS model gonzalez2012powergraph , but faces totally different challenges in system and interface design, such as expressiveness, scheduling for batched execution of different graphs, guaranteeing the memory continuity, etc., as we have discussed in §3.1.
7 Conclusion
We present Cavs as a vertexcentric programming interface as well as an efficient system for dynamic deep learning. Cavs represents a dynamic NN structure as static vertex functions and dynamic input graphs. It provides four novel APIs to allow users to easily program these types of NNs. With designed scheduling policy, memory management strategy, and graph execution optimizations, Cavs avoids substantial graph construction overhead suffered by dynamic declaration, and reports new stateoftheart system performance for various notable dynamic NN architectures.
References
 [1] Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. Tensorflow: A system for largescale machine learning. arXiv preprint arXiv:1605.08695 (2016).

[2]
Andreas, J., Rohrbach, M., Darrell, T., and Klein, D.
Neural module networks.
In
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
(2016), pp. 39–48.  [3] Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014).
 [4] BartholomewBiggs, M., Brown, S., Christianson, B., and Dixon, L. Automatic differentiation of algorithms. Journal of Computational and Applied Mathematics 124, 1 (2000), 171–190.
 [5] Bergstra, J., Bastien, F., Breuleux, O., Lamblin, P., Pascanu, R., Delalleau, O., Desjardins, G., WardeFarley, D., Goodfellow, I. J., Bergeron, A., and Bengio, Y. Theano: Deep Learning on GPUs with Python. In NIPSW (2011).

[6]
Buckman, J., Ballesteros, M., and Dyer, C.
Transitionbased dependency parsing with heuristic backtracking.
InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing; 2016 Nov 15; Austin, Texas, USA. Stroudsburg (USA): Association for Computational Linguistics (ACL); 2016. p. 231318.
(2016), ACL (Association for Computational Linguistics).  [7] Chen, R., Shi, J., Chen, Y., and Chen, H. Powerlyra: Differentiated graph computation and partitioning on skewed graphs. In Proceedings of the Tenth European Conference on Computer Systems (2015), ACM, p. 1.
 [8] Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., and Zhang, Z. Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv:1512.01274 (2015).
 [9] Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J., Catanzaro, B., and Shelhamer, E. cudnn: Efficient primitives for deep learning. arXiv preprint arXiv:1410.0759 (2014).
 [10] Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 (2014).
 [11] Cui, H., Zhang, H., Ganger, G. R., Gibbons, P. B., and Xing, E. P. Geeps: Scalable deep learning on distributed gpus with a gpuspecialized parameter server. In Proceedings of the Eleventh European Conference on Computer Systems (2016), ACM, p. 4.
 [12] Dave, C., Bae, H., Min, S.J., Lee, S., Eigenmann, R., and Midkiff, S. Cetus: A sourcetosource compiler infrastructure for multicores. Computer 42, 12 (2009).
 [13] Donahue, J., Anne Hendricks, L., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., and Darrell, T. Longterm recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE conference on computer vision and pattern recognition (2015), pp. 2625–2634.
 [14] Dyer, C., Ballesteros, M., Ling, W., Matthews, A., and Smith, N. A. Transitionbased dependency parsing with stack long shortterm memory. arXiv preprint arXiv:1505.08075 (2015).
 [15] DyNet Variable Length LSTM. https://github.com/neulab/dynetbenchmark.
 [16] Elman, J. L. Finding structure in time. Cognitive science 14, 2 (1990), 179–211.
 [17] Facebook. http://pytorch.org/.
 [18] Facebook Open Source. Caffe2 is a lightweight, modular, and scalable deep learning framework. https://github.com/caffe2/caffe2, 2017.
 [19] Gonzalez, J. E., Low, Y., Gu, H., Bickson, D., and Guestrin, C. Powergraph: Distributed graphparallel computation on natural graphs.
 [20] Google TensorFlow XLA. https://www.tensorflow.org/performance/xla/.
 [21] Gormley, M. R., Dredze, M., and Eisner, J. Approximationaware dependency parsing by belief propagation. arXiv preprint arXiv:1508.02375 (2015).
 [22] Grave, É., Joulin, A., Cissé, M., Grangier, D., and Jégou, H. Efficient softmax approximation for gpus. arXiv preprint arXiv:1609.04309 (2016).

[23]
Graves, A., Fernández, S., Gomez, F., and Schmidhuber, J.
Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks, 2006.
 [24] Guennebaud, G., Jacob, B., et al. Eigen v3. http://eigen.tuxfamily.org, 2010.
 [25] Gustafson, J. L. Reevaluating amdahl’s law. Communications of the ACM 31, 5 (1988), 532–533.
 [26] Gysi, T., Osuna, C., Fuhrer, O., Bianco, M., and Schulthess, T. C. Stella: A domainspecific tool for structured grid methods in weather and climate models. In High Performance Computing, Networking, Storage and Analysis, 2015 SCInternational Conference for (2015), IEEE, pp. 1–12.
 [27] Hinton, G. E., and Salakhutdinov, R. R. Reducing the dimensionality of data with neural networks. science 313, 5786 (2006), 504–507.
 [28] Hochreiter, S., and Schmidhuber, J. Long shortterm memory. Neural computation 9, 8 (1997), 1735–1780.
 [29] Intel Open Source Technology Center. Intel(r) math kernel library for deep neural networks (intel(r) mkldnn). https://github.com/01org/mkldnn, 2017.
 [30] Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093 (2014).
 [31] Kong, L., Dyer, C., and Smith, N. A. Segmental recurrent neural networks. arXiv preprint arXiv:1511.06018 (2015).
 [32] Krizhevsky, A., Sutskever, I., and Hinton, G. E. ImageNet Classification with Deep Convolutional Neural Networks. In NIPS (2012).
 [33] Liang, X., Shen, X., Feng, J., Lin, L., and Yan, S. Semantic object parsing with graph lstm. In European Conference on Computer Vision (2016), Springer, pp. 125–143.
 [34] Looks, M., Herreshoff, M., Hutchins, D., and Norvig, P. Deep learning with dynamic computation graphs. arXiv preprint arXiv:1702.02181 (2017).
 [35] Malewicz, G., Austern, M. H., Bik, A. J., Dehnert, J. C., Horn, I., Leiser, N., and Czajkowski, G. Pregel: a system for largescale graph processing. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of data (2010), ACM, pp. 135–146.
 [36] Mikolov, T., Chen, K., Corrado, G., and Dean, J. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013).
 [37] Murray, D. G., McSherry, F., Isaacs, R., Isard, M., Barham, P., and Abadi, M. Naiad: a timely dataflow system. In Proceedings of the TwentyFourth ACM Symposium on Operating Systems Principles (2013), ACM, pp. 439–455.
 [38] Neubig, G., Dyer, C., Goldberg, Y., Matthews, A., Ammar, W., Anastasopoulos, A., Ballesteros, M., Chiang, D., Clothiaux, D., Cohn, T., et al. Dynet: The dynamic neural network toolkit. arXiv preprint arXiv:1701.03980 (2017).
 [39] Neubig, G., Goldberg, Y., and Dyer, C. Onthefly operation batching in dynamic computation graphs. arXiv preprint arXiv:1705.07860 (2017).
 [40] NVIDIA. http://docs.nvidia.com/cuda/nvrtc/index.html.

[41]
Pang, B., Lee, L., and Vaithyanathan, S.
Thumbs up?: sentiment classification using machine learning techniques.
In Proceedings of the ACL02 conference on Empirical methods in natural language processingVolume 10 (2002), Association for Computational Linguistics, pp. 79–86.  [42] Quinlan, D. Rose: Compiler support for objectoriented frameworks. Parallel Processing Letters 10, 02n03, 215–226.
 [43] RaganKelley, J., Barnes, C., Adams, A., Paris, S., Durand, F., and Amarasinghe, S. Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines. ACM SIGPLAN Notices 48, 6 (2013), 519–530.
 [44] Rumelhart, D. E., Hinton, G. E., Williams, R. J., et al. Learning representations by backpropagating errors. Cognitive modeling 5, 3 (1988), 1.
 [45] Socher, R., Lin, C. C., Manning, C., and Ng, A. Y. Parsing natural scenes and natural language with recursive neural networks. In Proceedings of the 28th international conference on machine learning (ICML11) (2011), pp. 129–136.
 [46] Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing (2013), pp. 1631–1642.
 [47] Sundaram, N., Satish, N., Patwary, M. M. A., Dulloor, S. R., Anderson, M. J., Vadlamudi, S. G., Das, D., and Dubey, P. Graphmat: High performance graph analytics made productive. Proceedings of the VLDB Endowment 8, 11 (2015), 1214–1225.
 [48] Sundermeyer, M., Schlüter, R., and Ney, H. Lstm neural networks for language modeling. In Thirteenth Annual Conference of the International Speech Communication Association (2012).
 [49] Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. In Advances in neural information processing systems (2014), pp. 3104–3112.
 [50] Tai, K. S., Socher, R., and Manning, C. D. Improved semantic representations from treestructured long shortterm memory networks. arXiv preprint arXiv:1503.00075 (2015).
 [51] Tan, M., Santos, C. d., Xiang, B., and Zhou, B. Lstmbased deep learning models for nonfactoid answer selection. arXiv preprint arXiv:1511.04108 (2015).
 [52] TensorFlow Fixedsized LSTM Language Model. https://github.com/tensorflow/models/blob/master/tutorials/rnn/ptb/ptb_word_lm.py.
 [53] TensorFlow Fold Benchmark Code. https://github.com/tensorflow/fold/tree/master/tensorflow_fold/loom/benchmarks.
 [54] The Penn Tree Bank (PTB) Dataset. http://www.fit.vutbr.cz/~imikolov/rnnlm/simpleexamples.tgz.
 [55] Tian, Y., Balmin, A., Corsten, S. A., Tatikonda, S., and McPherson, J. From think like a vertex to think like a graph. Proceedings of the VLDB Endowment 7, 3 (2013), 193–204.
 [56] Tokui, S., Oono, K., Hido, S., and Clayton, J. Chainer: a nextgeneration open source framework for deep learning. In Proceedings of workshop on machine learning systems (LearningSys) in the twentyninth annual conference on neural information processing systems (NIPS) (2015), vol. 5.
 [57] Tokui, S., Oono, K., Hido, S., and Clayton, J. Chainer: a nextgeneration open source framework for deep learning. In Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Twentyninth Annual Conference on Neural Information Processing Systems (NIPS) (2015).
 [58] Vinyals, O., Kaiser, Ł., Koo, T., Petrov, S., Sutskever, I., and Hinton, G. Grammar as a foreign language. In Advances in Neural Information Processing Systems (2015), pp. 2773–2781.
 [59] Walt, S. v. d., Colbert, S. C., and Varoquaux, G. The numpy array: a structure for efficient numerical computation. Computing in Science & Engineering 13, 2 (2011), 22–30.
 [60] Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., and Bengio, Y. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning (2015), pp. 2048–2057.
 [61] Yan, Z., Zhang, H., Jagadeesh, V., DeCoste, D., Di, W., and Piramuthu, R. Hdcnn: Hierarchical deep convolutional neural network for image classification. ICCV (2015).
 [62] Yan, Z., Zhang, H., Jia, Y., Breuel, T., and Yu, Y. Combining the best of convolutional layers and recurrent layers: A hybrid network for semantic segmentation. arXiv preprint arXiv:1603.04871 (2016).
 [63] Yan, Z., Zhang, H., Wang, B., Paris, S., and Yu, Y. Automatic photo adjustment using deep neural networks. ACM Transactions on Graphics (TOG) 35, 2 (2016), 11.
 [64] Zaremba, W., Sutskever, I., and Vinyals, O. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329 (2014).
 [65] Zhang, H., Hu, Z., Wei, J., Xie, P., Kim, G., Ho, Q., and Xing, E. Poseidon: A system architecture for efficient gpubased deep learning on multiple machines. arXiv preprint arXiv:1512.06216 (2015).
 [66] Zhang, H., Zheng, Z., Xu, S., Dai, W., Ho, Q., Liang, X., Hu, Z., Wei, J., Xie, P., and Xing, E. P. Poseidon: An efficient communication architecture for distributed deep learning on GPU clusters. In 2017 USENIX Annual Technical Conference (USENIX ATC 17) (Santa Clara, CA, 2017), USENIX Association, pp. 181–193.
 [67] Zheng, S., Jayasumana, S., RomeraParedes, B., Vineet, V., Su, Z., Du, D., Huang, C., and Torr, P. H. Conditional random fields as recurrent neural networks. In Proceedings of the IEEE International Conference on Computer Vision (2015), pp. 1529–1537.
Comments
There are no comments yet.