1 Introduction
The field of microfluidics has been a popular locus of study for biological, chemical, and manufacturing research communities for nearly two decades, largely due to modern fabrication methods making the small length scales ( m) more accessible to highprecision analysis for biomedical technology, microscale engineering, and the study of fundamental physical systems Whitesides2006 ; Sackmann2014 . Microfluidic flows are typically dominated by viscous forces, as characterized by the Reynolds number, (the ratio of inertial forces to viscous forces in a fluid, with fluid density , velocity , viscosity , and microchannel hydraulic diameter ), which becomes due to small in microfluidic devices. In the last decade, however, inertially dominated flows (, often via increased fluid velocity ) have seen a rapid increase in attention due to their interesting nonlinear behavior Amini2014 . One such effect is seen in fluid flowing past an obstacle within a microchannel, as the finite inertia breaks the foreaft symmetry around the obstacle, irreversibly deforming the fluid. Amini et al. amini2013engineering leveraged this effect by sequencing cylindrical pillars in a programmatic way to “sculpt” the flow. Stoecklein et al. labonachip used an efficient forward model for simulating inertial flow sculpting to explore different flow shape deformations, illustrating a diverse and richly populated design space with many different possible flow shapes.
The ability to engineer the crosssectional shape of fluid streams has since been used to fabricate tailored microfibers Nunes2014 and 3D microparticles Paulsen2015 ; Wu2015 ; Paulsen2018 , perform solution exchange Sollier2015 , and create 3D shaped microcarriers for cellular analysis Wu2018 . Flow sculpting shows additional promise for creating useful microstructures for tissue engineering Du2008 , biosensing LeGoff2015 , as well as diagnostics and drug delivery Yu2018 . But design in flow sculpting is extremely difficult: even the limited set of 32 pillar configurations used by Stoecklein et al. labonachip offered pillar sequence permutations, with recent developments providing a continuous set of infinite pillar geometries Stoecklein2018 , vastly increasing flow shape possibilities, but making manual design impossible. This motivates the need for a fast solution to the inverse problem in flow sculpting, which asks: given a desired fluid flow shape, what is the pillar sequence which will create the closest matching shape?
The idea of designing shapes from a microfluidic flow is not new. In gaoptimization
, the inverse problem is posed as an optimization problems of choosing from a sequence of 32 pillar configurations. The authors utilize the Markov property of the pillar transformations – i.e. only the outcome of the previous pillar affects the subsequent pillar – to formulate an optimization approach based on Genetic Algorithms. While successful, this approach is quite time consuming taking several dozens of minutes for a solution, thus precluding realtime design exploration which is especially crucial in the bioengineering community for device design. For such problems with an combinatorially large exploration space, it is natural to use Reinforcement Learning (RL). The RL community has made significant advances over the past few years, and we leverage successes such as DQN
DQN , Double DQN DDQN to solve the design problem. There exist many works on RL which are interesting in this context but in this study we restrict ourselves to Double DQN DDQN . In future works, we would explore several other algorithms to improve the performance of the inverse design problem.1.1 Related Work
The initial attempts for this design problem was done by Stoecklein et.al.gaoptimization and Amini et. al.amini2013engineering using Genetic Algorithm (GA) for solving this inverse design problem. The key disadvantage with such an approach is that the search space in a GA based approach depends on the size of the pillar sequence chosen and search space for a pillar sequence of size is sequences. Such a large search space is very difficult to explore, which naturally makes GA a very time consuming option. Further, this approach requires the same exploration to be performed any time a new design problem is being solved. Also, multiple trials for each configuration required to ensure statistical convergence of optima compounds the cost even further.
In this context, several deep learning based approaches were attempted. Lore et. al
smccnnshow that it is possible to combine a convolutional neural network (CNN) with a simultaneous multiclass classification (SMC) problem formulation to predict a fixed length sequence of pillars given a target flow shape. In this formulation, the network predicts the pillar type for each index of the sequence simultaneously using a modified loss function which sums the negative loglikelihood across each pillar in the predicted sequence. However, the performance of this model depended heavily on the sample set used for generation. In this context, Stoecklein et.al
scirep explored the sampling complexity of the data generation by performing PCA and HDMR and also understood the need for a nonuniform sampling to solve this problem. Further, Lore et. al ppnitnexplored another approach in representing the problem using two networks, (i) Pillar prediction network and (ii) Intermediate transformation network. The idea is to subdivide the inverse problem by first identifying the intermediate shapes using the intermediate transformation network and then use pillar prediction network to predict the pillar sequence. However, the only knob controlling when to subdivide is based on the parameter called complexity measure. In reality, this methodology ends up generating pillar sequences that are longer compared to the actual pillar sequence. This indicates the inefficiency in representing the problem appropriately. Additionally, having a longer pillar sequence also implies higher cost for construction of the microfluidic device. Hence, obtaining pillar sequences which are optimal in length and naturally adaptive to sampling and performing better than evolutionary algorithms is necessary. In particular, learning such a design domain is useful for design problems where we need results in real time. Therefore, it is infeasible to explore the search space of the domain repeatedly for every design problem as performed by an evolutionary algorithm.
Our approach is also motivated with several other related works. In manochasiggraph18 , RL was leveraged to control the flow direction in an application for rigid body control. In stephanguy , RL was used to produce safer and shorter trajectories than traditional motion planning. In rlvortexflow ; rlvortexflow2 , they attempted to use RL for control over vortex induced flows. In rlgliding , RL was applied to achieve gliding with either minimum energy expenditure, or fastest time of arrival, at a predetermined location. However, to the best of the knowledge of the authors, there are none or very sparse work in the area of FlowShape design for microfluidic devices.
1.2 Contributions
In this work we build a generic framework for solving engineering design problems using RL. Our specific contributions are:

We build an RL environment using OpenAI gym gym for addressing the flow shape design in a microfluidic device.

We further build Double DQN agents for solving the design problem for obtaining few key shapes as a proofofconcept.
The flow of the rest of the paper is as follows: In section 2, we explain the details about the design environment. In section 3, we use the design environment constructed with Double DQN algorithm and then learn the RL environment. Further, in section 4
, we show some preliminary results obtained and also show how transfer learning is helpful in this context. Finally, we share some conclusions and future work in section
5.2 Flow Shape Design
We formally pose the flowsculpting problem (i.e., obtaining the pillar sequence to reach a target flow shape) as a deep reinforcement learning problem. In the RL problem, an agent interacts with the environment that contains certain dynamics governed by the physics of the real world. For each time step , the agent receives a state from the environment and selects an action according to it’s policy function . Here, we use a deep neural network for the policy that represents the mapping from the state space to the action space. The agent then receives a scalar reward computed from a reward function and transitions to the next state based on dynamics of the environment. The process continues until the maximum episode length is reach or the goal of the RL agent is met.
We embed the Markovian process of moving from one flow shape to another using the transition matrix as a RL problem using an OpenAI gym environment (as shown in Fig. 1). Since the dynamics of the environment is represented by a set of equations, the environment is deterministic. We consider that the current flow shape as the fully observable state of the environment and therefore we make no distinction between the observation and state returned by the environment. For each new episode, the target flow shape is initialized and given as the goal for the agent. At each time step , the RL agent is tasked with choosing the best action corresponding to the different possible pillar configurations. The action is passed to the environment where the corresponding transformed flow shape for the current flow shape is computed using the markovian transition matrix discussed in gaoptimization . This transformed shape is considered as the new observation for the agent. The episode terminates only when the agent places a sequence of pillars which deforms the flow shape to a shape that is similar to target flow shape or when the number of pillars placed exceeds a certain constraint. A metric (Pixel Match Rate, PMR) is defined as the norm of each pixel between the flow shape generated by the agent and the target flow shape. The PMR ranges from 0 where no pixels match between the target and generated flow shape to 1 if both the target and generated flow shape are identical for each pixel. Thus, the goal is achieved if a threshold of the PMR is reached.
(1) 
Since the structure of the reward is instrumental to the learning capabilities of the RL agent, we shaped the reward as function of the PMR. To incentivise the agent to achieve the target flow shape with the least amount of pillars, we structure the reward function in the gym environment to be negative reward. That is, for each step the agent takes, it receives a reward equivalent to the PMR difference between the flow shape it generated and the target flow shape, scaled by some constant. We find that the reward function shown in Equation 2, which is shaped by subtracting the baseline of the worse case scenario and scaled by the remainder of the PMR range after subtracting the baseline, is rich enough for the agent to learn the dynamics of environment and achieve the target flow shape. Here, we empirically find that the general baseline for the PMR metric is approximately 0.5 from running mutliple experiments with different target flow shapes.
(2) 
In formulating this problem as a reinforcement learning problem, we consider two scenarios. In the first scenario, we generate a target flow shape using a sequence of pillars defined by the user. In this scenario, the target flow shape remains the same throughout the agent’s training process and is analogous to the agent learning to navigate to a single goal. We ensure that the agent explores the action space and finds multiple possible paths (i.e. designs of pillar sequences) to reach the target (i.e. the crosssection shape of the flow).
We then formulate a second problem as a transfer reinforcement learning problem. In this formulation, we first train a RL agent on a flow shape that is easily achievable. Using the parameters of the learned agent, we then train the agent on sequence of flow shapes that are increasingly harder to achieve. We hope that by transferring the learned parameters of the agent, the more complex flow shapes can be more easily learned by the agent in a shorter time or with less exploration or both.
3 Algorithms and Model
To demonstrate that RL agent can learn a broad spectrum of flow shapes, we randomly sample flows from a set of 148,026 pillar sequences that was previously generated in scirep . These pillar sequences were generated from 7.5 million random sequences with 32 possible pillar configurations. A PCA was performed on the resulting 7.5 million flow shapes to produce a dataset of 148,026 unique flow shapes. Thus, we demonstrate that the agent in a fixedflow environment can sufficiently learn a target flow shape for a wide range of flow shapes. Since, the 10 sequences we sampled have a pillar sequence length of 7, we set the maximum episodic length of our environment to be 7 so that the RL agent either reach an approximation of the target flow shape using pillar sequence of length 7 or less. The action space of the agent is a set of 32 discrete actions corresponding to the 32 possible pillar configurations.
To test the efficacy of our proposed framework, we used a DDQN algorithm on the flowsculpting environment because value iteration methods are known to be more sample efficient compared to the policy iteration methods DDQN . In DQN DQN , the deep neural network acting as the Qfunction approximator approximates the Qvalue for each stateaction pair. The RL agent then selects the action with the maximum Qvalue. The DDQN algorithm is an improvement over the DQN introduced by DDQN
. In essence, the DDQN algorithm reduces the over estimation of a state’s value by decoupling the network used to select the next best action and the network used to estimate the value of the state. The policy of the DDQN agent is represented by a deep convolutional neural network with 2 convolutional layers with 32 and 64 output filters respectively. Each convolution layer is also followed by a subsequent layer of batch normalization and maxpooling. After the convolution layers, we use two fullyconnected layers with 128 and 64 hidden units respectively and a batch normalization layer in between them to give an output of the Qvalues for the 32 possible actions.
The termination criteria that we use for the agent was PixelMatchRate (PMR). A 90% PMR is set as the termination threshold if the finer details of the target flow shape is required. However, if the application of the microfluidic requires just a bulk approximation of the target flow shape, 85% PMR is typically sufficient scirep .
We also used RMSProp with a learning rate of 0.001 as the optimizer for the agent and a linearly decaying exploration policy which decays from 1 to 0.1 over 1 million steps. The target network update interval was set at
steps and the update only begins after the agent has taken random steps. For each our sampled flow shape, the agent is trained for episodes.4 Results and Discussion
In Figure 2(a), we show the average cumulative reward achieved by the agent for 5 of the target flow shapes (Flow ID 3, 5, 7, 8 and 9) where the agent performed well using a threshold of 90% PMR. As seen in the figure, the cumulative rewards attained by the RL agent converges after episodes of training. This convergence is similarly reflected in Figure 2(b) where the RL agent’s frequency of success per 1000 episode is shown. It can be seen that the RL agent achieves almost a success rate of 100 percent for all of the flows.
Flow Shape ID  Target Sequence  RL Agent 

Flow 3  15, 25, 6, 20, 18, 13, 15  24, 23 
Flow 5  22, 11, 31, 15, 18, 4, 1  25 
Flow 7  22, 29, 24, 16, 13, 12 , 4  29, 30 
Flow 8  20, 14, 16, 23, 17, 4 , 6  30,30,24 
Flow 9  23, 6, 3, 5, 5, 14, 11  20 
A comparison of the target flow shapes and flow shapes generated by the RL agent is shown in Figure 3. An interesting observation here is that the flow shapes sculpted by the RL agent were all generated with a minimal number of pillars. While the target flow shapes were all generated using a sequence of 7 pillars, the longest sequence of pillars used by the RL agent was 3 with a majority of the shapes being achievable with 2 pillars or less. Besides the obvious benefit of being more cost efficient, this can be also extremely advantageous in a microfluidic setting as often times there are physical constraints which limits the size of a microfluidic device. Therefore, the ability to generate a bulk approximation of the target flow shape with significantly shorter pillar sequence is extremely exciting. Although ultimately, it is up to the user of the microfluidic device to specify the threshold of accuracy required for his/her respective application.
We also demonstrate that it is possible for the RL agent to arrive at multiple possible solutions for the same inverse problem. We use the example of Flow 5 to illustrate this occurrence. Table LABEL:Tab:Best_5 shows the top 5 pillar sequences generated by the RL agent where the resulting deformed flow shapes has at least 90 percent PMR with the target flow. This can also be leveraged greatly in multiple ways. First, the user of the microfluidic device can utilize the RL agent as a recommender system. If the practitioner of the microfluidic device has a general idea of the required flow shape, the flow shape can be first used as target flow shape and the RL agent can be used to further finetuned the structure of the flow through the multiple solutions proposed by the agent. Secondly, if the user only has certain types of pillars at his/her disposal due to various factors, such as manufacturing capability constraints, this can be used to generate an alternate solution which might be more feasible for the user.
Method  Modality  Frequency 

Target  20, 14, 16, 23, 17, 4 , 6   
Best1  30, 30, 24  50294 
Best2  2, 29, 34, 31  20374 
Best3  30, 31, 31  10431 
Best4  29, 31, 24  7861 
Best5  30, 29, 25  6356 
For the second formulation of the problem as a transfer learning problem, we first train the RL agent to reach target flow 7. After the agent has successfully learn to generate sequences for the flow, we use the trained agent and give it a new task of learning flow 6. We repeat the procedure above by using the agent that has been trained on 2 flows and train it again with a third and fourth flow. The results show that using an RL agent previously trained on a different flow enables the agent to learn the new flow shape much quicker than training the RL agent from scratch. It is also apparent that trained RL agent success frequency has significantly less variation as compared to the RL agent trained from scratch. This illustrates that it is possible to transfer the learning of an RL agent learning to solve one design problem to another design problem. Being able to transfer the knowledge of an RL agent has huge ramifications since it drastically reduces the amount of exploration or number of forward function evaluations required to arrive at a solution. Figure 7 shows the comparisons between a vanilla DDQN RL agent trying to learn 4 different flow shapes from scratch and an RL agent that has previously been trained on multiple flow shapes.
5 Conclusions
In this study, we propose a Deep Reinforcement Learning framework to solve an example of an inverse problem that could possibly have multiple solutions. We demonstrate that by formulating the inverse problem using RL, the RL agent is able to achieve a desired target. In particular, we build an RL agent for the inverse problem of designing pillar sequence for obtaining a target flow shape by learning multiple pillar sequences to arrive at a good approximation of the target flow shape. In addition, these sequences are also often shorter than the original target sequence, thus the solutions presented by the RL agent can be more optimal and cost efficient. In the second part of the study, we show that it is possible to speed up the learning of an RL agent by using an agent that has previously been trained on a different flow shape. This demonstrates the phenomena of transfer learning where the knowledge of one RL agent can be transferred to solve another set of inverse problem. Future efforts for this work include creating/using algorithms which are more sample efficient and are generalizable for any complexity of flow shape.
6 Acknowledgements
This work has been supported in part by the U.S. Air Force Office of Scientific Research under the YIP grant FA95501710220. Any opinions, findings and conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the sponsoring agency. We would also like to thank Sambit Ghadai for providing valuable insights and feedback on this work.
References
 [1] George M Whitesides. The origins and the future of microfluidics. Nature, 442(7101):368–373, 2006.
 [2] Eric K Sackmann, Anna L Fulton, and David J Beebe. The present and future role of microfluidics in biomedical research. Nature, 507(7491):181–9, 2014.
 [3] Hamed Amini, Wonhee Lee, and Dino Di Carlo. Inertial microfluidic physics. Lab Chip, 14(15):2739–2761, aug 2014.
 [4] Hamed Amini, Elodie Sollier, Mahdokht Masaeli, Yu Xie, Baskar Ganapathysubramanian, Howard A Stone, and Dino Di Carlo. Engineering fluid flow using sequenced microstructures. Nature communications, 4:1826, 2013.
 [5] Daniel Stoecklein, ChuehYu Wu, Keegan Owsley, Yu Xie, Dino Di Carlo, and Baskar Ganapathysubramanian. Micropillar sequence designs for fundamental inertial flow transformations. Lab on a Chip, 14(21):4197–4204, 2014.
 [6] Janine K. Nunes, Chueh Yu Wu, Hamed Amini, Keegan Owsley, Dino Di Carlo, and Howard A Stone. Fabricating shaped microfibers with inertial microfluidics. Advanced Materials, 26:3712–3717, 2014.
 [7] Kevin S. Paulsen, Dino Di Carlo, and Aram J. Chung. Optofluidic fabrication for 3Dshaped particles. Nature Communications, 6:6976, 2015.
 [8] Chueh Yu Wu, Keegan Owsley, and Dino Di Carlo. Rapid SoftwareBased Design and Optical Transient Liquid Molding of Microparticles. Advanced Materials, 27(48):7970–7978, 2015.
 [9] Kevin S. Paulsen, Yanxiang Deng, and Aram J. Chung. DIY 3D Microparticle Generation from Next Generation Optofluidic Fabrication. Advanced Science, 5(7):1–6, 2018.
 [10] Elodie Sollier, Hamed Amini, Derek E Go, Patrick a. Sandoz, Keegan Owsley, and Dino Di Carlo. Inertial microfluidic programming of microparticleladen flows for solution transfer around cells and particles. Microfluidics and Nanofluidics, pages 1–13, 2015.
 [11] ChuehYu Wu, Daniel Stoecklein, Aditya Kommajosula, Jonathan Lin, Keegan Owsley, Baskar Ganapathysubramanian, and Dino Di Carlo. Shaped 3D microcarriers for adherent cell culture and analysis. Microsystems & Nanoengineering, 4(1):21, 2018.
 [12] Yanan Du, Edward Lo, Shamsher Ali, and Ali Khademhosseini. Directed assembly of cellladen microgels for fabrication of 3D tissue constructs. Proceedings of the National Academy of Sciences of the United States of America, 105(28):9522–7, 2008.
 [13] Gaelle C. Le Goff, Rathi L. Srinivas, W. Adam Hill, and Patrick S. Doyle. Hydrogel microparticles for biosensing. European Polymer Journal, 72:386–412, 2015.
 [14] Bing Yu, Hailin Cong, Qiaohong Peng, Chuantao Gu, Qi Tang, Xiaodan Xu, Chao Tian, and Feng Zhai. Current status and future developments in preparation and application of nonspherical polymer particles. Advances in Colloid and Interface Science, 256:126–151, 2018.
 [15] Daniel Stoecklein, Keegan Owsley, ChuehYu Wu, Dino Di Carlo, and Baskar Ganapathysubramanian. uFlow: software for rational engineering of secondary flows in inertial microfluidic devices. Microfluidics and Nanofluidics, 22(7):74, jun 2018.
 [16] Daniel Stoecklein, ChuehYu Wu, Donghyuk Kim, Dino Di Carlo, and Baskar Ganapathysubramanian. Optimization of micropillar sequences for fluid flow sculpting. Physics of Fluids, 28(1):012003, 2016.
 [17] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.
 [18] Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double qlearning. CoRR, abs/1509.06461, 2015.

[19]
Kin Gwn Lore, Daniel Stoecklein, Michael Davies, Baskar Ganapathysubramanian,
and Soumik Sarkar.
Hierarchical feature extraction for efficient design of microfluidic flow patterns.
In Feature Extraction: Modern Questions and Challenges, pages 213–225, 2015.  [20] Daniel Stoecklein, Kin Gwn Lore, Michael Davies, Soumik Sarkar, and Baskar Ganapathysubramanian. Deep learning for flow sculpting: Insights into efficient learning using scientific simulation data. Scientific reports, 7:46368, 2017.
 [21] Kin Gwn Lore, Daniel Stoecklein, Michael Davies, Baskar Ganapathysubramanian, and Soumik Sarkar. A deep learning framework for causal shape transformation. Neural Networks, 98:305–317, 2018.
 [22] Pingchuan Ma, Yunsheng Tian, Zherong Pan, Bo Ren, and Dinesh Manocha. Fluid directed rigid body control using deep reinforcement learning. ACM Transactions on Graphics (TOG), 37(4):96, 2018.
 [23] Bobby Davis, Nicholas Sohre, and Stephen J Guy. Multiworld motion planning. IEEE Robotics and Automation Letters, 3(4):3968–3974, 2018.
 [24] Jean Rabault, Ulysse Reglade, Nicolas Cerardi, Miroslav Kuchta, and Atle Jensen. Deep reinforcement learning achieves flow control of the 2d karman vortex street. arXiv preprint arXiv:1808.10754, 2018.
 [25] Rabault Jean, Kuchta Miroslav, and Jensen Atle. Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control. arXiv preprint arXiv:1808.07664, 2018.
 [26] Guido Novati, Lakshminarayanan Mahadevan, and Petros Koumoutsakos. Deepreinforcementlearning for gliding and perching bodies. arXiv preprint arXiv:1807.03671, 2018.
 [27] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/1606.01540, 2016.