In a typical Japanese agricultural town, the length of water canals extends to dozens of kilometers while the dry season during which inspection and repair can be conducted lasts only months per year. However, the current inspection process involves technicians walking along the canals and manually measuring and marking the damaged areas in a log book. Each technician can inspect only km a day. Furthermore, it takes over one month to convert the recorded data into digital information for guiding repairs. Water canal inspection using UAVs (Figure 1) would significantly reduce inspection time and labor cost by automatically identifying defects and registering their GPS coordinates.
We will focus our efforts on a town in Niigata, an agricultural district along the northwest coast of Japan. In the area of interest shown in Figure 2, there are about km of water canals spread over tens of square kilometers of farmland. The scale of water canals is beyond the cruise and communication range of most commercially available UAVs, which can typically fly for minutes within km of the ground station. As a result, inspection of all water canals requires heterogeneous vehicles, i.e. a combination of multiple UAVs and ground vehicles.
A viable inspection strategy first divides all water canals into sub-regions that a UAV (or a fleet of UAVs) can cover within one battery charge. Once the UAVs complete inspection of one sub-region, they return to ground vehicles (cars driven by human operators) and are transported to the next sub-region while their batteries are swapped.
There are several constraints that the UAVs and cars have to satisfy. First, as the cars are used as ground stations to monitor and recharge the UAVs, each UAV needs to have at least one car within its communication range at all time, and each UAV will return to a car when it runs out of battery. Second, as the cars travel on public roads, they have to abide by local traffic regulations and their speed is limited by real-time traffic conditions. Moreover, it is also necessary to quickly re-plan the paths should unexpected situations (such as UAV breakdown or change of traffic conditions) arise.
This paper presents an MIQP-based planning framework that generates optimal inspection plans for a given fleet of UAVs and cars, accounting for the aforementioned constraints. The plan includes recharging and transportation between different regions. The proposed framework can also quickly generate feasible new plans if a current plan is interrupted.
The structure of this paper is as follows: Section III describes the coverage planning problem we are trying to solve; Section IV provides detailed descriptions of our proposed planning framework; simulation results are presented in Section V; the conclusion and future work are stated in Section VI.
Ii Related Works
Cooperative control of a multi-agent system reduces operation time, introduces redundancy and robustness to the system to better handle adversarial situations such as malfunction or failure of one or more agents. Araki et al. built a robot swarm that can function as both ground and aerial vehicles, and developed planning methods using mixed integer program (MIP) . Schillinger et al. studied optimal planning algorithms for multiple ground robots using linear temporal logic . Kim et al. generates dynamically-feasible trajectories for multiple UAVs to cooperatively transport large objects .
Due to the expressiveness of integer variables, MIPs have been used to solve planning problems with complex constraints. Although current MIP solvers have worst-case exponential complexity , MIPs can be solved fast enough for many nontrivial problems that are practically useful. Classical graph planning problems such as traveling salesman problem (TSP)  and vehicle routing problem (VRP) 
can be written down and solved as mixed-integer linear programs (MILP). Avellaret al. solves a minimum-time ground area coverage problem by converting it to a VRP . Dynamically-feasible, obstacle-free UAV trajectories can also be obtained by solving MIPs [8, 9]. Grotli et al. studied UAV planning problems with fuel and communication constraints, but the communication ground stations are fixed . Evers et al. deals with the coverage problem and considers the uncertainty of fuel usage and weather conditions to provide a robust planning solution . However, no refueling is planned, and only target nodes instead of edges are required to be visited. Lim et al. covers both edges and nodes of a graph representing a power network using a fleet of UAVs and minimizes overall inspection time .
To the best of our knowledge, although a wide variety of planning problems has been solved with MIPs, the planning problem of heterogeneous vehicles, which allows multiple recharging/refueling of some of the vehicles and has constraints involving multiple types of vehicles, has yet to be addressed.
Iii Problem statement
Iii-a Graph representations of water canals and roads
The first step in our planning framework is to abstract canal and road networks from a map into graphs (Figure 3). GPS information of roads is extracted from the open-source map named OpenStreetMap  using OSMnx . NetworkX  is used to assign nodes to end and intersection points on the road network and identify the adjacency matrix of the graph.
Water canals and roads are represented as two weighted graphs:
, where is an intersection of or a point along the canals, and is a canal segment. The canal graphs are constructed in such a way that all edges in the graph have comparable weights, i.e. for all , so that it takes a UAV the same amount of time to inspect all edges in . An important observation is that is usually a tree;
, where is an intersection of roads, and a road segment.
Iii-B Heterogeneous planning problem
Given the following information:
graphs of water canals and roads;
number of available cars and UAVs;
UAV battery life and transmission range;
we want to find paths for UAVs and cars that inspect all water canals as quickly as possible, subject to the following constraints:
cars drive on the road graph ;
UAVs fly on the canal graph , except when taking off from and returning to the cars;
every UAV needs to have at least one car within its transmission distance, as shown in Figure 4;
every UAV takes off with a fully-charged battery, and needs to return to a car when its battery is depleted.
Iv Proposed computational methods
Using the canal and road graphs defined in Section III-A, the inspection paths for UAVs and cars are found through three steps:
Iv-A1 graph partitioning
as the entire canal graph is usually much larger than what a UAV fleet can inspect with a single battery charge, the canal graph is partitioned into subgraphs that a given fleet can possibly inspect within the UAV’s battery life ( minutes). In addition, partitioning the canal graph significantly reduces the size and computation time of each of the path planning problems.
Iv-A2 heterogeneous vehicle path planning
for each subgraph generated from partitioning, the heterogeneous vehicle path planning algorithm generates paths for UAVs and cars that inspect all canals in the given subgraph, subject to the constraints given in Section III-B.
Iv-A3 car routing between subgraphs
while being recharged, the UAVs are also being transported from one subgraph to another by cars. The car routing algorithm gives the minimum-length path that visits and inspects all canal subgraphs.
Iv-B Graph partitioning
To minimize the time for swapping batteries and traveling between partitioned subgraphs, the objective is that partitions have comparable sizes. Let be the total number of UAVs and the number of edges a UAV can inspect with one fully charged battery. The water canal graph consists of vertices and edges. Therefore, the maximum number of edges in a subgraph is , and an initial guess for the number of subgraphs is .
The actual number of subgraphs is increased from until the following mixed-integer quadratic program (MIQP) becomes feasible. The MIQP of minimizing the traveling distance of UAVs has the following objective function and constraints:
where represents whether Node is in Subgraph s, represents whether Edge in Subgraph . Constraint (2) requires that each edge belongs to one and only one subgraph. Constraint (3) sets and as the lower and upper bounds of the number of edges in every subgraph. As a connected subgraph of a tree is also a tree, Constraint (4) implies that each subgraph must be connected. Constraint (5) shows that if Edge is in Subregion s, both Node and Node are also in Subgraph .
For example, given a fleet of 4 UAVs, in which every UAV can cover edges on a single battery charge (, ), it is calculated that the graph of water canal should be partitioned into 8 subgraphs. The partitioning result is shown in Figure 5.
Iv-C Heterogeneous vehicles path planning for subgraphs
Given fixed numbers of UAVs and cars, the problem of generating paths for UAVs and cars to inspect a canal subgraph can be formulated as an MIQP with the objective of minimizing the traveling distance of UAVs. In addition to the constraints in Section III-B, we make the following assumptions:
UAVs and cars only travel to adjacent nodes over one time step;
All canal edges in a subgraph must be inspected by a UAV;
It takes a UAV one time unit to inspect an edge in the canal subgraph; and
The time taken to fly between the canals and the vehicles at the beginning and end of the inspection is negligible compared to the inspection time. This is reasonable because the UAVs need to fly very slowly in a zig-zag pattern in order to acquire clear images of the canal walls. In comparison, the UAVs can fly reasonably fast when traveling between the canals and the cars.
Let be the total number of edges in the canal subgraph indexed by , the planning time horizon is increased from the initial guess until the following MIQP becomes feasible. The MIQP of minimizing the traveling distance of UAVs has the following objective function:
where represents whether UAV at time is at Node of the canal subgraph; represents whether Car at time is at Node of the road subgraph; represents whether UAV is within the communication range of Car at time . is the total number of cars in the fleet. is the transmission matrix containing only s and s. if the Euclidean distance between Node and Node is less than the UAV’s maximum transmission distance. and , containing only s and s, are the adjacency matrices of the water canal and road subgraphs, respectively. if node and node are adjacent.
Constraint (7) means that all edges must be inspected once. represents whether Edge is visited by the UAV starting at time in the direction . For an edge , means that when the edge is inspected by an UAV, the UAV first visits Node and then Node . Similarly, means that the UAV visits first Node and then . implies that , , and implies that and . This relationship between and is summarized in Constraint (8) and (9).
Constraints (10) and (11) imply that for every UAV at all time, there must be at least one car within its transmission distance. Constraint (12) shows that at any time step, all cars and UAVs must appear at only one node. Constraints (13) and (14) require that UAVs and cars only travel to adjacent nodes between consecutive time steps.
The planning result for one subgraph is shown in Figure 6. The magenta graph is a subgraph of and the grey graph a subgraph of . The fleet has 4 UAVs and 2 cars. The UAV inspection paths are shown on top of as dashed lines with different colors. The car paths are shown as dotted lines on top of .
When executing a plan on a subgraph, re-planning is necessary in the event of UAV failure or traffic jam. To re-plan at , an MIQP similar to the one in Section IV-C is constructed, with the following modifications:
Canal edges already inspected at time are removed from the canal subgraph.
Congested roads are removed from the road subgraph.
Battery life of UAVs is updated to the remaining battery life.
All UAVs and cars start at their positions in the original plan at time .
As shown in Figure 7, the proposed re-planning method successfully generates new paths for UAVs and cars when a UAV fails or the traffic changes.
Iv-E Car routing between subgraphs
After finding an optimal plan to inspect a single subgraph, we need to find the shortest car route that visits all subgraphs. It is also required that the car route starts from and ends at a fixed location called the office. To plan car routes between canal subgraphs, a new graph whose nodes correspond to the canal subgraphs is constructed. As the cars are free to travel from any canal subgraph to any other canal subgraph, is fully connected but asymmetric (because of one-way roads). After determining the weights of the edges of the new fully-connected graph, the car routing problem, which searches for the shortest cycle that visits all nodes in , can be solved as an Asymmetric Traveling Salesman Problem (ATSP).
Iv-E1 Determining edge weights in
Let be the total number of subgraphs generated from canal graph partitioning. The total number of nodes in is because the office is also included as a node. Let be the matrix of edge weights of , in which is the weight of the directed edge pointing from Node to Node . We want a that minimizes the total travel distances of all cars when they carry UAVs from one subgraph to another (i.e. Subgraph A to Subgraph B).
In the inspection planning of subgraphs (Section IV-C), the planner gives a starting and leaving node for every car in every subgraph. The UAVs take off at the starting nodes (shown as in Figure 8) and land at the leaving positions ( in Figure 8). can be determined by finding the shortest of all paths connecting a leaving node () of Subgraph A and a starting node () in Subgraph B. This can be formulated as the following MIQP:
where is the length of the shortest path between Node and Node in the road graph, and is whether car travels from Node in Subgraph to Node in Subgraph . Constraint (16) says that all ending nodes of Subgraph must be visited only once and that all starting nodes of Subgraph must be visited only once.
After finding , the car routing problem is reduced to an ATSP for a complete directed graph. The objective is to find the shortest closed path that starts from the office and visits all subgraphs:
where denotes whether subgraph s is visited at time step . Constraint (18) says that cars must begin and end the journey from/at the office. Constraint (19) means at the same time, only one subgraph is visited, and each subgraph can be visited only once. Figure 9 shows the shortest paths for cars.
V Results and Discussion
The computation time of all methods in our inspection planning framework is benchmarked and summarized in Table I. The problem used for benchmarking considers the inspection of a water canal graph with nodes and edges. The inspection is conducted by a fleet consisting of UAVs and cars. Every UAV can cover 3 edges with a fully-charged battery. The MIPs are solved using GUROBI on a laptop with Intel® Core™ i7-3520M CPU (dual core four threads).
The planning result shown in Figure 10 illustrates that our algorithm ensures the coverage of water canals and solves the problem caused by limited transmission distance and UAV battery life. Colored lines on graph of water canal represent the path of UAVs and colored lines on graph of road denote path of cars. Dotted lines between graph of water canal and graph of road are path of UAVs between cars and water canal. The entire inspection plan involving all cars and UAVs is also simulated in Simulink, as shown in Figure 11.
A 3D reconstruction of a water canal subgraph using Octomap  (Figure 12(b)) is generated by simulating a UAV with stereo-camera flying inside the CAD model of water canals (Figure 12(a)) following a planned path. The local octomap for all the UAVs are obtained from the disparity depth-map of a virtual stereo camera sensor in Gazebo, a realistic physics simulator. These local octomap are stitched using the ground truth poses from the simulator to obtain a unified octomap of the model. Figure 12 proves that our method inspects both walls of water canals and is expected to work effectively in real world scenarios.
It is shown in Figure 13 that the maximum planning time for canal subgraphs (algorithm in Section IV-C) is 140, which can be further reduced by parallelization. Based on experiments, the time needed by cars to travel from one subgraph to another is about 10 minutes. If a new inspection plan is needed for the next subgraph due to different traffic conditions or the malfunctioning of some of the UAVs, the new plan will be ready before arriving at the next subgraph.
It has been assumed in Section IV-C that the time needed by one UAV to inspect one edge in is 1 unit time, which is 10 minutes based on experiments. If we further assume that the time taken by cars to travel from one subgraph to another is 10 minutes, then the total inspection time for UAV-car fleets of different sizes can be calculated, and is summarized in Table II. There is a 75% reduction in inspection time by increasing the fleet size from 1 UAV and 1 car to 4 UAVs and 2 cars.
Based on experimental results, the efficiencies of different inspection methods are summarized in Table III
. Manual inspection involves a technician walking along the canals and looking for defects. In manual navigation, an operator flies a UAV manually along the canals, and another inspector looks for defects from the UAV’s video feed. In our approach (heterogeneous vehicle routing), the UAV flies autonomously, records video of the canal walls, which are then analyzed with computer vision algorithms. Compared with current manual inspection and manual UAV flying method, the proposed heterogeneous vehicle planning framework usingUAVs and cars would reduce inspection time by and , respectively. These results could be improved further by using more UAVs and cars.
|Fleet size||1 UAV 1 car||3 UAVs 3 cars||4 UAVs 2 cars|
|Resolution||NA||0.2 mm-25 mm||0.2 mm|
Using UAV and car combinations to inspect large area of water canal reduces inspection time. A innovative algorithm is formulated to solve the new heterogeneous planning problem resulted from the UAV and car combinations. For the current map of water canal in an agricultural town in Japan, the planning algorithm runs online. The planning time is seconds for each subgraphs and re-planning time is less than seconds. Compared with manual inspection, using UAVs and cars can be expected to reduce inspection time by at most .
In the future, collision avoidance between vehicles, aerial constraints (weather, flying zones, busy aerial traffic, aerial regulations, etc.), ground traffic regulations (one-way, non-parking zones, speed limits, etc.) and uncertainty in maps and measurements of UAV/car localization will be added to the system to increase safety during inspection flights.
The authors would like to thank TOPRISE Co., LTD. for their sponsor for this work and Mayuko Nemoto for help on problem formulation and running experiments.
-  B. Araki, J. Strang, S. Pohorecky, C. Qiu, T. Naegeli, and D. Rus, “Multi-robot path planning for a swarm of robots that can both fly and drive,” in Robotics and Automation (ICRA), 2017 IEEE International Conference on. IEEE, 2017, pp. 5575–5582.
-  P. Schillinger, D. V. Dimarogonas, and M. Bürger, “Multi-objective search for optimal multi-robot planning with finite ltl specifications and resource constraints,” in IEEE International Conference on Robotics and Automation (ICRA), Singapore, May 29-June 3, 2017, 2017.
-  H. Kim, H. Lee, S. Choi, Y.-k. Noh, and H. J. Kim, “Motion planning with movement primitives for cooperative aerial transportation in obstacle environment,” in Robotics and Automation (ICRA), 2017 IEEE International Conference on. IEEE, 2017, pp. 2328–2334.
-  D. Bertsimas and J. N. Tsitsiklis, Introduction to linear optimization. Athena Scientific Belmont, MA, 1997, vol. 6.
-  K. L. Hoffman, M. Padberg, and G. Rinaldi, “Traveling salesman problem,” in Encyclopedia of operations research and management science. Springer, 2013, pp. 1573–1578.
-  G. Laporte, “The vehicle routing problem: An overview of exact and approximate algorithms,” European journal of operational research, vol. 59, no. 3, pp. 345–358, 1992.
-  G. S. Avellar, G. A. Pereira, L. C. Pimenta, and P. Iscold, “Multi-uav routing for area coverage and remote sensing with minimum time,” Sensors, vol. 15, no. 11, pp. 27 783–27 803, 2015.
-  R. Deits and R. Tedrake, “Efficient mixed-integer planning for uavs in cluttered environments,” in Robotics and Automation (ICRA), 2015 IEEE International Conference on. IEEE, 2015, pp. 42–49.
-  A. Richards, J. Bellingham, M. Tillerson, and J. How, “Coordination and control of multiple uavs,” in AIAA guidance, navigation, and control conference, Monterey, CA, 2002.
-  E. I. Grøtli and T. A. Johansen, “Path planning for uavs under communication constraints using splat! and milp,” Journal of Intelligent & Robotic Systems, vol. 65, no. 1, pp. 265–282, 2012.
-  L. Evers, T. Dollevoet, A. I. Barros, and H. Monsuur, “Robust uav mission planning,” Annals of Operations Research, vol. 222, no. 1, pp. 293–315, 2014.
-  G. J. Lim, S. Kim, J. Cho, Y. Gong, and A. Khodaei, “Multi-uav pre-positioning and routing for power network damage assessment,” IEEE Transactions on Smart Grid, 2016.
-  OpenStreetMap contributors, “Planet dump retrieved from https://planet.osm.org ,” https://www.openstreetmap.org, 2017.
-  G. Boeing, “Osmnx: New methods for acquiring, constructing, analyzing, and visualizing complex street networks,” Computers, Environment and Urban Systems, vol. 65, pp. 126–139, 2017.
-  A. Hagberg, P. Swart, and D. S Chult, “Exploring network structure, dynamics, and function using networkx,” Los Alamos National Lab.(LANL), Los Alamos, NM (United States), Tech. Rep., 2008.
-  Gurobi Optimization, Inc., “Gurobi optimizer reference manual,” 2017. [Online]. Available: http://www.gurobi.com/
-  A. Hornung, K. M. Wurm, M. Bennewitz, C. Stachniss, and W. Burgard, “Octomap: An efficient probabilistic 3d mapping framework based on octrees,” Autonomous Robots, vol. 34, no. 3, pp. 189–206, 2013.