1.1 Problem Statement
Unmanned aerial vehicles (UAVs) equipped with visual sensors are rapidly emerging as the solution of choice for gathering information in many applications, especially in hostile or challenging environments, such as sensing of landslides niethammer2010uav, search and rescue missions doherty2007uav and forest fire inspection alexis_coordination_2009. Furthermore, UAVs are capable of providing high-resolution data/images that can be analyzed and used to investigate area characteristics, produce sparse and dense surface reconstructions mansouri2018cooperative, as well as hazard maps.
However, UAVs are subject to limitations related to payload and flight time Sinabattry2017. Payload limitations can prevent the usage of certain visual sensors and have a direct impact on flight time, thus lightweight cameras are preferred. Moreover, time limitations need to be compensated by optimization-based path planners, which aim at minimizing total flight length and duration.
In the presented approach, the UAV can maneuver in six degrees of freedom (DoF) (), while having a downward-looking camera fixed to the UAV frame. In this case, the camera footprint can be modeled by a polygon with four vertices called a cell at time instant . The size and shape of a cell varies with the position and orientation of the UAV. The general concept is illustrated in Fig. 1 where the red and yellow cells are the camera footprints of the UAV at an upright and a tilted attitude.
As one may observe, the covered area is different when the UAV deviates from the upright forward-facing orientation, while the description of the overall problem statement will be presented in Section 2. Disregarding the fact that the camera footprint depends on the attitude of the UAV may lead to insufficient coverage and suboptimal behavior. Thus, the main objective of this article is to propose a path planner while maximizing coverage. The proposed method takes into account the target area and generates attitude and thrust commands, while considering all DoF of the UAV/camera, resulting in a time-varying and attitude dependent camera footprint.
1.2 Background & Motivation
The 2D coverage path planning (CPP) problem, has been extensively investigated galceran2013survey; raffo2011path. Most CPP methods rely on the decomposition of the target area in sub-regions, which the agent subsequently sweeps torres2016coverage; perez2016selecting; conesa2016mix. A common taxonomy of CPP algorithms is based on the type of decomposition galceran2013survey: a) exact cellular decomposition methods choset1998coverage where the area is broken down to simple non-overlapping regions which the UAV traverses, b) Morse-based cellular decomposition methods acar2002morse where the area is decomposed based on critical points of Morse functions milnor2016morse and a motion planner algorithm guarantees that the agent passes through the critical points in the target area, c) landmark-based topological coverage methods where natural landmarks guide the decomposition process wong2006qualitative, d) contact sensor-based coverage of rectilinear environments butler1999contact, where the vehicle follows a cyclic path while building up a cellular decomposition of the area, e) grid-based methods shivashankar2011real where the target area is decomposed into a collection of uniform grid cells, and f) graph-based methods xu2011graph where the environment is represented by a graph, which is updated using available UAV sensors, while performing a coverage task.
However, none of the aforementioned methods considers the sensors’ placement and alignment on the UAV and the changing shape and size of the camera footprint with the attitude of the UAV. Not taking into consideration these factors can lead to low visual feedback quality or result in an unsatisfactory coverage outcome. Moreover, in most of the related approaches, the agent is considered to maintain a constant altitude and attitude of flight, while the segmentation is performed by a top-level procedure valente2011multi; avellar2015multi. Few works consider a three DoF movement and a camera footprint in path planning, e.g., in popovic2017online the informative path planning algorithm is proposed. In that approach, the global viewpoint selection and evolutionary optimization are combined to refine the UAVs trajectory. However, that approach is minimizing the path without taking into account of the camera orientation. Similarly, in MANSOURI20181, the target area is decomposed in relation to the camera footprint and the shortest path, which pass through all the camera footprints, is generated. However, the UAV altitude and attitude (roll and pitch) are assumed to be constant and only three DoF () are considered, while that is an off-line method that requires high computation power.
To the best of our knowledge no work so far has considered path planning taking into account the motion and orientation of the camera with six DoF, which results in non-constant attitude-dependent camera footprints.
Based on the aforementioned state of the art, the main contribution of this article is two-fold. The first major contribution, we address the problem of coverage of a path planner coupled with a time-varying attitude-dependent camera footprint for the first time, to the best of our knowledge. In this approach, the downward camera on the UAV possesses six DoF. The optimization procedure, aims at maximizing the covered area, and the coverage quality and provides input to the low level attitude controller. To provide close to real time computation for this highly complex problem of path planning instead of calculating the union of camera footprint cells and intersection of them to the target area, the target area is filled by a set of randomly generated particles and the path planner tries to harvest all particles by camera footprint cells. At each time instant the area is updated based on the previous covered area, points from visited areas are removed and the cells are calculated based on the position and orientation of the camera. Moreover, the UAV can maneuver in all directions and changes its altitude, this can effect the quality of visual feedback. As the UAV increases the altitude, the larger area is covered which results to shorter path however, the quality of the coverage is reduced. Thus, coverage quality function is proposed and added to the optimization problem; the term calculates the quality of the coverage based on the altitude and attitude of the UAV.
The second major contribution stems from the fact that the segmentation of the area and the path planner are integrated, thus establishing an overall framework for the path planning which considers changes in the shape and size of the camera footprint of the UAV with a downward camera, which results to reasonably smooth and shorter paths as there is no need to sweep the area for full coverage.
The efficiency of the proposed method is evaluated on multiple case studies. In the proposed approach the camera movement and orientation are coupled with the UAV, while for cameras with gimbals, the camera remains horizontal regardless of the motion around them, a fact that limits the DoF of the camera motion in relation to the UAV.
The rest of the article is structured as follows. Initially some mathematical preliminaries of the proposed problem are presented in Sections 2.1 and 2.2 followed by the presentation of the coverage quality function in Section 2.3. The proposed path planner is presented and discussed in Section 3, while in Section 4 several simulation results are presented with a corresponding comparison and discussion. Finally Section 5 concludes the article by summarizing the findings and offering some directions for future research.
2 Problem Statement
2.1 UAV Kinematics
In order to describe the UAV kinematics, coordinate frames need to be defined. One coordinate system is fixed to the aerial vehicle and it is called the body frame , while the other one is called the fixed (or inertial) frame and it is fixed to the earth . The schematic structure of the UAV with different coordinate systems is illustrated in Fig. 1, where is the altitude of the UAV and is the distance from the center of the UAV to the target area. The state of the system is , where , and define the position of the UAV’s center of mass and follow the following dynamics.
where is the total thrust which is exercised by propellers, is the mass of the system, is the gravity acceleration and and are the pitch, roll, and yaw of the UAV’s attitude. More details about the modeling part can be found in alexis2011switching.
2.2 Coverage with Camera Cells
Let us assume is a given region to be visually covered and the cells represent the camera’s footprint at the time instant. The goal is to generate attitude and thrust commands in order to maximize the coverage of the cells on the target area . In other words, the union of cells over time should cover the whole target area:
In this approach, the attitude and total thrust of the UAV is calculated by the path planner to maximize the covered area and the quality of the coverage. To be more specific, each cell can be identified by its vertices’ positions . Naturally, the -coordinate of is zero because we need to cover an area at zero altitude but the proposed framework allows to accommodate uneven and non-horizontal surfaces. The vertices’ positions obtained from the camera position , camera orientation , horizontal field of view (HFOV) and vertical field of view (VFOV) of the camera MANSOURI20181. The and are constants and can be obtained from camera specifications.
Furthermore, the orientation of the vehicle is represented by a rotation matrix. The three elementary rotation matrices , and
, which rotate vectors by angles, and about the , and axes respectively, are:
As it is shown in Fig. 1, the red rectangle is the camera footprint without consideration of the UAV’s orientation. The vertices of the rectangle centered at can be calculated from the position of the UAV, and camera’s VFOV and HFOV. In order to obtain the camera’s footprint resulting from the orientation of the UAV, the vertices of the rectangle should be rotated by the rotation matrices and translated to the UAV’s position. Therefore, the cells can be calculated by:
where are the vertices of the cell at time , and is the camera position at time . As a remark, the area to be covered may not be flat (a subset of ). In such cases, the proposed approach is still applicable provided that the angle of visual perception is not of importance. This is the case when there are not any too high objects such as buildings, trees or hills that may obstruct the coverage.
2.3 Image Quality
During the inspection, the UAV can maneuver in six DoF and fly at different altitudes. One solution for decreasing the length of the path is to increase the altitude of the UAV and allow higher tilts, which results to larger covered area. However, this solution results to images of lower resolution and reduces the quality of the coverage. Therefore, to take in account the quality of images in relation to the altitude of the UAV for the path planner, the coverage quality function is introduced as in papatheodorou2017collaborative (See Eq. (5) below). Without loss of generality, it is assumed that the coordinates of the camera are the same as those of the UAV and the distance from the camera to the center of mass of the UAV is negligible. The coverage quality is given by:
where and correspond to the minimum and maximum altitudes respectively.
Additionally, in order to consider the tilt of the camera, the image quality is evaluated at the actual distance between the camera and the area along the shooting direction. In case and , and will be equal, yet, nonzero values for and are necessary for the movement of the UAV, as it can be seen from (1).
The distance increases with an increase in and at constant . This means that the distance between the camera and the area can vary based on the orientation of the UAV. The corrected distance is given by:
By virtue of (6), the coverage quality is evaluated as .
3 Path Planning by Particle Harvesting
The path planner’s objective is to generate attitude and thrust commands taking into account the camera footprint while trying to capture high quality images by avoiding high altitudes and high tilts. A block diagram representation of the proposed path planner and the corresponding low-level controller is shown in Fig. 2. The path planner will generate a desired thrust and a desired orientation for the inner loop controller. The low-level attitude controller XIONG2014725
is able to track the desired roll, pitch and yaw and to calculate the corresponding rotor speeds for the UAV. A multi-sensor-fusion extended Kalman filter (MSF-EKF)msf
fuses the obtained pose information from the localization systems and the inertial measurement unit (IMU) measurements to provide orientation and position estimates of the vehicle. The estimated position and velocity are provided to an outer control loop (the navigation controller) which controls the position and velocity of the UAV by providing references to an inner loop. The inner loop controller (the attitude controller), uses the attitude and angular velocity estimates to control the UAV’s attitude.
Given the highly complex nature of the path planning problem, a formulation is proposed, which is based on filling the target domain with a set of particles (points in )
, generated by randomly sampling from a uniform distribution over. Then, the path planner harvests all particles in by commanding thrust and tilt signals to the inner loop controller (low-level controller), while taking into account the system dynamics and constraints.
The state of the system is , is the estimated state, and is a discretization of (1), which may be obtained by the Euler method, a Runge-Kutta method, or any other discretization method stetter1973analysis. Moreover, and are the state and control action ahead of steps form the current time and is difference of control action between and steps ahead of current time .
For the path planner presented in Section 2 the following finite horizon cost function is introduced.
The cost function of the optimization problem involves five terms. The first term minimizes the movement of the UAV which means the next position and orientation of the UAV should be close to the current one. This way the UAV avoids jerks. The second term minimizes the remaining area by removing the observed points from target area. The operation denotes the cardinality of the set and the operator denotes the relative complement and is defined as follows:
In the third term of (3) the quality of the visual feedback is considered. Large values of lead to higher coverage quality. The fourth term, penalizes the aggressiveness of the obtained control actions. The fifth term is the soft constraint on minimum hovering altitude of the UAV to reduce the ground effect disturbance. What is more, altitude constraints guarantee that the UAV will not fly at too high an altitude, which is a safety as well as legal requirement222The EU parliament are discussing laws that will regulate the maximum altitude for civilian drones. Already all EU countries have in place regulations for civilian drones prohibiting flights above a certain altitude (typically 100m) EUregulation. The ground effect is the change in the thrust generated by the rotors when UAV is close to the ground due to the interaction of the rotor airflow with the ground surface sanchez2017characterization. The operation is the plus operator which is defined as:
Additionally, , , , and are the weights for each term of the objective function which reflect the relative importance of each term; the highest importance is on to enforce high coverage.
The following optimization problem is defined:
At every time instant , a finite-horizon optimal path with prediction horizon is solved and a corresponding optimal sequence of control actions are generated. The first control action in that sequence is applied to the system in a receding horizon fashion as shown in Alg. 1, and the covered points are removed from . This way, the remaining area which can be covered along the prediction horizon is minimized and the path planner aims at maximizing coverage. The optimal control problem is solved in a receding horizon fashion rawlings2009model. Eventually, the target area is approximately covered and the mission comes to an end. Note that the problem is solved subject to the system dynamics (10b), actuation constraints (10d) and the altitude constraints (10e).
Furthermore, to calculate if the particles located inside of the cell, the ray casting algorithm 514556 is used. The ray casting method is a concept from computational geometry that determines if a point lies inside a bounded region (cell). The method counts the number of times that the ray casting from each point to a point outside of the cell intersects the cells boundaries. If the number of intersections is 0 or an even number, the point lies outside the cell otherwise lies inside. Moreover, the sequential quadratic programming method (nocedal2006sequential, Chap. 18) of MATLAB’s optimization toolbox (fmincon) running on a single core was used to solve the optimization problem. Overall, Alg. 1 summarizes the steps of the proposed path planner.
4 Simulation Results
The proposed method has been evaluated in a simulation environment. A validated model of the Ascending Technologies NEO hexacopter has been selected for the simulations. The overall mass of NEO considered in this work is 3.3 and the total thrust is constrained between 0 and 50 . The tuning parameters of the path planner are , , , , and and time of prediction is 0.1 . The constraints on altitude, attitude and velocity of the UAV are presented in (11). In all the following cases the agent’s initial condition is and it is assumed that the downward camera has a VFOV and a HFOV of 1.2 .
In order to reduce the computation time, the target area is filled with random points and the points inside of each cell are enumerated, providing an estimate of the actual area coverage. The average computation time at each step for all scenarios is and in the vast majority of tests the runtime is below
. Additionally, it is assumed that a low-level attitude controller is able to track the desired roll, pitch, yaw and thrust. Additive noise with standard deviation of 3for the positioning system and of for the thrust are considered, to simulate the effect of wind gust or other possible external disturbances. In what follows, different scenarios are presented in order to evaluate the performance of the proposed method. All simulations have been performed on a computer with an Intel Core i7-6600U CPU, 2.6GHz and 8GB RAM. For visualization purposes and without loss of generality the orientation of the camera in the figures was down sampled.
In the first scenario, the target area is considered to be a rectangle with a size of . The generated path is depicted in Fig. 3. Moreover the camera footprints of the motion in relation to the target area are shown. It can be seen that despite the fact the UAV is inside the target area, the camera captures a larger area based on the UAV’s attitude. This shows the importance of considering the time-varying camera’s footprint in path planner algorithms.
In the second case, an octagonal polygon with an area of is considered. The result of the path planner’s trajectory and footprints of the camera are depicted in Fig. 4. The area is covered completely and there is sufficient overlap between the camera footprints, which is critical for further image post processing brown2007automatic.
Furthermore, Fig. 5 presents the obtained results in the case of a complex nonconvex polygon. In this case the area is fully covered by the path planner. Additionally, it should be highlighted that, in all scenarios, the UAV does not sweep the area to achieve full coverage; by considering the camera footprints in the path planner, the path becomes reasonably smooth and shorter as there is no need to sweep the area for full coverage. As an example, it can be seen in Fig. 4 that the UAV covered all of the area, while simply moving on a rather simple path towards the center of the domain of the target area.
Moreover, Table 1 summarizes the path length and compares the proposed path planner, the path generates by the grid based method and the method in MANSOURI20181. The flight time depends on the technical characteristics of the UAV, whereas the total length of the path is more appropriate to be used for comparisons. In the benchmark methods, the UAV maneuvers at a fixed altitude of 1 . In case of grid based method the UAV sweeps the grids with a minimum number of turns to cover the target area and the method in MANSOURI20181 segments the area with fixed size camera footprint and provides shortest path that pass through all camera footprints. It is shown that the proposed path planner significantly reduces the length of the path in all cases, while the increase in prediction horizon results in shorter paths. This shows the impact of the time-varying footprint in path planners.
|Case 1||Case 2||Case 3|
|7.1 m||4.2 m||8.1 m|
|7.1 m||3.6 m||6.9 m|
|proposed path planner with||5.6 m||2.3 m||3.5 m|
|proposed path planner with||4.8 m||1.7 m||2.8 m|
Furthermore, Fig. 6 depicts the performance of the proposed path planner in relation to the number of random points inside of the target area. It is shown that the proposed path planner performance increases by the number of particles, however, more points can affect the computation time. Figure 7 depicts that the average and maximum value of computation time change in relation to the number of random points. Thus, more investigation is needed in order to determine the optimal number of random particles.
In this article we presented the particle harvesting methodology: a novel approach to the problem of visual area coverage and path planning with attitude-dependent footprints. The path planner takes into account the kinematics of the UAV, the image quality and the camera footprint to generate attitude references for the low-level controller. This way, the segmentation of the area and the path planner are integrated and the UAV does not require to sweep the area for complete coverage. This leads to shorter paths compared to other methods as shown in Table 1, which is an important advantage due to the limited flight time of the UAVs. The navigation problem using the proposed methodology is solved at every time instant in a receding horizon fashion.
Furthermore, the path planner is adequately fast which establishes an overall framework for the path planning of the UAV with a downward camera in order to solve the coverage problem. The presented method has been tested in different convex and non-convex polygons and in all examined scenarios, a highly satisfactory coverage has been obtained. The coverage is found to be consistently close to in all experiments. In order to achieve coverage, the original area can be slightly enlarged. The provided path for the UAV leads to frames with substantial overlapping, which is necessary for the algorithm to provide a visual overview of the scene.
Future work will focus on the investigation of the effect of multiple UAVs, time-varying areas, collision avoidance, as well as the extensive experimentation with the overall suggested scheme, including the optimal number of particles generated inside the area of interest.