Ego-Lane Estimation by Modelling Lanes and Sensor Failures

02/05/2020 ∙ by Augusto Luis Ballardini, et al. ∙ 18

We present a probabilistic ego-lane estimation algorithm for highway-like scenarios that is designed to increase the accuracy of the ego-lane estimate, which can be obtained relying only on a noisy line detector and tracker. The contribution relies on a Hidden Markov Model (HMM) with a transient failure model. The proposed algorithm exploits the OpenStreetMap (or other cartographic services) road property lane number as the expected number of lanes and leverages consecutive, possibly incomplete, observations. The algorithm effectiveness is proven by employing different line detectors and showing we could achieve much more usable, i.e. stable and reliable, ego-lane estimates over more than 100 Km of highway scenarios, recorded both in Italy and Spain. Moreover, as we could not find a suitable dataset for a quantitative comparison with other approaches, we collected datasets and manually annotated the Ground Truth about the vehicle ego-lane. Such datasets are made publicly available for usage from the scientific community.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 2

page 3

page 4

page 7

page 10

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Autonomous vehicles require an accurate understanding of the surrounding environment in order to safely plan their actions. One such fundamental perception task concerns the localization of the vehicle. Autonomous vehicles cannot always rely on global positioning based on Global Navigation Satellite System (GNSS) signals (e.g., GPS, Beidu, Glonass, Galileo, etc.) because signals from satellites use to undergo multi-paths and physical barriers, leading sporadically to very poor position accuracy or even to no estimate at all. Therefore navigation modules usually couple the GNSS systems with cartographic maps and methods that leverage the road network graph as well as other common features [Steder2015, Raaijmakers2015, Gevers2014, Ballardini2016, Ballardini2017], like buildings, crossings or roundabouts, which are retrieved from cartographic services like OpenStreetMap. Therefore maps represent an important piece of information to exploit as prior in vehicle localization. Even though methodologies based on GNSS and maps, usually known as lock-on-road procedures, see e.g., [Alonso2012, Ballardini2015], lead to remarkable increases in the localization accuracy, they still do not use to achieve lane-level localization, i.e., accuracies in the order of 0.1m [Levinson2011].

Lane-level localization is actually a term with broad meaning and it usually might refer to two different problems: on the one hand it might refer to the determination of the lane currently occupied by the vehicle, a problem which is also known as host-lane or ego-lane estimation; on the other hand it might refer to the estimation of the lateral position of the vehicle inside the lane or whole road. The latter problem is relevant for the lower level control of the vehicle, while the first is relevant in the higher-level (tactical) control of the vehicle, i.e., trajectory and manoeuvre planning. Solutions to the latter live in , while solutions to the first live in . This work deals with the ego-lane estimation problem.

In both cases, some form of optical detection of the road lines is usually exploited. Quite frequently images obtained from the forward-looking camera(s) of the vehicle are used, and processed in order to detect the road lines. Typically, such lines are in turn processed to infer the vehicle ego-lane at the time of the image capture.

In this paper we present a probabilistic algorithm aimed at enhancing the ego-lane estimation obtained from a line detector. The algorithm exploits a GNSS measure as well as the number of lanes of the road, retrieved from a service like OpenStreetMap, as prior. Differently from other works available in the literature, ours is a modular, hence reusable, algorithm for improving the ego-lane estimation that could be obtained from a generic line detector. Our algorithm relies on a Hidden Markov Model (HMM) with a transient failure model, which allows us to accommodate inaccurate or missing road line detections.

The paper is organized as follows. Section II provides a short overview of the existing ego-lane estimation literature, Section III describes the proposed algorithm and Section IV introduces the experimental configurations and datasets. Finally, Section V critically presents the experimental results of the algorithm, and is followed by concluding remarks.

Ii Related work

Ego-lane estimation for autonomous driving has been extensively investigated in the last decades. The first achievements were obtained by the group of Prof. Dickmanns [Dickmanns1992]

, they introduced a road representation model based on clothoids, which were then updated with the image measurements by using Kalman filters.

Basing on these results, an active research has been conducted in the subsequent years [Pomerleau1995, Bertozzi1998, Taylor1996, Wang2004a]. Heterogeneous modeling techniques for the lane markings, including parabolas, clothoids, poly-lines or b-splines, were proposed, typically computed from images, after some preprocessing phases designed to remove clutter and irrelevant areas.

One of the most difficult tasks that are to be solved to identify the ego-lane is the detection of the road surface. Achieving a good discrimination of the road surface from other parts is crucial since it is the basis for further processing, but this detection is usually adversely affected by the large amount of clutter usually found on real roads. While faded road markings, unusual or specific weather conditions, or even light variations might severely affect the road surface detection, the visibility of the road surface is quite frequently hampered by the presence of other vehicles, thus requiring different considerations to solve the problem.

Most of the current Advanced Driver Assistance Systems (ADAS), like Lane Departure Warning (LDW) or Adaptive Cruise Control (ACC), require just a partial understanding of the whole observed scene, like the lines of the vehicle’s lane or the lane crossing points, in highway-like scenarios [BarHillel2014, Mccall2006a].

For what concerns sensing, even though LIDAR-based algorithms sport the advantage of active lightening, vision-based algorithms are, as for today, the most frequently used sensing approach for line detection and ego-lane estimation, since road markings are designed to be human-visible in mostly all driving conditions, see e.g., [BarHillel2014], which has an excellent review of approaches in lane localization.

With the objective to pursue lane-level localization, the authors in [stefanorf2016] propose to exploit the objects present in the surrounding of the vehicle and to describe the probabilistic dependencies between the object measurements, by means of a factor graph model. A similar proposal comes from the authors of [Flade2016], where Histogram of Oriented Gradients are used to align the images acquired from a front facing camera to the road lane markings, to improve the vehicle localization.

Many authors propose, in order to increase the performance of ego-lane estimation algorithms, to exploit additional road information gathered by map services as well as information provided by GNSS. In this regard, an interesting approach is presented in [Gao2009]

, where the authors tackled the ego-lane estimation as a scene-classification problem. They infer the lane number in a holistic fashion, leveraging both spatial information and objects around the vehicle, and finally training the best classifier with different learning algorithms. In

[Kim2008] the author presented a robust lane-detection-and-tracking algorithm combining a particle filtering technique for lane tracking and RANSAC for the detection of lane boundaries. The work detects left and right lane boundaries separately, without exploiting fixed width lane models, and combining lane detection and tracking within a common probabilistic framework.

The authors in [Kuhnl2013, Rabe2016], respectively in highway and urban scenarios, propose to exploit boosting classifiers and particle filtering approaches. A similar research was performed by [Lee2015]

, where multiple evidence from a visual processing pipeline was combined within a Bayesian Network approach.

Closer to our proposal are the works in [Nieto2008, Jiang2010, Kang2014], where the authors specifically address the multiple-lane detection problem. In [Nieto2008] multiple lane detections are performed after a first processing phase, where the authors identify the ego-lane geometry. Then, adjacent lanes are first hypothesized and then tested, assuming same curvature and width for all lanes, a fair assumption for most of multi-lane roads, including highways. Similarly, the work proposed in [Jiang2010] also considers highway scenarios and parallel lane markings, with respect to the detected ego-lane. More recently, the authors in [Kang2014] proposed a multi-lane detection algorithm based also on a hypothesis generation and testing scheme, ensuring an accurate geometric estimation by means of a robust line fitting pipeline and vanishing point estimation.

Differently from the other contributions, where the authors propose new detection pipelines for the ego-lane estimation problem, here we introduce a generic scheme aimed at improving the ego-lane estimation capabilities of potentially every line detector. Also, the output of a lane detection algorithm could be fed into our algorithm, to increase its performance in ego-lane estimation. Our aim was to enhance the localization capabilities of the scene understanding framework proposed in 

[Ballardini2015], introducing a lane-awareness module capable of reducing localization errors in highway-like environments. As a by-product of the modularity of the framework, we could easily compare the localization results obtained with and without the new proposed algorithm.

Iii Proposed algorithm

The goal of the proposed algorithm is to estimate the vehicle ego-lane in highway scenarios, i.e., when the topology of the roadway does not change, e.g., because of exit ramps, bifurcations, etc. The input of the proposed algorithm are both a global localization, at the level of accuracy provided by today GNSS, and the detections of the road line markings.

The algorithm is designed to tolerate occasional temporary failures of the underlying line detector as well as its noisy measurements. A line detector is a software component that detects and tracks the relative position of both dashed and continuous road lines, with respect to the vehicle.

Our algorithm relies on a probabilistic model of the functioning of the ego-lane estimation, and is designed to be independent from the line detector, so we can compare the results of our algorithm when working on the output of different line detectors.

Indeed, the estimation of the vehicle ego-lane can be regarded as a consequence of the outcome of the line detection procedure. Actually, the position of all the road lines w.r.t. the vehicle, allows to determine the ego-lane by using simple geometric considerations, on a per-frame basis. Unfortunately, line detections are usually not fully reliable, being hampered by faded road markings, cluttering elements from the nearby traffic, or weather conditions, see e.g., Figure 1, (a)a and 4.

Fig. 1: An example of a moderately congested condition on the A4 (Turin - Milan - Venice - Trieste) highway, Italy. Even at this moderate level of congestion most road markings are hidden by traffic.

However, we combine the line detections with an index about how reliable each detection is, we call this index Line Reliability Index (LRI). This index, together with our proposed probabilistic model, allows an appropriate handling of the noisy output of the line detectors.

Furthermore, consider the situation depicted in Figure (a)a, surely a critical situation for ego-lane estimation. Even though the exact lane cannot be estimated from the only detected line, the distance measured from such line would allow to limit the uncertainty only to the compatible lanes, as depicted by the green highlighted lanes in Figure (b)b.

(a)
(b)
Fig. 2:

In (a) only one line out of four is detected, thus the highlighted lanes in (b) have a higher probability of being the vehicle ego-lane, as implicated by the distances to the detected lines.

Fig. 3: If only the line indicated with the arrow were detected, the probability of being in Lane would be {0,0.5,0.5}. The idea of exploiting the plausibility given by each line, repeated for all the detected lines, and together with our model, allows to solve the ego-lane estimation problem. Here the green and blue visually suggest the reliability of the lines (green is higher).

Our proposal is to tackle the ego-lane estimation with a probabilistic model, in order to allow the system to infer the ego-lane by leveraging consecutive, yet incomplete, observations over time. An HMM is proposed, with a Lane variable that can take values, corresponding to the number of lanes retrieved from an OpenStreetMap-like service.

Iii-a Line detection and tracking

In this section, we shortly describe the line detection and tracking algorithms used in the experimental activity. We first introduce a “simple” line detector and tracker that is capable to work with both a stereo and a monocular camera configuration, requiring the projection of the camera(s) to be calibrated w.r.t. the vehicle frame before its usage. This algorithm consists of the following steps:

  • The contours of the road markings are extracted from the Bird Eye / Inverse Perspective view (BEV / IPV) of the left camera image and discarded if their area is below a threshold. To compute the BEV / IPV image an homography matrix is computed, basing on the intrinsic values of the projection model, and the extrinsic values w.r.t. the road surface. The contours in the BEV image are then determined using the algorithm proposed in [suzuki1985topological].

  • The algorithm then tries to fit, onto the detected contours, a fixed number of lines or clothoids (both the number and the type are parameters), trying to cover the highest number of contour areas; if the stereo configuration is available, the algorithm exploits it to exclude lines / clothoids not lying on the ground plane. The ground plane equation is evaluated using the output of the SGBM [sgbm] or the ELAS [Geiger2010ACCV] stereo-matching algorithm (this choice is a parameter). Concluding, both a monocular and stereo version of the algorithm is available.

  • The parameters of each line / clothoid are then updated by means of a Kalman filter.

W.r.t. the previous (e.g., ) frames, the number of times the line is detected is taken as the Line Reliability Index (LRI). Furthermore, again for each line, the LRI allows to set a flag, named isValid, once the counter reaches its maximum value; to reset the flag the counting goes on with hysteresis, so that the flag is not reset until LRI goes below a certain fraction of its maximum value.

Line1: isValid = 1; continuous=1; LRI: 10; offset: -9.15m Line2: isValid = 0; continuous=0; LRI: 09; offset: -6.47m Line3: isValid = 1; continuous=0; LRI: 07; offset: -2.15m Line4: isValid = 0; continuous=1; LRI: 00; offset: +0.99m
Box 1 The line detector output for the image in Figure 3. The isValid flag is set to TRUE when LRI=10, and reset using a hysteresis counting procedure. A negative offset refers to lines on the left of the vehicle.

This simple line detector and tracker achieves good performances only under optimal illumination conditions and, as depicted in Figure 3 and shown in the corresponding results in Box 1, dashed lines and shadows are not always handled correctly. However, it allowed to evaluate the effectiveness of our contribution, which is designed to enhance the vehicle ego-lane estimation by exploiting a noisy sensor as well as the road lane properties gathered from an OpenStreetMap-like service.

Moreover, to effectively assess the proposed probabilistic model, we performed an extensive search to find other line detectors, sporting a measure of the reliability of the detection of each line. Although the literature about line generic detection is huge, as the problem has been investigated since the beginning of digital image processing, solutions dedicated to detecting road lines that are also available with software reduce, to the best of our knowledge, to the following two.

  • The proposal by Mohamed Aly [aly2008real], which exploits a robust approach based on a line / bezier line tracker and a RANSAC procedure. As far as their publicly available results show, this algorithm is able to detect all the lines on the road surface, making it one of the most complete and well performing algorithms. Nevertheless, on one hand the software that is publicly available does not include the tracking module, and on the other the number of parameters is overwhelming (about one hundred). These aspects make this software extremely hard to use. We have not been able to find a reasonably good configuration in the limited time available. In conclusion, this option, although very appealing, could not be used in our experimental activity.

  • The solutions by Hur [hur2013multilane] (MLD in the following), which detects a maximum of four lines, corresponding to the markings of the current lane as well as of the two neighboring lanes. This software could be adapted to our requirements for this work, e.g., introducing a procedure for determining whether a line is dashed or continuous.

Neither of the two solutions can exploit a stereo camera configuration, and both rely on the extrinsic projection parameters to determine the distance of each line from the vehicle, exploiting the BEV / IPM image.

Fig. 4: In this figure, just two out of the five lines are correctly detected and tracked. The shadow created by the Jersey barrier prevents the correct detection of the leftmost line, although in our opinion the line might be detected, perhaps with a different set of parameters, which might in turn bring in other misdetections. An error might also arise with dashed lines, whenever the space between two consecutive detected dashes is increased by some vehicle. The two rightmost lines are not detected because of their limited thickness; also in this case a different processing pipeline could detect such lines, usually introducing other errors.

Iii-B Tentative vector and reliability of the whole detection

To exploit the measurements provided by a line detector and tracker like the ones mentioned before, we derived a probabilistic (inverse) sensor model, which exploits both the spatial information carried by the lines and the LRIs, both produced by the line detector and tracker. The processing pipeline is therefore composed as follows. First, the lines are sorted, in ascending order, based on their lateral offset w.r.t.

the vehicle. Then, a vector of counters, of the size of the number of lanes, is created. This vector is called

tentative (implied: distribution of the belief on the state as from the measurements, as usual for an inverse sensor model). The values in this vector are determined by iterating the following steps for all the valid lines, i.e., the lines whose isValid flag is set, taking into consideration whether each line is dashed or continuous.

  • 1 is added to the i-th tentative vector value, if it is in accordance with the measurement, i.e., if being in the i-th lane is compatible with the line position; this has the objective to cumulate the plausibility of being in the i-th lane, given the detected lines.

  • If the line has the continuous flag set, an additional Bonus Value (BV) is added to the tentative vector position (based on the distance w.r.t. the line); this has the objective to represent the fact that continuous lines are more informative, as they usually are the leftmost / rightmost lines of the road.

After the evaluation of the line indicated with an arrow in Figure 3, the resulting tentative vector would be .

During the iteration on all the lines, we also accumulate all the LRI counters, and compute the fraction w.r.t. the maximum LRI value times the current number of expected lines. This value is taken as an index of the overall reliability of all the line detections of the current frame, namely a Whole Output Reliability (WOR) of the detector. This in turn can be taken as an observation of the sensor being properly functioning or not.

It has to be noticed that some of these rules could be not adequate for all line detectors. For example, if a line detector could not provide a continuous flag or a reliability index for each line, the set of rules must be modified, in order to provide a frame-level tentative vector and an overall WOR.

Iii-C HMM with Transient Failure Model

To tackle the unavoidable problem of sensor failures, i.e., the cases of the sensor not being able to correctly determine the value of the ego-lane state variable, we applied a filtering algorithm based on an HMM, which includes a state representing the functioning of the sensor itself. For an introduction to HMM see [russellnorvig]

. The proposed model allows to take advantage of incomplete and/or noisy road line observations in a probabilistic fashion, to better estimate the current ego-lane as well as whether the sensor is properly working or not. In our opinion, this extra degree of freedom,

i.e., the explicit modeling of the sensor functioning state, allows a better performance, w.r.t. considering the ego-lane as the only unknown, as it gives an extra area of accommodation for matching the unknown value of the state variables to the observations.

The HMM implements a filtering procedure over discrete random variables, where each iteration depends on the parameterization in Equation 

1, see below for an explanation of the parameters.

(1)

There are four variables, for each time frame, see Figure 5:

  • Lane: represents the lane where the vehicle is (believed to be) located. This is a scalar variable, taking discrete values, which identify one of the road lanes.

  • Sensor State (also SS in the following): a scalar variable, taking one out of two values, which represents whether the sensor is properly working or not, and the taken values could be OK or BAD.

  • Detector Output: as we developed an inverse sensor model, this variable is homogenous to the Lane variable; in principle it would be a scalar that can assume one out of values, in practice, to represent a belief over the ego-lane, it will be an components vector, summing to 1, where each one represents the probability of that lane being the vehicle ego-lane.

  • Reliability Index (WOR): analogously to Detector Output, here we have in principle a scalar variable, taking one value out of the possible values; in practice WOR will be a 2 components vector, where each of the 2 values represents respectively how much the detector is believed to be properly working or not properly working.

Fig. 5: Two time frames of the HMM. The single circled variables are hidden, and the double circled variables are observable.

The dependencies between these variables are described using Conditional Probability Tables (CPTs), i.e.,

tables that describe the probability distribution of a variable (the values the variable can take are in the columns), given the state of its parents in the graph (the values the state variables can take are in the rows) of the model in Figure 

5. In other words, the CPT tables allow to forward compute the expectations on the variables, which will be then updated with the observations.

The first two variables (Lane and Sensor State) have temporal dependencies, while the others only depend from variables within the same time frame.

Table I basically describes the dynamics of the lane variable, i.e., the dynamics of the vehicle in the roadway. Thus, given a value of Lane at time , we expect the Lane variable at time

to follow a normal distribution with the Lane at time

as mean and

as variance. This description needs a refinement, which is below; its functioning is indicated with

, applied to a normal. The variance controls how likely is for the vehicle to change lane during a time interval. Notice that is set according to both the actual speed of a vehicle in changing lane and the frequency of the ego-lane estimation, i.e., the image frame rate. For the belief on the lane to have the discrete domain of lanes, we transform (

) the normal into a discrete set of values as follows: consider the domain of the probability density function (PDF) to be orthogonal to the lane axis. The PDF is fixed on the center of the lane as in the Table 

I; the integral across the lane is taken, from one sideline to the other. The lane widths could be assumed known, which is the approach taken in this work or, if available, they could be retrieved from an an OpenStreetMap-like service, as for the number of lanes. As the numbers obtained integrating across the lanes does not sum up to 1, e.g., because of the infinite extension of the normal vs. the finiteness of the roadway, the last step represented by is their normalization.

Lane
Lane 1 2
1
2
TABLE I: Lane CPT

Table II describes the sensor dynamics w.r.t. its mistakes, which in turn is related to the conditions of the road markings, and to the lightening conditions. If the sensor is working properly, it will remain in the properly working state with probability , and switch to giving a wrong output with probability . If the sensor is not in a properly working state, then it will remain in such a state with probability , and switch back to properly working with probability . This idea, by suitably setting and , aims to represent the real experience of the sensor working properly most of time, and then presenting a sequence of frames with the sensor providing a wrong output.

Sensor State
Sensor State OK BAD
OK
BAD
TABLE II: Sensor State CPT

Table III describes the detector output w.r.t. the state of its parents in the HMM: Sensor State and Lane. Therefore, the table has as many rows as the combinations of the values that the two conditioning variables can take, while it has as many columns as the values that the conditioned variable can take. As we have an inverse sensor model, the output of the detector is homogeneous to the Lane variable and we will have columns. If Sensor State is OK, we expect the detector to provide an output represented by a discrete set of numbers, drawn from normal distributions like it was for the Lane variable, i.e., expected value set onto the real lane, function steps like for the Lane variable, but variance described by the parameter , to represent the accuracy of the sensor in determining the ego-lane when properly working; this is clearly a different value w.r.t.

the lane-change dynamics of the vehicle. However, if Sensor State is BAD, the detector output will be independent of the real lane, thus the output will be uniformly distributed.

Detector Output
Sensor State Lane 1 2
OK 1
2
BAD 1
2
TABLE III: Detector Output CPT

We can similarly define the CPT for WOR, the expected overall reliability of the output of the sensor, e.g., as in Table IV. As in our model the WOR has only one parent, Sensor State, only two rows are needed in the Table. Notice though that a different model could be considered, where WOR depends on the Lane also, which might be more appropriate for certain situations and / or line detectors. The parameters and represents the probability of a correct evaluation of the Sensor State when the Sensor State is, respectively, OK and BAD.

WOR
Sensor State OK BAD
OK
BAD
TABLE IV: WOR CPT

Iii-D Inference

To perform inference with our model means to compute the most probable lane, given the tentative vector, i.e., the output of the (inverse) sensor for the Lane and the Sensor State, which are in turn based on the output of the line detector and tracker.

To compute the belief on the HMM state (i.e., Lane and Sensor State) at time , we start from the HMM state at time . Firstly, leveraging Table I and Table II, we compute the expectation at time on these variables. Sensor State is shortened in SS.

(2)
(3)

Then, in order to incorporate the new evidence carried by the Tentative vector and the WOR index, the Bayes formula has to be applied, so obtaining the belief over the HMM state at time . There are in general two ways of applying the Tentative vector and WOR index as evidence in the inference. The first way is to simply consider the most probable value of the Tentative vector and WOR index as an hard evidence for the belief on the state. The other way, instead, is to consider the Tentative vector and the WOR index as soft evidence [bilmes2004virtual]. Since this second way allows a more complete representation of the evidences, we have only considered this approach, also because the additional computation required for the inference is irrelevant, given the reduced size of the model.

In Figure 6 is depicted an overview of the proposed model. The blue box is the core of our model, i.e., the HMM, which is independent from the detector. The red box is the line detector and tracker, on which we tried to impose no constraint, keeping our model as general as possible. Lastly, the green box is the set of rules used to connect the line detector and tracker output to the HMM. This part may need to be changed with the output format of the line detector and tracker.

Fig. 6: An overview of the proposed model.

Iv Experimental Configuration

To effectively verify the improvements achieved by our model, we collected two datasets in real driving conditions. The first dataset was recorded in the A4 highway, Italy, from Bergamo to Milan. The second dataset is from the A2 highway area of Alcalá de Henares, Spain. Both the datasets were recorded at 10 fps and have a resolution of 1312x540 and 1392x400 pixels respectively. Differently from standard datasets like KITTI, in which the highway sequences only contain few lanes, we drove our vehicles on wider highways with 3 and 4 lanes (Spain and Italy respectively), including more than 100 lane changes in the A4 highway sequences. We manually annotated the ground truth (GT) about the correct lane for more than 20K frames, considering as the leftmost lane as in (a)a. For each frame, we also included a “crossing flag” to indicate whether the vehicle is changing lane, so to exclude ambiguous lane assignments, see Figure 7.

(a)
(b)
Fig. 7: Two frames from the proposed annotated dataset, the detector output is overlaid. In the top image, the vehicle was traveling in the A4 highway, Italy, and performing a lane change, i.e., in the dataset the crossing flag is set. The bottom image depicts a frame from the A-2 highway, Spain.

For each experimental setting, the parameterization of Equation 1 was empirically defined after an optimization phase, aimed at identifying the best parameter set w.r.t. the GT. Please, notice that the parameters are different since we changed either the detector configuration or the detector itself. The parameter values used during the experiments in Italy and Spain are reported in Table V and  Table VI.

Run
Det.
Alg.
s1 s2 p1 p2 p3 p4 BV
01 ML 0.336 0.696 0.895 0.894 0.690 0.461 7
02 SL 0.481 0.296 0.160 0.970 0.613 0.975 9
03 MC 0.407 0.360 0.853 0.993 0.303 0.640 4
04 SC 0.386 0.598 0.906 0.994 0.311 0.595 7
05 MLD 0.324 0.707 0.223 0.963 0.779 0.873 1
  • M = mono, L = line, S = stereo, C = clothoid, MLD = [hur2013multilane]

TABLE V: Parameterizations used for the different experimental settings - Italy ( = 4).
Run
Det.
Alg.
s1 s2 p1 p2 p3 p4 BV
06 ML 0.407 0.258 0.692 0.590 0.180 0.459 9
07 SL 0.364 0.460 0.640 0.556 0.409 0.812 8
08 MC 0.313 0.532 0.092 0.255 0.971 0.605 7
09 SC 0.382 0.483 0.941 0.977 0.885 0.984 9
10 MLD 0.343 2.907 0.283 0.991 0.903 0.060 5
  • M = mono, L = line, S = stereo, C = clothoid, MLD = [hur2013multilane]

TABLE VI: Parameterizations used for the different experimental settings - Spain ( = 3.

As further research is required for this problem, and to allow future researchers to compare their work with ours, we published our datasets and the associated GT values online111The dataset and the annotations are available on our lab’s website: http://www.ira.disco.unimib.it/ego-lane-estimation-by-modeling-lanes-and-sensor-failures.

(a)
(b)
(c)
Fig. 8: A short section of the 4-lanes A4 highway in Italy. More saturated colors correspond to a higher probability of being in a lane. The figure presents a comparison between our model (a) w.r.t. the results achieved using the detector only (b). (c) is the GT, in grey are the transitions between lanes. Our proposal yields good improvements, with more stable detections, w.r.t. the detector’s results.

V Results

We evaluated the localization performances of our proposal comparing the ego-lane estimates w.r.t. the GT, for a set of configurations (with / without the proposed model, line detector, etc.). The results are presented in the Tables, on a per-frame basis, reporting whether correct lane classifications were achieved.

Actual Class
Lane 1 Lane 2 Lane 3 Lane 4 Total

Predicted Class

Lane 1 1781 21 5 1 1808
Lane 2 59 1613 190 46 1908
Lane 3 147 42 730 166 1085
Lane 4 1 21 17 215 254
Unassigned 196 754 1310 456 2716
Support 2184 2451 2252 884 7771
  • Mean Precision: 0.83 Mean Recall: 0.51 Mean F1: 0.61 Accuracy 0.55 Logloss: 0.99

TABLE VII: Run#1 - Detector Only
Actual Class
Lane 1 Lane 2 Lane 3 Lane 4 Total

Predicted Class

Lane 1 2079 27 0 0 2106
Lane 2 67 2284 346 67 2764
Lane 3 17 117 1853 409 2396
Lane 4 21 23 53 405 502
Unassigned 0 0 0 3 3
Support 2184 2451 2252 884 7771
  • Mean Precision: 0.84 Mean Recall: 0.79 Mean F1: 0.80 Accuracy 0.85 Logloss: 0.51

TABLE VIII: Run#1 - Model
Actual Class
Lane 1 Lane 2 Lane 3 Lane 4 Total

Predicted Class

Lane 1 1941 31 2 4 1978
Lane 2 9 1611 147 31 1798
Lane 3 6 21 762 171 960
Lane 4 3 3 5 223 234
Unassigned 225 785 1336 455 2801
Support 2184 2451 2252 884 7771
  • Mean Precision: 0.90 Mean Recall: 0.53 Mean F1: 0.64 Accuracy 0.58 Logloss: 0.93

TABLE IX: Run#2 - Detector Only
Actual Class
Lane 1 Lane 2 Lane 3 Lane 4 Total

Predicted Class

Lane 1 2117 94 2 0 2213
Lane 2 64 2288 352 49 2753
Lane 3 3 69 1883 385 2340
Lane 4 0 0 15 450 465
Unassigned 0 0 0 0 0
Support 2184 2451 2252 884 7771
  • Mean Precision: 0.89 Mean Recall: 0.81 Mean F1: 0.83 Accuracy 0.86 Logloss: 0.94

TABLE X: Run#2 - Model
Actual Class
Lane 1 Lane 2 Lane 3 Lane 4 Total

Predicted Class

Lane 1 1438 30 51 0 1519
Lane 2 79 1795 238 124 2236
Lane 3 350 56 1207 269 1882
Lane 4 5 47 33 217 302
Unassigned 312 523 723 274 1832
Support 2184 2451 2252 884 7771
  • Mean Precision: 0.77 Mean Recall: 0.54 Mean F1: 0.62 Accuracy 0.59 Logloss: 1.07

TABLE XI: Run#3 - Detector Only
Actual Class
Lane 1 Lane 2 Lane 3 Lane 4 Total

Predicted Class

Lane 1 1967 69 48 1 2085
Lane 2 85 2268 313 70 2736
Lane 3 104 99 1817 317 2337
Lane 4 28 15 74 496 613
Unassigned 0 0 0 0 0
Support 2184 2451 2252 884 7771
  • Mean Precision: 0.83 Mean Recall: 0.79 Mean F1: 0.81 Accuracy 0.84 Logloss: 0.63

TABLE XII: Run#3 - Model
Actual Class
Lane 1 Lane 2 Lane 3 Lane 4 Total

Predicted Class

Lane 1 1596 27 12 8 1643
Lane 2 20 1910 190 20 2140
Lane 3 11 27 1283 327 1648
Lane 4 7 0 6 206 219
Unassigned 550 487 761 323 2121
Support 2184 2451 2252 884 7771
  • Mean Precision: 0.89 Mean Recall: 0.57 Mean F1: 0.67 Accuracy 0.64 Logloss: 0.80

TABLE XIII: Run#4 - Detector Only
Actual Class
Lane 1 Lane 2 Lane 3 Lane 4 Total

Predicted Class

Lane 1 2095 38 0 0 2133
Lane 2 86 2286 240 17 2629
Lane 3 3 127 1907 306 2343
Lane 4 0 0 105 552 657
Unassigned 0 0 0 9 9
Support 2184 2451 2252 884 7771
  • Mean Precision: 0.87 Mean Recall: 0.84 Mean F1: 0.85 Accuracy 0.88 Logloss: 0.49

TABLE XIV: Run#4 - Model
Actual Class
Lane 1 Lane 2 Lane 3 Lane 4 Total

Predicted Class

Lane 1 0 0 0 0 0
Lane 2 1 0 45 0 46
Lane 3 39 0 6 0 45
Lane 4 0 0 0 0 0
Unassigned 2122 2413 2171 876 7582
Support 2162 2413 2222 876 7673
  • Mean Precision: 0.03 Mean Recall: 0.00 Mean F1: 0.00 Accuracy 0.00 Logloss: 1.80

TABLE XV: Run#5 - Detector Only
Actual Class
Lane 1 Lane 2 Lane 3 Lane 4 Total

Predicted Class

Lane 1 1120 549 163 59 1891
Lane 2 448 1320 812 160 2740
Lane 3 576 407 1037 515 2535
Lane 4 18 137 210 142 507
Unassigned 0 0 0 0 0
Support 2162 2413 2222 876 7673
  • Mean Precision: 0.44 Mean Recall: 0.42 Mean F1: 0.42 Accuracy 0.47 Logloss: 1.26

TABLE XVI: Run#5 - Model
Actual Class
Lane 1 Lane 2 Lane 3 Total

Predicted Class

Lane 1 1405 162 2 1569
Lane 2 68 3004 275 3347
Lane 3 27 6 886 919
Unassigned 479 424 1132 2035
Support 1979 3596 2295 7870
  • Mean Precision: 0.91 Mean Recall: 0.64 Mean F1: 0.73 Accuracy 0.67 Logloss: 0.73

TABLE XVII: Run#6 - Detector Only
Actual Class
Lane 1 Lane 2 Lane 3 Total

Predicted Class

Lane 1 1762 263 0 2025
Lane 2 150 3323 326 3799
Lane 3 66 10 1969 2045
Unassigned 1 0 0 1
Support 1979 3596 2295 7870
  • Mean Precision: 0.90 Mean Recall: 0.89 Mean F1: 0.89 Accuracy 0.89 Logloss: 0.49

TABLE XVIII: Run#6 - Model
Actual Class
Lane 1 Lane 2 Lane 3 Total

Predicted Class

Lane 1 1380 163 2 1545
Lane 2 69 2850 216 3135
Lane 3 29 19 901 949
Unassigned 501 564 1176 2241
Support 1979 3596 2295 7870
  • Mean Precision: 0.91 Mean Recall: 0.62 Mean F1: 0.72 Accuracy 0.65 Logloss: 0.73

TABLE XIX: Run#7 - Detector Only
Actual Class
Lane 1 Lane 2 Lane 3 Total

Predicted Class

Lane 1 1749 219 0 1968
Lane 2 155 3322 278 3755
Lane 3 74 55 2017 2146
Unassigned 1 0 0 1
Support 1979 3596 2295 7870
  • Mean Precision: 0.90 Mean Recall: 0.89 Mean F1: 0.89 Accuracy 0.90 Logloss: 0.48

TABLE XX: Run#7 - Model
Actual Class
Lane 1 Lane 2 Lane 3 Total

Predicted Class

Lane 1 1263 82 3 1348
Lane 2 57 2527 208 2792
Lane 3 69 38 748 855
Unassigned 590 949 1336 2875
Support 1979 3596 2295 7870
  • Mean Precision: 0.90 Mean Recall: 0.55 Mean F1: 0.67 Accuracy 0.57 Logloss: 0.95

TABLE XXI: Run#8 - Detector Only
Actual Class
Lane 1 Lane 2 Lane 3 Total

Predicted Class

Lane 1 1671 199 49 1919
Lane 2 149 3262 565 3976
Lane 3 158 135 1681 1974
Unassigned 1 0 0 1
Support 1979 3596 2295 7870
  • Mean Precision: 0.84 Mean Recall: 0.82 Mean F1: 0.83 Accuracy 0.84 Logloss: 0.64

TABLE XXII: Run#8 - Model
Actual Class
Lane 1 Lane 2 Lane 3 Total

Predicted Class

Lane 1 1375 90 11 1476
Lane 2 46 2882 251 3179
Lane 3 55 19 824 898
Unassigned 503 605 1209 2317
Support 1979 3596 2295 7870
  • Mean Precision: 0.91 Mean Recall: 0.61 Mean F1: 0.72 Accuracy 0.64 Logloss: 0.79

TABLE XXIII: Run#9 - Detector Only
Actual Class
Lane 1 Lane 2 Lane 3 Total

Predicted Class

Lane 1 1777 166 10 1953
Lane 2 88 3348 363 3799
Lane 3 113 82 1922 2117
Unassigned 1 0 0 1
Support 1979 3596 2295 7870
  • Mean Precision: 0.89 Mean Recall: 0.88 Mean F1: 0.89 Accuracy 0.89 Logloss: 0.56

TABLE XXIV: Run#9 - Model
Actual Class
Lane 1 Lane 2 Lane 3 Total

Predicted Class

Lane 1 153 104 0 257
Lane 2 58 1714 365 2137
Lane 3 72 138 203 413
Unassigned 1653 1590 1727 4970
Support 1936 3546 2295 7777
  • Mean Precision: 0.62 Mean Recall: 0.21 Mean F1: 0.29 Accuracy 0.26 Logloss: 1.16

TABLE XXV: Run#10 - Detector Only
Actual Class
Lane 1 Lane 2 Lane 3 Total

Predicted Class

Lane 1 1436 292 0 1728
Lane 2 317 2841 326 3484
Lane 3 183 411 1969 2563
Unassigned 0 2 0 2
Support 1936 3546 2295 7777
  • Mean Precision: 0.80 Mean Recall: 0.80 Mean F1: 0.80 Accuracy 0.80 Logloss: 1.08

TABLE XXVI: Run#10 - Model

Figure 8 shows a short area of the A4 highway together with qualitative results of the algorithm, while in  Figures (j)j, (i)i, (h)h, (g)g, (f)f, (e)e, (j)j, (i)i, (h)h, (g)g, (d)d, (c)c, (f)f, (e)e, (b)b, (a)a, (d)d, (c)c, (b)b and (a)a we report the dispersion over the ego-lane detection, considering all the frames of all experiments. Here the line detectors alone appear clearly unable to correctly detect the ego-lane, mostly because of missing detections due to clutter or illumination issues. In the figures this is depicted in black. A further confirmation of this can be found in Figure 8 middle: the detector only results are extremely noisy, resulting in an unreliable ego-lane determination. For instance, the detector is completely missing the final transition from to , leaving the vehicle without almost any ego-lane localization clue. On the other hand, the filtering effect of the HMM is clearly shown in the same Figure top. The proposed model correctly identified the lane transitions even without a complete set of line measurements. Promising results are summarized in Figures 10 and 9. Our model outperformed the basic detector in all tests.

Concerning the confusion matrices Tables XXV, XXVI, XXIII, XXIV, XXI, XXII, XIX, XX, XVII, XVIII, XV, XVI, XIII, XIV, XI, XII, IX, X, VII and VIII, it is worth noting that, for the dataset recorded in Spain, both all the algorithms and configuration settings achieve a better performance w.r.t. Italy. This is most likely related to the better view of the whole road in front of the vehicle, which contains 3 lanes instead of 4. We can also conclude that the localization results achieved by using our line detector are consistently higher, although this comparison might be unfair for the Italy A4 highway dataset, because of the limitations MLD algorithm (maximum 4 lines). However, our experiments prove the effectiveness of our proposal, as depicted in Figures (i)i / (j)j, and (i)i / (j)j.

From the result of the experiments we can observe that:

  • using a line detector that is capable to classify the road line as dashed or continuous results in a better performance, see in Figure 11 an example of how the performance decreases when not using the BV.

  • in difficult situations that hampers the output of line detection and tracking, our approach is able to provide a valuable performance boost.

(a) Run#1 - Detector Only
(b) Run#1 - Model
(c) Run#2 - Detector Only
(d) Run#2 - Model
(e) Run#3 - Detector Only
(f) Run#3 - Model
(g) Run#4 - Detector Only
(h) Run#4 - Model
(i) Run#5 - Detector Only
(j) Run#5 - Model
Fig. 9: Comparison graphs between the localization accuracies using our proposal w.r.t. using both our proposal, in both monocular/stereo and line/clothoid tracking accordingly, and the MLD detector. The charts depict the results obtained on the Italian dataset in the A4 highway (4-lanes highway). A green color coding represents correct vehicle’s lane localization, yellow represent one-lane range mismatches, orange 2-lane range mismatches and red 3-lane range mismatches. The black color represents the inability to assign a lane, due to missing information.
(a) Run#6 - Detector Only
(b) Run#6 - Model
(c) Run#7 - Detector Only
(d) Run#7 - Model
(e) Run#8 - Detector Only
(f) Run#8 - Model
(g) Run#9 - Detector Only
(h) Run#9 - Model
(i) Run#10 - Detector Only
(j) Run#10 - Model
Fig. 10: As in Figure 9 except it refers to the Spanish dataset, taken in the A2 highway (3-lanes highway).
(a) Run#3 (No BV) - Detector Only
(b) Run#3 (No BV) - Model
Fig. 11: Comparison of Run#3 between the localization accuracies without the BV (see Figures (f)f and (e)e).

Conclusions

We presented an ego-lane estimation algorithm aimed at enhancing the accuracy of the vehicle localization at the lane level, in highway-like scenarios. Differently from other works, we proposed a method designed to cooperate with an existing line detector and tracker. With respect to the existing ego-lane estimation literature, our algorithm achieves good localization even when fed with noisy and/or occasionally missing data, i.e., the typical output of a real, and therefore faulty, line detector and tracker. We exploited an HMM-based scheme to take advantage of real road line observations in a probabilistic fashion. The proposed algorithm improves the localization robustness in conditions where lane markings are missing or are hidden by traffic clutter and/or by lightening issues, i.e., realistic conditions. We are currently preparing to collect and analyze the results of the proposed algorithm in urban scenarios.

References