Personalized Federated Learning of Driver Prediction Models for Autonomous Driving

Autonomous vehicles (AVs) must interact with a diverse set of human drivers in heterogeneous geographic areas. Ideally, fleets of AVs should share trajectory data to continually re-train and improve trajectory forecasting models from collective experience using cloud-based distributed learning. At the same time, these robots should ideally avoid uploading raw driver interaction data in order to protect proprietary policies (when sharing insights with other companies) or protect driver privacy from insurance companies. Federated learning (FL) is a popular mechanism to learn models in cloud servers from diverse users without divulging private local data. However, FL is often not robust – it learns sub-optimal models when user data comes from highly heterogeneous distributions, which is a key hallmark of human-robot interactions. In this paper, we present a novel variant of personalized FL to specialize robust robot learning models to diverse user distributions. Our algorithm outperforms standard FL benchmarks by up to 2x in real user studies that we conducted where human-operated vehicles must gracefully merge lanes with simulated AVs in the standard CARLA and CARLO AV simulators.

READ FULL TEXT VIEW PDF
12/31/2020

PFL-MoE: Personalized Federated Learning Based on Mixture of Experts

Federated learning (FL) is an emerging distributed machine learning para...
06/15/2021

Privacy Assessment of Federated Learning using Private Personalized Layers

Federated Learning (FL) is a collaborative scheme to train a learning mo...
02/05/2021

Federated Learning on the Road: Autonomous Controller Design for Connected and Autonomous Vehicles

A new federated learning (FL) framework enabled by large-scale wireless ...
02/04/2021

Semi-Synchronous Federated Learning

There are situations where data relevant to a machine learning problem a...
10/16/2020

Flow-FL: Data-Driven Federated Learning for Spatio-Temporal Predictions in Multi-Robot Systems

In this paper, we show how the Federated Learning (FL) framework enables...
02/19/2021

Personalized Federated Learning: A Unified Framework and Universal Optimization Techniques

We study the optimization aspects of personalized Federated Learning (FL...
01/04/2019

Breaching the privacy of connected vehicles network

Connected Vehicles network is designed to provide a secure and private m...

I Introduction

Future robotic fleets must operate amongst diverse humans with heterogeneous preferences on human-robot interaction, in applications ranging from nursing assistance robots to home robots and autonomous vehicles (AVs). Given such heterogeneity, there is a large incentive to share data from robotic fleet deployments to improve computer vision, prediction, and control modules based on diverse interactions with human users. For example, AVs can learn human trajectory forecasting models to proactively anticipate the behavior of nearby humans to aid decision-making

[schmerling2018multimodal]. Sharing rare and risk-sensitive driver styles during challenging contexts (e.g. traffic disruptions) can potentially help learn more robust forecasting models that generalize to new cities and driver populations. At the same time, raw trajectory data should be kept private to avoid revealing proprietary logic to competitor companies or individual driver behavior to insurance companies. In this paper, we address how to balance the competing objectives of user privacy and utility of data sharing in large-scale robotic fleet learning.

Federated learning (FL) is a promising approach to train machine learning (ML) models from distributed datasets while preserving privacy

[google2017federated, mcmahan2016federated, li2020federated]. FL trains a model locally at a user’s device (e.g. a robot) and simply shares model parameter updates with a central server to learn from diverse users while protecting privacy by not uploading raw training data. Typically, FL performs poorly (e.g., converges to a poor global model) when user data is highly heterogeneous [kairouz2019advances, fallah2020personalized], which is a key problem for AVs that must interact with a variety of diverse human driving styles. The recently-proposed personalized variant of FL [fallah2020personalized, dinh2020personalized] is a promising approach to handle user diversity, but, to the best of our knowledge, has hitherto not been applied in robotics. Moreover, our subsequent results illustrate poor performance for standard FL in robotics applications owing to diverse human-robot interactions. As such, the key contribution of this work is to contribute novel algorithms for personalized FL for human trajectory forecasting models in AV deployments.

Contributions and Organization: The technical contributions and organization of this paper are as follows. To the best of our knowledge, we present the first user study that assesses the efficacy of personalized federated learning in robotics, especially for trajectory forecasting models for AVs. Then, we introduce a simple experiment where a fleet of robots have similar dynamics models, but widely different cost functions, for which standard applications of personalized FL are sub-optimal. Then, to mitigate these problems, we introduce novel algorithmic extensions to personalized FL that effectively learn from global experience for common parameters (e.g., shared dynamics models) while adaptively specializing local cost functions. Finally, we show strong performance gains for our algorithm in a user-study with real human drivers on photo-realistic simulators like CARLA [Dosovitskiy17] and lightweight simulators like CARLO [cao2020reinforcement] for a lane merging scenario requiring challenging human-robot interaction.

(a)
(b)
Fig. 3: We evaluate a lane-merging scenario in CARLA with autonomous and human-driven vehicles to test the efficacy of personalized federated learning of trajectory forecasting models.

Ii Related Work

Motivation for Personalized Federated Learning in Robotics: FL [google2017federated] is a method to train a global ML model from multiple networked devices, such as mobile phones, that each have local labelled datasets. The key benefits of FL are privacy protection and reduced communication overhead since locally-trained model parameters, instead of raw user data, are sent to the cloud for knowledge sharing. FedAVG [mcmahan2016federated] is a simple, widely-used implementation of the FL algorithm that averages model gradients from diverse clients to learn a global model, which is then synced back to individual clients for continual learning. Standard applications of FL, including FedAVG, lose robust performance when individual mobile clients’ (e.g., robots) data is highly heterogenous [kairouz2019advances, li2020federated, bagdasaryan2020backdoor]. Such vulnerability to dataset heterogeneity is a crucial problem for networked robotic fleets such as AVs, since perception and forecasting models can fail to converge when robots observe sensory data from diverse environment distributions and interactions with radically different humans.

Recently, personalized FL [deng2020adaptive, dinh2020personalized, fallah2020personalized] has emerged as an effective method to learn robust models under such heterogeneity by specializing potentially unique models for each client (e.g. robot). It aims to first learn a global model and then efficiently adapt it to individual robots while minimizing the extra training cost of personalization. This technique is inspired by multi-task learning [ruder2017overview, zhang2017survey] and meta-learning [vilalta2002perspective], and is especially important in situations where the cost of personalization is relatively high, such as for low-power robots or mobile phones. Another approach to personalized FL is to weigh the parameters of a global model and unique personalized local model. For example, Deng et al. [deng2020adaptive] propose an algorithm that adaptively changes the relative weighting of parameters of a global model during local training. This approach is generally able to improve the accuracy of personalized models, although it leads to increased computation and communication costs.

Our key observation is that many models in robotics have common internal structures – dynamics models are often shared by similar robots, but cost functions or risk sensitivities can be unique. As such, our approach embraces the fact that robotic models can have a subset of parameters that are relatively invariant to data measured on each robot (e.g., shared dynamics) and should therefore resemble global parameters. On the other hand, other subsets of parameters should be personalized for each client, such as those that model risk-sensitivity and cost functions. Thus, our method flexibly adjusts the learning rate for each class of parameters based on its variance across heterogenous robots, which leads to more accurate and stable learning. As such, we embrace that different sub-sets of parameters represent global and local patterns, which makes our work different from

[deng2020adaptive] which simply produces a weighted average of global and local models.

Adaptive Learning Rates in Optimization:

In optimization, the learning rate is a hyper-parameter that governs how quickly parameters can be updated during one step of gradient descent, such as in the standard Stochastic Gradient Descent (SGD) algorithm. For example, the AdaGrad

[duchi2011adaptive]

algorithm adjusts the learning rate for each parameter based on its cumulative gradient. This algorithm helps to stabilize and speed up the learning process because it can apply a more appropriate learning rate than a uniform one depending on the progress of learning individual parameters. SGD algorithms such as RMSProp

[tieleman2012lecture] and Adam [kingma2014adam], which are widely used in modern ML training, also use this core idea, which can easily be used in personalized FL. In key contrast to these algorithms, our algorithm adjusts the learning rate based on the progress of learning across distributed robots. Specifically, our algorithm calculates the variance of each individual parameter across FL clients (robots) to gauge the progress of learning and set an appropriate learning rate for that parameter. Moreover, our method can be used alongside standard SGD optimizers for local datasets, such as Adam, as we show in our evaluation.

Trajectory Forecasting for AVs: Our work is complementary to a rich body of research on trajectory forecasting models in robotics, which attempt to predict the motion of pedestrians or the future trajectories of human drivers conditioned on behaviors of AVs. For example, Gupta et al. [gupta2018social]

proposes a Generative Adversarial Network (GAN) model that predicts pedestrians trajectories while considering their interactions. Likewise, Ivanovic et al.

[ivanovic2019trajectron, salzmann2020trajectron++]

propose recurrent neural network models that predict distributions over future agent trajectories by learning from past timeseries of agent interactions. Schmerling et al.

[schmerling2018multimodal] incorporate such trajectory forecasting models to construct control policies for AVs that anticipate the behavior of nearby human drivers to smoothly negotiate lane changes. Our work is complementary to such prior research – rather than develop novel forecasting models, we instead develop novel learning techniques to specialize models from heterogeneous agent interactions while protecting data privacy.

Federated Learning in Robotics: We note prior work has applied FL to robotics, but to learn vision models in private scenarios [li2019fc, liu2019lifelong, liu2020federated]. Instead, we address personalized federated learning to adapt to heterogenous human interactions, which is the key novelty of our paper. Moreover, a key novelty of our work is our new algorithm for personalization with differentiates between common parameters, such as for dynamics models, that apply to many robots and those that are robot, human, or scenario-specific.

Iii Problem Statement and Proposed Method

Fig. 4: Personalized Federated Learning (FL) for robotics: Each robot sees a potentially diverse local dataset and must learn a local model with parameters . These local models are occasionally pooled and synced with a cloud server to yield a global model which is further specialized locally at each robot .

We now formalize our problem statement to show how FL is applied to robotic systems. Fig. 4 illustrates an overview of the system we assume in our problem. First, we introduce a fleet of robots, such as autonomous vehicles, consisting of robots denoted by which measure sensory data such as an image or LIDAR point cloud where is discrete time. connects to a cloud environment with a wireless network. Our goal is to learn a model that predicts future system states or controls of other objects, such as pedestrians and vehicles, based on measured data and a parameter to be learned. Additionally, we introduce which is a parameter of the global model. The global model is held in the cloud and trained without sharing the measured data with robots.

The measured dataset is used to learn a machine learning model, such as a trajectory forecasting model, that predicts future values with model parameters . A series of measured from time to , such as a trajectory of a specific object or a segment of video, is denoted by . We note that our problem especially applies when each is very different from others. This is because personalization is not needed if has the same distribution for all .

The learning algorithms of our federated learning system aim to learn the parameters of a prediction model. Therefore, similar to general ML algorithms, we assume an objective function

to be optimized. Typically, this is a loss function of a prediction model, such as mean squared error for future predicted system states. For personalization, the learning algorithms aim to find optimal parameter

which minimizes for each robot . Thus, we formalize our problem as a minimization problem of the sum of the objective functions :

This means learning algorithms for our system should learn that minimizes the expected value of the objective functions by training the prediction model at robot . We note that are the same value for all if the algorithm trains only a single global model, like general FedAVG [mcmahan2017communication] without personalization.

Iii-a Proposed Method

Even in the setting we showed above, general FL algorithms such as FedAVG can potentially perform well. Namely, we can obtain a better set of parameters than randomly initialized without sharing in a cloud environment . However, we hypothesize that we can improve by syncing parameters from a global model as in standard FL, but also specializing the model for robot with the local data . Therefore, we propose a variant of FL algorithm with an additional training step for personalization, which is shown in Alg. 1. There are two major differences from general FL. First, we add a training step executed on each robot for personalization in Alg. 1 line 7. Unlike some prior personalized FL works such as [fallah2020personalized], we assume

can execute many more training steps (e.g. not one step of SGD but a few epochs) because today’s robot systems can do so with modern low-power deep learning accelerators. Our main innovation is to carefully limit parameter updates to avoid overfitting by personalized training. This is inspired by standard methods in fine-tuning DNNs using transfer learning, which freeze some layers’ parameters. Also, retraining all the parameters on a local dataset, which can be of limited size and highly biased, often causes overfitting. We found that it is effective to slow down the update of parameters that have common features across many robots’ data during personalized training. To do so, our key insight is to estimate the variance of different parameters

across robots during FL, which is codified in Algorithm lines 7, 13, and 14.

Specifically, we now discuss how to properly adjust the extent of parameter updates. In short, we propose to apply an adaptive learning rate to FL for robotics systems. Generally, it is hard to accurately determine which parameters learn common dynamics and shared trends across robots, especially when training a model such as a DNN with a huge number of parameters. However, we can expect such common parameters tend to eventually converge to a similar value on any dataset. Therefore, we propose a method to estimate how much parameters have in common from the variation of parameters learned by each robot in a fleet. The estimated degree of similarity finally affects the learning rate of each parameter (lines 13-14). As a result, higher learning rates are assigned to parameters with higher variation in personalized training, which might indicate they need specialization

per robot. Specifically, we introduce a variation vector

whose elements represent the degree of variability of each parameter. Then, the learning rate of each parameter is determined based on a ratio of the variability. Namely, the learning rate of the -th parameter of parameter vector is denoted by , where is a constant representing the maximum learning rate. In practical systems, we can cluster parameters (e.g. parameters of a layer of a DNN) and calculate learning rates for each cluster for computational scalability.

1 Randomly initialize parameter Initialize Initialize repeat
2        for  to  do
               Update with and learning rate , Update with and learning rate Send to cloud environment Result: Return
3              
4        end for
5       Update by aggregating Update : Update :
until  converges
Algorithm 1 Federated Control Learning: is a scalar constant representing the basic learning rate, and is the id of an individual parameter in . Therefore, represents the -th parameter in (the number of individual parameters is ). The key process is line 7 where a model is trained with the adaptive learning rate . , which is used to update , which represents a vector of the deviation sum of squares for each . We assume every robot (client) participates in every training round.

Iv Experimental Results

Fig. 5: The motion of a mass point in 1-dimension space: A mass point (circle in the fig) has a state consisting of velocity () and position (), and can receive control input (). (left) The goal of this task is to make position and velocity zero from the init state which is randomly given. (middle and right)

We now evaluate our proposed method (Algorithm 1) on three diverse tasks. We start with a simple toy example to benchmark the proposed method. The second and third are more practical tasks in autonomous driving where a robot vehicle changes a lane while anticipating the behavior of other human-driven vehicles.

Iv-a Illustrative Toy Example: LQR control of a 1-dimensional point mass “robot”

In this section, we introduce a very simple task where a point mass must be controlled by a linear-quadratic regulator (LQR) controller in a 1-dimensional space, as illustrated in Fig. 5. The goal is to learn parameters of an LQR controller that moves the point mass from an arbitrary initial state to the origin. Importantly, we have robots with identical dynamics, but different LQR cost functions. Our goal is to estimate both the dynamics model and LQR control policy parameters using local datasets for robot and personalized federated learning to share knowledge amongst robots . We now introduce the system dynamics and LQR controller for each robot .

Each robot ’s system state at discrete time is denoted by , which is a vector of its velocity and position . The dynamics of this simple linear system are given by , where is the dynamics matrix, is the control matrix, and is the control input. Assuming a unit-mass system, the dynamics of our linear system are given by:

Next, we define a cost function to identify the optimal control input. The standard LQR cost is defined as , where and are weight matrices ( ). It is well-known that the optimal control input , where is a feedback matrix that arises from the solution to the discrete time Riccati Equation.

In our toy experiment, we assume each robot knows the parametric form of the linear dynamics equations and linear feedback policy, but does not know the specific parameters , , and . Instead, each robot must learn these parameters from measured rollouts (e.g., trajectory data) of an expert controlling the system. We note that it is sufficient to learn dynamics matrices and to predict the next system state from the current state and applied control, as well as the LQR feedback matrix to general controls from the current system state.

Crucially, since each robot has the same dynamics, our experiments should show that and quickly converge to the same global value using knowledge transfer. However, since the cost functions differ, we should see each robot converge to its unique LQR feedback matrix, illustrating the benefits of personalization.

Heterogenous Robot Datasets: We generated different synthetic datasets (one per robot) containing rollouts of the LQR system using common dynamics but different cost functions. For simplicity, we fixed the weight matrix

as an identity matrix but varied

for each robot . The first robot has the reference controller whose . The second and third are 50 and 100, respectively, which indicate they prefer larger magnitude actuation. During simulation, we added Gaussian noise to the system state and control inputs with a small variance of . Using the optimal controllers for each robot’s cost function, we collected expert trajectories of a point mass for 40 randomly initialized states and a simulation time horizon of . Therefore, the number of total states collected was , with of the data held-out as testing data.

Benchmark Algorithms: To compare to our proposed method (Algorithm 1), we evaluated five algorithms shown in Table I. These benchmarks rigorously cover a spectrum of algorithms used in practice today, ranging from purely local training without data sharing (“Local”), simply pooling all data in the cloud without privacy considerations (“Cloud”), standard FL with FedAvg (“SFL”), standard personalized FL (“SPFL”), and finally our proposed method with adaptive, parameter-wise learning rates for personalized FL (“APFL”). Of course, each robot only has access to its own training dataset locally.

Name Description
Local All robots train their models only on data they measure themselves.
Cloud Training is processed on the cloud environment using all raw data, without privacy guarantees.
Standard FL (SFL) Standard FL, which is the same as Alg. 1 except there is no personalization (line 7).
Standard Personalized FL (SPFL) Standard personalized FL, which is Alg. 1 without applying our contribution of a parameter-wise learning rate.
Adaptive Personalized FL (APFL, Ours) Our proposed method in Algorithm 1.
TABLE I: Description of training algorithms

Evaluation Metrics: The overall loss function is the error in predicting a trajectory rollout using learned parameters , , and instead of the true parameters , , and . This is a sum of the mean squared error (MSE) of the control loss and state loss. Therefore, the loss function is denoted by where and are prediction vectors of the system state and control input computed by and , respectively. We implemented the training algorithms in Table 1 and evaluated each robot on test data drawn from its local data distribution. Then, we report the loss averaged across each robot, which corresponds to the objective function in Sec. III. We used the ADAM optimizer [kingma2014adam], and both the default learning rate and initial uniform learning rate in Algorithm 1 are set to .

Algorithm Local Cloud SFL SPFL APFL
State Loss 0.01101 0.01012 0.01023 0.01023 0.01018
Control Loss 0.01251 0.19666 0.20123 0.01095 0.01096
Total Loss 0.02352 0.20678 0.21146 0.02118 0.02114
TABLE II: Prediction loss for each algorithm: We present the total loss and constituent terms of state loss and control loss for each scheme, averaged over 10 test trials. Clearly, our method of APFL has the lowest total loss (our optimization objective) since it learns the global dynamics but learns unique cost functions per robot, which is crucial to achieve low control loss. We learn the dynamics and control with a small, non-zero loss since our simulations had Gaussian dynamics and actuation noise.
Fig. 6: Divergence of learned parameters from true parameters for Toy LQR Example: We quantify the divergence as the Euclidean distance between learned and true parameters for LQR in terms of , for dynamics (x-axis) and the feedback matrix (y-axis). Clearly, our method of APFL (purple) efficiently learns global insights for the shared dynamics to achieve low error for dynamics parameters. Moreover, it also effectively specializes the model to learn local, heterogeneous cost functions and achieve low error for control parameters.

Results: Table II shows the prediction losses for each training method. We show the prediction loss for next system states ( predicted by and ) and control loss ( predicted by ) separately to investigate how learning affects shared and robot-specific parameters. Most methods are able to learn the common state dynamics that depend on the global and . However, purely local training achieves the worst loss since each robot has a smaller amount of local training data. Our key result is that for control loss, the methods with personalization (Local, SPFL, and our APFL) outperform others that train a global model by mixing robots’ heterogenous data. Crucially, our method of APFL has the lowest total loss.

Fig. 6 provides further evidence for the benefits of our APFL method by showing how close the learned parameters , , and differ from the ground-truth parameters. Clearly, our method of APFL (purple) shows strong benefits of not only personalization, but also our scheme of adaptively adjusting the learning rate of parameters based on their variation across robots. Specifically, we achieve both low errors for dynamics and control matrices (x-axis) and the estimated LQR feedback matrix (y-axis). Next we evaluate the benefits of APFL on two challenging AV tasks featuring rich human-robot interaction in our user studies.

Iv-B Lane swapping in the CARLO driving simulator

The second case study is shown on the left side of Fig. 7, where a human-driven vehicle and AV must safely interchange lanes in a short distance, inspired by [schmerling2018multimodal]. The AV is equipped with a trajectory forecasting model that predicts the future motion of the human-driven car, conditioned on past interaction history and a candidate robot future control decision. A key feature of our work is we performed a study with 7 real human drivers, who exhibited a mixture of aggressive and cautious driving styles when deciding whether to overtake or yield to the AV. To quickly collect data with diverse human volunteers, we used the CARLO 2D driving simulator [cao2020reinforcement], which is a light-weight version of CARLA without photo-realistic rendered scenes. Crucially, our FL framework ensures that the human-robot interaction datasets, which often show risky driving styles from human subjects, are kept private and never shared with a central server.

Fig. 7: Traffic scenarios: The left and right figures show scenarios tested in Sec. IV-B and Sec. IV-C, respectively. The red car (upper lane) is a human-driven car, and the blue car (bottom lane) is an AV. The AV uses a forecasting model, learned using our personalized FL method, to anticipate the future trajectory of the human-driven car (for heterogenous driver styles) to efficiently and safely change lanes. In the right scenario, the gray car (lower right) is always going straight but at variable distances from the blue AV, requiring the AV to deliberate where it has a window to swap lanes.

System Definition: We define the joint robot and human (i.e., system) state as where and are the 2D position vectors of the robot and human car at discrete time , and and are the velocity vectors. The 2D vectors are composed of and components. For example, a robot’s velocity is denoted by where and are scalar values representing a robot’s velocity in the lateral and longitudinal directions. The control space of each car involves throttle and steering which are finite sets. For example, the throttle of a robot is denoted by . Similarly, the steering of a human is denoted by . Thus, the control of the cars at time is denoted by .

Prediction Models for Federated Learning: The robot controller has to make predictions of future system states to choose appropriate controls. Our scenario is inspired by [schmerling2018multimodal], and makes the same practical assumption that we know the AV’s internal dynamics and can fully observe robot and human states. Crucially, all we need is to predict the human’s future control inputs and trajectory conditioned on candidate future robot controls. Then, given such a prediction model, we can embed it into a Model Predictive Controller (MPC) [MPC] for the AV to choose low cost robot actions that anticipate the response of humans. This means that the prediction model needs to estimate a future series

of human control inputs at a single time-step. To achieve this, we introduce a Conditional Variational Autoencoder (CVAE) illustrated in Fig.

8 with recurrent subcomponents [schmerling2018multimodal] which are able to predict time-series data. Our MPC control inputs are decided periodically at a discrete interval .

Robot Controller: We define the robot controller as a two-phase controller. In the first phase, the controller is responsible for negotiating the longitudinal distance between human vehicles with different driving styles. This is the most important part in our experiment because it requires accurate, personalized predictions of human driver behavior. For the first phase, we introduce a cost function to evaluate candidate control inputs at time as follows:

The first term represents longitudinal cost based on the fact that cars need sufficient longitudinal distance during a lane change. The second term represents a distance cost between cars to avoid collisions. and are parameters that weight each objective. We chose and to heavily emphasize safety in this experiment.

When the robot controller decides a series of control inputs at time , it computes the sum of the expected cost function in the time horizon for all possible robot control candidates. The controller chooses a series of control inputs that minimizes the sum of the expected cost, which naturally depends on anticipated future human controls. We set the control interval and planning horizon to and respectively, which performed well with an acceptable compute budget.

Once there is enough longitudinal distance to swap lanes, the controller switches to the second phase, and just works as a lane change controller. Since we observed humans’ driving styles almost do not differ during the lane change (and largely govern when to safely initiate it), the lane change controller is a static LQR controller. This static LQR controller drives the robot to a target lateral position representing the x-coordinate of a target lane with longitudinal velocity . In the following quantitative evaluation, we incorporate human trajectory prediction models trained with the benchmarks in Table 1 into the MPC-based AV controller. Then, we compare these controllers in terms of how safe and efficiently the cars are able to swap lanes given the different human’s driving styles.

Fig. 8: Architecture of prediction model: We learn a variational autoencoder for trajectory forecasting using LSTM cells for the encoder and decoder [schmerling2018multimodal] using our personalized FL method. The inputs are 1) a history of past system states and controls, 2) a candidate robot future control and 3) an optional indicator of context (such as location). The encoder maps 1) and 2) to latent variables and . The decoder generates a prediction of the human-driver’s trajectory from the latent variables, 2) and 3).

Dataset: Our dataset consists of system trajectories (including robot and human states, controls, and costs) from time to the time when both cars finished swapping lanes. The dataset consists of a training set, a held-out evaluation set, and a third held-out test set. In order to add diversity to each session’s data, we randomly created 50 initial states .

For the training set, seven humans play the lane swapping scenario running on the CARLO simulator with 50 initial states above. Since the prediction model with CVAE shown in Fig. 8 is not trained at this time, the robot controller uses a very simple model based on only the current state. Specifically, the predicted human control for future time is always the current human control for any .

Once we gather initial data for how humans respond to a baseline AV without a sophisticated model of human behavior, we can train a better driven prediction model. This prediction model is in turn used in the AV’s MPC and resulting rollouts against human drivers allows us to generate an evaluation set. Notably, this evaluation set is used to fine-tune and re-train the human trajectory forecasting model.

Similar to the experiment IV-A, each of the 7 robot cars observes only one type of data corresponding to one human driver. During training, both the default learning rate and are 0.001, and the loss function is the evidence lower bound objective (ELBO) [kingma2013auto, rezende2014stochastic]. The number of epochs is 30 for each training step in Alg. 1. We used 40 sessions for training and 10 challenging held-out sessions for testing. We trained the model with four algorithms shown in Tab. I. Similar to the experiment in Sec. IV-A, the federated learning algorithms use FedAVG to aggregate trained parameters. In our method (APFL), we used layer-wise learning rate instead of parameter-wise to improve compute efficiency, which essentially groups parameters in one layer.

Fig. 9: Learning curve of FL algorithms in the lane swapping scenario: Clearly, our method of APFL (purple) achieves a lower test loss that is personalized to heterogenous drivers in our user study for a lane merging scenario in CARLO.

Evaluation Results: We evaluate the test losses of the prediction models trained with our benchmark algorithms. Fig. 9 illustrates the held-out test dataset loss for model’s successively trained using our benchmark algorithms. In contrast to standard FL, algorithms involving personalization adapt better to heterogenous human driver interactions to achieve a lower loss. Crucially, our method of APFL achieves the lowest loss, which is consistent with the results of the experiment in Sec. IV-A.

Additionally, evaluation results show an AV (with our learned prediction models in-the-loop) can safely and efficiently swap lanes with actual human drivers in test episodes in the CARLO simulator. Fig. 10 shows distributions of key quantitative metrics over the test episodes, such as the elapsed time until the AV starts a lane change, distance between cars when lane changes commence, and the mean cost for each session. Clearly, an AV using our APFL model can start to change lanes quickly with a larger safe distance to the human car.

Fig. 10: Performance comparison of lane swapping scenario: The non-proactive scheme (blue) means a simple benchmark case where the robot controller uses a naïve constant-velocity prediction model. APFL shows statistically-significant benefits from the other training algorithms in mean cost. Indeed, the maximum p-value for the Wilcoxon Signed-Rank test was 0.002476 when we compare the closest competitor of SPFL with APFL. Moreover, our method of APFL maintains a larger safe distance between cars.

Iv-C Lane change with the CARLA AV simulator

We now consider a more sophisticated AV driving scenario in the photo-realistic, standard CARLA [Dosovitskiy17] simulator. As shown on the right of Fig. 7, our scenario is similar to the lane-swap scenario of the previous section. However, there is additional complexity since there is a gray human-driven car that starts at random distances from the AV with random velocities, which requires the AV to reason about whether there is enough safety margin to overtake the red car.

Experimental Setup: The settings of this experiment closely follow those of the lane swapping case in IV-B. The control space is the same as before, and the state space is the same except that we add the state of the gray car. The architecture of the prediction model is exactly the same as the one in IV-B, shown in Fig. 8. Further, the robot controller is also the same except it does not switch to the second phase where the robot is going to change a lane because robot cars do not change lanes in the scenario. Additionally, we set a lower limit on the cost function to prevent the robot cars from unnecessarily increasing the distance to the human car after both cars have a safe and sufficient distance, which will artificially decrease the cost. Therefore, the new cost function is denoted by .

Unlike the lane swapping case, we use a programmed synthetic controller for human cars to test on uniformly diverse driving styles, including risky behaviors. This synthetic human controller is also a two-phase controller, but control inputs are decided according to a target velocity in the first phase. The target velocity is chosen as high or low , where the high velocity is only chosen if the human car has a safe relative difference from the other cars, modulated by its risk tolerance parameter :

In the above equation, is a parameter representing the degree of preference for overtaking robot cars, which we henceforth called the risk-tolerance parameter. Further, in the above equation, indicates the relative gap between cars based on the positions and velocities denoted by , and and are a longitudinal position of the gray car and a safety distance constant, respectively.

The target velocity for the human-driven vehicle is set depending on the relative velocity to the robot car driving alongside it. Once the target velocity is determined, the synthetic human-driver controller computes control inputs with a PID controller in CARLA. We tested 5 kinds of driving styles whose risk-sensitivity () parameters were in the range . These driving styles include a mixture of cautious and aggressive styles to ensure a diverse dataset. For instance, a human car with sometimes tries to pass the robot car even when the current relative position to the robot car is minimal.

We collected a diverse dataset in the same manner as in the previous scenario. Specifically, we generated 54 initial states where the relative position between cars and velocities are uniformly distributed. We trained trajectory forecasting models according to Table

I with the same hyper-parameters and LSTM encoder-decoder model as the previous scenario. Finally, we integrated the trained trajectory forecasting models into the AV’s controller and evaluated it in 9 challenging scenarios where the robot cars need to change their behavior depending on humans’ driving styles.

Algorithm Cloud SFL SPFL APFL(Ours)
Loss 0.01643 0.01760 0.01600 0.01160
TABLE III: Prediction loss for each training scheme: All losses are on the test data after training has converged.

Evaluation Results: Using the same metrics as the lane swapping case in Sec. IV-B, we evaluate the test loss of the prediction model and the controllers’ performance in terms of how safe and efficiently the cars are able to change a lane. The prediction losses on test data for each model, which are shown in Tab. III, have the same trend as in the previous scenario of lane swapping, clearly illustrating the benefits of our APFL method. Moreover, Fig. 11 illustrates APFL provides the lowest cost while maintaining a safe distance from other cars.

Fig. 11: Performance comparison of CARLA lane change scenario: Similar to the result of the lane swapping case in Sec. IV-B, APFL shows significant differences from the other training algorithms in mean cost (the max p-value for the wilcoxon signed-rank test was 0.021346 for the comparison with SPFL) while maintaining a safe distance between cars when lane changes commence.

V Discussion and Conclusions

This paper presents a novel personalized federated learning framework for deployments of robots that measure diverse sensory streams in varied environmental contexts. Our first contribution is to show the drawbacks, and potential, of standard personalized FL through a case study where robotic models share a common structure for dynamics but heterogeneous cost functions. Then, our second contribution is to propose a novel algorithm which mitigates these drawbacks and effectively leverages both local and global knowledge to improve robotic control. Specifically, we demonstrate strong experimental performance of our algorithm in state-of-the-art driving simulators featuring real human driver data.

In future work, we plan to test our algorithm on large-scale deployments of AVs using a combination of public trajectory datasets from Waymo, Aptiv, Lyft, etc. Moreover, we plan to provide theoretical convergence guarantees for our personalized FL algorithm in distributed convex optimization settings that arise in multi-agent systems and robotics. Overall, our work is a timely first step to address how to learn privacy-preserving, specialized deep learning models for robot fleets that will increasingly interact with diverse humans in diverse operating scenarios.

References