1 Introduction
Our longterm aim is to develop an AI system that can learn about a person’s intent and goals by continuously observing their behavior. In this work, we progress towards this aim by proposing an online Inverse Reinforcement Learning (IRL) technique to learn a decisiontheoretic human activity model from video captured by a wearable camera. The use of a wearable camera is critical to our task, as human activities must be observed up close and across large environments. Imagine a person’s daily activities—perhaps they are at home today, moving about, completing tasks. Perhaps they are a scientist that conducts a long series of experiments across various stations in a laboratory, or they work in an office building where they walk about their floor, get coffee, . As people tend to be very mobile, a wearable camera is ideal for observing a person’s behavior.
Since our task is to continuously learn human behavior models (., a policy) from observed behavior captured with a wearable camera, our task is best described as an online IRL problem. The problem is an inverse Reinforcment Learning problem because the underlying reward or cost function of the person is unknown. We must infer it along with the policy from the demonstrated behaviors. Our task is also an online learning problem, because our algorithm must continuously learn as a part of a lifelong process. From this perspective, we must develop an online learning approach that learns effectively over time.
We present an algorithm that incrementally learns spatial and semantic intentions (where you will go and what you will do) of a firstperson camera wearer. By tracking the goals a person achieves, the algorithm builds a set of possible futures. At any time, the user’s future is predicted among this set of goals. We term our algorithm “Discovering Agent Rewards for Kfutures Online” (Darko), as it learns to associate rewards with semantic states and actions from demonstrations to predict among possible goals.
Contributions:
To the best of our knowledge, we present the first application of ideas from online learning theory and inverse reinforcement learning to the task of continuously learning human behavior models with a wearable camera. Our proposed algorithm is distinct from traditional IRL problems as we jointly discover states, transitions, goals, and the reward function of the underlying Markov Decision Process model. Our proposed human behavior model also goes beyond firstperson trajectory forecasting and allows for the prediction of future human activities that can happen outside the immediate field of view and far into the future.
2 Related Work
Firstperson vision (FPV): Wearable cameras have been used for various human behavior understanding tasks [4, 10, 12, 16, 22] because they give direct access to detailed visual information about a person’s actions. Leveraging this feature of FPV, recent work has shown that it is possible to predict where people will look during actions [12] and how people will use the environment [20].
DecisionTheoretic Modeling: Given agent demonstrations, the task of inverse reinforcement learning (IRL) is to recover a reward function of an underlying Markov Decision Process (MDP) [1]. IRL has been used to model taxi driver behavior [34] and pedestrian behavior [35, 7]. In contrast to previous work, we go beyond physical trajectory forecasting by reasoning over future object interactions and predicting future goals in terms of scene types.
Online Learning Theory: The theory of learning to making optimal predictions from streaming data is well studied [23]
but is rarely used in computer vision, compared to the more prevalent application of supervised learning theory. We believe, however, that the utility of online learning theory is likely to increase as the amount of available data for processing is ever increasing. While the concept of online learning has been applied to inverse reinforcement learning
[18], the work was primarily theoretic in nature and has found limited application.Trajectory forecasting: Physical trajectory forecasting has received much attention from the vision community. Multiple human trajectory forecasting from a surveillance camera was investigated by [14]. Other trajectory forecasting approaches use demonstrations observed from a bird’seye view; [31] infers latent goal locations and [2] employ LSTMs to jointly reason about trajectories of multiple humans. In [25], the model forecasted shortterm future trajectories of a firstperson camera wearer by retrieving the nearest neighbors from a dataset of firstperson trajectories under an obstacleavoidance cost function, with each trajectory representing predictions of where the user will move in view of the frame; in [26], a similar model with learned cost function is extended to multiple users.
Predicting Future Behavior: In [6, 21], the tasks are to recognize an unfinished event or activity. In [6], the model predicts the onset for a single facial action primitive, the completion of a smile, which may take less than a second. Similarly, [21] predicts the completion of short human to human interactions. In [9], a hierarchical structured SVM is employed to forecast actions about a second in the future, and [28] demonstrates a semisupervised approach for forecasting human actions a second into the future. Other works predict actions several seconds into the future [3, 8, 11, 29]. In contrast, we focus on highlevel transitions over a sequence of future actions that may occur outside the frame of view, and take a longer time to complete (in our dataset, the mean time to completion is seconds).
3 Online IRL with DARKO
Our goal is to forecast the future behaviors of a person from a continuous stream of video captured by a wearable camera. Given a continuous stream of FPV video, our approach extracts a sequence of state variables
using a portfolio of visual sensing algorithms (., SLAM, stop detection, scene classification, action and object recognition). In an online fashion, we segment this state sequence into episodes (short trajectories) by discovering terminal goal states (., when a person stops). Using the most recent episode, we adaptively solve the inverse reinforcement learning problem using online updates. Solving the IRL problem in an online fashion means that we incrementally learn the underlying decision process model.
3.1 FirstPerson Behavior Model
A Markov Decision Process (MDP) is commonly used to model the sequential decision process of a rational agent. In our case, we use it to describe the activity of a person with a wearable camera. In a typical reinforcement learning problem, all elements of the MDP are assumed to be known and the task is to estimate an optimal policy
, that maps a state to an action , by observing rewards. In our novel online inverse formulation, every element of the MDP, including the policy, is unknown and must be inferred as new video data arrives. Formally, an MDP is defined as:States: is the state space: the set of states an agent can visit. In our online formulation, is initially empty, and must be expanded as new states are discovered. We define a state
as a vector that includes the location of the person (3D position), the last place the person stopped (a previous goal state), and information about any object that the person might be holding. Formally, a state
is denoted as:The triplet is a discrete 3D position. To obtain the position, we use a monocular visual Slam algorithm [15] to localize the agent in a continuously built map.
The vector encodes any objects that the person is currently holding. We include this information in the state vector because the objects a user acquires are strongly correlated to the intended activity [4]. if the user has object in their possession and zero otherwise. is a set of predefined objects available to the user. is a set of predefined scene types available to the user, which can be larger than the true number of scene types. The vector encodes the last scene type the person stopped. Example scene types are kitchen and office. if the user last arrived at scene type and is zero otherwise.
Goals: We also define a special type of state called a goal state , to denote states where the person has achieved a goal. We assume that when a person stops, their location in the environment is a goal. We detect goal states by using a velocitybased stop detector. Whenever a goal state is encountered, the sequence of states since the last goal state to the current goal state is considered a completed episode . The set of goals states expands with each detection. We explain later how is used to perform goal forecasting.
Actions: is the set of actions. can be decomposed into two parts: . The act of moving from one location in the environment to another location is denoted as . Like , must be built incrementally. The set is the set of possible acquire and release actions of each object: . The act of releasing or picking up an object is denoted as . Each action
must be detected. We do so with an imagebased firstperson action classifier. More complex approaches could improve performance
[13].Transition Function: The transition function represents how actions move a person from one state to the next state. is constructed incrementally as new states are observed and new actions are performed. In our work, is built by keeping a table of observed triplets, which describes the connectivity graph over the state space. More advanced methods could also be used to infer more complex transition dynamics [27, 30].
Reward Function: is an instantaneous reward function of action at state . We model as the inner product between a vector of features and a vector of weights . The reward function is essential in valuebased reinforcement learning methods (in contrast to policy search methods) as it is used to compute the policy . In the maximum entropy setting, the policy is given by , where the value functions and are computed from the reward function by solving the Bellman equations [34]. In our context, we learn the reward function online.
Intuitively, we would like the features of the reward function to incorporate information such as the position in an environment or objects in possession, since it is reasonable to believe that the goal of many activities is to reach a certain room or to retrieve a certain object. To this end, we define the features of the reward to mirror the information already contained in the state : the position, previous scene type, and objects held. To be concrete, the feature vector is the concatenation of the 3d position coordinates , a dimensional indicator vector over previous goal state type and a dimensional indicator vector over held objects. We also concatenate a dimensional indicator vector over actions .
3.2 The DARKO Algorithm
We now describe our proposed algorithm for incrementally learning all MDP parameters, most importantly the reward function, given a continuous stream of firstperson video (see Darkoin Algorithm 1). The procedure begins by initializing , reward parameters , empty state space , goal space , transition function , and current episode .
State Space Update: Image frames are obtained from a firstperson camera (the NewFrame function), and Slam is used to track the user’s location (lines 4 and 5). An imagebased action detection algorithm, ActDet, detects handobject interactions and decides movements as a function of current and previous position. While we provide an effective method for ActDet, our focus is to integrate (rather than optimize) its outputs. Lines 7 and 8 show how the trajectory is updated and MDP parameters of state space and transition function are expanded. Line 9 represents a collection of generalized forecasting tasks (see Section 4.4), such as the computation of future goal posterior and trajectory forecasting.
Goal Detection: In order to discover goals, we use a stopdetection algorithm GoalDet, by using the camera velocity computed from SLAM (Line 10). If a goal state has been detected, that terminal state is added to the set of goal states . The detection of a terminal state also marks the end of an episode . The previous goal state type is also updated for the next episode. Again, while we provide an effective method for GoalDet, our focus is to integrate (rather than optimize) its outputs.
Online IRL: With the termination of each episode , the reward function and corresponding policy are updated via the reward parameters (Line 13). The parameter update uses a sequence of demonstrated behavior via the episode , and the current parameters of the MDP. More specifically, OnlineIRL (Algorithm 2) performs online gradient descent on the likelihood under the maximum entropy distribution by updating current parameters of the reward function. The gradient of the loss can be shown to be the difference between expected feature counts and empirical feature counts of the current episode. Computing the gradient requires solving the soft value iteration algorithm of [33]. We include a projection step to ensure .
To the best of our knowledge, this is the first work to propose an online algorithm for maximum entropy IRL in the streaming data setting. Following the standard procedure for ensuring good performance of an online algorithm, we analyze our algorithm in terms of the regret bound. The regret of any online algorithm is defined as:
The regret is the cumulative difference between the performance of the online model using current parameter versus the best hindsight model using the best parameters . The loss is a function of the ’th demonstrated trajectory, and measures how well the model explains the trajectory.
In our setup, the loss function is defined as
. It can be shown^{1}^{1}1Proof of the regret bound is given in the Appendix that the regret of our online algorithm is bounded as follows:(1) 
where is a norm bound on , is feature dimensionality, and is the episode. The average regret approaches zero as grows since the bound is sublinear in . Verification of this noregret property is shown in the experiments.
4 Generalized Activity Forecasting
Without Line 9, Algorithm 1 only describes our online IRL process to infer the reward function. In order to make incremental predictions about the person’s future behaviors online, we can leverage the current MDP and reward function. An important function which lays the basis for predicting future behaviors is the state visitation function, denoted . We now show how can be modified to perform generalized queries about future behavior.
4.1 State Visitation Function
Using the current estimate of the MDP and the reward function, we can compute the policy of the agent. Using the policy, we can forward simulate a distribution of all possible futures. This distribution is called the state visitation distribution [33]. More formally, the posterior expected count of future visitation to a state can be defined as
(2) 
This quantity represents the agent’s expectation to visit each state in the future given the partial trajectory. indicates a partial trajectory starting at time and ending at time . The expectation is taken under the maximum causal entropy distribution,
, which gives the probability of a future trajectory given the current trajectory.
is the indicator function, which counts agent visits to . Equation 2 is estimated by sampling trajectories from .4.2 Activity Forecasting with State Subsets
In this work, we extend the idea of state visitations to a single state to a more general subset of states . While a generalized prediction task was not particularly meaningful in the context of trajectory prediction [7, 34], predictions over a subset of states now represents semantically meaningful concepts in our proposed MDP. By using the state space representation of our firstperson behavior model, we can construct subsets of the state space that have interesting semantic meaning, such as “having an object ” or “all states closest to goal with set of objects.”
Formally, we define the expected count of visitation to a subset of states satisfying some property :
(3)  
(4) 
Equation 4 is essentially marginalizing over the state subspace of Equation 2. Derivation of two other inference tasks is given in the Appendix: (1) expected visitation prediction of performing an action after arriving at a subspace, (2) expected joint actionstate subspace visitation prediction.
4.3 Forecasting Trajectory Length
In the following, we present a method to predict the length of the future trajectory. Formally, we can denote the expected trajectory length:
(5) 
Consider evaluating from Equation 4 by setting , that is, by considering the expected future visitation count to the entire state space. Then,
(6) 
where indicates the number of states in trajectory .
4.4 Future Goal Prediction
As previously described, we wish to predict the final goal of a person’s action sequence. For example, if I went to the study to pick up a cup, how likely am I to go to the kitchen versus the living room? This problem can be posed as solving for the MAP estimate of , the posterior over goals. It describes what goal the user seeks given their current trajectory, defined as:
(7) 
where is the value of with respect to a partial trajectory that ends in . Notice that the likelihood term is exponentially proportional to the value difference between the start state and the current state . In this way, the likelihood encodes the progress made towards a goal in terms of the value function.
5 Experiments
We first present the dataset we collected. Then, we discuss our methods for goal discovery and action recognition. To reiterate, our focus is not to engineer these methods, but instead to make intelligent use of their outputs in Darkofor the purpose of behavior modeling. We compare Darko’s performance versus several baselines on the task of goal forecasting, and show Darko’s performance is superior. Then, we analyze Darko’s performance under less noisy conditions, to illustrate how it improves when provided with more robust goal discovery and action detection algorithms. Next, we illustrate Darko’s empirical noregret performance, which further shows it is an efficient online learning algorithm. Finally, we present trajectory length forecasting results, and find that our length forecasts exhibit low median error. Additional analyses including feature ablation and incorporating uncertainty from goal discovery are presented in the Appendix.
5.1 FirstPerson Continuous Activity Dataset
We collected a dataset of sequential goalseeking behavior in several different environments such as home, office and laboratory. The users recorded a series of activities that naturally occur in each scenario. Each user helped design the script they followed, which involved their prior assumptions about what objects they will use and what goal they will seek. An example direction from a script is “obtain a snack and plate in kitchen, eat at dining room table.”
Users wore a hatmounted GoPro Hero camera with vertical, horizontal FOVs. Our dataset is comprised of 5 user environments, and includes over 250 actions with 19 objects, 17 different scene types, at least 6 activity goals per environment, and about 200 highlevel activities (trajectories). In each environment, the user recorded 3–4 long sequences of highlevel activities, where each sequence represents a full day of behavior. Our dataset represents over 15 days of recording.
For evaluation, all ground truth labels of objects (cup, backpack), actions (acquire, release) and goals (kitchen, bedroom) were first manually annotated. A goal label correspond to when a highlevel direction was completed, and in which scene it was completed, (dining room, time=). An action label indicates when an activity was performed, (acquire, cup, time=). Further details are presented in the Appendix.
5.2 Goal Discovery and Action Recognition
We describe two goal discovery methods and an action recognition method that can serve as input to Darko. With respect to Algorithm 1, these are GoalDet and ActDet.
Scenebased Goal Discovery: This model assumes that if a scene classifier is very confident in the scene type for several images frames, the camera wearer must be in a meaningful place in the environment (., kitchen, bedroom, office). We use the output of a scene classifier from [32] (GoogLeNet model) on every frame from the wearable camera. If the mean scene classifier probability for a scene type is above a threshold for 20 consecutive image frames, then we add the current state to the set of goals .
Stopbased Goal Discovery: This model assumes that when a person stops, they are at an important location. Using Slam’s 3D camera positions, we apply a threshold on velocity to detect stops in motion. When a stop is detected, we add the current state to the set of goals . In Table 1, temporal accuracies are computed by counting detections within second windows of ground truth labels as true positives; for the scenebased method, a true positive also requires the scene type to match the ground truth scene type. Stopbased discovery is reliable across all environments, thus, we use it as our primary goal discovery method.
Method  Home 1  Home 2  Office 1  Office 2  Lab 1 

Scene Discovery  
Stop Discovery  
Act. Recognition 
Imagebased Object Recognition: We designed an object recognition approach that classifies the object the user interacts with at every temporallylabeled window. It overwrites the groundtruth object label with its detection. The approach first detects regions of person in each frame with [19]
to focus on objects near the visible hands, which are cropped with context and fed into an imageclassifier trained on ImageNet
[24]. The outputs are remapped to our object set, and a final classification is produced by taking the maximum across objects. The peraction classification accuracies in Table 1 demonstrate the method can produce reasonable action classifications across all scenes. While imperfect, these detections serve as useful input to Darko.5.3 Goal Forecasting Performance
At every time step, our method predicts the user’s goal or final destination (., bedroom, exit) as described in Section 4.4 and shown in Figure 3. To understand the goal prediction reliability, we compare our approach to several baseline methods for estimating the goal posterior , where is a goal and is the observed state sequence up to the current time step. Each baseline requires the state tracking and goal discovery components of Darko.
Uniform Model (Uniform): This model returns a uniform posterior over possible goals known at the current episode , defining worst case performance.
MaxMargin Event Detection (MMED) [6]: A set of maxmargin models are trained to map features of a step history of state vectors to a goal score. We use the best performing sumL1norm features provided with the publicly available code.
RNN Classifier (RNN): An RNN is trained to predict . We experimented with a variety of parameters (see the Appendix) and report the best results.
Since all methods above are online algorithms, each of the models is updated after every episode . In order quantify performance with a single score, we use the mean probability assigned to the ground truth goal type over all episodes as (also denoted ). The goal forecasting performance results are summarized in Table 2 using the above metric.
5.4 Goal Forecasting with Perfect Visual Detectors
The experimental results up to this point have exclusively used visual detectors as input (., SLAM, scene classification, object recognition). While we have shown that our approach learns meaningful human activity models from real computer vision input, we would also like to understand how our online IRL method performs when decoupled from the noise of the visionbased input. We perform the same experiments described in Section 5.3 but with idealized (ground truth) inputs for goal discovery and action recognition. We still use Slam for localization.
Table 3 summarizes the mean true goal probability for each of the dataset environments. We observe a mean absolute performance improvement of by using idealized inputs. Our proposed model continues to perform the best against baselines methods. This performance indicates that as visionbased component technologies improve, we can expect significant improvements in the ability to predict the goals of human activities.
We also measure performance when the action detection is built from ground truth and the goal discovery is built from our described methods. Our expectation is that Darkowith stopbased discovery should outperform Darkowith scenebased based discovery, given the stopdetector’s more reliable goal detection performance (Table 1). The results over the dataset are given in Table 4, confirming our expectation.
Method  Home 1  Home 2  Office 1  Office 2  Lab 1 

Darko  
MMED [6]  
RNN  
Logistic  
Uniform 
Method  Home 1  Home 2  Office 1  Office 2  Lab 1 

Scenebased  
Stopbased 
5.5 Goal Forecasting Performance over Time
In additional to understanding the performance of goal prediction with a single score, we also plot the performance of goal prediction over time. We evaluate the goal forecasting performance as a function of the fraction of time until reaching the goal. In Figure 4, we plot the mean probability of the true goal at each fractional time step . Using fractional trajectory length allows for a performance comparison across trajectories of different lengths.
As shown in Figure 4, Darko
exhibits the property of maintaining uncertainty early in the trajectory and converging to the correct prediction as time elapses in most cases. In contrast, the logistic regression, RNN, and MMED perform worse at most time steps. As it approaches the goal, our method always produces a higher confidence in the correct goal with lower variance. We tried
argmax and Platt scaling [17] to perform multiclass prediction with MMED; argmax yielded higher , in addition to making noisier. While the RNN sees many states, its trajectorycentric hiddenstate representation may not have enough data to generalize as well as the statecentric baselines.5.6 Empirical Regret Analysis
We empirically show that our model has noregret with respect to the best model computed in hindsight under the MaxEntIRL loss function (negative logloss). In particular, we compute the regret (cumulative loss difference) between our online algorithm and the best hindsight model using the batch MaxEntIOC algorithm [34] at the end of all episodes. We plot the average regret for each environment in the dataset in Figure 5. The average regret of our algorithm approaches zero over time, matching our analysis.
5.7 Evaluation of Trajectory Length Estimates
Our model can also be used to estimate how long it will take a person to reach a predicted goal state. We predict the expected trajectory length as derived in Section 4.1. For the th episode, we use the normalized trajectory length prediction error defined as , where is the true trajectory length and (Eq. 6) is the predicted trajectory length. Proper evaluation of trajectory length towards a goal is challenging because our approach must learn valid goals in an online fashion. When a person approaches a new goal, our approach cannot accurately predict the goal because it has yet to learn that it is a valid goal state. As a result, our algorithm makes wrong goal predictions during episodes that terminate in new goal states. If we simply evaluate the mean performance, it will be dominated by the errors of the first episode terminating in a new goal state.
We evaluate median over all episodes. The median is not dominated by the errors of the first episode toward a new goal. We find most trajectory length forecasts are accurate, evidenced by the median of the normalized prediction error in Table 5. We include a partialtrajectory nearest neighbors baseline (NN). In Lab 1, the median trajectory length estimate is within of the true trajectory length.
Statistic  Home 1  Home 2  Office 1  Office 2  Lab 1 

Med. % Err.  
Med. % Err. NN  
Mean 
6 Conclusion
We proposed the first method for continuously modeling and forecasting a firstperson camera wearer’s future semantic behaviors at farreaching spatial and temporal horizons. Our method goes beyond predicting the physical trajectory of the user to predict their future semantic goals, and models the user’s relationship to objects and their environment. We have proposed several efficient and extensible methods for forecasting other semantic quantities of interest. Exciting avenues for future work include building upon the semantic state representation to model more aspects of the environment (which enables forecasting of more detailed futures), validation against human forecasting performance, and further generalizing the notion of a “goal” and how goals are discovered.
Acknowledgements: We acknowledge the support of NSF NRI Grant 1227495 and JST CREST Grant JPMJCR14E1. The authors thank Drew Bagnell for technical discussion, Katherine Lagree and Gunnar A. Sigurdsson for data collection assistance, and the anonymous reviewers for their helpful comments.
References

[1]
P. Abbeel and A. Y. Ng.
Apprenticeship learning via inverse reinforcement learning.
In
Proceedings of the twentyfirst international conference on Machine learning
, page 1. ACM, 2004. 
[2]
A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. FeiFei, and S. Savarese.
Social lstm: Human trajectory prediction in crowded spaces.
In
The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
, June 2016.  [3] Y. Cao, D. Barrett, A. Barbu, S. Narayanaswamy, H. Yu, A. Michaux, Y. Lin, S. Dickinson, J. Mark Siskind, and S. Wang. Recognize human activities from partially observed videos. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2013.
 [4] A. Fathi, A. Farhadi, and J. M. Rehg. Understanding egocentric activities. In 2011 International Conference on Computer Vision, pages 407–414. IEEE, 2011.
 [5] Y. Furukawa, B. Curless, S. M. Seitz, and R. Szeliski. Towards internetscale multiview stereo. In CVPR, 2010.
 [6] M. Hoai and F. De la Torre. Maxmargin early event detectors. International Journal of Computer Vision, 107(2):191–202, 2014.
 [7] K. M. Kitani, B. D. Ziebart, J. A. Bagnell, and M. Hebert. Activity forecasting. In European Conference on Computer Vision, pages 201–214. Springer, 2012.
 [8] H. S. Koppula and A. Saxena. Anticipating human activities using object affordances for reactive robotic response. IEEE transactions on pattern analysis and machine intelligence, 38(1):14–29, 2016.
 [9] T. Lan, T.C. Chen, and S. Savarese. A hierarchical representation for future action prediction. In European Conference on Computer Vision, pages 689–704. Springer, 2014.
 [10] Y. J. Lee, J. Ghosh, and K. Grauman. Discovering important people and objects for egocentric video summarization. In CVPR, volume 2, page 7, 2012.
 [11] K. Li and Y. Fu. Prediction of human activity by discovering temporal sequence patterns. IEEE transactions on pattern analysis and machine intelligence, 36(8):1644–1657, 2014.
 [12] Y. Li, Z. Ye, and J. M. Rehg. Delving into egocentric actions. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
 [13] M. Ma, H. Fan, and K. M. Kitani. Going deeper into firstperson activity recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1894–1903, 2016.
 [14] W.C. Ma, D.A. Huang, N. Lee, and K. M. Kitani. A gametheoretic approach to multipedestrian activity forecasting. arXiv preprint arXiv:1604.01431, 2016.
 [15] R. MurArtal, J. Montiel, and J. D. Tardós. Orbslam: a versatile and accurate monocular slam system. IEEE Transactions on Robotics, 31(5):1147–1163, 2015.
 [16] H. Pirsiavash and D. Ramanan. Detecting activities of daily living in firstperson camera views. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 2847–2854. IEEE, 2012.

[17]
J. Platt et al.
Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.
Advances in large margin classifiers, 10(3):61–74, 1999.  [18] N. D. Ratliff, J. A. Bagnell, and M. A. Zinkevich. Maximum margin planning. In Proceedings of the 23rd international conference on Machine learning, pages 729–736. ACM, 2006.
 [19] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. You only look once: Unified, realtime object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 779–788, 2016.
 [20] N. Rhinehart and K. M. Kitani. Learning action maps of large environments via firstperson vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 580–588, 2016.
 [21] M. S. Ryoo. Human activity prediction: Early recognition of ongoing activities from streaming videos. In Computer Vision (ICCV), 2011 IEEE International Conference on, 2011.
 [22] M. S. Ryoo and L. Matthies. Firstperson activity recognition: What are they doing to me? In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2730–2737, 2013.
 [23] S. ShalevShwartz et al. Online learning and online convex optimization. Foundations and Trends® in Machine Learning, 4(2):107–194, 2012.
 [24] K. Simonyan and A. Zisserman. Very deep convolutional networks for largescale image recognition. arXiv preprint arXiv:1409.1556, 2014.
 [25] H. Soo Park, J.J. Hwang, Y. Niu, and J. Shi. Egocentric future localization. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
 [26] S. Su, J. P. Hong, J. Shi, and H. S. Park. Social behavior prediction from first person videos. arXiv preprint arXiv:1611.09464, 2016.
 [27] W. Sun, A. Venkatraman, B. Boots, and J. A. Bagnell. Learning to filter with predictive state inference machines. In Proceedings of The 33rd International Conference on Machine Learning, pages 1197–1205, 2016.
 [28] C. Vondrick, H. Pirsiavash, and A. Torralba. Anticipating visual representations from unlabeled video. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
 [29] J. Walker, A. Gupta, and M. Hebert. Patch to the future: Unsupervised visual prediction. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 3302–3309. IEEE, 2014.
 [30] W. W.S. Wei. Time series analysis. AddisonWesley publ Reading, 1994.
 [31] D. Xie, S. Todorovic, and S.C. Zhu. Inferring “dark matter” and “dark energy” from videos. In Proceedings of the IEEE International Conference on Computer Vision, pages 2224–2231, 2013.

[32]
B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva.
Learning deep features for scene recognition using places database.
In Advances in neural information processing systems, pages 487–495, 2014.  [33] B. D. Ziebart. Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy. PhD thesis, Carnegie Mellon University, 2010.

[34]
B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey.
Maximum entropy inverse reinforcement learning.
In
AAAI Conference on Artificial Intelligence
, pages 1433–1438, 2008.  [35] B. D. Ziebart, N. Ratliff, G. Gallagher, C. Mertz, K. Peterson, J. A. Bagnell, M. Hebert, A. K. Dey, and S. Srinivasa. Planningbased prediction for pedestrians. In 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 3931–3936. IEEE, 2009.
Appendix
Appendix A Reward Function Feature Ablation Analysis
Feature Type  Home 1  Home 2  Office 1  Office 2  Lab 1 

State+Action  
State only  
Position only 
In Table 6, we show the mean true goal probability when labels are used as detectors (to isolate performance in the ideal case). While the purely positional representation of state performs well, it is almost always outperformed by the full representation of rewards that include features of both the full state and action. In Lab 1, the simpler representations slightly outperform the full, due to the relative simplicity of the highlevel activities in Lab 1. Here, knowledge of the state and previous goal alone is highly predictive of future goal.
Appendix B Incorporating Detection Noise
Current paradigms in vision often yield noise in the action and goal detectors necessary for Darko. We first describe our method for incorporating uncertainty in each goal detection, then conduct a performance analysis with synthetic noise. Then, we analyze the performance with real, noisy goal detection. We find Darkocan still perform well with forms of noisy goal and action detection. We find incorporating goal uncertainty significantly improves performance with synthetic noise, and shows improvements in the real goal detector setting. These results show that Darkocan tolerate the effects of noise, and further support the claim that it can enjoy the benefits of better scene and activity detection algorithms.
Harnessing goal detection confidence: In many scenarios, probability may be associated with each goal detection. We designed an effective method for handling realworld uncertainty. For known perfect goals, SoftValueIteration uses . Each goal is a maxima of and represents a reward of in log space. We replace each goal value with its logprobability: , which has the effect of biasing the policy towards goals with greater certainty. This results from the value iteration assigning higher value to states and actions closer to more certain goals, which makes the policy likelier to visit them. For example, if the goal detector yields a false positive of bathroom in the same area as a true positive detection of kitchen, the goal prediction posteriors for both goals will suffer, unless the false positive has an associated low (high uncertainty), in which case the policy is biased towards the correct goal of kitchen.
Noise analysis: We first analyze Darkounder the effect of adding noise to the GT. We add incorrect goal detections with probabilities , under various amounts of noise inserted uniformly at random across time: from of the number of original goal detections. For every scene, at each noise amount, we sample noise 5 times, and run Darkowith and without goal uncertainty for each corrupted sample, resulting in 225 paired experiments that evaluate the average goal forecasting probability. Perscene results are shown in Figures 6.
A onesided Wilcoxon signedrank test supports the hypothesis that incorporating high goal uncertainty yields better goal posterior prediction performance than not incorporating the uncertainty with
.Appendix C Proof of Regret Bound
Our regret bound is:
(8) 
where is a bound on the norm of , is feature dimensionality, and is the episode count (regret after the ’th episode).
Proof.
By Equation 2.5 of [23], the regret of online gradient descent is bounded:
(9) 
By using bounds on , and a minimizing choice of , we will prove the result.
Writing the general gradient in terms of the expected features (and omitting the subscript ):
(10) 
Using:
then Equation 10 becomes:
(11) 
Thus, using Equation 11 in Equation 9, and that the projection step of (constraining the set of to be the convex ball with radius ) ensures :
With the minimizing choice of ,
∎
Appendix D Derivation of Other Inference Tasks
d.1 ActionSubspace Visitation
To derive the actionsubspace visitation, we first use the posterior expected visitation count of performing an action immediately after arriving at a state is given in Equation 12, from [33].
(12)  
(13) 
Our definition of the posterior expected action subspace visitation count is given in Equation 14. This expresses the future expectation the user will perform an action while in a subspace , given their current trajectory .
(14)  
(15) 
.
Thus, the posterior expected action subspace visitation is straightforward to compute with . Various inference tasks can be constructed by choosing and appropriately.
d.2 Joint ActionState Subspace Visitation
We additionally derive the expected transition count from a subspace of states to a subspace of actions. This expresses the expectation that the user will perform an from a . It is defined as:
(16)  
(17) 
Again, computing this quantity is straightforward with . By marginalizing over various action and state subspaces that have semantic meaning, different inference quantities can be expressed and computed.
Appendix E Additional Dataset Information
Objects: The set of objects available in each environment is shown in Table 7.
Scene types: The set of scene types in each environment is shown in Table 8.
Labels: A small snippet of ground truth for Home 1 is shown in Table 9. The ground truth pairs frame indices (timestamps) with actions and goal arrivals.
Environment  Object Set 

Home 1  
Home 2  
Office 1  
Office 2  
Lab 1 
Environment  Scene Type Set 

Home 1  
Home 2  
Office 1  
Office 2  
Lab 1 
Frame Index  6750  6900  7200  7400  7630  7700 

Action/Arrival  Release Coat  Acquire Bookbag  Arrive Office  Acquire Mug  Arrive Kitchen  Release Mug 
Appendix F Regret with Detectors
We additionally show the empirical regret when using our goal discovery and action detection methods in Figure 7. We observe somewhat noisier regret behavior than in the original case, as the underlying demonstrations are noisier. The number of trajectories in Office 2 is higher here due to errors in the goal forecasting method, resulting in more goals being detected, which segments the demonstrations into more trajectories.
Appendix G Visualizations
We provide example 3D visualizations of 1) goal posterior 2) future state visitation and 3) the value function.
g.1 Goal posterior visualization
To emphasize our empirical finding that modeling the interaction with objects is useful for goal posterior prediction, we show an example sequence of predictions in Figure 8. Before the user grabs the mug, our algorithm predicts roughly equivalent probability to both the bedroom and kitchen. We see that after the user grabs the mug, Darkohas high confidence that the user will go to the kitchen.
g.2 Future state visitation visualizations
See Figure 9 for example visualizations of the expected future visitation counts. In order to visualize in 3 dimensions, we take the max visitation count across all states at each position. In rows 1, 3, and 4, a single demonstration is shown, which adapts to the agent’s trajectory (history). In row 2, the future state distribution drastically changes after each time the agent reaches a new goal.
g.3 Value function visualizations
See Figure 10 for example visualizations of the value function over time. Note 1) the state space size changes, and 2) that the value function changes over time, as the component of state that indicates the previous goal affects the value function.
g.4 RNN baseline settings
We experimented with a variety of settings for the RNN baseline. After each goal is detected, the RNN is refit. The settings we experimented with are cell {GRU, Basic}, learning rate , hidden dimension
, epochs after each goal
.
Comments
There are no comments yet.