World models , or models of environments , are an appealing way to represent an agent’s knowledge about its surroundings. An agent with a world model can predict its future by ‘imagining’ the consequences of a series of proposed actions. This capability can be used for sampling-based planning [16, 57], learning policies directly from the model (i.e., learning in a dream) [17, 23, 64, 25], and for counterfactual reasoning 
. Model-based approaches such as these also typically improve the sample efficiency of deep reinforcement learning[72, 62]. However, world models that generate high-dimensional visual observations (i.e., images) have typically been restricted to relatively simple environments, such as Atari games  and tabletops .
Our goal is to develop a generic visual world model for agents navigating in indoor environments. Specifically, given one or more previous observations and a proposed navigation action sequence, we aim to generate plausible high-resolution visual observations for viewpoints that have not been visited, and do so in buildings not seen during training. Beyond applications in video editing and content creation, solving this problem would unlock model-based methods for many embodied AI tasks, including navigating to objects , instruction-guided navigation [3, 66, 40] and dialog-guided navigation [74, 26]. For example, an agent asked to find a certain type of object in a novel building, e.g. ‘find a chair’, could perform mental simulations using the world model to identify navigation trajectories that are most likely to include chair observations – without moving.
Building such a model is challenging. It requires synthesizing completions of partially visible objects, using as few as one previous observation. This is akin to novel view synthesis from a single image [19, 80], but with potentially unbounded viewpoint changes. There is also the related but considerably more extreme challenge of predicting around corners. For example, as shown in Fig. 1, any future navigation trajectory passing the entrance of an unseen room requires the model to plausibly imagine the entire contents of that room (we dub this the room reveal problem). This requires generalizing from the visual, spatial and semantic structure of previously explored environments—which in our case are photo-realistic 3D captures of real indoor spaces in the Matterport3D dataset . A third problem is temporal consistency: predictions of unseen building regions should ideally be stochastic (capturing the full distribution of possible outcomes), but revisited regions should be rendered in a consistent manner to previous observations.
Towards this goal, we introduce Pathdreamer. Given one or more visual observations (consisting of RGB, depth and semantic segmentation for panoramas) from an indoor scene, Pathdreamer synthesizes high-resolution visual observations (RGB, depth and semantic segmentations) along a specified trajectory through future viewpoints, using a hierarchical two-stage approach. Pathdreamer’s first stage, Structure Generator, generates depth and semantic segmentations. Inspired by work in video prediction 
, these outputs are conditioned on a latent noise tensor capturing the stochastic information about the next observation (such as the layout of an unseen room) that cannot be predicted deterministically. The second stage’s Image Generator renders the depth and semantic segmentations as realistic RGB images using modified Multi-SPADE blocks[63, 51]. To maintain long-term consistency in the generated observations, both stages use back-projected 3D point cloud representations which are re-projected into image space for context .
As illustrated in Figure 1, Pathdreamer can generate plausible views of previously unseen indoor scenes under large viewpoint changes, while also addressing the room reveal problem – in this case correctly hypothesizing that the unseen room revealed at position 2 will most likely resemble a kitchen. Empirically, using the Matterport3D dataset  and observations, we evaluate both stages of our model against prior work and reasonable baselines and ablations. We find that the hierarchical structure of the model is essential for predicting over large viewpoint changes, that maintaining both RGB and semantic context is required, and that prediction quality degrades gradually when we evaluate with trajectory rollouts of up to 13m (with viewpoints 2.25m apart on average).
Encouraged by these results, we further evaluate whether RGB predictions from Pathdreamer can improve performance on a downstream visual navigation task. We focus on Vision-and-Language Navigation (VLN) using the R2R dataset . VLN requires agents to interpret and execute natural language navigation instructions in a photorealistic 3D environment. A robust finding from previous VLN research is that task success is dramatically increased by allowing an agent to look ahead at unobserved parts of the environment while following an instruction . We find that replacing look-ahead observations with Pathdreamer predictions maintains around half of this improvement, a finding we expect to have significant implications for research in this area. In summary, our main contributions include:
Proposing the study of visual world models for generic indoor environments and defining evaluation protocols and baselines for future work.
Extensive experiments characterizing the performance of Pathdreamer and demonstrating improved results on the downstream VLN task .
2 Related Work
Our work is closely related to the task of video prediction, which aims to predict the future frames of a video sequence. While some video prediction methods predict RGB video frames directly [76, 1, 41, 44], many others use hierarchical models to first predict an intermediate representation (such as semantic segmentation) [47, 35, 77, 82, 42], which improves the fidelity of long-term predictions . Several approaches have also incorporated 3D point cloud representations, using projective camera geometry to explicitly infer aspects of the next frame [75, 51, 43]. Inspired by this work, we adopt and combine both the hierarchical two-stage approach and 3D point cloud representations. Further, since our interest is in action-conditional world models, we provide a trajectory of future viewpoints to the model rather than assuming a constant frame rate and modeling camera motion implicitly, which is more typical in video generation [44, 42].
Action-Conditional Video Prediction
Conditional video prediction to improve agent reasoning and planning has been explored in several tasks. This includes video prediction for Atari games conditioned on control inputs [60, 10, 62, 25] and 3D game environments like Doom . In robotics, action-conditional video prediction has been investigated for object pushing in tabletop settings to improve generalization to novel objects [15, 16, 14]. This work has been restricted to simple environments and low-resolution images, such as 6464 images of objects in a wooden box. To the best of our knowledge, we are the first to investigate action-conditional video prediction in building-scale environments with high-resolution (1024512) images.
World Models and Navigation Priors
World models  are an appealing way to summarize and distill knowledge about complex, high-dimensional environments. However, world models can differ in their outputs. While Pathdreamer predicts visual observations, there is also a vast literature on world models that predict compact latent representations of future states [38, 24, 25] or other task-specific measurements  or rewards . This includes recent work attempting to learn statistical regularities and other priors for indoor navigation—for example, by mining spatial co-occurrences from real estate video tours , learning to predict top-down belief maps over room characteristics , or learning to reconstruct house floor plans using audio and visual cues from a short video sequence . In contrast to these approaches, we focus on explicitly predicting visual observations (i.e., pixels) which are generic, human-interpretable, and apply to a wide variety of downstream tasks and applications. Further, recent work identifies a close correlation between image prediction accuracy and downstream task performance in model-based RL .
Embodied Navigation Agents
High-quality 3D environment datasets such as Matterport3D , StreetLearn [56, 53], Gibson  and Replica  have triggered intense interest in developing embodied agents that act in realistic human environments . Tasks of interest include ObjectNav  (navigating to an instance of a particular kind of object), and Vision-and-Language Navigation (VLN) , in which agents must navigate according to natural language instructions. Variations of VLN include indoor navigation [3, 33, 66, 40], street-level navigation [9, 53], vision-and-dialog navigation [59, 74, 26], VLN in continuous environments , and more. Notwithstanding considerable exploration of pretraining strategies [46, 27, 50, 87], data augmentation approaches [20, 21, 73]
, agent architectures and loss functions[86, 48, 49], existing work in this space considers only model-free approaches. Our aim is to unlock model-based approaches to these tasks, using a visual world model to encode prior commonsense knowledge about human environments and thereby relieve the burden on the agent to learn these regularities. Underscoring the potential of this direction, we note that using the ground-truth environment for planning with beam search typically improves VLN success rates on the R2R dataset by 17-19% [20, 73].
Novel View Synthesis
Finally, we position our work in the context of novel view synthesis [19, 37, 29, 18, 70, 85, 54]. Many methods have been proposed to represent 3D scenes, including point cloud representations , layered depth images , and mesh representations . More recently, neural radiance fields (NeRF) [55, 52, 83]
achieved impressive results by capturing volume density and color implicitly with a neural network. NeRF models can synthesize very high quality 3D scenes, but a significant drawback for our purposes is that they require a large number of input views to render a single scene (e.g., 20–62 images per scene in). More importantly, these models are typically trained to represent a single scene, and currently do not generalize well to unseen environments. In contrast, our problem demands generalization to unseen environments, using as little as one previous observation.
Pathdreamer is a world model that generates high-resolution visual observations from a trajectory of future viewpoints in buildings it has never observed. The input to Pathdreamer is a sequence of previous observations consisting of RGB images , semantic segmentation images , and depth images
(where the depth and segmentations could be ground-truth or estimates from a model). We assume that a corresponding sequence of camera posesis available from an odometry system, and that the camera intrinsics are known or estimated. Our goal is to generate realistic RGB, semantic segmentation and depth images for a trajectory of future poses , which may be provided up front or iteratively by some agent interacting with the returned observations. Note that we generate depth and segmentation because these modalities are useful in many downstream tasks. We assume that the future trajectory may traverse unseen areas of environment, requiring the model to not only in-fill minor object dis-occlusions, but also to imagine entire room reveals (Figure 1).
Figure 2 shows our proposed hierarchical two-stage model for addressing this challenge. It uses a latent noise tensor to capture the stochastic information about the next observation (e.g. the layout of an unseen room) that cannot be predicted deterministically. Given a sampled noise tensor , the first stage (Structure Generator) generates a new depth image and segmentation image to provide a plausible high-level semantic representation of the scene, using as context the previous semantic and depth images , . In the second stage (Image Generator), the predicted semantic and depth images , are rendered into a realistic RGB image using previous RGB images as context. In each stage, context is provided by accumulating previous observations as a 3D point cloud which is re-projected into 2D using .
3.1 Structure Generator: Segmentation & Depth
Pathdreamer’s first stage is the Structure Generator, a stochastic encoder-decoder network for generating diverse, plausible segmentation and depth images. Like , to provide the previous observation context, we first back-project the previous segmentations into a unified 3D semantic point cloud using the depth images and camera poses . We then re-project this point cloud back into pixel space using to create sparse segmentation and depth guidance images , which reflect the current pose.
The input to the encoder is a one-hot encoding of the semantic guidance image, concatenated with the depth guidance image . The architecture of the encoder-decoder model is based on RedNet  – a ResNet-50  architecture designed for indoor RGB-D semantic segmentation. RedNet uses transposed convolutions for upsampling in the decoder and skip connections between the encoder and decoder to preserve spatial information. Since the input contains a segmentation image, and segmentation classes differ across datasets, the encoder-decoder is not pretrained. We introduce the latent spatial noise tensor into the model by concatenating it with the feature map between the encoder and the decoder. The final output of the encoder-decoder model is a segmentation image and a depth image , with segmentation predictions generated by a -way softmax and depth outputs normalized in the range
and generated via a sigmoid function. At each step during inference, the segmentation predictionis back-projected and added to the point cloud to assist prediction in future timesteps.
To generate the noise tensor , we take inspiration from SVG  and learn a conditional prior noise distribution . Intuitively, there are many possible scenes that may be generated for an unseen building region. We would like to carry the stochastic information about the next observation that the deterministic encoder cannot capture, and we would like for the decoder to make good use of that information. During training, we encourage the first outcome by using a KL-divergence loss to force the prior distribution to be close to the posterior distribution which is conditioned on the ground-truth segmentation and depth images. We encourage the second outcome by providing the decoder with sampled values from the posterior distribution (conditioned on the ground-truth outputs) during training. During inference, the latent noise is sampled from the prior distribution and the posterior distribution is not used. Both distributions are modeled using 3-layer CNNs that take their input from the encoder and output two channels representing and
to parameterize a multivariate Gaussian distribution. As shown in Figure 3, the noise is useful in encoding diverse, plausible representations of unseen regions.
Overall, the Structure Generator is trained to minimize a joint loss consisting of a cross-entropy loss for semantic predictions, a mean absolute error term for depth predictions, and the KL-divergence term for the noise tensor:
where , , and are weights determined by a grid search. We set these to 1, 100, and 0.5 respectively.
3.2 Image Generator: RGB
The Image Generator is an image-to-image translation GAN[22, 78] that converts the semantic and depth predictions , from the first stage into a realistic RGB image . Our model architecture is based on SPADE blocks  that use spatially-adaptive normalization layers to insert context into multiple layers of the network. As with our Structure Generator, we maintain an accumulating 3D point cloud containing all previous image observations. This provides a sparse RGB guidance image when re-projected. Similar to Multi-SPADE , we insert two SPADE normalization layers into each residual block: one conditioned on the concatenated semantic and depth inputs , and one conditioned on the RGB guidance image . The sparsity of the RGB guidance image is handled by applying partial convolutions . In total Image Generator consists of 7 Multi-SPADE blocks, preceded by a single convolution block.
Following SPADE , the model is trained with the GAN hinge loss, feature matching loss , and perceptual loss  from a pretrained VGG-19  model. During training, the generator is provided with the ground-truth segmentation image and ground-truth depth image . Our discriminator architecture is based on PatchGAN , and takes as input the concatenation of the ground-truth image or generated image , the ground-truth depth image and the ground-truth semantic image . The losses for the generator and the discriminator are:
where denotes the complete set of inputs to the generator, denotes the intermediate output of the layer of the pretrained VGG-19 network, denotes the output of the discriminator’s -th layer, and the conditioning inputs to the discriminator have been dropped to save space. We follow  for the choice of weights . Like the Structure Generator, the Image Generator is not pretrained.
3.3 Training and Inference
For training and evaluation we use Matterport3D , a dataset of 10.8k RGB-D images from 90 building-scale indoor environments. For each environment, Matterport3D also includes a textured 3D mesh which is annotated with 40 semantic classes of objects and building components. To align with downstream VLN tasks, in all experiments the RGB, depth and semantic images are panoramas in equirectangular format.
To train Pathdreamer, we sampled 400k trajectories from the Matterport3D training environments. To define feasible trajectories, we used the navigation graphs from the Room-to-Room (R2R) dataset , in which nodes correspond to panoramic image locations, and edges define navigable state transitions. For each trajectory 5–8 panoramas were sampled, choosing the starting node and the edge transitions uniformly at random. On average the viewpoints in these trajectories are 2m apart. Training with relatively large viewpoint changes is desirable, since the model learns to synthesize observations with large viewpoint changes in a single step (without the need to incur the computational cost of generating intervening frames). However, this does not preclude Pathdreamer from generating smooth video outputs at high frame rates111See https://youtu.be/HNAmsdk7lJ4 for our video generation results..
The first and second stages of the model are trained separately. For the Image Generator, we use the Matterport3D RGB panoramas as training targets at 1024512 resolution. We use the Habitat simulator  to render ground-truth depth and semantic training inputs and stitch these into equirectangular panoramas. We perform data augmentation by randomly cropping and horizontally rolling the RGB panoramas, which we found essential due to the limited number of panoramas available.
To train the Structure Generator, we again used Habitat to render depth and semantic images. Since this stage does not require aligned RGB images for training, in this case we performed data augmentation by perturbing the viewpoint coordinates with a random Gaussian noise vector drawn fromindependently along each 3D axis. The Structure Generator was trained with equirectangular panoramas at 512256 resolution.
To avoid heading discontinuities during inference, we use circular padding on the image x-axis for both the Structure Generator and the Image Generator. The 512256 resolution semantic and depth outputs of the Structure Generator are upsampled to 1024
512 using nearest neighbor interpolation before they are passed to the Image Generator. In quantitative experiments, we set the Structure Generator noise tensorto the mean of the prior.
For evaluation we use the paths from the Val-Seen and Val-Unseen splits of the R2R dataset . Val-Seen contains 340 trajectories from environments in the Matterport3D training split. Val-Unseen contains 783 trajectories in Matterport3D environments not seen in training. Since R2R trajectories contain 5-7 panoramas and at least 1 previous observation is given as context, we report evaluations over 1–6 steps, representing predictions over trajectory rollouts of around 2–13m (panoramas are 2.25m apart on average). See Figure 4 for an example rollout over 8.6m. We characterize the performance of Pathdreamer in comparison to baselines, ablations and in the context of the downstream task of Vision-and-Language Navigation (VLN).
4.1 Pathdreamer Results
A key feature of our approach is the ability to generate semantic segmentation and depth outputs, in addition to RGB. We evaluate the generated semantic segmentation images using mean Intersection-Over-Union (mIOU) and report results for:
Nearest Neighbor: A baseline without any learned components, using nearest-neighbor interpolation to fill holes in the projected semantic guidance image .
Ours (Teacher Forcing): Structure Generator trained using the ground truth semantic and depth images as the previous observation at every time step.
Ours (Recurrent): Structure Generator trained while feeding back its own semantic and depth predictions as previous observations for the next step prediction. This reduces train-test mismatch and may allow the model to compensate for errors when doing longer roll-outs.
We also tried training the hierarchical convolutional LSTM from , but found that it frequently collapsed to a single class prediction. We attribute this to the large viewpoint changes and heavy occlusion in the training sequences; we believe this can be more effectively modeled with point cloud geometry than with a geometry-unaware LSTM.
As illustrated in Table 1, Pathdreamer performs far better than the Nearest Neighbor baseline regardless of the number of steps in the rollout or the number of previous observations used as context. As expected, performance in seen environments is higher than unseen. Perhaps surprisingly, in Figure 4(a) we show that Recurrent training improves results during longer rollouts in the training environments (Val-Seen), but this does not improve results on Val-Unseen, perhaps indicating that the error compensation learned by the Image Generator does not easily generalize.
In addition to accurate predictions, we also want generated results to be diverse. Figure 3 shows that our model can generate diverse semantic scenes by interpolating the noise tensor , and that the RGB outputs closely reflect the generated semantic image. This allows us to generate multiple plausible alternatives for the same navigation trajectory.
Pathdreamer semantic segmentation mean-IOU (above) and RGB generation FID (below). Results are shown for Val-Seen (left) and Val-Unseen (right). Confidence intervals indicate the range of outcomes with 1, 2 or 3 previous observations as context.
To evaluate the quality of RGB panoramas generated by the Image Generator, we compute the Fréchet Inception Distance (FID)  between generated and real images for each step in the paths. We report results using the semantic images generated by the Structure Generator as inputs (i.e., our full model). To quantify the potential for uplift with better Structure Generators, we also report results using ground truth semantic segmentations as input. We compare to two ablated versions of our model:
No Semantics: The semantic and depth inputs , are removed from the Multi-SPADE blocks.
SPADE: An ablation of the RGB inputs to the model, comprising the previous RGB image and the re-projected RGB guidance image . The semantic image replaces as input to the model and the input layers are removed from the Multi-SPADE blocks, making this effectively the SPADE model .
|Model||Context||1 Step||1–6 Steps||1 Step||1–6 Steps|
|Ours (Teacher Forcing)||1||84.9||59.2||78.3||50.8|
|Ours (Teacher Forcing)||2||85.4||64.6||77.4||55.5|
|Ours (Teacher Forcing)||3||85.1||68.5||77.3||60.4|
|Model||Context||Obs||Sem||RGB||1 Step||1–6 Steps||1 Step||1–6 Steps|
As shown in Table 2, SPADE performs the best in Val-Seen, indicating that the model has the capacity to memorize the training environments. In this case, RGB inputs are not necessary. However, our model performs noticeably better in Val-Unseen, highlighting the importance of maintaining RGB context in unseen environments (which is our focus). Performance degrades significantly in the No Semantics setting in both Val-Seen and Val-Unseen. We observed that without semantic inputs, the model is unable to generate meaningful images over longer horizons, which validates our two-stage hierarchical approach. These results are reflected in the FID scores, as well as qualitatively (Figure 6); Image Generator’s outputs are significantly crisper, especially over longer horizons. Due to the benefit of guidance images, the Image Generator’s textures are also generally better matched with the unseen environment, while SPADE tends to wash out textures, usually creating images of a standard style. Figure 4(b) plots performance for every setting step-by-step. FID of the Image Generator improves substantially when using ground truth semantics, particularly for longer rollouts, highlighting the potential to benefit from improvements to the Structure Generator.
4.2 VLN Results
Finally, we evaluate whether predictions from Pathdreamer can improve performance on a downstream visual navigation task. We focus on Vision-and-Language Navigation (VLN) using the R2R dataset . Because reaching the navigation goal requires successfully grounding natural language instructions to visual observations, this provides a challenging task-based assessment of prediction quality.
In our inference setting, at each step while moving through the environment we use a baseline VLN agent based on  to generate a large number of possible future trajectories using beam search. We then rank these alternative trajectories using an instruction-trajectory compatibility model  to assess which trajectory best matches the instruction. The agent then executes the first action from the top-ranked trajectory before repeating the process. We consider three different planning horizons, with future trajectories containing 1, 2 or 3 forward steps.
The instruction-trajectory compatibility model is a dual-encoder that separately encodes textual instructions and trajectories (encoded using visual observations and path geometry) into a shared latent space. To improve performance on incomplete paths, we introduce truncated paths into the original contrastive training scheme proposed in . The compatibility model is trained using only ground truth observations. However, during inference, RGB observations for future steps are drawn from three different sources:
Ground truth: RGB observations from the actual environment, i.e., look-ahead observations.
Pathdreamer: RGB predictions from our model.
Repeated pano: A simple baseline in which the most recent RGB observation is repeated in future steps.
Note that in all cases the geometry of the future trajectories is determined by the ground truth R2R navigation graphs. In Table 3, we report Val-Unseen results for this experiment using standard metrics for VLN: navigation error (NE), success rate (SR), shortest path length (SPL), normalized Dynamic Time Warping (nDTW) , and success weighted by normalized Dynamic Time Warping (sDTW) .
Consistent with prior work [20, 73], we find that looking ahead using ground truth visual observations provides a robust performance boost, e.g., success rate increases from 44.6% with 1 planning step (top panel) to 59.3% with 3 planning steps (bottom panel). At the other extreme, the Repeated pano baseline is weak, with a success rate of just 35.7% with 1 planning step (top row). This is not surprising: repeating the last pano denies the compatibility model any useful visual representation of the next action, which is crucial to performance [20, 73]. However, increasing the planning horizon does improve performance even for the Repeated pano baseline, since the compatibility model is able to compare the geometry of alternative future trajectories. Finally, we observe that using Pathdreamer’s visual observations closes about half the gap between the Repeated pano baseline and the ground truth observations, e.g., 48.9% success with Pathdreamer vs. 40.6% and 59.3% respectively for the others. We conclude that using Pathdreamer as a visual world model can improve performance on downstream tasks, although existing agents still rely on using a navigation graph to define the feasible action space at each step. Pathdreamer is complementary to current SOTA model-based approaches, and a combination would likely lead to further boosts in VLN performance, which is worth investigating in future work.
In this paper, we presented Pathdreamer, a stochastic hierarchical visual world model. Pathdreamer is capable of synthesizing realistic and diverse panoramic images for unseen trajectories in real buildings. As a visual world model, Pathdreamer also shows strong promise in improving performance on downstream tasks, such as VLN. Most notably, we show that Pathdreamer captures around half the benefit of looking ahead at actual observations from the environment. The efficacy of Pathdreamer in the VLN task may be attributed to its ability to model fundamental constraints in the real world – relieving the agent from having to learn the geometry and visual and semantic structure of buildings. Applying Pathdreamer to other embodied navigation tasks such as Object-Nav , VLN-CE  and street-level navigation [9, 53] are natural directions for future work.
-  (2018) FutureGAN: anticipating the future frames of video sequences using spatio-temporal 3d convolutions in progressively growing gans. arXiv preprint arXiv:1810.01325. Cited by: §2.
-  (2018) On evaluation of embodied navigation agents. arXiv preprint arXiv:1807.06757. Cited by: §2.
-  (2018) Vision-and-language navigation: interpreting visually-grounded navigation instructions in real environments. In CVPR, pp. 3674–3683. Cited by: 3rd item, §1, §1, §2, §3.3, §4.2, §4.
-  (2020) Models, pixels, and rewards: evaluating design trade-offs in visual model-based reinforcement learning. arXiv preprint arXiv:2012.04603. Cited by: §2.
-  (2020) Objectnav revisited: on evaluation of embodied agents navigating to objects. arXiv preprint arXiv:2006.13171. Cited by: §1, §2, §5.
-  (2019) Woulda, coulda, shoulda: counterfactually-guided policy search. In ICLR, Cited by: §1.
-  (2017) Matterport3D: learning from RGB-D data in indoor environments. International Conference on 3D Vision (3DV). Cited by: §1, §1, §2, §3.3.
-  (2020) Semantic visual navigation by watching youtube videos. NeurIPS. Cited by: §2.
-  (2019) Touchdown: natural language navigation and spatial reasoning in visual street environments. In CVPR, pp. 12538–12547. Cited by: §2, §5.
-  (2017) Recurrent environment simulators. ICLR. Cited by: §2.
-  (2018) Stochastic video generation with a learned prior. icml. Cited by: 2nd item, §1, §3.1.
-  (2019) Peeking behind objects: layered depth prediction from a single image. Pattern Recognition Letters 125, pp. 333–340. Cited by: §2.
-  (2017) Learning to act by predicting the future. In ICLR, Cited by: §2.
-  (2017) Self-supervised visual planning with temporal skip connections. Conference on Robot Learning (CoRL). Cited by: §2.
-  (2016) Unsupervised learning for physical interaction through video prediction. In NeurIPS, pp. 64–72. Cited by: §2.
-  (2017) Deep visual foresight for planning robot motion. In ICRA, pp. 2786–2793. Cited by: §1, §2.
Deep spatial autoencoders for visuomotor learning. In ICRA, pp. 512–519. Cited by: §1.
-  (2019) Deepview: view synthesis with learned gradient descent. In CVPR, pp. 2367–2376. Cited by: §2.
-  (2016) DeepStereo: learning to predict new views from the world’s imagery. In CVPR, Cited by: §1, §2.
-  (2018) Speaker-follower models for vision-and-language navigation. NeurIPS. Cited by: §2, §4.2.
-  (2020) Counterfactual vision-and-language navigation via adversarial path sampler. In ECCV, pp. 71–86. Cited by: §2.
-  (2014) Generative adversarial networks. NeurIPS. Cited by: §3.2.
-  (2018) World models. NeurIPS. Cited by: §1, §2, §2.
-  (2019) Dream to control: learning behaviors by latent imagination. ICLR. Cited by: §2.
-  (2020) Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193. Cited by: §1, §2, §2.
-  (2020) Where are you? localization from embodied dialog. Cited by: §1, §2.
-  (2020-06) Towards learning a generic agent for vision-and-language navigation via pre-training. In CVPR, Cited by: §2.
-  (2016) Deep residual learning for image recognition. In CVPR, pp. 770–778. Cited by: §3.1.
-  (2018) Single-image tomography: 3d volumes from 2d cranial x-rays. In Computer Graphics Forum, Vol. 37, pp. 377–388. Cited by: §2.
-  (2017) GANs trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS, Cited by: §4.1.
-  (2019) General evaluation for instruction conditioned navigation using dynamic time warping. NeurIPS Workshop on Visually Grounded Interaction and Language (ViGIL). Cited by: §4.2.
-  (2017) Image-to-image translation with conditional adversarial networks. CVPR. Cited by: §3.2.
-  (2019) Stay on the path: instruction fidelity in vision-and-language navigation. Cited by: §2.
-  (2018) Rednet: residual encoder-decoder network for indoor rgb-d semantic segmentation. arXiv preprint arXiv:1806.01054. Cited by: §3.1.
-  (2017) Predicting scene parsing and motion dynamics in the future. In NeurIPS, pp. 6915–6924. Cited by: §2.
Perceptual losses for real-time style transfer and super-resolution. In ECCV, pp. 694–711. Cited by: §3.2.
-  (2017) Learning a multi-view stereo machine. In NeurIPS, pp. 365–376. Cited by: §2.
-  (2017) Deep variational bayes filters: unsupervised learning of state space models from raw data. ICLR. Cited by: §2.
-  (2020) Beyond the nav-graph: vision and language navigation in continuous environments. In ECCV, Cited by: §2, §5.
-  (2020) Room-across-room: multilingual vision-and-language navigation with dense spatiotemporal grounding. EMNLP. Cited by: §1, §2.
-  (2019) Predicting future frames using retrospective cycle gan. In CVPR, pp. 1811–1820. Cited by: §2.
-  (2021) Revisiting hierarchical approach for persistent long-term video prediction. In ICLR, Cited by: §2, §4.1.
-  (2020) Street-view panoramic video synthesis from a single satellite image. arXiv preprint arXiv:2012.06628. Cited by: §2.
-  (2020) Infinite nature: perpetual view generation of natural scenes from a single image. arXiv preprint arXiv:2012.09855. Cited by: §2.
-  (2018) Image inpainting for irregular holes using partial convolutions. In ECCV, pp. 85–100. Cited by: §3.2.
-  (2019) Vilbert: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NeurIPS, pp. 13–23. Cited by: §2.
-  (2017) Predicting deeper into the future of semantic segmentation. In ICCV, pp. 648–657. Cited by: §2.
-  (2019) Self-monitoring navigation agent via auxiliary progress estimation. ICLR. Cited by: §2.
The regretful agent: heuristic-aided navigation through progress estimation. In CVPR, pp. 6732–6740. Cited by: §2.
-  (2020) Improving vision-and-language navigation with image-text pairs from the web. ECCV. Cited by: §1, §2.
-  (2020) World-consistent video-to-video synthesis. ECCV. Cited by: 2nd item, §1, §2, §3.1, §3.2.
-  (2021) NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR, Cited by: §2.
-  (2020) Retouchdown: adding touchdown to streetlearn as a shareable resource for language grounding tasks in street view. EMNLP Workshop on Spatial Language Understanding (SpLU). Cited by: §2, §5.
-  (2019) Local light field fusion: practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG) 38 (4), pp. 1–14. Cited by: §2.
-  (2020) Nerf: representing scenes as neural radiance fields for view synthesis. ECCV. Cited by: §2.
-  (2019) The streetlearn environment and dataset. arXiv preprint arXiv:1903.01292. Cited by: §2.
-  (2018) Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In ICRA, pp. 7559–7566. Cited by: §1.
-  (2020) Seeing the un-scene: learning amodal semantic maps for room navigation. In ECCV, pp. 513–529. Cited by: §2.
Help, anna! visual navigation with natural multimodal assistance via retrospective curiosity-encouraging imitation learning. EMNLP. Cited by: §2.
-  (2015) Action-conditional video prediction using deep networks in atari games. In NeurIPS, pp. 2863–2871. Cited by: §2.
-  (2017) Value prediction network. In NeurIPS, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Cited by: §2.
-  (2020) Model-based reinforcement learning for atari. In ICLR, Cited by: §1, §2.
-  (2019) Semantic image synthesis with spatially-adaptive normalization. In CVPR, pp. 2337–2346. Cited by: 2nd item, §1, §3.2, §3.2, 2nd item.
-  (2019) Learning real-world robot policies by dreaming. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7680–7687. Cited by: §1.
-  (2020) Audio-visual floorplan reconstruction. arXiv preprint arXiv:2012.15470. Cited by: §2.
-  (2020) REVERIE: remote embodied visual referring expression in real indoor environments. In CVPR, Cited by: §1, §2.
-  (2019) Habitat: A Platform for Embodied AI Research. In ICCV, Cited by: §3.3.
-  (2020-06) 3D photography using context-aware layered depth inpainting. In CVPR, Cited by: §2.
-  (2015) Very deep convolutional networks for large-scale image recognition. ICLR. Cited by: §3.2.
-  (2019) Pushing the boundaries of view extrapolation with multiplane images. In CVPR, pp. 175–184. Cited by: §2.
-  (2019) The Replica dataset: a digital replica of indoor spaces. arXiv preprint arXiv:1906.05797. Cited by: §2.
-  (2018) Reinforcement learning: an introduction. A Bradford Book, Cambridge, MA, USA. Cited by: §1.
-  (2019) Learning to navigate unseen environments: back translation with environmental dropout. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2610–2621. Cited by: §2, §4.2.
-  (2020) Vision-and-dialog navigation. In Conference on Robot Learning (CoRL), pp. 394–406. Cited by: §1, §2.
-  (2018) Future semantic segmentation leveraging 3d information. ECCV 3D Reconstruction meets Semantics Workshop. Cited by: §2.
-  (2014) Patch to the future: unsupervised visual prediction. In CVPR, pp. 3302–3309. Cited by: §2.
-  (2018) Video-to-video synthesis. NeurIPS. Cited by: §2.
-  (2018) High-resolution image synthesis and semantic manipulation with conditional gans. In CVPR, pp. 8798–8807. Cited by: §3.2, §3.2.
-  (2019) Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In CVPR, pp. 6629–6638. Cited by: §4.2.
-  (2020) Synsin: end-to-end view synthesis from a single image. In CVPR, pp. 7467–7477. Cited by: §1, §2.
-  (2018) Gibson env: real-world perception for embodied agents. In CVPR, Cited by: §2.
-  (2018) Structure preserving video prediction. In CVPR, pp. 1460–1469. Cited by: §2.
-  (2020) PixelNeRF: neural radiance fields from one or few images. arXiv preprint arXiv:2012.02190. Cited by: §2.
-  (2021) On the evaluation of vision-and-language navigation instructions. In Conference of the European Chapter of the Association for Computational Linguistics (EACL), Cited by: §4.2, §4.2.
-  (2018) Stereo magnification: learning view synthesis using multiplane images. SIGGRAPH. Cited by: §2.
-  (2020) Vision-language navigation with self-supervised auxiliary reasoning tasks. In CVPR, pp. 10012–10022. Cited by: §2.
-  (2021) Multimodal text style transfer for outdoor vision-and-language navigation. EACL. Cited by: §2.