Local-to-Global Panorama Inpainting for Locale-Aware Indoor Lighting Prediction

by   Jiayang Bai, et al.

Predicting panoramic indoor lighting from a single perspective image is a fundamental but highly ill-posed problem in computer vision and graphics. To achieve locale-aware and robust prediction, this problem can be decomposed into three sub-tasks: depth-based image warping, panorama inpainting and high-dynamic-range (HDR) reconstruction, among which the success of panorama inpainting plays a key role. Recent methods mostly rely on convolutional neural networks (CNNs) to fill the missing contents in the warped panorama. However, they usually achieve suboptimal performance since the missing contents occupy a very large portion in the panoramic space while CNNs are plagued by limited receptive fields. The spatially-varying distortion in the spherical signals further increases the difficulty for conventional CNNs. To address these issues, we propose a local-to-global strategy for large-scale panorama inpainting. In our method, a depth-guided local inpainting is first applied on the warped panorama to fill small but dense holes. Then, a transformer-based network, dubbed PanoTransformer, is designed to hallucinate reasonable global structures in the large holes. To avoid distortion, we further employ cubemap projection in our design of PanoTransformer. The high-quality panorama recovered at any locale helps us to capture spatially-varying indoor illumination with physically-plausible global structures and fine details.


page 1

page 3

page 4

page 5

page 7

page 8


GRIG: Few-Shot Generative Residual Image Inpainting

Image inpainting is the task of filling in missing or masked region of a...

Deep Graph Learning for Spatially-Varying Indoor Lighting Prediction

Lighting prediction from a single image is becoming increasingly importa...

Diverse Image Inpainting with Bidirectional and Autoregressive Transformers

Image inpainting is an underdetermined inverse problem, it naturally all...

Layout-guided Indoor Panorama Inpainting with Plane-aware Normalization

We present an end-to-end deep learning framework for indoor panoramic im...

Image Inpainting via Generative Multi-column Convolutional Neural Networks

In this paper, we propose a generative multi-column network for image in...

Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding

Image inpainting has made significant advances in recent years. However,...

Image Fine-grained Inpainting

Image inpainting techniques have shown promising improvement with the as...

Please sign up or login with your details

Forgot password? Click here to reset