In this paper we present an end-to-end deep learning framework to turn images that show dynamic content, such as vehicles or pedestrians, into realistic static frames. This objective encounters two main challenges: detecting the dynamic objects, and inpainting the static occluded background. The second challenge is approached with a conditional generative adversarial model that, taking as input the original dynamic image and the computed dynamic/static binary mask, is capable of generating the final static image. The former challenge is addressed by the use of a convolutional network that learns a multi-class semantic segmentation of the image. The objective of this network is producing an accurate segmentation and helping the previous generative model to output a realistic static image. These generated images can be used for applications such as virtual reality or vision-based robot localization purposes. To validate our approach, we show both qualitative and quantitative comparisons with other methods by removing the dynamic objects and hallucinating (inpainting) the static structure behind them.
Check some of our results on images coming from the driving simulator
CARLA.
Slide the blue button to see the input and output of our framework.
Here we post some of our results on real-world images. The two left images are from the
CITYSCAPES dataset,
and the two right images are from the SVS dataset.
In our GitHub page we have some scripts available to generate the dataset with CARLA. Also, you can download a small dataset from here. Note that this dataset is valid for testing, but it contains very few images for training.
Here we show 4 examples of the training data format. These images are sourced from the CARLA driving simulator:
@inproceedings{bescos2018empty,
author = {Bescos, Berta and Neira, Jos{\'e} and Siegwart, Roland and Cadena, Cesar},
title = {{Empty Cities: Image Inpainting for a Dynamic-Object-Invariant Space}},
journal = {IEEE International Conference on Robotics and Automation (ICRA)},
year = {2019}
}