Empty Cities
Teaser Image

EmptyCities

EmptyCities

EmptyCities: Image Inpainting for a Dynamic-Object-Invariant Space

Berta Bescos, Jose Neira, Roland Siegwart, Cesar Cadena

University of Zaragoza, Spain     ETH Zurich, Switzerland

IEEE ICRA 2019

Abstract

In this paper we present an end-to-end deep learning framework to turn images that show dynamic content, such as vehicles or pedestrians, into realistic static frames. This objective encounters two main challenges: detecting the dynamic objects, and inpainting the static occluded background. The second challenge is approached with a conditional generative adversarial model that, taking as input the original dynamic image and the computed dynamic/static binary mask, is capable of generating the final static image. The former challenge is addressed by the use of a convolutional network that learns a multi-class semantic segmentation of the image. The objective of this network is producing an accurate segmentation and helping the previous generative model to output a realistic static image. These generated images can be used for applications such as virtual reality or vision-based robot localization purposes. To validate our approach, we show both qualitative and quantitative comparisons with other methods by removing the dynamic objects and hallucinating (inpainting) the static structure behind them.

Check some of our results on images coming from the driving simulator CARLA. Slide the blue button to see the input and output of our framework.

Here we post some of our results on real-world images. The two left images are from the CITYSCAPES dataset, and the two right images are from the SVS dataset.

Try Our Code

Torch: you can clone our GitHub repository here
PyTorch: we are currently working on this implementation

Check Out Our Video

Dataset

In our GitHub page we have some scripts available to generate the dataset with CARLA. Also, you can download a small dataset from here. Note that this dataset is valid for testing, but it contains very few images for training.

Here we show 4 examples of the training data format. These images are sourced from the CARLA driving simulator:

DatasetImage
DatasetImage
DatasetImage
DatasetImage

Paper

You can download our arXiv paper from HERE. When using this paper in your research, we will be happy if you cite us :)

@inproceedings{bescos2018empty,
  author = {Bescos, Berta and Neira, Jos{\'e} and Siegwart, Roland and Cadena, Cesar},
  title = {{Empty Cities: Image Inpainting for a Dynamic-Object-Invariant Space}},
  journal = {IEEE International Conference on Robotics and Automation (ICRA)},
  year = {2019}
}