Generative Visual Manipulation on the Natural Image Manifold
Realistic image manipulation is challenging because it requires modifying the image appearance in a user-controlled way, while preserving the realism of the result. Unless the user has considerable artistic skill, it is easy to “fall off” the manifold of
- PDF / 7,511,272 Bytes
- 17 Pages / 439.37 x 666.142 pts Page_size
- 71 Downloads / 201 Views
University of California, Berkeley, USA {junyanz,philkr,efros}@eecs.berkeley.edu 2 Adobe Research, San Jose, USA [email protected]
Abstract. Realistic image manipulation is challenging because it requires modifying the image appearance in a user-controlled way, while preserving the realism of the result. Unless the user has considerable artistic skill, it is easy to “fall off” the manifold of natural images while editing. In this paper, we propose to learn the natural image manifold directly from data using a generative adversarial neural network. We then define a class of image editing operations, and constrain their output to lie on that learned manifold at all times. The model automatically adjusts the output keeping all edits as realistic as possible. All our manipulations are expressed in terms of constrained optimization and are applied in near-real time. We evaluate our algorithm on the task of realistic photo manipulation of shape and color. The presented method can further be used for changing one image to look like the other, as well as generating novel imagery from scratch based on user’s scribbles.
1
Introduction
Today, visual communication is sadly one-sided. We all perceive information in the visual form (through photographs, paintings, sculpture, etc.), but only a chosen few are talented enough to effectively express themselves visually. This imbalance manifests itself even in the most mundane tasks. Consider an online shopping scenario: a user looking for shoes has found a pair that mostly suits her but she would like them to be a little taller, or wider, or in a different color. How can she communicate her preference to the shopping website? If the user is also an artist, then a few minutes with an image editing program will allow her to transform the shoe into what she wants, and then use image-based search to find it. However, for most of us, even a simple image manipulation in Photoshop presents insurmountable difficulties. One reason is the lack of “safety wheels” in image editing: any less-than-perfect edit immediately makes the image look completely unrealistic. To put another way, classic visual manipulation paradigm does not prevent the user from “falling off” the manifold of natural images. Understanding and modeling the natural image manifold has been a longstanding open research problem. But in the last two years, there has been rapid c Springer International Publishing AG 2016 B. Leibe et al. (Eds.): ECCV 2016, Part V, LNCS 9909, pp. 597–613, 2016. DOI: 10.1007/978-3-319-46454-1 36
598
J.-Y. Zhu et al.
(a) original photo
(e) different degree of image manipulation
Project
Edit Transfer (c) Editing UI
(b) projection on manifold
(d) smooth transition between the original and edited projection
Fig. 1. We use generative adversarial networks (GAN) [1, 2] to perform image editing on the natural image manifold. We first project an original photo (a) onto a lowdimensional latent vector representation (b) by regenerating it using GAN. We then modify the color and shape of the generated image
Data Loading...
