UnrealCV: Connecting Computer Vision to Unreal Engine
Computer graphics can not only generate synthetic images and ground truth but it also offers the possibility of constructing virtual worlds in which: (i) an agent can perceive, navigate, and take actions guided by AI algorithms, (ii) properties of the wor
- PDF / 759,064 Bytes
- 8 Pages / 439.37 x 666.142 pts Page_size
- 15 Downloads / 247 Views
Abstract. Computer graphics can not only generate synthetic images and ground truth but it also offers the possibility of constructing virtual worlds in which: (i) an agent can perceive, navigate, and take actions guided by AI algorithms, (ii) properties of the worlds can be modified (e.g., material and reflectance), (iii) physical simulations can be performed, and (iv) algorithms can be learnt and evaluated. But creating realistic virtual worlds is not easy. The game industry, however, has spent a lot of effort creating 3D worlds, which a player can interact with. So researchers can build on these resources to create virtual worlds, provided we can access and modify the internal data structures of the games. To enable this we created an open-source plugin UnrealCV (Project website: http://unrealcv.github.io) for a popular game engine Unreal Engine 4 (UE4). We show two applications: (i) a proof of concept image dataset, and (ii) linking Caffe with the virtual world to test deep network algorithms.
1
Introduction
Computer vision has benefited enormously from large datasets [7,8]. They enable the training and testing of complex models such as deep networks [13]. But performing annotation is costly and time consuming so it is attractive to make synthetic datasets which contain large amounts of images and detailed annotation. These datasets are created by modifying open-source movies [2] or by constructing a 3D world [9,17]. Researchers have shown that training on synthetic images is helpful for real world tasks [11,14,16,18,21]. Robotics researchers have gone further by constructing 3D worlds for robotics simulation, but they emphasize physical accuracy rather than visual realism. This motivates the design of realistic virtual worlds for computer vision where an agent can take actions guided by AI algorithms, properties of the worlds can be modified, physical simulations can be performed, and algorithms can be trained and tested. Virtual worlds have been used for autonomous driving [5], naive physics simulations [1] and evaluating surveillance system [19]. But creating realistic virtual worlds is time consuming. The video game industry has developed many tools for constructing 3D worlds, such as libraries of 3D object models. These 3D worlds are already realistic and the popularity of games and Virtual Reality (VR) drives towards even c Springer International Publishing Switzerland 2016 G. Hua and H. J´ egou (Eds.): ECCV 2016 Workshops, Part III, LNCS 9915, pp. 909–916, 2016. DOI: 10.1007/978-3-319-49409-8 75
910
W. Qiu and A. Yuille
greater realism. So modifying games and movies is an attractive way to make virtual worlds [5]. But modifying individual games is time-consuming and almost impossible for proprietary games. Hence our strategy is to modify a game engine, so that all the games built on top of it can be used. We develop a tool, UnrealCV, which can be used in combination with a leading game engine, Unreal Engine 4 (UE4), to use the rich resources in the game industry. UnrealCV can also be applied to 3D worl
Data Loading...
