• PDF / 918,203 Bytes
• 12 Pages / 439.642 x 666.49 pts Page_size
• 70 Downloads / 183 Views

DOWNLOAD

REPORT

---

Neural relational inference for disaster multimedia retrieval Samuel G. Fadel1

· Ricardo da S. Torres2

Received: 10 April 2019 / Revised: 22 June 2020 / Accepted: 26 June 2020 / © Springer Science+Business Media, LLC, part of Springer Nature 2020

Abstract Events around the world are increasingly documented on social media, especially by the people experiencing them, as these platforms become more popular over time. As a consequence, social media turns into a valuable source of data for understanding those events. Due to their destructive potential, natural disasters are among events of particular interest to response operations and environmental monitoring agencies. However, this amount of information also makes it challenging to identify relevant content pertaining to those events. In this paper, we use a relational neural network model for identifying this type of content. The model is particularly suitable for unstructured text, that is, text with no particular arrangement of words, such as tags, which is commonplace in social media data. In addition, our method can be combined with a CNN for handling multimodal data where text and visual data are available. We perform experiments in three different scenarios, where different modalities are evaluated: visual, textual, and both. Our method achieves competitive performance in both modalities by themselves, while significantly outperforms the baseline on the multimodal scenario. We also demonstrate the behavior of the proposed method in different applications by performing additional experiments in the CUB-200-2011 multimodal dataset. Keywords Machine learning · Neural networks · Information retrieval · Natural language processing · Multimodal

 Samuel G. Fadel

[email protected] Ricardo da S. Torres [email protected] 1

Institute of Computing, University of Campinas, Av. Albert Einstein, 1251, Campinas, 13083-852, Brazil

2

Department of ICT and Natural Sciences, Faculty of Information Technology and Electrical ˚ Engineering, NTNU - Norwegian University of Science and Technology, Alesund, Norway

Multimedia Tools and Applications

1 Introduction How to properly identify events of interest (e.g., natural disasters) based on multimodal data available? This is the research question that guides our research. In particular, we are interested in solutions that can effectively infer events based on complementary views provided by different modalities (e.g., visual properties and textual descriptions). This is still an open problem in several applications, such as those related to the proper management of social media multimedia collections [5]. The pervasiveness of social media has led many to use them as a means of communication and a source of information and news. Using social media, events around the world are documented not only by news outlets, but also by the people experiencing them. As a consequence, social media streams are becoming a valuable source of data for understanding, detailing, and assessing such events. Among those events are natu