• PDF / 1,192,761 Bytes
• 25 Pages / 439.37 x 666.142 pts Page_size
• 61 Downloads / 227 Views

DOWNLOAD

REPORT

---

Enhancing image retrieval for complex queries using external knowledge sources Haitham Samih 1 & Sherine Rady 2

& Tarek F. Gharib

2

Received: 24 August 2019 / Revised: 12 June 2020 / Accepted: 13 July 2020 # Springer Science+Business Media, LLC, part of Springer Nature 2020

Abstract

Annotation-based image retrieval associates textual descriptions to images based on human perception. A user query, composed of keywords of choice and for retrieval, are usually matched lexically with the textual descriptions associated for stored images to extract the best matches. This paradigm will not produce appropriate desired results for complex queries if a semantic approach is not considered. This paper proposes an image retrieval framework which integrates external knowledge sources for obtaining a higher-level inference that can both handle complex queries and increase the number of relevant retrievals. The framework includes a parser where a semantic representation graph is initially generated from both image captions and query. The semantic representation of image captions is stored in the form of Resource Description Framework (RDF) triples, while the user query is translated into a SPARQL language query. For better query understanding, the external knowledge sources (ConceptNet, WordNet), are next fused together with the parser’s output in a significant process named query expansion to infer combined and expanded knowledge about the terms used in the query. Also, the expansion process generates a set of expansion rules to semantically expand the user query to adapt the inferred knowledge. The expanded query is matched against the stored RDF triplets to indicate the best matched image retrievals. Retrievals are eventually ranked using a relation similarity metric to obtain a ranked list of relevant images. Experimental studies carried on two Flickr datasets show that the proposed framework outperforms related work with 40% increase in the number of relevant retrievals at almost full accuracy. The framework achieves additionally an average increase for the accuracy at given k in the range of 50–72% for up to the tenth retrieval. Keywords Annotation-based image retrieval . Semantic search and retrieval . Commonsense knowledge . Knowledge inference . Query understanding . Query expansion

* Sherine Rady [email protected] Extended author information available on the last page of the article

Multimedia Tools and Applications

1 Introduction With the growing use of computers, mobile devices and other digital products, there have been a great number of digital images and videos generated. The huge increase in such digital content demands effective and efficient indexing and retrieval systems. Image retrieval is the science of finding images that fulfill a user specified need. Image retrieval systems are generally classified into two types: content-based image retrieval systems (CBIR) and annotation-based image retrieval systems. In CBIR systems, the retrieval is done based on matching low-level feature descriptors extra