Automated 3D sign language caption generation for video
- PDF / 1,570,637 Bytes
- 14 Pages / 595.276 x 790.866 pts Page_size
- 5 Downloads / 260 Views
LONG PAPER
Automated 3D sign language caption generation for video Nayan Mehta1 · Suraj Pai2 · Sanjay Singh1
© Springer-Verlag GmbH Germany, part of Springer Nature 2019
Abstract Efforts to make online media accessible to a regional audience have picked up pace in recent years with multilingual captioning and keyboards. However, techniques to extend this access to people with hearing loss are limited. Further, owing to a lack of structure in the education of hearing impaired and regional differences, the issue of standardization of Indian Sign Language (ISL) has been left unaddressed, forcing educators to rely on the local language to support the ISL structure, thereby creating an array of correlations for each object, hindering the language building skills of a student. This paper aims to present a useful technology that can be used to leverage online resources and make them accessible to the hearingimpaired community in their primary mode of communication. Our tool presents an avenue for the early development of language learning and communication skills essential for the education of children with a profound hearing loss. With the proposed technology, we aim to provide a standardized teaching and learning medium to a classroom setting that can utilize and promote ISL. The goals of our proposed system involve reducing the burden of teachers to act as a valuable teaching aid. The system allows for easy translation of any online video and correlation with ISL captioning using a 3D cartoonish avatar aimed to reinforce classroom concepts during the critical period. First, the video gets converted to text via subtitles and speech processing methods. The generated text is understood through NLP algorithms and then mapped to avatar captions which are then rendered to form a cohesive video alongside the original content. We validated our results through a 6-month period and a consequent 2-month study, where we recorded a 37% and 70% increase in performance of students taught using Sign captioned videos against student taught with English captioned videos. We also recorded a 73.08% increase in vocabulary acquisition through signed aided videos. Keywords Indian Sign Language (ISL) · Profound and severe hearing loss · Bilingual education · Oralism · Total communication (TC)
1 Introduction
* Sanjay Singh [email protected] Nayan Mehta [email protected] Suraj Pai [email protected] 1
Department of Information and Communication Technology, Manipal Institute of Technology, MAHE, Manipal 576104, India
Department of Electronics and Communication Engineering, Manipal Institute of Technology, MAHE, Manipal 576104, India
2
India, a country with a population of 1.3 billion people, nearly a fifth of the world population [7], is estimated to have people with hearing loss of the order of 5 million [8]. According to the Government of India Disabled Persons Statistics Survey 2016 [35], 32.5% of this number is constituted of children. In the survey, for the age-group 5–9 years old, 209 of a sample set of
Data Loading...
