Sign language generator for video platforms

Abstract

Over 5% of the world population (430 million people) suffer from hearing loss and deafness. Therefore, communication with them is a challenging task. One of the popular communication methods with deaf people is sign language. It is achieved by simultaneously combining hand gestures, body language, and facial expressions. Even though it is possible to communicate in written format, deaf people still find difficulties in reading natural language texts. Therefore, formal sign languages have been introduced to fill this gap. There are different standards for sign languages. For example, the USA has American Sign Language (ASL), UK has British Sign Language (BSL), and Sri Lanka has Sri Lanka Sign Language (SSL). However, modern video platforms do not have sign language support. This research introduces and evaluates the user experience of a customizable sign language converting extension for video platforms. The proposed machine translation model translates English sentences in videos into equivalent Sri Lanka Sign Language. Moreover, the position, size, and background colour of the 3D human animator are customizable. This system was evaluated with deaf people from different demographics. The user test was conducted as a questionnaire survey. The participants were deaf or hard-of-hearing under three categories (deaf, hard-of-hearing, and severely deaf) and belonged to different age groups (0-10 years,11-20 years, 21-30 years, 31-40 years, 41-50 years, 51-60 years, and above 60). Personal health information and accessibility barriers faced by each participant was questioned in the questionnaire. The user test was conducted by providing participants with three videos with the proposed extension, where participants should complete a series of tasks according to the provided guidelines. The video platform considered for the evaluation was YouTube. The usability issues of the proposed extension were recorded. In addition, new requirements requested by the participants were also recorded. According to the results, 88% of participants identified the correct and incorrect Sinhala sign language generators based on the accuracy of Sinhala sign language, 96% of participants identified the sign language generator as user-friendly. Participants found that the most critical features of this extension were adjusting background colour according to the video and the human animator.