Real-time system for place recognition by interpreting Sri Lankan sign language into text using machine learning approach.

Abstract

Sri Lankan sign language (SSL) serves as a vital visual-gestural communication system for the deaf community in Sri Lanka. However, effective communication between hearing-impaired individuals and the general population is limited due to challenges in understanding sign language. Further interpreting dynamic signs is more challenging due to the complexity involved in the sequence of unique expressions in SSL. To address this issue, a novel SSL to Sinhala text interpreting technology was developed, focusing specifically on dynamic signs associated with Sri Lankan locations. The research dataset encompassed a dynamic sign dataset comprised of 30 videos per category, each precisely divided into 30 frames. This robust dataset further strengthened the effectiveness of our approach in enhancing location-specific dynamic sign recognition. This study contributes to bridging a critical gap in Sri Lankan sign language recognition by assessing our model's performance on dynamic signs across three and five distinct locations. To recognise these dynamic gestures, a vision-based approach was chosen, providing a simpler and cost-effective solution compared to sensor-based systems. The study integrated Media Pipe and a Long short-term memory (LSTM) neural network as part of a combined methodology to enable gesture detection and interpretation. By leveraging these techniques, a camera-based, low-cost solution was successfully developed for interpreting SSL’s dynamic gestures. The study systematically tested multiple models, tuning LSTM and dense layers with varying neurons, resulting in an optimal model. Following rigorous 50-epoch training, our model exhibited an accuracy of 98.89% for dynamic signs across three distinct locations and an accuracy of 93.33% for dynamic signs across five locations. Cross-validation techniques were employed to assess the system’s performance and ensure its generalizability across different datasets. By validating the system through cross-validation, its robustness and reliability were tested, enabling a more accurate interpretation of SSL’s dynamic gestures. The proposed SSL to Sinhala text interpreting technology has the potential to significantly improve communication between hearing-impaired individuals and the general population in Sri Lanka. By leveraging vision-based methods and incorporating dynamic gesture recognition, this technology can enhance the accessibility and inclusivity of communication for the deaf community. Further research and enhancements are being carried out to expand the system’s capabilities for more place recognition and address the challenges associated with dynamic gestures and facial expressions in SSL recognition.