DEEP LEARNING-BASED SIGN LANGUAGE INTERPRETATION FOR ENHANCED COMMUNICATION IN SPEECH-IMPAIRED INDIVIDUALS

Abstract

Sign language plays a crucial role in facilitating communication for individuals with speech or hearing impairments. However, traditional sign language interpretation relies on human translators, which can be costly and not always available. To address this challenge, this study proposes an AI-based sign language recognition system using the YOLOv11 model for real-time gesture detection. We trained YOLOv11 on a custom Pakistan Sign Language (PSL) dataset consisting of 13 gestures and a background class. Various preprocessing techniques, including data augmentation and normalization, were applied to enhance model performance. The model achieved 99.7% test accuracy, 98% F1-score, and 0.987 mAP@50, demonstrating its effectiveness in real-time sign recognition. While minor misclassifications occurred in similar gestures, the overall results showed strong reliability. To showcase real-world usability, we developed a desktop-based application using Tkinter and OpenCV, allowing real-time recognition through a webcam. Future improvements will focus on expanding the dataset, optimizing model performance for mobile devices, and integrating speech synthesis to enhance accessibility. These findings highlight the potential of deep learning in bridging communication gaps for the hearing-impaired, paving the way for more inclusive assistive technologies.