Pengenalan Kata Kolok Secara Real Time Menggunakan Mediapipe dan Algoritma Long Short-Term Memory (LSTM)
DOI:
https://doi.org/10.31598/sintechjournal.v8i2.1945Keywords:
Bahasa Isyarat Kolok, Tuna Rungu, Tuna Wicara, MediaPipe, Long Short-Term Memory (LSTM), Deep Learning.Abstract
Sign language is the primary form of communication for individuals who are deaf or mute. In the Bengkala community, there exists a unique sign language known as Kolok. Modern technology enables this local knowledge to be integrated into recognition systems. Building on this opportunity, the present study aims to develop a recognition system for Kolok Sign Language using the Long Short-Term Memory (LSTM) algorithm. The dataset was obtained directly from the Bengkala Village community, collected through two methods: manual recording using a smartphone camera and real-time frame capture with OpenCV assisted by a webcam. Ten vocabulary categories were utilized in a dataset comprising 300 time-series videos, each consisting of 30 frames. MediaPipe was employed to analyze each frame and extract keypoint coordinates of body and hand positions. The features used for modeling were the x, y, and z coordinates of body poses and hand landmarks. The extracted data were then processed using a two-layer LSTM architecture. The training results demonstrated highly consistent performance. At epochs 500 and 550, the model achieved accuracies of 0.91 and 0.93, along with F1-scores of 0.92 and 0.93, respectively. The highest performance was recorded at epoch 450, with a precision of 0.95, recall of 0.96, and an F1-score of 0.94.
References
[1] D. K. Jain, A. Kumar, and S. R. Sangwan, "TANA: The amalgam neural architecture for sarcasm detection in Indian indigenous language combining LSTM and SVM with word-emoji embeddings," Pattern Recognition Letters, vol. 160, pp. 11–18, Aug. 2022, doi: 10.1016/j.patrec.2022.05.026.
[2] Kementerian Koordinator Bidang Pembangunan Manusia dan Kebudayaan, "Pemerintah Penuhi Hak Penyandang Disabilitas di Indonesia," Kementerian Koordinator Bidang Pembangunan Manusia dan Kebudayaan. [Daring]. Tersedia: https://www.kemenkopmk.go.id/pemerintah-penuhi-hak-penyandang-disabilitas-di-indonesia. [Diakses: 24 Juni 2024].
[3] A. Aljabar and Suharjito, “Bisindo (Bahasa Isyarat Indonesia) Sign Language Recognition Using CNN and LSTM,” Advances in Science, Technology and Engineering Systems, vol. 5, no. 5, pp. 282–287, 2020, doi: 10.25046/aj050535.
[4] B. Natarajan et al., "Development of an End-to-End Deep Learning Framework for Sign Language Recognition, Translation, and Video Generation," in IEEE Access, vol 10, pp. 104358-104374, 2022, doi: 10.1109/ACCESS.2022.3210543.
[5] L. Arisandi and B. Satya, "Sistem klarifikasi bahasa isyarat Indonesia (Bisindo) dengan menggunakan algoritma convolutional neural network," Jurnal Sistem Cerdas, vol. 5, no. 3, pp. 135–146, 2022, doi: 10.37396/jsc.v5i3.262.
[6] N. H. A. E. and M. I. Zul, “Aplikasi Penerjemah Bahasa Isyarat Indonesia menjadi Suara berbasis Android menggunakan Tensorflow,” Jurnal Komputer Terapan, vol. 7, no. 1, pp. 74–83, 2021. doi: 10.35143/jkt.v7i1.4629.
[7] S. Ghosh, “Proposal of a Real-time American Sign Language Detector using MediaPipe and Recurrent Neural Network,” International Journal of Computer Sciences and Engineering, vol. 9, no. 7, pp. 46–52, 2021, doi: 10.26438/ijcse/v9i7.4652.
[8] F. Daniel Tanugraha, H. Pratikno, M. Musayanah, and W. Indah Kusumawati, “Pengenalan Gerakan Olahraga Berbasis (Long Short-Term Memory) Menggunakan Mediapipe,” Journal of Advances in Information and Industrial Technology, vol. 4, no. 1, pp. 37–45, 2022, doi: 10.52435/jaiit.v4i1.182.
[9] A. Nurmansyah, N. R. Rhamadhani, S. A. N. Hakim, S. A. Agustin, and S. Hamidah, "Permasalahan komunikasi yang kerap terjadi pada penyandang disabilitas," Jurnal Pendidikan, Bahasa dan Budaya, vol. 2, no. 2, pp. 200–210, 2023, doi: 10.55606/jpbb.v2i2.1515.
[10] A. Widiarti and H. Rahayu, “Pemberdayaan Penyandang Disabilitas Dalam Proses Pembangunan Di Indonesia Ditinjau Dari Perspektif Perundang-Undangan,” Jurnal Ilmu Hukum, vol. 3, no. 2, pp. 274–283, 2020, doi: 10.32493/rjih.v3i2.8095.
[11] A. Halder and A. Tayade, “Real-Time Vernacular Sign Language Recognition Using Mediapipe And Machine Learning,” International Journal of Research Publication and Reviews, vol. 2, pp. 9–17, 2021, doi: 10.13140/RG.2.2.32364.03203.
[12] F. M. Dewanto, H. A. Santoso, G. F. Shidik, and Purwanto, "Scoping Review of Sign Language Recognition: An Analysis of MediaPipe Framework and Deep Learning Integration," in Proc. IEEE Int. Seminar on Application for Technology of Information and Communication (iSemantic), Semarang, Indonesia, Sep. 21–22, 2024, pp. 451–458, doi: 10.1109/iSemantic63362.2024.10761983.
[13] B. Sundar and T. Bagyammal, "American sign language recognition for alphabets using MediaPipe and LSTM," Procedia Computer Science, vol. 215, pp. 642–651, 2022, doi: 10.1016/j.procs.2022.12.066.
[14] A. N. Ramdhon and F. Febriya, “Penerapan Face Recognition Pada Sistem Presensi,” Journal of Applied Computer Science and Technology, vol. 2, no. 1, pp. 12–17, 2021, doi: 10.52158/jacost.v2i1.121.
[15] S. B. Abdullahi and K. Chamnongthai, "American Sign Language words recognition using spatio-temporal prosodic and angle features: A sequential learning approach," IEEE Access, vol. 10, pp. 15911–15923, 2022, doi: 10.1109/ACCESS.2022.3148132.
[16] J. Huang, J. Chaijaruwanich, and V. Chouvatut, "Video-based sign language recognition with R(2+1)D and LSTM networks," in Proc. IEEE Int. Conf. on Knowledge and Smart Technology (KST), Krabi, Thailand, Feb. 28–Mar. 2, 2024, pp. 214–219, doi: 10.1109/KST61284.2024.10499646.
[17] I. Suyudi, S. Sudadio, and S. Suherman, “Pengenalan Bahasa Isyarat Indonesia Menggunakan Mediapipe Dengan Model Random Forest Dan Multinomial Logistic Regression,” Jurnal Ilmu Siber dan Teknologi Digital (JISTED), vol. 1, no. 1, pp. 65–80, 2022, doi: 10.35912/jisted.v1i1.1899.
[18] L. Wiranda, M. Sadikin, and Jurusan Teknik Informatika, Fakultas Ilmu Komputer, Universitas Mercu Buana, “Penerapan Long Short Term Memory pada Data Time Series untuk Memprediksi Penjualan Produk PT. Metiska Farma,” Jurnal Nasional Pendidikan Teknik Informatika, no. 3, p. 184, 2019, doi: 10.23887/janapati.v8i3.19139.
[19] W. A. F. B. Nordin, R. Alfred, C. P. Yee, S. H. Tanalol, R. V. Loudin, and Z. Iswandono, "Malay fake news classification using a deep learning approach," in Proc. 9th Int. Conf. on Computational Science and Technology (ICCST 2022), Lecture Notes in Electrical Engineering, vol. 983, D. K. Kang, R. Alfred, Z. I. B. A. Ismail, A. Baharum, and V. Thiruchelvam, Eds. Singapore: Springer, 2023, doi: 10.1007/978-981-19-8406-8_2.
[20] H. M. Putri, Fadlisyah, and W. Fuadi, "Pendeteksian Bahasa Isyarat Indonesia secara Real-Time Menggunakan Long Short-Term Memory (LSTM)," Skripsi, Fakultas Teknik, Universitas Malikussaleh, 2021, doi: 10.29103/tts.v3i1.6853
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 I Made Panji Prayoga, I Gusti Ayu Agung Diatri Indradewi, Putu Hendra Suputra
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
1.png)
