Optimization of Multimodal Deep Learning for Depression Detection

Hermawan, Aditya and Daniawan, Benny and Edy, Edy and Nathaniel, Joese (2024) Optimization of Multimodal Deep Learning for Depression Detection. IJCCS (Indonesian Journal of Computing and Cybernetics Systems), 19 (4). pp. 383-394. ISSN 2460-7258

	Text Optimization of Multimodal Deep Learning for Depression Detection.pdf - Published Version Download (726kB)
	Text Turnitin - Optimization of Multimodal Deep Learning for Depression Detection.pdf - Published Version Download (672kB)

Abstract

Depression is a complex and often underdiagnosed mental health condition that manifests through subtle verbal, acoustic, and behavioral cues. Traditional unimodal detection systems struggle to capture the full spectrum of depressive symptoms, often leading to inaccurate or incomplete assessments. This study proposes a multimodal deep learning framework that integrates textual, audio, and visual modalities to improve the robustness and reliability of automatic depression detection, achieving an overall classification accuracy of 74%. The approach prioritizes privacy and interpretability by using facial keypoints and gaze direction rather than raw video frames, and applies attention mechanisms to align and fuse features across modalities. Each modality is processed through dedicated neural architectures tailored to its data type, and their outputs are combined within a fusion model that learns to capture cross-modal emotional patterns. Experimental results demonstrate that the proposed multimodal system significantly outperforms its unimodal counterparts in terms of classification performance. The visual modality was found to contribute most strongly to detection accuracy, as confirmed by ablation analysis. These findings highlight the value of multimodal integration in capturing complex psychological signals and support the development of intelligent, non-invasive screening tools for use in digital mental health applications.

Item Type:	Article
Uncontrolled Keywords:	Attention Mechanism, Depression Detection, Facial Keypoints, Mental Health Screening, Multimodal Deep Learning
Subjects:	000 Karya Umum > 006 Metode Komputer Tertentu > 006.242 Kode Bar > 006.3 Kecerdasan Buatan 000 Karya Umum > 006 Metode Komputer Tertentu > 006.3 Kecerdasan Buatan
Divisions:	Fakultas Sains & Teknologi > Sistem Informasi Fakultas Sains & Teknologi > Teknik Informatika Fakultas Sains & Teknologi > Teknik Perangkat Lunak
Depositing User:	Hariyanto Rie
Date Deposited:	29 Aug 2026 01:49
Last Modified:	29 Aug 2026 01:49
URI:	https://repositori.buddhidharma.ac.id//id/eprint/3461

Actions (login required)

View Item