Deep multimodal habit tracking system: A user-adaptive approach for low-power embedded systems Déniz Cerpa, José Daniel Jimenez-Perera, Gabriel Nolasco, Ricardo Corral, Javier Barranco Expósito, Francisco Cyber-Physical System e-Health Multimodal Machine Learning User-adaptive Edge computing The pace of population ageing is increasing and is currently becoming one of the challenges our society faces. The introduction of Cyber-Physical Systems (CPS) has fostered the development of e-Health solutions that ease the associated economic and social burden. In this work, a CPS-based solution is presented to partially tackle the problem: a Deep Multimodal Habit Tracking system. The aim is to monitor daily life activities to alert in case of life-threatening situations improving their autonomy and supporting healthy lifestyles while living alone at home. Our approach combines video and heart rate cues to accurately identify indoor actions, running the processing locally in embedded edge nodes. Local processing provides inherent protection of data privacy since no image or vital signs are transmitted to the network, and reduces data bandwidth usage. Our solution achieves an accuracy of more than 80% in average, reaching up to a 95% for specific subjects after adapting the system. Although, the average F1-score improves by 2.4% compared to only using video information, the precision and recall for critical actions such as falls reaches up to 93.75%. Critical action detection is crucial due to their dramatic consequences, it helps to reduce false alarms, leading to building trust in the system and reducing economic cost. Also, the model is optimized and integrated in a Nvidia Jetson Nano embedded device, reaching real-time performance below 3.75 Watts. Finally, a dataset specifically designed for indoor action recognition using synchronized video and heart rate pulses has been collected. 2021-11-18T12:58:34Z 2021-11-18T12:58:34Z 2022 info:eu-repo/semantics/preprint http://hdl.handle.net/10481/71605 eng info:eu-repo/grantAgreement/H2020/783162 http://creativecommons.org/licenses/by-nc-nd/3.0/es/ info:eu-repo/semantics/openAccess Atribución-NoComercial-SinDerivadas 3.0 España