Approach for the Development of a Framework for the Identification of Activities of Daily Living Using Sensors in Mobile Devices
Abstract
:1. Introduction
2. Related Work
2.1. Sensors
2.2. Data Acquisition
2.3. Data Processing
2.3.1. Data Cleaning
2.3.2. Data Imputation
2.3.3. Feature Extraction
2.4. Data Fusion
2.5. Identification of Activities of Daily Living
2.5.1. Machine Learning
2.5.2. Pattern Recognition
2.6. Relation between the Identification of Activities of Daily Living and User Agenda
3. Methods and Expected Results
- Firstly, the ADL are recognized with motion and magnetic/mechanical sensors;
- Secondly, the identification of the environments is performed with acoustic sensors;
- Finally, there are two options, being these:
- ○
- The identification of standing activities with the fusion of the data acquired from motion and magnetic/mechanical sensors, and the environment recognized, where the number of ADL recognized depends on the number of sensors available;
- ○
- The identification of standing activities with the fusion of the data acquired from motion, magnetic/mechanical and location sensors, and the environment recognized, where the number of ADL recognized depends on the number of sensors available.
- MLP with Backpropagation;
- FNN with Backpropagation;
- DNN.
4. Discussion and Conclusions
Acknowledgments
Author Contributions
Conflicts of Interest
References
- Salazar, L.H.A.; Lacerda, T.; Nunes, J.V.; von Wangenheim, C.G. A Systematic Literature Review on Usability Heuristics for Mobile Phones. Int. J. Mob. Hum. Comput. Interact. 2013, 5, 50–61. [Google Scholar] [CrossRef]
- Foti, D.; Koketsu, J.S. Activities of daily living. In Pedretti’s Occupational Therapy: Practical Skills for Physical Dysfunction; Elsevier Health Sciences: Amsterdam, Netherlands, 2013; Volume 7, pp. 157–232. [Google Scholar]
- Garcia, N.M. A Roadmap to the Design of a Personal Digital Life Coach. In Advances in Intelligent Systems and Computing; Springer: Cham, Switzerland, 2016; Volume 399. [Google Scholar]
- Kleinberger, T.; Becker, M.; Ras, E.; Holzinger, A.; Müller, P. Ambient intelligence in assisted living: Enable elderly people to handle future interfaces. In Lecture Notes in Computer Science; Springer: Berlin, Germany, 2007; Volume 4555. [Google Scholar]
- Singh, D.; Kropf, J.; Hanke, S.; Holzinger, A. Ambient Assisted Living Technologies from the Perspectives of Older People and Professionals. In Lecture Notes in Computer Science; Springer: Cham, Switzerland, 2017; Volume 10410. [Google Scholar]
- Pires, I.M.; Garcia, N.M.; Pombo, N.; Flórez-Revuelta, F. From Data Acquisition to Data Fusion: A Comprehensive Review and a Roadmap for the Identification of Activities of Daily Living Using Mobile Devices. Sensors 2016, 16, 184. [Google Scholar] [CrossRef] [PubMed]
- Pires, I.M.; Garcia, N.M.; Pombo, N.; Flórez-Revuelta, F.; Zdravevski, E.; Spinsante, S. Machine Learning Algorithms for the Identification of Activities of Daily Living Using Mobile Devices: A Comprehensive Review. engr** for online temporal fusion in multisensor systems. Inf. Fusion 2008, 9, 370–388. [Google Scholar] [CrossRef]
- Singh, D.; Merdivan, E.; Psychoula, I.; Kropf, J.; Hanke, S.; Geist, M.; Holzinger, A. Human activity recognition using recurrent neural networks. In Proceedings of the International Cross-Domain Conference for Machine Learning and Knowledge Extraction, Reggio, Italy, 29 August–1 September 2017. [Google Scholar]
- Zhao, L.; Wu, P.; Cao, H. RBUKF Sensor Data Fusion for Localization of Unmanned Mobile Platform. Res. J. Appl. Sci. Eng. Technol. 2013, 6, 3462–3468. [Google Scholar] [CrossRef]
- Walter, O.; Schmalenstroeer, J.; Engler, A.; Haeb-Umbach, R. Smartphone-based sensor fusion for improved vehicular navigation. In Proceedings of the 2013 10th Workshop on Positioning Navigation and Communication (WPNC), Dresden, Germany, 20–21 March 2013. [Google Scholar]
- Grunerbl, A.; Muaremi, A.; Osmani, V.; Bahle, G.; Ohler, S.; Tröster, G.; Mayora, O.; Haring, C.; Lukowicz, P. Smart-Phone Based Recognition of States and State Changes in Bipolar Disorder Patients. IEEE J. Biomed. Health Inform. 2015, 15, 140–148. [Google Scholar] [CrossRef] [PubMed]
- Thatte, G.; Li, M.; Lee, S.; Emken, B.A.; Annavaram, M.; Narayanan, S.; Narayanan, D.; Mitra, U. Optimal Time-Resource Allocation for Energy-Efficient Physical Activity Detection. IEEE Trans. Signal Process 2011, 59, 1843–1857. [Google Scholar] [CrossRef] [PubMed]
- Bhuiyan, M.Z.H.; Kuusniemi, H.; Chen, L.; Pei, L.; Ruotsalainen, L.; Guinness, R.; Chen, R. Performance Evaluation of Multi-Sensor Fusion Models in Indoor Navigation. Eur. J. Navig. 2013, 11, 21–28. [Google Scholar]
- Bellos, C.; Papadopoulos, A.; Rosso, R.; Fotiadis, D.I. Heterogeneous data fusion and intelligent techniques embedded in a mobile application for real-time chronic disease management. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2011, 2011, 8303–8306. [Google Scholar] [PubMed]
- Ayachi, F.S.; Nguyen, H.; Goubault, E.; Boissy, P.; Duval, C. The use of empirical mode decomposition-based algorithm and inertial measurement units to auto-detect daily living activities of healthy adults. IEEE Trans. Neural Syst. Rehabilit. Eng. 2016, 24, 1060–1070. [Google Scholar] [CrossRef] [PubMed]
- Debes, C.; Merentitis, A.; Sukhanov, S.; Niessen, M.; Frangiadakis, N.; Bauer, A. Monitoring activities of daily living in smart homes: Understanding human behavior. IEEE Signal Process. Mag. 2016, 33, 81–94. [Google Scholar] [CrossRef]
- Koza, J.R.; Bennett, F.H.; Andre, D.; Keane, M.A. Automated design of both the topology and sizing of analog electrical circuits using genetic programming. In Artificial Intelligence in Design’96; Springer: Berlin, Germany, 1996; pp. 151–170. [Google Scholar]
- Russell, S.; Norvig, P.; Canny, C.F.; Malik, J.M.A. Artificial Intelligence: A Modern Approach; Prentice Hall: Upper Saddle River, NJ, USA, 1995. [Google Scholar]
- Du, K.-L.; Swamy, M.N.S. Fundamentals of Machine Learning. In Neural Networks and Statistical Learning; Springer: Berlin, Germany, 2014; pp. 15–65. [Google Scholar]
- Zhang, Y.; Rajapakse, J.C. Machine Learning in Bioinformatics; John Wiley & Sons: Hoboken, NJ, USA, 2009. [Google Scholar]
- Witten, I.H.; Frank, E.; Hall, M.A. Data Mining: Practical Machine Learning Tools and Techniques; Morgan Kaufmann: Burlington, MA, USA, 2016. [Google Scholar]
- Schapire, R.E. The boosting approach to machine learning: An overview. In Nonlinear Estimation and Classification; Springer: Berlin, Germany, 2003; pp. 149–171. [Google Scholar]
- Michalski, R.S.; Carbonell, J.G.; Mitchell, T.M.X. Machine Learning: An Artificial Intelligence Approach; Springer Science & Business Media: Berlin, Germany, 2013. [Google Scholar]
- Bishop, C.M. Pattern Recognition and Machine Learning; Springer: Berlin, Germany, 2006. [Google Scholar]
- Lorenzi, P.; Rao, R.; Romano, G.; Kita, A.; Irrera, F. Mobile Devices for the Real-Time Detection of Specific Human Motion Disorders. IEEE Sens. J. 2016, 16, 8220–8227. [Google Scholar]
- Lau, S.L.; König, I.; David, K.; Parandian, B.; Carius-Düssel, C.; Schultz, M. Supporting patient monitoring using activity recognition with a smartphone. In Proceedings of the 2010 7th International Symposium on Wireless Communication Systems (ISWCS), York, UK, 19–22 September 2010. [Google Scholar]
- Lau, S.L. Comparison of orientation-independent-based-independent-based movement recognition system using classification algorithms. In Proceedings of the 2013 IEEE Symposium on Wireless Technology and Applications (ISWTA), Kuching, Malaysia, 22–25 September 2013. [Google Scholar]
- Duarte, F.; Lourenco, A.; Abrantes, A. Activity classification using a smartphone. In Proceedings of the 2013 IEEE 15th International Conference on e-Health Networking, Applications & Services (Healthcom), Lisbon, Portugal, 9–12 October 2013. [Google Scholar]
- Fahim, M.; Lee, S.; Yoon, Y. SUPAR: Smartphone as a ubiquitous physical activity recognizer for u-healthcare services. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2014, 2014, 3666–3669. [Google Scholar] [PubMed]
- Bajpai, A.; Jilla, V.; Tiwari, V.N.; Venkatesan, S.M.; Narayanan, R. Quantifiable fitness tracking using wearable devices. In Proceedings of the 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milan, Italy, 25–29 August 2015. [Google Scholar]
- Nguyen, P.; Akiyama, T.; Ohashi, H.; Nakahara, G.; Yamasaki, K.; Hikaru, S. User-friendly Activity Recognition Using SVM Classifier and Informative Features. In Proceedings of the 2015 International Conference on Indoor Positioning and Indoor Navigation (Ipin), Banff, AB, Canada, 13–16 October 2015; pp. 1–8. [Google Scholar]
- Wang, C.; Xu, Y.; Zhang, J.; Yu, W. SW-HMM: A Method for Evaluating Confidence of Smartphone-Based Activity Recognition. In Proceedings of the 2016 IEEE Trustcom/BigDataSE/ISPA, Tian**, China, 23–26 August 2016. [Google Scholar]
- Lau, S.L.; David, K. Movement recognition using the accelerometer in smartphones. In Proceedings of the Future Network and Mobile Summit, Florence, Italy, 16–18 June 2010. [Google Scholar]
- Zhang, L.; Wu, X.; Luo, D. Real-Time Activity Recognition on Smartphones Using Deep Neural Networks. In Proceedings of the Ubiquitous Intelligence and Computing and 2015 IEEE 12th Intl Conf on Autonomic and Trusted Computing and 2015 IEEE 15th Intl Conf on Scalable Computing and Communications and Its Associated Workshops (UIC-ATC-ScalCom), Bei**g, China, 10–14 August 2015. [Google Scholar]
- Cardoso, N.; Madureira, J.; Pereira, N. Smartphone-based Transport Mode Detection for Elderly Care. In Proceedings of the IEEE 18th International Conference on E-Health Networking, Applications and Services (Healthcom), Munich, Germany, 14–16 September 2016; pp. 261–266. [Google Scholar]
- Vallabh, P.; Malekian, R.; Ye, N.; Bogatinoska, D.C. Fall Detection Using Machine Learning Algorithms. In Proceedings of the 24th International Conference on Software, Telecommunications and Computer Networks (SoftCOM), Split, Croatia, 22–24 September 2016; pp. 51–59. [Google Scholar]
- Filios, G.; Nikoletseas, S.; Pavlopoulou, C.; Rapti, M.; Ziegler, S. Hierarchical Algorithm for Daily Activity Recognition via Smartphone Sensors. In Proceedings of the 2015 IEEE 2nd World Forum on Internet of Things (WF-IOT), Milan, Italy, 14–16 December 2015; pp. 381–386. [Google Scholar]
- Tang, C.X.; Phoha, V.V. An Empirical Evaluation of Activities and Classifiers for User Identification on Smartphones. In Proceedings of the 2016 IEEE 8th International Conference on Biometrics Theory, Applications and Systems (BTAS), Niagara Falls, NY, USA, 6–9 September 2016; pp. 1–8. [Google Scholar]
- Li, P.; Wang, Y.; Tian, Y.; Zhou, T.S.; Li, J.S. An Automatic User-Adapted Physical Activity Classification Method Using Smartphones. IEEE Trans. Biomed. Eng. 2017, 64, 706–714. [Google Scholar] [CrossRef] [PubMed]
- Kim, Y.J.; Kang, B.N.; Kim, D. Hidden Markov Model Ensemble for Activity Recognition using Tri-axis Accelerometer. In Proceedings of the 2015 IEEE International Conference on Systems, Man, and Cybernetics (Smc 2015): Big Data Analytics for Human-Centric Systems, Kowloon, China, 9–12 October 2015; pp. 3036–3041. [Google Scholar]
- Brdiczka, O.; Bellotti, V. Identifying routine and telltale activity patterns in knowledge work. In Proceedings of the Fifth IEEE International Conference on Semantic Computing (ICSC), Palo Alto, CA, USA, 18–21 September 2011. [Google Scholar]
- Costa, Â.; Castillo, J.C.; Novais, P.; Fernández-Caballero, A.; Simoes, R. Sensor-driven agenda for intelligent home care of the elderly. Exp. Syst. Appl. 2012, 39, 12192–12204. [Google Scholar] [CrossRef]
Categories: | Sensors: | Availability |
---|---|---|
Motion sensors | Accelerometer Gyroscope | Always present Present in most models |
Magnetic/mechanical sensors | Magnetometer | Present in most models |
Location sensors | GPS | Always present |
Acoustic sensors | Microphone | Always present |
Force sensors | Touch screen | Always present |
Imaging/video sensors | Camera | Always present |
Methods: | Advantages: |
---|---|
ACQUA framework [17] | Controls of the order of the data acquisition; Controls the correct segments of the data requested; Controls the calibration of the data acquisition rates; Controls the packet sizes and radio characteristics; Controls the adaptation of the dynamic changes in query selective properties; Controls the support of multiple queries and heterogeneous time window semantics; Adapted for low processing, memory, and energy capabilities. |
Orchestrator framework [25] | Distributed execution of the data acquisition using several mobile devices; Adapted for low processing, memory, and energy capabilities. |
ErdOS framework [26] | Distributed execution of the data acquisition using several mobile devices; Adapted for low processing, memory, and energy capabilities. |
LittleRock prototype [27] | Adapted for low processing, memory, and energy capabilities. |
Jigsaw continuous sensing engine [28] | Controls the different sample rates; Adapted for low processing, memory, and energy capabilities. |
SociableSense framework [29] | Cloud-based framework; Needs a constant Internet connection; Adapted for low processing, memory, and energy capabilities. |
CHG technique [30] | Stores the sensory data in the smartphone memory; Adapted for low processing, and energy capabilities. |
BBQ framework [31] | Uses a multi-dimensional Gaussian probability density function from all sensors; Adapted for low processing, memory, and energy capabilities. |
Cursor movement algorithm [36] | Stores the sensory data in the smartphone memory; Adapted for low processing, and energy capabilities. |
No framework | Adapted for low processing, memory, and energy capabilities. |
Types of Sensors: | Data Cleaning Techniques: |
---|---|
Motion sensors; Magnetic/mechanical sensors. | Low-Pass Filter; High-Pass Filter; KALMAN Filter; Weighted moving average (WMA) algorithm; Moving average filter. |
Location sensors | The data cleaning technique is not important for this type of data acquired. |
Acoustic sensors | Moving average filter; Discrete Fourier Transform (DFT); Inverse Discrete Fourier Transform (IDFT); Fast Fourier Transform (FFT). |
Force sensors Imaging/video sensors | The data cleaning technique is not important for this type of data acquired. |
Types of Sensors: | Features: |
---|---|
Motion sensors; Magnetic/mechanical sensors. | Mean [67,70,71,72,73,74,75], average of peak frequency (APF) [66], maximum [66,70,71,73], minimum [66,70,71,73], standard deviation [66,67,70,71,72,73,74,75], Root Mean Square (RMS) [66,70], cross-axis signals correlation [66,67,69,73,76], skewness [67], kurtosis [67], average absolute deviation [67], slope [74], binned distribution [68], and zero crossing rate for each axis [69]; Mean [67,70,71,72,73,74,75], median [70,74], variance [70,71], maximum [66,70,71,73], minimum [66,70,71,73], standard deviation [66,67,70,71,72,73,74,75], Root Mean Square (RMS) [66,70], Fast Fourier Transform (FFT) spectral energy [70,76], frequency domain entropy [76], FFT coefficients [70,73], logarithm of FFT [76], Interquartile range [71,73], skewness [67], kurtosis [67], wavelet energy [73], and percentiles of MV [75]; Time between peaks [72], average of peak values [77], average of peak rising time [77], average of peak fall time [77], average time between peaks [77]. |
Location sensors | Distance between two points. |
Acoustic sensors | Average [78], Thresholding [78], Minimum [78], Maximum [78], Distance [78], MFCC (Mel-frequency cepstrum coefficients) [79,80]. |
Force sensors; Imaging/video sensors. | These sensors are not useful for the development of the framework for the Identification of ADL and their environments. |
Types of sensors: | Data fusion methods: |
---|---|
Motion sensors; Magnetic/mechanical sensors; Location sensors; Acoustic sensors. | Autoregressive-Correlated Gaussian Model; Fuzzy Logic; Dempster-Shafer; Evidence Theory; Recursive Operators; Support Vector Machine (SVM); Random Forests; Artificial Neural Networks (ANN); Decision Trees; Naïve Bayes classifier; Bayesian analysis; Kalman Filter; k-Nearest Neighbor (k-NN); Least squares-based estimation methods; Optimal Theory; Long Short Term Memory (LSTM) Recurrent Neural Networks (RNN); Uncertainty Ellipsoids. |
Force sensors; Imaging/video sensors. | These sensors are not useful for the development of the framework for the Identification of ADL and their environments. |
Types of Sensors: | Pattern Recognition Methods: | ADL Recognized: |
---|---|---|
Motion sensors; Magnetic/mechanical sensors; Location sensors; Acoustic sensors. | Support Vector Machines (SVM); Decision trees (J48, C4.5); Artificial Neural Networks (ANN); Probabilistic Neural Networks (PNN); Deep Neural Networks (DNN); Long Short Term Memory (LSTM) Recurrent Neural Networks (RNN); k-Nearest Neighbour (KNN); Naïve Bayes; Random Forest; Logistic Regression; Bayesian network; Sequential minimal optimization (SMO); Logistic Model Trees (LMT); Simple Logistic Logit Boost. | Walking; running; jogging; jum**; dancing; driving, cycling; sitting; standing; lying; walking on stairs; going up on an escalator; laying down; walking on a ramp. |
Support Vector Machines (SVM); Artificial Neural Networks (ANN); Probabilistic Neural Networks (PNN); Deep Neural Networks (DNN); Long Short Term Memory (LSTM) Recurrent Neural Networks (RNN); Hidden Markov model (HMM); Random Forest. | Cleaning; cooking; medication; swee**; washing hands; watering plants. | |
Hidden Markov model (HMM). | Walking; walking on stairs; standing; running; sitting; laying. | |
Force sensors; Imaging/video sensors. | These sensors are not useful for the development of the framework for the Identification of ADL and their environments. |
Accelerometer | Gyroscope | Magnetometer | Microphone | GPS | ||
---|---|---|---|---|---|---|
Activities | Going Downstairs | ✓ | ✓ | ✓ | ||
Going Upstairs | ✓ | ✓ | ✓ | |||
Running | ✓ | ✓ | ✓ | |||
Walking | ✓ | ✓ | ✓ | |||
Standing | ✓ | ✓ | ✓ | ✓ | ✓ | |
Slee** | ✓ | ✓ | ✓ | ✓ | ✓ | |
Driving | ✓ | ✓ | ✓ | ✓ | ✓ | |
Environments | Bar | ✓ | ||||
Classroom | ✓ | |||||
Gym | ✓ | |||||
Library | ✓ | |||||
Kitchen | ✓ | |||||
Street | ✓ | |||||
Hall | ✓ | |||||
Watching tv | ✓ | |||||
Bedroom | ✓ |
© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Pires, I.M.; Garcia, N.M.; Pombo, N.; Flórez-Revuelta, F.; Spinsante, S. Approach for the Development of a Framework for the Identification of Activities of Daily Living Using Sensors in Mobile Devices. Sensors 2018, 18, 640. https://doi.org/10.3390/s18020640
Pires IM, Garcia NM, Pombo N, Flórez-Revuelta F, Spinsante S. Approach for the Development of a Framework for the Identification of Activities of Daily Living Using Sensors in Mobile Devices. Sensors. 2018; 18(2):640. https://doi.org/10.3390/s18020640
Chicago/Turabian StylePires, Ivan Miguel, Nuno M. Garcia, Nuno Pombo, Francisco Flórez-Revuelta, and Susanna Spinsante. 2018. "Approach for the Development of a Framework for the Identification of Activities of Daily Living Using Sensors in Mobile Devices" Sensors 18, no. 2: 640. https://doi.org/10.3390/s18020640