Computer Vision-Based Automatic Emergency Notification System: Interpreting Construction Workers’ Hand Gestures
Publication: Computing in Civil Engineering 2023
ABSTRACT
With more than 900 fatalities and over 200,000 non-fatal injuries annually, construction is undoubtedly one of the most dangerous industries. While efforts have been made to develop autonomous safety surveillance systems, post-accident emergency response remains underexplored. This study addresses this gap by proposing a vision-based autonomous notification system using real-time data from multiple unmanned aerial vehicles (UAVs) at construction sites. The system identifies workers’ hand gestures, alerting a centralized command center with localization data when a distressed worker is detected. The system consists of two modules: (1) hand gesture recognition and interpretation and (2) localization module. A lightweight long short-term memory (LSTM) network was developed for gesture recognition, achieving a 94% accuracy rate in the test set.
Get full access to this article
View all available purchase options and get full access to this chapter.
REFERENCES
Amit, M. L., A. C. Fajardo, and R. P. Medina. 2022. “Recognition of Real-Time Hand Gestures Using Mediapipe Holistic Model and LSTM with MLP Architecture.” 2022 IEEE 10th Conference on Systems, Process and Control, ICSPC 2022 - Proceedings, 292–95. https://doi.org/10.1109/ICSPC55597.2022.10001800.
Dalal, N., and B. Triggs. 2005. “Histograms of Oriented Gradients for Human Detection.” Proceedings - 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005 I: 886–93. https://doi.org/10.1109/CVPR.2005.177.
Fang, W., B. Zhong, N. Zhao, P. E. D. Love, H. Luo, J. Xue, and S. Xu. 2019. “A Deep Learning-Based Approach for Mitigating Falls from Height with Computer Vision: Convolutional Neural Network.” Advanced Engineering Informatics. https://doi.org/10.1016/j.aei.2018.12.005.
Gheisari, M., and B. Esmaeili. 2016. “Unmanned Aerial Systems (UAS) for Construction Safety Applications.” Construction Research Congress 2016: Old and New Construction Technologies Converge in Historic San Juan - Proceedings of the 2016 Construction Research Congress, CRC 2016, 2642–50. https://doi.org/10.1061/9780784479827.263.
Hochreiter, S., and J. Schmidhuber. 1997. “Long Short-Term Memory.” Neural Computation 9 (8): 1735–80. https://doi.org/10.1162/NECO.1997.9.8.1735.
Huang, J., W. Zhou, Q. Zhang, H. Li, and W. Li. 2018. “Video-Based Sign Language Recognition Without Temporal Segmentation.” Proceedings of the AAAI Conference on Artificial Intelligence 32 (1): 2257–64. https://doi.org/10.1609/AAAI.V32I1.11903.
Jeelani, I., K. Asadi, H. Ramshankar, K. Han, and A. Albert. 2021. “Real-Time Vision-Based Worker Localization & Hazard Detection for Construction.” Automation in Construction. https://doi.org/10.1016/j.autcon.2020.103448.
Liao, Y., P. Xiong, W. Min, W. Min, and J. Lu. 2019. “Dynamic Sign Language Recognition Based on Video Sequence with BLSTM-3D Residual Networks.” IEEE Access 7: 38044–54. https://doi.org/10.1109/ACCESS.2019.2904749.
Miao, Q., Y. Li, W. Ouyang, Z. Ma, X. Xu, W. Shi, and X. Cao. 2017. “Multimodal Gesture Recognition Based on the ResC3D Network.” Proceedings - 2017 IEEE International Conference on Computer Vision Workshops, ICCVW 2017 2018-January (July): 3047–55. https://doi.org/10.1109/ICCVW.2017.360.
Mitra, S., and T. Acharya. 2007. “Gesture Recognition: A Survey.” IEEE Transactions on Systems, Man and Cybernetics Part C: Applications and Reviews 37 (3): 311–24. https://doi.org/10.1109/TSMCC.2007.893280.
Tran, D. S., N. H. Ho, H. J. Yang, E. T. Baek, S. H. Kim, and G. Lee. 2020. “Real-Time Hand Gesture Spotting and Recognition Using RGB-D Camera and 3D Convolutional Neural Network.” Applied Sciences 2020, Vol. 10, Page 722 10 (2): 722. https://doi.org/10.3390/APP10020722.
Wang, C., Z. Liu, and S. C. Chan. 2015. “Superpixel-Based Hand Gesture Recognition With Kinect Depth Camera.” IEEE Transactions on Multimedia 17 (1): 29–39. https://doi.org/10.1109/TMM.2014.2374357.
Wang, H., Q. Wang, M. Gao, P. Li, and W. Zuo. 2018. “Multi-Scale Location-Aware Kernel Representation for Object Detection.” https://github.com/.
Wang, H., A. Kläser, C. Schmid, and C. L. Liu. 2013. “Dense Trajectories and Motion Boundary Descriptors for Action Recognition.” International Journal of Computer Vision 103 (1): 60–79. https://doi.org/10.1007/S11263-012-0594-8/TABLES/7.
Wang, X., and Z. Zhu. 2021. “Vision-Based Hand Signal Recognition in Construction: A Feasibility Study.” Automation in Construction 125 (May): 103625. https://doi.org/10.1016/J.AUTCON.2021.103625.
Yasen, M., and S. Jusoh. 2019. “A Systematic Review on Hand Gesture Recognition Techniques, Challenges and Applications.” PeerJ. Computer Science 5 (9). https://doi.org/10.7717/PEERJ-CS.218.
Information & Authors
Information
Published In
Computing in Civil Engineering 2023
Pages: 469 - 475
History
Published online: Jan 25, 2024
ASCE Technical Topics:
- Automation and robotics
- Business management
- Computer vision and image processing
- Computing in civil engineering
- Construction engineering
- Construction management
- Construction sites
- Disaster preparedness
- Disaster risk management
- Emergency management
- Employment
- Engineering fundamentals
- Labor
- Methodology (by type)
- Occupational safety
- Personnel management
- Practice and Profession
- Public administration
- Public health and safety
- Safety
- Systems engineering
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.