Robust Activity Recognition for Adaptive Worker-Robot Interaction Using Transfer Learning
Publication: Computing in Civil Engineering 2023
ABSTRACT
Human activity recognition (HAR) using machine learning has shown tremendous promise in detecting construction workers’ activities. HAR has many applications in human-robot interaction research to enable robots’ understanding of human counterparts’ activities. However, many existing HAR approaches lack robustness, generalizability, and adaptability. This paper proposes a transfer learning methodology for activity recognition of construction workers that requires orders of magnitude less data and compute time for comparable or better classification accuracy. The developed algorithm transfers features from a model pre-trained by the original authors and fine-tunes them for the downstream task of activity recognition in construction. The model was pre-trained on Kinetics-400, a large-scale video-based human activity recognition dataset with 400 distinct classes. The model was fine-tuned and tested using videos captured from manual material handling (MMH) activities found on YouTube. Results indicate that the fine-tuned model can recognize distinct MMH tasks in a robust and adaptive manner, which is crucial for the widespread deployment of collaborative robots in construction.
Get full access to this article
View all available purchase options and get full access to this chapter.
REFERENCES
Construction Robotics. (2022). “Smart Lifting for Construction and Masonry.” Accessed May 12, 2023. https://www.construction-robotics.com/.
Escorcia, V., Dávila, M. A., Golparvar-Fard, M., and Niebles, J. C. (2012). Automated vision-based recognition of construction worker actions for building interior construction operations using RGBD cameras. Construction Research Congress 2012: Construction Challenges in a Flat World, 879–888.
Haurilet, M., Roitberg, A., Martinez, M., and Stiefelhagen, R. (2019). Wise—slide segmentation in the wild. 2019 International Conference on Document Analysis and Recognition (ICDAR), 343–348.
Kim, H., Kim, H., Hong, Y. W., and Byun, H. (2018). Detecting construction equipment using a region-based fully convolutional network and transfer learning. Journal of Computing in Civil Engineering, 32(2), 4017082.
Kim, K., and Cho, Y. K. (2020). Effective inertial sensor quantity and locations on a body for deep learning-based worker’s motion recognition. Automation in Construction, 113, 103126.
Kolar, Z., Chen, H., and Luo, X. (2018). Transfer learning and deep convolutional neural networks for safety guardrail detection in 2D images. Automation in Construction, 89, 58–70.
Liu, R., Ramli, A. A., Zhang, H., Henricson, E., and Liu, X. (2022). An overview of human activity recognition using wearable sensors: Healthcare and artificial intelligence. Internet of Things–ICIOT 2021: 6th International Conference, Held as Part of the Services Conference Federation, SCF 2021, Virtual Event, December 10–14, 2021, Proceedings, 1–14.
Luo, H., Xiong, C., Fang, W., Love, P. E. D., Zhang, B., and Ouyang, X. (2018). Convolutional neural networks: Computer vision-based workforce activity assessment in construction. Automation in Construction, 94, 282–289.
Ni, B., Peng, H., Chen, M., Zhang, S., Meng, G., Fu, J., Xiang, S., and Ling, H. (2022). Expanding language-image pretrained models for general video recognition. Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part IV, 1–18.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021). Learning transferable visual models from natural language supervision. International Conference on Machine Learning, 8748–8763.
Shen, J., Xiong, X., Li, Y., He, W., Li, P., and Zheng, X. (2021). Detecting safety helmet wearing on construction sites with bounding‐box regression and deep transfer learning. Computer‐Aided Civil and Infrastructure Engineering, 36(2), 180–196.
Sherafat, B., Ahn, C. R., Akhavian, R., Behzadan, A. H., Golparvar-Fard, M., Kim, H., Lee, Y.-C., Rashidi, A., and Azar, E. R. (2020). Automated methods for activity recognition of construction workers and equipment: State-of-the-art review. Journal of Construction Engineering and Management, 146(6), 3120002.
Zhang, S., Li, Y., Zhang, S., Shahabi, F., Xia, S., Deng, Y., and Alshurafa, N. (2022). Deep learning in human activity recognition with wearable sensors: A review on advances. Sensors, 22(4), 1476.
Zhang, S., Wei, Z., Nie, J., Huang, L., Wang, S., and Li, Z. (2017). A review on human activity recognition using vision-based method. Journal of Healthcare Engineering, 2017.
Information & Authors
Information
Published In
Computing in Civil Engineering 2023
Pages: 388 - 395
History
Published online: Jan 25, 2024
ASCE Technical Topics:
- Adaptive systems
- Artificial intelligence and machine learning
- Automation and robotics
- Business management
- Computer programming
- Computing in civil engineering
- Construction engineering
- Construction equipment
- Construction management
- Construction methods
- Employment
- Engineering fundamentals
- Equipment and machinery
- Human and behavioral factors
- Labor
- Personnel management
- Practice and Profession
- Systems engineering
- Systems management
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.