Vision-Based Control of Construction Robots Using Transfer Learning Approach
Publication: Computing in Civil Engineering 2023
ABSTRACT
This study explores the potential of vision-based control of robots for construction applications using a transfer learning approach. Repetitive and arduous construction tasks cause musculoskeletal disorders in human workers which can be performed safely by robots. Unlike manufacturing processes, construction tasks are complex. Collaborative robots utilize human intelligence to adapt to the complexities of the construction site. Interaction between humans and robots is a challenge because of the limited infrastructure and computational training of construction workers. Vision-based hand gesture control is implemented using transfer learning by using a pre-trained hand detection model called Mediapipe Hands and using its output as features to train another artificial neural net (ANN) to classify different hand gestures. After optimizing the model parameters, 99.43% training and 99.11% validation accuracies were achieved. The results of validation performed through experiments in a simulated environment show the potential of using hand gestures for the control of construction robots.
Get full access to this article
View all available purchase options and get full access to this chapter.
REFERENCES
Adami, P., P. B. Rodrigues, P. J. Woods, B. Becerik-Gerber, L. Soibelman, Y. Copur-Gencturk, and G. Lucas. 2022. “Impact of VR-based training on human–robot interaction for remote operating construction robots.” Journal of Computing in Civil Engineering, 36 (3): 04022006. American Society of Civil Engineers.
Afsari, K., S. Gupta, M. Afkhamiaghda, and Z. Lu. 2018. “Applications of Collaborative Industrial Robots in Building Construction.” 54th ASC Annual International Conference Proceedings, 472–479.
Alonso-Martin, F., M. Malfaz, J. Sequeira, J. F. Gorostiza, and M. A. Salichs. 2013. “A multimodal emotion detection system during human--robot interaction.” Sensors, 13 (11): 15549–15581.
Al-Shamayleh, A. S., R. Ahmad, M. A. M. Abushariah, K. A. Alam, and N. Jomhari. 2018. “A systematic literature review on vision based gesture recognition techniques.” Multimedia Tools and Applications, 77 (21): 28121–28184. Springer New York LLC.
Brosque, C., E. Galbally, Y. Chen, R. Joshi, O. Khatib, and M. Fischer. 2021. “Collaborative Welding and Joint Sealing Robots With Haptic Feedback.” Proceedings of the 38th International Symposium on Automation and Robotics in Construction (ISARC), 1–8. Dubai, UAE: IAARC.
Brunete, A., M. Hernando, J. E. Torres, and E. Gambao. 2012. “Heterogeneous multi-configurable chained microrobot for the exploration of small cavities.” Automation in Construction, 21 (1): 184–198. Elsevier. https://doi.org/10.1016/j.autcon.2011.06.003.
Chen, L., M. Zhou, W. Su, M. Wu, J. She, and K. Hirota. 2018. “Softmax regression based deep sparse autoencoder network for facial emotion recognition in human-robot interaction.” Information Sciences, 428: 49–61. Elsevier.
Davila Delgado, J. M., L. Oyedele, A. Ajayi, L. Akanbi, O. Akinade, M. Bilal, and H. Owolabi. 2019. “Robotics and automated systems in construction: Understanding industry-specific challenges for adoption.” Journal of Building Engineering, 26: 100868.
Droeschel, D., J. Stückler, and S. Behnke. 2011. “Learning to interpret pointing gestures with a time-of-flight camera.” 2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 481–488.
Entzel, P., J. Albers, and L. Welch. 2007. “Best practices for preventing musculoskeletal disorders in masonry: Stakeholder perspectives.” Applied Ergonomics, 38 (5): 557–566. https://doi.org/10.1016/j.apergo.2006.08.004.
Géron, A. 2019. Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow.
Kolar, Z., H. Chen, and X. Luo. 2018. “Transfer learning and deep convolutional neural networks for safety guardrail detection in 2D images.” Automation in Construction, 89: 58–70. Elsevier.
Kucuksubasi, F., and A. G. Sorguc. 2018. “Transfer Learning-Based Crack Detection by Autonomous UAVs.” Proceedings of the 35th International Symposium on Automation and Robotics in Construction (ISARC), J. Teizer, ed., 593–600. Taipei, Taiwan: IAARC.
de La Bourdonnaye, F., R. Setchi, and C. Zanni-Merk. 2016. “Gaze Trajectory Prediction in the Context of Social Robotics.” IFAC-PapersOnLine, 49 (19): 126–131. Elsevier.
Lakshmi, D., and R. Ponnusamy. 2021. “Facial emotion recognition using modified HOG and LBP features with deep stacked autoencoders.” Microprocessors and Microsystems, 82: 103834. Elsevier. https://doi.org/10.1016/J.MICPRO.2021.103834.
Liu, F., B. Du, Q. Wang, Y. Wang, and W. Zeng. 2017. “Hand gesture recognition using kinect via deterministic learning.” 2017 29th Chinese Control and Decision Conference (CCDC), 2127–2132. IEEE.
Liu, Y., and H. Jebelli. 2022. “Human-Robot Co-Adaptation in Construction: Bio-Signal Based Control of Bricklaying Robots.” Computing in Civil Engineering 2021, 304–312. Orlando, Florida: American Society of Civil Engineers.
Mukherjee, D., K. Gupta, L. H. Chang, and H. Najjaran. 2022. “A Survey of Robot Learning Strategies for Human-Robot Collaboration in Industrial Settings.” Robotics and Computer-Integrated Manufacturing, 73: 102231. Pergamon. https://doi.org/10.1016/J.RCIM.2021.102231.
Nuzzi, C., S. Pasinetti, R. Pagani, S. Ghidini, M. Beschi, G. Coffetti, and G. Sansoni. 2021. “MEGURU: a gesture-based robot program builder for Meta-Collaborative workstations.” Robotics and Computer-Integrated Manufacturing, 68: 102085. Pergamon.
Onososen, A. O., and I. Musonda. 2022. “Research focus for construction robotics and human-robot teams towards resilience in construction: scientometric review.” Journal of Engineering, Design and Technology, ahead-of-print (ahead-of-print).
Pan, S. J., and Q. Yang. 2010. “A Survey on Transfer Learning.” IEEE Transactions on Knowledge and Data Engineering, 22 (10): 1345–1359. https://doi.org/10.1109/TKDE.2009.191.
Plappert, M., C. Mandery, and T. Asfour. 2018. “Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks.” Robotics and Autonomous Systems, 109: 13–26. North-Holland.
Puzatova, A., P. Shakor, V. Laghi, and M. Dmitrieva. 2022. “Large-Scale 3D Printing for Construction Application by Means of Robotic Arm and Gantry 3D Printer: A Review.” Buildings, 12 (11).
Rautaray, S. S., and A. Agrawal. 2015. “Vision based hand gesture recognition for human computer interaction: a survey.” Artificial intelligence review, 43 (1): 1–54. Springer.
Siciliano, B., and O. Khatib, eds. 2016. Springer handbook of robotics. Springer Berlin, Heidelberg.
Unity-Technologies. 2023. Unity Robotics Demos. C#. Unity Technologies.
Universal Robots. 2023. “UR3e collaborative robot arm that automates almost anything.” Universal Robots. Accessed March 6, 2023. https://www.universal-robots.com/products/ur3-robot/.
Vysocky, A., and P. Novak. 2016. “Human-Robot collaboration in industry.” MM Science Journal, 9 (2): 903–906.
Xia, Z., Q. Lei, Y. Yang, H. Zhang, Y. He, W. Wang, and M. Huang. 2019. “Vision-based hand gesture recognition for human-robot collaboration: A survey.” 2019 5th International Conference on Control, Automation and Robotics (ICCAR), 198–205. IEEE.
Xu, X., and B. G. de Soto. 2020. “On-site Autonomous Construction Robots: A review of Research Areas, Technologies, and Suggestions for Advancement.” ISARC Proceedings, 385–392. IAARC.
Zhang, F., V. Bazarevsky, A. Vakunov, A. Tkachenka, G. Sung, C.-L. Chang, and M. Grundmann. 2020. “MediaPipe Hands: On-device Real-time Hand Tracking.”.
Information & Authors
Information
Published In
Computing in Civil Engineering 2023
Pages: 657 - 665
History
Published online: Jan 25, 2024
ASCE Technical Topics:
- Automation and robotics
- Business management
- Construction engineering
- Construction industry
- Construction management
- Construction sites
- Education
- Employment
- Engineering fundamentals
- Human and behavioral factors
- Infrastructure construction
- Labor
- Occupational safety
- Personnel management
- Practice and Profession
- Public administration
- Public health and safety
- Safety
- Systems engineering
- Training
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.