Prediction-Based Path Planning for Safe and Efficient Human–Robot Collaboration in Construction via Deep Reinforcement Learning
Publication: Journal of Computing in Civil Engineering
Volume 37, Issue 1
Abstract
Robotics has attracted broad attention as an emerging technology in construction to help workers with repetitive, physically demanding, and dangerous tasks, thus improving productivity and safety. Under the new era of human–robot coexistence and collaboration in dynamic and complex workspaces, it is critical for robots to navigate to the targets efficiently without colliding with moving workers. This study proposes a new deep reinforcement learning (DRL)–based robot path planning method that integrates the predicted movements of construction workers to achieve safe and efficient human–robot collaboration in construction. First, an uncertainty-aware long short-term memory network is developed to predict the movements of construction workers and associated uncertainties. Second, a DRL framework is formulated, where predicted movements of construction workers are innovatively integrated into the state space and the computation of the reward function. By incorporating predicted trajectories in addition to current locations, the proposed method enables proactive planning such that the robot could better adapt to human movements, thus ensuring both safety and efficiency. The proposed method was demonstrated and evaluated using simulations generated based on real construction scenarios. The results show that prediction-based DRL path planning achieved a 100% success rate (with a total of 10,000 episodes) for robots to achieve the destination along the near-shortest path. Furthermore, it reduced the collision rate with moving workers by 23% compared with the conventional DRL method, which does not consider predicted information.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request, including (1) data and codes for generating DRL environment and training and testing DRL path planning model, and (2) codes for training and testing the LSTM-based trajectory prediction model.
Acknowledgments
This research was partially funded by the University of Texas at San Antonio (UTSA), Office of the Vice President for Research, Economic Development, and Knowledge Enterprise, and the US National Science Foundation (NSF) via Grant Nos. 2138514 and 2129003. The authors gratefully acknowledge UTSA’s and NSF’s supports.
References
Ajeil, F. H., I. K. Ibraheem, A. T. Azar, and A. J. Humaidi. 2020. “Grid-based mobile robot path planning using aging-based ant colony optimization algorithm in static and dynamic environments.” Sensors (Switzerland) 20 (7): 1880. https://doi.org/10.3390/s20071880.
Alahi, A., K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese. 2016. “Social LSTM: Human trajectory prediction in crowded spaces.” In Proc., IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, 961–971. Manhattan, NY: IEEE. https://doi.org/10.1109/CVPR.2016.110.
Asadi, E., B. Li, and I. M. Chen. 2018. “Pictobot: A cooperative painting robot for interior finishing of industrial developments.” IEEE Rob. Autom. Mag. 25 (2): 82–94. https://doi.org/10.1109/MRA.2018.2816972.
Asadi, K., V. R. Haritsa, K. Han, and J.-P. Ore. 2021. “Automated object manipulation using vision-based mobile robotic system for construction applications.” J. Comput. Civ. Eng. 35 (1): 04020058. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000946.
Asadi, K., A. Kalkunte Suresh, A. Ender, S. Gotad, S. Maniyar, S. Anand, M. Noghabaei, K. Han, E. Lobaton, and T. Wu. 2020. “An integrated UGV-UAV system for construction site data collection.” Autom. Constr. 112 (Apr): 103068. https://doi.org/10.1016/j.autcon.2019.103068.
ASI (Autonomous Solution, Inc). 2019. “Robotic excavators.” Accessed October 12, 2019. https://www.asirobots.com/mining/excavator/.
Associated Builders and Contractors. 2021. “Construction spending and employment: History and forecast terms and sources.” Accessed June 2, 2021. https://www.abc.org/Portals/1/NewsReleases/explainer-2021.pdf?ver=2021-03-23-162431-727.
Botteghi, N., B. Sirmacek, K. A. A. Mustafa, M. Poel, and S. Stramigioli. 2020. “On reward shaping for mobile robot navigation: A reinforcement learning and SLAM based approach.” Preprint, submitted February 10, 2020. http://arxiv.org/abs/2002.04109.
Cai, J., and H. Cai. 2020. “Robust hybrid approach of vision-based tracking and radio-based identification and localization for 3D tracking of multiple construction workers.” J. Comput. Civ. Eng. 34 (4): 04020021. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000901.
Cai, J., A. Du, and S. Li. 2021. “Prediction-enabled collision risk estimation for safe human-robot collaboration on unstructured and dynamic construction sites.” In Computing in civil engineering 2021, 34–41. Reston, VA: ASCE.
Cai, J., Y. Zhang, L. Yang, H. Cai, and S. Li. 2020. “A context-augmented deep learning approach for worker trajectory prediction on unstructured and dynamic construction sites.” Adv. Eng. Inf. 46 (Oct): 101173. https://doi.org/10.1016/j.aei.2020.101173.
Cardno, C. A. 2018. “Robotic rebar-tying system uses artificial intelligence.” Civ. Eng. Mag. Arch. 88 (1): 38–39. https://doi.org/10.1061/ciegag.0001260.
Center for Construction Research and Training. 2018. “The construction chart book.” Accessed June 2, 2022. https://www.cpwr.com/wp-content/uploads/publications/5th-Edition-Chart-Book-Final.pdf.
Changali, S., A. Mohammad, and M. V. Nieuwland. 2015. “The construction productivity imperative.” Accessed June 14, 2021. https://www.mckinsey.com/business-functions/operations/our-insights/the-construction-productivity-imperative.
Chen, Z., K. Chen, C. Song, X. Zhang, J. C. P. Cheng, and D. Li. 2022. “Global path planning based on BIM and physics engine for UGVs in indoor environments.” Autom. Constr. 139 (Jul): 104263. https://doi.org/10.1016/j.autcon.2022.104263.
Dijkstra, E. W. 1959. “A note on two problems in connexion with graphs.” Numer. Math. 1 (1): 269–271. https://doi.org/10.1007/BF01386390.
Dong, C., H. Li, X. Luo, L. Ding, J. Siebert, and H. Luo. 2018. “Proactive struck-by risk detection with movement patterns and randomness.” Autom. Constr. 91: 246–255. https://doi.org/10.1016/j.autcon.2018.03.021.
Dong, X. S., E. Betit, A. M. Dale, G. Barlet, and Q. Wei. 2019. “Trends of musculoskeletal disorders and interventions in the construction industry.” Accessed June 2, 2022. https://www.cpwr.com/wp-content/uploads/2020/06/Quarter3-QDR-2019.pdf.
Du, A., and A. Ghavidel. 2022. “Parameterized deep reinforcement learning-enabled maintenance decision-support and life-cycle risk assessment for highway bridge portfolios.” Struct. Saf. 97 (Jul): 102221. https://doi.org/10.1016/j.strusafe.2022.102221.
Dusty Robotics. 2022. “Build better with BIM-Drien layout.” Accessed June 2, 2022. https://www.dustyrobotics.com/.
Freimuth, H., and M. König. 2018. “Planning and executing construction inspections with unmanned aerial vehicles.” Autom. Constr. 96 (Dec): 540–553. https://doi.org/10.1016/j.autcon.2018.10.016.
Fridovich-Keil, D., A. Bajcsy, J. F. Fisac, S. L. Herbert, S. Wang, A. D. Dragan, and C. J. Tomlin. 2020. “Confidence-aware motion prediction for real-time collision avoidance.” Int. J. Rob. Res. 39 (2–3): 250–265. https://doi.org/10.1177/0278364919859436.
Gao, J., W. Ye, J. Guo, and Z. Li. 2020. “Deep reinforcement learning for indoor mobile robot path planning.” Sensors (Switzerland) 20 (19): 1–15. https://doi.org/10.3390/s20195493.
Ge, S. S., and Y. J. Cui. 2002. “Dynamic motion planning for mobile robots using potential field method.” Auton. Rob. 13 (3): 207–222. https://doi.org/10.1023/A:1020564024509.
Gong, D., Y. Wang, J. Yu, and G. Zuo. 2019. “Motion mapping from a human arm to a heterogeneous excavator-like robotic arm for intuitive teleoperation.” In Proc., 2019 IEEE Int. Conf. on Real-Time Computing and Robotics, RCAR 2019, 493–498. Manhattan, NY: IEEE. https://doi.org/10.1109/RCAR47638.2019.9044131.
Hart, P. E., N. J. Nilsson, and B. Raphael. 1968. “A formal basis for the heuristic determination of minimum cost paths.” IEEE Trans. Syst. Sci. Cybern. 4 (2): 100–107. https://doi.org/10.1109/TSSC.1968.300136.
Hartley, R., and A. Zisserman. 2003. Multiple view geometry in computer vision. Cambridge, UK: Cambridge University Press.
Hospital Construction. 2019. “Hospital construction.” Accessed April 7, 2019. https://www.youtube.com/channel/UCEKwrM78pRv8WRcKvZNtE1w.
Hu, D., S. Li, J. Cai, and Y. Hu. 2020. “Toward intelligent workplace: Prediction-enabled proactive planning for human-robot coexistence on unstructured construction sites.” In Proc., Winter Simulation Conf. 2020, 2412–2423. New York: IEEE. https://doi.org/10.1109/WSC48552.2020.9384077.
Jacob-Loyola, N., F. Muñoz-La Rivera, R. F. Herrera, and E. Atencio. 2021. “Unmanned aerial vehicles (UAVs) for physical progress monitoring of construction.” Sensors 21 (12): 4227. https://doi.org/10.3390/s21124227.
Jeong, I., Y. Jang, J. Park, and Y. K. Cho. 2021. “Motion planning of mobile robots for autonomous navigation on uneven ground surfaces.” J. Comput. Civ. Eng. 35 (3): 04021001. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000963.
Kayhani, N., H. Taghaddos, and S. BehzadiPour. 2018. “Construction equipment collision-free path planning using robotic approach.” In Proc., 35th Int. Symp. on Automation and Robotics in Construction and International AEC/FM Hackathon: The Future of Building Things: ISARC 2018. London: International Association for Automation and Robotics in Construction. https://doi.org/10.22260/isarc2018/0169.
Khasawneh, A., H. Rogers, J. Bertrand, K. C. Madathil, and A. Gramopadhye. 2019. “Human adaptation to latency in teleoperated multi-robot human-agent search and rescue teams.” Autom. Constr. 99 (Mar): 265–277. https://doi.org/10.1016/j.autcon.2018.12.012.
Kim, P., J. Park, Y. K. Cho, and J. Kang. 2019. “UAV-assisted autonomous mobile robot navigation for as-is 3D data collection and registration in cluttered environments.” Autom. Constr. 106 (Oct): 102918. https://doi.org/10.1016/j.autcon.2019.102918.
Kim, S., M. Peavy, P. C. Huang, and K. Kim. 2021. “Development of BIM-integrated construction robot task planning and simulation system.” Autom. Constr. 127 (Jul): 103720. https://doi.org/10.1016/j.autcon.2021.103720.
Kim, S. K., J. S. Russell, and K. J. Koo. 2003. “Construction robot path-planning for earthwork operations.” J. Comput. Civ. Eng. 17 (2): 97–104. https://doi.org/10.1061/(ASCE)0887-3801(2003)17:2(97).
Kingma, D. P., and J. L. Ba. 2014. “Adam: A method for stochastic optimization.” Preprint, submitted December 22, 2014. http://arxiv.org/abs/1412.6980.
Lavalle, S. M. 1998. “Rapidly-exploring random trees: A new tool for path planning.” Accessed June 2, 2022. https://www.cs.csustan.edu/∼xliang/Courses/CS4710-21S/Papers/06RRT.pdf.
Lillicrap, T. P., J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. 2015. “Continuous control with deep reinforcement learning.” Preprint, submitted September 9, 2015. http://arxiv.org/abs/1509.02971.
Liu, Y., M. Habibnezhad, and H. Jebelli. 2021a. “Brain-computer interface for hands-free teleoperation of construction robots.” Autom. Constr. 123 (Mar): 103523. https://doi.org/10.1016/j.autcon.2020.103523.
Liu, Y., M. Habibnezhad, and H. Jebelli. 2021b. “Brainwave-driven human-robot collaboration in construction.” Autom. Constr. 124 (Apr): 103556. https://doi.org/10.1016/j.autcon.2021.103556.
Lu, T., S. Tervola, X. Lü, C. J. Kibert, Q. Zhang, T. Li, and Z. Yao. 2021. “A novel methodology for the path alignment of visual SLAM in indoor construction inspection.” Autom. Constr. 127 (Jul): 103723. https://doi.org/10.1016/j.autcon.2021.103723.
Madsen, A. J. 2019. “The SAM100: Analyzing labor productivity.” Accessed July 5, 2022. https://digitalcommons.calpoly.edu/cmsp/243/.
Min, H. Q., J. H. Zhu, and X. J. Zheng. 2005. “Obstacle avoidance with multi-objective optimization by PSO in dynamic environment.” In Proc., 2005 Int. Conf. on Machine Learning and Cybernetics, ICMLC 2005, 2950–2956. Manhattan, NY: IEEE. https://doi.org/10.1109/icmlc.2005.1527447.
Mnih, V., A. P. Badia, L. Mirza, A. Graves, T. Harley, T. P. Lillicrap, D. Silver, and K. Kavukcuoglu. 2016. “Asynchronous methods for deep reinforcement learning.” In Proc., 33rd Int. Conf. on Machine Learning, ICML 2016, 2850–2869. Norfolk, MA: Journal of Machine Learning Research.
Mnih, V., K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. 2013. “Playing Atari with deep reinforcement learning.” Preprint, submitted December 19, 2013. http://arxiv.org/abs/1312.5602.
Narazaki, Y., V. Hoskere, G. Chowdhary, and B. F. Spencer. 2022. “Vision-based navigation planning for autonomous post-earthquake inspection of reinforced concrete railway viaducts using unmanned aerial vehicles.” Autom. Constr. 137 (May): 104214. https://doi.org/10.1016/j.autcon.2022.104214.
Okishiba, S., R. Fukui, M. Takagi, H. Azumi, S. Warisawa, R. Togashi, H. Kitaoka, and T. Ooi. 2019. “Tablet interface for direct vision teleoperation of an excavator for urban construction work.” Autom. Constr. 102 (Jun): 17–26. https://doi.org/10.1016/j.autcon.2019.02.003.
OSHA (Occupational Safety and Health Administration). 2020. “Commonly used statistics.” Accessed March 25, 2020. https://www.osha.gov/oshstats/commonstats.html.
Roberts, D., and M. Golparvar-Fard. 2019. “End-to-end vision-based detection, tracking and activity analysis of earthmoving equipment filmed at ground level.” Autom. Constr. 105 (Sep): 102811. https://doi.org/10.1016/j.autcon.2019.04.006.
Stentz, A. 1994. “Optimal and efficient path planning for partially-known environments.” In Proc., IEEE Int. Conf. on Robotics and Automation, 3310–3317. New York: IEEE. https://doi.org/10.1007/978-1-4615-6325-9_11.
Tsuruta, T., K. Miura, and M. Miyaguchi. 2019. “Mobile robot for marking free access floors at construction sites.” Autom. Constr. 107 (Nov): 102912. https://doi.org/10.1016/j.autcon.2019.102912.
Wang, X., C.-J. Liang, C. C. Menassa, and V. R. Kamat. 2021. “Interactive and immersive process-level digital twin for collaborative human–robot construction work.” J. Comput. Civ. Eng. 35 (6): 04021023. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000988.
Wang, Z., H. Li, and X. Yang. 2020. “Vision-based robotic system for on-site construction and demolition waste sorting and recycling.” J. Build. Eng. 32 (Nov): 101769. https://doi.org/10.1016/j.jobe.2020.101769.
Wen, S., Y. Zhao, X. Yuan, Z. Wang, D. Zhang, and L. Manfredi. 2020. “Path planning for active SLAM based on deep reinforcement learning under unknown environments.” Intell. Serv. Rob. 13 (2): 263–272. https://doi.org/10.1007/s11370-019-00310-w.
Xiao, X., B. Liu, G. Warnell, and P. Stone. 2022. “Motion planning and control for mobile robot navigation using machine learning: A survey.” Auton. Rob. 46 (6): 569–597. https://doi.org/10.1007/s10514-022-10039-8.
Xie, L., S. Wang, A. Markham, and N. Trigoni. 2017. “Towards monocular vision based obstacle avoidance through deep reinforcement learning.” Preprint, submitted June 29, 2017. http://arxiv.org/abs/1706.09829.
Xu, H., N. Wang, H. Zhao, and Z. Zheng. 2019. “Deep reinforcement learning-based path planning of underactuated surface vessels.” Cyber-Phys. Syst. 5 (1): 1–17. https://doi.org/10.1080/23335777.2018.1540018.
Yan, N., S. Huang, and C. Kong. 2021. “Reinforcement learning-based autonomous navigation and obstacle avoidance for USVs under partially observable conditions.” Math. Probl. Eng. 2021: 1–13. https://doi.org/10.1155/2021/5519033.
Zhang, H. Y., W. M. Lin, and A. X. Chen. 2018. “Path planning for the mobile robot: A review.” Symmetry 10 (10): 450. https://doi.org/10.3390/sym10100450.
Zhang, S., J. Teizer, N. Pradhananga, and C. M. Eastman. 2015. “Workforce location tracking to model, visualize and analyze workspace requirements in building information models for construction safety planning.” Autom. Constr. 60: 74–86. https://doi.org/10.1016/j.autcon.2015.09.009.
Zhang, X., C. Wang, Y. Liu, and X. Chen. 2019. “Decision-making for the autonomous navigation of maritime autonomous surface ships based on scene division and deep reinforcement learning.” Sensors (Switzerland) 19 (18): 4055. https://doi.org/10.3390/s19184055.
Zhou, T., Q. Zhu, Y. Shi, and J. Du. 2022. “Construction robot teleoperation safeguard based on real-time human hand motion prediction.” J. Constr. Eng. Manage. 148 (7): 04022040. https://doi.org/10.1061/(ASCE)CO.1943-7862.0002289.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 37 • Issue 1 • January 2023
Copyright
© 2022 American Society of Civil Engineers.
History
Received: Apr 2, 2022
Accepted: Jul 14, 2022
Published online: Oct 6, 2022
Published in print: Jan 1, 2023
Discussion open until: Mar 6, 2023
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Guoliang Shi, Dechun Lu, Zhansheng Liu, Xiuli Du, Qingwen Zhang, Linlin Zhao, Zeqiang Wang, Material Distribution Planning Method and Experimental Verification under Multinode and Multivehicle Scene, Journal of Construction Engineering and Management, 10.1061/JCEMD4.COENG-15129, 150, 11, (2024).
- Xiaoyun Liang, Usman Rasheed, Jiannan Cai, Bastian Wibranek, Ibukun Awolusi, Impacts of Collaborative Robots on Construction Work Performance and Worker Perception: Experimental Analysis of Human–Robot Collaborative Wood Assembly, Journal of Construction Engineering and Management, 10.1061/JCEMD4.COENG-14390, 150, 8, (2024).
- Hongrui Yu, Vineet R. Kamat, Carol C. Menassa, Cloud-Based Hierarchical Imitation Learning for Scalable Transfer of Construction Skills from Human Workers to Assisting Robots, Journal of Computing in Civil Engineering, 10.1061/JCCEE5.CPENG-5731, 38, 4, (2024).
- Somin Park, Carol C. Menassa, Vineet R. Kamat, Joint BERT Model for Intent Classification and Slot Filling Analysis of Natural Language Instructions in Co-Robotic Field Construction Work, Computing in Civil Engineering 2023, 10.1061/9780784485224.055, (453-460), (2024).
- Corey Hannum, Rui Li, Weitian Wang, A Trust-Assist Framework for Human–Robot Co-Carry Tasks, Robotics, 10.3390/robotics12020030, 12, 2, (30), (2023).
- Jack C. P. Cheng, Changhao Song, Xiao Zhang, Zhengyi Chen, Pose Graph Relocalization with Deep Object Detection and BIM-Supported Object Landmark Dictionary, Journal of Computing in Civil Engineering, 10.1061/JCCEE5.CPENG-5301, 37, 5, (2023).