Vision and Trajectory–Based Dynamic Collision Prewarning Mechanism for Tower Cranes
Publication: Journal of Construction Engineering and Management
Volume 148, Issue 7
Abstract
Tower cranes are very common at construction sites. As workers focus most of their attention on their own tasks, their ability to detect changes in the surrounding environment is reduced, and it is difficult to avoid the collision risk of heavy falling objects. To solve this problem, this study establishes a dynamic collision prewarning mechanism for tower crane construction based on vision and trajectory analysis by tracking and predicting the trajectories of loads and workers. Specifically, the proposed dynamic collision prewarning mechanism consists of three parts. First, Fairmultiple object tracking (FairMOT), a multiple object tracking algorithm based on deep learning, is used to detect and track workers and loads, and time-series data of their positions are obtained. Then a trajectory prediction model based on a transformer is applied to predict the trajectories of objects in the future (10 s) based on the historical data. Finally, safety rules are established by considering the locations, speeds, shapes, and sizes of loads and workers and their trajectories over a period of time. Risk levels for each worker are assigned to reduce the risk of collisions between workers and loads. Finally, the performance of the models is evaluated at a construction site. FairMOT has good tracking performance and can continuously track objects with short occlusion (2 s). Transformer-based trajectory prediction model has higher accuracy than other methods [e.g., social generative adversarial network (GAN), social long short-term memory (LSTM)]. The results of the study show that the proposed method can accurately predict the unsafe approach of workers and loads. The safety prewarning mechanism proposed in this study can help improve the safety of tower crane construction.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code generated or used during the study are available in a repository or online in accordance with funder data retention policies. The repository address is https://github.com/dlutor/cranesafety.git.
Acknowledgments
The presented work is supported by Natural Science Foundation of Liaoning Province (2019-MS-052) and the Fundamental Research Funds for the Central Universities (DUT20ZD401).
References
Alahi, A., K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese. 2016. “Social lstm: Human trajectory prediction in crowded spaces.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 961–971. New York: IEEE.
Altché, F., and A. de La Fortelle. 2017. “An LSTM network for highway trajectory prediction.” In Proc., 2017 IEEE 20th Int. Conf. on Intelligent Transportation Systems (ITSC), 353–359. New York: IEEE.
Becker, S., R. Hug, W. Hubner, and M. Arens. 2018. “Red: A simple but effective baseline predictor for the trajnet benchmark.” In Proc., 15th European Conf. on Computer Vision (ECCV), 138–153. Berlin: Springer.
Bennewitz, M., W. Burgard, G. Cielniak, and S. Thrun. 2005. “Learning motion patterns of people for compliant robot motion.” Int. J. Rob. Res. 24 (1): 31–48. https://doi.org/10.1177/0278364904048962.
Bewley, A., Z. Ge, L. Ott, F. Ramos, and B. Upcroft. 2016. “Simple online and realtime tracking.” In Proc., 23rd IEEE Int. Conf. on Image Processing (ICIP), 3464–3468. New York: IEEE.
Cai, J., Y. Zhang, L. Yang, H. Cai, and S. Li. 2020. “A context-augmented deep learning approach for worker trajectory prediction on unstructured and dynamic construction sites.” Adv. Eng. Inf. 46 (Oct): 101173. https://doi.org/10.1016/j.aei.2020.101173.
Chen, J., Y. Fang, and Y. K. Cho. 2017a. “Mobile asset tracking for dynamic 3D crane workspace generation in real time.” In Proc., ASCE Int. Workshop on Computing in Civil Engineering (IWCCE), 122–129. Reston, VA: ASCE. https://doi.org/10.1061/9780784480830.016.
Chen, J., Y. Fang, and Y. K. Cho. 2017b. “Real-time 3D crane workspace update using a hybrid visualization approach.” J. Comput. Civ. Eng. 31 (5): 04017049. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000698.
Chen, Z. K., Y. F. Zhou, F. B. Wang, and Y. Zhou. 2013. “Structural design and experimental study of the crane avoidance warning visual system.” In Proc., 3rd Int. Conf. on Consumer Electronics, Communications and Networks (CECNet), 605–608. Beijing: IEEE.
Darkpgmr. 2020. “DarkLabel, video/image labeling and annotation tool.” Accessed May 7, 2021. https://github.com/darkpgmr/DarkLabel.
Dendorfer, P., H. Rezatofighi, A. Milan, J. Shi, D. Cremers, I. Reid, S. Roth, K. Schindler, and L. Leal-Taixé. 2020. “MOT20: A benchmark for multi object tracking in crowded scenes.” Preprint, submitted March 19, 2020. https://arxiv.org/abs/2003.09003.
Devlin, J., M.-W. Chang, K. Lee, and K. Toutanova. 2018. “BERT: Pre-training of deep bidirectional transformers for language understanding.” Preprint, submitted October 11, 2018. https://arxiv.org/abs/1810.04805.
Dong, C., H. Li, X. C. Luo, L. Y. Ding, J. Siebert, and H. B. Luo. 2018. “Proactive struck-by risk detection with movement patterns and randomness.” Autom. Constr. 91 (Jul): 246–255. https://doi.org/10.1016/j.autcon.2018.03.021.
Fang, W., L. Ding, H. Luo, and P. E. Love. 2018a. “Falls from heights: A computer vision-based approach for safety harness detection.” Autom. Constr. 91 (Jul): 53–61. https://doi.org/10.1016/j.autcon.2018.02.018.
Fang, Y. H., J. D. Chen, Y. K. Cho, K. Kim, S. J. Zhang, and E. Perez. 2018b. “Vision-based load sway monitoring to improve crane safety in blind lifts.” J. Struct. Integrity Maint. 3 (4): 233–242. https://doi.org/10.1080/24705314.2018.1531348.
Fu, X. Y., D. Niu, Q. Li, and J. B. Liu. 2019. “Position-pose measurement of crane sway based on monocular vision.” J. Eng.-JOE 2019 (22): 8330–8334. https://doi.org/10.1049/joe.2019.1072.
Giuliari, F., I. Hasan, M. Cristani, and F. Galasso. 2021. “Transformer networks for trajectory forecasting.” In Proc., 2020 25th Int. Conf. on Pattern Recognition (ICPR), 10335–10342. New York: IEEE.
Gupta, A., J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi. 2018. “Social gan: Socially acceptable trajectories with generative adversarial networks.” In Proc., 31st IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2255–2264. New York: IEEE.
Gutierrez, R., M. Magallon, and D. C. Hernandez. 2021. “Vision-based system for 3D tower crane monitoring.” IEEE Sens. J. 21 (10): 11935–11945. https://doi.org/10.1109/JSEN.2020.3042532.
Hellbach, S., J. P. Eggert, E. Körner, and H.-M. Gross. 2008. “Time series analysis for long term prediction of human movement trajectories.” In Proc., 15th Int. Conf. on Neuro-Information Processing, 567–574. Berlin: Springer.
Henriques, J. F., R. Caseiro, P. Martins, and J. Batista. 2014. “High-speed tracking with kernelized correlation filters.” IEEE Trans. Pattern Anal. Mach. Intell. 37 (3): 583–596. https://doi.org/10.1109/TPAMI.2014.2345390.
Huang, J. L., W. M. Xu, W. W. Zhao, H. S. Yuan, and Y. B. Hu. 2020. “Gray object recognition and angle measurement method based on mean shift to overhead crane systems.” In Proc., 39th Chinese Control Conf. (CCC), 2924–2929. New York: IEEE.
Hyla, P., and J. Szpytko. 2014. “Crane payload position measurement vision-based system dedicated for anti-sway solutions.” In Proc., 14th Transport Systems Telematics Conf. (TST), 404–413. Berlin: Springer.
Inukai, T., Y. Yoshida, and IEEE. 2012. “Control of a boom crane using installed stereo vision.” In Proc., 6th Int. Conf. on Sensing Technology (ICST), 189–194. Kolkata, India: IEEE.
Karwowski, W., and M. Rahimi. 1991. “Worker selection of safe speed and idle condition in simulated monitoring of two industrial robots.” Ergonomics 34 (5): 531–546. https://doi.org/10.1080/00140139108967335.
Kendall, A., Y. Gal, and R. Cipolla. 2018. “Multi-task learning using uncertainty to weigh losses for scene geometry and semantics.” In Proc., 31st IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 7482–7491. New York: IEEE.
Kim, D., H. Jebelli, S. Lee, and V. R. Kamat. 2020a. “Enhancing deep neural network-based trajectory prediction: Fine-tuning and inherent movement-driven post-processing.” In Proc., Construction Research Congress 2020: Safety, Workforce, and Education, 67–75. Reston, VA: ASCE. https://doi.org/10.1061/9780784482872.008.
Kim, D., S. Lee, and V. R. Kamat. 2020b. “Proximity prediction of mobile objects to prevent contact-driven accidents in co-robotic construction.” J. Comput. Civ. Eng. 34 (4): 04020022. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000899.
Kim, D., M. Liu, S. Lee, and V. R. Kamat. 2019. “Trajectory prediction of mobile construction resources toward pro-active struck-by hazard detection.” In Proc., the Int. Symp. on Automation and Robotics in Construction (IAARC). Waterloo, IA, Canada: International Association for Automation and Robotics in Construction (IAARC).
Kolar, Z., H. Chen, and X. Luo. 2018. “Transfer learning and deep convolutional neural networks for safety guardrail detection in 2D images.” Autom. Constr. 89 (May): 58–70. https://doi.org/10.1016/j.autcon.2018.01.003.
Kong, T., W. L. Fang, P. E. D. Love, H. B. Luo, S. J. Xu, and H. Li. 2021. “Computer vision and long short-term memory: Learning to predict unsafe behaviour in construction.” Adv. Eng. Inf. 50 (Oct): 101400. https://doi.org/10.1016/j.aei.2021.101400.
Lerner, A., Y. Chrysanthou, and D. Lischinski. 2007. “Crowds by example.” Comput. Graph. Forum 26 (3): 655–664. https://doi.org/10.1111/j.1467-8659.2007.01089.x.
Memarzadeh, M., M. Golparvar-Fard, and J. C. Niebles. 2013. “Automated 2D detection of construction equipment and workers from site video streams using histograms of oriented gradients and colors.” Autom. Constr. 32 (Jul): 24–37. https://doi.org/10.1016/j.autcon.2012.12.002.
Milan, A., L. Leal-Taixé, I. Reid, S. Roth, and K. Schindler. 2016. “MOT16: A benchmark for multi-object tracking.” Preprint, submitted March 2, 2016. https://arxiv.org/abs/1603.00831.
Ministry of Housing and Urban-Rural Development. 2001. Technical specification for safety of construction machinery. JGJ 33-2001. Beijing: China Construction Industry Press.
Nam, H., M. Baek, and B. Han. 2016. “Modeling and propagating cnns in a tree structure for visual tracking.” Preprint, submitted August 25, 2016. https://arxiv.org/abs/1608.07242.
OSHA (Occupational Safety and Health Administration). 2010. “Safety and health regulations for construction.” Accessed May 30, 2021. https://www.osha.gov/laws-regs/regulations/standardnumber/1926/1926.1425.
OSHA (Occupational Safety and Health Administration). 2020. “Accident search results.” Accessed May 7, 2021. https://www.osha.gov/pls/imis/AccidentSearch.search?acc_keyword=%22Tower%20Crane%22&keyword_list=on.
Park, J., E. Marks, Y. K. Cho, and W. Suryanto. 2016. “Performance test of wireless technologies for personnel and equipment proximity sensing in work zones.” J. Constr. Eng. Manage. 142 (1): 04015049. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001031.
Pellegrini, S., A. Ess, K. Schindler, L. van Gool, and IEEE. 2009. “You’ll never walk alone: Modeling social behavior for multi-target tracking.” In Proc., 12th IEEE Int. Conf. on Computer Vision, 261–268. New York: IEEE. https://doi.org/10.1109/iccv.2009.5459260.
Price, L. C., J. D. Chen, J. Park, and Y. K. Cho. 2021. “Multisensor-driven real-time crane monitoring system for blind lift operations: Lessons learned from a case study.” Autom. Constr. 124 (Apr): 103552. https://doi.org/10.1016/j.autcon.2021.103552.
Rahman, M. S., J. Vaughan, and ASME. 2014. “Simple near-realtime crane workspace mapping using machine vision.” In Proc., ASME 7th Annual Dynamic Systems and Control Conf. New York: ASME.
Rashid, K. M., and A. H. Behzadan. 2017. “Enhancing motion trajectory prediction for site safety by incorporating attitude toward risk.” In Proc., Computing Civil Engineering, 425–433. Reston, VA: ASCE. https://doi.org/10.1061/9780784480847.053.
Rashid, K. M., and A. H. Behzadan. 2018. “Risk behavior-based trajectory prediction for construction site safety monitoring.” J. Constr. Eng. Manage. 144 (2): 04017106. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001420.
Rashid, K. M., S. Datta, and A. H. Behzadan. 2017. “Coupling risk attitude and motion data mining in a preemtive construction safety framework.” In Proc., Winter Simulation Conf. (WSC), 2413–2424. New York: IEEE.
Roberts, D., T. Bretl, and M. Golparvar-Fard. 2017. “Detecting and classifying cranes using camera-equipped UAVs for monitoring crane-related safety hazards.” In Proc., ASCE Int. Workshop on Computing in Civil Engineering (IWCCE), 442–449. Reston, VA: ASCE. https://doi.org/10.1061/9780784480847.055.
Rosenfeld, J. 2021. “What OSHA safety data tells us about tower crane accidents.” Accessed May 7, 2021. https://www.rosenfeldinjurylawyers.com/news/what-osha-safety-data-tells-us-about-tower-crane-accidents/.
Sadeghian, A., V. Kosaraju, A. Gupta, S. Savarese, and A. Alahi. 2018. “Trajnet: Towards a benchmark for human trajectory prediction.” In Proc., Computer Vision and Pattern Recognition Workshops (CVPRW). New York: IEEE.
Son, H., H. Seong, H. Choi, and C. Kim. 2019. “Real-time vision-based warning system for prevention of collisions between workers and heavy equipment.” J. Comput. Civ. Eng. 33 (5): 04019029. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000845.
Tang, S., M. Golparvar-Fard, M. Naphade, and M. M. Gopalakrishna. 2020. “Video-based motion trajectory forecasting method for proactive construction safety monitoring systems.” J. Comput. Civ. Eng. 34 (6): 04020041. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000923.
Theios, J. 1973. “Reaction time measurements in the study of memory processes: Theory and data.” Psychol. Learn. Motivation 7 (Jan): 43–85. https://doi.org/10.1016/S0079-7421(08)60065-1.
Thipphavong, D. P., C. A. Schultz, A. G. Lee, and S. H. Chan. 2013. “Adaptive algorithm to improve trajectory prediction accuracy of climbing aircraft.” J. Guid. Control Dyn. 36 (1): 15–24. https://doi.org/10.2514/1.58508.
Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. 2017. “Attention is all you need.” Preprint, submitted June 12, 2017. https://arxiv.org/abs/1706.03762.
Vemula, A., K. Muelling, and J. Oh. 2018. “Social attention: Modeling attention in human crowds.” In Proc., 2018 IEEE Int. Conf. on Robotics and Automation (ICRA), 4601–4607. New York: IEEE.
Williams, C. K. 1998. “Prediction with Gaussian processes: From linear regression to linear prediction and beyond.” In Learning in graphical models, 599–621. Dordrecht, Netherlands: Springer.
Wojke, N., A. Bewley, and D. Paulus. 2017. “Simple online and realtime tracking with a deep association metric.” In Proc., 2017 IEEE Int. Conf. on Image Processing (ICIP), 3645–3649. New York: IEEE.
Xu, W., and T.-K. Wang. 2020. “Dynamic safety prewarning mechanism of human–machine–environment using computer vision.” Eng. Constr. Archit. Manage. 27 (8): 1813–1833. https://doi.org/10.1108/ECAM-12-2019-0732.
Yang, J. S., M. L. Huang, W. F. Chien, and M. H. Tsai. 2015. “Application of machine vision to collision avoidance control of the overhead crane.” In Proc., Int. Conf. on Automation, Mechanical and Electrical Engineering (AMEE), 361–364. Paris: Atlantis Press.
Yang, Z., Y. Yuan, M. Zhang, X. Zhao, Y. Zhang, and B. Tian. 2019. “Safety distance identification for crane drivers based on mask R-CNN.” Sensors 19 (12): 2789. https://doi.org/10.3390/s19122789.
Yu, Y., H. Guo, Q. Ding, H. Li, and M. Skitmore. 2017. “An experimental study of real-time identification of construction workers’ unsafe behaviors.” Autom. Constr. 82 (Oct): 193–206. https://doi.org/10.1016/j.autcon.2017.05.002.
Zhang, M., Z. Cao, Z. Yang, and X. Zhao. 2020a. “Utilizing computer vision and fuzzy inference to evaluate level of collision safety for workers and equipment in a dynamic environment.” J. Constr. Eng. Manage. 146 (6): 04020051. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001802.
Zhang, M., M. Zhu, and X. Zhao. 2020b. “Recognition of high-risk scenarios in building construction based on image semantics.” J. Comput. Civ. Eng. 34 (4): 04020019. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000900.
Zhang, X., W. Zhang, L. Jiang, and T. Zhao. 2020c. “Identification of critical causes of tower-crane accidents through system thinking and case analysis.” J. Constr. Eng. Manage. 146 (7): 04020071. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001860.
Zhang, Y., C. Wang, X. Wang, W. Zeng, and W. Liu. 2020d. “FairMOT: On the fairness of detection and re-identification in multiple object tracking.” Preprint, submitted April 4, 2020. https://arxiv.org/abs/2004.01888.
Zhou, H., S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang. 2020. “Informer: Beyond efficient transformer for long sequence time-series forecasting.” In Proc., 35th AAAI Conf. on Artificial Intelligence. Menlo Park, CA: Association for the Advancement of Artificial Intelligence (AAAI).
Zhou, X., D. Wang, and P. Krähenbühl. 2019. “Objects as points.” Preprint, submitted April 16, 2019. https://arxiv.org/abs/1904.07850.
Information & Authors
Information
Published In
Journal of Construction Engineering and Management
Volume 148 • Issue 7 • July 2022
Copyright
© 2022 American Society of Civil Engineers.
History
Received: Aug 26, 2021
Accepted: Mar 7, 2022
Published online: May 11, 2022
Published in print: Jul 1, 2022
Discussion open until: Oct 11, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Pinsheng Duan, Jianliang Zhou, Yang Miang Goh, Safety Risk Diagnosis Based on Motion Trajectory for Construction Workers: An Integrated Approach, Journal of Construction Engineering and Management, 10.1061/JCEMD4.COENG-13673, 149, 11, (2023).
- Peichen Zhai, Junjie Wang, Lite Zhang, Extracting Worker Unsafe Behaviors from Construction Images Using Image Captioning with Deep Learning–Based Attention Mechanism, Journal of Construction Engineering and Management, 10.1061/JCEMD4.COENG-12096, 149, 2, (2023).
- Zhuoya Zhao, Hang Yan, Yanhua Wang, Daoyuan Guo, A Coupled Analysis of Risk Factors of Tower Crane Collapse Accident in Extreme Winds Conditions Based on N-K Model, ICCREM 2023, 10.1061/9780784485217.096, (973-981), (2023).