Intelligent Construction Activity Identification for All-Weather Site Monitoring Using 4D Millimeter-Wave Technology
Publication: Journal of Construction Engineering and Management
Volume 150, Issue 11
Abstract
Site monitoring is indispensable for modern construction management. Contact approaches, represented by wearable devices, have problems such as privacy leaks and hindering working. Vision-based noncontact methods depend highly on light and environmental conditions, and have poor three-dimensional perception ability. To propose an all-weather noncontact activity identification approach on construction sites, four-dimensional (4D) millimeter-wave (MMW) radar is adopted in this study for the first time because of its excellent abilities of motion sensing, spatial sensing, and penetration. First, a feature processing method is proposed to convert the MMW signal to a seven-dimensional point cloud, which consists of the shape information (, , and ) and four attributes (, , , and ), representing the information of velocity, signal-to-noise ratio, height, and volume, respectively. Second, a novel deep learning framework is developed, which contains (1) one shape subnetwork, driven by the PointNet++ model, to capture the shape information of objects; (2) four attribute subnetworks to fully utilize the additional attribute features; and (3) a two-layer fusion module to combine all the outputs of the subnetworks. With precision of 0.963, recall of 0.961, and an F1 score of 0.962, the results show that the proposed method can accurately identify construction activities under different environmental conditions. It also can facilitate further development of MMW radar–based solutions for construction site analysis.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (Grant Nos. 42302322 and 72271186), the Internal Research Funding of the Hong Kong Polytechnic University (Grant No. P0047899), General Research Fund (GRF) Grant (15201621) titled “Monitoring and managing fatigue of construction plant and equipment operators exposed to prolonged sitting,” and General Research Fund (GRF) Grant (15210923) titled “Noninvasive noncontact mental workload and stress monitoring of construction equipment operators.”
References
Akbal, E., and T. Tuncer. 2022. “A learning model for automated construction site monitoring using ambient sounds.” Autom. Constr. 134 (May): 104094. https://doi.org/10.1016/j.autcon.2021.104094.
Almalioglu, Y., M. Turan, C. X. Lu, N. Trigoni, and A. Markham. 2021. “Milli-RIO: Ego-motion estimation with low-cost millimetre-wave radar.” IEEE Sens. J. 21 (3): 3314–3323. https://doi.org/10.1109/jsen.2020.3023243.
Arab, H., I. Ghaffari, L. Chioukh, S. O. Tatu, and S. Dufour. 2022. “A convolutional neural network for human motion recognition and classification using a millimeter-wave Doppler radar.” IEEE Sens. J. 22 (5): 4494–4502. https://doi.org/10.1109/JSEN.2022.3140787.
Asvadi, A., C. Premebida, P. Peixoto, and U. Nunes. 2016. “3D Lidar-based static and moving obstacle detection in driving environments: An approach based on voxels and multi-region ground planes.” Rob. Auton. Syst. 83 (Jun): 299–311. https://doi.org/10.1016/j.robot.2016.06.007.
Bai, J., S. Li, B. Tan, L. Q. Zheng, L. B. Huang, and L. F. Dong. 2022. “Traffic participants classification based on 3D radio detection and ranging point clouds.” IET Radar Sonar Navig. 16 (2): 278–290. https://doi.org/10.1049/rsn2.12182.
Bassier, M., S. Vincke, H. De Winter, and M. Vergauwen. 2020. “Drift invariant metric quality control of construction sites using BIM and point cloud data.” ISPRS Int. J. Geo-Inf. 9 (9): 545. https://doi.org/10.3390/ijgi9090545.
Chang, M. F., et al. 2019. “Argoverse: 3D tracking and forecasting with rich maps.” In Proc., 32nd IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 8740–8749. New York: IEEE.
Chen, B. W., S. Shi, J. Sun, W. Gong, J. Yang, L. Du, K. H. Guo, B. H. Wang, and B. W. Chen. 2019. “Hyperspectral lidar point cloud segmentation based on geometric and spectral information.” Opt. Express 27 (17): 24043–24059. https://doi.org/10.1364/oe.27.024043.
Cheng, Y., J. Su, H. Chen, and Y. Liu. 2021. “A new automotive radar 4D point clouds detector by using deep learning.” In Proc., ICASSP 2021–2021 IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 8398–8402. New York: IEEE.
Cong, D. H., L. Q. Zhang, P. P. Su, Z. Y. Tang, Y. H. Meng, and Y. Wang. 2017. “Design and implementation of LiDAR navigation system based on triangulation measurement.” In Proc., 29th Chinese Control and Decision Conf. (CCDC), 6060–6063. New York: IEEE.
Cui, H., J. Z. Wu, J. M. Zhang, G. Chowdhary, and W. R. Norris. 2021. “3D detection and tracking for on-road vehicles with a monovision camera and dual low-cost 4D mmWave radars.” In Proc., IEEE Intelligent Transportation Systems Conf. (ITSC), 2931–2937. New York: IEEE.
Czerniawski, T., B. Sankaran, M. Nahangi, C. Haas, and F. Leite. 2018. “6D DBSCAN-based segmentation of building point clouds for planar object classification.” Autom. Constr. 88 (Jun): 44–58. https://doi.org/10.1016/j.autcon.2017.12.029.
Dhiman, C., and D. K. Vishwakarma. 2019. “A robust framework for abnormal human action recognition using R-transform and Zernike moments in depth videos.” IEEE Sens. J. 19 (13): 5195–5203. https://doi.org/10.1109/jsen.2019.2903645.
Elyass, S. M., A. N. Abed, J. S. Hussei, A. G. Mahmoud, H. T. Ziboon, and T. R. Saeed. 2020. “Human gait recognition by shoulder movement’s Doppler signature using SVM classifier.” In Proc., AIP Conf. New York: AIP Publishing.
Fang, Q., H. Li, X. C. Luo, L. Y. Ding, H. B. Luo, T. M. Rose, and W. P. An. 2018a. “Detecting non-hardhat-use by a deep learning method from far -field surveillance videos.” Autom. Constr. 85 (Mar): 1–9. https://doi.org/10.1016/j.autcon.2017.09.018.
Fang, W., L. Ding, B. Zhong, P. E. D. Love, and H. Luo. 2018b. “Automated detection of workers and heavy equipment on construction sites: A convolutional neural network approach.” Adv. Eng. Inf. 37 (Mar): 139–149. https://doi.org/10.1016/j.aei.2018.05.003.
Fini, A. A. F., M. Maghrebi, P. J. Forsythe, and T. S. Waller. 2021. “Using existing site surveillance cameras to automatically measure the installation speed in prefabricated timber construction.” Eng. Constr. Archit. Manage. 29 (2): 573–600. https://doi.org/10.1108/ECAM-04-2020-0281.
Gao, X. Y., G. B. Xing, S. Roy, and H. Liu. 2021. “RAMP-CNN: A novel neural network for enhanced automotive radar object recognition.” IEEE Sens. J. 21 (4): 5119–5132. https://doi.org/10.1109/jsen.2020.3036047.
Gu, M., Z. Chen, K. Chen, and H. Pan. 2024. “RMPCT-Net: A multi-channel parallel CNN and transformer network model applied to HAR using FMCW radar.” Signal Image Video Process. 18 (3): 2219–2229. https://doi.org/10.1007/s11760-023-02894-4.
Han, S., J. Zhang, Z. S. Shaikh, J. Wang, and W. Ren. 2023. “Four-dimensional (4D) millimeter wave-based sensing and its potential applications in digital construction: A review.” Buildings 13 (6): 1454. https://doi.org/10.3390/buildings13061454.
Jebelli, H., C. R. Ahn, and T. L. Stentz. 2016. “Comprehensive fall-risk assessment of construction workers using inertial measurement units: Validation of the gait-stability metric to assess the fall risk of iron workers.” J. Comput. Civ. Eng. 30 (3): 04015034. https://doi.org/10.1061/(asce)cp.1943-5487.0000511.
Kakani, V., H. Kim, J. Lee, C. Ryu, and M. Kumbham. 2020. “Automatic distortion rectification of wide-angle images using outlier refinement for streamlining vision tasks.” Sensors 20 (3): 894. https://doi.org/10.3390/s20030894.
Kang, K. S., Y. W. Cho, K. H. Jin, Y. B. Kim, and H. G. Ryu. 2022. “Application of one-stage instance segmentation with weather conditions in surveillance cameras at construction sites.” Autom. Constr. 133 (Jan): 104034. https://doi.org/10.1016/j.autcon.2021.104034.
Kim, D., M. Y. Liu, S. Lee, and V. R. Kamat. 2019. “Remote proximity proximity monitoring between mobile construction resources using camera-mounted UAVs.” Autom. Constr. 99 (Dec): 168–182. https://doi.org/10.1016/j.autcon.2018.12.014.
Kim, J., and S. Chi. 2020. “Multi-camera vision-based productivity monitoring of earthmoving operations.” Autom. Constr. 112 (Apr): 103121. https://doi.org/10.1016/j.autcon.2020.103121.
Kim, S., S. H. Hong, H. Kim, M. Lee, and S. Hwang. 2023. “Small object detection (SOD) system for comprehensive construction site safety monitoring.” Autom. Constr. 156 (Dec): 105103. https://doi.org/10.1016/j.autcon.2023.105103.
Kosuge, A., S. Suehiro, M. Hamada, and T. Kuroda. 2022. “mmWave-YOLO: A mmWave imaging radar-based real-time multiclass object recognition system for ADAS applications.” IEEE Trans. Instrum. Meas. 71 (May): 1–10. https://doi.org/10.1109/tim.2022.3176014.
Lazaro, A., M. Lazaro, R. Villarino, and D. Girbau. 2021. “Seat-occupancy detection system and breathing rate monitoring based on a low-cost mm-wave radar at 60 GHz.” IEEE Access 9 (Jun): 115403–115414. https://doi.org/10.1109/access.2021.3105390.
Li, G., Y. L. Sit, S. Manchala, T. Kettner, A. Ossowska, K. Krupinski, C. Sturm, and U. Luebbert. 2019. “Novel 4D 79 GHz radar concept for object detection and active safety applications.” In Proc., 12th German Microwave Conf. (GeMiC), 87–90. New York: IEEE.
Li, H., H. Zhang, and X. Wang. 2020. “Vital signs detection based on millimeter wave radar.” In Proc., 5th Int. Conf. on Communication, Image and Signal Processing (CCISP), 6–9. New York: IEEE.
Liu, H., G. B. Wang, T. Huang, P. He, M. Skitmore, and X. C. Luo. 2020. “Manifesting construction activity scenes via image captioning.” Autom. Constr. 119 (Nov): 103334. https://doi.org/10.1016/j.autcon.2020.103334.
Maalek, R., and F. Sadeghpour. 2016. “Accuracy assessment of ultra-wide band technology in locating dynamic resources in indoor scenarios.” Autom. Constr. 63 (Jun): 12–26. https://doi.org/10.1016/j.autcon.2015.11.009.
Mohammadi, S. S., Y. Wang, and A. D. Bue. 2021. “Pointview-GCN: 3D Shape classification with multi-view point clouds.” In Proc., 2021 IEEE Int. Conf. on Image Processing (ICIP), 3103–3107. New York: IEEE.
Nobis, F., M. Geisslinger, M. Weber, J. Betz, and M. Lienkamp. 2019. “A deep learning-based radar and camera sensor fusion architecture for object detection.” In Proc., Symp. on Sensor Data Fusion—Trends, Solutions, Applications (SDF). New York: IEEE.
Oh, K., S. Park, J. Seo, J.-G. Kim, J. Park, G. Lee, and K. Yi. 2019. “Development of a predictive safety control algorithm using laser scanners for excavators on construction sites.” Proc. Inst. Mech. Eng., Part D: J. Automob. Eng. 233 (8): 2007–2029. https://doi.org/10.1177/0954407018764046.
Omar, H., L. Mandjoubi, and G. Kheder. 2018. “Towards an automated photogrammetry-based approach for monitoring and controlling construction site activities.” Comput. Ind. 98 (Jun): 172–182. https://doi.org/10.1016/j.compind.2018.03.012.
Price, L. C., J. Chen, J. Park, and Y. K. Cho. 2021. “Multisensor-driven real-time crane monitoring system for blind lift operations: Lessons learned from a case study.” Autom. Constr. 124 (Apr): 103552. https://doi.org/10.1016/j.autcon.2021.103552.
Puri, N., and Y. Turkan. 2020. “Bridge construction progress monitoring using lidar and 4D design models.” Autom. Constr. 109 (Jan): 102961. https://doi.org/10.1016/j.autcon.2019.102961.
Qi, C. R., H. Su, K. C. Mo, and L. J. Guibas. 2017a. “PointNet: Deep learning on point sets for 3D classification and segmentation.” In Proc., 30th IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 77–85. New York: IEEE.
Qi, C. R., L. Yi, H. Su, and L. J. Guibas. 2017b. “Pointnet++: Deep hierarchical feature learning on point sets in a metric space.” Adv. Neural Inf. Process Syst. 30: 5099–5108.
Rao, A. S., M. Radanovic, Y. G. Liu, S. B. Hu, Y. H. Fang, K. Khoshelham, M. Palaniswami, and T. Ngo. 2022. “Real-time monitoring of construction sites: Sensors, methods, and applications.” Autom. Constr. 136 (Apr): 104099. https://doi.org/10.1016/j.autcon.2021.104099.
Stolz, M., M. Wolf, F. Meinl, M. Kunert, and W. Menzel. 2018. “A new antenna array and signal processing concept for an automotive 4D radar.” In Proc., 15th European Radar Conf. (EuRAD), 63–66. New York: IEEE.
Svenningsson, P., F. Fioranelli, A. Yarovoy, and A. F. Martone. 2022. “A Bayesian network for the classification of human motion as observed by distributed radar.” IEEE Trans. Aerosp. Electron. Syst. 58 (6): 5661–5674. https://doi.org/10.1109/TAES.2022.3177589.
Tang, L., Y. Jia, Y. Qian, S. Yi, and P. Yuan. 2021. “Human activity recognition based on mixed CNN with radar multi-spectrogram.” IEEE Sens. J. 21 (22): 25950–25962. https://doi.org/10.1109/JSEN.2021.3118836.
Wang, S. H. Y., M. Kim, H. Hae, M. Q. Cao, and J. Kim. 2023. “The development of a rebar-counting model for reinforced concrete columns: Using an unmanned aerial vehicle and deep-learning approach.” J. Constr. Eng. Manage. 149 (11): 04023111. https://doi.org/10.1061/jcemd4.Coeng-13686.
Wang, Y. H., H. P. Liu, K. N. Cui, A. F. Zhou, W. S. Li, and H. D. Ma. 2021. “m-Activity: Accurate and real-time human activity recognition via millimeter wave radar.” In Proc., IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 8298–8302. New York: IEEE.
Wang, Z., V. Ramamoorthy, U. Gal, and A. Guez. 2020. “Possible life saver: A review on human fall detection technology.” Robotics 9 (3): 55. https://doi.org/10.3390/robotics9030055.
Wong, W., S. Tariq, S. R. Mohandes, and T. Zayed. 2020. “IoT-based application for construction site safety monitoring.” Int. J. Construct. Manage. 23 (1): 58–74. https://doi.org/10.1080/15623599.2020.1847405.
Woo, S., S. Jeong, E. Mok, L. Y. Xia, C. Choi, M. Pyeon, and J. Heo. 2011. “Application of WiFi-based indoor positioning system for labor tracking at construction sites: A case study in Guangzhou MTR.” Autom. Constr. 20 (1): 3–13. https://doi.org/10.1016/j.autcon.2010.07.009.
Wu, H., and J. Zhao. 2018. “An intelligent vision-based approach for helmet identification for work safety.” Comput. Ind. 100 (Mar): 267–277. https://doi.org/10.1016/j.compind.2018.03.037.
Wu, H. T., B. T. Zhong, H. Li, P. Love, X. Pan, and N. Zhao. 2021. “Combining computer vision with semantic reasoning for on-site safety management in construction.” J. Build. Eng. 42 (Oct): 103036. https://doi.org/10.1016/j.jobe.2021.103036.
Xiao, B., H. R. Xiao, J. W. Wang, and Y. Chen. 2022. “Vision-based method for tracking workers by integrating deep learning instance segmentation in off-site construction.” Autom. Constr. 136 (Apr): 104148. https://doi.org/10.1016/j.autcon.2022.104148.
Yadav, N., N. Sadeghi, and J. Kang. 2022. “On-body placement of wearable safety promotion devices based on wireless communication for construction workers-on-foot: State-of-the-art review.” Sensors 22 (9): 3134. https://doi.org/10.3390/s22093134.
Yaghoobi Ershadi, N., and J. M. Menéndez. 2017. “Vehicle tracking and counting system in dusty weather with vibrating camera conditions.” J. Sens. 2017 (1): 3812301. https://doi.org/10.1155/2017/3812301.
Yang, W. Y., and J. Ke. 2016. “Compressive full waveform LiDAR with low-cost sensor.” In Proc., Conf. on Optoelectronic Imaging and Multimedia Technology IV. Bellingham, WA: International Society for Optics and Photonics.
Yu, C. X., Z. Z. Xu, K. Yan, Y. R. Chien, S. H. Fang, and H. C. Wu. 2022. “Noninvasive human activity recognition using millimeter-wave radar.” IEEE Syst. J. 16 (2): 3036–3047. https://doi.org/10.1109/jsyst.2022.3140546.
Yu, Y. T., X. C. Yang, H. Li, X. C. Luo, H. L. Guo, and Q. Fang. 2019. “Joint-level vision-based ergonomic assessment tool for construction workers.” J. Constr. Eng. Manage. 145 (5): 04019025. https://doi.org/10.1061/(asce)co.1943-7862.0001647.
Yuan, X., F. Moreu, and C. D. Lippitt. 2021. “Bridge construction monitoring using LiDAR data.” In Proc., Tran-SET Conf. (Tran-SET), 317–323. Reston, VA: ASCE.
Zeid, K. A., J. Schult, A. Hermans, and B. Leibe. 2023. “Point2vec for self-supervised representation learning on point clouds.” In Proc., DAGM German Conf. on Pattern Recognition, 131–146. Cham, Switzerland: Springer.
Zhang, C., M. D. Cao, Y. Q. Gong, Y. Li, Y. M. Huang, and H. M. Wang. 2020. “Calibration of motional frequency spread for wide-band FMCW automotive millimeter-wave radar.” IEEE Access 8 (Dec): 14355–14366. https://doi.org/10.1109/access.2020.2966222.
Zhang, C., W. J. Shen, and Z. H. Ye. 2022. “Technical feasibility analysis on applying ultra-wide band technology in construction progress monitoring.” Int. J. Construct. Manage. 22 (15): 2951–2965. https://doi.org/10.1080/15623599.2020.1834928.
Zhang, R., and S. Cao. 2018. “Real-time human motion behavior detection via CNN using mmWave radar.” IEEE Sens. Lett. 3 (2): 1–4. https://doi.org/10.1109/LSENS.2018.2889060.
Zhang, S., W. Meng, H. Li, and X. Cui. 2019. “Multimodal spatiotemporal networks for sign language recognition.” IEEE Access 7 (Mar): 180270–180280. https://doi.org/10.1109/ACCESS.2019.2959206.
Zhao, Y. B., A. Yarovoy, and F. Fioranelli. 2022. “Angle-insensitive human motion and posture recognition based on 4D imaging radar and deep learning classifiers.” IEEE Sens. J. 22 (12): 12173–12182. https://doi.org/10.1109/jsen.2022.3175618.
Zhou, H., C. Xu, X. Tang, S. Wang, and Z. Zhang. 2022. “A review of vision-laser-based civil infrastructure inspection and monitoring.” Sensors 22 (15): 5882. https://doi.org/10.3390/s22155882.
Zhu, P. Q., X. F. Yin, J. Rodriguez-Pineiro, Z. Y. Chen, P. Wang, and G. Li. 2022. “Measurement-based wideband space-time channel models for 77GHz automotive radar in underground parking lots.” IEEE Trans. Intell. Transp. Syst. 23 (10): 19105–19120. https://doi.org/10.1109/tits.2022.3157849.
Information & Authors
Information
Published In
Journal of Construction Engineering and Management
Volume 150 • Issue 11 • November 2024
Copyright
© 2024 American Society of Civil Engineers.
History
Received: Dec 8, 2023
Accepted: May 28, 2024
Published online: Aug 22, 2024
Published in print: Nov 1, 2024
Discussion open until: Jan 22, 2025
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.