Automated Detection and Segmentation of Mechanical, Electrical, and Plumbing Components in Indoor Environments by Using the YOLACT++ Architecture
Publication: Journal of Construction Engineering and Management
Volume 150, Issue 8
Abstract
Indoor construction environments, with their high density and detailed components, are complex areas for progress monitoring and reporting. Traditional manual monitoring systems, often constrained by poor lighting and accessibility, are inaccurate and time-consuming. With recent technological advancements, deep learning-based object recognition models have achieved considerable attention in construction. This paper introduces a novel method for progress monitoring and reporting of construction operations, employing digital imaging and the You Only Look At CoefficienTs (YOLACT++) deep learning algorithm to automatically recognize mechanical, electrical, and plumbing (MEP) components in challenging indoor settings. Data augmentation techniques and transfer learning were applied to improve the model’s generalization and adaptability. The study distinctively focuses on complex components in complicated indoor environments, a less explored area in current research that mainly centered on outdoor or simpler indoor settings. To achieve this, the study enhanced the dataset quality by generating synthetic images that closely represent actual indoor conditions including different lighting, object complexity and scale, occlusion, clutter, and viewpoints. This study also evaluated different mixes of synthetic and real images to determine the optimum combination for effective training. Moving beyond commonly used algorithms such as Mask R-CNN and You Only Look Once (YOLO), the method applied in this work is the YOLACT++ with deformable convolutional neural networks v2 (DCNv2), enhancing the model’s ability to handle objects with different scales, postures, rotations, and viewpoints in the images that are essential in indoor environments. The model is validated on a large test dataset, including real images from construction sites, to cover different indoor scenarios. The model achieved a precision of 84.80% and a recall of 85.58% for HVAC duct detection and a precision of 86.87% and a recall of 73.93% for pipe detection, demonstrating its effectiveness under challenging conditions. This method contributes to more accurate automated progress monitoring in indoor environments by reducing manual and error prone inspections.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code generated or used during the study are available from the corresponding author by request.
Acknowledgments
The authors wish to acknowledge Dr. Golkhoo for collecting the Montréal set of images and Pomerleau Inc. for authorizing the use of these images for research and academic purposes. Further acknowledgment is extended to Mr. Behdad Mozhdehi for providing access to a site in Tehran, Iran, for image collection.
References
Ahmed, M., C. T. Haas, and R. Haas. 2012. “Using digital photogrammetry for pipe-works progress tracking.” Can. J. Civ. Eng. 39 (9): 1062–1071. https://doi.org/10.1139/l2012-055.
Akhil, R. P., and B. B. Das. 2019. “Cost reduction techniques on MEP projects.” Lecture Notes Civ. Eng. 25 (Jun): 495–517. https://doi.org/10.1007/978-981-13-3317-0_44.
Athira, M. V., and D. M. Khan. 2020. “Recent trends on object detection and image classification: A review.” In Proc., 2020 Int. Conf. Computational Performance Evaluation (ComPE 2020), 427–435. New York: IEEE.
Barrera-Animas, A. Y., and J. M. Davila Delgado. 2023. “Generating real-world-like labelled synthetic datasets for construction site applications.” Autom. Constr. 151 (Jul): 104850. https://doi.org/10.1016/j.autcon.2023.104850.
Bolya, D., C. Zhou, F. Xiao, and Y. J. Lee. 2019. “YOLACT: Real-time instance segmentation.” In Proc., IEEE Int. Conf. Computer Vision (ICCV), 9156–9165. New York: IEEE.
Bolya, D., C. Zhou, F. Xiao, and Y. J. Lee. 2020. “YOLACT++ better real-time instance segmentation.” IEEE Trans. Pattern Anal. Mach. Intell. 44 (2): 1108–1121. https://doi.org/10.1109/TPAMI.2020.3014297.
Bosché, F., M. Ahmed, Y. Turkan, C. T. Haas, and R. Haas. 2015. “The value of integrating Scan-to-BIM and Scan-vs-BIM techniques for construction monitoring using laser scanning and BIM: The case of cylindrical MEP components.” Autom. Constr. 49 (Jan): 201–213. https://doi.org/10.1016/j.autcon.2014.05.014.
Carranza-García, M., J. Torres-Mateo, P. Lara-Benítez, and J. García-Gutiérrez. 2021. “On the performance of one-stage and two-stage object detectors in autonomous vehicles using camera data.” Remote Sens. 13 (1): 89. https://doi.org/10.3390/rs13010089.
Czerniawski, T., and F. Leite. 2018. “3DFacilities: Annotated 3D reconstructions of building facilities.” Adv. Comput. Strategies Eng. 2018 (1): 186–200. https://doi.org/10.1007/978-3-319-91635-4_10.
Dai, J., H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei. 2017. “Deformable convolutional networks.” In Proc., IEEE Int. Conf. on Computer Vision (ICCV), 764–773. New York: IEEE.
Deng, J., W. Dong, R. Socher, L.-J. Li, Kai Li, and L. Fei-Fei. 2009. “ImageNet: A large-scale hierarchical image database.” In Proc., IEEE Conf. Computer Vision and Pattern Recognition, 248–255. New York: IEEE.
Duan, R., H. Deng, M. Tian, Y. Deng, and J. Lin. 2022. “SODA: A large-scale open site object detection dataset for deep learning in construction.” Autom. Constr. 142 (Oct): 104499. https://doi.org/10.1016/j.autcon.2022.104499.
Dutta, A., and A. Zisserman. 2019. “The VIA annotation software for images, audio and video.” In Proc., 27th ACM Int. Conf. on Multimedia, 2276–2279. New York: Association for Computing Machinery.
Ekanayake, B., A. Ahmadian Fard Fini, J. K. W. Wong, and P. Smith. 2022. “A deep learning-based approach to facilitate the as-built state recognition of indoor construction works.” Constr. Innov. 2022 (Dec): 20. https://doi.org/10.1108/CI-05-2022-0121.
Ekanayake, B., J. K. W. Wong, A. A. F. Fini, and P. Smith. 2021. “Computer vision-based interior construction progress monitoring: A literature review and future research directions.” Autom. Constr. 127 (Jul): 103705. https://doi.org/10.1016/j.autcon.2021.103705.
Everingham, M., L. van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. 2010. “The PASCAL visual object classes (VOC) challenge.” Int. J. Comput. Vis. 88 (Jun): 303–338. https://doi.org/10.1007/s11263-009-0275-4.
Everingham, M., L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. 2012. “The PASCAL visual object classes challenge 2012 (VOC2012) development kit.” Accessed July 8, 2022. http://host.robots.ox.ac.uk/pascal/VOC/voc2012/htmldoc/index.html.
Fang, W., B. Zhong, N. Zhao, P. E. D. Love, H. Luo, J. Xue, and S. Xu. 2019. “A deep learning-based approach for mitigating falls from height with computer vision: Convolutional neural network.” Adv. Eng. Inf. 39 (Jan): 170–177. https://doi.org/10.1016/j.aei.2018.12.005.
Girshick, R. 2015. “Fast R-CNN.” In Proc., 2015 IEEE Int. Conf. on Computer Vision (ICCV), 1440–1448. New York: IEEE.
Golkhoo, F. 2020. “Material management framework based on near real-time monitoring of construction operations.” Doctoral dissertation, Dept. of Building, Civil and Environmental Engineering, Concordia Univ.
Golparvar-Fard, M., J. Bohn, J. Teizer, S. Savarese, and F. Peña-Mora. 2011. “Evaluation of image-based modeling and laser scanning accuracy for emerging automated performance monitoring techniques.” Autom. Constr. 20 (8): 1143–1155. https://doi.org/10.1016/j.autcon.2011.04.016.
Hamledari, H., B. McCabe, and S. Davari. 2017. “Automated computer vision-based detection of components of under-construction indoor partitions.” Autom. Constr. 74 (Feb): 78–94. https://doi.org/10.1016/j.autcon.2016.11.009.
Hao, S., Y. Zhou, and Y. Guo. 2020. “A brief survey on semantic segmentation with deep learning.” Neurocomputing 406 (Sep): 302–321. https://doi.org/10.1016/j.neucom.2019.11.118.
He, K., G. Gkioxari, P. Dollar, and R. Girshick. 2017. “Mask R-CNN.” In Proc., IEEE Int. Conf. on Computer Vision, 2980–2988. New York: IEEE.
He, K., X. Zhang, S. Ren, and J. Sun. 2016. “Deep residual learning for image recognition.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 770–778. New York: IEEE.
Hou, X., Y. Zeng, and J. Xue. 2020. “Detecting structural components of building engineering based on deep-learning method.” J. Constr. Eng. Manage. 146 (2): 04019097. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001751.
Hwang, J., J. Kim, and S. Chi. 2023. “Site-optimized training image database development using web-crawled and synthetic images.” Autom. Constr. 151 (Jul): 104886. https://doi.org/10.1016/j.autcon.2023.104886.
Kim, J., J. Hwang, S. Chi, and J. O. Seo. 2020. “Towards database-free vision-based monitoring on construction sites: A deep active learning approach.” Autom. Constr. 120 (Dec): 103376. https://doi.org/10.1016/j.autcon.2020.103376.
Kim, J., D. Kim, S. H. Lee, and S. Chi. 2023. “Hybrid DNN training using both synthetic and real construction images to overcome training data shortage.” Autom. Constr. 149 (May): 104771. https://doi.org/10.1016/j.autcon.2023.104771.
Koo, B., and M. Fischer. 2000. “Feasibility study of 4D CAD in commercial construction.” J. Constr. Eng. Manage. 126 (4): 251. https://doi.org/10.1061/(ASCE)0733-9364(2000)126:4(251).
Kopsida, M., and I. Brilakis. 2020. “Real-time volume-to-plane comparison for mixed reality–based progress monitoring.” J. Comput. Civ. Eng. 34 (4): 04020016. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000896.
Korman, T. M., M. A. Fischer, and C. B. Tatum. 2003. “Knowledge and reasoning for MEP coordination.” J. Constr. Eng. Manage. 129 (6): 627. https://doi.org/10.1061/(ASCE)0733-9364(2003)129:6(627).
Li, Y., and J. Chen. 2022. “Computer vision–based counting model for dense steel pipe on construction sites.” J. Constr. Eng. Manage. 148 (1): 04021178. https://doi.org/10.1061/(ASCE)CO.1943-7862.0002217.
Lin, J. J., and M. Golparvar-Fard. 2020. “Visual and virtual progress monitoring in construction 4.0.” Construction 40 (1): 240–263. https://doi.org/10.1201/9780429398100-13.
Lin, T. Y., P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. 2017. “Feature pyramid networks for object detection.” In Proc., 30th IEEE Conf. Computer Vision and Pattern Recognition, 936–944. New York: IEEE.
Lin, T. Y., M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. 2014. “Microsoft COCO: Common objects in context.” In Proc., Computer Vision–ECCV 2014, 740–755. New York: Springer.
Liu, W., D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Y. Fu, and A. C. Berg. 2016. “SSD: Single shot MultiBox detector.” In Proc., Computer Vision.–ECCV 2016, 21–37. New York: Springer.
Man, K., and J. Chahl. 2022. “A review of synthetic image data and its use in computer vision.” J. Imaging 8 (11): 310. https://doi.org/10.3390/jimaging8110310.
Nath, N. D., and A. H. Behzadan. 2020. “Deep convolutional networks for construction object detection under different visual conditions.” Front. Built Environ. 6 (Aug): 97. https://doi.org/10.3389/fbuil.2020.00097.
Navon, R., and Y. Shpatnitsky. 2005. “Field experiments in automated monitoring of road construction.” J. Constr. Eng. Manage. 131 (4): 487. https://doi.org/10.1061/(ASCE)0733-9364(2005)131:4(487).
NIBS (National Institute of Building Sciences). 2017. “B. alliance-common building information model files and tools.” Accessed July 31, 2020. https://www.nibs.org/?page=bsa_commonbimfiles.
Niu, L., W. Cong, L. Liu, Y. Hong, B. Zhang, J. Liang, and L. Zhang. 2021. “Making images real again: A comprehensive survey on deep image composition.” Preprint, Submitted June 28, 2021. http://arxiv.org/abs/2106.14490.
O’Mahony, N., S. Campbell, A. Carvalho, S. Harapanahalli, G. V. Hernandez, L. Krpalkova, D. Riordan, and J. Walsh. 2019. “Deep learning vs. traditional computer vision.” In Proc., Computer Vision Conf., 128–144. New York: Springer.
Omar, T., and M. L. Nehdi. 2016. “Data acquisition technologies for construction progress tracking.” Autom. Constr. 70 (Jun): 143–155. https://doi.org/10.1016/j.autcon.2016.06.016.
Oquab, M., L. Bottou, I. Laptev, and J. Sivic. 2014. “Learning and transferring mid-level image representations using convolutional neural networks.” In Proc., IEEE Conf. Computer Vision and Pattern Recognition, 1717–1724. New York: IEEE.
Padilla, R., S. L. Netto, and E. A. B. Da Silva. 2020. “A survey on performance metrics for object-detection algorithms.” In Proc., Int. Conf. Systems, Signals, and Image Processing, 237–242. New York: IEEE.
Pal, A., and S. H. Hsieh. 2021. “Deep-learning-based visual data analytics for smart construction management.” Autom. Constr. 131 (Nov): 103892. https://doi.org/10.1016/j.autcon.2021.103892.
Paneru, S., and I. Jeelani. 2021. “Computer vision applications in construction: Current state, opportunities & challenges.” Autom. Constr. 132 (Dec): 103940. https://doi.org/10.1016/j.autcon.2021.103940.
Park, M. W., and I. Brilakis. 2016. “Continuous localization of construction workers via integration of detection and tracking.” Autom. Constr. 72 (Dec): 129–142. https://doi.org/10.1016/j.autcon.2016.08.039.
Paszke, A., et al. 2019. “PyTorch: An imperative style, high-performance deep learning library.” Accessed January 10, 2022. https://proceedings.neurips.cc/paper_files/paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf.
Poucin, F., A. Kraus, and M. Simon. 2021. “Boosting instance segmentation with synthetic data: A study to overcome the limits of real world data sets.” In Proc., IEEE Int. Conf. on Computer Vision (ICCV), 945–953. New York: IEEE.
Redmon, J., S. Divvala, R. Girshick, and A. Farhadi. 2016. “You only look once: Unified, real-time object detection.” In Proc., IEEE Computation Social Conf. Computer Vision and Pattern Recognition, 779–788. New York: IEEE.
Reja, V. K., K. Varghese, and Q. P. Ha. 2022. “Computer vision-based construction progress monitoring.” Autom. Constr. 138 (Jun): 104245. https://doi.org/10.1016/j.autcon.2022.104245.
Ren, S., K. He, R. Girshick, and J. Sun. 2015. “Faster R-CNN: Towards real-time object detection with region proposal networks.” In Proc., Advances in Neural Information Processing Systems (NIPS), 91–99. La Jolla, CA: Neural Information Processing Systems.
Seo, J., S. Han, S. Lee, and H. Kim. 2015. “Computer vision techniques for construction safety and health monitoring.” Adv. Eng. Inf. 29 (2): 239–251. https://doi.org/10.1016/j.aei.2015.02.001.
Shamsollahi, D., O. Moselhi, and K. Khorasani. 2021. “A timely object recognition method for construction using the Mask R-CNN architecture.” In Proc., 38th Int. Symp. Automation and Robotics in Construction (ISARC), 372–378. Bogotá, Colombia: IAARC.
Shamsollahi, D., O. Moselhi, and K. Khorasani. 2022. “Construction progress monitoring and reporting using computer vision techniques—A review.” In Proc., 38th Int. Symp. Automation and Robotics in Construction (ISARC), 467–474. Bogotá, Colombia: IAARC.
Shamsollahi, D., O. Moselhi, and K. Khorasani. 2023. “Integrated framework using computer vision and Ultra-Wide Band techniques for progress reporting in construction projects.” In Proc., Canadian Society of Civil Engineering Annual Conference (CSCE) 2023. Surrey, BC, Canada: CSCE.
Shekhar, S., P. Shukla, and B. B. Das. 2021. “Developing a standard template for activity linkage and resource estimation of MEP works.” Lect. Notes Civ. Eng. 105 (Aug): 485–496. https://doi.org/10.1007/978-981-15-8293-6_40.
Soltani, M. M., Z. Zhu, and A. Hammad. 2016. “Automated annotation for visual recognition of construction resources using synthetic images.” Autom. Constr. 62 (5): 14–23. https://doi.org/10.1016/j.autcon.2015.10.002.
Teizer, J. 2015. “Status quo and open challenges in vision-based sensing and tracking of temporary resources on infrastructure construction sites.” Adv. Eng. Inf. 29 (2): 225–238. https://doi.org/10.1016/j.aei.2015.03.006.
Teo, Y. H., J. H. Yap, H. An, S. C. M. Yu, L. Zhang, J. Chang, and K. H. Cheong. 2022. “Enhancing the MEP coordination process with BIM technology and management strategies.” Sensors 22 (13): 4936. https://doi.org/10.3390/s22134936.
Wang, J., Y. Ma, L. Zhang, R. X. Gao, and D. Wu. 2018. “Deep learning for smart manufacturing: Methods and applications.” J. Manuf. Syst. 48 (Jul): 144–156. https://doi.org/10.1016/j.jmsy.2018.01.003.
Wei, W., Y. Lu, Y. Lin, R. Bai, Y. Zhang, H. Wang, and P. Li. 2023. “Augmenting progress monitoring in soil-foundation construction utilizing SOLOv2-based instance segmentation and visual BIM representation.” Autom. Constr. 155 (Nov): 105048. https://doi.org/10.1016/j.autcon.2023.105048.
Wei, W., Y. Lu, T. Zhong, P. Li, and B. Liu. 2022. “Integrated vision-based automated progress monitoring of indoor construction using mask region-based convolutional neural networks and BIM.” Autom. Constr. 140 (Aug): 104327. https://doi.org/10.1016/j.autcon.2022.104327.
Wu, X., D. Sahoo, and S. C. H. Hoi. 2020. “Recent advances in deep learning for object detection.” Neurocomputing 396: 39–64. https://doi.org/10.1016/j.neucom.2020.01.085.
Xiao, B., and S.-C. Kang. 2021. “Development of an image data set of construction machines for deep learning object detection.” J. Comput. Civ. Eng. 35 (2): 05020005. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000945.
Xiao, B., Y. Zhang, Y. Chen, and X. Yin. 2021. “A semi-supervised learning detection method for vision-based monitoring of construction sites by integrating teacher-student networks and data augmentation.” Adv. Eng. Inf. 50 (Oct): 101372. https://doi.org/10.1016/j.aei.2021.101372.
Xuehui, A., Z. Li, L. Zuguang, W. Chengzhi, L. Pengfei, and L. Zhiwei. 2021. “Dataset and benchmark for detecting moving objects in construction sites.” Autom. Constr. 122 (Apr): 103482. https://doi.org/10.1016/j.autcon.2020.103482.
Yang, J., M. W. Park, P. A. Vela, and M. Golparvar-Fard. 2015. “Construction performance monitoring via still images, time-lapse photos, and video streams: Now, tomorrow, and the future.” Adv. Eng. Inf. 29 (2): 211–224. https://doi.org/10.1016/j.aei.2015.01.011.
Yang, J., Z. Shi, and Z. Wu. 2016. “Vision-based action recognition of construction workers using dense trajectories.” Adv. Eng. Inf. 30 (3): 327–336. https://doi.org/10.1016/j.aei.2016.04.009.
Yang, Q., E. Dong, and L. Zhu. 2020. “An instance segmentation algorithm based on Improved Mask R-CNN.” In Proc., 2020 Chinese Automation Congress, 4804–4809. New York: IEEE.
Zheng, Z., Z. Zhang, and W. Pan. 2020. “Virtual prototyping- and transfer learning-enabled module detection for modular integrated construction.” Autom. Constr. 120 (Aug): 103387. https://doi.org/10.1016/j.autcon.2020.103387.
Zhu, X., H. Hu, S. Lin, and J. Dai. 2019. “Deformable ConvNets V2: More deformable, better results.” In Proc., IEEE Computation Social Conf. Computer Vision and Pattern Recognition, 9300–9308. New York: IEEE.
Information & Authors
Information
Published In
Journal of Construction Engineering and Management
Volume 150 • Issue 8 • August 2024
Copyright
© 2024 American Society of Civil Engineers.
History
Received: Jan 30, 2024
Accepted: Mar 27, 2024
Published online: Jun 13, 2024
Published in print: Aug 1, 2024
Discussion open until: Nov 13, 2024
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.