Deep Learning–Based Automated Generation of Material Data with Object–Space Relationships for Scan to BIM
This article has been corrected.
VIEW CORRECTIONPublication: Journal of Management in Engineering
Volume 39, Issue 3
Abstract
Conventional scan to building information modeling (BIM) automation mainly deals with geometry. However, one of its limitations is the time it takes and the costs in generating material. Therefore, this study proposes an automated scan-to-BIM method considering both the geometry and material of building objects. It recognizes the geometry from a point cloud and the material from panorama images through deep learning–based semantic segmentation. The two extracted pieces of data are merged, and the BIM objects with material are automatically generated by using Dynamo. Here, the object–space relationships were applied to increase the accuracy of the material data to be included in the BIM object. As the result, the accuracy was improved by 48.66% compared with before the application. The proposed method can contribute to the improvement of the as-built BIM model usability because it can automatically generate a BIM model by reflecting the material, as well as the geometry of the existing building.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some of all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This research was supported by grants from the National Research Foundation of Korea funded by the Ministry of Education (NRF-2020R1A4A2002855) and Ministry of Science and ICT (NRF-2020R1A2C1010421) of the Korean government.
References
Ahmed, M., R. Seraj, and S. M. S. Islam. 2020. “The -means algorithm: A comprehensive survey and performance evaluation.” Electron 9 (8): 1295. https://doi.org/10.3390/electronics9081295.
Ali Khan, S., Y. Shi, M. Shahzad, and X. Xiang Zhu. 2020. “FGCN: Deep feature-based graph convolutional network for semantic segmentation of urban 3D point clouds.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition Workshops, 778–787. New York: IEEE. https://doi.org/10.1109/CVPRW50498.2020.00107.
Armeni, I., O. Sener, A. R. Zamir, H. Jiang, I. Brilakis, M. Fischer, and S. Savarese. 2016. “3D semantic parsing of large-scale indoor spaces.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 1534–1543. New York: IEEE. https://doi.org/10.1109/CVPR.2016.170.
Arora, P., and S. Varshney. 2016. “Analysis of -means and k-medoids algorithm for big data.” Procedia Comput. Sci. 78: 507–512. https://doi.org/10.1016/j.procs.2016.02.095.
Baumberg, A., and S. Gu. 2000. “Reliable feature matching across widely separated views 1. 1. Current approaches to stereo correspondence 1. 2. Current approaches to wide baseline stereo.” Current. https://doi.org/10.1109/CVPR.2000.855899.
Belsky, M., R. Sacks, and I. Brilakis. 2016. “Semantic enrichment for building information modeling.” Comput. Civ. Infrastruct. Eng. 31 (4): 261–274. https://doi.org/10.1111/mice.12128.
Besl, P. J., and N. D. McKay. 1992. “A method for registration of 3-D shapes.” IEEE Trans. Pattern Anal. Mach. Intell. 14 (2): 239–256. https://doi.org/10.1109/34.121791.
Bigda, K. C. 2012. “Interior finish, contents, and furnishings.” In Fire and life safety inspection manual. Edited by R. E. Solomon. Burlington, MA: Jones & Bartlett Learning.
Boufares, N., A. Motamedi, I. Iordanova, and D. Forgues. 2022. “Assessment of existing BIM implementation processes of a public organization to improve building assets management and maintenance.” In Vol. 1218 of Proc., IOP Conf. Series: Materials Science and Engineering, 012023. Bristol, UK: IOP. https://doi.org/10.1088/1757-899X/1218/1/012023.
Bryde, D., M. Broquetas, and J. M. Volm. 2013. “The project benefits of building information modelling (BIM).” Int. J. Project Manage. 31 (7): 971–980. https://doi.org/10.1016/j.ijproman.2012.12.001.
By, W., and B. Isakov. 2016. Developing a knowledge-based evaluation tool to support user selection. Toronto: Univ. of Toronto.
Chen, L., Q. Lu, and X. Zhao. 2022. “A semi-automatic image-based object recognition system for constructing as-is IFC BIM objects based on fuzzy-MAUT.” Int. J. Constr. Manage. 22 (1). https://doi.org/10.1080/15623599.2019.1615754.
Chen, Y., and G. Medioni. 1991. “Object modelling by registration of multiple range images.” Image Vision Comput. 10 (3): 145–155. https://doi.org/10.1016/0262-8856(92)90066-C.
da Silva, T. F. L., M. M. de Carvalho, and D. R. Vieira. 2022. “BIM critical-success factors in the design phase and risk management: Exploring knowledge and maturity mediating effect.” J. Constr. Eng. Manage. 148 (10): 04022104. https://doi.org/10.1061/(ASCE)CO.1943-7862.0002343.
Daszykowski, M., and B. Walczak. 2009. “Density-based clustering methods.” Compr. Chemom. 2: 635–654. https://doi.org/10.1016/B978-044452701-1.00067-3.
Deng, F., X. Zhu, and J. Ren. 2017. “Object detection on panoramic images based on deep learning.” In Proc., 3rd Int. Conf. on Control, Automation and Robotics, 375–380. New York: IEEE. https://doi.org/10.1109/ICCAR.2017.7942721.
Dimitrov, A., and M. Golparvar-Fard. 2014. “Vision-based material recognition for automated monitoring of construction progress and generating building information modeling from unordered site image collections.” Adv. Eng. Inf. 28 (1): 37–49. https://doi.org/10.1016/j.aei.2013.11.002.
Gan, V. J. L. 2022. “BIM-based building geometric modeling and automatic generative design for sustainable offsite construction.” J. Constr. Eng. Manage. 148 (10): 04022111. https://doi.org/10.1061/(ASCE)CO.1943-7862.0002369.
Ghadai, S., X. Y. Lee, A. Balu, S. Sarkar, and A. Krishnamurthy. 2019. “Multi-level 3D CNN for learning multi-scale spatial features.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition Workshops, 1152–1156. New York: IEEE. https://doi.org/10.1109/CVPRW.2019.00150.
Grilli, E., F. Menna, and F. Remondino. 2017. “A review of point clouds segmentation and classification algorithms.” Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. - ISPRS Arch. XLII-2/W3 (2W3): 339–344. https://doi.org/10.5194/isprs-archives-XLII-2-W3-339-2017.
Hossin, M., and M. N. Sulaiman. 2015. “A review on evaluation metrics for data classification evaluations.” Int. J. Data Min. Knowledge Manage. Process 5 (2): 1. https://doi.org/10.5121/ijdkp.2015.5201.
Ilter, D., and E. Ergen. 2015. “BIM for building refurbishment and maintenance: Current status and research directions.” Struct. Surv. 33 (3): 228–256. https://doi.org/10.1108/SS-02-2015-0008.
Jönsson, Å., A. M. Tillman, and T. Svensson. 1997. “Life cycle assessment of flooring materials: Case study.” Build. Environ. 32 (3): 245–255. https://doi.org/10.1016/S0360-1323(96)00052-2.
Jung, J., C. Stachniss, S. Ju, and J. Heo. 2018. “Automated 3D volumetric reconstruction of multiple-room building interiors for as-built BIM.” Adv. Eng. Inf. 38 (Oct): 811–825. https://doi.org/10.1016/j.aei.2018.10.007.
Kalkan, E., F. Y. Okur, and A. C. Altunışık. 2018. “Applications and usability of parametric modeling.” J. Constr. Eng. Manage. Innov. 1 (3): 139–146. https://doi.org/10.31462/jcemi.2018.03139146.
Kisantal, M., Z. Wojna, J. Murawski, J. Naruniec, and K. Cho. 2019. “Augmentation for small object detection.” Preprint, submitted Feburary 19, 2019. https://arxiv.org/abs/1902.07296.
Koo, H. J., and J. T. O’Connor. 2022. “A strategy for building design quality improvement through BIM capability analysis.” J. Constr. Eng. Manage. 148 (8): 04022066. https://doi.org/10.1061/(ASCE)CO.1943-7862.0002318.
Kovacic, I., and M. Honic. 2021. “Scanning and data capturing for bim-supported resources assessment: A case study.” J. Inf. Technol. Constr. 26 (32): 624–638. https://doi.org/10.36680/j.itcon.2021.032.
Kumar, K. S. C., and S. Al-Stouhi. 2021. “Multi-scale voxel class balanced ASPP for LIDAR pointcloud semantic segmentation.” In Proc., 2021 IEEE Winter Conf. Applications of Computer Vision. WACVW 2021, 117–124. New York: IEEE. https://doi.org/10.1109/WACVW52041.2021.00017.
Li, M., D. Meng, S. Gu, and S. Liu. 2016. “Research and improvement of DBSCAN cluster algorithm.” In Proc., 2015 7th Int. Conf. on Information Technology in Medicine and Education, ITME 2015, 537–540. New York: IEEE. https://doi.org/10.1109/ITME.2015.100.
Liao, W., Y. Liu, and A. Choudhary. 2004. “A grid-based clustering algorithm using adaptive mesh refinement.” In Proc., 7th Workshop on Mining Scientific and Engineering Datasets of SIAM Int. Conf. on Data Mining, 61–69. Livermore, CA: Lawrence Livermore National Laboratory.
Lu, Q., S. Lee, and L. Chen. 2018. “Image-driven fuzzy-based system to construct as-is IFC BIM objects.” Autom. Constr. 92 (Aug): 68–87. https://doi.org/10.1016/j.autcon.2018.03.034.
Macher, H., T. Landes, and P. Grussenmeyer. 2017. “From point clouds to building information models: 3D semi-automatic reconstruction of indoors of existing buildings.” Appl. Sci. 7 (10): 1030. https://doi.org/10.3390/app7101030.
Minu, R. I., and K. K. Thyagharajan. 2014. “Semantic rule based image visual feature ontology creation.” Int. J. Autom. Comput. 11 (5): 489–499. https://doi.org/10.1007/s11633-014-0832-3.
Nagai, T., T. Matsushima, K. Koida, Y. Tani, M. Kitazaki, and S. Nakauchi. 2015. “Temporal properties of material categorization and material rating: Visual vs non-visual material features.” Vision Res. 115 (Part B): 259–270. https://doi.org/10.1016/j.visres.2014.12.011.
Nakagawa, M. 2018. “Point cloud clustering using panoramic layered range image.” In Recent applications in data clustering, 75–92. London: IntechOpen. https://doi.org/10.5772/intechopen.76407.
Park, J., J. Kim, D. Lee, K. Jeong, J. Lee, H. Kim, and T. Hong. 2022. “Deep learning–based automation of scan-to-BIM with modeling objects from occluded point clouds.” J. Manage. Eng. 38 (4): 04022025. https://doi.org/10.1061/(ASCE)ME.1943-5479.0001055.
Patiño-Cambeiro, F., G. Bastos, J. Armesto, and F. Patiño-Barbeito. 2017. “Multidisciplinary energy assessment of tertiary buildings: Automated geomatic inspection, building information modeling reconstruction and building performance simulation.” Energies 10 (7): 1032. https://doi.org/10.3390/en10071032.
Peeran, S., M. V. Reddy, P. R. Rani, and S. Devi. 2015. “Consumer preferences on the choice of flooring material for residential buildings in Hyderabad city.” Int. J. Appl. Res. 1 (13): 660–662.
Pishdad-Bozorgi, P. 2017. “Future smart facilities: State-of-the-art BIM-enabled facility management.” J. Constr. Eng. Manage. 143 (9): 02517006. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001376.
Qi, C. R., H. Su, K. Mo, and L. J. Guibas. 2017. “PointNet: Deep learning on point sets for 3D classification and segmentation.” In Proc., 30th IEEE Conf. on Computer Vision and Pattern Recognition, CVPR 2017. New York: IEEE.
Ramaji, I. J., J. I. Messner, and E. Mostavi. 2020. “IFC-based BIM-to-BEM model transformation.” J. Comput. Civ. Eng. 34 (3): 04020005. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000880.
Rausch, C., Y. Zhao, and C. Haas. 2020. “Parametric or Non-parametric? Understanding the inherent trade-offs between forms of object representation.” In Proc., 37th Int. Symp. on Automation and Robotics in Construction, 248–255. Oulu, Finland: IAARC. https://doi.org/10.22260/isarc2020/0036.
Roldao, L., R. De Charette, and A. Verroust-Blondet. 2019. “3D surface reconstruction from voxel-based lidar data.” In Proc., 2019 IEEE Intelligent Transportation Systems Conf. (ITSC), 2681–2686. New York: IEEE. https://doi.org/10.1109/ITSC.2019.8916881.
Sarode, V., X. Li, H. Goforth, Y. Aoki, R. A. Srivatsan, S. Lucey, and H. Choset. 2019. “PCRNet: Point cloud registration network using pointnet encoding.” Preprint, submitted August 21, 2019. https://arxiv.org/abs/1908.07906.
Shahi, K., B. Y. McCabe, and A. Shahi. 2019. “Framework for automated model-based e-permitting system for municipal jurisdictions.” J. Manage. Eng. 35 (6): 04019025. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000712.
Shorten, C., and T. M. Khoshgoftaar. 2019. “A survey on image data augmentation for deep learning.” J. Big Data 6 (1): 1–48. https://doi.org/10.1186/s40537-019-0197-0.
Svirsky, Y., and A. Sharf. 2021. “A non-linear differentiable CNN-rendering module for 3D data enhancement.” IEEE Trans. Vis. Comput. Graph. 27 (7): 3238–3249. https://doi.org/10.1109/TVCG.2020.2968062.
Tang, P., D. Huber, B. Akinci, R. Lipman, and A. Lytle. 2010. “Automatic reconstruction of as-built building information models from laser-scanned point clouds: A review of related techniques.” Autom. Constr. 19 (7): 829–843. https://doi.org/10.1016/j.autcon.2010.06.007.
Tayeh, R., and R. R. A. Issa. 2020. “Interactive holograms for construction coordination and quantification.” J. Manage. Eng. 36 (6): 04020079. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000847.
Thomson, C., and J. Boehm. 2015. “Automatic geometry generation from point clouds for BIM.” Remote Sens. 7 (9): 11753–11775. https://doi.org/10.3390/rs70911753.
Valero, E., A. Forster, F. Bosché, E. Hyslop, L. Wilson, and A. Turmel. 2019. “Automated defect detection and classification in ashlar masonry walls using machine learning.” Autom. Constr. 106 (Oct): 102846. https://doi.org/10.1016/j.autcon.2019.102846.
Volk, R., J. Stengel, and F. Schultmann. 2014. “Building information modeling (BIM) for existing buildings—Literature review and future needs.” Autom. Constr. 38 (Mar): 109–127. https://doi.org/10.1016/j.autcon.2013.10.023.
Wada, K. 2016. “Labelme: Image polygonal annotation with Python.” Accessed February 5, 2022. https://github.com/wkentaro/labelme.
Wang, B., C. Yin, H. Luo, J. C. P. Cheng, and Q. Wang. 2021a. “Fully automated generation of parametric BIM for MEP scenes based on terrestrial laser scanning data.” Autom. Constr. 125 (May): 103615. https://doi.org/10.1016/j.autcon.2021.103615.
Wang, L., D. Li, H. Liu, J. Peng, L. Tian, and Y. Shan. 2021b. “Cross-dataset collaborative learning for semantic segmentation in autonomous driving.” In Vol. 36 of Proc., AAAI Conf. on Artificial Intelligence, 2487–2494. Palo Alto, CA: Association for the Advancement of Artificial Intelligence.
Wang, X. 2012. BIM Handbook: A guide to building information modeling for owners, managers, designers, engineers and contractors. New York: Wiley.
Xu, Y., K. Wang, K. Yang, D. Sun, and J. Fu. 2019. “Semantic segmentation of panoramic images using a synthetic dataset.” In Vol. 11169 of Artificial intelligence and machine learning in defense applications, 90–104. Washington, DC: SPIE.
Yang, J., Z. K. Shi, and Z. Y. Wu. 2016. “Towards automatic generation of as-built BIM: 3D building facade modeling and material recognition from images.” Int. J. Autom. Comput. 13 (4): 338–349. https://doi.org/10.1007/s11633-016-0965-7.
Yang, L., J. C. P. Cheng, and Q. Wang. 2020. “Semi-automated generation of parametric BIM for steel structures based on terrestrial laser scanning data.” Autom. Constr. 112 (Apr): 103037. https://doi.org/10.1016/j.autcon.2019.103037.
Yüksek, Í. 2015. “The evaluation of building materials in terms of energy efficiency.” Period. Polytech. Civ. Eng. 59 (1): 45–58. https://doi.org/10.3311/PPci.7050.
Zabin, A., B. Khalil, T. Ali, J. A. Abdalla, and A. Elaksher. 2020. “A semi-automated method for integrating textural and material data into as-built BIM using TIS.” Adv. Comput. Des. 5 (2): 127–146. https://doi.org/10.12989/acd.2020.5.2.127.
Zhang, Y., S. Song, P. Tan, and J. Xiao. 2014. “PanoContext: A whole-room 3D context model for panoramic scene understanding.” In Proc., European Conf. on Computer Vision, 668–686. Cham, Switzerland: Springer.
Zhao, C., L. Sun, and R. Stolkin. 2017. “A fully end-to-end deep learning approach for real-time simultaneous 3D reconstruction and material recognition.” In Proc., 18th Int. Conf. on Advanced Robotics ICAR 2017, 75–82. New York: IEEE. https://doi.org/10.1109/ICAR.2017.8023499.
Zhou, Y., L. Wang, Y. Xu, L. Ding, and Z. Tang. 2022. “Intelligent Fangcang shelter hospital systems for major public health emergencies: The case of the Optics Valley Fangcang Shelter Hospital.” J. Manage. Eng. 38 (1): 1–13. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000976.
Information & Authors
Information
Published In
Journal of Management in Engineering
Volume 39 • Issue 3 • May 2023
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Jun 25, 2022
Accepted: Dec 1, 2022
Published online: Feb 3, 2023
Published in print: May 1, 2023
Discussion open until: Jul 3, 2023
ASCE Technical Topics:
- Architectural engineering
- Benefit cost ratios
- Building information modeling
- Building management
- Buildings
- Business management
- Computer networks
- Computer vision and image processing
- Computing in civil engineering
- Construction engineering
- Construction materials
- Construction methods
- Design (by type)
- Engineering fundamentals
- Engineering materials (by type)
- Financial management
- Geometrics
- Highway and road design
- Materials engineering
- Methodology (by type)
- Practice and Profession
- Structural engineering
- Structures (by type)
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Yejin Shin, Yujin Choi, Jaeseung Won, Taehoon Hong, Choongwan Koo, A New Benchmark Model for the Automated Detection and Classification of a Wide Range of Heavy Construction Equipment, Journal of Management in Engineering, 10.1061/JMENEA.MEENG-5630, 40, 2, (2024).
- Sohyun Kim, Hyeonggyun Kim, Jaewook Lee, Taehoon Hong, Kwangbok Jeong, An Integrated Assessment Framework of Economic, Environmental, and Human Health Impacts Using Scan-to-BIM and Life-Cycle Assessment in Existing Buildings, Journal of Management in Engineering, 10.1061/JMENEA.MEENG-5600, 39, 5, (2023).