Exploiting BIM Objects for Synthetic Data Generation toward Indoor Point Cloud Classification Using Deep Learning
Publication: Journal of Computing in Civil Engineering
Volume 36, Issue 6
Abstract
Advances in technology are leading to more and more devices integrating sensors capable of acquiring data quickly and with high accuracy. Point clouds are no exception. Therefore, there is increased research interest in the large amount of available light detection and ranging (LiDAR) data by point cloud classification using artificial intelligence. Nevertheless, point cloud labeling is a time-consuming task. Hence the amount of labeled data is still scarce. Data synthesis is gaining attention as an alternative to increase the volume of classified data. At the same time, the amount of Building Information Models (BIMs) provided by manufacturers on website databases is increasing. In line with these recent trends, this paper presents a deep-learning framework for classifying point cloud objects based on synthetic data sets created from BIM objects. The method starts by transforming BIM objects into point clouds deriving a data set consisting of 21 object classes characterized with various perturbation patterns. Then, the data set is split into four subsets to carry out the evaluation of synthetic data on the implemented flexible two-dimensional (2D) deep neural framework. In the latter, binary or greyscale images can be generated from point clouds by both orthographic or perspective projection to feed the network. Moreover, the surface variation feature was computed in order to aggregate more geometric information to images and to evaluate how it influences the object classification. The overall accuracy is over 85% in all tests when orthographic images are used. Also, the use of greyscale images representing surface variation improves performance in almost all tests although the computation of this feature may not be robust in point clouds with complex geometry or perturbations.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request. Data sets are available at https://dpv.uvigo.es/index.php/s/ywPWjr8cKc4P3dk.
Acknowledgments
This work has been partially supported by the European Association of International Cooperation Galicia–North of Portugal, program IACOBUS VI through a stay at Institute for Systems and Computer Engineering, Technology and Science (INESC TEC) to E. Frías (IACOBUS VII-35563149). This project has received funding from the Xunta de Galicia through project ED431C 2020/01, and from the Government of Spain through project PID2019-105221RB-C43 funded by MCIN/AEI/10.13039/501100011033 and through human resources Grant RYC2020-029193-I funded by MCIN/AEI/10.13039/501100011033 and FSE “El FSE invierte en tu futuro.” The open access fee has received funding from the University of Vigo/CISUG. This document reflects only the views of the authors. The statements made herein are solely the responsibility of the authors.
References
Abanda, F., and L. Byers. 2016. “An investigation of the impact of building orientation on energy consumption in a domestic building using emerging BIM (building information modelling).” Energy 97 (Feb): 517–527. https://doi.org/10.1016/j.energy.2015.12.135.
Abdel-Hamid, O., A. Mohamed, H. Jiang, and G. Penn. 2012. “Applying convolutional neural networks concepts to hybrid NN-HMM model for speech recognition.” In Proc., 2012 IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 4277–4280. New York: IEEE.
Alawadhi, M., and W. Yan. 2021. “BIM hyperreality: Data synthesis using BIM and hyperrealistic rendering for deep learning.” Preprint, submitted May 10, 2021. https://arxiv.org/abs/2105.04103.
Armeni, I., A. Sax, A. R. Zamir, and S. Savarese. 2017. “Joint 2D-3D-semantic data for indoor scene understanding.” Preprint, submitted February 3, 2017. https://arxiv.org/abs/1702.01105.
Balado, J., L. Díaz-Vilariño, E. Verbree, and P. Arias. 2020. “Transfer learning for indoor object classification: From images to point clouds.” ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci. 5 (Jun): 65–70. https://doi.org/10.5194/isprs-annals-V-4-2020-65-2020.
Bazazian, D., J. Casas, and J. Ruiz-Hidalgo. 2015. “Fast and robust edge extraction in unorganized point clouds.” In Proc., 2015 Int. Conf. on Digital Image Computing: Techniques and Applications (DICTA), 1–8. New York: IEEE.
Belford, M., B. MacNamee, and D. Greene. 2018. “Synthetic dataset generation for online topic modeling.” In Proc., 25th Irish Conf. on Artificial Intelligence and Cognitive Science, edited by J. McAuley and S. McKeever. Dublin, Ireland: Science Foundation Ireland.
Bim&Co. 2022. “Bim&co homepage.” Accessed January 18, 2022. https://www.bimandco.com.
Bimobject. 2022. “BIM object homepage.” Accessed January 15, 2022. https://www.bimobject.com.
Bosché, F., A. Guillemet, Y. Turkan, C. Haas, and R. Haas. 2014. “Tracking the built status of MEP works: Assessing the value of a scan-vs-BIM system.” J. Comput. Civ. Eng. 28 (4): 05014004. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000343.
Bowyer, K. W., N. V. Chawla, L. O. Hall, and W. P. Kegelmeyer. 2011. “SMOTE: Synthetic minority over-sampling technique.” Preprint, submitted June 9, 2011. https://arxiv.org/abs/1106.1813.
Broquetas, M., D. Bryde, and J. M. Volm. 2013. “The project benefits of building information modelling (BIM).” Int. J. Project Manage. 31 (7): 971–980. https://doi.org/10.1016/j.ijproman.2012.12.001.
Cano, I., and V. Torra. 2009. “Generation of synthetic data by means of fuzzy -regression.” In Proc., IEEE Int. Conf. on Fuzzy Systems. New York: IEEE.
Chang, A., A. Dai, T. Funkhouser, M. Halber, M. Niebner, M. Savva, S. Song, A. Zeng, and Y. Zhang. 2017. “Matterport3D: Learning from RGB-D data in indoor environments.” In Proc., 2017 Int. Conf. on 3D Vision (3DV), 667–676. Los Alamitos, CA: IEEE Computer Society. https://doi.org/10.1109/3DV.2017.00081.
Charles, R. Q., H. Su, M. Kaichun, and L. J. Guibas. 2017. “Pointnet: Deep learning on point sets for 3D classification and segmentation.” In Proc., 2017 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 77–85. New York: IEEE.
Chollet, F. 2017. “Xception: Deep learning with depthwise separable convolutions.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 1251–1258. New York: IEEE.
Dey, A. 2016. “Machine learning algorithms: A review.” Int. J. Comput. Sci. Inf. Technol. 7 (3): 1174–1179.
Goyal, A., H. Law, B. Liu, A. Newell, and J. Deng. 2021. “Revisiting point cloud shape classification with a simple and effective baseline.” Preprint, submitted June 9, 2021. https://arxiv.org/abs/2106.05304.
Griffiths, D., and J. Boehm. 2019. “A review on deep learning techniques for 3D sensed data classification.” Remote Sens. 11 (12): 1499. https://doi.org/10.3390/rs11121499.
Hajiaghayi, M., and E. Vahedi. 2019. “Code failure prediction and pattern extraction using LSTM networks.” In Proc., 2019 IEEE 5th Int. Conf. on Big Data Computing Service and Applications (BigDataService), 55–62. New York: IEEE.
Han, X.-F., J. S. Jin, M.-J. Wang, W. Jiang, L. Gao, and L. Xiao. 2017. “A review of algorithms for filtering the 3D point cloud.” Signal Process. Image Commun. 57 (Sep): 103–112. https://doi.org/10.1016/j.image.2017.05.009.
Hazan, E., A. Klivans, and Y. Yuan. 2017. “Hyperparameter optimization: A spectral approach.” Preprint, submitted June 2, 2017. https://arxiv.org/abs/1706.00764.
He, K., X. Zhang, S. Ren, and J. Sun. 2016. “Deep residual learning for image recognition.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 770–778. New York: IEEE.
Hearn, D., and M. Baker. 1997. Computer graphics, C version. London: Pearson Education.
Hermosilla, P., T. Ritschel, and T. Ropinski. 2019. “Total denoising: Unsupervised learning of 3D point cloud cleaning.” Preprint, submitted April 16, 2019. https://arxiv.org/abs/1904.07615.
Iglesias, F., D. Ojdanic, A. Hartl, and T. Zseby. 2020. “MDCStream: Stream data generator for testing analysis algorithms.” In Proc., 13th EAI Int. Conf. on Performance Evaluation Methodologies and Tools, 56–63. New York: Association for Computing Machinery. https://doi.org/10.1145/3388831.3388832.
Jaritz, M., J. Gu, and H. Su. 2019. “Multi-view pointnet for 3D scene understanding.” In Proc., IEEE/CVF Int. Conf. on Computer Vision (ICCV) Workshops. New York: IEEE.
Karim, R. 2020. “Illustrated: 10 CNN architectures.” Accessed July 29, 2019. https://towardsdatascience.com/illustrated-10-cnn-architectures-95d78ace614d.
Khoshelham, K., L. Díaz-Vilariño, M. Peter, Z. Kang, and D. Acharya. 2017. “The ISPRS benchmark on indoor modelling.” In Vol. 42 of Proc., ISPRS Geospatial Week 2017, edited by D. Li, 367–372. Hannover, Germany: International Society for Photogrammetry and Remote Sensing.
Krizhevsky, A., I. Sutskever, and G. E. Hinton. 2017. “Imagenet classification with deep convolutional neural networks.” Commun. ACM. 60 (6): 84–90. https://doi.org/10.1145/3065386.
Landrieu, L., and M. Simonovsky. 2018. “Large-scale point cloud semantic segmentation with superpoint graphs.” In Proc., Conf. on Computer Vision and Pattern Recognition, 4558–4567. New York: IEEE.
Le, T., and Y. Duan. 2018. “Pointgrid: A deep network for 3D shape understanding.” In Proc., Conf. on Computer Vision and Pattern Recognition, 9204–9214. New York: IEEE.
LeCun, Y., Y. Bengio, and G. Hinton. 2015. “Deep learning.” Nature 521 (7553): 436–444. https://doi.org/10.1038/nature14539.
LeCun, Y., B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. 1989. “Backpropagation applied to handwritten zip code recognition.” Neural Comput. 1 (4): 541–551. https://doi.org/10.1162/neco.1989.1.4.541.
Li, L., K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. 2016. “Hyperband: A novel bandit-based approach to hyperparameter optimization.” J. Mach. Learn. Res. 18 (1): 6765–6816. https://doi.org/10.48550/ARXIV.1603.06560.
Liu, W., Z. Wang, X. Liu, N. Zeng, Y. Liu, and F. E. Alsaadi. 2017. “A survey of deep neural network architectures and their applications.” Neurocomputing 234 (Apr): 11–26. https://doi.org/10.1016/j.neucom.2016.12.038.
Ma, J. W., T. Czerniawski, and F. Leite. 2020. “Semantic segmentation of point clouds of building interiors with deep learning: Augmenting training datasets with synthetic BIM-based point clouds.” Autom. Constr. 113 (May): 103144. https://doi.org/10.1016/j.autcon.2020.103144.
Maturana, D., and S. Scherer. 2015. “Voxnet: A 3D convolutional neural network for real-time object recognition.” In Proc., Int. Conf. on Intelligent Robots and Systems (IROS), 922–928. New York: IEEE.
NBS (National Building Specification). 2022. “NBS national BIM library.” Accessed January 15, 2022. https://www.nationalbimlibrary.com.
Pan, S. J., and Q. Yang. 2010. “A survey on transfer learning.” IEEE Trans. Knowl. Data Eng. 22 (10): 1345–1359. https://doi.org/10.1109/TKDE.2009.191.
Park, Y., and J.-M. Guldmann. 2019. “Creating 3D city models with building footprints and LiDAR point cloud classification: A machine learning approach.” Comput. Environ. Urban Syst. 75 (May): 76–89. https://doi.org/10.1016/j.compenvurbsys.2019.01.004.
Perez, L., and J. Wang. 2017. “The effectiveness of data augmentation in image classification using deep learning.” Preprint, submitted December 13, 2017. https://arxiv.org/abs/1712.04621.
Polantis. 2022. “Polantis: Thousands of free CAD and BIM objects.” Accessed January 15, 2022. https://www.polantis.com.
Qi, C. R., L. Yi, H. Su, and L. J. Guibas. 2017. “Pointnet++: Deep hierarchical feature learning on point sets in a metric space.” Preprint, submitted June 7, 2017. https://arxiv.org/abs/1706.02413.
Rakotosaona, M., V. L. Barbera, P. Guerrero, N. J. Mitra, and M. Ovsjanikov. 2019. “PointCleanNet: Learning to denoise and remove outliers from dense point clouds.” Preprint, submitted January 4, 2019. https://arxiv.org/abs/1901.01060.
Ruizhongtai Qi, C., H. Su, M. NieBner, A. Dai, M. Yan, and L. Guibas. 2016. “Volumetric and multi-view CNNs for object classification on 3D data.” In Proc., 2016 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 5648–5656. New York: IEEE.
Sánchez-Monedero, J., P. A. Gutiérrez, M. Pérez-Ortiz, and C. Hervás-Martínez. 2013. “An -spheres based synthetic data generator for supervised classification.” In Advances in computational intelligence, 613–621. Berlin: Springer.
Silberman, N., D. Hoiem, P. Kohli, and R. Fergus. 2012. “Indoor segmentation and support inference from RGBD images.” In Proc., Computer Vision—ECCV 2012, edited by A. Fitzgibbon, S. Lazebnik, P. Perona, Y. Sato, and C. Schmid, 746–760. Berlin: Springer.
Simonyan, K., and A. Zisserman. 2014. “Very deep convolutional networks for large-scale image recognition.” Preprint, submitted September 4, 2014. https://arxiv.org/abs/1409.1556.
Su, H., S. Maji, E. Kalogerakis, and E. Learned-Miller. 2015. “Multi-view convolutional neural networks for 3D shape recognition.” In Proc., IEEE Int. Conf. on Computer Vision, 945–953. New York: IEEE.
Szegedy, C., S. Ioffe, V. Vanhoucke, and A. A. Alemi. 2016a. “Inception-v4, inception-resnet and the impact of residual connections on learning.” Preprint, submitted February 23, 2016. https://arxiv.org/abs/1602.07261.
Szegedy, C., W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. 2014. “Going deeper with convolutions.” Preprint, submitted September 17, 2014. https://arxiv.org/abs/1409.4842.
Szegedy, C., W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. 2015. “Going deeper with convolutions.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 1–9. New York: IEEE.
Szegedy, C., V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. 2016b. “Rethinking the inception architecture for computer vision.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 2818–2826. New York: IEEE.
Tan, C., F. Sun, T. Kong, W. Zhang, C. Yang, and C. Liu. 2018. “A survey on deep transfer learning.” Preprint, submitted August 6, 2018. https://arxiv.org/abs/1808.01974.
Taylor, A., S. Leblanc, and N. Japkowicz. 2016. “Anomaly detection in automobile control network data with long short-term memory networks.” In Proc., 2016 IEEE Int. Conf. on Data Science and Advanced Analytics (DSAA), 130–139. New York: IEEE.
Tchapmi, L., C. Choy, I. Armeni, J. Gwak, and S. Savarese. 2017. “Segcloud: Semantic segmentation of 3D point clouds.” In Proc., 2017 Int. Conf. on 3D Vision (3DV), 537–547. Piscataway, NJ: IEEE. https://doi.org/10.48550/ARXIV.1710.07563.
Tomás, J. T., N. Spolaôr, E. A. Cherman, and M. C. Monard. 2014. “A framework to generate synthetic multi-label datasets.” Electron. Notes Theor. Comput. Sci. 302 (Feb): 155–176. https://doi.org/10.1016/j.entcs.2014.01.025.
Uy, M. A., Q.-H. Pham, B.-S. Hua, T. Nguyen, and S.-K. Yeung. 2019. “Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data.” In Proc., IEEE/CVF Int. Conf. on Computer Vision, 1588–1597. New York: IEEE.
Wang, J., W. Sun, W. Shou, X. Wang, C. Wu, H.-Y. Chong, Y. Liu, and C. Sun. 2014. “Integrating BIM and LiDAR for real-time construction quality control.” J. Intell. Rob. Syst. 79 (3): 417–432. https://doi.org/10.1007/s10846-014-0116-8.
Weiss, U., P. Biber, S. Laible, K. Bohlmann, and A. Zell. 2010. “Plant species classification using a 3D LiDAR sensor and machine learning.” In Proc., 2010 9th Int. Conf. on Machine Learning and Applications, 339–345. Washington, DC: IEEE Computer Society.
Wong, S. C., A. Gatt, V. Stamatescu, and M. D. McDonnell. 2016. “Understanding data augmentation for classification: When to warp?” Preprint, submitted September 28, 2016. https://arxiv.org/abs/1609.08764.
Wu, Z., S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 2015. “3D shapenets: A deep representation for volumetric shapes.” In Proc., 2015 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 1912–1920. New York: IEEE.
Xiao, J., A. Owens, and A. Torralba. 2013. “Sun3D: A database of big spaces reconstructed using SfM and object labels.” In Proc., 2013 IEEE Int. Conf. on Computer Vision, 1625–1632. New York: IEEE.
Yuksel, C. 2015. “Sample elimination for generating Poisson disk sample sets.” Comput. Graphics Forum 34 (2): 25–32. https://doi.org/10.1111/cgf.12538.
Zhou, Q.-Y., J. Park, and V. Koltun. 2018. “Open3D: A modern library for 3D data processing.” Preprint, submitted January 30, 2018. https://arxiv.org/abs/1609.08764.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 36 • Issue 6 • November 2022
Copyright
© 2022 American Society of Civil Engineers.
History
Received: Nov 30, 2021
Accepted: May 5, 2022
Published online: Aug 27, 2022
Published in print: Nov 1, 2022
Discussion open until: Jan 27, 2023
ASCE Technical Topics:
- Architectural engineering
- Artificial intelligence and machine learning
- Building information modeling
- Building management
- Computer networks
- Computer programming
- Computer vision and image processing
- Computing in civil engineering
- Design (by type)
- Engineering fundamentals
- Equipment and machinery
- Geometrics
- Highway and road design
- Methodology (by type)
- Neural networks
- Probe instruments
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Liu Yang, Yi-Chun Lin, Hubo Cai, Ayman Habib, From Scans to Parametric BIM: An Enhanced Framework Using Synthetic Data Augmentation and Parametric Modeling for Highway Bridges, Journal of Computing in Civil Engineering, 10.1061/JCCEE5.CPENG-5640, 38, 3, (2024).