Instance Segmentation of Industrial Point Cloud Data
Publication: Journal of Computing in Civil Engineering
Volume 35, Issue 6
Abstract
The challenge that this paper addresses is how to efficiently minimize the cost and manual labor for automatically generating object oriented geometric digital twins (gDTs) of industrial facilities, so that the benefits provide even more value compared to the initial investment to generate these models. Our previous work achieved the current state-of-the-art class segmentation performance (75% average accuracy per point and average area under the ROC curve, AUC, 90% in the CLOI dataset classes) and directly produces labelled point clusters of the most important to model objects (CLOI classes) from laser scanned industrial data. CLOI stands for C-shapes, L-shapes, O-shapes, I-shapes and their combinations. However, the problem of automated segmentation of individual instances that can then be used to fit geometric shapes remains unsolved. We argue that the use of instance segmentation algorithms has the theoretical potential to provide the output needed for the generation of gDTs. We solve instance segmentation in this paper through (1) using a CLOI-Instance graph connectivity algorithm that segments the point clusters of an object class into instances, and (2) boundary segmentation of points that improves Step 1. Our method was tested on the CLOI benchmark dataset and segmented instances with 76.25% average precision and 70% average recall per point among all classes. This proved that it is the first to automatically segment industrial point cloud shapes with no prior knowledge other than the class point label and is the bedrock for efficient gDT generation in cluttered industrial point clouds.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code used during the study were provided by a third party. Direct requests for these materials (the datasets used for the evaluation of the methods in the work) may be made to the provider as indicated in the Acknowledgements.
Acknowledgments
We thank our colleague Graham Miatt, who has provided insight, expertise, and data that greatly assisted this research. We also express gratitude to Bob Flint from BP International Centre for Business and Technology (ICBT), who provided data for evaluation. The research leading to these results has received funding from the Engineering and Physical Sciences Research Council (EPSRC) and the US National Academy of Engineering (NAE). AVEVA Group Plc. and BP International Centre for Business and Technology (ICBT) partially sponsor this research under grant agreements RG83104 and RG90532, respectively. We gratefully acknowledge the collaboration of all academic and industrial project partners. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the institutes mentioned above.
References
Agapaki, E. 2020. “Automated object segmentation in existing industrial facilities.” Ph.D. thesis, Dept. of Engineering, Univ. of Cambridge.
Agapaki, E., and I. Brilakis. 2017. “Prioritising object types of industrial facilities to reduce as-is modelling time.” In Proc., 33rd Annual ARCOM Conf., 402–411. Glasgow, UK: Association of Researchers in Construction Management.
Agapaki, E., and I. Brilakis. 2018. “State-of-practice on as-is modelling of industrial facilities.” In Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics). Plymouth, UK: European Group for Intelligent Computing in Engineering.
Agapaki, E., and I. Brilakis. 2020. “Cloi-net: Class segmentation of industrial facilities’ point cloud datasets.” Adv. Eng. Inf. 45 (Aug): 101121. https://doi.org/10.1016/j.aei.2020.101121.
Agapaki, E., A. Glyn-Davies, S. Mandoki, and I. Brilakis. 2019. “CLOI: A shape classification benchmark dataset for industrial facilities.” In Proc., 2019 ASCE Int. Conf. on Computing in Civil Engineering. Reston, VA: ASCE.
Agapaki, E., G. Miatt, and I. Brilakis. 2018. “Prioritizing object types for modelling existing industrial facilities.” Autom. Constr. 96 (Dec): 211–223. https://doi.org/10.1016/j.autcon.2018.09.011.
Agapaki, E., and M. Nahangi. 2020. “Scene understanding and model generation.” In Infrastructure computer vision, 65–167. Amsterdam, Netherlands: Elsevier.
Armeni, I., O. Sener, H. Jiang, M. Fischer, and S. Savarese. 2016. “3D semantic parsing of large-scale indoor spaces.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 1534–1543. New York: IEEE.
Arshad, S., M. Shahzad, Q. Riaz, and M. M. Fraz. 2019. “Dprnet: Deep 3d point based residual network for semantic segmentation and classification of 3d point clouds.” IEEE Access 7: 68892–68904. https://doi.org/10.1109/ACCESS.2019.2918862.
Bauer, F. L., and H. Wössner. 1972. “The ‘Plankalkül’ of Konrad Zuse: A forerunner of today’s programming languages.” Commun. ACM 15 (7): 678–685. https://doi.org/10.1145/361454.361515.
Beale, R., C. Tech, P. Bowers, and P. Smith. 2010. Process piping design handbook: The planning guide to piping design. 3rd ed. Houston: Gulf Publishing Company.
Behley, J., V. Steinhage, and A. Cremers. 2012. “Performance of histogram descriptors for the classification of 3D laser range data in urban environments.” In Proc., IEEE Int. Conf. on Robotics and Automation (ICRA). New York: IEEE.
Bokrantz, J., A. Skoogh, T. Ylipää, and J. Stahre. 2016. “Handling of production disturbances in the manufacturing industry.” J. Manuf. Technol. Manage. 27 (8): 1054–1075. https://doi.org/10.1108/JMTM-02-2016-0023.
Brilakis, I., and C. T. M. Haas. 2019. Infrastructure computer vision. Oxford, UK: Butterworth-Heinemann.
BuildingSMART. 2018. “International home of openBIM.” Accessed April 16, 2021. https://www.buildingsmart-tech.org.
Che, E., J. Jung, and M. J. Olsen. 2019. “Object recognition, segmentation, and classification of mobile laser scanning point clouds: A state of the art review.” Sensors 19 (4): 810. https://doi.org/10.3390/s19040810.
ClearEdge. 2019. “Plant modeling capabilities.” Accessed April 16, 2021. https://www.clearedge3d.com/products/edgewise-plant/.
Climate Policy Initiative Berlin. 2011. Thermal efficiency retrofit of residential buildings: The German experience. Berlin: DIW Berlin.
Comaniciu, D., and P. Meer. 2002. “Mean shift: A robust approach toward feature space analysis.” In Proc., IEEE Trans. Pattern Anal. Mach. Intell. New York: IEEE.
Czerniawski, T., and F. Leite. 2020. “Automated digital modeling of existing buildings: A review of visual object recognition methods.” Autom. Constr. 113 (May): 103131. https://doi.org/10.1016/j.autcon.2020.103131.
Deng, Z., and L. J. Latecki. 2017. “Amodal detection of 3D objects: Inferring 3D bounding boxes from 2D ones in RGB-depth images.” In Proc., 30th IEEE Conf. on Computer Vision and Pattern Recognition, CVPR 2017. New York: IEEE.
Dimitrov, A., and M. Golparvar-Fard. 2015. “Segmentation of building point cloud models including detailed architectural/structural features and MEP systems.” Autom. Constr. 51 (Mar): 32–45. https://doi.org/10.1016/j.autcon.2014.12.015.
Dimitrov, A., R. Gu, and M. Golparvar-Fard. 2016. “Non-uniform b-spline surface fitting from unordered 3D point clouds for as-built modeling.” Comput.-Aided Civ. Infrastruct. Eng. 31 (7): 483–498. https://doi.org/10.1111/mice.12192.
Edwards, J., and A. Townsend. 2011. “Buildings under refurbishment and retrofit.” Accessed April 16, 2021. https://www.carbonaction2050.com/sites/carbonaction2050.com/files/document-attachment/BuildingsunderRefurbandRetrofit.pdf.
Elich, C., F. Engelmann, J. Schult, T. Kontogianni, and B. Leibe. 2019. “3D-BEVIS: Birds-eye-view instance segmentation.” Accessed April 16, 2021. https://arxiv.org/pdf/1904.02199.pdf.
Fumarola, M., and R. Poelman. 2011. “Generating virtual environments of real world facilities: Discussing four different approaches.” Autom. Constr. 20 (3): 263–269. https://doi.org/10.1016/j.autcon.2010.08.004.
Geiger, A., P. Lenz, and R. Urtasun. 2012. “Are we ready for autonomous driving? the KITTI vision benchmark suite.” In Proc., IEEE Computer Society Conf. on Computer Vision and Pattern Recognition. New York: IEEE.
Girshick, R., J. Donahue, T. Darrell, and J. Malik. 2014. “Rich feature hierarchies for accurate object detection and semantic segmentation.” In Proc., IEEE Computer Society Conf. on Computer Vision and Pattern Recognition. New York: IEEE.
Hackel, T., N. Savinov, L. Ladicky, J. D. Wegner, K. Schindler, and M. Pollefeys. 2017. “SEMANTIC3D.NET: A new large-scale point cloud classification benchmark.” In Proc., ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences. Amsterdam, Netherlands: Elsevier.
Hariharan, B., P. Arbeláez, R. Girshick, and J. Malik. 2014. “Simultaneous detection and segmentation.” In Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics). Cham, Switzerland: Springer.
He, K., P. Gkioxari, P. Dollar, and R. Girshick. 2017. “Mask R-CNN.” In Proc., Int. Conf. on Computer Vision. New York: IEEE.
Hensley, D. L. 2017. “Dav’s bending book.” Accessed April 16, 2021. https://lu903.com/wp-content/uploads/2017/06/Bending-Book-rev-2-full.pdf.
Hou, J., A. Dai, and M. Niebner. 2019. “3D-SIS: 3D semantic instance segmentation of RGB-D scans.” Preprint, submitted December 17, 2018. https://arxiv.org/abs/1812.07003.
Huang, J., et al. 2017. “Speed/accuracy trade-offs for modern convolutional object detectors.” In Proc., 30th IEEE Conf. on Computer Vision and Pattern Recognition, CVPR 2017. New York: IEEE.
Hullo, J.-F., G. Thibault, C. Boucheny, F. Dory, and A. Mas. 2015. “Multi-sensor as-built models of complex industrial architectures.” Remote Sens. 7 (12): 16339–16362. https://doi.org/10.3390/rs71215827.
Ibrahim, Z., H. Aris, and A. Mansur. 2018. “Automated validation of crowdsourced data.” In Proc., 16th Student Conf. on Research and Development (SCOReD). New York: IEEE.
ISO. 2004. Industrial automation systems and integration—Product data representation and exchange. Part 11: Description methods: The express language reference manual. ISO 10303-11. Geneva: ISO.
Ketchen, D. J., and C. L. Shook. 1996. “The application of cluster analysis in strategic management research: An analysis and critique.” Strategic Manage. J. 17 (6): 441–458. https://doi.org/10.1002/(SICI)1097-0266(199606)17:6%3C441::AID-SMJ819%3E3.0.CO;2-G.
Li, B., Y. Shi, Z. Qi, and Z. Chen. 2019. “A survey on semantic segmentation.” In Proc., IEEE Int. Conf. on Data Mining Workshops, ICDMW. New York: IEEE.
Li, D., and C. Feng. 2019. “Primitive fitting using deep geometric segmentation.” In Proc., 36th Int. Symp. on Automation and Robotics in Construction, ISARC 2019. Edinburgh, UK: International Association for Automation and Robotics in Construction.
Liang, M., B. Yang, S. Wang, and R. Urtasun. 2018. “Deep continuous fusion for multi-sensor 3D object detection.” In Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics). New York: Computer Vision Foundation.
Liang, Z., M. Yang, L. Deng, C. Wang, and B. Wang. 2019. “Hierarchical depthwise graph convolutional neural network for 3d semantic segmentation of point clouds.” In Proc., 2019 Int. Conf. on Robotics and Automation (ICRA), 8152–8158. New York: IEEE.
Liu, S., C. Chen, Y. Lu, F. Ouyang, and B. Wang. 2019. “An interactive method to improve crowdsourced annotations.” In Proc., IEEE Transactions on Visualization and Computer Graphics New York: IEEE.
Lu, Q., C. Chen, W. Xie, and Y. Luo. 2020. “Pointngcnn: Deep convolutional networks on 3d point clouds with neighborhood graph filters.” Comput. Graphics 86 (Feb): 42–51. https://doi.org/10.1016/j.cag.2019.11.005.
Lu, R., and I. Brilakis. 2017. “Recursive segmentation for as-is bridge information modelling.” In Proc., Joint Conf. on Computing in Construction (JC3), 209–217. Edinburgh, UK: Heriot-Watt Univ.
Ma, L., R. Sacks, U. Kattel, and T. Bloch. 2018. “3D Object Classification Using Geometric Features and Pairwise Relationships.” Comput.-Aided Civ. Infrastruct. Eng. 33 (2): 152–164. https://doi.org/10.1111/mice.12336.
Maneewongvatana, S., and D. Mount. 2002. “Analysis of approximate nearest neighbor searching with clustered point sets.” In Vol. 59 of Data structures, near neighbor searches, and methodology, 105–123. Providence, RI: American Mathematical Society.
Munoz, D., J. A. Bagnell, N. Vandapel, and M. Hebert. 2009. “Contextual classification with functional max-margin markov networks.” In Proc., 2009 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition Workshops, 2009. New York: IEEE.
Newell, A., Z. Huang, and J. Deng. 2017. “Associative embedding: End-to-end learning for joint detection and grouping.” Preprint, submitted November 16, 2016. https://arxiv.org/abs/1611.05424.
Newell, A., K. Yang, and J. Deng. 2016. “Stacked hourglass networks for human pose estimation.” In Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics). New York: Computer Vision Foundation.
NIST. 2018. The costs and benefits of advanced maintenance in manufacturing. Gaithersburg, MD: US Department of Commerce.
Perez-Perez, Y., M. Golparvar-Fard, and K. El-Rayes. 2016. “Semantic and geometric labeling for enhanced 3D point cloud segmentation.” In Proc., Construction Research Congress 2016, 2542–2552. Reston, VA: ASCE.
Perez-Perez, Y., M. Golparvar-Fard, and K. El-Rayes. 2017. “Semantic-rich 3d cad models for built environments from point clouds: An end-to-end procedure.” In Proc., Computing in Civil Engineering 2017, 166–174. Reston, VA: ASCE.
Pham, Q., D. T. Nguyen, B. Hua, G. Roig, and S. Yeung. 2019. “JSIS3D: Joint semantic-instance segmentation of 3D point clouds with multi-task pointwise networks and multi-value conditional random fields.” Preprint, submitted April 1, 2019. https://arxiv.org/abs/1904.00699.
Qi, C. R., W. Liu, C. Wu, H. Su, and L. J. Guibas. 2018. “Frustum PointNets for 3D object detection from RGB-D data.” In Proc., IEEE Computer Society Conf. on Computer Vision and Pattern Recognition. New York: IEEE.
Qi, C. R., L. Yi, H. Su, and L. J. Guibas. 2017a. “PointNet++: Deep hierarchical feature learning on point sets in a metric space.” In Proc., Computer Vision and Pattern Recognition (CVPR). New York: IEEE.
Qi, R., H. Su, K. Mo, and L. J. Guibas. 2017b. “PointNET: Deep learning on point sets for 3D classification and segmentation.” In Proc., Computer Vision and Pattern Recognition (CVPR). New York: IEEE.
Rethage, D., J. Wald, J. Sturm, N. Navab, and F. Tombari. 2018. “Fully-convolutional point networks for large-scale point clouds.” In Proc., European Conf. on Computer Vision (ECCV), 596–611. New York: Computer Vision Foundation.
Roynard, X., J.-E. Deschaud, and F. Goulette. 2018. Paris-Lille-3D: A point cloud dataset for urban scene segmentation and classification. New York: IEEE.
Sacks, R., M. Girolami, and I. Brilakis. 2020. “Building information modelling, artificial intelligence and construction tech.” Dev. Built Environ. 4 (Nov): 100011. https://doi.org/10.1016/j.dibe.2020.100011.
Steder, B., G. Grisetti, and W. Burgard. 2010. “Robust place recognition for 3D range data based on point features.” In Proc., IEEE Intl. Conf. on Robotics and Automation (ICRA). New York: IEEE.
US Energy Information Administration. 2006. 2003 commercial buildings energy consumption survey. Washington, DC: Energy Information Administration.
Wang, W., R. Yu, Q. Huang, and U. Neumann. 2018. “SGPN: Similarity group proposal network for 3D point cloud instance segmentation.” In Proc., Computer Vision and Pattern Recognition. New York: IEEE.
Wang, X., X. Shen, C. Shen, and J. Jia. 2019. “Associatively segmenting instances and semantics in point clouds.” Preprint, submitted February 26, 2019. https://arxiv.org/abs/1902.09852.
Xie, Y., J. Tian, and X. X. Zhu. 2019. “Linking points with labels in 3D: A review of point cloud semantic segmentation.” Preprint, submitted August 23, 2019. https://arxiv.org/abs/1908.08854.
Yang, B., J. Wang, R. Clark, Q. Hu, S. Wang, A. Markham, and N. Trigoni. 2019. “Learning object bounding boxes for 3D instance segmentation on point clouds.” Accessed April 16, 2021. https://arxiv.org/pdf/1906.01140v2.pdf.
Ye, X., J. Li, H. Huang, L. Du, and X. Zhang. 2018. “3d recurrent neural networks with context fusion for point cloud semantic segmentation.” In Proc., European Conf. on Computer Vision (ECCV), 403–417. New York: Computer Vision Foundation.
Yi, L., W. Zhao, H. Wang, M. Sung, and L. Guibas. 2019. “GSPN: Generative shape proposal network for 3D instance segmentation in point cloud.” Preprint, submitted December 8, 2018. https://arxiv.org/abs/1812.03320.
Zhang, J., X. Zhao, Z. Chen, and Z. Lu. 2019. “A review of deep learning-based semantic segmentation for point cloud.” IEEE Access 7: 179118–179133. https://doi.org/10.1109/ACCESS.2019.2958671.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 35 • Issue 6 • November 2021
Copyright
© 2021 American Society of Civil Engineers.
History
Received: Sep 9, 2020
Accepted: Feb 5, 2021
Published online: Aug 23, 2021
Published in print: Nov 1, 2021
Discussion open until: Jan 23, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Sandeep Jalui, Evangelia Agapaki, Investigation of Graph Neural Networks for Instance Segmentation of Industrial Point Cloud Data, Learning and Intelligent Optimization, 10.1007/978-3-031-24866-5_30, (411-428), (2023).
- Eva Agapaki, Ioannis Brilakis, CLOI: An Automated Benchmark Framework for Generating Geometric Digital Twins of Industrial Facilities, Computing in Civil Engineering 2021, 10.1061/9780784483893.068, (546-553), (2022).
- Yuandong Pan, Alexander Braun, Ioannis Brilakis, André Borrmann, Enriching geometric digital twins of buildings with small objects by fusing laser scanning and AI-based image recognition, Automation in Construction, 10.1016/j.autcon.2022.104375, 140, (104375), (2022).