Structural Material Condition Assessment through Human-in-the-Loop Incremental Semisupervised Learning from Hyperspectral Images
Publication: Journal of Computing in Civil Engineering
Volume 38, Issue 6
Abstract
Engineering materials in constructed systems in service exhibit complex patterns, including structural damage, environmental artifacts, and artificial anomalies. In recent years, machine vision methods have been extensively studied, most of which train models using regular grey or color images in the visible bands and label at pixel levels with a large volume of data. The authors propose using hyperspectral imaging (HSI) for structural material condition assessment in this work. Compared with visible images, the research challenge is that HSI pixels with high-dimensional spectral profiles are beyond human perceptive capabilities with hidden discriminative power. Learning from labeled and unlabeled data is one direct approach to unlocking this power. A deep neural network-enabled spatial-spectral feature extraction and a semisupervised learning architecture were developed in this work. A human-in-the-loop (HITL) framework was comparatively studied with three incremental training-data configuration schemes. The paper concludes with the following empirical findings: (1) fully supervised learning determines the baseline of the detection performance; (2) an extensive range of ratio values exists between the unlabeled and the labeled data for incremental semisupervised learning, and a ratio can be taken as a conservative and operational ratio; and (3) with parametric semisupervised learning with equal labeled and unlabeled data participation, the proposed HITL operational workflow can be implemented as a practical approach for HSI-based structural material and damage detection.
Formats available
You can view the full content in the following formats:
Data Availability Statement
All data and models generated or used during the study appear in the published article. The hyperspectral imaging dataset used in this study is publicly shared through Figshare (Chen 2020).
Acknowledgments
This material is based partially upon work supported by the National Science Foundation (NSF) under Award number No. IIA-1355406 and work supported by the United States Department of Agriculture’s National Institute of Food and Agriculture (USDA-NIFA) under Award No. 2015-68007-23214. Shimin Tang contributed equally to this work. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of NSF or USDA-NIFA.
References
Agnisarman, S., S. Lopes, K. C. Madathil, K. Piratla, and A. Gramopadhye. 2019. “A survey of automation-enabled human-in-the-loop systems for infrastructure visual inspection.” Autom. Constr. 97 (Dec): 52–76. https://doi.org/10.1016/j.autcon.2018.10.019.
Alzubaidi, L., J. Zhang, A. J. Humaidi, A. Al-Dujaili, Y. Duan, O. Al-Shamma, J. Santamara, M. A. Fadhel, M. Al-Amidie, and L. Farhan. 2021. “Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions.” J. Big Data 8 (Dec): 1–74. https://doi.org/10.1186/s40537-021-00444-8.
Aryal, S., Z. Chen, and S. Tang. 2021. “Mobile hyperspectral imaging for material surface damage detection.” J. Comput. Civ. Eng. 35 (1): 04020057. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000934.
Belkin, M., and P. Niyogi. 2004. “Semisupervised learning on Riemannian manifolds.” Mach. Learn. 56 (1): 209–239. https://doi.org/10.1023/B:MACH.0000033120.25363.1e.
Bennett, K., and A. Demiriz. 1999. “Semisupervised support vector machines.” In Advances in neural information processing systems, 368–374. La Jolla, CA: Neural Information Processing Systems Foundation.
Bishop, C. 2006. Pattern recognition and machine learning. New York: Springer.
Browne, M., and S. S. Ghidary. 2003. “Convolutional neural networks for image processing: An application in robot vision.” In Proc., Australasian Joint Conf. on Artificial Intelligence, 641–652. New York: Springer.
Chen, Y., H. Jiang, C. Li, X. Jia, and P. Ghamisi. 2016. “Deep feature extraction and classification of hyperspectral images based on convolutional neural networks.” IEEE Trans. Geosci. Remote Sens. 54 (10): 6232–6251. https://doi.org/10.1109/TGRS.2016.2584107.
Chen, Y., K. Zhu, L. Zhu, X. He, P. Ghamisi, and J. A. Benediktsson. 2019. “Automatic design of convolutional neural network for hyperspectral image classification.” IEEE Trans. Geosci. Remote Sens. 57 (9): 7048–7066. https://doi.org/10.1109/TGRS.2019.2910603.
Chen, Z. 2020. “Hyperspectral imagery for material surface damage.” Figshare. Accessed May 29, 2020. https://doi.org/10.6084/m9.figshare.12385994.v1.
Chen, Z., R. Derakhshani, C. Halmen, and J. T. Kevern. 2011. “A texture-based method for classifying cracked concrete surfaces from digital images using neural networks.” In Proc., 2011 Int. Joint Conf. on Neural Networks, 2632–2637. New York: IEEE. https://doi.org/10.1109/IJCNN.2011.6033562.
Chien, C., W. Martin, A. Meyer, and J. Aggarwal. 1983. Detection of cracks on highway pavements. Austin, TX: Univ. of Texas at Austin.
Crammer, K., and Y. Singer. 2001. “On the algorithmic implementation of multiclass kernel-based vector machines.” J. Mach. Learn. Res. 2 (12): 265–292. https://doi.org/10.5555/944790.944813.
Cubert. 2022. “Hyperspectral vnir camera firefleye 185.” Cubert GmbH. Accessed January 1, 2022. https://www.cubert-hyperspectral.com/products/firefleye-185.
Cui, R., H. Yu, T. Xu, X. Xing, X. Cao, K. Yan, and J. Chen. 2022. “Deep learning in medical hyperspectral images: A review.” Sensors 22 (24): 9790. https://doi.org/10.3390/s22249790.
Dhillon, A., and G. K. Verma. 2020. “Convolutional neural network: A review of models, methodologies and applications to object detection.” Prog. Artif. Intell. 9 (2): 85–112. https://doi.org/10.1007/s13748-019-00203-0.
Didaci, L., G. Fumera, and F. Roli. 2012. “Analysis of co-training algorithm with very small training sets.” In Proc., Joint IAPR Int. Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR), 719–726. New York: Springer.
Ding, S., Z. Zhu, and X. Zhang. 2017. “An overview on semisupervised support vector machine.” Neural Comput. Appl. 28 (5): 969–978. https://doi.org/10.1007/s00521-015-2113-7.
Du, S., J. Lee, H. Li, L. Wang, and X. Zhai. 2019. “Gradient descent finds global minima of deep neural networks.” In Proc., 36th Int. Conf. on Machine, 1675–1685. Breckenridge, CO: Proceedings of Machine Learning Research.
Ehrenberg, H. R., J. Shin, A. J. Ratner, J. A. Fries, and C. Ré. 2016. “Data programming with ddlite: Putting humans in a different part of the loop.” In Proc., Workshop on Human-In-the-Loop Data Analytics, 1–6. New York: Association for Computing Machinery. https://doi.org/10.1145/2939502.2939515.
Feng, F., W. Li, Q. Du, and B. Zhang. 2017. “Dimensionality reduction of hyperspectral image with graph-based discriminant analysis considering spectral similarity.” Remote Sens. 9 (4): 323. https://doi.org/10.3390/rs9040323.
Fiebrink, R., P. R. Cook, and D. Trueman. 2011. “Human model evaluation in interactive supervised learning.” In Proc., SIGCHI Conf. on Human Factors in Computing Systems, 147–156. New York: Association for Computing Machinery. https://doi.org/10.1145/1978942.1978965.
Fralick, S. 1967. “Learning to recognize patterns without a teacher.” IEEE Trans. Inf. Theory 13 (1): 57–64. https://doi.org/10.1109/TIT.1967.1053952.
Gao, L., J. Li, M. Khodadadzadeh, A. Plaza, B. Zhang, Z. He, and H. Yan. 2014. “Subspace-based support vector machines for hyperspectral image classification.” IEEE Geosci. Remote Sens. Lett. 12 (2): 349–353. https://doi.org/10.1109/LGRS.2014.2341044.
Haas, C., H. Shen, W. Phang, and R. Haas. 1984. “Application of image analysis technology to automation of pavement condition surveys.” In Proc., Int. Transport Congress, 55–73. Cape Town, South Africa: A. A. Balkema.
Haut, J. M., M. E. Paoletti, J. Plaza, J. Li, and A. Plaza. 2018. “Active learning with convolutional neural networks for hyperspectral image classification using a new Bayesian approach.” IEEE Trans. Geosci. Remote Sens. 56 (11): 6440–6461. https://doi.org/10.1109/TGRS.2018.2838665.
He, K., X. Zhang, S. Ren, and J. Sun. 2016. “Deep residual learning for image recognition.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 770–778. New York: IEEE.
Ince, T., S. Kiranyaz, L. Eren, M. Askar, and M. Gabbouj. 2016. “Real-time motor fault detection by 1-d convolutional neural networks.” IEEE Trans. Ind. Electron. 63 (11): 7067–7075. https://doi.org/10.1109/TIE.2016.2582729.
Joachims, T. 1999. “Transductive inference for text classification using support vector machines.” In Vol. 99 of Proc., 16th Int. Conf. on Machine Learning, 200–209. San Francisco: Morgan Kaufmann Publishers. https://doi.org/10.5555/645528.657646.
Khodadadzadeh, M., J. Li, A. Plaza, and J. M. Bioucas-Dias. 2014. “A subspace-based multinomial logistic regression for hyperspectral image classification.” IEEE Geosci. Remote Sens. Lett. 11 (12): 2105–2109. https://doi.org/10.1109/LGRS.2014.2320258.
Kim, B. 2015. “Interactive and interpretable machine learning models for human machine collaboration.” Ph.D. thesis, Dept. of Aeronautics and Astronautics, Massachusetts Institute of Technology.
Kingma, D. P., S. Mohamed, D. Jimenez Rezende, and M. Welling. 2014. “Semisupervised learning with deep generative models.” In Advances in neural information processing systems, 27. La Jolla, CA: Neural Information Processing Systems Foundation.
Kiranyaz, S., T. Ince, and M. Gabbouj. 2015. “Real-time patient-specific ecg classification by 1-d convolutional neural networks.” IEEE Trans. Biomed. Eng. 63 (3): 664–675. https://doi.org/10.1109/TBME.2015.2468589.
Kirillov, A., et al. 2023. “Segment anything.” In Proc., IEEE/CVF Int. Conf. on Computer Vision, 4015–4026. New York: IEEE.
Krishnan, S., M. J. Franklin, K. Goldberg, J. Wang, and E. Wu. 2016. “Activeclean: An interactive data cleaning framework for modern machine learning.” In Proc., 2016 Int. Conf. on Management of Data, 2117–2120. New York: IEEE.
LeCun, Y., P. Haffner, L. Bottou, and Y. Bengio. 1999. “Object recognition with gradient-based learning.” In Shape, contour and grouping in computer vision, 319–345. New York: Springer.
Lee, H., and H. Kwon. 2017. “Going deeper with contextual CNN for hyperspectral image classification.” IEEE Trans. Image Process. 26 (10): 4843–4855. https://doi.org/10.1109/TIP.2017.2725580.
Lee, J. S., Y. Ham, H. Park, and J. Kim. 2022. “Challenges, tasks, and opportunities in teleoperation of excavator toward human-in-the-loop construction automation.” Autom. Constr. 135 (Jun): 104119. https://doi.org/10.1016/j.autcon.2021.104119.
Li, S., W. Song, L. Fang, Y. Chen, P. Ghamisi, and J. A. Benediktsson. 2019. “Deep learning for hyperspectral image classification: An overview.” IEEE Trans. Geosci. Remote Sens. 57 (9): 6690–6709. https://doi.org/10.1109/TGRS.2019.2907932.
Liu, Y., Y. Zhang, Y. Wang, F. Hou, J. Yuan, J. Tian, Y. Zhang, Z. Shi, J. Fan, and Z. He. 2024. “A survey of visual transformers.” IEEE Trans. Neural Networks Learn. Syst. 35 (6): 7478–7498. https://doi.org/10.1109/TNNLS.2022.3227717.
Liu, Z., S. A. Suandi, T. Ohashi, and T. Ejima. 2002. “Tunnel crack detection and classification system based on image processing.” In Vol. 4664 of Machine vision applications in industrial inspection X, 145–153. Bellingham, WA: International Society for Optics and Photonics.
Mäkinen, S., H. Skogström, E. Laaksonen, and T. Mikkonen. 2021. “Who needs MLOps: What data scientists seek to accomplish and how can MLOps help.” In Proc., 2021 IEEE/ACM 1st Workshop on AI Engineering-Software Engineering for AI (WAIN), 109–112. New York: IEEE.
McHugh, M. L. 2012. “Interrater reliability: The kappa statistic.” Biochem. Med. 22 (3): 276–282. https://doi.org/10.11613/BM.2012.031.
Mohan, A., and S. Poobal. 2017. “Crack detection using image processing: A critical review and analysis.” Alexandria Eng. J. 57 (2): 787–798. https://doi.org/10.1016/j.aej.2017.01.020.
Mosqueira-Rey, E., E. Hernández-Pereira, D. Alonso-Ros, J. Bobes-Bascarán, and Á. Fernández-Leal. 2023. “Human-in-the-loop machine learning: A state of the art.” Artif. Intell. Rev. 56 (4): 3005–3054. https://doi.org/10.1007/s10462-022-10246-w.
Niazmardi, S., S. Homayouni, and A. Safari. 2013. “An improved fcm algorithm based on the svdd for unsupervised hyperspectral data classification.” IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 6 (2): 831–839. https://doi.org/10.1109/JSTARS.2013.2244851.
Oullette, R., M. Browne, and K. Hirasawa. 2004. “Genetic algorithm optimization of a convolutional neural network for autonomous crack detection.” In Vol. 1 of Proc., 2004 Congress on Evolutionary Computation (IEEE Cat. No. 04TH8753), 516–521. New York: IEEE.
Pavia, D. L., G. M. Lampman, G. S. Kriz, and J. A. Vyvyan. 2014. Introduction to spectroscopy. 5th ed. Boston, MA: Cengage Learning.
Peng, B., C. Li, P. He, M. Galley, and J. Gao. 2023. “Instruction tuning with gpt-4.” Preprint, submitted April 6, 2023. https://arxiv.org/abs/2304.03277.
Rasti, B., D. Hong, R. Hang, P. Ghamisi, X. Kang, J. Chanussot, and J. A. Benediktsson. 2020. “Feature extraction for hyperspectral imagery: The evolution from shallow to deep: Overview and toolbox.” IEEE Geosci. Remote Sens. 8 (4): 60–88. https://doi.org/10.1109/MGRS.2020.2979764.
Ritchie, S. G. 1990. “Digital imaging concepts and applications in pavement management.” J. Transp. Eng. 116 (3): 287–298. https://doi.org/10.1061/(ASCE)0733-947X(1990)116:3(287).
Scudder, H. 1965. “Probability of error of some adaptive pattern-recognition machines.” IEEE Trans. Inf. Theory 11 (3): 363–371. https://doi.org/10.1109/TIT.1965.1053799.
Signoroni, A., M. Savardi, A. Baronio, and S. Benini. 2019. “Deep learning meets hyperspectral image analysis: A multidisciplinary review.” J. Imaging 5 (5): 52. https://doi.org/10.3390/jimaging5050052.
Tang, Y. Y., H. Yuan, and L. Li. 2014. “Manifold-based sparse representation for hyperspectral image classification.” IEEE Trans. Geosci. Remote Sens. 52 (12): 7606–7618. https://doi.org/10.1109/TGRS.2014.2315209.
Theodoridis, S., and K. Koutroumbas. 1999. “Pattern recognition and neural networks.” In Advanced course on artificial intelligence, 169–195. New York: Springer.
Van Engelen, J. E., and H. H. Hoos. 2020. “A survey on semisupervised learning.” Mach. Learn. 109 (2): 373–440. https://doi.org/10.1007/s10994-019-05855-6.
Vapnik, V. 1999. The nature of statistical learning theory. New York: Springer.
Windrim, L., A. Melkumyan, R. J. Murphy, A. Chlingaryan, and R. Ramakrishnan. 2018. “Pretraining for hyperspectral convolutional neural network classification.” IEEE Trans. Geosci. Remote Sens. 56 (5): 2798–2810. https://doi.org/10.1109/TGRS.2017.2783886.
Wu, X., L. Xiao, Y. Sun, J. Zhang, T. Ma, and L. He. 2022. “A survey of human-in-the-loop for machine learning.” Future Gener. Comput. Syst. 135 (Mar): 364–381. https://doi.org/10.1016/j.future.2022.05.014.
Yang, X., Z. Song, I. King, and Z. Xu. 2023. “A survey on deep semisupervised learning.” IEEE Trans. Knowl. Data Eng. 35 (9): 8934–8954. https://doi.org/10.1109/TKDE.2022.3220219.
Yao, D., Z. Zhi-li, Z. Xiao-feng, C. Wei, H. Fang, C. Yao-ming, and W.-W. Cai. 2023. “Deep hybrid: Multi-graph neural network collaboration for hyperspectral image classification.” Def. Technol. 23 (Apr): 164–176. https://doi.org/10.1016/j.dt.2022.02.007.
Yu, S., S. Jia, and C. Xu. 2017. “Convolutional neural networks for hyperspectral image classification.” Neurocomputing 219 (Jun): 88–98. https://doi.org/10.1016/j.neucom.2016.09.010.
Zanzotto, F. M. 2019. “Human-in-the-loop artificial intelligence.” J. Artif. Intell. Res. 64 (Apr): 243–252. https://doi.org/10.1613/jair.1.11345.
Zhang, C., J. Shin, C. Ré, M. Cafarella, and F. Niu. 2016. “Extracting databases from dark data with deepdive.” In Proc., 2016 Int. Conf. on Management of Data, 847–859. New York: Association for Computing Machinery. https://doi.org/10.1145/2882903.2904442.
Zhang, C. J., L. Chen, H. V. Jagadish, and C. C. Cao. 2013. “Reducing uncertainty of schema matching via crowdsourcing.” Proc. VLDB Endow. 6 (9): 757–768. https://doi.org/10.14778/2536360.2536374.
Zhang, S., J. Zhang, X. Wang, J. Wang, and Z. Wu. 2023. “ELS2T: Efficient lightweight spectral-spatial transformer for hyperspectral image classification.” IEEE Trans. Geosci. Remote Sens. 61: 1–16. https://doi.org/10.1109/TGRS.2023.3299442.
Zhao, C., B. Qin, S. Feng, W. Zhu, W. Sun, W. Li, and X. Jia. 2023. “Hyperspectral image classification with multi-attention transformer and adaptive superpixel segmentation-based active learning.” IEEE Trans. Image Process 32 (Jun): 3606–3621. https://doi.org/10.1109/TIP.2023.3287738.
Zhong, Y., A. Ma, and L. Zhang. 2014. “An adaptive memetic fuzzy clustering algorithm with spatial information for remote sensing imagery.” IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 7 (4): 1235–1248. https://doi.org/10.1109/JSTARS.2014.2303634.
Zhu, X. 2005. Semisupervised learning literature survey. Madison, WI: Univ. of Wisconsin–Madison.
Zhu, X., and Z. Ghahramani. 2002. Learning from labeled and unlabeled data with label propagation. Pittsburgh: Carnegie Mellon Univ.
Zhu, X., and A. B. Goldberg. 2009. Introduction to semi-supervised learning. Berlin: Springer.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 38 • Issue 6 • November 2024
Copyright
This work is made available under the terms of the Creative Commons Attribution 4.0 International license, https://creativecommons.org/licenses/by/4.0/.
History
Received: Jan 8, 2024
Accepted: May 31, 2024
Published online: Aug 24, 2024
Published in print: Nov 1, 2024
Discussion open until: Jan 24, 2025
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.