Deep Learning in Construction: Review of Applications and Potential Avenues
Publication: Journal of Computing in Civil Engineering
Volume 36, Issue 2
Abstract
Neural networks (NNs) have seen an increase in popularity in the last few years. As found in several papers, they outperformed other machine learning algorithms and have improved their applicability. NNs have shown potential when working with prediction and detection algorithms by addressing a large variety of problems, such as recognition of heavy machinery, project success prediction, workers’ pose assessment, price estimations, and project productivity estimation. To understand the future potential of NNs, we completed a bibliometric analysis of the existing literature, including publications since 2010. The areas in which NNs have been used in construction applications are categorized to understand their connections and underlying architectures. This work found a wider field of applications for NNs in the construction industry than originally known. New architectures such as transformer networks have not been explored fully in construction research but could lead to higher-performing networks. As far as the authors know, this is the first review to solely focus on construction, excluding areas such as structural engineering, indoor climate, occupancy modeling, and energy analysis. The limitations of NNs are discussed, and a path forward is proposed, which includes real-time models and examination of new architectures, which would allow the construction research to fully exploit the potential of NNs.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
No data, models, or code were generated or used during the study.
References
Abdelaty, A., K. J. Shrestha, and H. D. Jeong. 2020. “Estimating preconstruction services for bridge design projects.” J. Manage. Eng. 36 (4): 04020034. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000793.
Akinosho, T. D., L. O. Oyedele, M. Bilal, A. O. Ajayi, M. D. Delgado, O. O. Akinade, and A. A. Ahmed. 2020. “Deep learning in the construction industry: A review of present status and future innovations.” J. Build. Eng. 32 (Nov): 101827. https://doi.org/10.1016/j.jobe.2020.101827.
Amer, F., and M. Golparvar-Fard. 2021. “Modeling dynamic construction work template from existing scheduling records via sequential machine learning.” Adv. Eng. Inf. 47 (Jan): 101198. https://doi.org/10.1016/j.aei.2020.101198.
Antwi-Afari, M. F., H. Li, E. A. Pärn, and D. J. Edwards. 2018. “Critical success factors for implementing building information modelling (BIM): A longitudinal review.” Autom. Constr. 91 (Jul): 100–110. https://doi.org/10.1016/j.autcon.2018.03.010.
Baalousha, Y., and T. Çelik. 2011. “An integrated web-based data warehouse and artificial neural networks system for unit price analysis with inflation adjustment.” J. Civ. Eng. Manage. 17 (2): 157–167. https://doi.org/10.3846/13923730.2011.576806.
Badawy, M., A. Hussein, S. M. Elseufy, and K. Alnaas. 2019. “How to predict the rebar labours’ production rate by using ANN model?” Int. J. Construct. Manage. 21 (4): 427–438. https://doi.org/10.1080/15623599.2018.1553573.
Bangaru, S. S., C. Wang, S. A. Busam, and F. Aghazadeh. 2021. “ANN-based automated scaffold builder activity recognition through wearable EMG and IMU sensors.” Autom. Constr. 126 (Jun): 103653. https://doi.org/10.1016/j.autcon.2021.103653.
Bayram, S., M. E. Ocal, E. L. Oral, and C. D. Atis. 2016. “Comparison of multi layer perceptron (MLP) and radial basis function (RBF) for construction cost estimation: The case of Turkey.” J. Civ. Eng. Manage. 22 (4): 480–490. https://doi.org/10.3846/13923730.2014.897988.
Braun, A., and A. Borrmann. 2019. “Combining inverse photogrammetry and BIM for automated labeling of construction site images for machine learning.” Autom. Constr. 106 (Oct): 102879. https://doi.org/10.1016/j.autcon.2019.102879.
Braun, A., S. Tuttas, A. Borrmann, and U. Stilla. 2020. “Improving progress monitoring by fusing point clouds, semantic data and computer vision.” Autom. Constr. 116 (Aug): 103210. https://doi.org/10.1016/j.autcon.2020.103210.
Cai, J., Y. Zhang, L. Yang, H. Cai, and S. Li. 2020. “A context-augmented deep learning approach for worker trajectory prediction on unstructured and dynamic construction sites” Adv. Eng. Inf. 46 (Oct): 101173. https://doi.org/10.1016/j.aei.2020.101173.
Calderon, W. T., D. Roberts, and M. Golparvar-Fard. 2021. “Synthesizing pose sequences from 3D assets for vision-based activity analysis.” J. Comput. Civ. Eng. 35 (1): 04020052. https://doi.org10.1061/(ASCE)CP.1943-5487.0000937.
Cheng, M., Y. Chang, and D. Korir. 2019. “Novel approach to estimating schedule to completion in construction projects using sequence and nonsequence learning.” J. Constr. Eng. Manage. 145 (11): 04019072. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001697.
Chou, J., C. Lin, A. Pham, and J. Shao. 2015. “Optimized artificial intelligence models for predicting project award price.” Autom. Constr. 54 (Jun): 106–115. https://doi.org/10.1016/j.autcon.2015.02.006.
Cireşan, D. C., U. Meier, L. M. Gambardella, and J. Schmidhuber. 2010. “Deep, big, simple neural nets for handwritten digit recognition.” Neural Comput. 22 (12): 3207–3220. https://doi.org/10.1162/NECO_a_00052.
Dai, X., Y. Chen, B. Xiao, D. Chen, M. Lio, L. Yuan, and L. Zhang. 2021. “Dynamic head: Unifying object detection heads with attentions.” Preprint, submitted June 15, 2021. https://arxiv.org/abs/2106.08322.
Davis, P., F. Aziz, M. T. Newas, W. Sher, and L. Simon. 2021. “The classification of construction waste material using a deep convolutional neural network.” Autom. Constr. 122 (Feb): 103481. https://doi.org/10.1016/j.autcon.2020.103481.
Ding, L., W. Fang, H. Luo, P. E. D. Love, B. Zhong, and X. Ouyang. 2018. “A deep hybrid learning model to detect unsafe behavior: Integrating convolution neural networks and long short-term memory.” Autom. Constr. 86 (Feb): 118–124. https://doi.org/10.1016/j.autcon.2017.11.002.
El-Gohary, K. M., R. F. Aziz, and H. A. Abdel-Khalek. 2017. “Engineering approach using ANN to improve and predict construction labor productivity under different influences.” J. Constr. Eng. Manage. 143 (8): 04017045. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001340.
Fan, F., J. Xiong, M. Li, and G. Wang. 2021. “On interpretability of artificial neural networks: A survey.” Preprint, submitted January 8, 2020. https://arxiv.org/abs/2001.02522.
Fang, Q., H. Li, X. Luo, L. Ding, H. Luo, T. M. Rose, and W. An. 2018a. “Detecting non-hardhat-use by a deep learning method from far-field surveillance videos.” Autom. Constr. 85 (Jan): 1–9. https://doi.org/10.1016/j.autcon.2017.09.018.
Fang, Q. H., X. Li, L. Ding, T. M. Rose, W. An, and Y. Yu. 2018b. “A deep learning-based method for detecting non-certified work on construction sites.” Adv. Eng. Inf. 35 (Jan): 56–68. https://doi.org/10.1016/j.aei.2018.01.001.
Fang, W., L. Ding, H. Luo, and P. E. D. Love. 2018c. “Falls from heights: A computer vision-based approach for safety harness detection.” Autom. Constr. 91 (Jul): 53–61. https://doi.org/10.1016/j.autcon.2018.02.018.
Fang, W., L. Ding, B. Zhong, P. E. D. Love, and H. Luo. 2018d. “Automated detection of workers and heavy equipment on construction sites: A convolutional neural network approach.” Adv. Eng. Inf. 37 (Aug): 139–149. https://doi.org/10.1016/j.aei.2018.05.003.
Fang, W., H. Luo, S. Xu, P. E. D. Love, Z. Lu, and C. Ye. 2020. “Automated text classification of near-misses from safety reports: An improved deep learning approach.” Adv. Eng. Inf. 44 (Apr): 101060. https://doi.org/10.1016/j.aei.2020.101060.
Gerek, I. H., E. Erdis, G. Mistikoglu, and M. Usmen. 2015. “Modelling masonry crew productivity using two artificial neural network techniques.” J. Civ. Eng. Manage. 21 (2): 231–238. https://doi.org/10.3846/13923730.2013.802741.
Golnaraghi, S., Z. Zangenehmadar, O. Moselhi, and S. Alkass. 2019. “Application of artificial neural network(s) in predicting formwork labour productivity.” Artif. Intell. Appl. Civ. Eng. 2019 (Jan): 1. https://doi.org/10.1155/2019/5972620.
Goodfellow, I., Y. Bengio, and A. Courville. 2016. Deep learning. Cambridge, UK: MIT Press.
Goodfellow, I., J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Benigo. 2014. “Generative adversarial networks.” Preprint, submitted June 10, 2014. https://arxiv.org/abs/1406.2661.
Guo, Y., Y. Xu, and S. Li. 2020. “Dense construction vehicle detection based on orientation-aware feature fusion convolutional neural network.” Autom. Constr. 112 (Apr): 103124. https://doi.org/10.1016/j.autcon.2020.103124.
He, K., X. Zhang, S. Ren, and J. Sun. 2015. “Deep residual learning for image recognition.” Preprint, submitted December 10, 2015. https://arxiv.org/abs/1512.03385.
Hochreiter, S., and J. Schmidhuber. 1997. “Long short-term memory.” Neural Comput. 9 (8): 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735.
Hola, B., and K. Schabowicz. 2010. “Estimation of earthworks execution time cost by means of artificial neural networks.” Autom. Constr. 19 (5): 570–579. https://doi.org/10.1016/j.autcon.2010.02.004.
Huang, G., Z. Liu, L. van der Maaten, and K. Q. Weinberger. 2017. “Densely connected convolutional networks.” Preprint, submitted August 25, 2016. https://arxiv.org/abs/1608.06993.
Hyari, K. H., A. Al-Daraiseh, and M. El-Mashaleh. 2016. “Conceptual cost estimation model for engineering services in public construction projects.” J. Manage. Eng. 32 (1): 04015021. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000381.
Jha, K. M., and C. T. Chockalingam. 2011. “Prediction of schedule performance of Indian construction projects using an artificial neural network.” Construct. Manage. Econ. 29 (9): 901–911. https://doi.org/10.1080/01446193.2011.608691.
Jung, M., and S. Chi. 2020. “Human activity classification based on sound recognition and residual convolutional neural network.” Autom. Constr. 114 (Jun): 103177. https://doi.org/10.1016/j.autcon.2020.103177.
Juszczyk, M., K. Zima, and W. Lelek. 2019. “Forecasting of sports fields construction costs aided by ensembles of neural networks.” J. Civ. Eng. Manage. 25 (7): 715–729. https://doi.org/10.3846/jcem.2019.10534.
Kamari, M., and Y. Ham. 2021. “Vision-based volumetric measurements via deep learning-based point cloud segmentation for material management in jobsites.” Autom. Constr. 121 (Jan): 103430. https://doi.org/10.1016/j.autcon.2020.103430.
Kassem, M., E. Mahamedi, K. Rogage, K. Duffy, and J. Huntingdon. 2021. “Measuring and benchmarking the productivity of excavators in infrastructure projects: A deep neural network approach.” Autom. Constr. 124 (Apr): 103532. https://doi.org/10.1016/j.autcon.2020.103532.
Kawaguchi, K., L. P. Kaelbling, and Y. Bengio. 2020. “Generalization in deep learning.” Preprint, submitted October 16, 2017. https://arxiv.org/abs/1710.05468.
Khan N., M. R. Saleem, D. Lee, M. Park, and C. Park. 2021. “Utilizing safety rule correlation for mobile scaffolds monitoring leveraging deep convolution neural networks.” Comput. Ind. 129 (Aug): 103448. https://doi.org/10.1016/j.compind.2021.103448.
Kim, D., S. Lee, and V. R. Kamat. 2020a. “Proximity prediction of mobile objects to prevent contact-driven accidents in co-robotic construction.” J. Comput. Civ. Eng. 34 (4): 04020022. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000899.
Kim, H., Y. Ham, W. Kim, S. Park, and H. Kim. 2019a. “Vision-based nonintrusive context documentation for earthmoving productivity simulation.” Autom. Constr. 102 (Jun): 135–147. https://doi.org/10.1016/j.autcon.2019.02.006.
Kim, K., and Y. K. Cho. 2020. “Effective inertial sensor quantity and locations on a body for deep learning-based worker’s motion recognition.” Autom. Constr. 113 (Apr): 103126. https://doi.org/10.1016/j.autcon.2020.103126.
Kim, M., Q. Wang, and H. Li. 2019b. “Non-contact sensing based geometric quality assessment of buildings and civil structures: A review.” Autom. Constr. 100 (Apr): 163–179. https://doi.org/10.1016/j.autcon.2019.01.002.
Kim, Y., C. H. P. Nguyen, and Y. Choi. 2020b. “Automatic pipe and elbow recognition from three-dimensional point cloud model of industrial plant piping system using convolutional neural network-based primitive classification” Autom. Constr. 116 (Aug): 103236. https://doi.org/10.1016/j.autcon.2020.103236.
Ko, C. 2013. “Predicting subcontractor performance using web-based evolutionary fuzzy neural networks.” Sci. World J. 2013: 729525. https://doi.org/10.1155/2013/729525.
Kolar, Z., H. Chen, and X. Luo. 2018. “Transfer learning and deep convolutional neural networks for safety guardrail detection in 2D images.” Autom. Constr. 89 (May): 58–70. https://doi.org/10.1016/j.autcon.2018.01.003.
Krizhevsky, A., I. Sutskever, and G. E. Hinton. 2012. “Imagenet classification with deep convolutional neural networks.” Adv. in Neural Inf. Proc. Syst. 25 (May): 1097–1105.
Lee, H., K. Yang, N. Kim, and C. R. Ahn. 2020a. “Detecting excessive load-carrying tasks using a deep learning network with a Gramian angular field.” Autom. Constr. 120 (Dec): 103390. https://doi.org/10.1016/j.autcon.2020.103390.
Lee, Y., M. Scarpiniti, and A. Uncini. 2020b. “Advanced sound classifiers and performance analyses for accurate audio-based construction project monitoring.” J. Comput. Civ. Eng. 34 (5): 04020030. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000911.
Li, W., H. Li, Q. Wu, X. Chen, and K. N. Ngan. 2019. “Simultaneously detecting and counting dense vehicles from drone images.” IEEE Trans. Ind. Electron. 66 (12): 9651–9662. https://doi.org/10.1109/TIE.2019.2899548.
Luo, H., M. Wang, P. K. Wong, and J. C. P. Cheng. 2020. “Full body pose estimation of construction equipment using computer vision and deep learning techniques.” Autom. Constr. 110 (Feb): 103016. https://doi.org/10.1016/j.autcon.2019.103016.
Luo, H., C. Xiong, W. Fang, P. E. D. Love, B. Zhang, and X. Ouyang. 2018a. “Convolutional neural networks: Computer vision-based workforce activity assessment in construction.” Autom. Constr. 94 (Oct): 282–289. https://doi.org/10.1016/j.autcon.2018.06.007.
Luo, X., H. Li, D. Cao, and F. Dai. 2018b. “Recognizing diverse construction activities in site images via relevance networks of construction-related objects detected by convolutional neural networks.” J. Comput. Civ. Eng. 32 (3): 04018012. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000756.
Marzouk, M., and A. Amin. 2013. “Predicting construction materials prices using fuzzy logic and neural networks.” J. Constr. Eng. Manage. 139 (9): 1190–1198. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000707.
Michel, P., O. Levy, and G. Neubig. 2019. “Are sixteen heads really better than one?” Preprint, submitted May 25, 2019. https://arxiv.org/abs/1905.10650.
Mousavi, S. M., B. Vahdani, H. Hashemi, and S. Ebrahimnejad. 2015. “An artificial intelligence model-based locally linear neuro-fuzzy for construction project selection.” Multiple-Valued Logic Soft Comput. 25 (6): 589–604.
Nath, H. D., A. H. Behzadan, and S. G. Paal. 2020. “Deep learning for site safety: Real-time detection of personal protective equipment.” Autom. Constr. 112 (Apr): 103085. https://doi.org/10.1016/j.autcon.2020.103085.
Ng, A. 2021. “MLOps: From model-centric to data-centric AI by Andrew Ng.” Accessed October 13, 2021. https://www.deeplearning.ai/wp-content/uploads/2021/06/MLOps-From-Model-centric-to-Data-centric-AI.pdf.
Patel, D. A., and K. N. Jha. 2015. “Neural network model for the prediction of safe work behavior in construction projects.” J. Constr. Eng. Manage. 141 (1): 04014066. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000922.
Qazi, A. 2020. “Data-driven impact assessment of multidimensional project complexity on project performance.” Int. J. Prod. Perf. Manage. https://doi.org/10.1108/IJPPM-06-2020-0281.
Rashid, K. M., and J. Louis. 2019. “Times-series data augmentation and deep learning for construction equipment activity recognition.” Autom. Constr. 42 (Oct): 100944. https://doi.org/10.1016/j.aei.2019.100944.
Rasul, A., J. Seo, and A. Khajepour. 2021. “Development of integrative methodologies for effective excavation progress monitoring.” Sensors 21 (2): 364. https://doi.org/10.3390/s21020364.
Schmidhuber, J. 2015. “Deep learning in neural networks: An overview.” Neural Net. 61 (Jan): 85–117. https://doi.org/10.1016/j.neunet.2014.09.003.
Shen, T., Y. Nagai, and C. Gao. 2020. “Design of building construction safety prediction model based on optimized BP neural network algorithm.” Soft Comput. 24 (11): 7839–7850. https://doi.org/10.1007/s00500-019-03917-4.
Shi, H. 2012. “ACO trained ANN-based bid/no-bid decision-making.” Int. J. Model. Identif. Control. 15 (4): 290–296. https://doi.org/10.1504/IJMIC.2012.046408.
Shiha, A., E. M. Dorra, and K. Nassar. 2020. “Neural networks model for prediction of construction material prices in Egypt using macroeconomic indicators.” J. Civ. Eng. Manage. 146 (3): 04020010. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001785.
Silver, D., et al. 2018. ”A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play.” Science 362 (6419): 1140–1144. https://doi.org/10.1126/science.aar6404.
Slaton, T., C. Hernandez, and R. Akhavian. 2020. “Construction activity recognition with convolutional recurrent networks.” Autom. Constr. 113 (May): 103138. https://doi.org/10.1016/j.autcon.2020.103138.
Son, H., H. Choi, H. Seong, and C. Kim. 2019. “Detection of construction workers under varying poses and changing background in image sequences via very deep residual networks.” Autom. Constr. 99 (Mar): 27–38. https://doi.org/10.1016/j.autcon.2018.11.033.
Sonmez, R., and B. Sözgen. 2017. “A support vector machine method for bid/no bid decision making.” J. Civ. Eng. Manage. 23 (5): 641–649. https://doi.org/10.3846/13923730.2017.1281836.
Srivastava, R. K., K. Greff, and J. Schmidhuber. 2015. “Training very deep networks.” Preprint, submitted July 22, 2015. https://arxiv.org/abs/1507.06228.
Vahdani, B., S. M. Mousavi, H. Hashemi, M. Mousakhani, and S. Ebrahimnejad. 2014. “A new hybrid model based on least squares support vector machine for project selection problem in construction industry.” Arab. J. Sci. Eng. 39 (5): 4301–4314. https://doi.org/10.1007/s13369-014-1032-8.
Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. 2017. “Attention is all you need.” Preprint, submitted June 12, 2017. https://arxiv.org/abs/1706.03762.
Wang, H., and B. Raj. 2017. “On the origin of deep learning.” Preprint, submitted February 24, 2017. https://arxiv.org/abs/1702.07800.
Wang, X., and Z. Zhu. 2021. “Vision–based framework for automatic interpretation of construction workers’ hand gestures.” Autom. Constr. 130 (Oct): 103872. https://doi.org/10.1016/j.autcon.2021.103872.
Wang, Z., H. Li, and X. Zhang. 2019. “Construction waste recycling robot for nails and screws: Computer vision technology and neural network approach.” Autom. Constr. 97 (Jan): 220–228. https://doi.org/10.1016/j.autcon.2018.11.009.
Wang, Z., Q. Zhang, B. Yang, T. Wu, K. Lei, B. Zhang, and T. Fang. 2021a. “Vision-based framework for automatic progress monitoring of precast walls by using surveillance videos during the construction phase.” J. Comput. Civ. Eng. 35 (1): 04020056. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000933.
Wang, Z., Y. Zhang, K. M. Mosalam, Y. Gao, and S. Huang. 2021b. “Deep semantic segmentation for visual understanding on construction sites.” Comput. Aided Civ. Infrastruct. Eng. https://doi.org/10.1111/mice.12701.
Wei, R., P. E. D. Love, W. Fang, H. Luo, and S. Xu. 2019. “Recognizing people’s identity in construction sites with computer vision: A spatial and temporal attention pooling network.” Adv. Eng. Inf. 42 (Oct): 100981. https://doi.org/10.1016/j.aei.2019.100981.
Wu, C., X. Wang, P. Wu, J. Wang, R. Jiang, M. Chen, and M. Swapan. 2021. “Hybrid deep learning model for automating constraint modelling in advanced working packaging.” Autom. Constr. 127 (Jul): 103733. https://doi.org/10.1016/j.autcon.2021.103733.
Xiao, B., and S. Kang. 2021. “Vision-based method integrating deep learning detection for tracking multiple construction machines.” J. Comput. Civ. Eng. 35 (2): 04020071. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000957.
Zhang, H., X. Yan, and H. Li. 2018. “Ergonomic posture recognition using 3D view-invariant features from single ordinary camera.” Autom. Constr. 94 (Oct): 1–10. https://doi.org/10.1016/j.autcon.2018.05.033.
Zhang, M., M. Zhu, and X. Zhao. 2020. “Recognition of high-risk scenarios in building construction based on image semantics.” J. Comput. Civ. Eng. 34 (4): 04020019. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000900.
Zhao, J., and E. Obonyo. 2020. “Convolutional long short-term memory model for recognizing construction workers’ postures from wearable inertial measurement units.” Adv. Eng. Inf. 46 (Oct): 101177. https://doi.org/10.1016/j.aei.2020.101177.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 36 • Issue 2 • March 2022
Copyright
© 2021 American Society of Civil Engineers.
History
Published online: Dec 23, 2021
Published in print: Mar 1, 2022
Discussion open until: May 23, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Yuanxie Shen, Linfeng Wu, Shixue Liang, Explainable machine learning-based model for failure mode identification of RC flat slabs without transverse reinforcement, Engineering Failure Analysis, 10.1016/j.engfailanal.2022.106647, 141, (106647), (2022).