Content-Based Image Retrieval for Construction Site Images: Leveraging Deep Learning–Based Object Detection
Publication: Journal of Computing in Civil Engineering
Volume 37, Issue 6
Abstract
Visual data comprising images and videos has become an integral aspect of construction management, potentially supplanting traditional paper-based site documentation. With the vast amount of image data generated in construction projects, an efficient retrieval system that not only enhances visual data documentation but also promotes reutilization is needed. Existing label-based image retrieval methods for construction images require manual labeling and ignore visual information. Moreover, other content-based methods that consider visual properties of construction images are limited to utilizing simple visual features of the entire image. This poses a challenge when attempting to retrieve target images from the same construction site or those involving similar construction activities, particularly considering that construction images often share similar visual properties. This research introduces a content-based image retrieval method that employs object detection to identify significant subregions within construction images and convolutional neural networks to extract refined visual features of these subregions. By simply inputting a query image, the proposed method can accurately retrieve target construction images of interest. The proposed method was validated through experiments designed to retrieve target images in both same-site and same-activity retrieval scenarios. The proposed method achieved the best mean average precision (86.4%). This technology could contribute to construction data management and decision-making processes by providing an efficient information retrieval system.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
All data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This work was partly supported by the Hong Kong Polytechnic University (Project ID: P0044475) and the China Scholarship Council under Grant No. CSC202007970002. The authors would like to express appreciation to the volunteers who participated in collecting the data set, and Zhang et al. (2022) for sharing the worker image collection.
References
Brilakis, I., and L. Soibelman. 2005. “Content-based search engines for construction image databases.” Autom. Constr. 14 (4): 537–550. https://doi.org/10.1016/j.autcon.2004.11.003.
Brilakis, I., L. Soibelman, and Y. Shinagawa. 2005. “Material-based construction site image retrieval.” J. Comput. Civ. Eng. 19 (4): 341–355. https://doi.org/10.1061/(ASCE)0887-3801(2005)19:4(341).
Brilakis, I. K., and L. Soibelman. 2008. “Shape-based retrieval of construction site photographs.” J. Comput. Civ. Eng. 22 (1): 14–20. https://doi.org/10.1061/(ASCE)0887-3801(2008)22:1(14).
Cakir, F., K. He, X. Xia, B. Kulis, and S. Sclaroff. 2019. “Deep metric learning to rank.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 1861–1870. New York: IEEE.
Chen, C., Z. Zhu, and A. Hammad. 2020. “Automated excavators activity recognition and productivity analysis from construction site surveillance videos.” Autom. Constr. 110 (Feb): 103045. https://doi.org/10.1016/j.autcon.2019.103045.
Dalal, N., B. Triggs, N. Dalal, B. Triggs, O. Gradients, D. International, S. Diego, N. Dalal, and B. Triggs. 2010. “Histograms of oriented gradients for human detection to cite this version: HAL Id: Inria-00548512 histograms of oriented gradients for human detection.” In Proc., IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, 886–893. New York: IEEE.
Deng, J., W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. 2009. “Imagenet: A large-scale hierarchical image database.” In Proc., 2009 IEEE Conf. on Computer Vision and Pattern Recognition, 248–255. New York: IEEE.
Ding, L., W. Fang, H. Luo, P. E. D. Love, B. Zhong, and X. Ouyang. 2018. “A deep hybrid learning model to detect unsafe behavior: Integrating convolution neural networks and long short-term memory.” Autom. Constr. 86 (Mar): 118–124. https://doi.org/10.1016/j.autcon.2017.11.002.
Ding, Y., J. Ma, and X. Luo. 2022. “Applications of natural language processing in construction.” Autom. Constr. 136 (Apr): 104169. https://doi.org/10.1016/j.autcon.2022.104169.
Fang, Q., H. Li, X. Luo, L. Ding, T. M. Rose, W. An, and Y. Yu. 2018. “A deep learning-based method for detecting non-certified work on construction sites.” Adv. Eng. Inf. 35 (Jan): 56–68. https://doi.org/10.1016/j.aei.2018.01.001.
Fang, W., L. Ding, P. E. D. Love, H. Luo, H. Li, F. Peña-Mora, B. Zhong, and C. Zhou. 2020. “Computer vision applications in construction safety assurance.” Autom. Constr. 110 (Feb): 103013. https://doi.org/10.1016/j.autcon.2019.103013.
Fang, W., P. E. D. Love, H. Luo, and S. Xu. 2022. “A deep learning fusion approach to retrieve images of people’s unsafe behavior from construction sites.” Dev. Built Environ. 12 (Dec): 100085. https://doi.org/10.1016/j.dibe.2022.100085.
Felzenszwalb, P. F., R. B. Girshick, D. McAllester, and D. Ramanan. 2010. “Object detection with discriminatively trained part-based models.” IEEE Trans. Pattern Anal. Mach. Intell. 32 (9): 1627–1645. https://doi.org/10.1109/TPAMI.2009.167.
Gil, D., G. Lee, and K. Jeon. 2018. “Classification of images from construction sites using a deep-learning algorithm.” In Proc., Int. Symp. on Automation and Robotics in Construction, 1–6. Banff, AB, Canada: IAARC Publications.
Girshick, R., J. Donahue, T. Darrell, and J. Malik. 2014. “Rich feature hierarchies for accurate object detection and semantic segmentation.” In Proc., 2014 IEEE Conf. on Computer Vision and Pattern Recognition, 580–587. New York: IEEE.
Goodfellow, I., Y. Bengio, and A. Courville. 2016. Deep learning. Cambridge, MA: MIT Press.
Ha, I., H. Kim, S. Park, and H. Kim. 2018. “Image retrieval using BIM and features from pretrained VGG network for indoor localization.” Build. Environ. 140 (Feb): 23–31. https://doi.org/10.1016/j.buildenv.2018.05.026.
Hameed, I. M., S. H. Abdulhussain, and B. M. Mahmmod. 2021. “Content-based image retrieval: A review of recent trends.” Cogent Eng. 8 (1): 1927469. https://doi.org/10.1080/23311916.2021.1927469.
Han, K. K., and M. Golparvar-Fard. 2017. “Potential of big visual data and building information modeling for construction performance analytics: An exploratory study.” Autom. Constr. 73 (Jan): 184–198. https://doi.org/10.1016/j.autcon.2016.11.004.
He, K., G. Gkioxari, P. Dollár, and R. Girshick. 2017. “Mask r-cnn.” In Proc., IEEE Int. Conf. on Computer Vision, 2961–2969. New York: IEEE.
He, K., X. Zhang, S. Ren, and J. Sun. 2016. “Deep residual learning for image recognition.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 770–778. New York: IEEE.
Howard, A. G., M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. 2017. “MobileNets: Efficient convolutional neural networks for mobile vision applications.” Preprint, submitted April 17, 2017. https://arxiv.org/abs/1704.04861.
Hu, B., R.-J. Song, X.-S. Wei, Y. Yao, X.-S. Hua, and Y. Liu. 2020. “PyRetri: A PyTorch-based library for unsupervised image retrieval by deep convolutional neural networks.” In Proc., 28th ACM Int. Conf. on Multimedia, 4461–4464. New York: Association for Computing Machinery.
Jun, H., B. Ko, Y. Kim, I. Kim, and J. Kim. 2020. “Combination of multiple global descriptors for image retrieval.” Preprint, submitted March 26, 2019. https://arxiv.org/abs/1903.10663.
Kim, D., M. Liu, S. Lee, and V. R. Kamat. 2019. “Remote proximity monitoring between mobile construction resources using camera-mounted UAVs.” Autom. Constr. 99 (Mar): 168–182. https://doi.org/10.1016/j.autcon.2018.12.014.
Kim, H., H. Kim, Y. W. Hong, and H. Byun. 2017. “Detecting construction equipment using a region-based fully convolutional network and transfer learning.” J. Comput. Civ. Eng. 32 (2): 04017082. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000731.
Kim, J., and S. Chi. 2022. “Graph neural network-based propagation effects modeling for detecting visual relationships among construction resources.” Autom. Constr. 141 (Sep): 104443. https://doi.org/10.1016/j.autcon.2022.104443.
Kingma, D. P., and J. Ba. 2017. “Adam: A method for stochastic optimization.” Preprint, submitted December 22, 2014. https://arxiv.org/abs/1412.6980.
Krishna, R., et al. 2017. “Visual genome: Connecting language and vision using crowdsourced dense image annotations.” Int. J. Comput. Vis. 123 (1): 32–73. https://doi.org/10.1007/s11263-016-0981-7.
Kulchandani, J. S., and K. J. Dangarwala. 2015. “Moving object detection: Review of recent research trends.” In Proc., 2015 Int. Conf. on Pervasive Computing (ICPC), 1–5. New York: IEEE.
Latif, A., A. Rasheed, U. Sajid, J. Ahmed, N. Ali, N. I. Ratyal, B. Zafar, S. H. Dar, M. Sajid, and T. Khalil. 2019. “Content-based image retrieval and feature extraction: A comprehensive review.” Math. Probl. Eng. 2019 (Aug): 1. https://doi.org/10.1155/2019/9658350.
Li, N., Q. Li, Y. S. Liu, W. Lu, and W. Wang. 2020. “BIMSeek++: Retrieving BIM components using similarity measurement of attributes.” Comput. Ind. 116 (Apr): 103186. https://doi.org/10.1016/j.compind.2020.103186.
Li, Z., and D. Li. 2022. “Action recognition of construction workers under occlusion.” J. Build. Eng. 45 (Jan): 103352. https://doi.org/10.1016/j.jobe.2021.103352.
Lienhart, R., and J. Maydt. 2002. “An extended set of Haar-like features for rapid object detection.” In Proc., Int. Conf. on Image Processing, 900–903. New York: IEEE.
Liu, P. C.-Y., and N. El-Gohary. 2020. Semantic image retrieval and clustering for supporting domain-specific bridge component and defect classification, 809–818. Reston, VA: ASCE. https://doi.org/10.1061/9780784482858.087.
Liu, W., D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Fu, and A. C. Berg. 2016a. SSD: Single shot multibox detector. 21–37. Berlin: Springer.
Liu, Y., and J. K. Yeoh. 2021. “Robust pixel-wise concrete crack segmentation and properties retrieval using image patches.” Autom. Constr. 123 (Mar): 103535. https://doi.org/10.1016/j.autcon.2020.103535.
Liu, Z., P. Luo, S. Qiu, X. Wang, and X. Tang. 2016b. “DeepFashion: Powering robust clothes recognition and retrieval with rich annotations.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 1096–1104. New York: IEEE.
Ma, J. W., T. Czerniawski, and F. Leite. 2021. “An application of metadata-based image retrieval system for facility management.” Adv. Eng. Inf. 50 (Oct): 101417. https://doi.org/10.1016/j.aei.2021.101417.
Ma, K., B. Wang, Y. Li, and J. Zhang. 2022. “Image retrieval for local architectural heritage recommendation based on deep hashing.” Buildings 12 (6): 809. https://doi.org/10.3390/buildings12060809.
Manning, C. D., P. Raghavan, and H. Schütze. 2008. Introduction to information retrieval. Cambridge, MA: Cambridge University Press.
Nath, N. D., and A. H. Behzadan. 2019. “Deep learning models for content-based retrieval of construction visual data.” In Proc., ASCE Int. Conf. on Computing in Civil Engineering 2019, 66–73. Reston, VA: ASCE. https://doi.org/10.1061/9780784482438.009.
Nath, N. D., and A. H. Behzadan. 2020. “Deep convolutional networks for construction object detection under different visual conditions.” Front. Built Environ. 6 (Aug): 97. https://doi.org/10.3389/fbuil.2020.00097.
Paneru, S., and I. Jeelani. 2021. “Computer vision applications in construction: Current state, opportunities & challenges.” Autom. Constr. 132 (Dec): 103940. https://doi.org/10.1016/j.autcon.2021.103940.
Park, J., H. Cai, and D. Perissin. 2017. “Bringing information to the field: Automated photo registration and 4D BIM.” J. Comput. Civ. Eng. 32 (2): 04017084. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000740.
Radenović, F., G. Tolias, and O. Chum. 2018. “Fine-tuning CNN image retrieval with no human annotation.” IEEE Trans. Pattern Anal. Mach. Intell. 41 (7): 1655–1668. https://doi.org/10.1109/TPAMI.2018.2846566.
Raghavan, R., and K. John Singh. 2021. “A review on content based image retrieval and its methods towards efficient image retrieval.” In Intelligent computing paradigm and cutting-edge technologies, edited by M. N. Favorskaya, S.-L. Peng, M. Simic, B. Alhadidi, and S. Pal, 363–371. Berlin: Springer.
Ramzi, E., N. Thome, C. Rambour, N. Audebert, and X. Bitot. 2021. “Robust and decomposable average precision for image retrieval.” Preprint, submitted October 1, 2021. https://arxiv.org/abs/2110.01445.
Redmon, J., S. Divvala, R. Girshick, and A. Farhadi. 2016. “You only look once: Unified, real-time object detection.” In Proc., 2016 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 779–788. New York: IEEE.
Ren, S., K. He, R. Girshick, and J. Sun. 2017. “Faster R-CNN: Towards real-time object detection with region proposal networks.” IEEE Trans. Pattern Anal. Mach. Intell. 39 (6): 1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031.
Simonyan, K., and A. Zisserman. 2014. “Very deep convolutional networks for large-scale image recognition.” Preprint, submitted September 4, 2014. https://arxiv.org/abs/1409.1556.
Smith, D. K., and M. Tardif. 2009. Building information modeling: A strategic implementation guide for architects, engineers, constructors, and real estate asset managers. New York: Wiley.
Song, H. O., Y. Xiang, S. Jegelka, and S. Savarese. 2016. “Deep metric learning via lifted structured feature embedding.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition. New York: IEEE.
Szegedy, C., W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. 2015. “Going deeper with convolutions.” In Proc., 2015 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 1–9. New York: IEEE.
Teh, E. W., T. DeVries, and G. W. Taylor. 2020. “ProxyNCA++: Revisiting and revitalizing proxy neighborhood component analysis.” In Proc., Computer Vision–ECCV 2020: 16th European Conf., 448–464. Berlin: Springer.
Tolias, G., R. Sicre, and H. Jégou. 2016. “Particular object retrieval with integral max-pooling of CNN activations.” Preprint, submitted November 18, 2015. https://arxiv.org/abs/1511.05879.
Tong, X.-Y., G.-S. Xia, F. Hu, Y. Zhong, M. Datcu, and L. Zhang. 2020. “Exploiting deep features for remote sensing image retrieval: A systematic investigation.” IEEE Trans. Big Data 6 (3): 507–521. https://doi.org/10.1109/TBDATA.2019.2948924.
Wah, C., S. Branson, P. Welinder, P. Perona, and S. Belongie. 2011. The caltech-ucsd birds-200-2011 dataset. Pasadena, CA: California Institute of Technology.
Wang, H., Y. Song, L. Huo, L. Chen, and Q. He. 2023. “Multiscale object detection based on channel and data enhancement at construction sites.” Multimedia Syst. 29 (1): 49–58. https://doi.org/10.1007/s00530-022-00983-x.
Wang, Y., B. Xiao, A. Bouferguene, M. Al-Hussein, and H. Li. 2022. “Vision-based method for semantic information extraction in construction by integrating deep learning object detection and image captioning.” Adv. Eng. Inf. 53 (Aug): 101699. https://doi.org/10.1016/j.aei.2022.101699.
Wei, Y., and B. Akinci. 2022. “Panorama-to-model registration through integration of image retrieval and semantic reprojection.” Autom. Constr. 140 (Aug): 104356. https://doi.org/10.1016/j.autcon.2022.104356.
Wu, C., X. Li, Y. Guo, J. Wang, Z. Ren, M. Wang, and Z. Yang. 2022. “Natural language processing for smart construction: Current status and future directions.” Autom. Constr. 134 (Feb): 104059. https://doi.org/10.1016/j.autcon.2021.104059.
Wu, Y., A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick. 2019. “Detectron2.” GitHub. Accessed July 5, 2023. https://github.com/facebookresearch/detectron2.
Xiao, B., and S.-C. Kang. 2020. “Development of an image data set of construction machines for deep learning object detection.” J. Comput. Civ. Eng. 35 (2): 05020005. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000945.
Xu, S., J. Wang, W. Shou, T. Ngo, A.-M. Sadick, and X. Wang. 2021. “Computer vision techniques in construction: A critical review.” Arch. Comput. Methods Eng. 28 (5): 3383–3397. https://doi.org/10.1007/s11831-020-09504-3.
Xuehui, A., Z. Li, L. Zuguang, W. Chengzhi, L. Pengfei, and L. Zhiwei. 2021. “Dataset and benchmark for detecting moving objects in construction sites.” Autom. Constr. 122 (Feb): 103482. https://doi.org/10.1016/j.autcon.2020.103482.
Yang, J., B. Jiang, and H. Song. 2018. “A distributed image-retrieval method in multi-camera system of smart city based on cloud computing.” Future Gener. Comput. Syst. 81 (Apr): 244–251. https://doi.org/10.1016/j.future.2017.11.015.
Yang, J., M.-W. Park, P. A. Vela, and M. Golparvar-Fard. 2015. “Construction performance monitoring via still images, time-lapse photos, and video streams: Now, tomorrow, and the future.” Adv. Eng. Inf. 29 (2): 211–224. https://doi.org/10.1016/j.aei.2015.01.011.
Yeum, C. M., S. J. Dyke, B. Benes, T. Hacker, J. Ramirez, A. Lund, and S. Pujol. 2018. “Postevent reconnaissance image documentation using automated classification.” J. Perform. Constr. Facil. 33 (1): 04018103. https://doi.org/10.1061/(ASCE)CF.1943-5509.0001253.
Yu, H., H. Sun, J. Tao, C. Qin, D. Xiao, Y. Jin, and C. Liu. 2023. “A multi-stage data augmentation and AD-ResNet-based method for EPB utilization factor prediction.” Autom. Constr. 147 (Mar): 104734. https://doi.org/10.1016/j.autcon.2022.104734.
Zhang, F., H. Fleyeh, X. Wang, and M. Lu. 2019. “Construction site accident analysis using text mining and natural language processing techniques.” Autom. Constr. 99 (Mar): 238–248. https://doi.org/10.1016/j.autcon.2018.12.016.
Zhang, L., J. Wang, Y. Wang, H. Sun, and X. Zhao. 2022. “Automatic construction site hazard identification integrating construction scene graphs with BERT based domain knowledge.” Autom. Constr. 142 (Oct): 104535. https://doi.org/10.1016/j.autcon.2022.104535.
Zhang, S., and L. Zhang. 2022. “Construction site safety monitoring and excavator activity analysis system.” Constr. Rob. 6 (2): 151–161. https://doi.org/10.1007/s41693-022-00077-0.
Zhao, Z.-Q., P. Zheng, S.-T. Xu, and X. Wu. 2019. “Object detection with deep learning: A review.” In IEEE transactions on neural networks and learning systems, 1–21. New York: IEEE. https://doi.org/10.1109/TNNLS.2018.2876865.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 37 • Issue 6 • November 2023
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Apr 24, 2023
Accepted: Jul 23, 2023
Published online: Sep 15, 2023
Published in print: Nov 1, 2023
Discussion open until: Feb 15, 2024
ASCE Technical Topics:
- Artificial intelligence and machine learning
- Computer programming
- Computer vision and image processing
- Computing in civil engineering
- Construction engineering
- Construction management
- Construction methods
- Construction sites
- Engineering fundamentals
- Information management
- Information systems
- Integrals
- Mathematics
- Methodology (by type)
- Neural networks
- Systems engineering
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.