Bi-Directional Image-to-Text Mapping for NLP-Based Schedule Generation and Computer Vision Progress Monitoring
Publication: Construction Research Congress 2024
ABSTRACT
State-of-the-art in construction document analytics and progress detection has experienced accelerated growth over the last decade. However, each area encountered isolated growth, not considering their interactions. Today, progress monitoring practices are often neglected due to requiring manual input of visible progress against schedules. Such a challenge can be attributed to (1) vision-based progress tracking lacking formal construction work templates applied in common construction workflows, and (2) research in automated schedule generation and analytics lacking focus on extracting fragnets from a body of existing schedules. This study brings together insights on research trends for automated schedule generation and analytics using Natural Language Processing (NLP) and detection of under-construction objects using Computer Vision. Finally, the AIConstruct system is presented to demonstrate, for the first time, how the integration of text and image can create seamless data synchronization for construction progress monitoring and automated schedule generation, unlocking a new research paradigm.
Get full access to this article
View all available purchase options and get full access to this chapter.
REFERENCES
4D Construction Group. (2020). “4d survey: The 4d construction group, <https://www.4d.construction/reports>(Jan).
Alikhani, H., Le, C., and Jeong, H. D. (2020). “A deep learning algorithms to generate activity sequences using historical as-built schedule data.” Creative Construction e-Conference 2020, Budapest University of Technology and Economics, 2–6 (July).
Amer, F., and Golparvar-Fard, M. (2019). “Automatic understanding of construction schedules: Part-of-activity tagging.” EC3 Conference 2019, Vol. 1, 190–197.
Amer, F., Jung, Y., and Golparvar-Fard, M. (2023). “Construction schedule augmentation with implicit dependency constraints and automated generation of lookahead plan revisions.” Automation in Construction, 152, 104896.
Amer, F., Koh, H. Y., and Golparvar-Fard, M. (2021). “Automated methods and systems for construction planning and scheduling: Critical review of three decades of research.” Journal of Construction Engineering and Management, 147(7), 03121002.
Autodesk. (2021). “Autodesk: The bim adoption curve.” Autodesk, <https://aec.autodesk.com/global-bim-study>(Jan).
Bang, S., and Kim, H. (2020). “Context-based information generation for managing uav-acquired data using image captioning.” Automation in Construction, 112, 103116.
Cho, J., and Lee, G. (2019). “A chatbot system for construction daily report information management.” ISARC. Proceedings of the International Symposium on Automation and Robotics in Construction, Vol. 36, IAARC Publications, 429–437.
Dimitrov, A., and Golparvar-Fard, M. (2014). “Vision-based material recognition for automated monitoring of construction progress and generating building information modeling from unordered site image collections.” Advanced Engineering Informatics, 28(1), 37–49 Copyright: Copyright 2014 Elsevier B.V., All rights reserved.
Dong, N., Fischer, M., Haddad, Z., and Levitt, R. (2013). “A method to automate look-ahead schedule (las) generation for the finishing phase of construction projects.” Automation in Construction, 35, pp. 157–173.
Gheisari, M., Rashidi, A., and Esmaeili, B. (2018). Using Unmanned Aerial Systems for Automated Fall Hazard Monitoring. 62–72.
Golparvar-Fard, M., Peña-Mora, F., and Savarese, S. (2015). “Automated progress monitoring using unordered daily construction photographs and ifc-based building information models.” Journal of Computing in Civil Engineering, 29(1), 04014025.
Guo, D., Onstein, E., and La Rosa, A. D. (2021). “A semantic approach for automated rule compliance checking in construction industry.” IEEE Access, 9, 129648–129660.
Hamledari, H., McCabe, B., and Davari, S. (2017). “Automated computer vision-based detection of components of under-construction indoor partitions.” Automation in Construction, 74, 78–94.
Han, K., Degol, J., and Golparvar-Fard, M. (2018). “Geometry- and Appearance-Based Reasoning of Construction Progress Monitoring.” Journal of Construction Engineering and Management, 144(2), 1–12.
Han, K., and Golparvar-Fard, M. (2014). “Automated monitoring of operation-level construction progress using 4d bim and daily site photologs.” 1033–1042 (05).
Han, K. K., and Golparvar-Fard, M. (2015). “Appearance-based material classification for monitoring of operation-level construction progress using 4d bim and site photologs.” Automation in Construction, 53, 44–57.
Hong, Y., Xie, H., Bhumbra, G., and Brilakis, I. (2022). “Improving the accuracy of schedule information communication between humans and data.” Advanced Engineering Informatics, 53, 101645.
Jobsite Editorial Staff. (2022). “The anatomy of a request for information.” Jobsite, <https://www.procore.com/jobsite/the-anatomy-of-a-request-for-information-rfi/>.
Jung, Y., Hockenmaier, J., and Golparvar-Fard, M. (2023a). “Feasibility analysis on the use of nlp-based schedule analytics for 4d project planning and controls.” Computing in Civil Engineering 2023, in press.
Jung, Y., Núñez-Morales, J., Hockenmaier, J., and Golparvar-Fard, M. (2023b). “Uniformatbridge: A transformer language model for mapping construction schedule activities to astm uniformat categories.” Automation in Construction, in preparation.
Kim, T., and Chi, S. (2019). “Accident case retrieval and analyses: Using natural language processing in the construction industry.” Journal of Construction Engineering and Management, 145(3), 04019004.
Lin, J. J., and Golparvar-Fard, M. (2020). Construction Progress Monitoring Using Cyber-Physical Systems. Springer International Publishing, Cham, 63–87.
Lin, J. J., Ibrahim, A., Sarwade, S., and Golparvar-Fard, M. (2021). “Bridge inspection with aerial robots: Automating the entire pipeline of visual data capture, 3d mapping, defect detection, analysis, and reporting.” Journal of Computing in Civil Engineering, 35(2), 04020064.
Lin, J. J., Lee, J. Y., and Golparvar-Fard, M. (2019). “Exploring the Potential of Image-Based 3D Geometry and Appearance Reasoning for Automated Construction Progress Monitoring.” Computing in Civil Engineering 2019, 3, 105–113.
Liu, H., Wang, G., Huang, T., He, P., Skitmore, M., and Luo, X. (2020). “Manifesting construction activity scenes via image captioning.” Automation in Construction, 119, 103334.
Malsane, S., Matthews, J., Lockley, S., Love, P. E., and Greenwood, D. (2015). “Development of an object model for automated compliance checking.” Automation in construction, 49, 51–58.
Marks, E., and Teizer, J. (2012). Proximity sensing and warning technology for heavy construction equipment operation.
Núñez-Morales, J., Hsu, S.-H., Ibrahim, A., and Golparvar-Fard, M. (2023). “New metrics to benchmark and improve bim visibility within a synthetic image generation process for computer vision progress tracking.” Proceedings of the Canadian Society for Civil Engineering Annual Conference 2023, in press.
Pan, Z., Su, C., Deng, Y., and Cheng, J. (2021). “Video2entities: A computer vision-based entity extraction framework for updating the architecture, engineering and construction industry knowledge graphs.” Automation in Construction, 125, 103617.
Pučko, Z., Šuman, N., and Rebolj, D. (2018). “Automated continuous construction progress monitoring using multiple workplace real time 3d scans.” Advanced Engineering Informatics, 38, 27–40.
Salama, D. M., and El-Gohary, N. M. (2016). “Semantic text classification for supporting automated compliance checking in construction.” Journal of Computing in Civil Engineering, 30(1), 04014106.
Shin, Y.-S., and Kim, J. (2022). “A vision-based collision monitoring system for proximity of construction workers to trucks enhanced by posture-dependent perception and truck bodies’ occupied space.” Sustainability, 14(13).
Soibelman, L., Wu, J., Caldas, C., Brilakis, I., and Lin, K.-Y. (2008). “Management and analysis of unstructured construction data types.” Advanced Engineering Informatics, 22(1), 15–27 Intelligent computing in engineering and architecture.
Tang, S., and Golparvar-Fard, M. (2021). “Machine learning-based risk analysis for construction worker safety from ubiquitous site photos and videos.” Journal of Computing in Civil Engineering, 35(6), 04021020.
Tang, S., Golparvar-Fard, M., Naphade, M., and Gopalakrishna, M. M. (2020a). “Video-based motion trajectory forecasting method for proactive construction safety monitoring systems.” Journal of Computing in Civil Engineering, 34(6), 04020041.
Tang, S., Roberts, D., and Golparvar-Fard, M. (2020b). “Human-object interaction recognition for automatic construction site safety inspection.” Automation in Construction, 120, 103356.
Tixier, A. J.-P., Hallowell, M. R., Rajagopalan, B., and Bowman, D. (2016). “Automated content analysis for construction safety: A natural language processing system to extract precursors and outcomes from unstructured injury reports.” Automation in Construction, 62, 45–56.
Torres-Calderon, W., Chi, Y., Amer, F., and Golparvar-Fard, M. (2019). “Automated mining of construction schedules for easy and quick assembly of 4d bim simulations.” Computing in Civil Engineering 2019, ASCE, 432–438.
Turkan, Y., Bosche, F., Haas, C. T., and Haas, R. (2012). “Automated progress tracking using 4d schedule and 3d sensing technologies.” Automation in Construction, 22, 414–421 Planning Future Cities-Selected papers from the 2010 eCAADe Conference.
Tuttas, S., Braun, A., Borrmann, A., and Stilla, U. (2014). “Comparision of photogrammetric point clouds with bim building elements for construction progress monitoring.” Vol. 1, 341–345 (08).
Wang, J., and Razavi, S. N. (2016). “Low false alarm rate model for unsafe-proximity detection in construction.” Journal of Computing in Civil Engineering, 30(2), 04015005.
Xiao, B., Wang, Y., and Kang, S.-C. (2022). “Deep learning image captioning in construction management: A feasibility study.” Journal of Construction Engineering and Management, 148(7), 04022049.
Xu, Z., Guan, H., Kang, J., Lei, X., Ma, L., Yu, Y., Chen, Y., and Li, J. (2022). “Pavement crack detection from ccd images with a locally enhanced transformer network.” International Journal of Applied Earth Observation and Geoinformation, 110, 102825.
Zhai, P., Wang, J., and Zhang, L. (2023). “Extracting worker unsafe behaviors from construction images using image captioning with deep learning–based attention mechanism.” Journal of Construction Engineering and Management, 149(2), 04022164.
Zhang, C., and Arditi, D. (2013). “Automated progress control using laser scanning technology.” Automation in Construction, 36, 108–116.
Zhang, F., Fleyeh, H., Wang, X., and Lu, M. (2019). “Construction site accident analysis using text mining and natural language processing techniques.” Automation in Construction, 99, 238–248.
Zhang, J., and El-Gohary, N. M. (2016). “Extending building information models semi-automatically using semantic natural language processing techniques.” Journal of Computing in Civil Engineering, 30(5), C4016004.
Zhang, S., Sulankivi, K., Kiviniemi, M., Romo, I., Eastman, C. M., and Teizer, J. (2015). “Bim-based fall hazard identification and prevention in construction safety planning.” Safety Science, 72, 31–45.
Zhong, B., Xing, X., Luo, H., Zhou, Q., Li, H., Rose, T., and Fang, W. (2020). “Deep learning-based extraction of construction procedural constraints from construction regulations.” Advanced Engineering Informatics, 43, 101003.
Zou, Y., Kiviniemi, A., and Jones, S. W. (2017). “Retrieving similar cases for construction project risk management using natural language processing techniques.” Automation in Construction, 80, 66–76.
Information & Authors
Information
Published In
Construction Research Congress 2024
Pages: 826 - 835
History
Published online: Mar 18, 2024
ASCE Technical Topics:
- Analysis (by type)
- Automation and robotics
- Business management
- Computer analysis
- Computer vision and image processing
- Construction engineering
- Construction management
- Detection methods
- Engineering fundamentals
- Geomatics
- Management methods
- Mapping
- Methodology (by type)
- Practice and Profession
- Scheduling
- Surveying methods
- Systems engineering
- Tracking
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.