Practical Deployment of BIM-Interoperable Voice-Based Intelligent Virtual Agent Support Construction Worker Productivity
Publication: Journal of Construction Engineering and Management
Volume 150, Issue 11
Abstract
The architecture, engineering, and construction (AEC) industry in the US faces increasing labor shortages while accumulating downward productivity trends. Previous research efforts have shown that lack of timely access to information is one of the main factors hindering construction workers’ productivity, and prompt information extraction (IE) from BIM sources can help address this issue. This study develops a BIM-web interoperable system, featuring a voice-based intelligent virtual agent (VIVA), to provide workers with information on-demand when conducting their activities on the jobsite. VIVA integrates construction digital sources with web platforms to enable real-time data communication between the users, workers in this case, and BIM platforms. The voice interaction feature of VIVA provides an intuitive and safe tool that can easily be used by workers with varying ranges of experience, with minimal need for training. The responsiveness and accuracy of VIVA is evaluated through two proof-of-concept case studies. Results indicate that the BIM-VIVA link and underlying natural language understanding (NLU) algorithms in the Google Actions platform accurately understand user queries, with a performance of 97.50% correct responses. As a result, VIVA achieves superior performance compared with previous research works based on NLU algorithms with text-based queries. The developments made in this research not only contribute to improving the safety and productivity of construction workers but also can open the door for easier onboarding of less-experienced workers, thus addressing the industry’s severe skilled labor shortage issue.
Practical Applications
This study introduces an innovative web-based system featuring VIVA to address the pressing labor shortage and diminishing productivity in the AEC industry. VIVA seamlessly integrates construction digital sources with web platforms, enabling real-time communication with construction workers and facilitating their on-demand information retrieval. The intuitive voice interaction feature of VIVA offers a safe and easily navigable tool that demands minimal training for workers with diverse backgrounds and experience levels. This feature also makes VIVA a practical tool for hands-free communication and information retrieval on jobsites, which minimizes the imposed distraction inherent in technological solutions. Construction workers can use voice commands to access project plans, schedules, and safety protocols, allowing them to retrieve critical information without having to stop work or use their hands. This intuitive interface reduces the need for complex user interfaces or manual input, making it easier for workers to interact with technology while focusing on their tasks. Hence, VIVA promises to improve productivity and safety of diverse group of workers without causing distraction or safety issues itself, which is a common concern when implementing technology to assist workers in hazardous and ever-changing construction jobsites. Beyond enhancing safety and productivity, widespread use of VIVA can help address the AEC sector’s severe skilled labor shortage by providing an accessible means to onboard less-experienced workers. As such, VIVA provides opportunities to augment efficiency for seasonal/nonnative workers, thus facilitating the integration of a broader workforce and addressing the industry’s pressing labor challenges.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code generated or used during the study are available in an online repository. Data Stream is publicly available in the citation provided (Linares-Garcia 2024b). Code is also publicly available in the citation provided (Linares-Garcia 2024a).
References
Afsari, K., C. Eastman, and D. Shelden. 2017. “Building information modeling data interoperability for cloud-based collaboration: Limitations and opportunities.” Int. J. Archit. Comput. 15 (3): 187–202. https://doi.org/10.1177/1478077117731174.
Ahmed, A. L., and M. Kassem. 2018. “A unified BIM adoption taxonomy: Conceptual development, empirical validation and application.” Autom. Constr. 96 (Mar): 103–127. https://doi.org/10.1016/j.autcon.2018.08.017.
Akanmu, A., and J. Olayiwola. 2019. “Towards a cyber-physical postural training system for construction workers.” In Proc., Advances in ICT in Design, Construction & Management in Architecture, Engineering, Construction and Operations. Newcastle upon Tyne, UK: Northumbria Univ.
Alexakis, G., S. Panagiotakis, A. Fragkakis, E. Markakis, and K. Vassilakis. 2019. “Control of smart home operations using natural language processing, voice recognition and IoT technologies in a multi-tier architecture.” Designs 3 (3): 32. https://doi.org/10.3390/designs3030032.
Alsafouri, S., and S. K. Ayer. 2019. “Mobile augmented reality to influence design and constructability review sessions.” J. Archit. Eng. 25 (3): 04019016. https://doi.org/10.1061/(ASCE)AE.1943-5568.0000362.
Anumba, C. J., A. Akanmu, X. Yuan, and C. Kan. 2020. “Cyber—Physical systems development for construction applications.” Front. Eng. Manage. 8 (Mar): 72–87. https://doi.org/10.1007/s42524-020-0130-4.
Associated Builders and Contractors. 2023. Construction workforce shortage tops half a million in 2023, says ABC. Washington, DC: Associated Builders and Contractors.
Bilal, M., L. O. Oyedele, J. Qadir, K. Munir, S. O. Ajayi, O. O. Akinade, H. A. Owolabi, H. A. Alaka, and M. Pasha. 2016. “Big data in the construction industry: A review of present status, opportunities, and future trends.” Adv. Eng. Inf. 30 (3): 500–521. https://doi.org/10.1016/j.aei.2016.07.001.
Bråthen, K., and A. Moum. 2016. “Bridging the gap: Bringing BIM to construction workers.” Eng. Constr. Archit. Manage. 23 (6): 751–764. https://doi.org/10.1108/ECAM-01-2016-0008.
Calvetti, D., P. N. M. Magalhães, F. Sujan Sujesh, M. C. Gonçalves, and H. J. Campos de Sousa. 2020. “Challenges of upgrading craft workforce into Construction 4.0: Framework and agreements.” Proc. Inst. Civ. Eng. Manage. Procure. Law 173 (4): 158–165. https://doi.org/10.1680/jmapl.20.00004.
Chen, H., L. Hou, G. Zhang, and S. Moon. 2021. “Development of BIM, IoT and AR/VR technologies for fire safety and upskilling.” Autom. Constr. 125 (Dec): 103631. https://doi.org/10.1016/j.autcon.2021.103631.
Dai, J., P. M. Goodrum, and W. F. Maloney. 2009. “Construction craft workers’ perceptions of the factors affecting their productivity.” J. Constr. Eng. Manage. 135 (3): 217–226. https://doi.org/10.1061/(ASCE)0733-9364(2009)135:3(217).
Dave, B., A. Buda, A. Nurminen, and K. Främling. 2018. “A framework for integrating BIM and IoT through open standards.” Autom. Constr. 95 (Dec): 35–45. https://doi.org/10.1016/j.autcon.2018.07.022.
Dy, E. A. Z., D. C. Edusada, J. L. A. Robles, A. P. Triñona, I. K. B. Camacho, M. G. V. Calilung, and F. D. Poso. 2021. “Construction labor productivity in construction sites during the COVID-19 pandemic using relative importance index (RII).” In Proc., 2021 IEEE 13th Int. Conf. on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment, and Management (HNICEM), 1–6. New York: IEEE.
Fan, H., F. Xue, and H. Li. 2015. “Project-based as-needed information retrieval from unstructured AEC documents.” J. Manage. Eng. 31 (1): A4014012. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000341.
Gerami Seresht, N., and R. Fayek Aminah. 2018. “Dynamic modeling of multifactor construction productivity for equipment-intensive activities.” J. Constr. Eng. Manage. 144 (9): 04018091. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001549.
Han, T., T. Ma, Z. Fang, Y. Zhang, and C. Han. 2022. “A BIM-IoT and intelligent compaction integrated framework for advanced road compaction quality monitoring and management.” Comput. Electr. Eng. 100 (Dec): 107981. https://doi.org/10.1016/j.compeleceng.2022.107981.
He, T., and F. Jazizadeh. 2022. “Nudging occupants for energy-saving through voice-based proactive virtual assistants.” In Proc., Construction Research Congress 2022, 402–411. Reston, VA: ASCE.
Kan, C., C. J. Anumba, and J. I. Messner. 2017. “Potential use of cyber-physical systems (CPS) for planning and operation of mobile cranes on construction sites.” In Proc., ASCE Int. Workshop on Computing in Civil Engineering 2017, 139–146. Reston, VA: ASCE.
Kan, C., C. J. Anumba, F. Yihai, and J. I. Messner. 2018. “A cyber-physical system (CPS) for planning and monitoring mobile cranes on construction sites.” Proc. Inst. Civ. Eng. Manage. Procure. Law 171 (6): 240–250. https://doi.org/10.1680/jmapl.17.00042.
Karimi, H., T. R. B. Taylor, G. B. Dadi, P. M. Goodrum, and C. Srinivasan. 2018. “Impact of skilled labor availability on construction project cost performance.” J. Constr. Eng. Manage. 144 (7): 04018057. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001512.
Klötzer, C., J. Weißenborn, and A. Pflaum. 2017. “The evolution of cyber-physical systems as a driving force behind digital transformation.” In Proc., 2017 IEEE 19th Conf. on Business Informatics (CBI), 5–14. New York: IEEE.
Li, C. Z., F. Xue, X. Li, J. Hong, and G. Q. Shen. 2018. “An internet of things-enabled BIM platform for on-site assembly services in prefabricated construction.” Autom. Constr. 89 (Mar): 146–161. https://doi.org/10.1016/j.autcon.2018.01.001.
Li, J., N. Li, K. Afsari, Z. Wu, and H. Cui. 2019. “Integration of building information modeling and web service application programming interface for assessing building surroundings in early design stages.” Build. Environ. 153 (Mar): 91–100.
Li, R. Y. M., and T. H. Leung. 2020. “Computer vision and hybrid reality for construction safety risks: A pilot study.” In Proc., 4th Int. Congress on Information and Communication Technology, 17–22. Singapore: Springer.
Lin, J.-R., Z.-Z. Hu, J.-P. Zhang, and F.-Q. Yu. 2016. “A natural-language-based approach to intelligent data retrieval and representation for cloud BIM.” Comput.-Aided Civ. Infrastruct. Eng. 31 (1): 18–33. https://doi.org/10.1111/mice.12151.
Linares-Garcia, D. A. 2024a. “VIVA cloud functions code.” Accessed March 27, 2024. https://doi.org/10.5281/zenodo.10888401.
Linares-Garcia, D. A. 2024b. “VIVA JSON Stream.” Accessed March 27, 2024. https://sheetdb.io/api/v1/59icmymv3xo4w.
Linares-Garcia, D. A., G. Flores-Linares, and N. Roofigari-Esfahan. 2022a. “Framework and case studies for context-aware AR system (CaARS) for ubiquitous applications in the AEC Industry.” In Proc., Construction Research Congress 2022, 1278–1288. Amsterdam, Netherlands: Elsevier.
Linares-Garcia, D. A., N. Roofigari-Esfahan, K. Pratt, and M. Jeon. 2022b. “Voice-based intelligent virtual agents (VIVA) to support construction worker productivity.” Autom. Constr. 143 (Dec): 104554. https://doi.org/10.1016/j.autcon.2022.104554.
Moreira, L. C., P. P. Mota, and F. A. Machado. 2021. “BIM, IoT and MR integration applied on risk maps for construction.” In Proc., 18th Int. Conf. on Computing in Civil and Building Engineering, 895–906. New York: Springer.
Muñoz-La Rivera, F., J. Mora-Serrano, I. Valero, and E. Oñate. 2020. “Methodological-technological framework for construction 4.0.” Arch. Comput. Methods Eng. 28: 689–711. https://doi.org/10.1007/s11831-020-09455-9.
Neve, H. H., S. Wandahl, S. Lindhard, J. Teizer, and J. Lerche. 2020. “Determining the relationship between direct work and construction labor productivity in North America: Four decades of insights.” J. Constr. Eng. Manage. 146 (9): 04020110. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001887.
Niknam, M., and S. Karshenas. 2017. “A shared ontology approach to semantic representation of BIM data.” Autom. Constr. 80 (Dec): 22–36. https://doi.org/10.1016/j.autcon.2017.03.013.
Omar, H., L. Mahdjoubi, and G. Kheder. 2018. “Towards an automated photogrammetry-based approach for monitoring and controlling construction site activities.” Comput. Ind. 98 (Jun): 172–182. https://doi.org/10.1016/j.compind.2018.03.012.
Ozturk, G. B., and I. Yitmen. 2019. “Conceptual model of building information modelling usage for knowledge management in construction projects.” IOP Conf. Ser.: Mater. Sci. Eng. 471 (Jun): 022043. https://doi.org/10.1088/1757-899X/471/2/022043.
Pan, Y., and L. Zhang. 2021. “A BIM-data mining integrated digital twin framework for advanced project management.” Autom. Constr. 124 (May): 103564. https://doi.org/10.1016/j.autcon.2021.103564.
Park, J., and Y. K. Cho. 2018. “Use of a mobile BIM application integrated with asset tracking technology over a cloud.” In Proc., 21st Int. Symp. on Advancement of Construction Management and Real Estate, 1535–1545. Singapore: Springer.
Park, M., K.-W. Lee, H.-S. Lee, P. Jiayi, and J. Yu. 2013. “Ontology-based construction knowledge retrieval system.” KSCE J. Civ. Eng. 17 (7): 1654–1663. https://doi.org/10.1007/s12205-013-1155-6.
Preidel, C., S. Daum, and A. Borrmann. 2017. “Data retrieval from building information models based on visual programming.” Visualization Eng. 5 (1): 18. https://doi.org/10.1186/s40327-017-0055-0.
Puri, N., and Y. Turkan. 2018. “A review of technology supplemented progress monitoring techniques for transportation construction projects.” In Proc., Construction Research Congress 2018. Reston, VA: ASCE. https://doi.org/10.1061/9780784481264.050.
Pyae, A., and P. Scifleet. 2018. “Investigating differences between native English and non-native English speakers in interacting with a voice user interface: A case of Google home.” In Proc., 30th Australian Conf. on Computer-Human Interaction, Association for Computing Machiner, 548–553. New York: Association for Computing Machinery.
Rajpurkar, P., R. Jia, and P. Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD, 784–789. Stroudsburg, PA: Association for Computational Linguistics. https://doi.org/10.18653/v1/P18-2124.
Rüßmann, M., M. Lorenz, P. Gerbert, M. Waldner, J. Justus, P. Engel, and M. Harnisch. 2015. “Industry 4.0: The future of productivity and growth in manufacturing industries.” Boston Consulting Group 9 (1): 54–89. https://doi.org/10.1016/j.autcon.2011.05.012.
Ruwanpura, J. Y., K. N. Hewage, and L. P. Silva. 2012. “Evolution of the i-Booth© onsite information management kiosk.” Autom. Constr. 21 (Dec): 52–63. https://doi.org/10.1016/j.autcon.2011.05.012.
Sacks, R., M. Radosavljevic, and R. Barak. 2010. “Requirements for building information modeling based lean production management systems for construction.” Autom. Constr. 19 (5): 641–655. https://doi.org/10.1016/j.autcon.2010.02.010.
Sciullo, L., I. D. R. Zyrianoff, A. Trotta, and M. D. Felice. 2021. “WoT micro servient: Bringing the W3C web of things to resource constrained edge devices.” In Proc., 2021 IEEE Int. Conf. on Smart Computing (SMARTCOMP) 161–168. New York: IEEE.
Shahinmoghadam, M., and A. Motamedi. 2021. “An ontology-based mediation framework for integrating federated sources of BIM and IoT data.” In Proc., 18th Int. Conf. on Computing in Civil and Building Engineering, 907–923. New York: Springer.
Shin, S., and R. A. Issa Raja. 2021. “BIMASR: Framework for voice-based BIM information retrieval.” J. Constr. Eng. Manage. 147 (10): 04021124. https://doi.org/10.1061/(ASCE)CO.1943-7862.0002138.
Son, H., C. Kim, and Y. Cho. 2017. “Automated schedule updates using as-built data and a 4D building information model.” J. Manage. Eng. 33 (4). https://doi.org/10.1061/(ASCE)ME.1943-5479.0000528.
Srewil, Y., and R. J. Scherer. 2017. Construction objects recognition in framework of CPS, 2472. New York: IEEE.
Sveikauskas, L., S. Rowe, J. Mildenberger, J. Price, and A. Young. 2016. “Productivity growth in construction.” J. Constr. Eng. Manage. 142 (10): 04016045. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001138.
Tang, S., D. R. Shelden, C. M. Eastman, P. Pishdad-Bozorgi, and X. Gao. 2019. “A review of building information modeling (BIM) and the internet of things (IoT) devices integration: Present status and future trends.” Autom. Constr. 101 (Mar): 127–139. https://doi.org/10.1016/j.autcon.2019.01.020.
Turkan, Y., F. Bosché, C. T. Haas, and R. Haas. 2013. “Toward automated earned value tracking using 3D imaging tools.” J. Constr. Eng. Manage. 139 (4): 423–433. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000629.
Valinejadshoubi, M., O. Moselhi, A. Bagchi, and A. Salem. 2021. “Development of an IoT and BIM-based automated alert system for thermal comfort monitoring in buildings.” Sustainable Cities Soc. 66 (Jun): 102602. https://doi.org/10.1016/j.scs.2020.102602.
Wang, H., and X. Meng. 2019. “Transformation from IT-based knowledge management into BIM-supported knowledge management: A literature review.” Expert Syst. Appl. 121 (May): 170–187. https://doi.org/10.1016/j.eswa.2018.12.017.
Wang, H., X. Meng, and X. Zhu. 2022a. “Improving knowledge capture and retrieval in the BIM environment: Combining case-based reasoning and natural language processing.” Autom. Constr. 139 (May): 104317. https://doi.org/10.1016/j.autcon.2022.104317.
Wang, J., and S. Razavi. 2019. “Integrated & automated systems for safe construction sites.” Prof. Saf. 21 (3): 41–48. https://doi.org/10.1109/MITP.2018.2876986.
Wang, N., R. Issa, and C. Anumba. 2021. “A framework for intelligent building information spoken dialogue system (iBISDS).” In Proc., EG-ICE 2021 Workshop on Intelligent Computing in Engineering. Berlin: Technische Universität Berlin.
Wang, N., R. R. A. Issa, and C. J. Anumba. 2022b. “NLP-based query-answering system for information extraction from building information models.” J. Comput. Civ. Eng. 36 (3): 04022004. https://doi.org/10.1061/(ASCE)CP.1943-5487.0001019.
Wang, X., M. Truijens, L. Hou, Y. Wang, and Y. Zhou. 2014. “Integrating augmented reality with building information modeling: Onsite construction process controlling for liquefied natural gas industry.” Autom. Constr. 40 (Apr): 96–105. https://doi.org/10.1016/j.autcon.2013.12.003.
Wu, Y., D. Rough, A. Bleakley, J. Edwards, O. Cooney, P. R. Doyle, L. Clark, and B. R. Cowan. 2020. “See what I’m saying? Comparing intelligent personal assistant use for native and non-native language speakers.” In Proc., 22nd Int. Conf. on Human-Computer Interaction with Mobile Devices and Services, New York: Association for Computing Machinery.
Yeh, K.-C., M.-H. Tsai, and S.-C. Kang. 2012. “On-site building information retrieval by using projection-based augmented reality.” J. Comput. Civ. Eng. 26 (3): 342–355. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000156.
Zhou, H., M. O. Wong, H. Ying, and S. Lee. 2018. “A framework of multi-user voice-driven BIM-based navigation system for fire emergency response.” In Proc., 26th Int. Workshop on Intelligent Computing in Engineering. Berlin: Technische Universität Berlin.
Information & Authors
Information
Published In
Journal of Construction Engineering and Management
Volume 150 • Issue 11 • November 2024
Copyright
© 2024 American Society of Civil Engineers.
History
Received: Sep 1, 2023
Accepted: May 14, 2024
Published online: Aug 24, 2024
Published in print: Nov 1, 2024
Discussion open until: Jan 24, 2025
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.