Journal of Computing in Civil Engineering

Digital representation of robots as Avatars, called “RoboAvatars,” is a premise for value-added construction applications such as simulation, layout design, and task planning. Existing RoboAvatars are described in data schemas predominantly from the robotics community, which prevents their smooth applications in the built environment. To fully unleash the power of robotics, this research aims to develop a Building RoboAvatar by adopting the industry foundation classes (IFC) as the de facto standard in the building industry. First, the Building RoboAvatar is defined from a built environment perspective, and then substantiated with IFC. A translator called RoboIFCTrans is developed to facilitate the exploitation of the numerous readily available RoboAvatars represented by the Unified Robot Description Format. Experiments demonstrated the effectiveness of Building RoboAvatar in representing robot information needed for the built environment, which encompasses the “whole-part” robot structure and properties in terms of productivity, capability, etc. The RoboIFCTrans can accurately generate IFC representations of diverse robots (TurtleBot, UR-5, Diablo) within 41.9 s. Practical implications of the IFC-based Building RoboAvatars were illustrated by two use cases. The research contributes to building a “Tower of Babel” between the construction and robotics communities. The source code is made publicly open, in the hope of encouraging future research to explore more exciting opportunities (e.g., robot-oriented design, digital twin) enabled by the Building RoboAvatar.

Abstract

The rapid identification of structural damage in pavements remains a challenging issue. This paper proposes an automatic identification method that combines nondestructive testing, the ground penetrating radar (GPR), and object detection (OD) models for pavement structural assessment. Radar wave image data were collected from a 40 km single-lane track road and processed to create the training data set of 2,000 labeled images with two types of structural distress. Four OD models, including Faster R-CNN, YOLOv5, YOLOv8, and DETR, were adapted and trained on the data set to recognize structural damage in the images, and their prediction performances were compared. The results demonstrated that YOLOv8 was the state-of-the-art (SOTA) model, which achieved precise identification of the location and type of pavement structural damage with a mean average precision (mAP) of 98.5%. This study contributes to the improvement of latest OD models in the non-destructive inspection, facilitating the development of real-time automated pavement assessment and maintenance decision systems.

Practical Applications

This research addresses a common problem in civil engineering: how to quickly detect all hidden damage in pavement structures over long stretches of highway projects. Although there has been previous research in this area, finding a more reliable solution has proven to be a challenging goal. This study presents a method that utilizes the geological radar, a nondestructive testing technology, meaning it will not harm the road, along with artificial intelligence (AI) computer models to identify damage under the pavement surface. Think of it as teaching a computer to recognize cracks or looseness in the radar images of the road. Four different computer models were trained using 2,000 road images with known damage to teach them how to identify different damage features. The results were impressive—the computer could accurately identify the location and type of damage with a high level of confidence. In fact, the best model achieved a 98.5% score measured by mean average precision, which indicates its effectiveness. This research has significant practical applications as it streamlines the process of inspecting roads for structural problems, making it faster and more efficient. It can help with real-time evaluations and decisions about road maintenance.

Remotely operated vehicle (ROV) is a widely used subsea vehicle in offshore oil and gas industries to assist in the development and inspection of offshore oil fields, given its agility, safety, and endurance. As the offshore industry and subsea engineering fields advance, the demands on ROV technology, particularly in terms of control accuracy and safety, have been greatly increasing. Traditional methods require ROV operators to be stationed on vessels in challenging conditions, relying on video streams that offer limited spatial information about the ROV working environment. All these factors make conventional joystick controls even harder for novices. This research aims to propose a novel approach to human-autonomy collaboration aiming to diminish the learning and operational burdens on operators. A virtual reality (VR)-based sensory augmentation and body motion teleoperation method was applied, which allows operators to teleoperate the ROV in a much more comfortable environment. An interactive user interface was developed to enable seamless engagement with the autonomous system and facilitate dynamic switching between human and autonomous controls. To assess the system, a human subject experiment was conducted to compare the performance of an inspection task between human control, full autonomy, and human autonomy teaming methods. The result indicated that our solution could enhance human understanding of the ROV work status as well as reduce human workload during operation.

Artificial intelligence (AI) is increasingly aiding human decision makers in construction bidding processes by analyzing competitors’ bidding patterns. However, concerns are emerging about the potential for AI-driven algorithmic collusion, which might inflate prices and disrupt fair competition in various sectors. Given the unique dynamics of the construction sector and its growing reliance on AI, understanding the impact of these algorithms on the bidding landscape is essential, both academically and practically. Thus, this study investigates the impact of AI in the construction bidding market on bid pricing patterns to predict how the landscape of the market might change as AI starts to play a more prominent role. We subjected AI bidding agents to repeated competitions with each other in computer-simulated construction bidding marketplaces. We focused on the markup decisions made by the AI bidders. Our findings indicate that AI bidders tend to develop cooperative strategies over time, leading to higher bids overall compared to lower, competitive bids. This collusive behavior was facilitated by frequent interactions (previous bidding competitions over time) between AI bidders. This collusive behavior was enabled by algorithms that aimed to maximize the profit and was hindered by algorithms that aimed to maximize the number of project wins. These findings highlight potential fairness and competitiveness issues in construction bidding with dominant AI bidders.

Human–robot collaboration (HRC) has emerged as a promising frontier within the construction industry, offering the potential to enhance productivity, safety, and efficiency. The effectiveness of HRC critically depends on the degree of trust that workers place in their robots, and establishing a seamless level of trust in robots is essential to realize the full benefits of HRC. Despite the extensive exploration of trust dynamics in various industries, there is a notable research gap with regard to trust in construction robots, which possess distinctive characteristics in terms of appearance, capabilities, and interaction compared to robots in other sectors. Therefore, in this study, we analyzed trust dynamics within the context of HRC during construction tasks. Both subjective survey data and objective psychophysiological data—including heart rate variability (HRV), electrodermal activity (EDA), and electroencephalogram (EEG)-based emotional valence and arousal—were employed as human trust measures. We conducted experiments for bricklaying tasks in an immersive virtual construction environment and analyzed multifaceted robot factors—including workspace environment, level of interaction, and robot speed, proximity, and angle of approach—and their relationships with human trust measures using statistical analysis, such as $t$-test, two-way ANOVA, Spearman’s rank correlation, and moderation analysis. The results indicated that workspace environment and level of interaction were the most significant robot factors affecting human trust. EDA exhibited the most sensitivity to variations in robot factors. It was also observed that the effect of speed, proximity, and angle of approach were also dependent on level of interaction and type of workspace environment. There was a significant positive correlation between proximity and perceived trust. The findings of this study contribute to the optimization of robot design and interaction protocols for construction tasks, fostering greater worker trust, and enhancing project productivity and efficiency.

Abstract

Ensuring the alignment of perspectives between unmanned ground vehicles (UGVs) and Building Information Modeling (BIM) is crucial for the precise retrieval and analysis of BIM-stored information during inspection tasks. However, accumulative localization errors often result in deviations between the viewpoints of UGV cameras and their corresponding representations in BIM at specific waypoints. Therefore, this study introduces a sequential rectification method to correct the UGV’s pose within the BIM environment to ensure seamless alignment of perspectives. By leveraging visual features and geometric strategies in sequence, this method correlates the UGV-captured point cloud data with the BIM, thereby deriving accurate and robust pose rectification. Experimental validation in a featureless indoor environment demonstrated that this method reduced the angle and distance error of reference lines in two-dimensional (2D) views to approximately 2° and 7 pixels, respectively, and the root mean square error (RMSE) of three-dimensional (3D) lines to approximately 17 cm. The validation also demonstrated that the proposed method was particularly robust in correcting the pose and improving the alignment of perspectives between BIM and the UGV, even in cases of significant misalignment. Hence, this study improves the reliability of decisions made by UGVs for indoor inspection when cross-referencing with BIM data.

Practical Applications

UGVs have been leveraged to automate inspection tasks, thereby reducing human participation. Generally, these UGVs are programmed to perform analyses by comparing the as-built condition with the as-designed BIM. However, for these tasks to accurately reference the data stored in BIM, it is crucial to have a seamless alignment between the UGV’s perspectives and the as-designed BIM. Unfortunately, due to the accumulation of localization errors, discrepancies often arise between the perspectives in BIM and UGV cameras at waypoints. Such misalignments can compromise the reliability of decisions made for assigned tasks compared with BIM, such as verifying measurements of elements or confirming equipment installation during the project handover phase. Hence, this study, having demonstrated superior performance and robustness, could provide an effective solution to this alignment challenge. By ensuring seamlessly aligned perspectives, this method improves the reliability of UGV inspection assessments when accessing corresponding data stored in BIM.

Assigning repetitive and physically demanding construction tasks to robots can alleviate human workers’ exposure to occupational injuries, which often result in significant downtime or premature retirement. However, the successful delegation of construction tasks and the achievement of high-quality robot-constructed work requires transferring necessary dexterous and adaptive construction craft skills from workers to robots. Predefined motion planning scripts tend to generate rigid and collision-prone robotic behaviors in unstructured construction site environments. In contrast, imitation learning (IL) offers a more robust and flexible skill transfer scheme. However, the majority of IL algorithms rely on human workers repeatedly demonstrating task performance at full scale, which can be counterproductive and infeasible in the case of construction work. To address this concern, in this paper, we propose an immersive and Cloud Robotics-based virtual demonstration framework that serves two primary purposes. First, it digitalizes the demonstration process, eliminating the need for repetitive physical manipulation of heavy construction objects. Second, it employs a federated collection of reusable demonstrations that are transferable for similar tasks in the future and can, consequently, reduce the requirement for repetitive illustration of tasks by human agents. In addition, to enhance the trustworthiness, explainability, and ethical soundness of the robot training, this framework utilizes a hierarchical imitation learning (HIL) model to decompose human manipulation skills into sequential and reactive subskills. These two layers of skills are represented by deep generative models; these models enable adaptive control of robot action. The proposed framework has the potential to mitigate technical adoption barriers and facilitate the practical deployment of full-scale construction robots to perform a variety of tasks with human supervision. By delegating the physical strains of construction work to human-trained robots, this framework promotes the inclusion of workers with diverse physical capabilities and educational backgrounds within the construction industry.

Flaggers are always required to work close to open traffic lanes. They may be hit by distracted, speeding, or intoxicated drivers, leading to injuries and fatalities. To protect them, the concept of an automated flagging system device (AFSD) has been proposed. One of the core functions of an AFSD is to recognize the turn signals of vehicles in the lanes in order to guide the traffic. However, existing studies on vehicle turn-signal recognition have mainly focused on autonomous driving scenarios. Research on vehicle turn-signal recognition for AFSDs has been limited. In this study, we propose a novel method for vehicle turn-signal recognition using a video camera on an AFSD. The method first uses object detection and tracking to locate the vehicles and identify their front lighting areas (FLAs). Then, the luminance of each vehicle’s FLA is extracted. Based on the captured luminance features over time, a convolutional operator is applied to figure out whether the left or right FLA is flashing. The proposed method was implemented and tested on real traffic videos. The results showed that the overall signal recognition accuracy of the method reached 78.62%.

Abstract

Road markings are essential features to convey important information to various roadway users such as pedestrians, bicyclists, and motorists. Although mobile laser scanning (MLS) technology provides dense and spatially accurate data, automated identification of road markings remains a challenge. Although approaches have been developed to detect road markings from point clouds, prior studies have not thoroughly investigated road discretization and road marking clustering, which are essential for effective instance-level identification and management of road markings. To improve road discretization, our approach uses line segmentation to divide the trajectory of MLS data and fit circles to individual segments. By calculating the intersection between the road points and the circle’s center for each trajectory segment, this method ensures no overlap occurs when discretizing roads with sharp turns. To improve road marking clustering, the discretized points are rasterized onto a two-dimensional (2D) image and clustered using connected component labeling. Individual markings are skeletonized to detect junctions and corners, which enables the separation of underclustered road markings. Overclustered road markings are then merged using a rule-based approach optimized with a traffic line manual. The proposed approach was evaluated through extensive experiments using 28 MLS data sets containing 2,340 road marking clusters with complex geometry and wear. Out of these, only 64 instances (2.7%) were falsely clustered, achieving a precision rate of 97.5% and a recall rate of 99.7%.

Practical Applications

Efficient road maintenance and improved road safety rely heavily on the clear visibility of road markings, which guide diverse users, such as pedestrians, cyclists, and drivers. Traditional methods for inspecting these markings are often time-consuming and prone to human error. In this study, we use MLS technology to collect comprehensive and precise data on roadways. We have developed an automated technique that significantly enhances the identification and management of road markings. This method accurately rasterizes and segments roadway lidar data—even in areas with sharp turns—to apply advanced image processing for the recognition of individual markings, including those that are substantially worn or have complex geometries. This simplifies the geometric data in a GIS database such that each line object is more representative of how someone would manually digitize the marking. Tested on numerous data sets, our approach achieved an accuracy rate of over 97% in clustering individual road markings. The developed approach not only streamlines roadway maintenance but also offers potential benefits for intelligent transportation systems. It contributes to safer autonomous vehicle navigation and supports informed decision-making regarding road infrastructure, ultimately improving all road users’ experience.

Abstract

The high occurrence of work-related musculoskeletal disorders (WMSDs) in construction remains a pressing concern, causing numerous nonfatal injuries. Preventing WMSDs necessitates the implementation of an ergonomic process, encompassing the identification of ergonomic problems and corresponding solutions. Finding ergonomic problems and solutions within active construction sites requires significant efforts from personnel possessing ergonomics expertise. However, ergonomic experts and training programs are often lacking in construction. To address this issue, the authors applied deep learning (DL)–based explainable image captioning to identify ergonomic problems and their corresponding solutions from images that are prevalent in construction sites. To this end, the authors proposed a vision-language model (VLM) capable of identifying ergonomic problems and their solutions, aided by data augmentation. The bilingual evaluation understudy (BLEU) score was used to measure the similarity between ergonomic problems and solutions identified by the proposed VLM and those specified in an ergonomic guideline. Testing with 222 real-site images, the proposed VLM achieved the highest BLEU-4 score, 0.796, compared with the traditional convolutional neural network-long short-term memory and a state-of-the-art VLM, the bootstrapping language-image pretraining. In addition, the authors developed an explainability module, visualizing which specific areas of the images the proposed VLM focuses on when identifying ergonomic problems and the important words for identifying ergonomic solutions. The highest BLEU score and the visual explanations demonstrate the potential and credibility of the proposed VLM in identifying ergonomic problems and their solutions. The proposed VLM and explainability module greatly contribute to implementing the ergonomic process in construction, identifying ergonomic problems and their solutions only with site images.

Practical Applications

To prevent WMSDs, the National Institute of Occupational Safety and Health (NIOSH) recommends implementing an ergonomic process, which encompasses ergonomic problem identification, ergonomic risk assessment, and ergonomic solution identification. The current practice on sites relies on the intermittent implementation of manual ergonomic processes, and thus often falls short in protecting workers against WMSDs due to rapidly changing site conditions and the lack of on-site ergonomic expertise. Addressing this, many automated tools have been developed for ergonomic risk assessment, but none for ergonomic problem and solution identification. Therefore, with these assessment tools, we aim to streamline the recommended ergonomic process in an automated manner. To this end, we propose a deep learning–based explainable image captioning model for automated ergonomic problem and solution identification. Utilizing an ordinary camera (e.g., smartphones and site surveillance cameras), safety managers can easily identify ergonomic problems, assess risk levels, and identify corresponding solutions. Additionally, our model provides justification for its identification by visualizing the reason behind the identified ergonomic problems and solutions. With such an easily accessible and trustworthy model, the on-site ergonomic process can be streamlined, potentially reducing workers’ WMSDs.

Abstract

Tracking the postures of construction workers can provide precious information for safety management, occupational illness prevention, and productivity investigation. However, the posture data of construction workers is rarely utilized due to a lack of appropriate methods to track it. This research proposes a real-time multiworker posture tracking (MWPT) method to accurately track the postures of multiple workers onsite from video streams. It consists of three elements: image enhancement to adapt varying light conditions, posture detection for obtaining workers’ postures, and matching for tracking and retracking postures. In the field experiment, MWPT performed satisfactorily with an average of two ID switches (IDS), an average frame per second (FPS) of 11.0, and an average precision (AP@50) of 86.33. The results prove the capability of MWPT for tracking multiworker postures in real construction environments with high robustness and effectiveness. This research not only contributes an innovative tracking algorithm but also lays a stepping stone toward further worker posture-related research.

Practical Applications

This research introduces a posture tracking method for multiple construction workers. The motivation behind this research stems from the absence of an effective tracking method tailored to the demanding conditions of construction sites, such as variable lighting, extensive occlusions, and the challenge of distinguishing workers with similar appearances. The proposed solution leverages posture similarity for tracking workers and incorporates a retracking mechanism to address visual occlusions. The contrast limited adaptive histogram equalization method is employed to recover high-contrast visual features from dimly lit images, effectively addressing the issue of insufficient lighting. This method has been validated as both effective and efficient through customized test videos from actual construction sites. It shows promise for use in unsafe behavior detection, occupational disease prevention, and other posture-related research or applications as an effective tool.