Automatic Pixel-Level Detection of Structural Steel Elements Using U-Net and Transfer Learning

Leveraging deep learning models to automatically detect structural steel elements at the pixel level from visual data can support many potential use cases for construction projects, including automated construction progress monitoring, quality control, safety monitoring, and perception systems for construction robotics. Many researchers have been applying deep learning models to detect various construction entities on sites. However, limited research has focused on detecting structural steel elements at the pixel level. This research focused on implementing U-Net, a state-of-the-art deep learning model for semantic segmentation, to segment steel beams, girders, and columns from construction images. All images were captured by a drone from a three-story steel building construction site. A data augmentation process was performed to increase the quantity of the training data. Two pre-trained convolutional neural networks (CNNs), VGG16 and ResNet50, were applied to the U-Net as backbones, respectively, for transfer learning purposes. The U-Net models with different backbones were trained using different sizes of the training data, and the performance was analyzed and compared. Four metrics were used to quantitatively analyze the model performance: precision, recall, F1-score, and Intersection over Union (IoU). This research found that the transfer learning technique improved the model performance by 2%−6%, depending on the analyzed metric. The data augmentation also slightly boosted the model performance. However, both transfer learning and data augmentation increased the overfitting issue. The future work includes combining the segmented steel elements with the as-designed building information modeling (BIM) for automated construction progress monitoring.