Retracted: Object-Oriented Damage Information Modeling Concepts and Implementation for Bridge Inspection

Five implementation methods to extend the BIM concept are described in the literature. First, the BIM Collaboration Format (BCF) was designed for sharing change requirements of building models during design, planning, and construction (buildingSMART International Ltd. 2009). BCF files are additional files besides the building information and contain one or more issues with a camera position, photos, and descriptions. These files do not contain building information and hence are not capable of providing damage data, such as geometries or properties of a defect.

Another possibility is to use proprietary and custom data formats, e.g., Autodesk Revit or Microsoft Excel files (McGuire et al. 2016). The most noteworthy advantages of proprietary data formats are the availability of existing tools and hence less effort for software implementation. However, those concepts are not capable of storing all information related to defects, for example, Excel is not made for storing geometry data.

In addition to the mentioned proprietary data formats, open data formats are available, for example, the IFC (Borrmann et al. 2018). IFC is a comprehensive data format for BIM and provides numerous possibilities for modeling semantics and geometries. Additionally, several authoring tools allow exports to IFC and numerous IFC viewers exist. Hence, less implementation efforts are required if a damage-loaded BIM could be modeled by means of IFC. Although the IFC standard is extensive, it lacks specific relationships and entities for defects and damaged components. To overcome this gap, extensions of the IFC have been developed, for example, by Tanaka et al. (2018). The biggest disadvantage of such extensions is that extending the IFC requires further implementation work to read, write, visualize, and edit the resultant data model.

Linked data models incorporate IFC data and associate data via links to other models or model types. One example is the linking of IFC data with semantic web ontologies. All building information is stored in an IFC file and damage information is stored in a separate damage model. Both are linked via a link model (Hamdan and Scherer 2018; Hamdan et al. 2019). On the one hand, a linked data or multimodel concept is highly flexible and semantic web allows defining custom relationships (W3C 2012). On the other hand, distributing data over two models leads to data dispersion and again the necessity of implementing your own viewers and authoring tools arises.

Several open-source IFC viewers exist, such as the Java IFC Toolkit or the xBIM Xplorer. These viewers and related libraries cope with reading, writing, visualizing, and editing IFC files. Extensions to these software tools require less effort compared to developing software from scratch. This leads to the decision to implement the conceptual model by using the IFC 4 standard (buildingSMART International Ltd. 2016).

Current and future inspection practices require a suitable DIM with geometric, visual, and semantic damage data. Heretofore, several DIMs with regard to visualization and semantic information have been developed. However, each focuses on a specific aspect, such as visualization or simulation. Some models include semantic data and some include visual data while others include geometries. This study focuses on a comprehensive DIM that incorporates all three aspects and is usable for multiple use cases. Simply combining existing approaches would not be possible because this would lead to redundant information in the model, for example, the different approaches of modeling defect geometries by Isailovic et al. (2020) and Hamdan and Scherer (2018). Instead, a requirement analysis under the consideration of the existing work is necessary to identify the model requirements and address them in the data model. This incorporates synergizing existing models and concepts and adding new aspects, such as multiple defect representations and mapping information.

The objectives of this study were to address the following research questions:

Semantic data refer to data that describe the meaning or content of a defect. Table 1 gives the content of the semantic data. The first semantic information is that there is a defect or damaged component. Hearn (2007) showed in his comparison of national inspection practices that nations acquire either defect- or component-centered data for bridge assessment. Using a defect entity enables supporting both assessment practices. A component with a related defect may be easily abstracted to a damaged component. The other way around would be considerably more difficult. Hence, a class for an individual defect is necessary. Such a defect object requires an identifier, name, and description to cope with textual information.

Bridge assessment relies on defect relationships. The first relationship is with the damaged component. Second, a defect may consist of several part defects, for example, a crack map consists of several cracks. Third, a defect can have another defect as a reason; for example, a spalling is the reason behind an exposed and corroded rebar. Last, a defect has associated documents, like reports.

The entire data of a defect require a relation to time because inspections are snapshots in time. The defect entity should stay the same because only the parameters of the defect vary with time and not the defect itself. A special case could be if two defects become one, such as two cracks growing and becoming one crack. It can be concluded that the defect entity persists over time. In contrast, the associated data represent a snapshot.

Different defect types have different effects, which gives rise to a variation in assessment results. Artus and Koch (2020) defined 12 damage types. In general, different damage types, which may occur at bridges, need to be supported. Some examples are cracks, spalling, and corrosion. A damage of a specific type is represented by a damage type class and a relationship between the defect and the damage type. Finally, semantic data contain defect properties. Such properties may contain textual descriptions, implications, measurements, or condition ratings. These properties may be grouped together on the basis of their context, meaning, or time. This leads to using one class to group properties and one class for each property itself. A property must have a name and value. A description and unit are optional. All aforementioned semantic information may be used to automatically check for specified rules later (Ren et al. 2019). Furthermore, probabilistic simulations use semantic data for predicting bridge conditions (Calvert et al. 2020).

Hearn (2007) emphasized the importance of photos and sketches during inspection, which has necessitated their inclusion in the data model. Photos or sketches are stored in files, which are linked to the defect. Findings of state-of-the-art research have confirmed the benefit of using UAS for visual inspection (Morgenthal et al. 2019). Using video-capturing drones leads to a large number of photos or even video streams. Hence, a DIM must offer storing several photos of a defect or a video stream. This approach is similar to the aforementioned reports and hence the same approach is applicable. Finally, photos may be used as textures for depicting the defect’s image at the correct model position. For this purpose, rectified photos are necessary. Furthermore, a texture-mapping algorithm or a mapping table is necessary to properly attach the image. Both need to be stored in case of using textures. If the texture should be depicted only at a specific part of a component geometry, this part needs to be marked.

Geometry data consist of one-dimensional (1D), 2D, and 3D geometry data. One-dimensional geometry denotes the defect position. The position is important for subsequent assessment; for example, cracks in the deck above the expansion joint could indicate a too small or missing expansion joint.

Two-dimensional geometry data could be poly lines or plane sketches. Instead of adding photos of sketches, inspectors could add those sketches to the model, which would be machine-readable. Sketches are relevant in case of bad illumination because inspectors create sketches to provide additional geometric information for the assessment afterward. Furthermore, on-site inspections primarily rely on paper-based plans. To support this use case, a model of a damaged bridge should contain plans in addition to the 3D model. This leads to the point where a defect has multiple geometries with different contexts.

Automated damage detection systems generate 3D damage geometry data. UASs capture photos or videos during their flight around the bridge. Structure-from-motion (SfM) algorithms are employed to generate point clouds and meshes. Within point clouds, further annotations are possible to accentuate damaged regions (Morgenthal et al. 2019). The points of a point cloud are used to generate meshes of a defect to retrieve solid geometries (Isailovic et al. 2020). Three-dimensional geometries provide damage-related information for structural bridge simulations and bridge assessments. Refer to the “Damage Information Modeling” section for explanations on how to use 3D geometry data for structural bridge simulations.

In summary, the following essential requirements must be addressed:

Hamdan and Scherer (2018) and Sacks et al. (2018a) covered semantic data in their studies in the form of defect relations and measurements. The concepts presented in this paper are based on their work in a sense that relationships and measurements are included. Fig. 3 shows the unified modeling language (UML) diagram for semantic defect data. Building products and subclasses represent bridge parts and semantic damage data are the annotation and relation. To address the target of a defect entity, a DefectAnnotation is defined, which consists of a name, ID, and description. Relationships are the next point in the list of requirements. The relationship between the defect and the building product is designed via an objectified relationship DefectProductRelation. This relationship has two attributes: relatingProduct points to the affected bridge component and relatedDefect points to the affecting defect. Such an objectified relationship has the advantage that it can contain additional information, for example, the relation type. The CauseEffectRelation offers the possibility to represent causing and resulting defects. The causingDefect points to the defect that is the reason for the resultingDefect. This relationship is an $m$ to $n$ relationship because a defect may have multiple causing defects and one defect may have several resulting defects. An objectified relationship has been used to cope with this requirement of an $m$ to $n$ relationship. Related documents may be referenced via the DocumentReference class. This points to a general object because the documents may be related to a specific defect, such as photos of a defect, or to a building product, such as an ultrasonic survey of a bridge component. Furthermore, the class contains an identifier, name, description, and uniform resource identifier (URI) about the referenced document.

To address the requirement of a defect type, the DefectAnnotation is provided with the defectType attribute, which is of type DefectType. The DefectType class has a name and description. Possible names could be crack, spalling, corrosion, or similar. The defect properties are added by the class Measurement and grouped via a MeasurementSet. A measurement contains a name, value, and an optional unit. In summary, this part of the model covers the requirements of modeling semantic data as shown by Table 1.

Hüthwohl et al. (2018) proposed defect representations using textures; however, their approach misses the requirement of including multiple photos and necessary texturing information. Addressing the requirement of storing multiple photos, they can be referenced via DocumentReference. Fig. 4 shows the use of a photo as texture. The DefectAnnotation has a Texturing with the texture area, mapping information, and URI. A texture is applied to the texture area by mapping the image from the URI with the given texture mapping. TextureMappingAlgorithm is a subclass of TextureMapping and represents generic algorithms to calculate texture mappings, like spherical texture mapping. TextureGeometryMap represents a point-based texture map also as subclass of TextureMapping. A TextureArea defines the location of a texture. This may be the entire geometry of a defect or component or it is a part of those geometries.

Physical defects lead to cutouts in component geometries. Either this is derived from the relationship between the damage and the component, or the relationship between the component and the damage is modeled separately from the geometry of the damaged component. The advantage of interpreting the geometry on the basis of the relationship between the component and its defect is that fewer entities are necessary, which entails less risk for incoherence. However, including multiple geometries is defined as a requirement, but calculating the geometry on the basis of the relationship does not offer involving multiple geometries for a defect or damaged component. A distinction of relationships and geometries enables including multiple geometries of defects and damaged components. For the sake of completeness, both methods are modeled, implemented, and tested. These methods have in common that the defect and component have individual geometries; furthermore, the concept of constructive solid geometry (CSG) s used to calculate the geometry of a damaged component for both approaches.

Fig. 7 depicts the entire UML model including both geometry modeling approaches. The geometry may be included by the DamagedGeometryCutout relation or by the DefectProductRelation in combination with CSG geometries.

Up to now, there have been no specific defect entities implemented in the IFC. However, the IFC offers several alternatives: IfcProxy, IfcAnnotation, IfcSurfaceFeature, and IfcVoidingFeature. Table 2 compares the advantages and disadvantages of these IFC entities. IfcProxy is a generic entity, but IFC 4 lists the proxy as deprecated and recommends using IfcBuildingElementProxy instead. A look at newer versions of the IFC reveals the proxy is marked deprecated no longer. The DIM should be usable in future standards and hence the proxy is taken as an option instead of BuildingElementProxy. A proxy is very flexible and thus suitable for every defect type. However, a proxy is treated as an individual element or building element, which conflicts with the nature of a defect. A defect cannot exist without the affected component. An IfcAnnotation may be used to add further annotations to a component. However, IfcAnnotations are only meant to have zero-dimensional, 1D, or 2D geometries. IfcSurfaceFeatures and IfcVoidingFeatures are suitable for specific defects. Surface features are suitable for modeling corrosion or other defects that only affect the surface of a component. Voiding features are suitable for defects like cracks or spalling, which add a void to a component. However, they are not suitable for modeling further damage types, e.g., material changes below the surface, divergences from specifications, or washouts. On the whole, depending on the defect, a suitable IFC entity must be chosen. Corrosion or other surface changes that affect surface features. Cracks and spalling are represented by voiding features at best. Other defects could use either annotations or proxies.

In the next step, the analysis of suitable relationships from the IFC standard is presented. IfcRelAssigns “is a generalization of ‘link’ relationships among instances of IfcObjects and its various $1^{\mathrm{st}}$ level subtypes” (buildingSMART International Ltd. 2018b). A specific identification of the relationship is stored as the name of the relationship. In case of the DefectProductRelation, the name of IfcRelAssignsToProdcut would be defect product relation. However, a defect is part of a component and if the component is destroyed the defect no longer exists. Hence, a composition is more precise. Strict compositions are modeled with IfcRelAggregates. Construction and design practice understands aggregations as a sum of different products. This would imply a defect is a product if an aggregation is used, which is questionable. Altogether, aggregations seem to be the most precise relationship for physical defects. Both relationships of the aggregation and the assignment may be used for other defects. In case of using IfcVoidingFeature to represent defects, the decomposition relationship IfcRelVoidsElement is suitable. “IfcRelVoidsElement is an objectified relationship between a building element and one opening element that creates a void in the element” (buildingSMART International Ltd. 2018b). As stated previously, the voiding feature and the voids relationship are only applicable to cracks or spalling and not in case of material changes or other damage types. An example is depicted in Fig. 8. IfcRelAssignsToProdcut may be used for effect-cause relations with the name “cause” or “reason.” Additional information about the relation might be given by the description of the relationship. Table 3 provides an overview of the existing relationships and related advantages and disadvantages.

To model damage types, IFC type objects are suitable entities. Furthermore, property sets or templates are suitable for defining type-related properties or necessary properties for entities of a type, for example, measurements, the condition rating, dates, or further descriptions. Document references link to existing inspection reports, results of additional surveys, or photos. Fig. 9 depicts an IFC file that includes a damage type, some measurements, and a document reference.

This subsection discusses the implementation of geometries of the DIM by using the IFC. In addition to modeling the geometry of the damaged component, the method of modeling a defect geometry and the use of geometric representation contexts are illustrated.

Visual data may be stored as document references or as textures, which are depicted on a 3D surface. Document references were discussed in the “IFC Classes for Semantic Data” section. Coming to the implementation of textures, Fig. 12 illustrates how to include a texture in an IFC file. To position an image, for example, a png file, within the 3D model, a geometry is necessary. This geometry is represented by the IfcRepresentationItem. Such a geometry could be a plane, which carries the texture slightly above the related position of the affected component. A listing example can be found in the study by Hüthwohl et al. (2018). In addition to the texture itself, the mapping is necessary. The IFC offers the class IfcTextureCoordinate to add texture-mapping information and subclasses, such as IfcTextureCoordinateGenerator and IfcIndexedTriangleTextureMap, to either define an algorithmic or point-based texture mapping.

During the tests, which are explained in the next section, two problems could be observed:

To verify the entire model, one software tool had to be extended; xBIM was chosen for the same.

In the first step, the functionality of visualizing semenatic data was tested. All four IFC entities, i.e., IfcAnnotation, IfcProxy, IfSurfaceFeature, and IfcVoidingFeature, were tested. The expectation is that the software provides a geometric view, a hierarchical tree view, and a view for the properties. Table 5 presents an overview of the test results. Revit, Desite BIM, and Solibri Model Viewer lack the hierarchical view of the model and hence the defects without geometries could not be selected. Furthermore, none of the three includes a hierarchical view of the model. All other software visualizes the test files properly.

Next, the visualization of the relationships was tested. For this purpose, typification, external references, and defect relationships were added. Classification could be visualized via a property view or by using the correct product type. Table 6 summarizes the test results. IFC viewers do not access product catalogs and hence the type is shown as property in the view. Revit uses its internal type catalog to select the corresponding type of an entity. However, this is only possible if the typification is stored with correct Revit family names. The same problem arises with measurements or properties in Revit. External references should be shown at least in the property view with their URI. The apstex IFC viewer and xBIM show external references in such a way. None of the other software tools showed the external document references. Last, defect relationships, i.e., aggregation, association, or voids element, should be shown in the hierarchical view or as properties. xBIM and apstex show aggregations in the hierarchical view and associations as properties. BIM Vision was able to show aggregations but not the associations.

Textures are the second requirement in the data model. To test texturing, an image was attached to an additional plane, which is at the defect position. Other geometries may be used instead of a plane. As depicted by the last row in Table 7, none of the available software was able to properly visualize the texture. Most ignored the texture parameter. usBIM only shows the plane where the texture should be depicted. xBIM Xplorer has received an extension to visualize textures on a geometries with a given texture mapping. Fig. 14 shows an example of a beam with a texture representing a defect.

Geometric representations are very common in the AEC sector. However, the software programs support the geometric representations in different quality, which is evidenced in Table 7. The visualization of CSG geometries was done properly by all IFC viewers except Desite BIM and the Solibri Model Viewer. None of the viewers that are available by the software vendors offer a selection of representation context. This requirement was achieved only by Revit. Revit includes 2D plans and 3D views for its building models; however, multiple 3D geometries are not possible in Revit. The custom extended version of xBIM was able to offer the context selection with multiple 3D geometries.

The next step tested the visualization of an IfcVoidingFeature with an IfcRelVoidsElement relationship in accordance with the relationship-based cutout. The voiding feature is correctly supported by apstex’s IFC Viewer and xBIM Xplorer. Other programs do not respect an IfcVoidingFeature with an IfcRelVoidsElement relationship. Many viewers are able to handle an opening in conjunction with an IfcRelVoidsElement. However, defining a defect as an opening is semantically wrong. Fig. 15 shows the visualization of an IfcVoidingFeature with an IfcRelVoidsElement relationship in the original xBIM Xplorer. Fig. 15(a) shows a beam with typical spalling. Fig. 15(b) depicts a close-up screenshot of the cutout of the defect in the beam. Finally, in Fig. 15(c) one can see the highlighted defect geometry of the spalling. A similar result was achieved with the apstex IFC viewer.

Fig. 16 shows the selection and output of different visualization contexts in the extended xBIM Xplorer. Figs. 16(a–c) show the selected representation context, Figs. 16(d–f) present an overview of the model, and Figs. 16(g–i) depict a close-up view of the damaged section. Figs. 16(d and g) show the visualization of the undamaged beam, Figs. 16(e and h) show the defect geometry, and Figs. 16(f and i) show the damaged beam after subtracting the defect geometry. If the defect geometry and the geometry of the damaged component are activated, the used defect element, which is a proxy in this case, is shown as filling in the damaged beam. This is disadvantageous because the defect geometry should not be a filling. If the relationship-based cutout is used, i.e., IfcVoidingFeature with IfcRelVoidsElement, only the damaged component geometry is visible; however, the cutout is never shown as filling.

The test with the CSG representation revealed there is another feature missing in xBIM Xplorer. xBIM Xplorer is not able to use IfcBooleanResults in combination with IfcTriangulatedFaceSets. Because of this, the example in Fig. 16 uses a simplified defect geometry.

Altogether, with the use of IFC and an extension of xBIM Xplorer, it was possible to address all requirements stated in the “Requirement Analysis” section. Table 8 shows an overview of the requirements and finally used entities of the standardized IFC 4. All implementations could be verified using an extended version of xBIM Xplorer.

On the basis of the damaged bridge model, an inspection review may be performed. Fig. 18 shows a scenario of discussing defects using the as-damaged bridge model. All defects, their properties, and related documents may be reviewed by a team of engineers. Instead of using drawings and textual descriptions only, a 3D model can be examined by moving around, selecting images, showing related data, and discussing defect geometries and their impact on the condition assessment.

After the overall review of the bridge, some components may need further investigation. For this step, a geometry-based structural analysis, e.g., FEA, is applicable to determine the impact of the defect on internal forces and stresses. Fig. 23 illustrates an FEA in ANSYS with an individual beam. As an example, the equivalent von Mises stresses were calculated. Fig. 23(a) presents the 3D model views of the beam and the spalling. Fig. 23(b) shows the colored beam in ANSYS and a close look at the beam. The color legend is shown in Fig. 23(b). For the FEA, the IFC file that contains the beam is converted into a step file using IfcConvert (IfcOpenShell 2015). The engineer can add load conditions, bearings, and simulation parameters. With this workflow, the geometry of the beam can be imported directly instead of redrawing it. Subsequently, the FEA can be performed. This FEA model is used as an example, not to perform an in-depth analysis but to show the capability of the information model.

A 3D visualization of the damaged bridge allows model-based assessment reviews. For this purpose, a 3D BIM model of the bridge with all related components forms the basis. Necessary data are photos, measurements, geometry, textures, documents, and typification. Furthermore, the data model must respect that a defect may have multiple measurements, photos, or geometries because of several consecutive inspections. When using a photo as texture, a rectified photo and a texture map are necessary. Structural simulations benefit from an automatic exchange of geometry data. This needs the 3D BIM model, semantic data, and defect geometries.

The defect should be represented by an object because it groups all related data, for example, photos, measurements, and documents. Classifications should be defined once and used several times. Hence, a typification object, which can be used for all defects of the same type, is necessary. The defect has a geometry that influences the geometry of the related component. To calculate the geometry of damaged components, the geometry of the component and the defect geometry are required. This leads to the fact that a component has two geometries: an intact and a damaged geometry. Hence, a selection of the visualized geometry is necessary, which is respected by using a representation context. Finally, a defect photo may be used as texture. This texture must be mapped to the geometry either by an algorithm, for example, spherical mapping, or by pointwise mapping.

The work of Hamdan and Scherer (2018) and Sacks et al. (2018a) included alphanumeric damage properties in their model but omitted geometry data, visual data, and relationships to additional documents. Geometric data were incorporated by Isailovic et al. (2020) and McGuire et al. (2016). However, these studies did not incorporate multiple defect representations in parallel, for example, a defect representation as texture and geometry. Although the approach of Hüthwohl et al. (2018) showed how to visualize a defect as a texture, it does not explain how the required texture-mapping information is provided to the visualization software. The proposed concept synergizes semantic, geometric, and image data as well as adding the possibility of multiple representations and multiple images.

Out of five possibilities, IFC was chosen for implementating the object-oriented model because it is standardized and supported by numerous software tools. By using IfcBooleanResult, IfcVoidingFeature, IfcImageTexture, IfcGeometricRepresentationContext, IfcTextureCoordinate, IfcDocumentReference, IfcPropertxSet, and IfcProperty, all designed concepts could be implemented without extensions of the IFC. This has the advantage that all models based on this concept can be used in current and future inspection software.

Several software programs lack proper implementation or do not support all concepts of the IFC. Most software can handle proxies, aggregations, and assignments. If a software has a hierarchical view that is not fulfilled by all tested software tools, the semantic data are displayed properly. Some problems occur using classifications because several software tools struggle to properly include classification information. Even fewer software programs supported the concept of cutouts, especially if it is implemented using IfcRelVoidsElement. None of the tested tools offered the user the option for selecting a representation context or visualizing a texture. To test textures and geometry selection, xBIM Xplorer was extended. The extended version of xBIM Xplorer was capable of all the required concepts.

In summary, the paper has the following contributions: