A Framework for Characterizing Uncertainty Factors in Postdisaster Structural Performance Assessment Data

Postdisaster structural performance assessments provide quantifiable evidence of infrastructure system performance under hazard conditions, help to prioritize emergency response and recovery activities, and identify broader design issues. Such assessments can form a basis for empirical fragility functions, refine existing process-based models based on validation exercises, and establish baseline conditions for recovery tracking and modeling. Therefore, understanding the uncertainty associated with an assessment and its effect on damage, loss, or recovery models is critical for the verification and validation processes of these models. Uncertainties in a model’s characterization of a hazard and the resulting structural response should be quantified when possible. Many models propagate these uncertainties to evaluate their effects on model outputs or perform sensitivity tests to changes in model parameters. However, modal, temporal, and human factors that may cause variability in the assessment of a given system’s condition or state at a specific point in time are rarely characterized to assess how sources of uncertainty can compound in postdisaster performance assessments. This study identified and analyzed sources of uncertainty in postdisaster performance assessments including how and when assessments are performed and who is involved with postdisaster assessments. The discussion of how covers the mode of assessment, including field-based assessments, remote observation, and/or combined observation and measurement. When considers the temporal nature of an assessment, including the timing of performance assessments following a disaster and the effects of cascading or successive disasters. Last, who captures human factors, including surveyor bias, training, and familiarity with the affected area. A framework for quantifying uncertainty factors in field-based performance assessments is proposed. The framework follows the methodology presented in FEMA P-695 (FEMA 2009), which characterizes total system collapse uncertainty for seismic hazards based on four uncertainty ($\beta$) factors. Repositories of publicly available damage data hosted on platforms such as DesignSafe-CI [a cyberinfrastructure component of the Natural Hazards Engineering Research Infrastructure (NHERI) collaboration funded by the National Science Foundation] (Rathje et al. 2017) may be leveraged in future work to develop quantitative values for proposed uncertainty parameters in field study measurements and modeling applications.

While many previous studies have investigated infrastructure and social systems subjected to natural hazard events through postevent reconnaissance efforts, few report uncertainty as a standard element in assessed damages or performance. In recent years, several studies have explicitly addressed uncertainty in various aspects of performance assessment; several recent examples are presented in Table 1. However, a robust, standardized framework for systematically and consistently quantifying these inherent uncertainties is necessary for comparison across hazard types and individual events. Such a framework is critical for verification and validation processes when collected data and assessments are used to inform vulnerability and resilience models.

Uncertainty in postdisaster performance assessments can stem from various factors, all of which are affected by the nature of the event as well as the typology and scale of the assessed system. A framework for characterizing these uncertainties is shown in Fig. 1. The framework comprises four steps to identify, characterize, quantify, and propagate uncertainty in postevent performance assessment and modeling. This study focused on the first two steps of the framework: (1) identifying factors affecting sources of uncertainty; and (2) characterizing sources of uncertainty in postevent performance assessments.

In Step 1, factors affecting sources of uncertainty include the nature of the event and the type and scale of the assessed system. The nature of the event (red box) refers to hazard intensity and damage mechanisms that may impact a system or component of a system. It also includes factors that may alter the capabilities of researchers to establish baseline conditions: for example, for characterizing multihazard events, cascading or successive hazard events, or longitudinal recovery. The type and scale of the assessed system (cyan box) affect the sample size, sample methodology, and system archetypes chosen for performance assessment.

In Step 2, nature, type, and scale affect epistemic uncertainties in performance assessments due to modal (blue box, the method by which damage data are collected), temporal (orange box, assessment timelines in comparison to damage or recovery timelines), and human (yellow box, potential bias in damage observations) factors, all of which comprise aleatory and epistemic uncertainties. While aleatory uncertainty cannot be reduced, aleatory and epistemic uncertainties arising from modal, temporal, and human factors can be characterized. By evaluating the uncertainties in these three factors, research teams can (1) take steps to mitigate epistemic uncertainty; and (2) quantify uncertainty factors that can be used in damage or recovery models to better communicate risks to decision makers and stakeholders. The following sections characterize factors affecting aleatory and epistemic uncertainties and offer mechanisms to reduce epistemic uncertainties.

By recognizing modal, temporal, and human factors leading to uncertainty in performance assessment, steps can be taken to reduce variation associated with these factors. Standardizing damage measurements and conducting longitudinal studies will likely lead to reductions in assessment uncertainties collectively and enhance the ability to quantify sources of uncertainty.

Using and assigning common archetypes provides a mechanism for comparing damages across buildings or infrastructure systems by grouping units with similar attributes. Eisenack et al. (2019) highlighted four major insights on the use of archetypes applicable in the study of hazard and resilience impacts on infrastructure and social systems that can be mapped to community assessment. These insights included the following: (1) identify domains of validity for specific archetypes, (2) ensure that archetypes can be aggregated to assess a community impact, (3) clearly establish levels of abstraction (i.e., differences between essential characteristics of idealized archetypes and characteristics of actual systems), and (4) determine relationships between attribute configurations, theoretical models, and ranges of validity over hazard conditions. The use of archetypes can be either inductive or deductive in nature. For example, the US Geological Survey developed detailed structure type categories based on construction type (e.g., wood frame, steel, reinforced concrete, reinforced masonry) and details such as low- or high-rise construction in order to develop a country-specific inventory for assessing earthquake vulnerability (Jaiswal and Wald 2008). Additionally, Nofal and van de Lindt (2020b) developed 15 building archetypes for investigating building vulnerability to flood hazards. Appropriate use of archetypes should avoid overgeneralization by identifying reappearing but nonuniversal patterns that hold for well-defined subsets of cases. There is a need for quality criteria and practices in the selection and use of archetypes—which may range from defining built infrastructure to organizational structures and hazard types—to guide the design of theoretically rigorous and practical archetype analyses. Challenges in the use of archetypes include, but are not limited to, demonstration of the validity of the analysis, delineation of archetype boundaries, and selection of appropriate attributes to define them. For example, varying hazard intensity–system response relationships may exist among different archetypes. Therefore, grouping archetypes or inaccurately delineating archetype boundaries can increase the uncertainty in empirical fragility modeling. When possible, archetypes should be determined or planned for in designing a postevent assessment sample.

A standardized performance assessment module with associated field protocols for disaggregating archetypes and determining the performance of affected systems can be a breakthrough for identifying uncertainties over multiple field-based performance assessments. Proper equipment (e.g., safety gear, horizontal levels, batteries for cameras or specialized equipment), preparation (e.g., safety briefings, preparations for working in conditions with no power, water, or restrooms), and documentation can contribute to the success of data collection, interpretation, and dissemination (Peraza et al. 2014; Roueche et al. 2019, 2020, 2021). Designing and adopting valid and consistent measurements will provide damage data that are directly comparable to published and publicly available data based on the same measurements. For example, the USDA Food Security Survey Module has been incorporated into national level surveys (Pérez-Escamilla et al. 2004) as well as independent studies, because researchers can use a core module for required measurements and optional modules that can accommodate varying needs in field studies (Bickel et al. 2000). Using the USDA compatible survey modules has enabled researchers to test the validity and reliability of their own data by comparing them with national statistics and shared data from other researchers. In addition, subsidiary damage data offer a comparable layer for performance assessment outcomes from field studies. For example, pre- and postevent property tax records can provide an indirect measurement of damage by decreased appraised values (Zhang and Peacock 2009; Hamideh et al. 2018). Google street view and satellite images can also be employed to check preevent physical conditions related to nondisaster damage by deferred maintenance and housing abandonment (van de Lindt et al. 2018). Last, follow-up assessments and longitudinal studies can enable comparison of assessments made at different points in time to evaluate the optimal timing for measuring initial damage and can provide a means of quantifying modal, temporal, and human-induced uncertainty.

This paper introduced a framework for identifying and characterizing uncertainties in postdisaster performance assessments. The nature of an event, the type of system being assessed, the scale of assessment, and mode, timing, and human factors all contribute to uncertainty in performance assessment. The study did not attempt to quantify sources of uncertainty but rather articulated the scope of potential sources of uncertainty and the interdependencies among them. The framework is an attempt to isolate each source. However, many sources of uncertainty inherently overlap and may be correlated. For example, cognitive biases are affected by the mode and timing of performance assessment. More data are required for robust quantification of the various sources of uncertainty across hazards, events, and geographies and their correlations. It is critical to measure how each source independently influences performance assessment and how sources combine to jointly influence performance assessment. A method for addressing correlations is also needed to identify factors that may increase other sources of epistemic uncertainty.

Uncertainty quantification for performance assessments requires a significant and coordinated effort from researchers from multiple hazards and fields. A foundation for quantifying these uncertainties has been utilized by FEMA P-695 (FEMA 2009), which quantified a series of $\beta$ factors capturing uncertainty for seismic hazard analysis. Four independent sources of uncertainty were identified (seismic records, design requirements, test data, and modeling) to estimate total structural collapse uncertainty. Record-to-record uncertainty was found using a lognormal distribution, while uncertainty in design requirements, test data, and modeling were determined using expert opinions based on the completeness and robustness of the uncertainty factor and confidence in the basis of these factors (FEMA 2009). A similar approach could be employed to inform hazard, damage, and recovery models such that fragility functions and subsequent use of such functions provide more accurate depictions of system performance. For example, comparisons for the same structure of performance assessments based on field surveys, remote sensing, and/or resident surveys may identify trends in modal uncertainties, allowing for similar uncertainty factors to be determined based on the completeness, robustness, or confidence of a given performance assessment data set. Longitudinal studies of the same sample (e.g., Sutley et al. 2021; Helgeson et al. 2021) can provide insight into parameters characterizing temporal effects on damage assessment, while assessments by multiple researchers of the same sample (or sample subset) can be analyzed to quantify inherent variability due to human uncertainties. Identification and quantification of these sources of uncertainty may also lead to future research on strategies to reduce uncertainty.

Deliberate uncertainty quantification can be time consuming and challenging during already time- and resource-intensive reconnaissance efforts following disasters. Therefore, it may not always be feasible to document uncertainty parameters during field reconnaissance, given that field teams prioritize collection of perishable performance data for a sufficient sample. A subset of a sample may be analyzed in the field or evaluated based on imagery and other data collected during reconnaissance efforts using photography or emerging street-view image capture technology (e.g., Roueche et al. 2021). However, it is important for teams to recognize trade-offs and to achieve a balance between uncertainty quantification and robust data collection. Efforts to characterize modal uncertainties from photographic and field-based assessments may allow teams to perform uncertainty characterization after returning from field efforts focused primarily on data collection.

To fully characterize the modal, temporal, and human uncertainties affecting postdisaster performance assessments, a comprehensive data-synthesis effort spanning hazards, geographies, and systems is required.

Some or all data, models, or code generated or used during the study are available from the corresponding author by request, including framework, performance assessment data from the longitudinal Lumberton study, and performance assessment data collected following Hurricane Ike.

This research was conducted as part of the NIST Center of Excellence for Risk-Based Community Resilience Planning under Cooperative Agreement 70NANB20H008 between the NIST and Colorado State University. The views expressed in this article are those of the authors and do not necessarily represent the opinions or views of NIST or the US Department of Commerce. The authors would like to thank Professor John van de Lindt, Professor Andre Barbosa, and Dr. Omar Nofal, who contributed valuable discussions informing the content of this article.