Validation of Urban Flood Inundation Models Applied Using Nationally Available Data Sets: Novel Analyses of Observed High Water Information

We used the NWS Analysis of Record for Calibration (AORC; Kitzmiller et al. 2014) as the primary meteorological forcing for ADHydro and WRF-Hydro. The AORC is a multidecade, internally-consistent data set of precipitation, temperature, solar radiation (shortwave and longwave downward at the surface), dew point, wind vectors, and terrain-level pressure. AORC data cover the contiguous US (CONUS) at a 1-km grid resolution and have an hourly time interval. Noting the flashy hydrograph response of the Sugar Creek stream gages, we utilized the 2 and 5 min precipitation estimates from the NWS Multiradar Multisensor System (MRMS; Zhang et al. 2016) to time disaggregate the hourly AORC precipitation estimates into 15 min intervals.

We selected two extreme storms to evaluate the models’ ability to simulate flood inundation. The flood of record in August 2008 resulted from over 28 cm of rain falling over the entire basin in a 36–48 h period. We also selected a large convective event in August 2011 in which 7–15 cm fell in 3–4 h. Hereafter, we refer to these events by year, e.g., the 2008 and 2011 events.

Anticipating that there would be future expanded applications of HRMs, we selected only static geographic data sets that had national coverage. The 10 m National Elevation Data set (NED; USGS 2017) was selected to represent topography and define flow directions. Street vectors were derived using Open Street Map and used to define major urban flow paths for both models. We used the 2011 USGS National Land Cover Dataset (NLCD; Wickham et al. 2017) for land use and land cover. Soil texture information was taken from the Soil Survey Geographic (SSURGO) data set (Soil Survey Staff 2021). We chose to forego strict definitions of building footprints to define surface flow paths. Finally, we used the National Hydrography Dataset Plus (NHD+) version 2 data to define the location of the channel network (Moore et al. 2019). Storm sewer networks were not modeled. Cross-section data were available but not used due to our self-imposed limitation to use only data sets with national coverage. Instead, both ADHydro and WRF-Hydro used empirical stream-order relationships to define channel shape parameters.

The CMSWS provided three types of observed HWMs. The first type was surveyed HWMs collected by a local engineering firm on behalf of CMSWS. The vertical datum used was NAVD 88, and the horizontal datum was NAD83/2007. Map projection data utilized the North Carolina 3200 projection. Categorically, CMSWS rated the surveyed HWMs as good, fair, or poor. However, no additional information was available regarding how this rating was determined. The types of surveyed HWMs were mud lines, wrack lines, debris lines, seed lines, stain lines, and witness marks. Second, flood damage depths were geo-encoded parcel locations of damage in which interior surveys were conducted to determine the depth of water within the structure. The structures impacted include private, business, and utility locations. The residential structures impacted include single-family homes, apartments, condos, and townhomes. Business structures include retail, offices, and warehouses. Recorded water depths were relative depths measured within the structure. Considering 2008 and 2011 together, average damage depths ranged from approximately 43–78 cm depending on the type of measurement (i.e., living area, crawl space, etc). Third, flooded streets were geo-encoded locations where flooding was observed. Of the three types of inundation data, flooded streets are the least quantitative and should be considered subjective. No indication of water depth was provided. Sources of the observations were witnesses, news reports, emergency management (such as police, fire, and local government), and photographs of flooding. Appendix I presents examples of the three types of HWMs from the Charlotte basin. Table 1 presents the number of observed HWMs obtained for this study. Examination of the 2011 flood damage (100 values) and flooded streets data (1,951 values) revealed that the latitude/longitude location information between the two types was redundant. As such, the 100 flood damage depths in 2011 were treated as a subset of the 1,951 flooded street observations. Hereafter, we use the term flood damage/flooded streets to refer to this group of observations.

In an idealized setting, predicted maximum water depths in the computational element containing the HWM would be extracted and directly compared to the surveyed depth at each high-water mark location for a direct or reference comparison. Uncertainties in the surveyed HWM value and quality and differences in the underlying mesh structures and size (e.g., Fig. 1) of computational elements of models such as WRF-Hydro and ADHydro, make direct comparisons complicated. Therefore, in addition to the reference comparison, we used the areal sector approach of Patrick et al. (2018) to minimize the impacts of computational mesh differences and potential differences in the representation of terrain in the models. This approach provided five additional metrics to objectively assess simulation performance when the model predicts water in the general vicinity of the HWM.

We use Fig. 3 to illustrate the derivation of areal sectors. The background image in Fig. 3 shows the WRF-Hydro maximum water depth grid cells. For ease of visualization, the background grid cells have been aggregated $4:1$ in size. (Recall that all WRF-Hydro calculations were performed at the $\sim 10\mathrm{m}$ grid scale). The blue line is the NHD+ stream channel vector. The triangular ADHydro mesh element is outlined in red. The yellow square is the WRF-Hydro element, and the black dot is the surveyed HWM location. The yellow triangle outlined in red is the point on the NHD+ stream network vector that is closest to the surveyed HWM. The steps of Patrick et al. (2018) are listed below:

Using the areal sectors, we computed a variety of predicted water depths for each model as shown in the list below.

One concern about using areal spatial analysis is the impact of uncertainty in ground elevations. Patrick et al. (2018; Appendix II) performed several analyses to explore this concern. Elevation analysis confirmed that there was minimal NED DEM elevation variation within the areal sectors. In addition, distributions of different NED DEM elevations within the areal sectors were explored and compared to the surveyed HWM ground elevation. The variance between surveyed HWM ground elevations and NED 10 m HWM elevations was consistent with the variance between surveyed HWM ground elevations and the areal sector mean elevations. Typically, these relationships were valid except when areal sector areas were large. Overall, median distances from the surveyed HWMs to the NHD+ stream segment intersections were less than 45 m, and the median areal sector areas are less than $\mathrm{4,400}{\mathrm{m}}^2$. This information suggests that using an areal analysis does not significantly enhance uncertainty or increase errors for most HWMs. Thus, differences in the models’ computed maximum depths at surveyed HWM locations should largely be a function of model physics and discretization.

We also developed a novel method to analyze predictions at flood damage/flooded street locations. Recall that these flood reports consist of measurements of flood damage depths in structures and locations where photos, news, and witness reports indicated flooding. Referring to Fig. 4, the analysis steps are as follows:

We computed maximum depths in each computational element to compare to the total of 172 surveyed HWMs and over 2,000 flood-damage/flooded-street observations. For WRF-Hydro, the maximum depth data files consisted of overland and channel flow depths. The ADHydro team submitted maximum depths in all nonchannel computational mesh elements. These depths should be taken into account when interpreting the results. For brevity, we focus on the calibrated results. The uncalibrated results are presented by Smith et al. (2020).

Table 2 presents the analysis of predicted inundation depths at surveyed HWMs. Column 2 lists the six types of predicted maximum depths. The reference value refers to the maximum depth for the model element which contains the surveyed HWM. The other types (maximum, mean, median, IDW, and areal mean) are depths computed using the predicted maximum depths in all model elements residing within the areal sectors which contain each surveyed HWM. Note that the maximum category most likely included flow depths in channels for WRF-Hydro. There are 131 surveyed HWMs for the 2008 event and 41 for the 2011 event. Of these, 73 and 15 were above-ground HWMs for 2008 and 2011, respectively.

Table 2 presents root mean square errors and mean absolute errors (MAE) between the predicted and observed HWM depths. We computed these errors for the 2008 and 2011 storm events for the reference depth and the five areal sector computed depths. An observed depth of zero was used in the case of surveyed on-ground HWMs. Columns 3–6 present the errors at all surveyed HWMs, while columns 7–10 highlight the errors for only above-ground surveyed HWMs. Values in bold font indicate the smallest errors in a column.

The reference results (Row 1) in Table 2 indicate that WRF-Hydro generated lower RMSE and MAE values than ADHydro (Row 7) at all locations of surveyed HWMs. All the various areal sector RMSE and MAE values support this result by assessing the predicted depths at and in the vicinity of the surveyed HWM. In six of eight columns, the best overall results were generated by the WRF-Hydro mean depth, as seen by the values in bold in Row 3.

Considering the within-model results, it would be reasonable to expect that a model’s best results would be achieved by the reference simulation at the HWM. For ADHydro, the reference depth (Row 7) did indeed generate the lowest RMSE and MAE errors in three cases, as seen in columns four through six. The IDW sector depth (Row 11) resulted in the lowest RMSE and MAE values in three other cases (columns 3, 7, and 8). For WRF-Hydro, none of the reference depths achieved the lowest RMSE and MAE values. Rather, the lowest WRF-Hydro values in six out of eight columns resulted from the mean sector depth (Row 3). The areal weighted mean depth achieved the second-lowest results (columns 3, 5, and 7–10). Our results in Table 2 suggest that the predictive strength of WRF-Hydro was achieved by mean or areal weighted mean depths rather than reference depths at the specific HWM locations. In contrast, ADHydro’s best results were achieved at or near the HWM locations using reference and IDW depths, respectively.

The overall lower values of RMSE and MAE from WRF-Hydro compared to ADHydro in Table 2 are likely the result of two factors. First, because it is limited to a one-way coupling between overland and channel flow, WRF-Hydro frequently underestimated inundation depths. This resulted in lower RMSE and MAE values when observed depths were also shallow. Second, channel capacities in ADHydro were likely underestimated, causing that model to more frequently overpredict inundations depths by an amount greater than what WRF-Hydro underestimated them. This is supported by the general improvement in ADHydro’s RMSE and MAE values when surveyed on-ground watermarks are removed (21 out of 24 cases in columns 7–10). Contrary to ADHydro, when the surveyed on-ground watermarks were removed, WRF-Hydro’s RMSE and MAE values typically worsened (20 out of 24 cases in columns 7–10). Quantitatively, WRF-Hydro may be producing slightly lower RMSE and MAE values overall, but qualitatively, we believe ADHydro more realistically predicts inundation near the surveyed HWM and provides additional actionable information.

We present further analyses of predicted depths at surveyed HWMs in Table 3. This table shows the percentage of hits each model recorded at the surveyed HWMs. A hit occurs when the model predicts a maximum depth that is greater than the observed depth multiplied by a threshold. Hits are computed using the reference depth and the areal weighted mean depth of the sector containing the surveyed HWM. We prescribed hit categories with depth thresholds of 10%, 30%, and 50% of the observed high water depth. For example, suppose the surveyed observed depth is 1.0 m. At the 10% threshold, a model must compute a depth greater than $1.0\mathrm{m}\times 10\%$ or 0.1 m to record a hit. Note that for on-ground HWMs, a hit is counted anytime the flow depth is greater than zero. We interpret Table 3 as follows. The WRF-Hydro value of 39 in column 5 means that at 39% of the 131 surveyed HWMs, WRF-Hydro predicted a maximum depth which was equal to or greater than 50% of the observed high water depth.

Considering first the reference case, the results in Table 3 indicate ADHydro computed the highest number of hits at all depth thresholds. Looking at the case for all HWMs (columns 3–8) for each depth threshold, ADHydro predicted exceedance depths at nearly twice the number of HWMs compared to WRF-Hydro. The differences are greater when only above-ground HMWs are considered (columns 9–14). Similarly, ADHydro areal weighted mean flood depths exceeded the thresholds at a greater number of HWM locations than WRF-Hydro. These results indicate ADHydro properly placed floodwaters at HWM locations more often than WRF-Hydro. ADHydro also computed more hits than WRF-Hydro for uncalibrated simulations of surveyed HWMs (Smith et al. 2020).

Table 4 presents the analysis of simulated depths at property parcels where damage surveys were conducted, and photos and news outlets reported flooding. We used five threshold depths ($<2.54$, $\ge 2.54$, $\ge 5.08$, $\ge 15.24$, and $\ge 30.48\mathrm{cm}$) to calculate hit flooded percentages for the parcel polygons. We computed the parcel-threshold hits or the percentage of parcels at which the models predicted a maximum depth greater than a threshold depth.

In the case of WRF-Hydro simulations of the 2008 event, 31% of the parcel polygons had a predicted mean areal maximum water depth less than 2.54 cm, and 21% of the parcels had a predicted water depth greater than 30.48 cm. For the 2008 event, ADHydro achieved higher percentages of hits for deeper thresholds compared to WRF-Hydro. For example, 76% of the 280 parcels had a predicted maximum depth greater than 30.48 cm compared to 21% for WRF-Hydro.

For the 2011 event, ADHydro also achieved more hits at the deeper thresholds than WRF-Hydro. Interestingly, both models generated more hits for the $<2.54\mathrm{cm}$ threshold in 2011 than 2008. For WRF-Hydro, 84% of the parcels in 2011 had a computed depth of less than 2.54 cm compared to 31% in 2008. ADHydro generated max depths less than 2.54 cm in 43% of the cases in 2011, compared to only 1% in 2008.

The results in Table 4 indicate WRF-Hydro tended to generate many instances of minimal flood depths ($<2.54\mathrm{cm}$) in 2008 and 2011 at locations where flooding was observed. Such large numbers of minimal depths are suspect, given the nature of the observations and storm severity. On the other hand, ADHydro computes a greater number of hits at all thresholds deeper than 2.54 cm. Hits at deeper thresholds seem to be realistic given the nature of the flooded streets/flood damage observations.

Fig. 5 shows the spatial distribution of a subset of the 1,144 flooded property parcels for the 2011 storm described in Table 4. The preponderance of black parcels in Fig. 5 shows that WRF-Hydro had many more cases of flood depths less than 2.54 cm compared to ADHydro. The large number of parcel inundation depths less than 2.54 cm for WRF-Hydro seems unrealistic because 7–15 cm of rain fell within a period of 3–4 h.