Deep Learning–Based Building Attribute Estimation from Google Street View Images for Flood Risk Assessment Using Feature Fusion and Task Relation Encoding

Floods are a very common natural disaster, occurring worldwide and causing economic losses and human casualties. Expected climate changes over the next century, including sea level rise (Michael 2007; Neumann et al. 2015), more frequent extreme precipitation events (Meehl et al. 2005; Lehmann et al. 2015), and more intense cyclone activity (Emanuel 2005; Landsea et al. 2006), pose existential threats to coastal cities hosting the large majority of human life and activity (Hallegatte et al. 2013). Governments are working to adapt to the changing environment by investing in large-scale risk mitigation. For example, the State of Louisiana in the United States has produced its Comprehensive Master Plan for a Sustainable Coast, a fifty-year, legislatively-mandated plan consisting of approximately $50 billion in coastal protection and restoration projects (Louisiana Coastal Protection and Restoration Authority 2012a, b). In coastal areas, governments, individual homeowners, landlords, and businesses all need accurate information about current and future flood risk to make effective decisions about risk mitigation. Decision makers such as state and federal governments may have the resources to invest in large-scale protection projects (e.g., levees, floodwalls, and pumps) that alter the local probability distribution of flood depths, but others generally do not. However, homeowners can still reduce their risk through measures such as elevation-in-place that raise the home’s foundation. In this paper, we focus on identifying building attributes that provide individuals with the decision support they need to make informed, cost-effective decisions about risk mitigation of their own properties (Kellens et al. 2013).

Damage calculations in the Coastal Louisiana Risk Assessment (CLARA) model primarily follow methods developed for the FEMA Hazus Multi-Hazard model (Hazus-MH) (Scawthorn et al. 2006; Johnson et al. 2013; Fischbach et al. 2017). Direct economic losses associated with flooding are calculated as a function of a multitude of variables, such as the building’s replacement cost, depth of flooding relative to the building’s first-floor elevation above grade, number of stories, foundation type, and building type. The last three characteristics can be collectively referred to as building characteristics.

Assume a structure of building characteristics $i$ with size $s$ and construction quality $q$ is being retrofitted to elevate its foundation to a height of $h$ feet above the current level. Moreover, assume that the elevation of flooding relative to the top of the building’s foundation is $e$. Then, the depth damage function for the building can be denoted as $Di(e)$, the replacement cost can be denoted as $V(s,q)$, and the probability distribution function of flood elevations occurring in a given year can be denoted as $f(e)$, where each of these depends on the variables inside their respective parenthesis. In the CLARA model, the depth damage function is expressed as the proportion of the structure’s replacement cost incurred as damage to repair or reconstruct the building after a flood event. Elevating the structure directly reduces the effective flood depth experienced in comparison. Therefore, the annual losses are

The damage curve $D_i(e)$ can vary substantially depending on the building characteristics, with building type being the most important driver of variability. For example, 0.3-m of inundation above the first-floor elevation is estimated to cause structural damage equal to 48% of the replacement cost for an average commercial building but 68% for a manufactured home. Damage to a single-family home on a slab foundation is 56% if one story and 44% if two stories; the respective values are 62% or 54% if the home is on a pier foundation (Scawthorn et al. 2006).

The set of structural attributes relevant to this calculation guided the selection of features to estimate in this study. Such data are expensive to collect manually and, as a result, seldom up to date. For instance, in large parts of the Louisiana coast, where significant efforts have been made toward studying flood risk since Hurricane Katrina struck in 2005, the most recent data on the height of building foundations above grade were from street-level surveys performed by the USACE in 1991 (USACE 2009). Frequent reconstructions and retrofits in the three decades since then have made it obsolete, but the state has no better estimates to rely on for making investment decisions about coastal protection measures in some areas. A small number of building-level data collection efforts utilizing post-Katrina reconstruction and tax records have somewhat improved the estimates of structural features in several Louisiana parishes; however, the coverage of high quality data is far from complete. In other states and, in particular, developing countries, individual structure-level data either do not exist or are scattered across multiple agencies and jurisdictions, making them prohibitively expensive and time-consuming to collect.

In recent years, deep learning (LeCun et al. 2015) techniques have helped engineering researchers improve solutions for a variety of problems, and the improvements have usually been significant (Cheng et al. 2021; Bao et al. 2019). To better tackle the grand challenge of flood risk management, this study proposes a deep-learning-based framework that can collect structure-level building attribute data in a more effective and efficient manner than the current approach of performing manual street-level surveys. First, the Google Street View (GSV) images of buildings in the areas of interest are gathered. Then, the proposed framework uses them to simultaneously estimate multiple structural attributes that are crucial for assessing flood risk. Consequently, combining the estimated structural attributes with geospatial data and flood risk models (e.g., CLARA model) will directly improve flood risk assessments for the areas of interest. Fig. 1 shows one view of an online flood risk decision support system developed by Louisiana’s Coastal Protection and Restoration Authority (CPRA) that visualizes flood risk information for public individuals and businesses in the coastal zone in which the proposed framework will analyze GSV images and estimate risk-relevant structural attributes.

The estimated attributes include each building’s foundation height, foundation type (pier, slab-on-grade, mobile home, or other), building type (commercial, residential, or mobile home), and number of stories (one story or more). Fig. 2 shows sample GSV images of typical buildings displaying these attributes.

By integrating the proposed framework into Louisiana’s master planning and community resilience programs, individual homeowners can directly benefit from the “hyper-local” flood risk assessments by informing risk mitigation decisions. This will be especially true in vulnerable communities that do not currently have structural protection provided by levee and floodwall systems. Although this study focuses on predicting building attributes for managing flood risk, the proposed framework can be extended to predict different attributes for other hazards, including hurricane, tornado, or seismic hazards.

This study strives to first model the building attribute estimation problem as a multiobjective one, that is, by using one model for predicting multiple attributes at once. Then, to improve the performance of this multiobjective model, this study proposes the addition of metadata and the encoding of known relationships between the different building attributes. Not all of the building attributes may have known relationships or relevant metadata readily available; however, those necessary for assessing flood risks usually do. For instance, piers are one of the most prevalent types of foundations in coastal regions, and we know that they are usually taller than slab foundations. Because both foundation height and foundation type are extremely important for the assessment and mitigation of flood risks, the knowledge that pier foundations are usually taller can be included in an automated building attribute extraction scheme to allow the modeled probabilities of the foundation type and height estimates to inform each other and improve overall performance.

Similarly, some information relevant to the estimation of attributes for flood risk management is not present in an image. For instance, a building’s actual scale is vital when predicting foundation height. However, using an object detector followed by scaling the image to a fixed size masks some of this information from the models. Meta information, such as the size of bounding boxes, distance between the camera and the building, building aspect ratio, and others, reintroduces some of the lost information and helps predict the attributes that are dependent on them. Adding meta information when predicting building attributes also has other advantages. Sometimes, they directly capture specific relationships between the tasks. For instance, commercial buildings in some neighborhoods tend to have shorter building-camera distances than residential buildings because the latter are usually farther from the street, and buildings with lower aspect ratios have a higher probability of being multistoried than those with higher aspect ratios. Being judicious in the selection of meta information that either characterizes or differentiates the relevant classes can significantly improve the estimation of building attributes relevant to flood risk estimation. Our study shows that this combination of multitask learning (MTL), relationship encoding, and feature fusion can improve the performance of building attribute prediction models.

We propose a MTL approach for predicting multiple building attributes at once for flood risk assessment. To the best of our knowledge, this study is the first to do so and the first to estimate foundation type and foundation height as a regression problem from GSV images. MTL, in addition to being cheap, also improves performance by allowing the transference of knowledge across tasks (Thrun 1996; Caruana 1997). Our study shows that this relatively overlooked area of research should be perceived as a viable candidate for building attribute estimation because of its potential to provide accurate predictions at a proverbial penny on the dollar when compared with single-task learning approaches. Similarly, to further increase the performance of MTL models, we propose the two methods of Task Relation Encoding Network (TREncNet) and feature fusion. Traditionally, each task in MTL has independent, fully connected layers. Recent studies (Meyerson and Miikkulainen 2017; Ma et al. 2018) have shown that the implicit relations among tasks can be learned by soft layer ordering or multiple gating with task-specific parameters. However, the case of tasks having explicit relations (i.e., relationships we have prior knowledge about) has been under-discussed. We encode such known relationships through an architectural modification that we call TREncNet. Moreover, we use another technique called feature fusion, which introduces metadata, such as camera-to-building distances and aspect ratios, into the MTL network. By combining MTL, TREncNet, and feature fusion, we report the best overall prediction scores among three separate single-task learning and four separate plain MTL experiments conducted using state-of-the-art architectures. Given its capabilities, we conclude that the proposed approach can effectively and efficiently process comprehensive data without using street surveys, which will save time and money for flood risk management. Finally, we provide an extensive evaluation of different CNN architectures for building attribute prediction and object detection, which provides practitioners a starting point when implementing this framework.

The remainder of this paper is organized as follows. The section on “Building Dataset Generation” describes the dataset of buildings collected for training and evaluation. The section on “Proposed Framework” elaborates on the details of the proposed framework. The section on “Experimental Results” discusses the evaluation results. The section on “Error Analysis” discusses why some errors might have occurred. The section on “Key Findings and Potential Implications” provides a detailed breakdown of the results from our proposed approaches and discusses some implications of this study. The section on “Limitations and Future Work” lists some possible future studies that can build on this work, and the “Conclusion” summarizes the paper.

To train the CNNs in the proposed framework and validate the estimation performance, ground-truth building attributes along with the corresponding GSV images need to be collected. Several field surveys have been conducted in the post-Katrina coastal Louisiana that collected the attributes and the GPS coordinates of 80,109 buildings. For 73,781 buildings, the records had all of the desired attribute information, and the other 6,328 records had only foundation height information. This building-level data primarily originated from three studies performed by the USACE: the Morganza to the Gulf Reformulation study (noa, a), Southwest Coastal Louisiana Feasibility study (noa, b), and West Shore Lake Pontchartrain Feasibility study (noa, c). Coverage includes part or all of Calcasieu, Cameron, Iberia, Jefferson Davis, Lafourche, St. Charles, St. James, St. John, and Terrebonne parishes (i.e., county-level units of governance in Louisiana). Foundation heights from FEMA Elevation Certificates for 2,471 buildings in Jefferson Parish were also obtained from a parish floodplain manager. The buildings’ GSV images ($640\times 640$ resolution and 75° field of view) were autonomously extracted using GSV’s application programming interface (API) and the GPS coordinates. Then, the bounding boxes of buildings in the GSV images were manually annotated, as shown in Fig. 3.

Not all of the GSV images had usable views of buildings. Sometimes, major parts of the buildings were blocked by objects (e.g., fences, trees, or cars), or the buildings appeared too small in the images (e.g., with width or height less than 80 pixels). Given the inconsistencies between the collected building coordinates and the GSV API, some images did not contain a building in the scenes, and some coordinates did not have a GSV image available. Fig. 3 also shows sample GSV images with unusable building views. Images with unacceptable views were removed from the dataset; the remaining dataset contained 42,415 usable GSV images. Fig. 4 shows the distributions of the building attributes in the training dataset. The distributions are imbalanced, and the dominant attributes with the largest prevalence are 0.15-m foundation heights (0.5-ft), concrete slab foundations, residential buildings, and one-story buildings.

Fig. 5 shows the overview of the proposed framework, consisting of the following steps. (1) “CNN building detection” detects the bounding box of a building from a GSV image. The pixels inside the bounding box are then scaled to a fixed-sized image of that building. (2) “CNN feature extraction” extracts the image feature vector from the fixed-sized image. (3) “Feature fusion” concatenates the image feature vector with the meta information vector to form a fused feature vector. (4) “Task relation encoding network” simultaneously predicts all of the building attributes from the fused feature vector, including the building’s foundation height in feet above grade, foundation type (pier, slab, mobile home, or other), building type (commercial, residential, or mobile home), and number of stories (one story or more). The details of each step are explained in the following subsections, and the training of CNNs is described in the experimental results section.

Some of the correctly predicted samples are shown in Fig. 12, and incorrectly predicted samples are shown in Fig. 13. In this section, we perform error analysis using the incorrectly predicted samples. For foundation height estimation, the system predicts significantly higher heights than the ground truth for buildings that have slab foundations but also pillars on their facade. Similarly, some mistakes also occur as a result of wrong labels. For instance, the system predicts 0.180 m as one of the buildings’ foundation height, which visually appears closer to the true value but has a large numerical disparity with the ground truth (2.440 m). When looking at the wrong predictions in foundation type classification, one of the key points to note is the number of labeling mistakes across sections. In some cases, the model actually predicts the correct class, but the labels are wrong. At other times, the errors are random, such as the prediction of mobile foundations as pier, mobile foundations as others, and others as mobile, which is not surprising. Foundation type estimation is a four class task, and even our best models fail to properly characterize the minor classes.

The classifications of mobile homes as residential and residential homes as mobile seem to occur because of unclear boundaries between some of the buildings in these two classes. Some mobile homes’ visual features resemble those of residential homes and vice-versa, and the model fails to capture these distinctions. The misclassification of some of the commercial buildings as residential seems to occur for the same reason; however, the misclassification of residential buildings as commercial seems random, suggesting that the model fails to characterize some of the patterns in residential homes that distinguish them from commercial homes.

The errors in the one-story category occur when tall slab foundations underlie the structures. Some mistakes may also have occurred due to noise. For instance, in one of the images, the building adjacent to the building of concern seems to be multistoried. Similarly, the errors in the “two stories” category do not seem to possess an alternate explanation other than the fact that they happen because of the model’s inability to properly categorize some of the multistoried buildings.

To conclude, two main kinds of errors exist in our predictions. The first one originates because of the inability of the models to properly characterize a specific class. The other is due to incorrect labeling. To prevent the former type of error from happening, more advanced versions of TREncNet and feature fusion that introduce even more relevant metadata and encode complex relationships need to be conceptualized. Similarly, to prevent the latter type of error from happening, local agencies need to be more careful when taking and keeping records of building attributes.

The first key finding of this study is that building attribute estimation is best modeled as a multiobjective problem if more than one attribute exists. By reducing the number of models, the inference time can be decreased significantly, which allows authorities to more frequently carry out projects, such as flood risk assessment. Furthermore, the performance of the single task learning approach in estimating the most important building attribute regarding flood risk management is concerned (i.e., foundation height) is subpar. The lower foundation height MAEs in multitask models suggests that some degree of correlation exists between foundation height and the other attributes, and deep learning architectures possess enough ability to exploit them. However, the performance gains in the classification tasks were not as high, suggesting that they are relatively easy tasks and, thus, are affected by the regularization that can sometimes be induced by MTL. This disparity in the performance gain between easy and difficult tasks is consistent with the findings of MTL studies from other fields.

The second finding of this study is that additional sources of information through either feature fusion or the encoding of known relationships can improve performance. This is especially the case when the two are used together rather than separately. Similarly, the type of metadata and relationship encoding may impact the performance of individual categories. Evidence of this idea is observed in our results.

For instance, TREncNet did not provide any improvement in foundation height estimation, but feature fusion did. We hypothesize that, although foundation height is correlated with other tasks (evident by the performance gain when MTL instead of STL was used), the encoded relationships apparent to us might have been equally apparent to the architectures, which made our encodings redundant. However, for feature fusion, some information such as camera-to-building distances were extraneous to the images, whereas others such as aspect ratios of bounding boxes might have been too difficult for the networks to estimate without outside help. Furthermore, the largest-to-smallest reduction in foundation height MAE was 15% for MobileNet V1, 12% for Inception V2, 9% for Inception V4, and 8% for Inception ResNet V2, which lends some credence to this theory because simpler architectures might be less prone to map harder-to-extract information by themselves.

Moreover, the results of foundation and building types show that TREncNet has a stronger impact than feature fusion. When compared with those two attributes, its impact on the number of stories and foundation height categories is negligible. To determine why, we must observe two factors: the architecture of TREncNet and the confusion matrices. Four total architectures exist; however, MobileNet V1 not only echoes the pattern in most other architectures but also accentuates it and serves as a case study for analyzing where TREncNet may or may not work or where feature fusion may or may not work. Therefore, we have illustrated its confusion matrices for the three classification tasks along with the boxplots for the foundation height task in Fig. 14, where the number of samples is 4,341, 1,252, 503, 737, 995, and 245, respectively, for each of the foundation height intervals. First, we must refer to Fig. 7.

As is observed, task encoding for the number of stories task is minimal. We only share the logits of the mobile home from building type task with one-story of the building stories task. However, from Fig. 14, the performance of the one-story class was already saturated even without TREncNet. For both foundation type and building type tasks, TREncNet improves the performance of minority classes. To understand why, let us refer back to Fig. 7. The building type category takes logits from the number of stories category. We performed a Cramer’s V test using the chi-square test of independence between the two categories. Cramer’s V value ranges from 0% to 100% in percentage form, and a higher value suggests a stronger relationship between nominal variables. With the residential buildings included, the result of the Cramer’s V test between the number of stories and the building type was 8%. When it was excluded, Cramer’s V output jumped to 20%. Although this is still a weak correlation in statistical terms, in the context of deep learning, even this small value might have been enough to significantly propel the performance of minority classes in building type estimations. The same can be said about the minority classes of foundation type and number of stories. With slab foundation excluded, the Cramer’s V value jumped from 13% to 15%. Here, however, the slab foundation prediction has increased in the confusion matrix as well, which means that the initial high value and the lower increase with the slab foundation excluded are still consistent with our theory. We could not perform the same test for foundation and building types because mobile homes were a part of both tasks.

This study is the first that proposes the inclusion of encoding and metadata for building attribute estimation in flood risk assessment, and we firmly believe that more room exists for experimentation. This version of TREncNet and these metadata are only meant as proofs of concepts rather than ends in themselves. Ultimately, we hope that the main implication of this paper lies in the message that both performance and cost-efficiency can be achieved at the same time when predicting building attributes.

One of the limitations of this work is that the coverage of the GSV database may present a problem in certain cases. In much of the developing world (including most of Africa), GSV is almost nonexistent. Even in parts of major world economies, such as China and India, GSV images mostly cover only cities and landmarks. Be that as it may, our approach is broadly applicable in developed countries due to GSV’s spatial coverage, having photographed more than 10 million miles of roads (Nieva 2019). Moreover, GSV’s coverage in developing countries is increasing every day, and Google has made significant efforts toward providing the needed resources (for instance, cars and all of the necessary sensors) to motivated native people such that GSV photographs can be obtained through semicrowdsourced ventures. Such efforts have been met with marked enthusiasm, primarily on account of the young and increasingly tech-savvy populace of low- and middle-income countries. Updates are also relatively frequent, with images being refreshed approximately every one to three years in coastal Louisiana (more frequent updates are in urban areas such as New Orleans). GSV also allows fully crowdsourced contributions in some cases, enabling researchers doing postevent reconnaissance after Hurricane Laura in 2020 to publish new imagery within days to document damage to the western part of the state. Similarly, although GSV images (where available) are good sources of information on a structure’s facade, they are typically information-poor for its sides and posterior (typically, only a few buildings in the GSV database have images of multiple sides). Some of the distinguishing features that are relevant to building and foundation type classifications may be apparent from these alternate vantage points as well; thus, the availability of such views may enhance performance.

One future study could be a data fusion scheme that aggregates the detection and prediction results from multiple GSV images of the same building to increase the robustness of the estimation. Moreover, we used only the camera’s metadata in our feature fusion scheme; however, other types of information, such as the building’s zoning and ownership, could also prove advantageous. In addition to processing only image data, combining data from audio or depth sensors may also be beneficial (Huang and Kingsbury 2013; Hazirbas et al. 2016; Wu et al. 2013; Kamel et al. 2018). To correctly train and evaluate the detection and prediction models, we manually removed the GSV images with unacceptable views (e.g., buildings blocked by trees). Although the quality of the views can be determined autonomously as well—for instance, a low detection score or a small bounding box during the inference phase may be used to filter them out—this is still not efficient. Future directions building on this work could be to implement an object recognition model that can estimate the view quality of details.

Last but not least, the performance of our approach in some of the key tasks and their classes needs further improvement. Regarding foundation height, we were able to achieve an MAE as low as 0.171 m. Although a significant improvement over a single-task learning approach’s MAE (0.223 m) and a plain MTL approach’s MAE (0.192 m), it is still quite high considering that the average foundation height of our data was 0.45 m. The overall MAE for foundation height is small compared with uncertainty in flood depth exceedance values arising from factors such as uncertainty in hydrodynamic models, variability in levee overtopping volumes, probability of system failures, noise in lidar measurements, and randomness in the characteristics of observed historic events (for example, studies of flood hazards in Louisiana have found that 80% confidence intervals for the 100-year flood depth may be several feet in many locations (Fischbach et al. 2017) but is not as low as it could be from a computational perspective. Of particular concern is the fact that our worst relative estimates (MAE/ground truth) were usually in the lowest foundation (0.153 m; Table 4) when lower foundations are actually at greater flood risks than higher ones.

Similarly, in some buildings with a foundation height of 0.153 m, the prediction is as high as 3.0 m (Fig. 14). Still, this study is the first of its kind for inexpensive and efficient regression of foundation heights from GSV images using deep learning techniques and in a MTL environment; therefore, we are positive that improvements will be made in the future by building on it. The performances in the number of stories task were promising for all the mentioned approaches. However, they were still heavily skewed toward the majority class. From a financial planning perspective, this is inefficient because the classification of a significant proportion of two-storied buildings as one-storied can drastically overestimate flood damage. Ideally, we want a balanced performance if improving the performances of both classes is not possible, such that we are neither more conservative nor more generous than optimal. The same can be said about foundation type and building type classification. By erroneously predicting a quarter of commercial buildings as residential, we may be underestimating flood damage (commercial buildings may have more to lose if flooded). Similarly, the detection of pier foundations might be more important than the detection of slab foundations in some areas because the former is typically located in riskier places (such as immediately next to coasts within a larger coastal region). The fact that our approach’s F1 score in pier foundation detection is only slightly higher than the baseline is a limitation. Overall, we improved the performances of some classes to an extent, but they still need significant improvements if we are to accurately assess flood risks (i.e., neither overestimate nor underestimate damages). The best way forward from this research is as follows: for classification tasks, to find ways to keep the performances in the majority classes at least at the current level and raise the performances of the minority classes to the level observed in the majority classes; and for foundation height, significantly improve the performance for smaller heights. In summary, a selective approach might be a more suitable next step to improve the overall performance.

To collect comprehensive and up-to-date structure attribute data to assess flood risks without labor-intensive street surveys, this study details a deep learning-based framework that can simultaneously estimate multiple building attributes from GSV imagery. An extensive evaluation is done to select the optimal building detection model. Furthermore, a feature fusion scheme is proposed that combines image features with meta information that improves the prediction of foundation heights. Additionally, TREncNet is introduced to encode task relations as network connections for MTL that enhance the predictions of classification tasks.

The proposed framework achieves a 0.177-m foundation height prediction MAE and classification f-scores of 77.96% for foundation type, 83.12% for building type, and 94.60% for building stories and requires less than five days to predict the attributes of 0.8 million buildings in the coastal Louisiana study region.

Given its capabilities and efficiency, the proposed framework saves time and money for flood risk studies. The method also has the potential for predicting other types of building attributes relevant to estimating hurricane, tornado, or seismic hazards. Consequently, the data set of inferred foundation heights (Chen et al. 2020) is planned for direct use by the risk model used in Louisiana’s 2023 Coastal Master Plan (Brown et al. 2020). However, the classification of nonresidential assets (e.g., distinguishing movie theaters, banks, and hospitals from each other) would be challenging without much larger ground-truth training sets, and GSV images often do not capture the full structure for large, multistory assets such as office towers or hotels.

Deep learning-based methods have been used in the geospatial context in the past; however, such methods have primarily focused on performance at the expense of cost. This study proves that both inexpensive and accurate building attribute prediction schemes are possible by combining three methods: MTL, feature fusion, and TREncNet.

Some or all data, models, or code that support the findings of this study are available from the corresponding author on reasonable request. The coastwide data set of estimated structural features is publicly available (Chen et al. 2020). The ground-truth data collected by government sources are available on request to Dr. Johnson. The availability of the images used in this work is restricted by the terms of use of the GSV API.

This study was funded in part by the Andrew W. Mellon Foundation. We also extend our sincere thanks to Mr. Ahmad Bassel Abdallah, a freshman at Purdue University, for helping us with some of the figures used in this paper.