Evaluation of Climate Model Performance for Water Supply Studies: Case Study for New York City

Meteorological measurements of Ppt, Tmax, Tmin, Tave, and Wind were used in the skill score comparisons described subsequently. Two types of observed data (OD1 and OD2) were used in this study for the skill score comparisons. OD1 is from the daily $1/8\text{-}\mathrm{degree}$ gridded reanalysis product produced by Maurer et al. (2002). Data for the five meteorological parameters was taken from seven grid cells surrounding the NYC WOH watershed [closest land grids (CLG) boxes in Fig. 1(a)]. These were then averaged to give a single daily value representative of the entire watershed area. OD2 is based on measurements made at meteorological stations (17 precipitation, and 3 temperature) distributed within the WOH watersheds [locations provided in Fig. 1(b)]. Spatial averages of air temperature and precipitation were used to calculate basin average values for each reservoir watershed (details provided in Supplemental Data). Wind data were collected from a single shore-based station near each reservoir [Fig. 1(b)]. These were also averaged to give a single WOH value.

Data associated with multiple realizations of the baseline scenario (20C3M) from 20 GCMs were downloaded, and from these data the five meteorological variables were extracted (Table S1). The number of useable GCM realizations ranged between 30 and 45 depending on the climate variables evaluated. The GCMs were from research groups participating in the World Climate Research Programme’s (WCRP’s) Couple Model Intercomparison Project Phase 3 (CMIP3) multimodel simulations. The grids surrounding the study region were extracted and then interpolated to a common 2.5º grid using bilinear interpolation (yellow boxes in Fig. 1).

The SS ranged from 0.65 to 0.95 for Ppt in all four seasons at CLG scale using OD1 dataset (Fig. 2). The solid red line is the mean while the shaded region represents the variation in PDFs for a meteorological data simulated by the different AR4 climate models in the CLG region (seven land-grid points surrounding NYC watersheds) for four seasons [December-January-February (DJF), March-April-May (MAM), June-July-August (JJA), and September-October-November (SON)]. In each panel, the black bold line represents the PDF obtained using daily observed data (OD1) for the study region. Differences between the PDFs of observed Ppt and that derived from most of the GCMs are larger during summer and fall seasons (smaller skill scores). The reasons for this may be that the GCM models tend to overestimate the number of small Ppt events ($1–3\mathrm{mm}/\mathrm{day}$, Fig. 2) and small to medium Ppt events (Fig. 3, minimum; 5th–75th percentiles). Similar overestimation of small events (GCM drizzle) were observed in Australia (Perkins et al. 2007) and India (Anandhi and Nanjundiah 2015). The boxplots in Fig. 3 are interpreted as follows: middle line shows the median value; top and bottom of box show the upper and lower quartiles (i.e., 75th and 25th percentile values); and whiskers show the minimum and maximum model values. The triangle and circle in the boxplots represent the observed and GCM ensemble mean of the statistics for seasons DJF, MAM, JJA, and SON. The gir GCM statistic values calculated were excluded from the plots. Box and whisker plots indicate statistics calculated for daily climate variable calculated for the various AR4 climate models across the four seasons, namely DJF, MAM, JJA, and SON for CLG spatial scale (seven land-grid points surrounding NYC watersheds) and OD1 dataset. The overestimation of small events contributes to an overestimation of total precipitation even though the models also tend to underestimate larger events in summer and fall (Figs. 2 and 3). Note that the models underpredicted the median and standard deviation of Ppt in all of the seasons (Fig. 2, median).

SS ranged from 0.55 to 0.95 for Tave, 0.3 to 0.95 for Tmax, and 0.4 to 0.95 for Tmin in the four seasons [Fig. 4(a)]. The figure is interpreted as follows: middle line shows the median value; top and bottom of box show the upper and lower quartiles (i.e., the 75th and 25th percentile values); and whiskers show the maximum and minimum percentile skill scores. The outliers are indicated by a “+.” The circle in the figure represents the mean skill score each of the seasons (DJF, MAM, JJA, and SON). Among the temperatures, Tave was better simulated than Tmax and Tmin. The reasons for lower SS may be that the GCM models were underestimating the number of cold days and overestimating the number of warm days especially during winter season (Tmax and Tmin in Fig. 2). However, the largest temperature biases—as well as the largest between-model variability—were found in summer (Columns 2–4 in Fig. 2).

SS ranged from 0.2 to 0.95 for Wind in the four seasons [Fig. 4(a)]. In most cases, the GCM simulated Wind distribution compared unfavorably to the observed distribution (Fig. 2). The reasons for the lower SS is because models overestimated smaller winds (Fig. 3, minimum; Fig. 2, $0–5\mathrm{m}/\mathrm{s}$) and underestimated the mean and median winds as well as the frequency of large events. The largest model biases and the largest between-model variability were found for smaller events. Additionally, the models tended to overestimate the frequency of small events ($0–5\mathrm{m}/\mathrm{s}$) and underestimate the frequency of large events. Similar results were observed for OD2 data set at CCG scale (figure not shown).

The results of the probability-based SS ranking procedure at the CLG scale using OD1 dataset for all ensemble members of a GCM are summarized as a function of season [Figs. 4(a) and S1] and the SS is arranged in descending order for each variable [Fig. 4(b)]. In the figure, the AR4 climate models are ranked based on average skill scores for spatial scale CLG using OD1 dataset. For a variable and GCM, the average skill score is calculated from the skill scores of different realizations for a GCM and four seasons (DJF, MAM, JJA, and SON) in each realization. The GCM with the highest skill score is given rank 1. While calculating the average ranks, only GCMs that have all five meteorological variables were used. The closeness of the statistical measures of the GCM data to equivalent observations can be seen in Fig. 3. In general, no one model was consistently ranked best by SS for all the meteorological variables (Ppt, Wind, Tave, Tmax, and Tmin), or during all the seasons (DJF, MAM, JJA, and SON). Overall, the magnitudes of SS did not vary between seasons for Ppt and Wind, although there was a higher variability in SS during summer for Ppt [Fig. 4(a)]. For temperature, there were generally lower magnitudes of SS during summer. Overall, spring had a higher mean/median skill score for all five variables. Fall’s mean/median SS were also high for temperature variables and wind. For each meteorological variable, different ensemble members of the same model had similar SS in the SS ranking procedure (i.e., cc4 and cc6). This can indicate that the skill scores were not due to random or chaotic processes but were in fact related to model formulation. Ensemble average SS showed no clear relationship between SS and three model characteristics (horizontal resolution, convective scheme, and flux correction).

Overall rankings are in Table 1. The cs5 seemed to have consistently low ranks in the region for Ppt and temperature variables. The gir had very different statistics (not shown due to being outside the range of figures) compared with the rest of the models for Ppt.

Our results show that when GCMs are ranked by skill score and a variety of statistics for different metrological variables, there is no obvious way to choose a subset of models that are clearly superior. First, we did not find certain models as clearly superior; instead, there was a gradual decrease in model skill along a continuum from highest to lowest skill score. Second, different models performed better for different meteorological variables and performance measures. This can greatly complicate choosing a subset of models when simulations depend on multiple meteorological drivers. The simplest way to choose a subset of models is to identify how many models are appropriate for the variable(s) of interest, then choose a subset from these based on the combined SS rankings that include all needed meteorological variables. For the NYC water supply watershed region and when evaluating multiple metrological parameters, we concluded that using as many GCM datasets as possible was the best strategy. We were not able to identify a clear subset of models that was superior for all the meteorological variables used in our water supply simulations. However, we were able to eliminate several GCM data sets that clearly underpreformed. Even though our evaluation was not able to clearly identify GCM models that preformed best for our purposes, we feel that documentation of this methodology is valuable. Results could be different when fewer meteorological parameters are needed, or in other geographical regions where the GCMs may agree to a greater extent.