Innovative Missing Data Replacement Methods Using Time Series Models

Missing data commonly exist in operational records, especially in the influent and effluent water quality records in water and wastewater treatment plants. For example, unexpected events cause the failure of measurements of plant data, and holidays or less experienced personnel shifts make the measurements unavailable. Traditional missing data replacement methods, such as average (AVE) method and average of nearest observations (ANO) method, depend on a MCAR (missing completely at random) assumption, which means the probability that an observation is missing shall not be related with its data structure. Unfortunately, MCAR observations are less likely in reality because they are commonly correlated with time (for example, most of missing data happen on weekends and holidays). In this paper, two innovative methods (TES and TESWN) are developed based on commonly used time series models that can accurately capture time series' statistical characteristics: trend, lag, and/or seasonality. The TES method is a combined two directional exponential smoothing method, where forward ES and backward ES are applied independently and the mean values of those two forecasts are used to replace the missing values. The TESWN method is known as the TES method with white noise added. It begins with the TES method, then adds a white noise term to account for random effects observed in the data but not captured by the autocorrelation function. These two innovative methods, together with AVE and ANO methods, are applied to real water and wastewater treatment plants' influent data (flow and concentrations). The results indicate that the TES method is the best overall performance method because it has more similar statistical characteristics (mean, standard deviation, percentiles, etc.) with the original data. The TESWN method can also be recommended if the goal is to capture the overall variability in the distribution instead of obtaining a close match to the exact time series.