Comparison of Interpolation, Statistical, and Data-Driven Methods for Imputation of Missing Values in a Distributed Soil Moisture Dataset
Publication: Journal of Hydrologic Engineering
Volume 19, Issue 1
Abstract
Missing values in in situ monitoring data is a problem often encountered in hydrologic research and applications. Values in a data set may be missing because of sensor error or failure of data recording devices. Whereas various imputation techniques have focused on hydrometeorological data, very few studies have investigated gap-filling methods for soil moisture data. This paper aims to fill that gap by investigating well-established statistical and data-driven methods for infilling missing values in a high resolution, soil moisture time series. Since 2006, the authors collected hourly soil moisture data in the Hamilton-Halton Watershed, Southern Ontario, Canada at four research sites. Each site contained nine stations with time domain reflectometry (TDR) soil sensors at six soil depths. From these distributed data sets, the authors removed values randomly () and systematically () from the data to evaluate the effectiveness of the monthly average replacement (MAR), soil layer relative difference (SLRD), linear and cubic interpolation, artificial neural networks (ANN), and evolutionary polynomial regression (EPR) infilling methods. When values were randomly removed, interpolation, ANN, and EPR were able to infill the missing values with similar efficiency, whereas MAR and SLRD were the least effective methods. Similarly, when large systematic gaps were present in the data, interpolation and ANN were the most effective methods of infilling, respectively. However, the effectiveness of both infilling methods is limited as serial gaps become larger than 72–100 h.
Get full access to this article
View all available purchase options and get full access to this article.
Acknowledgments
This work was partially funded by the Canadian Foundation for Innovation (CFI), the Natural Science and Engineering Research Council (NSERC), and the Ontario Ministry of Research and Innovation (OMRI). The authors are grateful to Dr. G. Dumedah for his assistance in data-processing and Dr. O. Giustolisi for his technical advice on the use of EPR. This work used EPR software developed by Orazio Giustolisi (Technical Univ. of Bari) and Dragan Savic (Univ. of Exeter). The authors are also grateful to three anonymous reviewers for their comments that helped to improve the manuscript.
References
ASCE. (1996). Hydrology handbook, 2nd Ed., ASCE, New York.
Bermudez, J. D., Corberan-Vallet, A., and Vercher, E. (2009). “Forecasting time series with missing data using Holt’s model.” J. Stat. Plan. Infer., 139(8), 2791–2799.
Casenave, A. (1982). “The small rain-simulator: Conditions of use and guidelines for data analysis, Ivory Coast.” Cahiers ORSTOM, Serie Hydrologie, 19(4), 207–227.
Cosh, M. H., Jackson, T. J., Bindlish, R., and Prueger, J. H. (2004). “Watershed scale temporal and spatial stability of soil moisture and its role in validating satellite estimates.” Remote Sens. Environ., 92(4), 427–435.
Coulibaly, P., and Evora, N. (2007). “Comparison of neural network methods for infilling missing daily weather records.” J. Hydrol., 341(1), 27–41.
Dorigo, W. A., et al. (2011). “The international soil moisture network: A data hosting facility for global in situ soil moisture measurements.” Hydrol. Earth Syst. Sci., 15(5), 1675–1698.
Dumedah, G., and Coulibaly, P. (2011). “Evaluation of statistical methods for infilling missing values in high-resolution soil moisture data.” J. Hydrol., 400(1–2), 95–102.
Elshorbagy, A., Corzo, G., Srinivasulu, S., and Solomatine, D. P. (2010a). “Experimental investigation of the predictive capabilities of data-driven modeling techniques in hydrology. Part 1: Concepts and methodology.” Hydrol. Earth Syst. Sci., 14(10), 1931–1941.
Elshorbagy, A., Corzo, G., Srinivasulu, S., and Solomatine, D. P. (2010b). “Experimental investigation of the predictive capabilities of data-driven modeling techniques in hydrology. Part 2: Application.” Hydrol. Earth Syst. Sci., 14(10), 1943–1961.
Elshorbagy, A., and El-Baroudy, I. (2009). “Investigating the capabilities of evolutionary data-driven techniques using the challenging estimation of soil moisture content.” J. Hydroinf., 11(3–4), 237–251.
Elshorbagy, A., and Parasuraman, K. (2008). “On the relevance of using artificial neural networks for estimating soil moisture content.” J. Hydrol., 362(1–2), 1–18.
Elshorbagy, A., Simonovic, S., and Panu, U. (2000). “Performance evaluation of artificial neural networks for runoff prediction.” J. Hydrol. Eng., 424–427.
Entekhabi, D., Rodriguez-Iturbe, I., and Castelli, F. (1996). “Mutual interaction of soil moisture state and atmospheric processes.” J. Hydrol., 184(1–2), 3–17.
Famiglietti, J. S., Ryu, D., Berg, A. A., Rodell, M., and Jackson, T. J. (2008). “Field observations of soil moisture variability across scales.” Water Resour. Res., 44(1), W01423.
Fernández-Gálvez, J., Verhoef, A., and Barahona, E. (2007). “Estimating soil water fluxes from soil water records obtained using dielectric sensors.” Hydrol. Processes, 21(20), 2785–2793.
Fritsch, F. N., and Carlson, R. E. (1980). “Monotone piecewise cubic interpolation.” SIAM J. Numer. Anal., 17(2), 238–246.
Gheyas, I. A., and Smith, L. S. (2010). “A neural network-based framework for the reconstruction of incomplete data sets.” Neurocomputing, 73(16–18), 3039–3065.
Giustolisi, O., Doglioni, A., and Savic, D. A. (2004a). “Data reconstruction and forecasting by evolutionary polynomial regression.” Proc. of the 6th Int. Conf. on Hydroinformatics, World Scientific Publishing Company, Toh Tuck Link, Singapore, 787–794.
Giustolisi, O., Doglioni, A., and Savic, D. A. (2004b). “Knowledge discovery by evolutionary polynomial regression.” Proc. of the 6th Int. Conf. on Hydroinformatics, World Scientific Publishing Company, Toh Tuck Link, Singapore, 1647–1654.
Giustolisi, O., Doglioni, A., Savic, D. A., and Webb, B. W. (2007). “A multi-model approach to analysis of environmental phenomena.” Environ. Modell. Softw., 22(5), 674–682.
Giustolisi, O., and Savic, D. A. (2003). “Evolutionary polynomial regression (EPR): Development and applications.”, School of Engineering, Computer Science and Mathematics, Centre for Water Systems, Univ. of Exeter, Exeter, U.K.
Haykin, S. (1999). Neural networks: A comprehensive foundation, Prentice Hall, Upper Saddle River, NJ.
Hopke, P. K., Liu, C. H., and Rubin, D. B. (2001). “Multiple imputation for multivariate data with missing and below-threshold measurements: Time-series concentrations of pollutants in the Arctic.” Biometrics, 57(1), 22–33.
Jain, S., Singh, V., and van Genuchten, M. (2004). “Analysis of soil water retention data using artificial neural networks.” J. Hydrol. Eng., 415–420.
Junninen, H., Niska, H., Tuppurainen, K., Ruuskanen, J., and Kolehmainen, M. (2004). “Methods for imputation of missing values in air quality data sets.” Atmos. Environ., 38(18), 2895–2907.
Kahaner, D. (1989). Numerical methods and software, Prentice Hall, Englewood Cliffs, NJ.
Khalil, M., Panu, U. S., and Lennox, W. C. (2001). “Groups and neural networks based streamflow data infilling procedures.” J. Hydrol., 241(3–4), 153–176.
Liu, X., Coulibaly, P., and Evora, N. (2008). “Comparison of data-driven methods for downscaling ensemble weather forecasts.” Hydrol. Earth Syst. Sci., 12(2), 615–624.
MacKay, D. J. C. (1992). “A practical Bayesian framework for backpropagation networks.” Neural Comput., 4(3), 448–472.
Mair, A., and Fares, A. (2010). “Assessing rainfall data homogeneity and estimating missing records in Mākaha Valley, O’ahu, Hawai’i.” J. Hydrol. Eng., 61–66.
Markewitz, D., Devine, S., Davidson, E. A., Brando, P., and Nepstad, D. C. (2010). “Soil moisture depletion under simulated drought in the Amazon: Impacts on deep root uptake.” New Phytol., 187(3), 592–607.
Monteith, J. L. (1965). “Evaporation and environment.” Symp. Soc. Exp. Biol., 19, 205–234.
Murtagh, F., Zheng, G., Campbell, J. G., and Aussem, A. (2000). “Neural network modelling for environmental prediction.” Neurocomputing, 30(1–4), 65–70.
NeuroSolutions [Computer software]. NeuroDimension Inc., Gainsville, FL.
Ng, W., Panu, U., and Lennox, W. (2009). “Comparative studies in problems of missing extreme daily streamflow records.” J. Hydrol. Eng., 91–100.
Rollenbeck, R., and Anhuf, D. (2007). “Characteristics of the water and energy balance in an Amazonian lowland rainforest in Venezuela and the impact of the ENSO-cycle.” J. Hydrol., 337(3–4), 377–390.
Savic, D. A., Giustolisi, O., and Laucelli, D. (2009). “Asset deterioration analysis using multi-utility data and multi-objective data mining.” J. Hydroinf., 11(3–4), 211–224.
Schwarzel, K., et al. (2009). “Soil water content measurements deliver reliable estimates of water fluxes: A comparative study in a beech and a spruce stand in the Tharandt forest (Saxony, Germany).” Agric. For. Meteorol., 149(11), 1994–2006.
Seneviratne, S. I., et al. (2010). “Investigating soil moisture-climate interactions in a changing climate: A review.” Earth Sci. Rev., 99(3–4), 125–161.
Ulke, A., Tayfur, G., and Ozkul, S. (2009). “Predicting suspended sediment loads and missing data for Gediz River, Turkey.” J. Hydrol. Eng., 954–965.
Vachaud, G. A., Passerat de Slians, A., Balabanis, P., and Vauclin, M. (1985). “Temporal stability of spatially measured soil water probability density function.” Soil Sci. Soc. Am. J., 49(4), 822–828.
Western, A. W., Blöschl, G., and Grayson, R. B. (1998). “Geostatistical characterization of soil moisture patterns in the Tarrawarra catchment.” J. Hydrol., 205(1–2), 20–37.
Western, A. W., and Grayson, R. B. (1998). “The Tarrawarra data set: Soil moisture patterns, soil characteristics and hydrological flux measurements.” Water Resour. Res., 34(10), 2765–2768.
Western, A. W., Grayson, R. B., and Blöschl, G. (2002). “Scaling of soil moisture: A hydrological perspective.” Annu. Rev. Earth Planet. Sci., 30, 149–180.
Wu, C. L., Chau, K. W., and Fan, C. (2010). “Prediction of rainfall time series using modular artificial neural networks coupled with data-preprocessing techniques.” J. Hydrol., 389(1–2), 146–167.
Information & Authors
Information
Published In
Journal of Hydrologic Engineering
Volume 19 • Issue 1 • January 2014
Pages: 26 - 43
Copyright
© 2014 American Society of Civil Engineers.
History
Received: Nov 11, 2011
Accepted: Dec 13, 2012
Published online: Dec 17, 2012
Discussion open until: May 17, 2013
Published in print: Jan 1, 2014
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.