Large-Scale Loop Detector Troubleshooting Using Clustering and Association Rule Mining
Publication: Journal of Transportation Engineering, Part A: Systems
Volume 146, Issue 7
Abstract
The archived data from traffic sensors are used in a wide range of traffic management applications. However, missing or invalid data are becoming an important concern. This study proposes a systematic approach to identify and characterize data error patterns to facilitate large-scale loop detector troubleshooting. Data were collected from loop detectors in Phoenix. A set of quality control criteria was applied on daily 20-s data to find the error percentage for each loop detector. A fuzzy -means clustering method was implemented on the data quality check results and preliminary clusters were identified. To discover the most frequent rules within the clusters, an association rule mining method was applied to the clusters’ data subsets. Loop detector stations with different error patterns were visited in the field to verify the clustering and association rule mining results, identify potential causes, and recommend appropriate solutions. The analysis identified four key patterns, indicating that the proposed approach successfully found the relationships in the data errors. The findings of this study help traffic engineers to more easily diagnose and troubleshoot large-scale loop detector errors.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
All data, models, or code generated or used during the study are confidential in nature. All these items are part of a funded project by the Arizona Department of Transportation, so they are not allowed to be shared.
Acknowledgments
The authors would like to thank the Arizona Department of Transportation (ADOT) for funding and data support. We also acknowledge Mr. Vahid N. Goftar, Mr. David Riley, and Mr. Reza Karimvand for their professional advice and project coordination. The authors wish to extend their thanks to Mr. Robert Kluger and Ms. Jan Szechi for valuable comments and proofreading.
References
Agrawal, R., T. Imieliński, A. Swami, R. Agrawal, T. Imieliński, and A. Swami. 1993. “Mining association rules between sets of items in large databases.” In Proc., 1993 ACM SIGMOD Int. Conf. on Management of Data—SIGMOD ’93, 207–216. New York: ACM Press.
Agrawal, R., and R. Srikant. 1994. “Fast algorithms for mining association rules.” In Proc., 20th Int. Conf. on Very Large Data Bases, VLDB, 487–499. New York: Association for Computing Machinery.
Al-Maolegi, M., and B. Arkok. 2014. “An improved Apriori algorithm for association rules.” Int. J. Nat. Lang. Comput. 3 (1): 21–29. https://doi.org/10.5121/ijnlc.2014.3103.
Bae, B., H. Kim, H. Lim, Y. Liu, L. D. Han, and P. B. Freeze. 2018. “Missing data imputation for traffic flow speed using spatio-temporal cokriging.” Transp. Res. Part C: Emerging Technol. 88 (Mar): 124–139. https://doi.org/10.1016/j.trc.2018.01.015.
Bezdek, J. C., R. Ehrlich, and W. Full. 1984. “FCM: The fuzzy -means clustering algorithm.” Comput. Geosci. 10 (2–3): 191–203. https://doi.org/10.1016/0098-3004(84)90020-7.
Bickel, P. J., C. Chen, J. Kwon, J. Rice, E. Van Zwet, and P. Varaiya. 2007. “Measuring traffic.” Stat. Sci. 22 (4): 581–597. https://doi.org/10.1214/07-STS238.
Cheevarunothai, P. 2006. “Identification and correction of dual-loop sensitivity problems.” Transp. Res. Rec. 1945 (1): 73–81. https://doi.org/10.1177/0361198106194500110.
Cheevarunothai, P., Y. Wang, and N. Nihan. 2005. “Development of an advanced loop event data analyzer (ALEDA) system for dual-loop detector malfunction detection and investigation.” In Proc., 12th World Congress on Intelligent Transport Systems. Washington, DC: Transportation Research Borad.
Chen, C., J. Kwon, J. Rice, A. Skabardonis, and P. Varaiya. 2003. “Detecting errors and imputing missing data for single-loop surveillance systems.” Transp. Res. Rec. 1855 (1): 160–167. https://doi.org/10.3141/1855-20.
Chen, X., Z. He, and L. Sun. 2019. “A Bayesian tensor decomposition approach for spatiotemporal traffic data imputation.” Transp. Res. Part C: Emerging Technol. 98 (Jan): 73–84. https://doi.org/10.1016/j.trc.2018.11.003.
Chen, Y., Y. Lv, and F.-Y. Wang. 2020. “Traffic flow imputation using parallel data and generative adversarial networks.” IEEE Trans. Intell. Transp. Syst. 21 (4): 1624–1630. https://doi.org/10.1109/TITS.2019.2910295.
Chen, Z., X. Qin, E. Schneider, Y. Cheng, S. Parker, and R. R. Shaon. 2019c. “Designing a comprehensive procedure for flagging archived traffic data: A case study.” Transp. Res. Rec. 2673 (6): 165–175. https://doi.org/10.1177/0361198119841286.
Coifman, B. 1999. “Using dual loop speed traps to identify detector errors.” Transp. Res. Rec. 1683 (1): 47–58. https://doi.org/10.3141/1683-07.
Coifman, B., and H. Lee. 2011. Diagnosing chronic errors in freeway loop detectors from existing field hardware. Washington, DC: Transportation Research Borad.
Duan, Y., Y. Lv, Y.-L. Liu, and F.-Y. Wang. 2016. “An efficient realization of deep learning for traffic data imputation.” Transp. Res. Part C: Emerging Technol. 72 (Nov): 168–181. https://doi.org/10.1016/j.trc.2016.09.015.
Dunn, J. C. 1973. “A fuzzy relative of the ISODATA process and its use in detecting compact well-separated clusters.” J. Cybern. 3 (3): 32–57. https://doi.org/10.1080/01969727308546046.
Ghosh, A., and B. Nath. 2004. “Multi-objective rule mining using genetic algorithms.” Inf. Sci. 163 (1–3): 123–133. https://doi.org/10.1016/j.ins.2003.03.021.
Hothorn, T., and B. S. Everitt. 2014. A handbook of statistical analyses using R. Boca Raton, FL: CRC Press.
Hu, P., R. Goeltz, and R. Schmoyer. 2001. Proof of concept of ITS as an alternative data resource: A demonstration project of Florida and New York data. Oak Ridge, TN: Oak Ridge National Laboratory.
Ishimaru, J. 1998. CDR user’s guide. Seattle: Univ. of Washington.
Jacobson, L. N., N. L. Nihan, and J. D. Bender. 1990. “Detecting erroneous loop detector data in a freeway traffic management system.” Transp. Res. Rec. 1287: 151–166.
Karimpour, A., A. Ariannezhad, and Y.-J. Wu. 2019. “Hybrid data-driven approach for truck travel time imputation.” IET Intel. Transp. Syst. 13 (10): 1518–1524. https://doi.org/10.1049/iet-its.2018.5469.
Klein, L. A. 2001. Sensor technologies and data requirements for ITS. Boston: Artech House.
Klein, L. A., M. K. Mills, and D. R. P. Gibson. 2006. Traffic detector handbook. 3rd ed. McLean, VA: USDOT.
Laña, I., I. I. Olabarrieta, M. Vélez, and J. Del Ser. 2018. “On the imputation of missing data for road traffic forecasting: New insights and novel techniques.” Transp. Res. Part C: Emerging Technol. 90 (May): 18–33. https://doi.org/10.1016/j.trc.2018.02.021.
Lee, H., and B. Coifman. 2011. “Identifying and correcting pulse-breakup errors from freeway loop detectors.” Transp. Res. Rec. 2256 (1): 68–78. https://doi.org/10.3141/2256-09.
Lu, X., Z. Kim, M. Cao, P. Varaiya, and R. Horowitz. 2010. Deliver a set of tools for resolving bad inductive loops and correcting bad data. Berkeley, CA: Univ. of California.
Lu, X.-Y., P. Varaiya, R. Horowitz, and J. Palen. 2008. “Faulty loop data analysis/correction and loop fault detection.” In Proc., 15th World Congress on Intelligent Transport Systems and ITS America’s 2008 Annual Meeting. Washington, DC: ITS America.
Ma, X., Z. Dai, Z. He, J. Ma, Y. Wang, and Y. Wang. 2017. “Learning traffic as images: A deep convolutional neural network for large-scale transportation network speed prediction.” Sensors 17 (4): 818. https://doi.org/10.3390/s17040818.
Payne, H. J., E. D. Helfenbein, and H. C. Knobel. 1976. Vol. 2 of Development and testing of incident detection algorithms. Olympia, WA: Federal Highway Administration.
Ran, B., H. Tan, Y. Wu, and P. J. Jin. 2016. “Tensor based missing traffic data completion with spatial–temporal correlation.” Physica A 446 (Mar): 54–63. https://doi.org/10.1016/j.physa.2015.09.105.
Rodrigues, F., K. Henrickson, and F. C. Pereira. 2019. “Multi-output Gaussian processes for crowdsourced traffic data imputation.” IEEE Trans. Intell. Transp. Syst. 20 (2): 594–603. https://doi.org/10.1109/TITS.2018.2817879.
Schmoyer, R., P. S. Hu, and R. T. Goeltz. 2001. “Statistical data filtering and aggregation to hour totals of intelligent transportation system 30-s and 5-min vehicle counts.” Transp. Res. Rec. 1769 (1): 79–86. https://doi.org/10.3141/1769-10.
Thambusamy, V. 2014. “Performance based analysis between k-means and fuzzy C-means clustering algorithms for connection oriented telecommunication data.” Appl. Soft Comput. 19 (Jun): 134–146. https://doi.org/10.1016/j.asoc.2014.02.011.
Turner, S. 2004. “Defining and measuring traffic data quality: White paper on recommended approaches.” Transp. Res. Rec. 1870 (1): 62–69. https://doi.org/10.3141/1870-08.
Turner, S. 2007. Quality control procedures for archived operations traffic data: Synthesis of practice and recommendations. Washington, DC: Federal Highway Administration.
Turner, S., R. Margiotta, and T. Lomax. 2004. Monitoring urban freeways in 2003: Current conditions and trends from archived operations data. College Station, TX: Texas Transportation Institute.
Turochy, R. E., and B. L. Smith. 2000. “New procedure for detector data screening in traffic management systems.” Transp. Res. Rec. 1727 (1): 127–131. https://doi.org/10.3141/1727-16.
Vlahogianni, E. I., M. G. Karlaftis, and J. C. Golias. 2014. “Short-term traffic forecasting: Where we are and where we’re going.” Transp. Res. Part C: Emerging Technol. 43 (Jun): 3–19. https://doi.org/10.1016/j.trc.2014.01.005.
Wang, Y., and N. L. Nihan. 2003. “Can single-loop detectors do the work of dual-loop detectors?” J. Transp. Eng. 129 (2): 169–176. https://doi.org/10.1061/(ASCE)0733-947X(2003)129:2(169).
Information & Authors
Information
Published In
Journal of Transportation Engineering, Part A: Systems
Volume 146 • Issue 7 • July 2020
Copyright
©2020 American Society of Civil Engineers.
History
Received: Jun 30, 2019
Accepted: Feb 19, 2020
Published online: May 12, 2020
Published in print: Jul 1, 2020
Discussion open until: Oct 12, 2020
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.