Systematic Analysis of After-Action Reports: A Plan Evaluation Methodological Approach
Publication: Natural Hazards Review
Volume 22, Issue 4
Abstract
After-action reports are often completed by local, state, or federal jurisdictions following a disaster to identify what did and did not go well and to provide recommendations to improve upon identified weaknesses. Emergency management scholars are increasingly using these reports as data, yet the variation of methodological approaches calls into question the validity of the studies. The purpose of this paper is not only to address the lack of methods for systematically analyzing after-action reports, but to also propose a method derived from the plan evaluation literature. This paper presents an overview of after-action reports as data in the existing literature, a review of the plan evaluation methodology from the planning literature, and concludes with an explanation of the methods applied to after-action reports.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request. Available with this paper is the codebook. Because this paper discusses the methodological approach, the data will be not available until after the larger case study is published.
Acknowledgments
I would like to acknowledge graduate research assistants, Juan Lugo and Jasmine Blais, who assisted in creation of the codebook and coding the AARs.
References
Baer, W. C. 1997. “General plan evaluation criteria: An approach to making better plans.” J. Am. Plann. Assoc. 63 (3): 329–344. https://doi.org/10.1080/01944369708975926.
Barnett, D. J., K. Strauss-Riggs, V. L. Klimczak, A. J. Rosenblum, and T. D. Kirsch. 2020. “An analysis of after action reports from Texas Hurricanes in 2005 and 2017.” J. Public Health Manage. Pract. 27 (2): E71–E78. https://doi.org/10.1097/phh.0000000000001120.
Berke, P., and D. Godschalk. 2009. “Searching for the good plan: A meta-analysis of plan quality studies.” J. Plann. Lit. 23 (3): 227–240. https://doi.org/10.1177/0885412208327014.
Brody, S. D. 2003. “Implementing the principles of ecosystem management through local land use planning.” Population Environ. 24 (6): 511–540. https://doi.org/10.1023/A:1025078715216.
Corbin, J., and A. Strauss. 2015. Basics of qualitative research: Techniques and procedures for developing grounded theory. 4th ed. Los Angeles: Sage.
Cutter, S. L. 2016. “The landscape of disaster resilience indicators in the USA.” Nat. Hazards 80 (2): 741–758. https://doi.org/10.1007/s11069-015-1993-2.
Cutter, S. L., C. G. Burton, and C. T. Emrich. 2010. “Disaster resilience indicators for benchmarking baseline conditions.” J. Homeland Secur.Emergency Manage. 7 (1): 51. https://doi.org/10.2202/1547-7355.1732.
Davies, R., E. Vaughan, G. Fraser, R. Cook, M. Ciotti, and J. E. Suk. 2019. “Enhancing reporting of after action reviews of public health emergencies to strengthen preparedness: A literature review and methodology appraisal.” Disaster Med. Public Health Preparedness 13 (3): 618–625. https://doi.org/10.1017/dmp.2018.82.
Deyle, R. E., and R. A. Smith. 1998. “Local government compliance with state planning mandates: The effects of state implementation in Florida.” J. Am. Plann. Assoc. 64 (4): 457–469. https://doi.org/10.1080/01944369808976004.
Feinberg, D. S., and C. M. Ryan. 2020. “Evaluating the quality and implementation of hazard mitigation plans in coastal Washington State.” Nat. Hazards Rev. 21 (2): 04020013. https://doi.org/10.1061/(ASCE)NH.1527-6996.0000372.
FEMA. 2020. “Homeland Security exercise and evaluation program.” Accessed December 1, 2020. https://www.fema.gov/emergency-managers/national-preparedness/exercises/hseep.
Guyadeen, D., J. Thistlethwaite, and D. Henstra. 2018. “Evaluating the quality of municipal climate change plans in Canada.” Clim. Change 152 (1): 121–143. https://doi.org/10.1007/s10584-018-2312-1.
Hawkins, C., and C. C. Knox. 2014. “Disaster events and policy change in Florida.” In Disaster and development, edited by N. Kapucu and T. Liou, 111–127. Cham, Switzerland: Springer.
Hilts, A. S., S. Mack, M. Eidson, T. Nguyen, and G. S. Birkhead. 2016. “New York State public health system response to Hurricane Sandy: An analysis of emergency reports.” Disaster Med. Public Health Preparedness 10 (3): 308–313. https://doi.org/10.1017/dmp.2015.142.
Hu, Q., C. C. Knox, and N. Kapucu. 2014. “What have we learned since September 11, 2001? A network study of the Boston Marathon bombings response.” Public Administration Rev. 74 (6): 698–712. https://doi.org/10.1111/puar.12284.
Knox, C. C. 2013. “Analyzing after action reports from Hurricanes Andrew and Katrina: Repeated, modified, and newly created recommendations.” J. Emergency Manage. 11 (2): 160–168. https://doi.org/10.5055/jem.2012.0135.
Knox, C. C. 2017. “A football field lost every 45 minutes: Evaluating local capacity to implement Louisiana’s coastal master plan.” Coastal Manage. 45 (3): 233–252. https://doi.org/10.1080/08920753.2017.1303757.
Knox, C. C., D. Linskey, and J. Tyler. 2021. “Preparing officers to build a culture of preparedness: A case study of the Boston Marathon bombings.” In The role of law enforcement in disasters, edited by M. Landahl and T. Neaves. London: Emerald.
Loftland, J., D. Snow, L. Anderson, and L. H. Loftland. 2006. Analyzing social settings: A guide to qualitative observation and analysis. 4th ed. Belmont, CA: Thomson/Wadsworth.
Lyles, W., and M. Stevens. 2014. “Plan quality evaluation 1994–2012: Growth and contributions, limitations, and new directions.” J. Plann. Educ. Res. 34 (4): 433–450. https://doi.org/10.1177/0739456X14549752.
Peacock, W. G., S. D. Brody, W. A. Seitz, W. J. Merrell, A. Vedlitz, S. Zahran, R. C. Harriss, and R. Stickney. 2010. Advancing resilience of coastal localities: Developing, implementing, and sustaining the use of coastal resilience indicators: A final report, 1–148. College Station, TX: Hazard Reduction and Recovery Center.
Rivera, J. D., and A. E. Nickels. 2014. “Social capital, community resilience, and faith-based organizations in disaster recovery: A case study of Mary Queen of Vietnam Catholic Church.” Risk Hazards Crisis Public Policy 5 (2): 178–211. https://doi.org/10.1002/rhc3.12050.
Stevens, M. R., W. Lyles, and P. R. Berke. 2014. “Measuring and reporting intercoder reliability in plan quality evaluation research.” J. Plann. Educ. Res. 34 (1): 77–93. https://doi.org/10.1177/0739456X13513614.
Tang, Z., M. K. Lindell, C. S. Prater, and S. D. Brody. 2008. “Measuring tsunami planning capacity on US pacific coast.” Nat. Hazard. Rev. 9 (2): 91–100. https://doi.org/10.1061/(ASCE)1527-6988(2008)9:2(91).
United Nations. 2015. “Sendai framework for disaster risk reduction 2015–2030.” Accessed April 14, 2021. https://www.undrr.org/publication/sendai-framework-disaster-risk-reduction-2015-2030.
Woodruff, S. C., S. Meerow, M. Stults, and C. Wilkins. 2018. “Adaptation to resilience planning: Alternative pathways to prepare for climate change.” J. Plann. Educ. Res. 0739456X18801057. https://doi.org/10.1177/0739456X18801057.
Information & Authors
Information
Published In
Natural Hazards Review
Volume 22 • Issue 4 • November 2021
Copyright
© 2021 American Society of Civil Engineers.
History
Received: Dec 11, 2020
Accepted: Jul 27, 2021
Published online: Sep 8, 2021
Published in print: Nov 1, 2021
Discussion open until: Feb 8, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.