The Warning Lexicon: A Multiphased Study to Identify, Design, and Develop Content for Warning Messages
Abstract
Emergency managers and alert and warning originators require tools to build effective warning messages. The purpose of this research is to develop one such tool—a Warning Lexicon—that can be used to construct consistent warning messages for Wireless Emergency Alerts. Specifically, the Warning Lexicon systematically establishes a common set of statements about hazard impacts and their associated recommended protective actions that can be used to quickly write effective warning message contents. Thus, the Warning Lexicon allows practitioners and risk communicators to (1) write effective warning messages at the time of the threat, (2) reduce message issuance delay, and (3) develop templated messages as part of their preparedness process. We built the Warning Lexicon through a theoretically informed, multiphased mixed methods process of content analysis and subject matter expert review to verify the accuracy of lexicon statements and validate the language used to instruct message receivers about protective actions. The resulting content incorporated into the Warning Lexicon includes 48 hazards, their associated impacts, and 112 protective action statements spanning atmospheric, technological, biological, and human-induced events. The result of this project is a comprehensive, theory-based, expert-informed, and practitioner-reviewed resource to support the composition of warning messages for imminent threat events.
Formats available
You can view the full content in the following formats:
Data Availability Statement
Anonymized data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This material is based upon work supported by the Federal Emergency Management Agency Integrated Public Alert and Warning System, or, FEMA-IPAWS (70FA5021C00000016) awarded to Jannette Sutton; we also extend thanks to the anonymous subject matter experts who shared their time and knowledge with us; and we acknowledge the unmatched contributions made to this field by Dr. Dennis Mileit and Dr. John Sorensen, upon whose shoulders this work stands. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of FEMA.
References
Agnew, D. 2021. “As texans endured days in the dark, the state failed to deliver vital emergency information.” Accessed September 30, 2023. https://www.texastribune.org/2021/02/19/texas-emergency-communication-power-outages/.
AT&T. 2022. “WEA capable phones.” Accessed March 15, 2023. https://www.att.com/idpassets/images/support/pdf/WEA-capablePhones.pdf.
Bean, H. 2019. Mobile technology and the transformation of public alert and warning. Santa Barbara, CA: ABC-CLIO.
Bean, H., and A. A. Hasinoff. 2022. “The social functions of idle alerts.” In Social media and crisis communication, edited by Y. Jin and L. Austin, 360–370. New York: Routledge.
Bean, H., B. F. Liu, S. Madden, D. Mileti, J. Sutton, and M. Wood. 2014. Comprehensive testing of imminent threat public messages for mobile devices. College Park, MD: National Consortium for the Study of Terrorism and Responses to Terrorism.
Bean, H., B. F. Liu, S. Madden, J. Sutton, M. M. Wood, and D. S. Mileti. 2016. “Disaster warnings in your pocket: How audiences interpret mobile alerts for an unfamiliar hazard.” J. Contingencies Crisis Manage. 24 (3): 136–147. https://doi.org/10.1111/1468-5973.12108.
Bean, H., J. Sutton, B. F. Liu, S. Madden, M. M. Wood, and D. S. Mileti. 2015. “The study of mobile public warning messages: A research review and agenda.” Rev. Commun. 15 (1): 60–80. https://doi.org/10.1080/15358593.2015.1014402.
Bender, A. 2022. “Wireless emergency alerts; amendment of Part 11 of the commission’s rules regarding the emergency alert system, PS Docket nos. 15-91, 15-94.” Accessed January 23, 2023. https://www.fcc.gov/ecfs/document/10728245959959/1.
Braun, V., and V. Clarke. 2006. “Using thematic analysis in psychology.” Qual. Res. Psychol. 3 (2): 77–101. https://doi.org/10.1191/1478088706qp063oa.
Buyuk, H. F. 2023. “Turkey blocks Twitter after public criticism of quake response.” BalkanInsight. Accessed March 15, 2023. https://balkaninsight.com/2023/02/08/turkey-blocks-twitter-after-public-criticism-of-quake-response/.
Cao, Y., B. J. Boruff, and I. M. McNeill. 2016. “Is a picture worth a thousand words? Evaluating the effectiveness of maps for delivering wildfire warning information.” Int. J. Disaster Risk Reduct. 19 (Oct): 179–196. https://doi.org/10.1016/j.ijdrr.2016.08.012.
CISA (Cytersecurity and Infrastructure Security Agency). 2019. “National emergency communications plan.” Accessed January 23, 2023. https://www.cisa.gov/sites/default/files/publications/19_0924_CISA_ECD-NECP-2019_1_0.pdf.
Creswell, J. W., and J. D. Creswell. 2017. Research design: Qualitative, quantitative, and mixed methods approaches. Thousand Oaks, CA: SAGE.
DHS S&T Directorate (Department of Homeland Security Science and Technology Directorate). 2018. “Report on alerting tactics.” Accessed March 15, 2023. https://www.dhs.gov/sites/default/files/publications/1051_IAS_Report-on-Alerting-Tactics_180807-508.pdf.
Doermann, J. L., E. D. Kuligowski, and J. Milke. 2021. “From social science research to engineering practice: Development of a short message creation tool for wildfire emergencies.” Fire Technol. 57 (Mar): 815–837. https://doi.org/10.1007/s10694-020-01008-7.
FCC (Federal Commuications Commission). 2022. “Communications security, reliability, and interoperability council VIII working group tasks.” Accessed March 15, 2023. https://www.fcc.gov/file/23814/download.
FEMA. n.d. “Ready.” Accessed September 29, 2023. https://www.ready.gov.
FEMA. 2013. “Improvised nuclear device response and recovery: Communicating in the immediate aftermath.” Accessed March 15, 2023. https://www.fema.gov/sites/default/files/documents/fema_improvised-nuclear-device_communicating-aftermath_june-2013.pdf.
FEMA. 2021a. “Improving public messaging for evacuation and shelter in pace. Findings and recommendations for emergency managers from peer-reviewed research.” Accessed July 5, 2023. https://www.fema.gov/sites/default/files/documents/fema_improving-public-messaging-for-evacuation-and-shelter-in-place_literature-review-report.pdf.
FEMA. 2021b. “Protective actions research.” Accessed March 15, 2023. https://community.fema.gov/ProtectiveActions/s/.
FEMA IPAWS (Integrated Public Alert and Warning System). 2021. “Tip 38: Imminent threat vs. public safety.” Accessed March 15, 2023. https://www.fema.gov/sites/default/files/documents/fema_ipaws-tip-38-it-vs-ps.pdf.
FEMA IPAWS (Integrated Public Alert and Warning System). 2022a. “IPAWS program planning toolkit.” Accessed March 15, 2023. https://www.fema.gov/emergency-managers/practitioners/integrated-public-alert-warning-system/public-safety-officials/toolkit.
FEMA IPAWS (Integrated Public Alert and Warning System). 2022b. “Wireless emergency alert capabilities by cellular handset and wireless provider.” Accessed March 15, 2023. https://www.fema.gov/sites/default/files/documents/fema_ipaws-guidance-wea-versions-provider-links.pdf.
FEMA IPAWS (Integrated Public Alert and Warning System). 2023. “IPAWS best practices.” Accessed July 5, 2023. https://www.fema.gov/sites/default/files/documents/fema_ipaws-best-practices-guide.pdf.
FEMA National Integration Center. 2021. “When minutes matter.” Accessed July 5, 2023. https://www.fema.gov/sites/default/files/documents/fema_2021-ipaws-users-conference-nic-technical-assistance-workshops-presentation.pdf.
Fischer, L., D. Huntsman, G. Orton, and J. Sutton. 2023. “You have to send the right message: Examining the influence of protective action guidance on message perception outcomes across prior hazard warning experience to three hazards.” Weather Clim. Soc. 15 (2): 307–326. https://doi.org/10.1175/WCAS-D-22-0092.1.
Frisby, B. N., S. R. Veil, and T. L. Sellnow. 2014. “Instructional messages during health-related crises: Essential content for self-protection.” Health Commun. 29 (4): 347–354. https://doi.org/10.1080/10410236.2012.755604.
Gao, S., and Y. Wang. 2021. “Assessing the impact of geo-targeted warning messages on residents’ evacuation decisions before a hurricane using agent-based modeling.” Nat. Hazards 107 (1): 123–146. https://doi.org/10.1007/s11069-021-04576-1.
Goldman, J. 2011. Words of intelligence: An intelligence professional’s lexicon for domestic and foreign threats. Lanham, MD: Scarecrow Press.
Gray, L., G. Wong-Wylie, G. Rempel, and K. Cook. 2020. “Expanding qualitative research interviewing strategies: Zoom video communications.” Qual. Rep. 25 (5): 1292–1301. https://doi.org/10.46743/2160-3715/2020.4212.
Harrison, J., C. McCoy, K. Bunting-Howarth, H. Sorensen, K. Williams, and C. Ellis. 2014. Evaluation of the national weather service impact-based warning tool. Washington, DC: National Oceanic and Atmospheric Administration.
Harrison, S. E., S. H. Potter, R. Prasanna, E. E. H. Doyle, and D. Johnston. 2021. “‘Where oh where is the data?’: Identifying data sources for hydrometeorological impact forecasts and warnings in Aotearoa New Zealand.” Int. J. Disaster Risk Reduct. 66 (Dec): 102619. https://doi.org/10.1016/j.ijdrr.2021.102619.
Hodas, N. O., G. Ver Steeg, J. Harrison, S. Chikkagoudar, E. Bell, and C. D. Corley. 2015. “Disentangling the lexicons of disaster response in Twitter.” In Proc., 24th Int. Conf. on World Wide Web, 1201–1204. New York: Association for Computing Machinery.
Kox, T., and A. H. Thieken. 2017. “To act or not to act? Factors influencing the general public’s decision about whether to take protective action against severe weather.” Weather Clim. Soc. 9 (2): 299–315. https://doi.org/10.1175/WCAS-D-15-0078.1.
Kuligowski, E. D., N. A. Waugh, J. Sutton, and T. J. Cova. 2023. “Ember alerts: Assessing wireless emergency alert messages in wildfires using the warning response model.” Nat. Hazards Rev. 24 (2): 04023009. https://doi.org/10.1061/NHREFO.NHENG-1724.
Linardos, V., M. Drakaki, P. Tzionas, and Y. Karnavas. 2022. “Machine learning in disaster management: Recent developments in methods and applications.” Mach. Learn. Knowl. Extr. 4 (2): 446–473. https://doi.org/10.3390/make4020020.
Lindell, M. K., and R. W. Perry. 2012. “The protective action decision model: Theoretical modifications and additional evidence.” Risk Anal. 32 (4): 616–632. https://doi.org/10.1111/j.1539-6924.2011.01647.x.
Margolis, J. 2020. “Confused about curfew? One of those messages was meant for Glendale.” LAist. Accessed October 25, 2023. https://laist.com/news/curfew-confusion-los-angeles-glendale-mistake-emergency-wireless-alerts.
McCrindle, M., and E. Wolfinger. 2011. Word up: A lexicon and guide to communication in the 21st century. Braddon, ACT, Australia: Halsted Press.
McGregor, J., J. P. Elm, E. T. Stark, J. Lavan, R. Creel, C. Alberts, C. Woody, R. J. Ellison, and T. Marshall-Keim. 2014. Best practices in wireless emergency alerts. Pittsburgh: Software Engineering Institute.
McKinley, C. 2022. “Marshall fire exposes holes in emergency notification systems.” The Gazette. Accessed July 5, 2023. https://gazette.com/premium/marshall-fire-exposes-holes-in-emergency-notification-systems/article_20114fdc-7594-11ec-ab04-3b0890a75587.html.
Merriam-Webster. 2023. “Lexicon.” Accessed March 15, 2023. https://www.merriam-webster.com/dictionary/lexicon.
Mileti, D. 2018. PrepTalks: Modernizing public warning messaging. Washington, DC: FEMA.
Mileti, D., M. Wood, H. Bean, F. Lui Brooke, J. Sutton, and S. Madden. 2012. Workshop briefing white paper: Comprehensive testing of imminent threat public messages for mobile devices. College Park, MD: National Consortium for the Study of Terrorism and Responses to Terrorism.
Mileti, D. S., and J. H. Sorensen. 1990. Communication of emergency public warnings: A social science perspective and state-of-the-art assessment. Oak Ridge, TN: Oak Ridge National Laboratory.
Minson, S. E., E. S. Cochran, J. K. Saunders, S. K. McBride, S. Wu, A. S. Baltay, and K. R. Milner. 2022. “What to expect when you are expecting earthquake early warning.” Geophys. J. Int. 231 (2): 1386–1403. https://doi.org/10.1093/gji/ggac246.
Munns, W. R., A. W. Rea, M. J. Mazzotta, L. A. Wainger, and K. Saterson. 2015. “Toward a standard lexicon for ecosystem services.” Integr. Environ. Assess. Manage. 11 (4): 666–673. https://doi.org/10.1002/ieam.1631.
Nagourney, A., D. E. Sanger, and J. Barr. 2018. “Hawaii panics after alert about incoming missile is sent in error.” NY Times. Accessed September 30, 2023. https://www.nytimes.com/2018/01/13/us/hawaii-missile.html.
NBC Los Angeles. 2021. “LA citywide alert issued to remind people to get the COVID-19 vaccine.” Accessed March 15, 2023. https://www.nbclosangeles.com/news/local/la-to-send-wireless-emergency-alert-to-cell-phones-to-urge-vaccinations/2587155/.
NRC (National Research Council). 2013. Public response to alerts and warnings using social media: Report of a workshop on current knowledge and research gaps, Washington, DC: National Academies Press.
NWS (National Weather Service). n.d. “Wireless emergency alerts (360 characters).” Accessed July 5, 2023. https://www.weather.gov/wrn/wea360.
Oakland, T., and H. B. Lane. 2004. “Language, reading, and readability formulas: Implications for developing and adapting tests.” Int. J. Test. 4 (3): 239–252. https://doi.org/10.1207/s15327574ijt0403_3.
Office of the Federal Register, National Archives and Records Administration. 2010. “Public law 111–274—Plain writing act of 2010.” US Government Printing Office. Accessed October 13, 2010. https://www.govinfo.gov/content/pkg/PLAW-111publ274/pdf/PLAW-111publ274.pdf.
Olteanu, A., C. Castillo, F. Diaz, and S. Vieweg. 2014. “CrisisLex: A lexicon for collecting and filtering microblogged communications in crises.” In Proc., Int. AAAI Conf. on Web and Social Media. Cambridge, MA: Association for the Advancement of Artificial Intelligence Press.
Plotnick, L., and S. R. Hiltz. 2018. “Software innovations to support the use of social media by emergency managers.” Int. J. Hum.-Comput. Interact. 34 (4): 367–381. https://doi.org/10.1080/10447318.2018.1427825.
Potter, S. H., P. V. Kreft, P. Milojev, C. Noble, B. Montz, A. Dhellemmes, R. J. Woods, and S. Gauden-Ing. 2018. “The influence of impact-based severe weather warnings on risk perceptions and intended protective actions.” Int. J. Disaster Risk Reduct. 30 (Sep): 34–43. https://doi.org/10.1016/j.ijdrr.2018.03.031.
Quad Cities. 2021. “Illinois sends emergency alert asking for healthcare workers to join COVID-19 fight.” Accessed March 15, 2023. https://www.ourquadcities.com/news/local-news/illinois-sends-emergency-alert-asking-for-healthcare-workers-to-join-covid-19-fight/.
Readability Formulas. n.d. “What are readability formulas and how do they help me?” Accessed March 15, 2023. https://www.readabilityformulas.com/.
Salafasky, N., et al. 2008. “A standard lexicon for biodiversity conservation: Unified classifications of threats and actions.” Conserv. Biol. 22 (4): 897–911. https://doi.org/10.1111/j.1523-1739.2008.00937.x.
Schmidt, E. 2022. “Reading the numbers: 130 million American adults have low literacy skills, but funding differs drastically by state.” Accessed March 15, 2023. https://www.apmresearchlab.org/10x-adult-literacy.
Schneider, D. 2022. “Green Bay residents encounter confusion after ‘emergency alert’ was sent out cellphones across city.” Green Bay Press Gazette. Accessed July 5, 2023. https://www.greenbaypressgazette.com/story/news/local/2022/12/01/green-bay-residents-encounter-confusion-after-alert-was-sent/69690451007/.
Sharon, A. J., and A. Baram-Tsabari. 2014. “Measuring mumbo jumbo: A preliminary quantification of the use of jargon in science communication.” Public Understanding Sci. 23 (5): 528–546. https://doi.org/10.1177/0963662512469916.
Shulman, H. C., and O. M. Bullock. 2020. “Don’t dumb it down: The effects of jargon in COVID-19 crisis communication.” PLoS One 15 (10): e0239524. https://doi.org/10.1371/journal.pone.0239524.
Sivle, A. D., and T. Aamodt. 2019. “A dialogue-based weather forecast: Adapting language to end-users to improve communication.” Weather 74 (12): 436–441. https://doi.org/10.1002/wea.3439.
Statcounter GlobalStats. 2023. “Mobile vendor market share in United States of America.” Accessed March 15, 2023. https://gs.statcounter.com/vendor-market-share/mobile/united-states-of-america.
Sutton, J., L. Fischer, L. E. James, and S. E. Sheff. 2020. “Earthquake early warning message testing: Visual attention, behavioral responses, and message perceptions.” Int. J. Disaster Risk Reduct. 49 (Oct): 101664. https://doi.org/10.1016/j.ijdrr.2020.101664.
Sutton, J., L. Fischer, and M. M. Wood. 2021. “Tornado warning guidance and graphics: Implications of the inclusion of protective action information on perceptions and efficacy.” Weather Clim. Soc. 13 (4): 1003–1014. https://doi.org/10.1175/WCAS-D-21-0097.1.
Sutton, J., and L. M. Fischer. 2021. “Understanding visual risk communication messages: An analysis of visual attention allocation and think-aloud responses to tornado graphics.” Weather Clim. Soc. 13 (1): 173–188. https://doi.org/10.1175/WCAS-D-20-0042.1.
Sutton, J., and E. D. Kuligowski. 2019. “Alerts and warnings on short messaging channels: Guidance from an expert panel process.” Nat. Hazards Rev. 20 (2): 04019002. https://doi.org/10.1061/(ASCE)NH.1527-6996.0000324.
Sutton, J., S. C. Vos, M. M. Wood, and M. Turner. 2018. “Designing effective tsunami messages: Examining the role of short messages and fear in warning response.” Weather Clim. Soc. 10 (1): 75–87. https://doi.org/10.1175/WCAS-D-17-0032.1.
Sutton, J., M. Wood, D. Huntsman, N. Waugh, and S. Crouch. 2023. “Communicating hazard location in earthquake early warnings.” Nat. Hazards Rev. 24 (4). https://doi.org/10.1061/NHREFO.NHENG-1723.
Temnikova, I., A. Varga, and D. Biyikli. 2014. “Building a crisis management term resource for social media: The case of floods and protests.” In Proc., 9th Int. Conf. on Language Resources and Evaluation (LREC’14), 740–747. Reykjavik, Iceland: European Language Resources Association (ELRA).
Trujillo-Falcón, J. E., O. Bermúdez, K. Negrón-Hernández, J. Lipski, E. Leitman, and K. Berry. 2021. “Hazardous weather communication en Español: Challenges, current resources, and future practices.” Bull. Am. Meteorol. Soc. 102 (4): E765–E773. https://doi.org/10.1175/BAMS-D-20-0249.1.
Tutten, J. 2023. “Officials apologize after ‘emergency alert’ test sent in ‘error’ to cell phones at 4:45 a.m.” WFTV9. Accessed July 5, 2023. https://www.wftv.com/news/local/floridians-outraged-after-alarm-during-emergency-alert-test-sent-445-am/65R2O6EDT5HTRCDST3NDRYHE4Q/.
USACE. 2019. “Public Warning for dam and levee failure.” Accessed July 5, 2023. https://www.publications.usace.army.mil/Portals/76/Users/182/86/2486/EP%201110-2-17.pdf?ver=2019-06-20-152050-550.
US Department of Justice. 2019. “Amber alert statistics.” Accessed March 15, 2023. https://amberalert.ojp.gov/statistics#:∼:text=As%20of%20January%202%2C%202023,plans%20throughout%20the%20United%20States.
US Social Security Administration. 2019. “What is the Flesch-Kincaid readability test.” Accessed March 15, 2023. https://secure.ssa.gov/poms.nsf/lnx/0910605105.
Verizon. n.d. “Wireless emergency alert compatible devices.” Accessed March 15, 2023. https://www.verizon.com/support/wireless-emergency-alerts-compatible-devices/.
Whitcomb, D. 2018. “After disaster alert failures, US moves toward national system.” Reuters. Accessed July 5, 2023. https://www.reuters.com/article/us-usa-disaster-alerts/after-disaster-alert-failures-u-s-moves-toward-national-system-idUSKBN1KU1C1.
Williams, C. A., and G. M. Eosco. 2021. “Is a consistent message achievable? Defining ‘message consistency’ for weather enterprise researchers and practitioners.” Bull. Am. Meteorol. Soc. 102 (2): E279–E295. https://doi.org/10.1175/BAMS-D-18-0250.1.
Wood, M., H. Bean, B. Liu, and M. Boyd. 2015. Comprehensive testing of imminent threat public messages for mobile devices: Updated findings. College Park, MD: National Consortium for the Study of Terrorism and Responses to Terrorism.
Wood, M., D. S. Mileti, H. Bean, B. F. Liu, J. Sutton, and S. Madden. 2018. “Milling and public warnings.” Environ. Behav. 50 (5): 535–566. https://doi.org/10.1177/0013916517709561.
Wood, M. M. 2017. “Geotargeted alerts and warnings.” In International encyclopedia of geography: People, the earth, environment and technology, 1–11. Oxford, UK: Wiley.
Woodworth, W. 2020. “‘Completely botched’: Failed emergency alerts raise questions for future disasters.” Accessed September 30, 2023. https://www.statesmanjournal.com/story/news/2020/09/28/failed-oregon-wildfire-emergency-evacuation-alerts-questions-future-disasters-earthquake/5792075002/.
Wylie Communications. n.d. “Measure reading levels with readability indexes: How to calculate readability with the Flesch Reading Ease, Flesch-Kincaid and Gunning Fog tests.” Accessed March 15, 2023. https://www.wyliecomm.com/2021/11/measure-reading-levels-with-readability-indexes/.
Information & Authors
Information
Published In
Natural Hazards Review
Volume 25 • Issue 1 • February 2024
Copyright
This work is made available under the terms of the Creative Commons Attribution 4.0 International license, https://creativecommons.org/licenses/by/4.0/.
History
Received: Mar 23, 2023
Accepted: Aug 22, 2023
Published online: Oct 30, 2023
Published in print: Feb 1, 2024
Discussion open until: Mar 30, 2024
ASCE Technical Topics:
- Business management
- Communication systems
- Disaster preparedness
- Disaster risk management
- Disaster warning systems
- Emergency management
- Engineering fundamentals
- Infrastructure
- Lifeline systems
- Managers
- Methodology (by type)
- Personnel (type)
- Personnel management
- Practice and Profession
- Research and development
- Research methods (by type)
- Validation
- Verification
- Wireless technologies
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Jeannette Sutton, Hamilton Bean, Lori Peek, Erica Kuligowski, Michele Wood, The Legacy of Dennis S. Mileti and the Future of Public Alert and Warning Research, Natural Hazards Review, 10.1061/NHREFO.NHENG-2113, 25, 3, (2024).