The Warning Lexicon: A Multiphased Study to Identify, Design, and Develop Content for Warning Messages

A lexicon is generally defined as a set of vocabulary related to a specific language, subject matter, and/or specific group of people (Merriam-Webster 2023). They are used to organize terms and concepts into a common language and provide definitions for words used across knowledge domains. A lexicon that is agreed upon by its users can help to standardize terminology and definitions and facilitate mutual understanding and communication among disparate groups of researchers, practitioners, policy makers, and the public (see, for example, Goldman 2011; McCrindle and Wolfinger 2011; Plotnick and Hiltz 2018).

Because the language used to describe technology and interventions can differ by domain and expertise, creating a lexicon enables a shared language that can, over time, become familiar to practitioners and the public alike. For example, Salafasky (2008) developed a standard lexicon related to biodiversity conservation that enables scientists and policy makers across multiple disciplines to communicate using shared definitions for their common vocabulary. Following this effort, the US Environmental Protection Agency (EPA) developed a standard lexicon for ecosystem services through a process of consensus building and iterative discourse among partners (Munns et al. 2015). The objective of the EPA lexicon is to facilitate mutual understanding of terminology used by multiple disciplines and foster transdisciplinary collaboration among scientists (Munns et al. 2015).

Within the space of disaster communication, lexicons have primarily been developed for the purpose of automatically detecting message content posted to social media during disasters (Plotnick and Hiltz 2018). Through a combination of machine learning and content analysis, researchers have identified vocabulary that is specific to crisis management (Temnikova et al. 2014), individual hazards (Olteanu et al. 2014), and disaster response (Hodas et al. 2015). By using a lexicon to quickly identify what members of the public post about online, public officials can direct resources to assist with rescue and recovery actions (see, for example, Buyuk 2023). This ability to detect public needs from large and expanding data sets in real time remains a priority for disaster responders (Linardos et al. 2022).

However, little attention has been devoted to building a lexicon that is useful for warning message creation, specifically. The ability to clearly communicate to the public about an imminent threat is critical for reducing the loss of life and injury. Public officials and responders depend on precision and clarity of warning language but lack access to a common vocabulary that can be used to consistently convey information about impending threats and the actions the public should take to reduce their impacts (see Hodas et al. 2015; Olteanu et al. 2014). For risk communicators, a warning lexicon is a valuable tool that can be used to write evidence-based messages with greater speed, confidence, and effectiveness.

Specifically, effective warning messages must include the hazard, its impacts, and protective actions. Thus, a warning lexicon must also include these types of information, resulting in a common set of actionable statements and impacts for a variety of hazards. We further discuss effective warning message design below.

Mileti and Sorensen (1990) document five key types of warning message content that motivate people to take timely and appropriate protective action in response to a warning message, which is referred to as the WRM. These contents include the following:

Decades of research investigating effective warnings has demonstrated the importance of including these contents to warn about imminent hazards, such as earthquake (Sutton et al. 2020), tsunami (Sutton et al. 2018), tornado (Sutton et al. 2021), dust-storm, and snow squall (Fischer et al. 2023), especially for hazards that are unfamiliar to message receivers (Wood et al. 2018).

Mileti (2018) later stated that warning messages need to also include statements describing a hazard’s potential impacts and consequences to populations as well as how protective actions can reduce those consequences. Described by scholars as “impact-based warnings” (Potter et al. 2018), these messages are intended to increase one’s perceived severity of the hazard (Harrison et al. 2014).

The WRM also emphasized the importance of message style—or the way content is communicated. Warning messages should be clear, without using ambiguous or unfamiliar terms (Sharon and Baram-Tsabari 2014; Sivle and Aamodt 2019), as well as certain and complete. They should also be internally consistent, lacking conflicting information (Williams and Eosco 2021), and specific (Sutton et al. 2023). Messages that are clear can also result in increased knowledge and feelings of self-efficacy, and decrease protective action delay (Fischer et al. 2023; Frisby et al. 2014; Wood et al. 2018). Message style can also increase a sense of urgency by using an imperative voice and exclamation marks or by calling attention to content in ALL CAPS (Sutton and Fischer 2021).

Within the WRM research record, responses to messages have largely been measured as perceptual outcomes, which are focused on acquiring new information from the alert or warning message. WRM scholars have focused on perceptions of message understanding, message believing, and message personalization by measuring the extent to which a message receiver agrees or disagrees with statements in a message to which they are exposed (see, for example, Fischer et al. 2023). WRM scholars also measure the extent to which a message receiver believes that they will be able to make a quick, effective decision about protective action response based upon the message contents (see, for example, Wood et al. 2017). More recently, WRM scholars have inquired about perceptions of self- and response-efficacy that affect a message receiver’s perceived ability to do the recommended action, which can also affect their potential decision-making (see, for example, Sutton et al. 2021). Notably, the focus within the warning literature has been on measuring perceptions rather than behavioral intention or behavioral change.

There are key limitations to assessing behavioral intention in warning message design research. First, this type of research commonly uses hypothetical scenarios, which cannot reflect the environmental or social cues people also use to make protective action decisions (Lindell and Perry 2012) and therefore will not faithfully represent actual responses to a warning message (NRC 2013). Second, people tend to overestimate their intention to perform protective actions when actual consequences are not present (Kox and Thieken 2017). Third, behavioral intention cannot capture barriers that may exist during actual decision making. Overall, the context in which an individual actually receives an alert or warning message cannot be reliably replicated, making it difficult to clearly assess whether an individual message will result in a behavioral outcome (FEMA 2021a).

Importantly, the content and style in which a warning is delivered increases in importance when it is sent over short messaging channels. Because of the limited character counts restricting the message writer’s ability to communicate details, effectively capturing the key contents, structured in a standardized manner, becomes vital in time-limited contexts. Wireless Emergency Alerts (WEAs) represent a national solution to delivering imminent threat messages to all persons through the United States. We turn to a discussion of WEAs next.

WEAs are the one channel in the United States that has the greatest potential to reach persons at risk of an imminent threat, without requiring them to sign up, opt in, or turn on a device to receive a warning message. By using the Integrated Public Alert and Warning System (IPAWS), risk communicators at the local, state, and national levels can send geotargeted, text-like messages to WEA-enabled mobile devices (Bean 2019). WEAs can alert persons of imminent threat using text, audio signal, and vibration for hazards of all types, ranging from short fuse events such as earthquakes via the USGS ShakeAlert service (Minson et al. 2022) to longer fuse threats such as hurricanes via the National Weather Service (Gao and Wang 2021). The WEA system is also used to deliver AMBER, presidential, and public safety alert messages.

Initially, WEAs were limited to 90 characters of text (known as WEA 1.0). In 2019, WEA messages were expanded by four times the number of characters to 360 characters of text (known as WEA 2.0). Since 2019, all the mobile devices built by the top five brands—or Apple, Samsung, Motorola, Google, and LG—are WEA 2.0 enabled, which represents more than 95% of all mobile devices sold in the US (AT&T 2022; Statcounter GlobalStats 2023). WEA 2.0 enabled devices, and some 1.0 devices (Verizon, n.d.), also facilitate the inclusion of a URL in messaging, allowing risk communicators to add a link for more complete information (see Kuligowski et al. 2023).

The design, dissemination, and conditions leading to the of delivery of WEA messages highlight the complexities of communicating warnings effectively. The character limitations pose a challenge for message writers to construct a terse message under heightened conditions of stress (Bean 2019). Researchers have also found that short messages, especially WEAs limited to 90 characters, tend to be incomplete and lack specificity; therefore, these types of messages are less actionable (Wood et al. 2018). Expanding WEA messages to 360 characters allows message writers to increase details about the hazard, impacts, and recommended guidance necessary to act. Risk communicators who have access to relevant content to design an effective warning message can reduce issuance delay and increase the likelihood of message receivers taking action.

The five message contents included in the WRM, presented in an order optimized for the message receiver (source, hazard, location, guidance, time; Bean et al. 2014) served as the initial foundation for WEA messages in the United States (Bean 2019). Indeed, this standardized structure was found to optimize message receiver outcomes (Bean et al. 2015), leading to greater understanding, believing, personalizing, and ability to decide in comparison with other message content ordering. As a result, this standardized message structure has been adopted and included in Federal trainings (FEMA National Integration Center 2021) and guidance documents (USACE 2019; CISA 2019; FEMA IPAWS 2023) for all warning communications issued via WEA and other short message channels. The structure should not differ due to the type of hazard, the location of the threat, or the population receiving the message. However, there is not yet a lexicon—or a consistent set of evidence-based warning message contents—that can be used to quickly communicate hazard impacts and their recommended actions.

In 2012, researchers carried out initial efforts to define and limit a set of protective actions that could be reduced to short phrases suitable for 90-character WEA 1.0 messages (Mileti et al. 2012). This research led to the creation of a Hazard Versus Protective Actions Message Matrix (aka “The Hazard Matrix”) that organized a set of natural, technological, and human-induced hazards ($n=59$) and their associated protective actions ($n=27$) into a grid, whereby action statements were sequentially numbered representing the initial, secondary, and tertiary warning message content, plus an “all clear” (see Fig. 1).

The Hazard Matrix represented an innovative and important contribution at the time it was developed; however, it is limited in scope and only includes a terse set of actions that contains no instruction or elaboration on each protective action recommendation. It provides only the name of each hazard, and lacks information about the hazard itself as well as potential impacts or consequences to populations. We use the Hazard Matrix as an initial starting point for the Warning Lexicon and build upon it by describing each hazard’s impacts and broadening the protective action categories into complete and specific action statements. We describe the method used to develop this more comprehensive Warning Lexicon next.

In this first phase of research, we deconstructed English language warning message templates ($n=470$) from eight metropolitan areas located across the United States. These templates were used as the starting point for developing Lexicon contents. The hazards included in this template data set far exceeded the number and type of hazards contained in the original Hazard Matrix, allowing the research team to draw from a wide breadth of message examples for message deconstruction. Each message was deconstructed into content categories representing the hazard, hazard impact, and protective action statements that would later be verified, vetted, and potentially changed; thus, we were not concerned with any prior assessment of their effects on WRM outcomes or the location from which they originated. For these templates, we focused on the warning messages that had been designed for short messaging channels, including social media (140–280 characters in length), opt-in messaging systems and Short Message Service (SMS; 160 characters), and WEAs (90 or 360-characters).

Deconstructed messages were organized by hazard type using Excel spreadsheets to show variations on language describing each hazard, hazard impact, and protective action statements (see example in Table 1). Messages were manually deconstructed by the third author in consultation with the first two authors of this manuscript. This step resulted in a comprehensive list of protective actions statements ($n=372$) across 91 hazards. We discuss next how we selected the list of hazards included in the Warning Lexicon.

After collecting and developing the initial Warning Lexicon protective action and hazard impact statements, hazard subject matter expert interviews were conducted to verify their accuracy. Fifteen hazard subject matter experts (SMEs) were identified through professional contacts with researchers and practitioners in local, state, and federal agencies, as well as recommendations from other hazard experts. Hazard SME participants included both practitioners ($n=10$) and researchers ($n=5$) with 12 to 29 years of service in their professional roles. All hazard SMEs had degrees, ranging from a bachelor’s degree to doctorate, in their relevant areas of practice or research, including atmospheric science or meteorology ($n=8$), emergency management/public administration ($n=1$), geosciences ($n=2$), criminal justice ($n=2$), and engineering and applied mechanics ($n=2$). Hazard SMEs work professionally as program coordinators and trainers ($n=9$), researchers ($n=7$), practitioners ($n=5$), and lab directors ($n=1$). Hazard SMEs were invited via email to participate in a virtual interview for the purpose of verifying the accuracy of and providing changes to Warning Lexicon contents. Hazard SMEs had expertise in each of the 48 hazards contained in the Warning Lexicon; in many cases a single SME had expertise with more than one hazard type (e.g., severe storm, tornado, hail, and lightning; hurricane and storm surge; volcano and earthquake).

Hazard SME interviews were conducted via Zoom, using guided discussion methods with open-ended response (Gray et al. 2020). The facilitator began with a short presentation summarizing the objectives and goals for developing the Warning Lexicon. Following this introduction, participants were told that they would be reviewing hazard impact and protective action statements for the hazards in which they had subject matter expertise.

First, the facilitator made their computer screen visible to the SME and presented a list of protective action statements associated with the hazard, which had been compiled in Phase 1 of the project. They were then asked to consider all possible protective actions that might be recommended for a single hazard type, rather than focusing narrowly on specific instances or conditions in which a single incident might occur. SMEs were asked to think about (1) protective action recommendations that should be made when an emerging hazardous condition would likely result in an impact on life and property; and (2) recommendations that should be made when the hazardous event is occurring or has already occurred, resulting in impacts that were verifiable. Each protective action statement was reviewed and confirmed by the SME as an appropriate protective action for a particular hazard. In some cases, the SME made suggestions for changes to the language describing the recommendations, and in other cases additional actions were recommended. In a few cases, action statements that had been included were removed. The facilitator typed these changes directly into the review document that was shared on the computer conferencing screen.

After the protective action statements were reviewed for a single hazard, the SME assessed the associated hazard impact statement developed in Phase 1 of the research. The SME was invited to review each statement, comment upon its accuracy, and, in cases where a statement was perceived to be incomplete or inaccurate, to offer alternative language. Changes were made directly on the shared review document to ensure that language was accurately captured.

At the conclusion of the hazard impact statement review, the SME cross validated the linkages between the impact and the protective action statements. In some cases, additional protective actions were warranted due to changes in the hazard impact statement; those new contents were added. In cases where the protective action list included actions that were considered vague, imprecise, or unwarranted relative to the hazard impact statement, they were eliminated.

Some hazards and protective actions in the Warning Lexicon were not confirmed directly with SMEs who were interviewed due to their perceived concerns about how the information might be used by emergency management alerting authorities in the future. For example, a defined set of WEAs is issued for hazards over which the National Weather Service has jurisdiction. These include dust storm, wind, flash flood, hurricane/typhoon, severe thunderstorm, snow squall, storm surge, tornado, and tsunami (NWS, n.d.). Concerns were expressed by some SMEs about the potential for overalerting, should an AA choose to replicate an alert previously issued. In other cases, SMEs felt a particular hazard was not appropriate to be communicated via WEA. However, there is no consistent national policy dictating “appropriate” WEA use, nor has there been a definitive, large scale study to determine the extent to which alerting via WEA is viewed as problematic by the public. These hazards for which concern was expressed include blizzard, ice storm, heavy snow, extreme cold, winter storm, snow squall, air quality, and infectious disease. In these cases, the protective action and impact statements were validated with pre-existing resources (see Appendix S2).

After the protective actions were validated by SMEs for each hazard, the final list of protective actions was cross-checked using the materials located on the FEMA (2021b) Protective Action Research website. This site presents research-based protective action recommendations for 15 hazards and served as an additional confirmation that the hazard, impact statements, and recommended actions aligned with Federal emergency management guidance documents.

At the conclusion of Phase 2, message contents had been verified by 15 hazard subject matter experts for 48 hazards resulting in 93 hazard impact statements, reflecting event timing, and 112 protective action statements. (See Supplemental Materials, S1, for the complete list.)

In Phase 3, we consulted communication and warning message design experts who provided feedback on the language used in each statement via focus group and written communication. Phase 3 experts included a medical anthropologist with expertise in infectious disease and public health, and technological and radiological hazards; a public health expert with expertise on public health, geological, and atmospheric hazards; and three social scientists with expertise in wildfire and a range of hydrological, geological, and atmospheric hazards. SMEs were asked to review hazard impact and protective action statements with a focus on message simplicity, eliminating the use of “jargon” (i.e., words and phrases used to convey technical information among scientists, which are not well understood by the public; Shulman and Bullock 2020), and identifying unintended consequences from specific message phrases (e.g., message content that may stigmatize certain populations).

An initial group meeting was held via online teleconference (i.e., Zoom). First, the goals and objectives of the Warning Lexicon project were shared by the facilitator. This introduction was followed by instructions on how the group would contribute to the review process, beginning with a review of protective action statements that applied to most of the hazards. The facilitator shared their screen and manually entered recommended changes into each statement. A key example highlighting this review is for one protective action recommendation associated with infectious disease. The original action statement included, “stay away from sick people.” During expert review, one participant noted that the language used in this statement could be stigmatizing for some populations, leading to unintended consequences that would later require additional communication interventions. They also noted the difficultly associated with discerning whether someone is indeed sick when out in public. Thus, this protective action statement was then eliminated.

Group concurrence was obtained for every protective action statement included in the Warning Lexicon. A portion of hazard impact statements was also jointly reviewed. SMEs independently completed inspection and made edits to any impact statements that were not discussed in the group call. Revised hazard impact statements were shared among the SMEs before the research team incorporated them into the final Warning Lexicon.

To utilize the Warning Lexicon, risk communicators are encouraged to draw from research guidance on designing an effective warning message (see Doermann et al. 2021; Sutton and Kuligowski 2019). As described previously, messages that include (1) the hazard and its impacts, (2) the location of the hazard, (3) protective action guidance, (4) the message source, and (5) the time to act or time the message expires have been found to increase the likelihood that individuals will take protective action (Mileti and Sorensen 1990; Sutton and Kuligowski 2019; Wood 2017). Placing an official, recognizable, and familiar source at the beginning of the message increases message receivers’ understanding and belief (Bean et al. 2015). Using common words that are unambiguous and require no interpretation to instruct message receivers on the actions they should take will increase understanding and perceptions of self-efficacy (Sutton et al. 2021; Wood et al. 2015). Using language that names the potential impacts or consequences of the hazard and accurately depicts the severity and certainty of the event increases perceptions of risk and personalization (Potter et al. 2018). Identifying the hazard location by towns, cities, counties, major landmarks, and roads that are known to most individuals will increase threat understanding and personalization (Cao et al. 2016). And clearly identifying the timing of the hazardous event, such as when it starts and stops, by when actions should be taken, or when updates will be provided, can reduce delayed action (Bean et al. 2014).

Message style conventions (e.g., the use of ALL CAPS and imperative sentences) can increase attention and perceptions of urgency (Sutton and Kuligowski 2019). Urgency can be further communicated with the inclusion of timing information, such as the words NOW or IMMEDIATELY. However, the use of acronyms, abbreviations, and technical jargon can decrease understanding and delay action (Bean et al. 2016). Furthermore, researchers have found that the order of contents affects message understanding, belief, and decision-making (Wood et al. 2015). For messages between 90 and 280 characters long, contents should be presented as follows: source, hazard, location, time, guidance. For messages reaching 360 characters, the following order is conventional: source, hazard, location, guidance, time (Bean et al. 2015). Frequently, messages will have some integration of content across categories. For example, location may be integrated into the description of the hazard and guidance information. Additionally, time is frequently integrated into guidance statements as part of the instruction, such as by the use of the words “now,” “immediately,” or “until further notice” (see example messages in Sutton and Kuligowski 2019).

Preparing warning message contents prior to an imminent threat event will reduce the time and effort required to write a complete message under conditions of heightened stress and uncertainty. As part of preparedness efforts, risk communicators can (1) identify a list of threats and hazards that are likely to occur within their community; (2) develop a list of trusted and familiar organizations or agencies to be included as the message source; (3) prepare a list of well-known landmarks, town/city/county names, and major roads or intersections to describe the location of a threat; and (4) establish a webpage or social media URL on which updates can be posted (see FEMA IPAWS 2022a). Message writers also can identify, download, and practice writing character-constrained warnings using existing software, such as hemingwayapp.com, or Grammarly.com, or build their own character counters using Excel or Google sheets.

Additionally, the Warning Lexicon can be used to develop an imminent threat message at the time of a hazardous event. However, message templates for known hazards written prior to an event will require minimal editing to include the specific impacts, location of the hazard, and time to act or to receive an update (see Supplemental Materials, S1, for example message templates for each hazard). Because WEA messages are delivered to all WEA-enabled mobile devices located within a geographically defined area, the language included in the Warning Lexicon is intended to be applicable to the broader, English-speaking public. Unlike targeted messages that can be delivered to individuals who opt in to specific messaging channels or groups, WEA messages are not tailored to the characteristics of individual populations. Therefore, the Warning Lexicon is useful for plain language messaging, using the empirically supported standardized format, to reach all populations at risk.

The following prompts adapted from Doermann et al. (2021) should inform the design of any warning message, regardless of the time at which it is written (pre, during, or post-initiation), and regardless of the short-messaging channel for which it is written (WEA, SMS, social media, etc.). Content from the Warning Lexicon can be used to address prompts about the hazard, hazard impacts, and protective actions; risk communicators should be prepared to populate contents about the source, location, time, and additional information.

The Warning Lexicon currently includes 48 hazards, which vary by frequency of occurrence. Hazards that are seasonal, such as tornado, hurricane, or wildfire, are routinely communicated to populations at risk, suggesting greater content familiarity by AAs. Hazards that occur less frequently, such as earthquake or tsunami, may result in greater reliance upon the Warning Lexicon by AAs. In both cases, AAs will benefit from reviewing the Warning Lexicon during the preparedness and planning phases of the hazard lifecycle, and to develop messaging templates early. There is also a high likelihood that new or emerging threats have not been captured and will require risk communicators to adapt these contents to additional contexts, such as complex events such as tornado and flash flood hazards that co-occur. The use of WEA for messaging is also likely to continue to adapt to new uses, as seen in 2019 when FEMA expanded its recommendations to include public safety messages unrelated to, but coinciding with, the start of the COVID-19 pandemic. Notably, WEA was used to update people on vaccine clinics (NBC Los Angeles 2021), curfews (Margolis 2020), and the need for medical personnel (Quad Cities 2021) to name only a few novel uses. Researchers have not yet conducted empirical studies to support message design recommendations for these nonimminent threats sent via WEA.

Furthermore, the technological affordances of WEA are also likely to change in the future (FEMA IPAWS 2022b). Specifically, WEA 1.0 supported 90-character messages, WEA 2.0 supported 360 characters, Spanish language messages, and URLs, and WEA 3.0, representing more than 60% of active smartphones, supports narrow geotargeting through device based geofencing (Bender 2022). Next, WEA 4.0 is likely to feature new capabilities such as those proposed by the Federal Communications Commission (FCC) to “identify the software or functional requirements necessary to allow WEA software to pull capabilities from other mobile device applications and firmware to enhance the presentation of WEA alert messages” (FCC 2022). These capabilities could also affect the ability to increase the number of languages in which WEAs are delivered. However, the design of messages in languages beyond English will remain constrained due to human resources; most emergency management agencies do not have in-house translators. At this time, the Warning Lexicon is limited to text only, in English, and does not include evidence-based guidance on icons, maps, or other graphics.

Similarly, because the Warning Lexicon is focused primarily on hazards requiring protective action response, it does not include things like missing persons, such as AMBER alerts. Although notifications for these types of events may help to raise public awareness and enlist their help (US Department of Justice 2019), they do not require message recipients to act due to an immediate safety threat (Bean and Hasinoff 2022). Furthermore, some hazards, such as water contamination and nuclear attack, have messaging that is informed by legal statutes for language that cannot be adapted. They are, therefore, not included in the Warning Lexicon.

Additionally, the Warning Lexicon does not include any policy recommendations about when a warning should be issued or under what conditions. Notably, there is no Federal government policy dictating the appropriate use of the WEA system by authorized Alerting Originators. The lack of consistent federal policy is a limitation that applies not only to research, but to practice as well. We also acknowledge that the Warning Lexicon is focused only on English language contents, suggesting significant gaps in communicating effectively to a sizeable portion of the US population (Trujillo-Falcón et al. 2021).

Finally, although the Warning Lexicon was designed to include plain language that is accessible to message receivers, impact and protective action statements were not rated for their readability or reading ease. We did, however, explore strategies to assess the reading level of each statement. One readability assessment found that more than half of Americans between the ages of 16 and 74 read below the equivalent of a sixth-grade level (Schmidt 2022). Because of this, some agencies within the US Government, such as the Department of Defense, Internal Revenue Service, and Social Services Administration, make use of the Flesch-Kincaid Reading Ease test, with a goal of meeting a sixth to eighth grade reading level (US Social Security Adminstration 2015; Wylie Communications, n.d.). Readability formulas are designed and used to estimate the difficulty of textual features, such as the frequency of familiar vocabulary and the syntax of text, including number of words and their syllables, number of sentences, and even number of characters that can be quantified. However, readability formulas were designed to examine extended text of at least 200–600 words in paragraph form, not brief statements such as those that are included in warning messages (Readability Formulas, n.d.). Scholars who have applied readability formulas to questions used on tests and exams, which are frequently similar in length to short messages used for warnings, argue that “given the brevity and density of information conveyed in [test items], readability formula data on them are likely to be unreliable” (Oakland and Lane 2004, p. 245). Although they are easy to access and use and demonstrate compliance with government directives such as the Plain Writing Act of 2010 (Office of the Federal Register, National Archives and Records Administration 2010), readability calculators may not provide valid assurance about ease of reading for short messages. Thus, assessing the readability of the Warning Lexicon cannot be completed using existing formulas and calculators, but will instead rely on future testing with populations that are likely to encounter content presented as short messages.

The contents in the Warning Lexicon were verified, and validated, using a multiphase strategy, drawing from the state-of-the-art knowledge on warning messages. At the time of this writing, the research team has conducted one public message testing experiment to determine the effectiveness of the Warning Lexicon contents for five different hazards on WRM outcomes described above. Results demonstrated that the inclusion of contents provided in the Warning Lexicon increased message understanding, belief, and self-efficacy in comparison with similar, complete messages. However, additional message testing and research needs remain. First, contents drawn from the Warning Lexicon can be used in additional message-testing experiments for emerging hazards to determine their effectiveness, in comparison with those using alternative contents. Second, the Warning Lexicon, as a content database, can be tested with message writers to determine how it can be incorporated it into their own preparedness and response practices as well as how it affects their ability to write effective messages. Third, the contents of the Warning Lexicon can be expanded to include new or emergent threats, to account for complex hazards, or to add public safety contents such as alerts for missing persons. Fourth, the Warning Lexicon can be translated into additional languages and tested with those language-speaking populations. And fifth, the Warning Lexicon can also adapt to include other lexical features, such as icons, to depict hazards and protective actions and maps to depict hazard location.

In the future, the Warning Lexicon will serve as the data backbone for software that will enable alert providers to systematically construct warning messages using drop-down menus and other features. Like the suggestion made by Doermann et al. (2021) for wildfire messaging, the Warning Lexicon will provide the foundation for an all-hazards message design dashboard.