Professional Societies Making Engineering Outreach Work: Good Input Results in Good Output

There are many stakeholders in engineering outreach, and their goals and objectives are not always compatible in practice. These disconnects can undermine outreach efforts. For example, while the universally stated goal of society outreach is to introduce children and young adults to engineering and recruit more people into the profession, an unstated but influential goal is to provide leadership training and volunteer activities for members. The latter too often leads to the volunteers themselves becoming the de facto audience of outreach efforts rather than the children and young adults that the activities supposedly serve. If keeping volunteers engaged is an unstated goal, then proven practices like vetting, training, and critiquing volunteers is difficult lest they become offended, disaffected, or less enthusiastic about society involvement or even membership (Bogue and Marra 2007; Wharton 1991). Nonetheless training for volunteers to increase their knowledge and understanding of targeted audiences and motivation to reach them is basic to outreach success (Haynes and Nykiel 2005).

Other stakeholders bring additional objectives. Many companies use a three-pronged philanthropic strategy of “do good, look good, and feel good” (Rosen 2012). Corporate sponsors value having their name associated with an activity acknowledged as successful by attendees and volunteers. Of course, sponsors expect that their philanthropy will “do good” and that the event will yield future engineers. But corporate sponsors, like the volunteers they sponsor, are often unaware of how to properly assess and evaluate the efficacy of programs. Given the time and training needed for performing research, developing objectives, and implementing pre- and post-event evaluations, often sponsors accept volunteers’ evaluations that rely on “look good” measures of quantity (i.e., number of children, adult influencers, and volunteers who gave positive feedback on the event) rather than quality (i.e., sustained impact on participants, continuous improvement, and measured outcomes). More than 100 corporate CEOs formed a coalition, Change the Equation (www.changetheequation.org), to “foster widespread literacy in science, technology, engineering and mathematics (STEM) that sparks an innovative spirit in students and prepares them for postsecondary options.” Recognizing that their philanthropic investments in STEM should see greater results, the coalition issued “Design Principles 2.0 for Effective STEM Philanthropy” (Change the Equation 2012). Consistent with the arguments presented here, the design principles will increasingly focus corporate sponsorship on programs that provide evidence of meeting objectives.

Volunteers are key to reaching—or, it has to be said, repelling—diverse audiences. The dilemma for traditional professional societies in developing activities that attract and are effective with boys and girls of all races and ethnicities is that the society members themselves are predominantly male and Caucasian. Husbands et al. (2002) noted that most organizations have an “unproductive image of ‘who a volunteer is,’” leading them to recruit leaders and volunteers they know and who look like them rather than those who are from the demographics that the activities are designed to attract and serve. Ultimately, this disconnect underscores the need to create inclusive environments in the engineering organizations, to attract diverse volunteers into outreach activities, and to build culture and gender awareness in all volunteers. Such initiatives will also make society membership more attractive to those underrepresented in engineering who are not now joining, including younger engineers. To be successful, organizations must go beyond asking nontraditional members to volunteer; such members must be recruited to leadership positions, have the ability to drive changes in existing programming and the organization overall, have the expectation of being listened to (Latting 1990), and be recognized for their contributions to the society and the discipline.

Another disconnect is that societies’ formal expertise in outreach is held by a small number of staff, while the implementation is typically done by very large numbers of volunteers geographically dispersed from each other and from the society leadership. Volunteers, when well trained, can make an impact (Baker et al. 2000; Wasik 1998; Haines 1990). While there is a dearth of literature on volunteerism in terms of motivation and effective management (Husbands et al. 2002), there are examples of effective training programs (Felix et al. 2004, Countryman 2009, www.Outreach4Change.org 2011). Managing volunteers—from recruiting to screening to requiring training—is a foundational challenge to providing effective outreach. The staff experts have little formal “authority” over the actual outreach interaction and can only work to persuade the volunteers to adopt best practices. If the outreach staff of a society is judged by how many activities are offered and how many volunteers are engaged, there is no motivation to understand whether the nominal audience—the participants—are learning what they are intended to learn, much less act on it.

Volunteer members, the backbone of society outreach efforts, are knowledgeable on engineering and enthusiastic about their profession. But typically, they have little training beyond the mechanics of the activity, have no knowledge of marketing to young people, lack cultural and gender awareness, are inexperienced in the delivery of education, and generally do not have the expertise to implement the best outreach activities. It is illogical that any engineering professional would undertake outreach projects without appropriate preparation. How many engineers would design a new process or component using only ideas observed from others or without understanding the requirements of the design? Successful projects survey internal and external conditions, define prospective audiences, research specs, review existing designs (especially for activities or programs that have run for years), identify outcome metrics and decide how to measure them, validate and verify the project design, and more. Yet, the former scenario is the prevailing practice in engineering outreach. For example, many organizations’ long-running but outdated outreach programs are considered successful when measured by how many children, volunteers, and sponsors it attracts.

Again, if we look at industry as a model, offering the same product year after year without assessing changing customer needs, emerging markets, cost-effectiveness, or quality improvement opportunities would be a nonstarter. Over time, the product would fail to attract new customers and would convince existing customers to find a competitor. Although this is a nonviable industry model, this is the typical society outreach activity today. Too many programs are developed based on experiences, information, or support volunteer leaders thought they would have like to have had or remembered experiences that “turned us on” to engineering. These activities have continued without change not because they effectively engage participants in engineering in the long term, but because they continue to attract volunteer involvement, participants, and funding. The latter are important but alone cannot achieve the long-term goals.

Outreach activities that succeed in the overarching goal of motivating participants to prepare for and ultimately pursue an engineering career are based on established best practices: reports on which kinds of activities and practices work, academic and demographic research, surveys and other assessment tools to measure participant response, participants’ experiences and expectations, and meaningful evaluation of results. These practices, covered in more depth in the sections that follow, too often are missing from society-promoted or -sponsored outreach efforts.

To achieve good outreach, as measured by whether it engages children in engineering, activities need to be based on a solid foundation. Assessment-based practice is the core of successful event planning and delivery in outreach, just as it is in every engineer’s career success. Ideally, assessment begins at inception, either of the event or activity itself or of the annual organization cycle for delivery. Three practices are key to successful assessment-based programming. The first and overarching practice is integrating assessment throughout planning—from the selection of a primary goal and supporting objectives, to the identification of activities that can achieve them, to measurement of whether they were achieved, to the application of results to continuous improvement. Two additional practices—(1) the use of proven practices and (2) relevant social science research that looks at the characteristics of targeted audiences and the efficacy of outreach approaches and methodologies (now available in user-friendly formats)—help organizers define those objectives, identify how they will measure success, and create a sound platform for development and continuous improvement. In the sections that follow, we discuss these key practices and highlight several initiatives that provide resources easily accessed and are designed to be user friendly.

Seeing happy kids at an event is heartwarming and satisfying, but the only way to understand the real impact of an activity is to measure it. Assessment should be integral to all activities or program planning and not limited to simple surveys and participant counts (Bogue and Marra 2007). Thinking about assessment from the beginning of the society’s planning cycle, rather than considering it as an add-on done primarily to meet funding requirements, is key to successful outreach efforts. Everyone involved in mounting an event, from the organizers to the volunteers, needs to understand whom they will be serving (audience), what problem requires action (needs assessment), what they aim to achieve through the event (goal), how that aim will be operationalized (objectives that can be measured), what activities or information will best serve the objectives (agenda), how to best design the activities or information delivery (proven practice of the society’s own and of others and insights from social science research), whether the goals and objectives were reached (e.g., surveys, formal observations, time and resource analysis) and finally how to achieve the continuous improvement of the event, activity, or overall programming.

For example, if the overarching goal is to introduce girls to engineering as an active, problem-solving profession, one of the organizers’ objectives may be to understand how participants describe engineering after an event. Knowing that they want participants to see engineering in this way, the organizers decide to offer a hands-on engineering challenge activity. The measurement of whether the activity achieves the objective is how participants respond to specific questions on a postevent survey. If participants actively connect the enjoyment of the hands-on experience to the enjoyment of an engineering career, then the organizers know their objective was met. If not, then they have the information to make the event better achieve their objectives—in addition to providing an enjoyable time for participants—the next time around. (A more detailed description of assessment for outreach practitioners is provided in the companion paper “Outcomes-Based Assessment: Driving Effectiveness of an Outreach Program for Girls” in this special issue.)

A proven practice is an initiative or program whose effectiveness in attaining its desired outcomes is supported by carefully crafted goals and objectives, comprehensive research, and assessment and evaluation. Developing and evaluating outreach activities that are research based, assessment based, and inclusive is an important endeavor that does require up-front investment but will yield greater results in the long run. Becoming aware of and reusing existing proven practices in professional society outreach, implementing consistent assessment practices, and integrating cultural awareness into outreach activities will improve the effectiveness of outreach. Creating an inclusive program cannot be seen as unnecessary or treated as an add-on, but must be integral to outreach efforts.

The Society of Women Engineers’ Assessing Women and Men in Engineering (AWE) Project is an NSF-funded initiative that provides easily adaptable surveys and other assessment tools for use in measuring typical STEM outreach activities (www.engr.psu.edu/awe; Marra and Bogue 2004). The AWE Project also offers literature resources and capacity-building tools for STEM outreach and education personnel. AWE surveys are tested, free, and easily adaptable to engineering society activities. Paper surveys can be downloaded from the website or SurveyMonkey online versions requested by writing to [email protected]. Surveys are available for Grade 2–16 outreach activities (Spanish versions available); college-level mentoring, leaving and persisting in engineering, climate, and self-efficacy activities; and peer observation. AWE collaborates actively with the National Girls Collaborative Project and the National Academy of Engineering’s Center for the Advancement of Scholarship on Engineering Education (CASEE).

The Program in Education, Afterschool and Resilieny (PEAR) at Harvard University offers a searchable online database of assessment tools for informal STEM learning called Assessment Tools in Informal Science (www.pearweb.org/atis). PEAR is designed to help researchers, evaluators, and practitioners access selected tools for assessing activity and program quality and outcomes. Based on research findings (Hussar et al. 2008), this database is a project of PEAR and Youth Development Researchers at 4-H.

The Applying Research to Practice (ARP) resources (www.engr.psu.edu/awe/ARPResources.aspx), developed jointly by SWE and CASEE, provide practitioner-friendly access to research and scholarly literature rzelevant to the development of effective outreach activities. The ARP resources include papers on the following topics:

In addition to the ARP resources, the website includes information on attribution theory, changing problem solving, cooperative learning, questions in the classroom, career development theory, family influence, sense of community, and factors supporting the retention and persistence of female undergraduates.

CASEE, through its Engineering Equity Extension Service (EEES) project sponsored by the GSE program of NSF, has several research-based resources for practitioners. The Research to Practice in Engineering Education series (CASEE 2012) summarizes research in educational psychology, sociology, and behavioral science into one-page documents with practical suggestions for teachers and outreach volunteers categorized by research and audience. The Teachers Integrating Prior Scholarship (TIPS) resources include topics relevant to outreach such as reaffirming students’ self-worth, allowing them to build on prior knowledge, setting goals, increasing diversity in teams, encouraging them to appraise their efforts, encouraging questions by all students, addressing misunderstanding of STEM concepts, and others. The Data-driven Engineering Education Practices (DEEP) resources summarize research in engineering education aimed at faculty but has documents relevant to outreach volunteers on how faculty attitudes affect retention and culture in the classroom, self-efficacy, team performance, and other topics. Finally, the Responding to Administrative Priorities (RAP) resources condense social and behavioral science research for administrators, but topics relevant to outreach activities include reactions to being outnumbered (especially important for girls and minority students in STEM), video games and spatial cognition, perceptions of intelligence, and leadership skills.

A recent study (Committee on Public Understanding of Engineering Messages 2008) examined how adults and youth regard engineering and what messages would present a more accurate picture of engineering as a prosocial and relevant profession. The Changing the Conversation report and its partner website (http://engineeringmessages.org/) allow the engineering community to share messages and resources more efficiently.

An AAUW (2010) research report presented compelling evidence that can help explain the puzzle of Why So Few?. The report presents in-depth yet accessible profiles of eight key research findings that point to environmental and social barriers—including stereotypes, gender bias, and the climate of science and engineering departments in colleges and universities—that continue to block women’s participation and progress in STEM fields. The report also includes up-to-date statistics on girls’ and women’s achievement and participation in these areas and offers new ideas for what each of us can do to more fully open scientific and engineering fields to girls and women.

The New Directions in Engineering Excellence resources were developed by CASEE as part of the EEES project. These are easy to use, visually attractive booklets, videos, and slide presentations for K–12, undergraduate, and informal educators to recruit, retain, and advance women in the field of engineering by better engaging the diverse skills, interests, and backgrounds of their female students and connecting educational activities to engineering opportunities. The resources emphasize evidence-based research on best practices in engineering education. A thorough resource list and bold design elements offer users the option of finding and using academic research or reading quick overviews. Users will find information to enable them to introduce female students to the practical applications of their STEM degrees, to keep those students engaged throughout their undergraduate careers and beyond, and to help them achieve their fullest potential.

Each multimedia product distills key points from research on recruitment, retention, and advancement in engineering education; provides examples of and strategies for successful recruitment, retention, and advancement; and identifies and describes formal and informal steps as well as specific activities for educators. Each booklet addresses differences in strategies based on type of learning environment (formal or informal); learning activities (academic, experiential, or attitudinal), outreach environments, age of audience, and highlighted populations of underrepresented students. The books are organized in a similar way, with an introduction to both the series and the topic followed by information on and the context of the specific topic (recruitment, retention, or advancement). Each product also covers fostering persistence, mentoring, and evaluation and assessment of efforts toward each of the topics.

Outreach 4 Change (www.outreach4change.org) is an NSF-funded initiative to increase outreach capacity and effectiveness among engineering society leadership, staff, and outreach volunteers who are striving to connect and engage girls of all races and ethnicities with engineering. The American Indian Science and Engineering Society, the National Society of Black Engineers (NSBE), the Society of Hispanic Professional Engineers, and the Society of Women Engineers are collaborating on the effort. On March 24, 2011, Outreach 4 Change held its first workshop in St. Louis during the NSBE annual convention. This one-day event focused on training engineering association leaders, staff, and outreach volunteers. Other workshops or sessions have been held for the American Association of Engineering Societies, Engineers Week Discover-E, and the National Girls Collaborative Project.

Beyond the workshops and sessions, Outreach 4 Change is a growing community educating and communicating about the need for outreach to all girls and the methods to do so effectively. The objectives are to build outreach capacity among the professional staff and volunteers who develop and offer these programs, engender a culture of assessment in the outreach community, create cultural awareness and integrate proven practices for underrepresented groups into outreach strategic planning and implementation, and create a network of knowledgeable society staff, members, and leaders committed to developing effective K–12 outreach activities with measureable outcomes.

The National Center for Women & Information Technology (www.ncwit.org) offers a clearinghouse of evidence-based resources relevant to girls, women, technology, and computing that are relevant for all STEM disciplines. Resources include statistics on women and girls in information technology (IT), links to assessment resources, information on careers in IT (including videotaped role models), lists of top-10 resources for promising practices for students, parents, and teachers; research reports like “How Can Encouragement Increase Persistence in Computing?” (Cohoon 2011), “How Can Companies Achieve Organizational Diversity?” (Ashcraft 2008), and “How Can Reducing Unconscious Bias Increase Women’s Success in IT?” (Ashcraft 2010); curriculum materials, posters, workbooks, talking points, webinars, and videos and programs-in-a-box; and turnkey solutions with components necessary for quick and strategic action.