Employing Technology to Enable Remote Research Charrettes as a Method for Engaging Industry and Uncovering Best Practices: A Novel Approach for a Post-COVID-19 World

A charrette is a structured focus group meeting used to elicit creative design ideas and practical input from project stakeholders in a workshop setting. Based on the French word for cart, charrette became part of the architectural vocabulary in the early 1900s at the Ecole des Beaux-Arts in Paris (Healey 1991). Nearing the end of a grueling architectural studio project, students would feverishly work to finish their projects and place them on a cart as it was being wheeled down the aisles between drafting tables to collect projects. The term is broadly used by architects and engineers today for structured, intensive early project planning sessions.

Research charrettes were inspired by the use of planning charrettes by designers to benefit from industry experience in producing early project planning ideas in a workshop setting (Gibson and Whittington 2010). The research charrettes method was derived from these planning charrettes and focused on a new element called social data collection (Gibson and Whittington 2010). As identified by Gibson and Whittington (2010), research charrettes combine some of the best features of surveys, interviews, and focus groups. For instance, they satisfy the objectives of focus groups, such as helping participants explore and clarify their thoughts around a particular topic, while also including surveys for structured data collection from participants. Research charrettes are typically aimed at collecting data while also capturing different types of discussions that focus on tool development or validation of an idea or method. They have been used in multiple studies between 1996 and 2019 (e.g., Cho and Gibson 2001; Griffith and Gibson 2001; Yussef et al. 2019) and have gained popularity in the field of CEM particularly between the years 2010 and 2021. Table 1 provides a sample of studies using research charrettes with a summary of the topic description of each study and the objective behind employing research charrettes.

Although this method was used before 2010, Gibson and Whittington (2010) were the first to provide a detailed description of how to plan and conduct such CEM research charrettes. Research charrettes are traditionally carried out by having industry experts and researchers colocated in a single venue. Charrettes support input and feedback from all participants and allow for project data collection in real time from participants that represent certain stakeholder groups (Yussef et al. 2019). Conducting a successful research charrette requires proper preplanning of logistics as well as availability and allocation of resources such as travel funds and venue funds to carry out the workshops. When performed correctly, they enable collection of a large amount of data by actively collaborating with subject matter experts (Grau et al. 2017) and offer an environment for practitioners to engage in a structured atmosphere while using multiple data collection approaches to obtain responses in less time (Gibson and Whittington 2010). Because of their unique feature as a social data collection approach, research charrette sessions allowed Grau et al. (2017) to monitor the behavior of subject matter experts and to capture their thoughts on applications of the research. Engaging in research charrettes is an opportunity for experts to interact supervised by senior researchers to identify best practices from successful or unsuccessful project environments (Franz et al. 2017). Among the research charrette’s planning limitations is finding a good time and location that suits all the invited participants without delaying the research project. Another limitation is the cost of conducting the session considering travel and venue expenses. Considering the advancement in technology to hold virtual meetings, these limitations can be partially overcome without compromising the value of social interaction.

The point of departure and foundation for this paper is Gibson and Whittington (2010), who identified a research charrette development process map to help ensure successful results. The steps identified in the process included plan and develop multiple data collection strategies using industry input; develop charrette agenda and data collection instruments; set up meeting logistics and solicit participation; and conduct research charrettes. They identified the prework as the most important component including involving the sponsor or oversight committee in putting in place the topic, objectives, issues for discussion, a developed and tested data collection strategy, and good background documents. Attendees must be selected based on specific background and experience criteria. As for the number of participants needed, that would depend on the purpose of the study and whether statistical analysis is to be performed or not.

Logistics must be well defined prior to the workshop such as material and equipment needs, room layout and seating arrangements, breakout rooms, food and drink availability, handouts, nearby hotels, and parking and meeting location access and security arrangements. Also, handout packets and data collection forms must be carefully prepared and provided to participants at the beginning of the workshop. The meeting agenda and flow should be shared with the participants and should accommodate periods of informal networking as well as structured discussion following the presentation of the topic and research background by the researchers. Gibson and Whittington (2010) concluded that research charrettes enable rich data collection and live, person-to-person communication in an accelerated duration while reducing geographic bias of participants. They also allow researchers to field questions during data collection and help gain more consistent results from the participants collectively. These benefits should not be impacted by the sudden shift to remote work if care is taken in structuring the charrette events. Hence, this change compelled the authors to utilize videoconferencing tools and virtual data collection software to continue using charrettes for their ongoing research projects, which originally planned to use in-person charrettes for data collection.

There are a number of types of data needed in CEM research, and accordingly different methods to help collect said data. Examples of such data and methods vary from collecting qualitative data or experts’ opinions using Delphi or modified Delphi methods (Cheng 2014; Tayeh and Issa 2021), collecting qualitative data through observational research methods (Leicht et al. 2010), and collecting data based on practitioners’ experience using research charrettes (Gibson and Whittington 2010). Other examples include using questionnaires to collect data for national surveys or opinions on relative importance of project management best practices; structured interviews or facilitated discussions to gain insights from small group of practitioners on a small group of structured questions and answers; source document reviews to collect data for validation; and focus groups to collect qualitative data that are difficult to leverage from practitioners in one-to-one interviews (Whittington and Whittington 2009). As discussed in the “Research Charrettes: Proven Benefits and Noted Limitations” section, research charrettes include the features and advantages of many of the aforementioned methods and have proven to be very efficient in collecting practitioners’ feedback on newly developed frameworks and tools (Whittington and Whittington 2009; Collins et al. 2017; Yussef et al. 2019) to ensure the usability and applicability of the tools, supporting practitioners in their work.

Considering that research charrettes are a hybrid method that include social data collection, most of the CEM studies that employ them and use surveys or questionnaires as an approach to collect data require having the participants answer survey questions while being colocated in one venue (Kang et al. 2014; Franz et al. 2017; El Asmar and Gibson 2018). However, more recent studies are utilizing Qualtrics weblink to collect responses by distributing the links using emails (Yussef et al. 2017, 2019; Feghaly et al. 2019, 2020). Qualtrics and Google Forms have proven to be reliable and widely used software platforms for data collection (Shrestha et al. 2016; Ezeldin and Abu Helw 2018; Bsisu 2020).

Other advanced technologies are also used for data collection purposes in the CEM field. These include the use of mobile robotic systems for autonomous navigation in indoor areas (Asadi and Han 2020). Drones are also considered to help in collecting more accurate and enhanced real-time data from construction sites (Charlesraj and Rakshith 2020). Similarly, recent studies prove the effectiveness of using virtual and augmented realities in construction engineering education and training while providing useful information for researchers on how to integrate such technologies in their programs (Wang et al. 2018).

Videoconferencing platforms are now an essential technical part of the CEM research world. Recent research has shown the benefits of using videoconferencing tools in carrying out remote research efforts and in collecting data for qualitative studies (Gray et al. 2020) as well as providing guidance through sharing best practices and field examples for researchers transitioning in-person assessments and efforts to a synchronous videoconferencing platform (Marhefka et al. 2020). Archibald et al. (2019) acknowledged the importance of Zoom among other advanced communication technologies in carrying out qualitative and mixed-methods research, and they investigated the feasibility and acceptability of using Zoom to collect qualitative interview data to better understand its suitability for such research. Gray et al. (2020) examined how the features of Zoom can be used to conduct successful qualitative research interviews and concluded with advantages discerned by the interviewers and interviewees using Zoom. These include ease of accessibility as well as saving time for other priorities where logistical hindrances such as travel funds and location to conduct face-to-face interviews are avoided. Despite the physical separation, researchers still felt a real connection with interviewees, unlike with telephone interviews.

In the prepandemic world, one study was identified that tested the use of an online approach to conducting design charrettes in a virtual environment using Web 2.0 (Ryan et al. 2008). They identified a number of online applications that can be used as alternatives for traditional design charrettes for planning purposes such as Wikiplanning, Meetup, Zebralog, PlaceMatters, and Planetizen.

In a recent publication, Dirrigl et al. (2021) discussed how they began adopting the charrettes method to collect data to identify regional environmental and sustainability issues in South Texas; however, their final charrette was disrupted by the COVID-19 pandemic. Accordingly, they used Zoom as a meeting platform to conduct a half-day online charrette in 2020, considering its features of allowing qualitative data collection (Archibald et al. 2019), which is critical to a charrette. They identified the practicability of utilizing the breakout rooms feature and assigning participants in facilitating the discussion. Using the Zoom share screen feature allowed the participants to visualize the shared information, which enhanced the overall process. This is an example of design charrettes where progress was hindered by the COVID-19 situation, but it was overcome with technology.

Certainly, in today’s environment, because of COVID-19 and because of the proven new technologies enabling remote work in general, remote research charrettes are ubiquitous; however, no studies were found outlining their efficacy.

RRCs are planned in a similar manner to traditional research charrettes by determining the issues to be discussed and the information to be collected, as shown in Fig. 1. Accordingly, the agenda for the charrette is planned and data collection instruments needed to capture the information are prepared. A Qualtrics-based survey (or other electronic platform) is developed to collect data instead of printing the forms to distribute to the participants during the workshop. Also, necessary documents about the research topic are prepared and sent via email to the participants before the workshop instead of distributing packets during the workshop. Choosing the participants is based on a list of certain criteria determined by the researchers and the sponsor. As for logistics, choosing a convenient geographic location, picking a venue, planning travel, and all other venue-related details such as reserving rooms, preparing audiovisual equipment, and ordering food are not applicable. Logistics are now related to picking a convenient time to meet across several time zones and having adequate resources such as electricity, computers, headset or speaker, microphone, and internet connection for all participants (participants are reminded of their requirements ahead of time). The senior researchers host the workshops and share details about the research and workshop objectives when asking practitioners if they are interested in participating. After receiving confirmations or declinations from the invitees, the research team sorts the participants list and sends save-the-date invitations, documents to read before the workshop, workshop link, and agenda to the confirmed invitees as well as directions on what to prepare before attending the workshop. Through the invitation links, the participants confirm or decline their attendance. The sponsor, who is also a practitioner, can help host and coordinate the workshop. How the remote workshop is carried will be discussed in more detail with an example in the following section.

The authors have anecdotally heard of other researchers moving toward using RRCs or similar forms of technology to continue their efforts. Whether this continues post-COVID-19 or not remains to be seen, but RRCs can be used in CEM research among other techniques that require human-based interaction between individuals from different areas of the world to exchange thoughts and conduct technical discussions while collecting data and feedback in the process. As shown by Dirrigl et al. (2021), different software and applications can be coupled with videoconferencing platforms to satisfy the objective of the charrette. The advancement of remote research–enabling technologies exacerbated by COVID-19 pandemic restrictions influenced the need to use the RRC method. Twelve remote workshops conducted by the authors on a single topic have proven the feasibility, effectiveness, and practicality of using RRCs as described subsequently.

To test the hypothesis regarding the difference between RRC and in-person charrettes, the authors followed a systematic research methodology described here. First, they conducted a number of RRCs, which were prepared, developed, and hosted following a consistent process. Second, the authors documented the process and improved it based on lessons learned. Third, the authors demonstrated an RRC example to specifically highlight the integration of the videoconferencing platform and the data collection technology, while maintaining the social, professional, and technical features of a traditional research charrette. Fourth, the authors collected research charrettes characteristics data from the recently conducted RRCs and compared these data to previous traditional research charrettes from projects conducted by two of the authors. With the identified metrics, they conducted comparative analyses to highlight the performance of RRCs compared to the traditional method. Fifth, throughout conducting the RRCs, the authors identified challenges faced and lessons learned underlying their experiences as well as through participants’ feedback. The authors then discussed the findings in reference to recent studies published on data collection and research mechanisms in a remote environment. Finally, a list of recommendations was developed around how to successfully conduct RRCs. The following sections detail each step of the methodology.

From June 2020 to August 2021, the authors conducted 12 RRCs focused on the topic of integrated project and program management as related to earned value management systems (EVMSs). These workshops were collectively aimed at developing and testing a decision-support tool, the Integrated Project/Program Management Maturity and Environment Total Risk Rating (referred to as IP2M METRR). This tool is intended for use on large, complex projects and programs that need to comply with EVMS guidelines. More background on this topic is discussed by Cho et al. (2020) and Aramali et al. (2022b).

Of the initial eight RRC workshops focused on developing the new decision-support tool, four tackled the first component of the tool. The researchers aimed to adequately describe and weight the relative importance of 56 EVMS maturity attributes (Aramali et al. 2022a). The other four workshops tackled the second component of the tool; they aimed to identify, describe, and weight 33 environment factors that impact the EVMS implementation. The final set consisted of four workshops that allowed practitioners to use the tool to assess their own projects and programs and to collect project or program quantitative and qualitative performance data such as cost and schedule-related information as well as compliance with the appropriate guidelines, customer satisfaction, and meeting the business objectives.

Emails were used to communicate workshop information and to send invitations to prospective participants. All the workshops were conducted via Zoom, with Qualtrics used to collect participant demographic information as well as their input to survey questions used to assess the tool. Proper use of the technology as well as management experience from traditional charrettes enabled the successful conduction of the workshops, while preserving as much as possible the interactive nature of charrettes. Based on experience, the authors feel this melding of technologies would work effectively for carrying out structured and unstructured interviews and focus groups meetings.

Similar to numerous research studies, a multitude of statistical analyses can be used to test the hypotheses. In this example, descriptive information about the participants, comments received about strengths and areas for improvement for the tool, as well as feedback and suggestions to advance the tool were collected and evaluated. Forced ranking and sensitivity analysis were used to determine the proper weights of the attributes and factors. For the data validation workshops, testing for statistically significant differences among performance metrics and correlation between the assessment results and performance results took place; also, sensitivity analyses were used to determine thresholds to categorize performance outcomes.

A research team consisting of four researchers and 27 industry practitioners cocreated the tool. The practitioners are experts in earned value management (EVM) as well as project and program control with an average of 19 years of relevant experience. Together with the authors, the research team drafted the initial IP2M METRR tool with 56 maturity attributes grouped within 10 subprocesses, and 33 environment factors grouped within four categories.

After drafting the initial tool, a set of eight RRCs were conducted to prioritize both components of the tool as well as provide practitioners’ feedback for refinement. The following description is for the set of workshops dedicated to the environment component of the tool (the maturity component was also assessed in a similar manner in four other workshops). Each of the workshops consisted of four major sections as shown in the agenda in Fig. 2. The first section consisted of introductions and objectives, starting with a welcome word by the sponsor, followed by brief introductions of each of the participants. Next came an overview of the workshop and research objectives, a confidentiality statement, and the research study milestones. Afterward, definitions of the project key terms set the stage for a presentation of the environment component of the tool with its list of environment factors as well as a detailed example of one environment factor.

The second section of the workshop consisted of collecting data to review and critique the environment factors and assessment mechanism, as well as to develop credible weights reflecting the relative importance of each environment factor. The workshops were structured in an organized manner to ensure consistent and efficient data collection while allowing for an open discussion with questions and answers for each factor. The Qualtrics link was shared with the participants in the Zoom chat box so they could start filling in background information about themselves, their organization, and the individual project they were using as the anchor for their input. The hosts had shared documents and instructions with the participants before the workshop so they could each come prepared to the workshop with a recent anchor project. Then instructions were clearly articulated to solicit evaluation input from the attendees. An example of the types of instructions used in this workshop included: “First, collectively participants will review each factor (each factor will be projected on Zoom, in order). Second, participants can ask questions and provide comments and suggestions on each of the environment factors, in order, by category. Participants should use the Qualtrics comment box for each category to provide edits/suggestions/input related to any of the factors.” These sentences show how the hosts are making use of the resources available to enable RRC. Basically, the workshop slides were shared all the time on the screen using the Zoom share screen feature to provide instructions and keep the participants well informed and literally on the same page. Once the participants access Qualtrics, a similar preview (screenshot of each Qualtrics screen) was also shown in the presentation slides to make sure participants can follow the session flow easily without the need to shift between multiple screens. The respondents were then asked to give feedback and suggest edits to each of the factors within each category through a comment box, as shown in Fig. 3. After all the factors for a given category were reviewed, the participants were asked to indicate the relative importance of each factor within this environment category. Questions and comments were encouraged during this process with two-way dialog that could be joined by anyone participating in the session. Fig. 4 shows one example where the rating is on a scale of 1 to 5, where 1 means the most impactful and 5 means the least impactful. The details of the method used to develop the prioritizations are described by Aramali et al. (2021).

Once each factor was weighted within each category, the third section of the workshop assessed these high-level categories against one another. The participants were asked to weight the four categories of the environment component by giving each a percentage out of 100 such that the four grades sum up to 100. Moreover, the proposed rating scheme and rating levels were reviewed to collect feedback on their functionality and clarity. Finally, the data collection was wrapped up by giving the respondents a last chance to provide general feedback and to review previous answers and input, if needed, before submitting their responses. Before exiting Qualtrics, the respondents were asked if they would like to receive continuing education unit (CEU) credits for their participation.

The fourth and final stage of the workshop concluded by sharing how the envisioned environment rating score sheet would look along with the proposed summary visuals in the form of a maturity and environment rating plot or matrix. Then the (virtual) floor was open for further questions and discussions. These four workshop stages, along with a planned break halfway through the allowed time, detail one example of how a successful RRC can be conducted. However, the process does not stop here. Follow-up work takes place to address the given comments and collected feedback, and to follow up on any missing information.

Based on the feedback and weighting data from 20 participants in the first two environment workshops, the authors analyzed the data and combined some factors, reducing the number of factors from 35 to 27. These factors were then ranked by 27 additional practitioners in the third and fourth environment workshops. In addition to prioritizing factors, the environment component of the tool received 368 comments from the first two environment workshops. Most of the comments were addressed by the authors and in some cases experts from the research team were called upon to help (requiring about 20 meeting hours over Zoom), resulting in a new and improved version of the factors that were then presented in the third and fourth workshops. During these final two environment workshops, a total of 308 comments were received from the participants and again were resolved collectively by the authors and industry experts over a second set of about 20 meeting hours via Zoom, before finalizing the tool to its most updated version. Table 2 shows an example of a few of the overarching comments received. The RRC setting as well as Zoom features encouraged and enabled receiving such a large and diverse number of rich comments evaluating the tool and associated approach. The participants were able to interact with other practitioners from diverse industries and locations and exchange thoughts about common challenges and best practices. Unlike in-person research charrettes, informal networking during breaks and breakout session reports were not doable; yet, the Zoom feature of individual or group chatting encouraged traditionally more reserved individuals to voice their opinions and contribute to the discussions.

Other observed advantages of this type of RRC data collection method are the valuable questions asked by the practitioners on how to use the envisioned tool, their feedback on the tool and how to further enhance it, the discussions about how the data they are providing will be used in determining performance assessment, and what they gained in the session in terms of knowledge around the EVMS strengths and gaps, hopefully enhancing their ongoing projects and future projects. This feedback was used to improve the tool development effort and to inform the authors’ and steering team’s research efforts.

As mentioned previously, four additional RRCs were conducted for tool validation in a similar manner. Qualtrics survey questions inquired about project performance data and included the developed tool to assess the maturity and environment of a sample of completed projects. A total of 35 projects and programs totaling more than $20 billion in cost were used to test the tool.

Overall, using RRCs is a cheaper way to get much more data. One observation is that remote workshops have enabled more individuals to share their thoughts through asking questions directly and confidentially to the hosts via Zoom’s individual chat feature. Also, a Qualtrics feature allows respondents to pause data entry and carry on with it at a later time, which provides flexibility that proved critical for some participants. The pandemic and travel restrictions played a key role in making RRC a possible and usable method. In the last 2 years, the technology became available to support this change, which was practically impossible a decade ago. And even if it were technologically possible several years ago, people were not ready to change. Today most people are trained and set up for using technologies for virtual communication and to work remotely, so asking practitioners to participate in remote charrettes is not a challenge anymore.

Recently, impacted by the COVID-19 pandemic and the resultant forced shift to remote work, many researchers, especially those involved with data collection that relies on human interaction, started accommodating to the situation and finding new methods and techniques to continue their work. Lobe et al. (2020) provide guidance on the videoconferencing services suitable for a project’s needs to transform from face-to-face qualitative data collection to socially distant methods. This study exemplifies such efforts from the medical sector and identifies Zoom among others as convenient platforms; however, they point out the issues of ensuring security and confidentiality of the data in the process. Opara et al. (2021) describe the implementation methodology as the benefits and challenges of using Google Docs to conduct synchronous, online, written interviews through presenting two of their projects’ case studies as live examples. Their study is within the social sciences field. The benefits identified include more privacy for the participants, and “shy” individuals can voice their thoughts better. Also, more flexibility and time savings with respect to “transcribing verbatim immediately.” Similarly, Marhefka et al. (2020) discuss best practices and key considerations in transitioning in-person data collection to videoconferencing platforms in the medical field.

Considering the performance results of RRCs, it is evident that they share common advantages with the methods and techniques described here and in the background section. These include saving time, voicing the thoughts of more individuals, increasing the diversity of participants, and reducing travel costs, among others. However, unlike RRCs, all these existing studies do not address the CEM field, do not quantify the performance metrics that were addressed in this paper, and mainly rely on one platform or tool to specifically collect a certain type of data rather than integrating a videoconferencing platform that connects the investigators to the attendees, a data collection tool that captures attendees’ interventions, and a research method that solicits practitioners’ experiences and direct feedback.

While RRCs allow for collecting a richer data set from experts, the sheer amount of data will require more time for researchers to go through, assess the input, and address it. This can be considered a challenge associated with RRCs. Indeed, the authors have identified several challenges associated with RRCs. The major challenges of this method are technical difficulties, associated with the use of an online meeting platform. Similar to any online tool, the successful use of the Zoom software is contingent on functional infrastructure and services of electricity and internet connection. In some cases, the functionality of these services is jeopardized by local weather conditions leading to cutting off an attendee from the call. Also, because of conducting all work remotely and keeping the computer turned on most of the time, computers might crash at any point in time causing disruptions to the workshop. The ability to rejoin the meeting after being disconnected is contingent on the speed of the systems update or the availability of other devices with installed and functional Zoom software. Also, sound settings of speakers, microphone, headsets, and cameras must be adjusted to the proper settings. None of these issues were required in the traditional charrette method.

In terms of workshop logistics, in traditional charrettes participants can usually hear the hosts’ instructions the first time, but this might not always be the case on Zoom. Also, participants arriving late to a virtual meeting (which has happened a number of times) can also exhaust the hosts if they need to repeat instructions several times or answer the same questions that were explained at an earlier time. During in-person sessions, hosts could walk around the room to guide the respondents; in virtual sessions this is not possible, so it requires very explicit instructions and continuous feedback on whether the pace of the workshop is too fast and if everyone is tracking together. Traditionally, presenters can interact with the facial expressions they receive from the audience; however, in Zoom many participants decide to turn the video off, which can be a communication barrier. Moreover, the hosts cannot guarantee that all the attendees at the start of the workshop will attend the whole session. For example, in one case a workshop started with 28 participants and finished with 19. Attrition was observed to be higher in RRCs compared to traditional charrettes. Finally, the workshop time must be carefully set during reasonable working hours across multiple time zones because the attendees are geographically dispersed. However, regardless of the issues discussed here, the benefits of RRCs are several and far outweigh the challenges.