Tulare County’s Peer Review

For ASCE members interested in the peer review program, there are several factors that might provide some insight and possibly motivation to either invite a peer review team (PRT) to visit their agency, or to become a peer reviewer. In fact, there are two stories to tell: the overall objectives and procedures of the peer review process, and the peer reviewer’s duties and responsibilities. Long working days were a part of one of these stories of a recent ASCE peer review conducted in Tulare County, California, for the Resource Management Agency (RMA) and its director, Henry Hash.

A peer review is a structured process that helps an agency raise their performance. A PRT does this by examining the agency’s mission, objectives, policies, resources, and management practices. ASCE undertakes a peer review at the invitation of an interested agency.

A team of experienced ASCE members works to identify key issues and opportunities an organization currently faces. The PRT is made up of a group of professional civil engineers. These engineers have a breadth of management and technical experience as well as expertise in public works and planning that helps them to help other public agencies improve service.

Once an agency decides to have a peer review, work begins immediately. The agency director—working with Alicia Karwoski, ASCE’s Manager of Professional Practice—assembles a PRT from reviewers that are available for the assignment. Even more important than availability, the reviewers must meet the agency’s approval. Agency directors often look for specific training or experience in reviewers’ backgrounds to insure an understanding of the agency. This step often takes a while to complete as schedules are mixed and matched and reviewer qualifications are assessed.

The first step for Director Henry Hash in Tulare County was to review and approve the members of the PRT that would perform the peer review of his organization. Once the team was approved, and dates for the review set, the PRT started work . . . and work it is!

Intensive is just one word that peer reviewers use to describe the peer review experience. For the ASCE members taking part in the Tulare County RMA peer review—Mike Quan, Ron Lundquist, Stu Moring, Dennis Randolph, and Kyle Schilling—work started several weeks before the on-site visit to RMA’s offices in Visalia. A large box for each PRT member arrived from Tulare County crammed with the RMA’s policy and procedure documents, staff resumes and organizational charts, and most importantly, questionnaires completed by many of the agency’s employees.

It is the peer reviewer’s job to review these materials and become familiar with the organization, its people, their processes and procedures, and some of the ideas they have about their organization. Besides reading the material, the team reviewed and analyzed the questionnaire results both subjectively and statistically.

A week or two before the on-site visit to RMA’s offices, team leader Mike Quan held a teleconference with the PRT. Discussions centered on some preliminary impressions based on the advance material sent to each reviewer. The PRT also talked about how the visit and especially the peer review might be structured. Mike also reported on discussions he had with Henry Hash to prepare for the visit. The conference call ended with scheduling the PRT’s first on-site meeting at 6:00 p.m. Sunday, the week of the visit.

How the agency visit is carried out is critical in two ways. First, it is important to guarantee the PRT answer an agency director’s questions about their organization. So, interviews need to be structured to hone in on these items. In addition, it is important that the PRT tailor their visit in a way that helps them to answer the questions they see as critical to their mission.

After meeting Sunday evening, the four-and-a-half-day visit to the RMA officially began early Monday morning with a 6:00 a.m. breakfast meeting of the PRT. The purpose was to review the schedule for that day and each following day of the week, and plan individual and group interview assignments. In addition, since the first meeting was a briefing for the director, and then his senior managers, PRT members needed to organize their discussions in preparation for these briefings.

The director is the PRT’s client and the PRT reports its comments and findings directly to him. While it is common to also brief senior managers of the PRT’s findings, this only happens if the director wants it done, and in RMA’s case, Henry Hash was enthusiastic about this.

Once the first meetings were completed, the PRT members were shown their offices for the next four days as well as their meeting rooms. Even more importantly, they were given take-out menus, as lunch, just like breakfast (and some dinners) would be working sessions. Finally, it was time to get down to work!

At the heart of the peer review process are one-on-one confidential interviews with agency staff by the peer reviewers. The ability to establish rapport quickly with the employees and to make them comfortable so they speak their minds freely is the first and by far the toughest job of a peer reviewer. Peer reviewers must be skilled in communicating with strangers and specifically must be able to “break the ice” with people who often feel threatened by the entire peer review process.

In preparation for the PRT’s visit, Henry Hash told the staff of the RMA of the coming peer review. This is a very important step in the entire process. Further notice is also provided when staff is asked to complete the program questionnaires. However, there can never be enough notice and information given out about what a peer review is all about.

By talking with staff, finding common threads among the many comments that they offer, and most importantly evaluating everything within the context of the peer reviewers’ experience and knowledge, the PRT can quickly learn about an agency.

What peer reviewers generally notice during the first morning is hesitancy by individual staff members to talk freely, or at least to be comfortable and talk. Confidentiality of the interviews is stressed to the agency employees to ease their concerns. Peer reviewers must be skilled in interactive interviewing. That is, they must know when to listen. They must also know when to ask questions to clarify comments and find out answers to questions that need to be answered.

The plain fact is that many people just do not know much about peer reviews. Often they view these professional reviews as similar to audits, and the PRT members as potential hatchet men. And their first view of a group of senior engineers in “suits and ties” often does not help them feel comfortable.

Therefore, the first interviews are important. The word quickly spreads throughout the agency once the interviews begin: What did they ask? Who are they? How did the interview go? A measure of success for the PRT is more talk and longer interviews as the first day and the rest of the week progress.

The PRT interviews a sample of employees from across the agency. The agency takes the lead in selecting employees for interviews, and scheduling them throughout the week of the PRT visit. The PRT makes sure the scheduled people represent a good cross section of the organization and adds or subtracts as necessary. The PRT also makes sure the employees are assigned to peer reviewers so each PRT member meets and interviews individuals from every work area of the agency. In this way, PRT members can confirm and corroborate comments, issues, and concerns heard during interviews. Much of the time spent in the frequent regroups of the PRT is used to discuss points and validating comments and ideas the staff offered them.

After meeting with Henry Hash, the PRT met and briefed the RMA’s senior managers and then got down to the business of interviewing staff. For the next four-and-a-half days, the PRT followed a crushing schedule of interviews, debriefings, and summarization of findings.

The daily routine of the PRT calls for a short team meeting, then interviews. A midmorning break follows to give the team an opportunity to discuss what it has heard, and after that, more interviews are conducted. Next comes a group working session during lunch to discuss findings, and then more interviews. In mid-afternoon there is a break to further discuss findings, and after that, more interviews. Then, there is a closing meeting on-site. Finally, after a short break, the PRT holds a dinner meeting to summarize the day’s interviews and efforts. Each meeting provides an opportunity to compare findings, elicit common issues, and make adjustments: adding or deleting certain interviewees, adding certain questions to the interview process, or seeking factual confirmation of a given theme.

For the RMA PRT this work session commonly ran to 7 or 8 p.m. After that, there was a drive back to the hotel. All in all the PRT conducted individual interviews with eighty-eight members of RMA staff. Various team members also visited offices in the central administration building and at project sites in the field. Two PRT members also attended the Tulare County Board of Supervisor’s meeting on June 14, 2005.

With a heavy flow of information pouring into the PRT, it is a major task to organize and make sense of it all. Frequent meetings between the team’s members are necessary to sort all this information out. However, a much more wide-ranging process is necessary to meet the high standards of ASCE’s peer review program.

Toward the end of the peer review process, the team’s leader engages the PRT members in just such a formal process to organize data and ideas. At the end of the fourth day of the review, Mike Quan asked the PRT members to spend about thirty minutes reviewing interview notes and documents and circling important items. Then each PRT member was directed to write a three-to-ten-word statement that answers the following question: “What are the agency’s three to five top key opportunities for change? Team members were asked to supply only three to five themes to force them to prioritize.

The team members use four criteria to help them discover key issues. First, the agency must have influence over the item. Second, the change or improvement will add significant (i.e., noticeable value) to the agency’s performance. Third, the agency must be able to carry out the effort with existing resources. Finally, the item must be a root cause, not an effect.

The PRT was able to develop a set of “opportunities” for the RMA after four days, and presented them to the director on Friday morning. The PRT gave the director a confidential briefing lasting about ninety minutes that followed the form of the final written report.

The exit debriefing is the high point of the entire process for both the PRT and the agency director. The verbal report is the first formal report the agency receives on the peer review. The agency director receives comments from a group of peers, and the PRT looks for affirmation that they have identified and confirmed some of the agency’s concerns.

While the PRT leaves no written report with the agency, ASCE may, if requested, provide a written report to the agency. This usually takes four to six weeks after the visit. The emphasis of a PRT’s work is always positive, and highlights the opportunities an agency has to make itself better. Both the verbal report on the last day of the visit and the written report usually have the following three parts. First and foremost are the key opportunities for change. Taking these actions that can significantly increase the likelihood of successfully carrying out the responsibilities of the organization. For Tulare County RMA, the PRT offered five key opportunities.

Next are the influences/drivers. These are issues that may be outside the control of the agency, yet may be significantly influencing the agency’s performance. In addition, there is potential that these issues may evolve into key issues and keeping an eye on them may present further opportunities for the agency. The PRT felt there were eight influences/drivers the RMA needed to monitor.

Finally, the PRT points out “other matters to note.” These are insights into the organization that do not rise to the level of a key opportunity, yet are easy to carry out, and can benefit the agency. The PRT noted four “other matters” in their reports.

Overall, the ASCE peer review program provides benefits to all involved. For the Tulare County RMA and Director Henry Hash, their peer review has given them an opportunity to make their organization better. The peer review was a huge commitment for the agency. In particular, the week of the on-site office visit by the PRT members means a large disruption in the routine of the staff. Over a four-and-a-half-day period eighty-eight staff members spent an average of about one hour each discussing the agency with PRT members. There is also a lot of time spent among employees discussing their PRT interviews during the entire period. Obviously many staff members took the peer review very seriously and had spent time before their interview preparing notes and organizing their thoughts.

This is a tremendous benefit for the agency. Often in the rush to do the daily work of an organization people just do not have a chance to really think about making their organization better. The ASCE peer review program provides just such an opportunity.

For the members of the peer review team, a peer review provides an opportunity to use the skills and experience gained over a career to help a colleague. It means hard work and long hours dedicated to listening to people and seriously considering their comments and points of view. Finally, it means a week of pro bono work, a contribution to the good of the profession.

—Dennis A. Randolph, P.E., is managing director of Calhoun County Road Commission in Marshall, Michigan, and can be contacted via e-mail at [email protected].