Follow this guidance to carry out independent project reviews of research projects that are both valuable and effective — and pain free
Independent project reviews (IPRs) are a common tool used to control, review and validate a project before making an irrevocable commitment to start the next phase. Common issues often reviewed during an IPR include necessary scope, cost and schedule validity, design readiness, unresolved issues, overlooked problems, validity of assumptions, and confirmation of economic analysis, among others. Often called “cold eyes reviews” (as they are intended to bring a fresh set of “cold” eyes to review the project) — or, less commonly, “murder boards” (due to their tendency to be used to identify areas to cut costs or even kill projects) — IPRs are an excellent tool if used properly.
While already commonly used in large-scale projects throughout the chemical process industries (CPI), IPRs are becoming increasingly common for research projects, as well. This is still a relatively new area, however. As discussed here, the use of an IPR for research projects raises some additional issues. These areas of concern — while often common with larger projects — are somewhat specialized and often slightly (but critically different) when it comes to research projects.
Figure 1. Defining which aspects of the research project should be reviewed, while recognizing those that are either ‘a given’ or are beyond the scope of the effort, can help make the most of the effort
Guidelines for a research IPR
Research projects are very different from process projects for a variety of reasons. The scale of the work is much smaller. The technical challenges are often much greater. Information is usually more limited. Project teams are typically much smaller. Goals and objectives are often less well defined. As a result, simply assuming that an IPR can be conducted the same way for a research project is unrealistic and may doom the effort to a poor outcome. Tips and proven strategies are provided here to help tailor the IPR effort to a research project, with the goal of improving both the process and the outcome.
The IPR sponsors must take the time to define the desired scope of the review in adequate detail in advance. While it may appear a safe path to simply open up any portion of the project to the review, the sponsors need to understand that this requires significantly more time, effort, experience and probably more personnel on the team. In many cases, some aspects of the research project are not realistic for the team to review, as the rationale or justifications are outside the scope of their expertise.
The sponsors should clearly identify decisions that are outside the scope of the review — fo instance, that the new facility is to be taken as “a given” — so that the IPR can focus on more germane issues. For example, the decision to open a new facility in a different region to meet some strategic goals is probably beyond the scope of a normal IPR, and should not necessarily be evaluated during the review.
Figure 2. All projects have different critical steps and a good ‘cold eyes’ review will help the team to focus on the most critical ones
The IPR sponsors should check in with the team periodically to make sure they are on track, but give the IPR team room and time to work. It is easy for a team to begin down a path that the sponsors did not originally envision (and may not feel is worthwhile). It may also be necessary for the sponsors to occasionally corral a reluctant member (or team) who is having difficulty remaining within the IPR scope. That said, the IPR sponsors also need to give the team room and time to do their work.
It is not uncommon for the sponsors to want frequent updates, in order to get a sense of the preliminary findings. While this can allow the sponsors to make sure the IPR is not drifting off track or encountering problems that may need sponsor help to overcome, it can also lead to “knee-jerk reactions” to issues that often turn out to have been adequately addressed or at least not as serious as first envisioned once further study and analysis have been completed. Strong, experienced and independent team leadership is essential on the IPR team. Such leaders should hold off on reporting issues prematurely or too frequently, arguing, correctly, that it is too early in the study for anything of real value to be reported.
Figure 3. The ‘cold eyes’ review team needs to focus on all of the aspects of the environment in which the project is proposed, to identify any potentially overlooked area of concern
Pick the best lead possible for the IPR, and make sure they focus on the quality of the result, not just the presentation.The IPR team lead is a visible and desirable position. If done well, it usually can provide a good career boost; however, if done poorly it can be career-deadening. There is a natural tendency for team leaders to work on the final presentation (formal and informal) perhaps more than the content, with adverse effects on the IPR quality.
In my experience, I have found the best team leaders tend to be those who are nearer the end of their careers — individuals who have both the broader experience and a lower fear of consequences compared to those earlier in their careers. Similarly, recent retirees and outside contractors are good candidates, as they are better able to focus on the content of the project review over their perceived performance.
Structure the IPR based on the amount of work that needs to be reviewed, but keep it as small as possible.The IPR team composition is not amenable to a ready formula. My experience suggests that at least three members are needed; rarely is a team with more than five members effective. With fewer members, you typically get a one-person view; any more and excessive time and effort is often expended trying to reach real agreement.
Make sure the IPR has research experience that is both broad and relevant. The IPR team’s experience is critical to its success, so members need to be chosen for their experience with research-type projects. There is an unfortunate tendency for sponsors to feel that an experienced project manager is suitable to review any project. Unfortunately, an IPR has limited time and ability to look at the entire continuum of a proposed project. Hence, the members need to have a clear appreciation of what areas they need to look at in detail, versus what areas can quickly be determined to appear satisfactory. There is a natural tendency for the sponsors to want a very “cold eye” — that is, someone from completely outside the field. This is particularly true if the project appears to have potential problems, such as a higher- than-anticipated budget, a longer than desirable schedule, or technical or implementation issues that were not originally envisioned.
However, my experience is that this level of “cold eyes” (selecting members based on them having little or no research experience) is usually self-defeating. The outsider, however competent and dedicated, does not have the frame of reference needed to quickly evaluate the issues. Too much time is spent with them trying to understand the differences between their scope of reference and a research program.
This results in less effective — and even occasionally wrong — recommendations. Often the outside member is viewed as having adequate research experience based on casual factors, such as participation in one or two other IPRs on research projects, or time spent in research either far in the past or of very limited duration. When assembling the IPR team, nothing can match the effectiveness of individuals with long-term, focused experience with research programs.
IPR members have to be able to look past their preferred way of doing a project and evaluate whether the project’s proposed approach is adequate. IPR members need to be able to look at another project team’s work and try to determine if the project team is missing anything of significance. It is often the case that the project team may be planning to do something differently than what the IPR member would have done, or that may have been less than efficient in some areas (especially on issues that become obvious in hindsight). A good IPR member must have the ability to put aside his or her personal preferences, opinions and prejudices, and only raise issues that they feel the project team’s planned approach does not adequately address. Too often, IPR reports read like a summary of the members preferred approaches — rather than an objective evaluation of the project team’s proposed plan.
Recognize the difference between an IPR for a major process project and a research project. Large-scale CPI projects have significant controls and procedures due to their sheer size and complexity. By comparison, research projects often have much more flexibility and are able to use greatly reduced levels of controls and planning. This lack of very defined procedures for every part of the research project can be a problem for those IPR members without research experience. They often become fixated on what they see as a plan lacking the process structure and controls they would see on a larger project. This leads them to focus on these missing structures as major issues rather than asking if the project-team plan addresses the research project adequately, albeit with less formal procedures and structures due to its lower complexity or scale.
IPRs require adequate time for the members to review all the information in a fast and efficient manner.To do this effectively, the project team needs to have adequate information available in an organized format. While this should be inherent in all project work, too often long-term team members are so familiar with the key practices and procedures that certain important items are not adequately documented.
Similarly, some approaches have developed over time, either overall within the organization or specific to the project, that are not documented at all, as they have become inherent to any research project. Failure to document these key elements — even those that are recognized organizationally and are already clearly followed methods — is a problem; these elements must be documented so that an outsider can understand fully what is going to happen and, more importantly, can assess the underlying logic that led to these practices. The absence of such information at the beginning not only wastes time for the IPR team, but can leave a lingering sense that the project team has not diligently thought through its approach. This will, unfortunately, increase the project team’s work and may delay the onset of the IPR team’s work.
The IPR should not be started until adequate information is available in an organized format. Rushing to get the IPR started always takes longer in the end, and almost always produces a product with poorer quality. Everyone involved in the project needs to be realistic about when adequate information is available. There is always a tendency to start with preliminary documents or a verbal review. Sometimes this is successful with a very experienced project team and IPR; but all too often it leads to more questions down blind ends.
IPR members need to avoid any tendency to delve too deeply into the minutiae of the project. This may be due to their personal style (members who are technical specialists versus generalists), a vague unease over a different or non-standard approach, or an excessive desire to be thorough. The IPR team lead needs to understand how to identify this problem and address it promptly.
Project team members should feel empowered to raise any concerns they have to the IPR and not try to keep them “in the family.” Project team members should recognize how valuable an IPR can be to their success. The review process gives the project team members an opportunity to raise concerns that they feel have not been addressed. It also allows them to often shape management’s occasionally irrational expectations. A frank discussion about the pressure to produce an overly optimistic schedule can often translate into a comment that extra time needs to be allotted for project completion. Demonstrating concern about an unresolved area can lead to sponsor pressure to address a hitherto intractable problem in a timely fashion.
Figure 4. ‘Cold eyes’ reviews can assist in making larger pilot plant and research projects proceed more smoothly, allowing issues to be addressed and rectified earlier
IPR reports need to be crisp and clear, focusing on major issues, and not creating pages of documentation on minor ones. IPR reports should clearly and unequivocally identify those areas the team thinks are going to create significant problems and avoid the all-too-human tendency to list every single item found. IPRs are intended to highlight significant problems that an experienced team thinks could adversely affect the success of a project. What is significant, of course, is open to a wide interpretation. Clearly some items that may have been overlooked or poorly thought through may jump out.
However, there is often a tendency for IPRs to raise every issue they can think of, in order to make sure none is overlooked. This must be strongly resisted. The IPR team has to work hard to separate out the significant issues from those that are routine. Potential issues related to, say, weather, productivity, estimate quality, schedule uncertainty and a host of other issues are always present to some degree or other.
The trick is to identify any issues or concerns that have the potential to significantly affect the project, have not yet been adequately addressed, and are within the project team’s control. If, for example, the project team has included no allotment for weather for an outside job in North Dakota in winter, potential schedule-related issues that could arise should be raised. If, however, the team has allocated a week for bad weather in Texas in the fall, raising the concern that a hurricane could delay the schedule further probably isn’t necessary.
There is no easy or formulaic approach to this identification. My experience suggests that for an issue to be significant it probably needs to meet most of the following criteria:
- It alone has the potential to adversely affect the cost or schedule by more than 10%
- It was completely or significantly overlooked by the project team
- The project team’s analysis of the effect is deemed by the IPR to be incorrect or at least badly flawed
- The IPR’s combined actual experience with the proposed approach has shown significant problems on similar past projects that have not been adequately ameliorated by the project team’s existing plan
These are fairly high hurdles, but they provide useful benchmarks to consider, to clearly separate significant issues from potential problems or simple concerns about alternate approaches. Most IPRs should not be finding many significant issues unless the project team is really not qualified or competent.
List less-critical observations in backup information or an appendix, providing clear documentation that these issues have been reviewed with the project team, are being addressed and pose no significant problems. Many, observations in my experience, require no real action and just perhaps a heightened sense of caution or wariness to make sure they do not escalate. Bringing them to the sponsor’s attention rarely is necessary. However, when all are raised in the body of the final report they often hide the main issues, create a backlash with the project team (who may feel “nitpicked”) and create a sense of unease in some sponsors, particularly those with less experience or more exacting standards, that the project team is not doing its job well enough.
Problems must be clearly and unequivocally stated in the final report. Trying to work the issue so as not to assign blame or needlessly provoke project personnel is certainly desirable — but not at the cost of losing the criticality of the issue. Deciding what is “politically appropriate wording” versus clear communication is never easy. My suggestion is to make sure to state the issue as clearly as possible without any rationale or softening.
IPRs that do not identify any major issues should say so and not fill the final report with minor issues. There is a strong tendency to dig even deeper, to raise any concern that can be seen and to produce something to justify the time and effort. Shouldn’t the IPR always seek to suggest better approaches, or identify areas of weakness or concern (no matter how minor), to help make the project better? I don’t think mentioning a host of minor issues (if that is all the IPR team could find) to the project team has any downside. However, I do think that raising them to the sponsor level provides little upside.
The project team must have the ability to give its view of the IPR’s concerns, to allow them to correct a misconception, clarify an issue or highlight an error. If the process has worked well, the project team will normally tend to agree with the concern and then present proposed corrective measures. The team may sometimes disagree with the IPRs assessment for valid reasons. Usually such a counter argument results from the project team’s failure to effectively explain a portion of the plans or rationale related to a given topic, or the occasional unwillingness of an IPR team to accept an alternate approach. Reconciling divergent views is not always easy. The sponsors will have to decide which view to follow or ask for further work. In the latter case, I would suggest that a different person or team address the specific issue in question. Sending the entire IPR and project teams back to work out the issue is often not productive.
Closing thoughts
IPRs for research projects are valuable and useful tools. But like any tool, they can be used less effectively if their limitations are not recognized, and if some basic guidelines are not followed. IPRs are intended to provide a review of the project team’s plan. That review is only focused on one specific moment in time. Regardless of how good the IPR is, how carefully its recommendations are followed, or how diligent the review, even the best IPR cannot guarantee a successful project. If the approaches decided upon (by the project team, or recommended by the IPR) are not effective, if the project team does not have the skill and expertise to execute the project, if outside events arise that were not considered, projects will still suffer. However, a good IPR can improve the project’s odds of success.
Author
Richard P. Palluzi, P.E., CSP, of Richard P Palluzi LLC (72 Summit Dr., Basking Ridge, NJ 07920; Email: [email protected]; Phone: 908-285-3782) is a consultant to the pilot plant and laboratory research community on safety, design and research project management. He retired as a Distinguished Engineering Associate after almost 40 years at ExxonMobil Research and Engineering, where he was involved in the design, construction, and support of pilot plants and laboratories for ExxonMobil’s research site in Clinton, N.J., as well as affiliates worldwide. Palluzi is the author of two books, and numerous articles and presentations. He is a past chair of the AIChE Pilot Plant Committee, ExxonMobil’s Pilot Plant and Laboratory Safety Standards Committee, and ExxonMobil’s Safe Operation Team for their Clinton Facility. He is on the National Fire Protection Association (NFPA) NFPA-45 Fire Protection for Laboratories Using Chemicals and NFPA-55 Industrial and Medical Gases committees. Palluzi also teaches several courses for the University of Wisconsin’s Dept. of Engineering Professional Development. He has B.E. and M.E. degrees in chemical engineering from Stevens Institute of Technology.