o The replaced legacy assets were decommissioned and no longer run.

o The project was completed roughly within time and cost parameters.

o The estimated business benefits that were used to justify the cost of the project were substantially realized.

So, with these four criteria in mind, let's ask the question again. Do the 50 percent of projects that don't fail by the initial implement/no implement criteria succeed by definition? Well, if the project costs twice, or even thrice, the agreed, original estimate, is that a success? What if the project costs about what originally was estimated but the benefits that were used to approve the project fail to materialize? Bear in mind the cost side of this equation is measured at least in millions and often in tens of millions of dollars.

The basis for these questions is the numbers that are formulated during the cost-benefit analysis (CBA), which takes place prior to initiation of the project. Usually a CBA leads to some estimate of how implementing the new system will pay for itself over a certain time horizon. This often is referred to as the ROI, or return on investment. While these kinds of calculations are a useful discipline and should be part of any significant project, they often lead to wildly optimistic conclusions. There are several reasons a CBA can be wrong. These include:

o Political pressure to minimize costs and maximize benefits.

o Lack of complete understanding of the cost components of a large, lengthy, and complex project.

o Inappropriate or inadequate assumptions about the future-state business environment on the basis of which to estimate post-project benefits.

Before we proceed, let me reiterate a verity of the legacy replacement project environment: The costs are front-end loaded, and the benefits are back-end loaded. The benefits do not kick in until the new system is largely implemented and the legacy system is decommissioned. By then, most of the costs have been incurred.

What are these costs, and how can they be accurately estimated? Assuming the project is a "package" implementation, there are some guidelines: First, the vendor will charge a license fee based on some set of parameters that usually involve the amount of premium or number of claims that will be processed by the system, the number of lines of business, the number of states, and the scope of functionality being provided. Second, the vendor will charge a maintenance fee, which each year will equal roughly 20 percent of the license cost. There may or may not be significant hardware and software purchase and support costs, for example, new servers and a new relational database management system (RDBMS).

So far the numbers are reasonably easy to estimate. Now comes the hard part. How much will the implementation cost? The implementation cost is both the largest cost component and the most difficult to estimate. There are three major areas of implementation cost. First, there are costs incurred by the software vendor to perform installation, configuration, tailoring, and integration tasks. Second, there are carrier IT and business costs to define business requirements, support legacy integration and retirement, design and execute system validation and acceptance testing, develop and deliver user training, and support system rollout and data migration. Third, there are system integrator or third-party consulting costs to perform tasks the carrier is unable to undertake for reasons of resource constraints, knowledge, or skill sets.

Defining these costs requires two disciplines: the identification and development of a work breakdown structure, and the estimation of effort for each identified work unit. The point here is a significant amount of project definition and planning needs to have taken place before a realistic cost estimate can be developed. Taking this one step further, we also should note the work breakdown structure and the estimation rely on a realistic definition of the project scope. If the scope is incomplete, the cost estimate will be the same.

Who is capable of creating an accurate cost estimate? The vendor should be capable of providing accurate numbers because it has done similar projects before (if it has not, the carrier should revisit its selection criteria), but the vendor will provide estimates only for its part of the project. There are third-party consultants who plan and estimate projects for living. Unfortunately, the least likely group to do this accurately is the in-house carrier staff. Why? Because, with a few exceptions, in-house people do major transformation projects once in their career and have no experience in estimating anything this large and complex.

Even more difficult to estimate accurately are the benefits used to justify the project to executive management. I'd like to expand on an analogy I quoted in February: Trying to quantify the benefits of replacing core insurance processing systems is like trying to justify the cost of putting your kid through college. Like replacing a core system, college represents a major financial commitment that must be sustained over several years, the costs are front-end loaded, and the benefits are unpredictable and all back-end loaded.

The reason the benefits are so hard to predict is the acquisition of a college degree, or the implementation of a new core system, presents its owner with potential more than an end result. It's what the owner does with the new asset that leads to the benefit. We know statistically college graduates earn more than non-grads. We also know, at least anecdotally, insurance carriers with modern systems seem to compete better than those with legacy systems. But the fact remains it is hard to predict the outcome or benefit for any one specific graduate or carrier. It all depends on how that newly acquired potential is used over time and what subsequent events occur and the associated decisions that are made.

Clearly, if a carrier implements a modern, highly configurable policy administration system and continues to do one rate change per year and minimal product changes, it will accrue a certain level of benefit. Suppose that carrier then acquired a new member company or found the need or opportunity to enter a new market. Suddenly the benefit of the new policy administration system is greatly increased because the company is capable of doing things it could not do before, or at least capable of doing them much more quickly. I worked with a client last year that concluded it could not cost justify replacing its policy administration system. That was correct, but only for so long as the inertial business model holds. Should that carrier find itself in a more dynamic business environment, its systems absolutely will hamper its ability to compete effectively. The other implication of this analogy is one is left with only the inertial model in order to build the benefits case. Clearly a benefit case cannot be built on what-if scenarios.

Benefits usually can be classified in three categories. These are expense savings, increased profitability, and premium growth. Expenses savings often are the result of calculated headcount reductions in business operations. The basic rationale is if the system does more, then fewer people are required to do the remaining work (think straight-through processing, for example). These types of savings sometimes fail to materialize as carriers choose to "write more premium with the same headcount" rather than lay people off. Increased profitability often comes from reduced "leakage." Examples of leakage include uncollected earned premium, claims paid in error, and policies written for inadequate premium due to lack of information. Premium growth comes from better agent support, higher retention, and greater product range, all of which can be enabled by modern systems. Let's assume the team doing the benefits analysis comes up with achievable numbers. The key question then becomes: "Will the carrier company make the operational, organizational, and process changes needed to realize these savings?"

My suspicion is the realization of the numbers found in most CBAs falls significantly short in many instances, either because the numbers were unachievable in the first place or because the carrier failed to make the choices required to realize the benefits. There is the human tendency to "look on the bright side," the perceived political pressure to "make the numbers work," the gut intuition that replacing the systems is the right thing to do, and finally as more than one cynic has confided, by the time results can be measured, so many other variables will have changed that isolating the benefits that flow from the system change will be very difficult to do.

There is no doubt the CBA is an inexact and imperfect process. It's a necessary and important validation step along the path to transformational systems change. No board of directors is going to write a blank check for a mysterious outcome. The objective here has been to point out that as an industry we need to understand the limitations of how we currently approach the cost-benefit analysis and work to improve our process, product, and outcome.

But even if this last paragraph is true, we still are left in the position it is better to do the right thing for the wrong reasons than to do nothing at all. And no CBA is a waste of time or effort but is rather a difficult process that should provide useful insights into project costs and current operational inefficiencies, even if it sometimes yields questionable outputs.