"The changes occurring in business today inevitably require some form of data migration – the process of moving data from an existing system to a new replacement system. Data migration is driven by business mergers and acquisitions, movement from legacy systems to packaged solutions (e.g., Lawson, Oracle, PeopleSoft, SAP, Siebel, etc.), implementation of data warehouses and data marts, application system expansion, web access to legacy systems, movement to outsourced and ASP solutions, as well as many other system development projects."

Cost of migrating data is often higher than anticipated 

Sadly, the actual cost of migrating the data for these projects is frequently much greater than was initially anticipated. Large data migration and ETL projects often fall into the vicious cycle of "Code, Load & Explode." The symptoms of Code, Load & Explode will sound familiar to most implementation teams: develop programming logic—run a test load—the load fails—discover that data assumptions were incorrect and unexpected data quality issues exist – react by cleansing data and changing specifications—delay to adjust programming—re-test the load—the load fails—find additional incorrect data assumptions and additional unexpected data quality problems . . . react again . . . delay again . . . react again. . . delay again . . .over and over. In all too many data migration projects, the vicious cycle of Code, Load & Explode goes on, and on, and on, increasing the cost beyond anyone's worst imagination, and often jeopardizing the success of the entire implementation.

The other issue is that there are a plethora of tools on the market to assist with migrations and conversions, but the tools are generally very hard to use, don't provide many insights into how to achieve the best results and are only as good as the analyst or programmer that is using them. Data migration is a very complex venture and it is one that most vendors cannot honestly say that they have expertise in, even if they have conducted thousands of migrations.

(Photo: Shutterstock)

The main reason is data is different, even if database structures are not. There is also a great deal of unstructured data, and there are many algorithms that have to be converted to work with the new system. The normal approach has been to deduplicate the data, then map it field by field, look for inconsistencies and then the true ETL (Extract/Transform/Load) process begins. While many companies attempt to set a timeline on completion of these types of projects, there are a great many variables that inevitably cause the vast majority of migrations to fail. 

It's very important to understand that a complex migration consists of many variables. The old adage of garbage out equals' garbage in has unfortunately reared its ugly head in many a migration. 

Companies that are planning migrations from an old legacy system probably face less of a challenge because in most cases, if it is a home grown system, the IT staff may know and understand the product really well, which is half of the equation. But the harsh reality is that most systems are not home grown. In many cases you are pitting the incumbent against the new kid on the block, and each holds one half of the equation. The other issue is that ETL tools that currently exist are really not built for the specific use case of dealing with migrations. 

Weigh enterprise risks associated with migrating

Business does not stop when you have systems to migrate. You have to balance your company's growth strategy and current initiatives with the fact that you may be dealing with the ramifications of the migration for months, and in many cases years to come. Companies that do not weigh the enterprise risks associated with migrating from one source to another are certainly brewing a recipe for failure. 

Migrations are generally about records, not necessarily individual pieces of data. When you look at the commonalities between a record in one source and migrating that record to another structure it does not necessarily seem difficult at first. I mean there are commonalities in every policy for example. You will have an insured, a primary address and potentially some common coverages. 

Mapping the commonalities is usually a pretty good strategy because then you can migrate what is normal and only focus on what is different from one record to another. But if you have varied lines and multi-state policies and claims to migrate, it may become very complex and more prone to failure.

(Photo: Shutterstock)

The other key is cooperation from both parties. Normally if your system is proprietary, you have a great advantage because your IT and Business Analysts will in most cases have very in depth knowledge about the way the system is built and the algorithms and data structure, but if it is an incumbent ceding to a new system or systems, the transition can be more difficult. 

Other considerations are the impact on reporting, auditing and potentially data warehousing. Migrations can put a damper on corporate strategy as it becomes difficult to focus on expansion when it directly impacts the migration. Companies that migrate may be putting their plans on hold as unexpected and unplanned changes may directly affect the corporate strategy for months to come. 

Questions to ask for a successful migration

If companies want to overcome the pitfalls of a potentially prolonged and painful migration, knowing the incumbent system, algorithms and having a very good handle on how a migration will effect corporate strategy is a must. Some of the questions you want to ask yourself and your new provider as part of a well-planned and successful migration are:

• How many migrations have you had that failed?
• What is your strategy for our success? 
• How is the data being migrated? (Lazy Migration, Continuous Migration or incremental migration)
• How much do you know about the incumbent system? 
• What level of cooperation will you get from the incumbent? How will it affect the timeline?
• How long will the migration take? 
• What is the strategy to keep the project on time if there are "change orders?"
• Do I have the expertise in house to manage the migration? 
• Can I allocate resources from my pool of talent to focus on this project for its entire duration?
• How are you assured the conversion/migration was successful?  Meaning, every claim, policy, etc… was migrated properly?
• What's the strategy to migrate your users and business processes?
• Do you plan to update your business processes to take advantage of new technology?  Automate manual processes?

Try to get a full customer list from the potential provider. Don't let them "cherry pick" who you speak to. Have your team fully document the results in a risk assessment and then weigh the chances of success or failure using a point system of high to low risks. 

Consider bringing in independent experts

Finally, look outside of the incumbent, your talent pool, and the new provider for migration experts. There are companies that focus solely on migrations and are experts, and they have seen it all. If it makes sense, bring in independent experts to evaluate your provider and help to craft a strategy that makes sense for your business. I have personally worked with a variety of companies that have failed in some aspect or another because they did not weigh the need for a detailed strategy for their migration.

Most companies assume that the provider is an expert, after all they are selling the software and should know how migrations work right? Wrong. As data becomes more complex and less structures, you need experts who can navigate the complex nature of a migration and insure your success. 

Migrations don't have to be prolonged or painful. If you are an insurance carrier facing a migration or just want to find ways to automate workflow processes as they apply to daily interactions with content, data and reporting, feel free to contact me.