Richmond, Va., is a little over an hours drive from the coastal cities of Norfolk and Virginia Beach. Most people would consider Richmond far enough inland so that worries over hurricanes or tropical storms wouldnt cause a run on plywood at Home Depot. That is until Hurricane Isabel struck the Virginia coast in September and pushed high winds all the way to Richmond. Jeff Fehn, director of IT for Richmond-based Virginia Farm Bureau Insurance, figured heavy rains would be the biggest problem for his city. At least thats what happened during past hurricanes. But for some reason Isabel was different. We got a pretty good brunt of it, he says.

Fehn and other insurance IT leaders may be shocked when huge storms hit their area, or when an electrical blackout occurs, or even when a virus attacks their systems. They no longer can afford to be surprised, though. Too much is riding on the data that sits inside their data warehouse. These disasters serve as a warning to insurers in other parts of the world, as well: You never know when trouble will arise, so take some steps to prepare for the worst.

When you are talking about the blackout, I think a lot of people were unprepared for a disaster that affected an entire region, says Mark Vanston, a program director for META Group. A good example is a lot of data centers have dual power feeds, so if one went down, they thought they were protected by a secondary source. The energy blackout that blanketed the Northeast section of the United States and parts of Canada on Aug. 16 made that strategy moot.

Power Is Out
In the aftermath of Isabel, power was out in Richmond for quite some time, but Virginia Farm Bureau was ready to go with two diesel-powered generators. The building had power, but not enough to fuel the air conditioning for the next 10 days, except in the data center. That was a minor inconvenience, Fehn says. The data center chillers ran just fine. That probably was the most popular spot in the building.

GHS Data Management is a claims processing outsourcer for pharmacy claims. Located in Augusta, Maine, GHS escaped the blackout, but the fierce New England winters have hardened businesses there to the realities of natural disasters. In 1998, we had a huge ice storm, and there were people out of power for a few weeks, says Jason Skeffington, GHS data warehouse ad-ministrator. We have all our servers and mission-critical systems on a large UPS that is supplemented by a backup generator. If the power was wiped out in this area for days or weeks, we still could operate, including our decision-support system, our data warehouse, and our Web server.

What to Do
The typical steps taken when a data warehouse is designed is to build duplication or redundancy into the architecture, according to Ron Barker, senior principal and insurance practice leader for Knightsbridge Solutions, a data warehouse and business intelligence consultancy. Companies will need a backup and disaster recovery plan in place for each of their operational systems, which will be some strategy to back the data up nightly, weekly, monthly, and annualized backups, Barker says. They either will store the files physically off site in the event of a disaster or have a hot standby operational data center, he says.
In the case of the big blackout, though, some insurers were relying on third-party vendors to protect the data and found that wasnt a great solution if the outsourcer also was affected by the power failure. A lot of people thought they were protected through their outsourcing contracts, says Vanston. Networks were down, power sources were out. You couldnt even get to the secondary providers.

Diesel Generators
Diesel generators have become a popular disaster recovery tool for insurers. The majority of the major players will have a diesel backup of some facility that allows them to run their major critical systems, says Barker. They may have 12 hours of hot standby through battery storage capability, and they may have enough fuel on site to last them for a week to run the diesels and generate the power.

Not all diesel generators passed the test in the blackout. The problem was a lot of them werent maintained, so they wouldnt start, says Vanston. Companies projected the amount of power they need, and they bought appropriate diesel generators and put in the correct supply limitations. The problem was they didnt do any load testing. A lot of companies started a big generator, ran it through the wires, and it just blew a bunch of circuits because it wasnt separated correctly. There are a whole bunch of minor things people didnt test until the actual disaster.
The generators worked fine for Virginia Farm Bureau. The building is wired so all the PCs can run off a generator, says Fehn. We have specially marked plugs.

Some less-fortunate businesses learned several lessons from the blackout. Specifically, computers need electrical power. A lot of people didnt know what to do because their processes or calling trees were stored on a central server, and they couldnt get the information, says Vanston.

Go to the Tapes
Both GHS and Virginia Farm Bureau make backup tapes that are stored at a secure facility away from the home office. We have tape backups stored on site and off site in fireproof cabinets, says Skeffington.

Fehn says Virginia Farm Bureau has a warehouse facility in another part of Richmond where it can operate in the event of an emergency. Weve configured it as a hot site, he says. If [the home office] was eliminated or rendered inhabitable, our contract with our network provider allows us to transfer all the data lines to a couple of T1s that have been pulled to that site. We have a contract where well have some servers dropped in, take backup tapes and load them onto those systems, and kind of hook everything together. Hopefully, within a short period of time, were back up and running.

Happily for Virginia Farm Bureau, Isabel did not damage the home office or the warehouse site, but the storm damage was so widespread in the community the company is considering moving the backup site further inland to Roanoke, in the western part of Virginia.

If we have a fire [at the home office], its not going to be an issue for our warehouse site, says Fehn. A tornado? Probably the same thing. A hurricane? Its big enough. [Isabel] wasnt a nice, straight, one-mile line. It pretty well knocked over everything in its path, and its path was huge. The western part of the state didnt take the brunt of the wind, though.

Synchronize Your Data?
One result of the blackout was it made some companies believe they were wasting money on having synchronized data stored by an outsourcer. That really teed off a lot of people because they spent a ton of money having these protection contracts that couldnt be satisfied, says Vanston. They pay a bunch of money, and when they cant get restored, they realize that money could have been used for other things.

Companies will be looking at different scenarios in the wake of the blackout, Vanston asserts, including doing things in house, such as building a secondary data center that can be used multiple ways. They can do app to app, testing, quality-assurance-type approaches on the secondary site and have it there in case of a disaster, he says. Its been kind of a fundamental shift instead of automatically outsourcing because its easier. I think people are looking at it from a view of if they had two architectures, could they do different things?

One problem is, due to fiber channel limitations, synchronous data replication cant be done beyond a 50- to 60-mile area, according to Vanston. So now you have this widespread disaster, and people consider maybe they dont need synchronous data replication; maybe they should start building architecture and applications from an a-synchronous-type approach. Then you could have geographically separated data centers.

There likely would be data-integrity and quality-of-service issues if information is sent by packets to another data center. But the reality is maybe our protection level changes, he says.

Money Drives Data
Whatever insurers do to develop a disaster recovery plan is going to be expensive. The investment for physical plant backups, such as diesel generators, is not a trivial number, says Barker. Vanston agrees but sees cost concerns being tempered somewhat. Cost drives everything, he says. But at the same time I would argue a few years ago 90 percent of the decision was based on cost. I think thats probably down to 70 percent now.

The insurance industry has an advantage over other financial services businesses because insurers dont view their data in a zero-latency environments, unlike securities and banking industries, where huge dollar amounts flow across the system in any given period of time.

If [securities or banking businesses] are down for 30 seconds or a minute, that can be an extremely expensive experience, says Barker. In the insurance industry, while a lot of carriers will tell you their desire is to get to zero latency, the truth of the matter is they probably are fine with one day.

That is because many insurance systems are monthly driven. The objective is to drive [the date] from a monthly cycle of refresh in the warehouse down to two weeks, or one week, or in some cases down to a day, says Barker. It depends on their focus around customer service and how closely they track transactions against a client. I would say the majority of insurers are in that one week to one month cycle range to refresh data in a warehouse.

Barker doesnt mean to imply insurers can afford to go a month before knowing a policy has been paid or a claim has been accepted. But is it critical to the business if the data is not available for a period of a day or two or three? he asks.

Man-made Disasters
Another kind of disaster insurers are preparing for is the attack on their data warehouse or their systems by a hacker or through a virus. An electronic hit really can throw an organization into a spasm, says Barker. It can bring you down for days, even weeks.

Risk managers might monitor power usage and weather reports quite often, but a carriers systems have to be monitored every second of the day. If a virus gets through your firewall, it can spread through your system in a heartbeat, says Barker. Once you get inside a secure network, the controls are off. Once you get inside the firewall, [the system] trusts you.

When an attack occurs, the insurer has to react quickly to the emergency. The different monitoring systems usually page key IT staff, says Skeffington. If something actually gets through [the firewall or virus scanners], a fast, planned, and careful response has to be ready ahead of time. You cant just go running around with your hair on fire trying to punch it out. Thats just not going to be prudent. Thinking it through ahead of time is key, and then having the staff in place. Even though it might be a large expenditure, redundancy is a huge saver if something does go south on you.

Time to Recover
Some disasters are easier to recover from than others, Barker says. Sometimes its as simple as quarantining the virus on the various systems, he explains. If its a damaging virus, it can mean either format the drive and start again fresh with a clean drive or, in some cases, you actually have to replace the PC because its rendered inoperable.

Insurance carriers are better prepared than some industries, he believes. There isnt an insurance company out there that doesnt have somebody whos responsible for the security of the data, says Barker. Then there are others that have extensive departments that do nothing but watch e-mail traffic, content, viruses, and so on. They manage the networks carefully.

Have a Policy
Fehn knows its a clich, but he says he learned firsthand from Isabel: Have a plan. He never had seen a hurricane travel so far inland before Isabel. Now he is a believer. Its one of those things you think cant happen to you, and sure enough it can, he says. Plan the best you can, and go through the process of regularly reviewing and fine-tuning it.

Is there such a thing as going too far in efforts to protect data and ensure its availability to users? We havent gone overkill yet, says Skeffington. The only time people decide its not overkill is when they get hacked and they find out they needed all those [protective] steps, he adds.

The only real overkill is to unplug the wire and manually grant access to the data, he continues. That, in my opinion, would be overkill. But any automatic system that you can put in place that monitors, that blocks, that scans for virusesI dont think any application that can do those jobs is overkill.

Vanston says IT has the charge of carrying out the plan, but the policy for disaster recovery and business continuity has to be developed by the business side. You want to do a business impact analysis and understand what doesnt need to be protected, he says. Its not just a case of lets protect everything. Its a question of where you really want to put the good dollars. Its a fairly monotonous and grueling process. Thats why you need high-level business people to sign off on it and say, We cant protect everything, so lets protect these three or four things.

Data Mining, Data Warehouse Tech Guide

ADP Claims_Services Group
San Ramon, Calif.
800-237-4968
www.adpclaims.com

Axiom Software Laboratories
New York, N.Y.
212-248-4188
www.axiomsl.com

Cargofound
Auckland, New Zealand
+64 9 817 4421
www.cargofound.com

CSC Financial Services
Austin, Tex.
800-345-7672
www.csc-fs.com

Data Advantage Group
San Francisco, Calif.
415-469-4300
www.dataag.com

DataQuick
San Diego, Calif.
888-604-3282
www.dataquick.com

DecisionPoint Applications
San Mateo, Calif.
800-970-5372
www.dpapplications.com

Delphi Technology
Cambridge, Mass.
617-494-8361
www.delphi-tech.com

Depository Trust & Clearing Corp.
New York, N.Y.
212-855-5651
www.dtcc.com

E-Z Data
Pasadena, Calif.
800-777-9188
www.ez-data.com

Fair Isaac
San Rafael, Calif.
415-472-2211
www.fairisaac.com

Fiserv
Pittsburgh, Pa.
412-577-3000
www.fiserv.com

Focus Solutions Inc.
Fort Washington, Pa.
215-643-9300
www.focus-inc.com

IBM
White Plains, N.Y.
800-426-4968
www.ibm.com

Insight Decision Solutions, Inc.
Markham, Ont., Canada
905-475-3282
www.insightdecision.com

Insurance Information Exchange
College Station, Tex.
800-683-8553
www.iix.com

Magnify
Chicago, Ill.
312-384-7161
www.magnify.com

MicroStrategy
McLean, Va.
703-848-8600
www.microstrategy.com

Millbrook, Inc.
Bethlehem, Pa.
610-867-7400
www.millbrookinc.com

Miningham & Oellerich, Inc.
New York, N.Y.
212-349-4410
www.serveme.com

MIS AG
Newark, N.J.
973-679-0724
www.mis-ag.com

NCR Teradata
Dayton, Ohio
937-445-5000
www.teradata.com

OUR-HR, LLC
Greenville, Miss.
663-334-6919
www.our-hr.com

Pinpoint Solutions, LLC
Livingston, N.J.
973-716-0700
www.pinpnt.com

Priority Data Systems
Omaha, Neb.
800-228-9410
www.priority-data.com

ReClaim Technologies
Newark, Ohio
740-344-6956
www.reclaimtech.com

Risk Laboratories
Marietta, Ga.
678-784-4600
www.risklabs.com

Sagent Technology
Mountain View, Calif.
650-815-3100
www.sagent.com

Thazar Solutions
Frisco, Tex.
972-377-1110
www.thazar.com

Universal Conversion Technologies
Addison, Tex.
214-348-2000
www.uctcorp.com