Aon Benfield says the insurance industry has recently witnessed evidence of models both underestimating and overestimating loss estimates.

The firm says that models overestimated the actual incurred loss in last year's Chile earthquake, while they underestimated losses for U.S. hurricane events.

In its 2009 study, Aon Benfield says that, using the RMS version 9.0 model, actual losses were approximately 1.4 times the model estimates for personal and small commercial-lines business, translating into a 40 percent model miss. Large commercial-lines businesses were over two times model estimates or 100 percent model miss.

Dan Dick, executive managing director of Aon Benfield Analytics, tells NU Online News Service that the study found this issue with all modelers. RMS was used as an example. He says it was not a question of the models being incorrect, but that the losses did not include exposures to risks such as business interruption, flood, or loss-adjustment expense that would drive the ultimate loss figures for an insurer.

Earlier this month, reinsurance brokerage firm Holborn said modelers were significantly underestimating market losses for the March 11 Japan earthquake and tsunami because some coverages were not included in loss estimates.

"For us it is not an indication of us saying the modeling firms are wrong, [but] we recognize that there are many areas of uncertainty within the modeling process and we are asking the modeling firms to be more forthright in putting forward what they are trying to model and what they are not trying to model," says Dick. "It is letting insurers understand how much of a gap there is in what the modelers are not trying to model, how that could affect [the carriers], and [wanting them to] make adjustments for that as well as any other potential adjustments that they feel need to be made to best reflect their risk."

Under Solvency II, Europe's set of regulations for the financial-services industry, insurers using models will "need to be able to demonstrate a robust understanding of the inner workings of the 'black box' models." Dick explains that the black box is the variance in what the models do measure and what they fail to measure.

"Model evaluation is a crucial part of ensuring a company is adequately capitalized to meet its catastrophe exposures," says Paul Miller, head of International Catastrophe Management at Aon Benfield Analytics, in a statement.

"There has long been appetite for more transparent catastrophe models as components remain hidden from the end users," says Ben Fox of the model evaluation team at Aon Benfield Analytics.

He says the new European regulations further drive the need for information, and until there is information available that details what goes into construction of the models, making additional evaluations of exposure will be necessary.

Aon Benfield says its team will review insurers' models to assess the reliability of loss estimations and identify the strongest model for each territory and peril.

The process involves:

• Initial overview of the new model loss estimates with a high-level review of the model changes.
• In-depth analysis of the key components: the hazard, vulnerability and financial loss.
• Benchmarking individual insurers against the industry and advising on reasons behind any deviations.
• Knowledge sharing with clients to assist with their discussions with regulators and rating agencies.