How to overcome AI shortcomings to benefit your insurance business

In the past year, business leaders have considered how to use artificial intelligence to its maximum potential.

Leaders in the insurance industry have a secondary consideration: How can we do this without negative impact, operating in a highly regulated and complex industry?

The answer, for many, is using existing large language models (LLMs), such as GPT-4, combined with your company’s proprietary data, to bring dramatic change to customer service operations while saving millions of dollars. And you can do it without hiring an AI expert team of data scientists and engineers.

I’ll explain how using one case study of a preexisting, small team that did this by launching a “super-chatbot” at Jerry.

But first, you shouldn’t set out to use AI simply because you feel like you should. You need a good use case.

Ask yourself two questions:

• What customer problem are you trying to solve?
• What business problem are you trying to solve?

In our case, both answers had a common denominator: Reducing customer response wait times. In April, before we deployed our super-chatbot, our human agents handled 100% of messages received from about 100,000 unique customers a month. We were able to reply to only about half of our customer messages within 24 hours.

Two months later, thanks to the chatbot, we replied to 93% of customer messages within 30 seconds, and only 30% of all messages needed to be escalated to human beings. By August, the volume of messages requiring human response had dropped almost 90%.

But how do you ensure that chatbot is providing accurate information when addressing customer needs?

This is a barrier that may be stopping many insurance professionals from using AI That’s because with an average chatbot, there’s a tradeoff between the breadth of the domain knowledge you can feed it and the control you can have over its responses. With the consumer-facing version of GPT-4, for instance, this often results in “hallucinations” as it draws on its enormous knowledge base (read: the Internet).

Consider national insurance brokers, who need to answer customer questions on an array of topics. They have customers in all 50 states and offer multiple lines of insurance. The amount of unique customer questions is essentially endless. And given the nature of insurance, we can’t afford hallucinations.

Here’s a three-step solution:

1. Build different “sub-agents,” each focused on a narrower domain, such as providing details about a state’s coverage requirements or how to make a payment.
2. Create extensive testing tools to eliminate hallucinations and ensure they aren’t re-introduced in subsequent versions of each sub-agent.
3. Finally, build an action framework so that the virtual sub-agent can take actions on behalf of a customer in addition to answering a customer’s questions.

Before you start building, consider these keys for success:

• Get good at writing prompts. The answers you get from an LLM will be directly correlated to how good your prompts are. If writing prompts is new to you, take the time to learn ongoing and tap subject matter experts, like your insurance agents.
• Iterate quickly. Build and test, again and again. GPT-4 can be unpredictable, especially for newer users. Iterate quickly and often to learn its idiosyncrasies.
• Invest in testing. Monitoring a chatbot’s performance without reading every single message thread is challenging. Build processes that showcase how changes to prompts affect answers to real customer questions. We read 1,000 messages a week and manually labeled responses, then consolidated findings to focus on the highest-priority items. This gave us an intuitive sense of how the LLM responded to prompt changes.
• Version control is critical. This enables you to quickly launch versions of sub-agents, test them, and revert to the previous version if necessary.
• Start small and grow. We started with a four-person team, including three senior software engineers. We maintained that size through most of the development, and eventually expanded to about 12 people.

Following these guidelines from the beginning will result in a chatbot that has a strong, scalable infrastructure. You’ll be able to continue improving your chatbot and growing your AI. capabilities.

Potential impact

If you focus on concrete problems, like customer wait time, and stay within your capabilities, you can dramatically transform customer experience. In our case that meant:

• A projected $4 million in annual savings by the end of 2023 (one year post project launch)
• A 2%-5% sales conversion jump and a similar increase in retention of existing customers

That’s what I call putting an LLM to work.

John Spottiswood (john@getjerry.com) is chief operating office for Jerry, an auto insurance comparison shopping platform. He is responsible for executing business strategy and operations; marketing, product development, data, partnerships, administration and customer experience; and expanding service offerings into new fintech and automotive categories.

See also:

• ChatGPT and the fight against insurance fraud
• Exploring the buzz around generative AI for insurers
• The insurance AI revolution: Challenges and opportunities