Everybody seems to be falling in love with bots. Bots are easily the hottest topic with clients running contact centers and customer care centers right now. The apparent speed of implementation, the promise of increased deflection, and the ability to provide something innovative for customers are big draws. But not everyone implementing these bots will have a long-term relationship with them.

A few months ago, my colleague Krishan Rajaram called out the speed trap in building bots: that the ease of leveraging chat bot platforms is leading some organizations to build poorly thought-out virtual agents with little value. The platform economy is certainly putting powerful tools into our hands and, as with any tool, we need to learn how to use them effectively. One place to start learning about bots is to understand how to measure their success.

Pay attention to the deflection rate
One of the key success factors for deploying any self-service technology is its deflection rate. I’d argue that this isn’t the primary success factor but most CFOs are likely to disagree so we’ll stick with it as the measure to dig into here. Deflection rate – note that this is the rate of live agent chat deflection, not deflection across all channels – is a function of three key measures of the chat bot: intent coverage, knowledge delivery rate, and knowledge success rate.

Understanding customer intent
That first measure, intent, is the reason a customer comes to you for support: customers wanting to reset their devices or change their phone numbers are examples of intent. One of the biggest issues with quickie bot building is that they don’t have very broad intent coverage. A bot that only knows how to answer a few questions will need to escalate to a human agent much more often than they deflect. To build a successful bot you’ll need to gather deep insight into why your customers are contacting you and make sure that your bot can cover a large percentage of those inbound requests.

The knowledge delivery rate
The second measure is what Microsoft calls the knowledge delivery rate. This is the ability of a chat bot to understand a customer’s stated intent. Simple chat bots can understand straightforward questions and be successful in answering them but as the vocabulary for your problem space becomes more complex it becomes more difficult for the chat bot to understand customer intent. This problem isn’t unique to bots. Human agents sometimes can’t figure out what a customer wants or needs either. The knowledge delivery rate for bots is largely a function of the natural language processing (NLP) backing up the bot. Success in this area is largely about choosing a good bot platform.

The knowledge success rate
The final measure is the knowledge success rate. This is the rate at which a bot actually solves a customer problem. And, just like with the knowledge delivery rate, failure isn’t unique to bots. If customers contact an agent multiple times regarding the same issue, it means the agent has failed to deliver knowledge successfully. You have tremendous control over this measure’s success by building bots that provide a great user experience. Bots that provide links to information in a knowledge base aren’t going to be as successful as ones that present information within the conversation itself, and a bot that can directly handle a customer’s intent, like updating the phone number on their account, will be more successful still.

All three of these measures can be expressed as a percentage, and your bot platform should be able to report on each of them in real-time. To get your deflection rate, simply multiply each of the measures together to get a percentage of traffic deflected from your chat agents. A bot that covers 75% of your customer intents, has an 80% knowledge delivery rate, and a 50% knowledge success rate will give you a 30% deflection rate.

Maintaining a long-term relationship
Companies on a journey to build an Intelligent Enterprise are building bots that can act as agents, removing the barriers between application silos so they act on behalf of customers. Just like great human agents, bots need monitoring and training. While deflection is arguably not the most important measure of bot success, understanding how to calculate that measure offers great insight into bot platforms and ensures that you know where to improve your bot by adding intent or increasing its ability to act on behalf of your customers. So, feel free to love your bots but be sure you’re setting yourself up for a long-term, satisfying relationship.

To learn more, listen to me and other panelists on a recent contact center-focused webinar from ICMI entitled The Future Is Here: What You Need to Know About Machine Learning and AI