View your bot's performance data in Certainly's KPI Relationship Report

The KPI Relationship Report offers a quantitative overview of a given bot's performance using data collected by a set of predefined KPI tags added to the bot's Modules. In this dashboard, we provide visualizations of this data, definitions of key terms, and descriptions of what the visualizations show.

In this article, we'll discuss the following:

This is simply the first iteration of this feature, and we welcome your feedback as we continue to improve the KPI Relationship Report based on customer needs.
Introduction to the KPI Relationship Report

The KPI Relationship Report allows you to visualize the number of Conversations, Fallbacks, Handovers, Bot Response Evaluations, and Bot Performance Evaluations. We'll explore a few examples of these visualizations below.

For instance, you can view the bot's daily traffic, by hour, to identify the most pressing potential needs for human support:

botdailytraffic.jpg

You can also determine how many conversations the bot has handled independently (without third-party involvement):

botconversations.png

Please note that we exclude all empty conversations (those in which bot users have not sent any messages to the bot) from the dataset that populates this report to ensure that we measure only conversations with expressed needs.

Predefined KPIs

We use KPI tags to collect the data that populates the KPI Relationship Report. Adding them to Starting, Fallback, Handover, and Evaluation Modules allows us to measure the number of users that have been through the various paths of a bot.

The KPIs identified below are predefined by Certainly to capture data at critical points in the conversational flows.

Starting, Fallback, and HandoverEvaluation and Customer Satisfaction (CSAT)
  • Start ['Start']
  • Explicit Handover ['Handover']
  • Implicit Handover ['KPI 13']
  • Fallback ['Fallback']
  • Evaluation Answer - Positive ['KPI 9']
  • Evaluation Answer - Negative ['KPI 8']
  • Bot's Performance Evaluation - Positive ['KPI 10']
  • Bot's Performance Evaluation - Negative ['KPI 11']
  • Bot's Performance Evaluation - Uncertain ['KPI 12']

If you don't see value in some of these predefined KPIs for your specific purposes, feel free to skip those. Any skipped KPIs will simply return "0" in the dashboard visualizations. Please note, however, that historical data will not be available if you decide to add them later, as the count for KPI tags starts when they're added to a bot's Modules.

How to access the KPI Relationship Report

Accessed via the drop-down Section menu, this dashboard can be easily added for any company upon request. If you're interested, please reach out to our Customer Success team.

We'll first check that your bot Modules and KPI tags have been configured according to the guidelines below, then we'll add the dashboard to the Section menu of your Certainly Platform view.

How to prepare your bot for the KPI Relationship Report

To benefit from the KPI Relationship Report, you must make sure that your bot is configured correctly. This may require changes to your bot's Modules and current KPI tag setup. Just let us know if you need help with the configuration!

Module configuration

Before setting your KPI tags, it's important that your bot has the corresponding Modules to catch those KPIs during the conversation. In this section, we'll discuss the specific Modules.

Starting, Fallback, and Handover Modules

When you build your bot, you'll have a default Starting Module on your bot's canvas. Note, however, that you can define a different starting Module if you'd like. This is important, as you will label it with the ['Start'] KPI tag later.

You will likely also want your bot to be able to address situations where it doesn't understand a user's request with a Fallback Module, and, in more complex situations, handle cases in which human support is needed with a Handover Module.

Handover Modules can be Explicit or Implicit. To help you decide how to use each type, we define them in greater detail here.

Not all requests can be standardized and added to the bot, and some users prefer to be handed over instantly. As such, it's valuable to present an option for the customer to connect with a human agent. Explicit Handover is when the handover is a selectable option for the user, like clicking a Suggested Reply, or they can request human support using a free text response.

In other cases, customers may not decide to be handed over on their own. They can end up in a loop, searching for something the bot hasn't been trained for. Implicit Handover is a configuration where handover is automatically triggered when the bot struggles to understand a user. This trigger is the clearest indicator of the bot's ability to handle conversations on its own.

Evaluation and CSAT Modules

To provide the best overview of a bot's performance, we highly recommended including opportunities for customers to evaluate individual bot responses as well as the overall conversation. Evaluations of the bot's answer to the end user indicate whether the bot is providing enough relevant information. Based on these results, you can investigate possible reasons for low performance and take action to improve your bot.

Similarly, the overall conversational evaluation assesses the user's perception of the bot's conversational performance (we can call this the customer satisfaction score, or CSAT). For example, at the end of a conversation, the bot user will have three options to choose from: Bot's Performance Evaluation - Good, - Bad, and - Unsure.

As noted before, you can omit any KPIs you don't feel are beneficial for your needs. If you don't wish to display, for instance, an "unsure" evaluation option to your customers, simply don't include a third Evaluation Module in your bot's configuration.

KPI tag setup

After you determine your need for these described Module types and you build the relevant Modules, you can use the list of predefined KPIs above to tag those Modules for the KPI Relationship Report data collection.

In an ideal case, the final basic bot configuration with relevant KPI tags will look similar to this:

mceclip2.png

As a reminder, the count for any KPI tags starts once they are added to a bot's Modules and that bot has been saved and published. Historical data prior to that point will not be included in the default KPI Relationship Report.>