Setting the expectations

Before unleashing your creativity in the bot-building process, specific components of the chatbot project need to be prepared. Setting the right expectations will play a significant role in the long run by guiding and aligning the stakeholders and the project itself.

Having a solid understanding of the application potential of Certainly's platform solutions allows for precise, unambiguous, and transparent project development that aligns with the company's business goals and strategy. Assessing whether the chatbot fulfills that purpose requires an explicitly defined measurement of success. Success measurement can be designed with a set of Key Performance Indicators (KPIs), either tracked directly on the platform or extracted from it.  

This article will help you define expectations and success by understanding the project's purpose and monitoring its performance. We will cover:

Making the chatbot project part of your business strategy  

When driving a chatbot project, it is crucial to determine the viability and effectiveness of such an endeavor. Even though the chatbot can function as intended from the start, an initial question must be raised to examine: “what is success”?  

The success of the chatbot project should tightly align with the business strategy and create value by resolving an issue or supporting business activity. Defining the scope and purpose of the chatbot allows for further technical and use case development, as well as success specification. By clarifying what these factors mean to you, a solid foundation of expectations will be developed and smoothly maintained over the project lifetime. 

Certainly Platform as a strategic tool  

In order to help you solve problems, deliver results, and reach business goals, Certainly's platform and services should serve as an excellent strategic tool. The ability to build a DIY bot allows for endless bot types and use cases. Once you summarize your needs and understand how the chatbot can support them, you'll be ready to determine the bot's action plan. We distinguish between two technical designs of chatbot that you can build: classic rule-based or the popular AI-fueled bot. 

  • The traditional keyword/rule-based bot leads the end user through a set of potential answers down the decision tree without allowing the end user to depart from the predefined flow. 
  • An AI chatbot comprised of Natural Language Understanding (NLU) technology grants a more dynamic interaction with the end user. It supports free question/answer by recognizing the intent behind the input.

We encourage you to combine the rule-based logic with the AI to get the best end user experience, optimal conversation flows, and rich data sources. 

Since the platform provides the freedom of creating a chatbot from scratch, there are multiple types of bots you can create and departments you can cover: 

Chatbot Type 

Purpose 

Business Department Activity 

Basic Support Bot 

Solves specific problems and guides to/ through a process with the end user's additional information. Can act as an intelligent assistant by answering FAQs, understanding end users' needs with the help of NLU, and/ or linking to human support. Often, integrations to third-party systems provide the bot with additional information.  

Sales, Customer Service/Support, Ecommerce 

Basic Support Bot: Transactional Element 

A transactional element can execute certain transactions- arranging a meeting, completing an order, making a reservation, and much more.  

Ecommerce, Sales, Customer Success 

Basic Support Bot: Informational Element 

Information provision feature by searching a specific knowledge base/ internet. Can also collect and map data about end users.  

Human Resources, Management 

Digital Twin Assistant Bot 

Has the essence of a Basic Support Bot but performs significantly better in conversations and makes interactions even more human-like. Acts as a proactive assistant by understanding end users' needs based on their input and browser/ in-app activity and dynamically interacting with the website content. 

Conversational Commerce, Ecommerce, Sales, Customer Success, Increase of products sold

Alongside chatbot types and applications, here is a list of typical metrics corresponding to those types:

Chatbot Type 

Recommendation for Typical Key Performance Indicators (KPIs) 

Basic Support Bot 

  • Fallback rate 
  • Ticket- and handover-related metrics 
  • Retention rate 
  • Recognition rate (NLU) 
  • Lead generation rate 

Basic Support Bot: Transactional Element 

  • Goal completion rate 
  • Bounce rate 

Basic Support Bot: Informational Element 

  • Customer satisfaction rate 
  • Number of new FAQs 

Digital Twin Assistant Bot 

(In addition to KPIs of Basic Support Bot 

  • Customer satisfaction rate 
  • Bounce rate 
  • Retention rate 
  • Bot-enabled purchases 
  • Browse-to-buy ratio 
  • Time before purchase/ goal completion length 
  • Cart abandonment rate 
  • Ecommerce return rate 

A detailed list of suggested metrics and their descriptions can be found here. By keeping these essential advantages and limitations in mind while designing the bot, you build an environment where success requirements are clearly defined, and expectations can be easily communicated. 

Internal and external communication of expectations

Once you have determined its type and purpose, it is time to design the bot. While doing so, keep in mind which of the Modules you are going to continue observing as performance indicators.  

For instance, you can identify an end user's visit to a specific Module in the flow either as a positive/ successful or negative/ unsuccessful outcome. For example, at the end of the conversation, the end users need to state whether they were happy with the answers the bot provided or not. The current Module the end user is visiting asks: "Are you satisfied with the provided answer?" If the end user says yes, the conversation can be marked as a positive outcome and vice-versa. 

The idea behind this is to capture the desired flow of conversation while still avoiding complexity that could hinder direct and easy observation of the indicator. By clearly establishing internally which Modules are identified as successful, you ensure that all project participants understand what should be focused on and how return on investment (ROI) can be defined. 

Channel, software, and platform integrations should also be considered when determining performance indicators, as they add a layer to the project dynamics and complexity. When these are taken into account, the bot itself could affect other products with which it is integrated and act as a source of data. This data could then be combined with the data extracted from other products.  

The so-called indirect metrics that result from combining data from the bot and other software are primarily quantitative. They will usually be related to amounts and totals that could be easily automated. While direct metrics are connected to both quantitative and qualitative data obtained from the Certainly Platform itself, the qualitative data relates to the content of the messages that the bot receives through conversation with the end user and will most likely remain manual to some extent. 

Once an internal agreement has been reached between the project leaders and participants, the success and KPI definitions should be communicated to Certainly. The Certainly team will then provide support and help refine the chatbot project along the way.  

Metrics and KPIs for the success of the project 

Just as the level of chatbot customization on Certainly's platform is high, so is the number of potential KPIs. That is why it is essential to closely monitor your bot the first time it goes live or while performing a soft launch. You will be able to use its initial performance as a benchmark for further improvements of both the bot design and the KPIs you are tracking.

Below you can find some of the fundamental and frequently used metrics: 

Direct Metrics 

Description 

Fallback rate 

Number of times bot fails to understand end user's input 

Customer satisfaction rate 

End user's experience during bot interaction, satisfaction of the answer provided 

Recognition rate 

Number of times bot is able to understand end user's question/intent 

Handover rate 

Number of end users requesting handover to a human agent 

Ticket creation rate 

Number of times end users requesting ticket/e-mail creation 

Ticket deflection rate 

Number of times end users fail to be transferred to the human agent, but reject to create a ticket or send an e-mail 

Retention rate (Total count of end users, Total count of conversations) 

Number of returning/new end users 

Bounce rate 

Number of times bot initiates conversation, but end user leaves the page without interaction (empty conversations) 

Number of new Frequently Asked Questions (FAQs)

Pattern of end user behavior and questions asked of the bot, and based on that discover new FAQs 

Indirect Metrics 

Description 

Handover waiting time 

Average time end user waits to be transferred from bot to human agent 

Handover abandonment rate 

Number of end users who leave chat mid-handover process 

Goal completion rate (for specialized bots) 

Number of end users filling out a form, completing a purchase with the bot 

Bounce rate 

Number of end users entering a website who don't use or see your bot 

Lead generation rate 

Total amount of newly captured leads compared to amount of channel visits 

Browse-to-buy ratio 

Ratio of items purchased to items viewed 

Bot-enabled purchases 

Number of units bought with the help of bot 

Resolution rate 

Compare the time and amount of human agents' ticket resolution to the ones resolved by the bot 

Time before purchase/ Goal completion length 

Amount of time it takes to complete a purchase from the moment the chat was initiated 

Cart abandonment rate 

Number of initiated sales that never reach purchase/ transaction completion 

Ecommerce return rate 

Number of items returned compared to items sold- measures bot impact of product return 

The list of metrics above is not exhaustive and can be interpreted differently by each party. This is why it is essential that there is an alignment between project participants, both internally and externally.

The results can be expressed both in relative and absolute values, depending on preference. What’s more, different integrations will offer other insights into the way end users interact with your bot and what type of data is received.  

Certainly offers various ways of fetching the received data, through:

  • the KPI Platform, which displays cumulative and chronological values of Module visits tagged with KPI tags;
  • the Inbox, which displays chats between the end user and bot; and
  • Message Reports, which provide the most accurate view of the conversations and the data that was collected. The majority of metrics and KPIs are located in the message reports; however, calculations need to be performed to extract KPIs and rich information. 

The ways of approaching the chatbot project and measuring its performance are endless. It is up to you and your team to agree on the unified term of what success is and how to measure it. Certainly will always support you in defining those metrics and boost your bot in fulfilling the desired business purpose.

Ultimately, it comes down to creating scalable, omnichannel chatbots that efficiently and effectively support your activities, follow your strategy, and fulfill their purpose of making the customer happy. In summary, it is essential to:

  • understand different business applications and bot types, to which the chatbot project can be applied;
  • determine which needs or problems the chatbot project should solve or add value to;
  • design the chatbot while keeping in mind what would (and would not) be the desired conversation outcome;
  • define what success is and how you measure it (based on the factors above);
  • understand the importance of metrics and decide which ones fit your bot the best; and
  • become informed of different ways you can reach the data and apply measurements.