Before unleashing your creativity in the bot-building process, specific components of the chatbot project need to be prepared. Setting the right expectations will play a significant role in the long run by guiding and aligning the stakeholders and the project itself.
Having a solid understanding of the application potential of Certainly's platform solutions allows for precise, unambiguous, and transparent project development that aligns with the company's business goals and strategy.
Assessing whether the chatbot fulfills that purpose requires an explicitly defined measurement of success. Success measurement can be designed with a set of Key Performance Indicators (KPIs), either tracked directly on the platform or extracted from it.
This article will help you define expectations and success by understanding the project's purpose and monitoring its performance. We will cover:
- Making the chatbot project part of your business strategy
- Internal and external communication of expectations
- Metrics and KPIs for the success of the project
Making the chatbot project part of your business strategy
When driving a chatbot project, it is crucial to determine the viability and effectiveness of such an endeavor. Even though the chatbot can function as intended from the start, an initial question must be raised to examine: “what is success”?
The success of the chatbot project should tightly align with the business strategy and create value by resolving an issue or supporting business activity. Defining the scope and purpose of the chatbot allows for further technical and use case development, as well as success specification.
By clarifying what these factors mean to you, a solid foundation of expectations will be developed and smoothly maintained over the project lifetime.
Certainly Platform as a strategic tool
In order to help you solve problems, deliver results, and reach business goals, Certainly's platform and services should serve as an excellent strategic tool.
The ability to build a DIY bot allows for endless bot types and use cases. Once you summarize your needs and understand how the chatbot can support them, you'll be ready to determine the bot's action plan.
We distinguish between two technical designs of chatbot that you can build: classic rule-based or the popular AI-fueled bot.
- The traditional keyword/rule-based bot leads the end user through a set of potential answers down the decision tree without allowing the end user to depart from the predefined flow.
- An AI chatbot comprised of Natural Language Understanding (NLU) technology grants a more dynamic interaction with the end user. It supports free question/answer by recognizing the intent behind the input.
We encourage you to combine the rule-based logic with the AI to get the best end user experience, optimal conversation flows, and rich data sources.
Since the platform provides the freedom of creating a chatbot from scratch, there are multiple types of bots you can create and departments you can cover:
| Chatbot Type | Purpose | Business Department Activity |
|
Basic Support Bot |
Solves specific problems and guides to/ through a process with the end user's additional information. Can act as an intelligent assistant by answering FAQs, understanding end users' needs with the help of NLU, and/ or linking to human support. Often, integrations to third-party systems provide the bot with additional information. |
Sales, Customer Service/Support, Ecommerce |
|
Basic Support Bot: Transactional Element |
A transactional element can execute certain transactions- arranging a meeting, completing an order, making a reservation, and much more. |
Ecommerce, Sales, Customer Success |
|
Basic Support Bot: Informational Element |
Information provision feature by searching a specific knowledge base/ internet. Can also collect and map data about end users. |
Human Resources, Management |
|
Digital Twin Assistant Bot |
Has the essence of a Basic Support Bot but performs significantly better in conversations and makes interactions even more human-like. Acts as a proactive assistant by understanding end users' needs based on their input and browser/ in-app activity and dynamically interacting with the website content. |
Conversational Commerce, Ecommerce, Sales, Customer Success, Increase of products sold |
Here is a list of typical metrics corresponding to those chatbot types:
| Chatbot Type | Recommendation for Typical Key Performance Indicators (KPIs) |
|
Basic Support Bot |
|
|
Basic Support Bot: Transactional Element |
|
|
Basic Support Bot: Informational Element |
|
|
Digital Twin Assistant Bot |
(In addition to KPIs of Basic Support Bot)
|
A detailed list of suggested metrics and their descriptions can be found below.
By keeping these essential advantages and limitations in mind while designing the bot, you build an environment where success requirements are clearly defined, and expectations can be easily communicated.
Internal and external communication of expectations
Once you've determined its type and purpose, it is time to design the bot! While doing so, keep in mind which of the Modules you are going to continue observing as performance indicators.
For instance, you can identify an end user's visit to a specific Module in the flow either as a positive/ successful or negative/ unsuccessful outcome. As an example, at the end of the conversation, the end users need to state whether they were happy with the answers the bot provided or not. The current Module the end user is visiting asks: "Are you satisfied with the provided answer?" If the end user says yes, the conversation can be marked as a positive outcome and vice-versa.
The idea behind this is to capture the desired flow of conversation while still avoiding complexity that could hinder direct and easy observation of the indicator. By clearly establishing internally which Modules are identified as successful, you ensure that all project participants understand what should be focused on and how return on investment (ROI) can be defined.
Channel, software, and platform integrations should also be considered when determining performance indicators, as they add a layer to the project dynamics and complexity. When these are taken into account, the bot itself could affect other products with which it is integrated and act as a source of data. This data could then be combined with the data extracted from other products.
The so-called indirect metrics that result from combining data from the bot and other software are primarily quantitative. They will usually be related to amounts and totals that could be easily automated. While direct metrics are connected to both quantitative and qualitative data obtained from the Certainly Platform itself, the qualitative data relates to the content of the messages that the bot receives through conversation with the end user and will most likely remain manual to some extent.
Once the project leaders and participants have reached an internal agreement, the success and KPI definitions should be communicated to Certainly. The Certainly team will then provide support and help refine the chatbot project along the way.
Metrics and KPIs for the success of the project
Just as the level of chatbot customization on Certainly's platform is high, so is the number of potential KPIs. That's why it's essential to closely monitor your bot the first time it goes live or during a soft launch. You'll be able to use its initial performance as a benchmark for further improvements of both the bot design and the KPIs you are tracking.
Below you can find some of the fundamental and frequently used metrics:
| Direct Metrics | Description |
| Fallback rate | Number of times bot fails to understand end user's input |
| Customer satisfaction rate | End user's experience during bot interaction, satisfaction of the answer provided |
| Recognition rate | Number of times bot is able to understand end user's question/intent |
| Handover rate | Number of end users requesting handover to a human agent |
| Ticket creation rate | Number of times end users requesting ticket/e-mail creation |
|
Ticket deflection rate |
Number of times end users fail to be transferred to the human agent, but reject to create a ticket or send an e-mail |
| Retention rate (Total count of end users, Total count of conversations) |
Number of returning/new end users |
|
Bounce rate |
Number of times bot initiates conversation, but end user leaves the page without interaction (empty conversations) |
| Number of new Frequently Asked Questions (FAQs) | Pattern of end user behavior and questions asked of the bot, and based on that discover new FAQs |
| Indirect Metrics | Description |
| Handover waiting time | Average time end user waits to be transferred from bot to human agent |
| Handover abandonment rate | Number of end users who leave chat mid-handover process |
| Goal completion rate (for specialized bots) | Number of end users filling out a form, completing a purchase with the bot |
| Bounce rate | Number of end users entering a website who don't use or see your bot |
| Lead generation rate | Total amount of newly captured leads compared to amount of channel visits |
| Browse-to-buy ratio | Ratio of items purchased to items viewed |
| Bot-enabled purchases | Number of units bought with the help of bot |
| Resolution rate | Compare the time and amount of human agents' ticket resolution to the tickets resolved by the bot |
| Time before purchase/ Goal completion length | Amount of time it takes to complete a purchase from the moment the chat was initiated |
| Cart abandonment rate | Number of initiated sales that never reach purchase/ transaction completion |
| Ecommerce return rate | Number of items returned compared to items sold - measures bot impact on product return |
The list of metrics above is not exhaustive and can be interpreted differently by each party. This is why it is essential that there is an alignment between project participants, both internally and externally.
The results can be expressed both in relative and absolute values, depending on preference. What’s more, different integrations will offer other insights into the way end users interact with your bot and what type of data is received.
Certainly offers various ways of fetching the received data, through:
- the Inbox, where you can view chats between the end user and bot;
- the Traffic Report, which allows you to track your bot's performance using Tags; and
- the Message Insights Report, where you can segment visitor messages by Module and filter by Tags.
The ways of approaching the chatbot project and measuring its performance are endless. It is up to you and your team to agree on the unified term of what success is and how to measure it. Certainly will always support you in defining those metrics and boost your bot in fulfilling the desired business purpose.
Ultimately, it comes down to creating scalable, omnichannel chatbots that efficiently and effectively support your activities, follow your strategy, and fulfill their purpose of making the customer happy.
In summary, it is essential to:
- understand different business applications and bot types, to which the chatbot project can be applied;
- determine which needs or problems the chatbot project should solve or add value to;
- design the chatbot while keeping in mind what would (and would not) be the desired conversation outcome;
- define what success is and how you measure it (based on the factors above);
- understand the importance of metrics and decide which ones fit your bot the best; and
- become informed of different ways you can reach the data and apply measurements.