Mobile TeleSystems (MTS) (Digital Trajectory: Action Track (ATC))

2021: MTS introduced monitoring of the Smarty chatbot service

MTS develops automated customer service services in order to reduce the staff of call centers throughout the country. One of these smart services is the Smarty chatbot.

As part of the MTS Startup HUB accelerator in 2020, the MTS Client Service Development Department identified the problem of low customer satisfaction and resolvability of calls by the chatbot service, and frequent failures in integration with external services led to an additional burden on call centers operators.

The ex Weigandt Consulting team proposed using the Action Track product (PBX) to monitor the chatbot service, its integrations with external systems and control the processing of each subscriber's request. As part of the pilot, the PBX product began to process the logs of the service, a dashboard of metrics of the efficiency of the chatbot was formed. The pilot was recognized as successful, and from the end of 2020, a full-fledged implementation project began.

Action Track was deployed in the MTS loop and integrated with the Graylog storage system. The PBX began to process the chatbot's real-time logs without masking user data, which was necessary within the pilot.

The objects of control for monitoring are chat bot and user interaction sessions. Processing up to 4 million events per day, the PBX began to identify cases when the bot did not respond to the client, errors in integrations with external systems, a long absence of requests from users. To control the operation of the service, metrics and thresholds for triggering and creating incidents on IT groups were determined.

In order to analyze the effectiveness of the chat bot scenarios for processing subscriber requests, reports and dashboards were formed to determine how the scenario affects business metrics. The PBX also made it possible to assess the impact of IT errors on key metrics: transfers to the operator, solvability and NPS.

As a result of the thin settings of the PBX monitoring response to errors and outages in the chatbot, it became possible to solve critical incidents 3 times faster. Business analysis of bot scenarios and correlation between errors and business metrics allowed the bot development team to identify the Top 10 scenarios for refinement and Top 5 services that will fail the most and affect business performance.

As a result of 6 months of industrial operation of PBX monitoring, the metric of satisfaction with the tNPS chatbot service increased by 14%, and the number of transfers to the call center from the chatbot decreased by 6 percentage points.