Citywide Contact Center began testing new language models to improve consultation quality

2024

Use large language models in pilot mode

The Citywide Contact Center began using large language models (LLM) in pilot mode. This is a type of artificial intelligence that allows the voice assistant to quickly find the desired answer in the knowledge base, answer several questions from subscribers at once and learn independently. This will help improve the quality and speed of advice on hotlines, and operators will be able to devote more time to solving more complex requests from residents. This was announced by the Department of Information Technologies of Moscow (DIT) on November 5, 2024.

The use of language models is piloted on the line of the unified reference service of the Moscow Government for two important tasks: to provide advice on incoming calls and to classify feedback based on service results. In combination, this approach allows not only to train a voice assistant without the help of operators, but also to improve the quality of consultations: the language model classifies the feedback of residents based on the results of calls, which will further help improve the knowledge base and consultation scenarios, and, if necessary, improve the qualifications of operators, - said the head of the citywide contact center Andrei Savitsky.

The use of language models on the line of the unified reference service of the Moscow Government is invisible to residents. When a person calls the line and asks a question, the speech recognition system is turned on, it records the message, translates it from audio format into text and sends it to the language model. It, in turn, using additional filters, finds the necessary articles with answers in the knowledge base, highlights the essence in them and then generates a response that the voice assistant informs the subscriber. At the same time, the most accurate comprehensive answer to all questions of the resident is formed, without unnecessary information.

Large language models are able to independently and in a split second find information on each new topic in the entire knowledge base of the contact center and voice it to the applicant. Unlike the standard voice assistant, which reads the answer from the knowledge base, an LLM-based assistant can maintain a lively dialogue, focusing on the intonation and manner of communication of the subscriber.

In addition, a large neural network-based language model trains a voice assistant. If earlier operators had to manually upload new topics and answers in various variations to the virtual assistant, now LLM helps the voice assistant to independently find the necessary information in the general knowledge base. The role of a person is not excluded: in addition to the fact that some of the calls are still processed by operators, they also monitor the relevance of information in the knowledge base. Thus, the neural network trains the neural network, but under the strict control of contact center specialists.

Another important area for using language models is a new approach in classifying feedback based on service results. The voice assistant is already able to collect feedback from residents on the contact center lines, but the contact center specialists were still involved in processing comments.

Now, thanks to the introduction of language models, receiving an assessment from a resident or a comment upon completion of a consultation, the neural network ranks them positive, negative and neutral, allowing quality control employees to quickly identify shortcomings and make the consultations provided even faster and more accurate.

The use of large language models on the OCC hotlines to work on the quality of consultations complements the already implemented artificial intelligence-based tools - a digital audit operating since 2023 and a speech analytics project launched this year.

Expanding the use of artificial intelligence

The Moscow Department of Information Technology has expanded the use of artificial intelligence technologies to improve the quality of consultations in a citywide contact center. DIT announced this on July 11, 2024. The Digital Auditor is already operating on three hotlines, and its scenario now includes several new parameters. For a year and a half of the project, the voice assistant helped to analyze more than 240 thousand dialogs: the results of the analysis are transferred to the quality control service of the contact center, where operators are assigned additional training if necessary.

Digital audit is a comprehensive project that, thanks to artificial intelligence technologies, has been helping city hotline operators answer residents' questions even faster and more accurately for a year and a half. To do this, we use the capabilities of a voice assistant: it simulates a person's call and communicates with a specialist in a predetermined scenario, analyzing many parameters of the dialogue. For the first time, such a decision was launched on the hotline of a single medical reference service, and now it is used on two more lines: a single reference service of the Moscow Government and a technical support service of the portal mos.ru. Also, some parameters were added to the scenarios of the "digital auditor." For example, it checks how well operators assimilate information on new query topics. For a year and a half, artificial intelligence analyzed the dialogues of 2.3 thousand employees, - said the head of the citywide contact center Andrei Savitsky.

The use of artificial intelligence for digital audit of the quality of consultations began in 2023 on the hotline of the unified medical reference service (122). During the dialogue with the specialist, the virtual assistant checks the availability of the line, the speed and duration of the response, as well as the completeness of the information provided.

Since 2024, the voice assistant began to analyze the quality of consultations on two more hot lines of the citywide contact center: the unified reference service of the Moscow Government +7 (495) 777-77-77 and the technical support service of the mos.ru +7 (495) 539-55-55 portal. The list of scenarios for which digital analysis is carried out has also expanded. Now artificial intelligence helps to quickly check how well information is provided to residents on new topics. After the specialist has familiarized himself with the new materials, the virtual assistant conducts a verification call. This allows you to make sure in a short time that all operators have assimilated new information with high quality, and residents who contact hotlines receive up-to-date information.

In addition, using artificial intelligence, it is now possible to analyze the correctness of the pronunciation of terms and stress setting in words, which is necessary to improve the quality of consultations of a citywide contact center.

The results of the analysis are transmitted to the quality control service. If the auditor identified shortcomings in the consultation, the operator is assigned additional training.

In addition, artificial intelligence helps to improve the knowledge base used by operators. Thanks to this, specialists can quickly find the information necessary for citizens. If during the verification dialogue the operator needs more time to find the necessary information, then the voice assistant not only records this fact, but also analyzes the information search scheme, and also transmits the results to the quality control service. This allows you to simplify the search system, as well as adjust the content of knowledge base topics for more convenient use.

2023

Analysis of 2.5 thousand dialogs of hotline specialists

Since the beginning of the year, artificial intelligence has analyzed about 2.5 thousand dialogues of specialists in hotlines of the citywide contact center. The audit found that more than 80 per cent of consultations are done correctly, with an average operator response time of less than 45 seconds. This was announced by the Department of Information Technologies of Moscow (DIT) on July 3, 2023.

{{quote 'The use of artificial intelligence makes it possible to conduct a comprehensive analysis of the quality of consultations on hotlines and take measures as soon as possible to eliminate shortcomings so that each resident can always quickly receive complete and accurate information on his issue. The robot communicates with operators according to a given scenario, checks the availability of the line, the duration of the call and dialogue, as well as the correspondence of operator responses to the knowledge base. During the digital audit, there was no case that specialists realized that they were not communicating with a person. To do this, we are constantly expanding the capabilities of artificial intelligence. Since the beginning of the year, the digital auditor has analyzed the dialogues of about 500 operators. The results of the analysis are used for training and advanced training of specialists. In the future, it is planned to expand the thematic cases of the robot and use it on all 70 hot lines of the contact center, - said the head of the citywide contact center Andrei Savitsky. }}

The digital auditor conducts about 500 verification dialogues each month on the hotlines of the citywide contact center. The results of the review of each dialogue are transmitted to quality control specialists who use them to organize training and advanced training for employees working on hotlines.

Using AI to improve the quality of citizens' consultations

In the Moscow citywide contact center (OCC), they began to use artificial intelligence in test mode to improve the quality of consultations of citizens. The digital auditor simulates a call to the hotline and communicates with operators according to a predetermined scenario, then analyzes the quality of responses and transmits information to quality control specialists. Based on this data, recommendations for specialists are developed. Including additional training can be organized. This was announced on March 30, 2023 by the Moscow Department of Information Technologies (DIT).

{{quote 'Every month more than five million calls from citizens are received on the hotlines of the citywide contact center. It is important that each resident receives complete and accurate information on his issue as quickly as possible. Quality control specialists regularly conduct random checks, analyzing the quality of operators' responses and helping them correct shortcomings. Now the voice assistant is also involved in this work. For three months, within the framework of the pilot project, the digital auditor analyzed 263 dialogues of contact center operators, showing its effectiveness. In the future, it can be used on all 70 hot lines of the contact center. The results of the analysis of dialogues in the future can be used for training and advanced training of the center's employees, - said the head of the citywide contact center Andrei Savitsky. }}

When creating a digital auditor, they used voice the assistant model, which has successfully processed more than 50 percent of calls to hotlines since 2014. At the same time, the robot has important additional functions, in particular, it can imitate human communication. He speaks to the operator not in a mechanical voice, but uses a wide variety of speech capabilities: female and male voices, various intonations, as well as conversational turns.

Before starting verification calls for a digital auditor, a number of articles are selected from the knowledge base on which he will communicate with a specialist. For example, a new consultation topic is selected or changes are made to an existing knowledge base article. Next, a dialogue scenario is developed for the robot, taking into account various possible clarifying questions of the operator. All components of the knowledge base are collected on the basis of an earlier analysis of real appeals of citizens to employees of the OCC.

During the dialogue, the robot analyzes the dialogue and answers according to several professional parameters, such as the correctness of the greeting, the correctness of using the background Hold or Mute modes during a conversation, the total duration of the dialogue, the correspondence of the content of the answer to the essence of the question and the knowledge base, the correctness of the operator's actions in the system when processing the appeal. In addition, during the audit, the robot also records general data on the operation of the center, including the average call time, the percentage of calls lost due to waiting, and so on. The results of the test are transmitted to the quality control service of the contact center.

2019: Virtual Operator Development and Implementation Experience

Speaking at the TAdviser "Artificial Intelligence Systems" conference in October 2019, the head of the citywide contact center DITNDREI Savitsky shared his experience in developing and implementing a virtual operator. This is not only the largest government contact center in Russia, but also in Europe, Savitsky noted. It started in 2011 with the processing of 1 million requests per year, and in 2019 it is predicted to reach an indicator of 41.8 million per year. They are processed through all remote communication channels, including phone, social networks, online chats and forms.

The center employs about 2.5 thousand operators, the speaker cited these. It collectively turns out that it processes more than one case per second. The center gives answers on all issues of the city's life, including education, public services, housing and communal services, medicine, transport, etc.

Andrey Savitsky, head of the citywide contact center of the DIT of the Moscow government

In connection with the constantly growing number of requests, a virtual operator was introduced, which now processes about 40% of requests from citizens. The prerequisites for its appearance were the understanding that with some major events in the city, the center does not always manage to scale up and answer questions from the population.

Andrei Savitsky spoke in detail about the evolution of the virtual operator of the contact center from simple voice messages (IVR - Interactive Voice Response, a system of pre-recorded voice messages) to an advanced, smart bot developing on the basis of AI technologies . In 2014, they began to design the first solution with speech recognition and synthesis. First, a bot was created that could respond to the most typical appeals.

Slide from Andrey Savitsky's presentation

The initially created system did not have citizens to interact: often, when a person calls with an appeal and hears the metal voice of an answering machine, he tries to connect with a live operator. The main task was to force a citizen to communicate with a robot.

To do this, we not only invented various tricks, but also improved the robot itself. For example, synthesized speech was made from pre-recorded speech, a preliminary sound signal was removed. As a result, we came to a combined speech - these are pre-written human phrases and synthesis, when synthesized information is spoken, for example, the addresses of houses, - the speaker explained.

Slide from Andrey Savitsky's presentation

After numerous improvements, the so-called Wise bot now works in the contact center. In some cases, he knows who is reaching out to him and is capable of predictive service. When communicating, he uses big data, builds a dialogue depending on the age of the subscriber.

The creation of a virtual operator made it possible to speed up the processing of requests, reduce the expectation of a response, which increased the loyalty of the population, as well as save funds from the city budget: since it is cheaper than a live operator.

Slide from Andrey Savitsky's presentation

Regarding the speed of processing, an experiment was conducted at the contact center when a person and a virtual operator answered the same request. He showed that the bot processes the appeal twice as quickly and ends the conversation. The reason is that AI searches faster for answers in large amounts of information. In addition, when communicating with a virtual operator, citizens usually formulate requests more accurately and do not ask questions about distracted topics.

At the same time, not a single living employee was fired, assured Andrei Savitsky, since the volume of urban tasks is large.

We do not plan to dismiss operators, but plan to improve their qualifications, because there are many life situations when a person will not use a mobile application and communicate with a robot. These are some kind of emergency situations, for example, - the speaker explained.

Slide from Andrey Savitsky's presentation

The contact center plans to introduce a Mega bot. He recognizes the subscriber, adjusts to the dialogue, to the life situation and gives hints on its solution, delves into the essence of the problem, can create applications and assign them an ID. This is AI integrated with urban Big Data and many other systems, Savitsky explained.