MegaFon and HPE signed the settlement agreement on the claim for $28 million

2020: The settlement agreement with HPE in the claim for $28 million

On July 11, 2020 it became known that MegaFon and Hewlett Packard Enterprise signed the settlement agreement on the claim for $28 million within which the American producer of IT solutions was accused of failures of the IT system of MegaFon. In the Russian company told about it to Reuters agency. HPE confirmed settlement of the claim.

According to the representative MegaFon, a dispute was settled in July, 2020, the claim is withdrawn. Any agreement details of the company did not begin to be disclosed. Extrajudicial dispute settlement "completely resolves the conflict" between the parties, the representative of Hewlett Packard Enterprise reported.

MegaFon specified the following damage in the claim:

• cost of the equipment of HPE and its implementation ($28 million);
• expenses on unsuccessful completion of a system to exclude failures;
• the cost of the new equipment which should be bought on replacement, and implementation cost (unknown value, at least $28 million);
• the half-received profit because of loss of clients (in day of accident at MegaFon of sale of contracts of MTS in Moscow grew twice, and in the Volga region – five times. The representative of Tele2 Russia announced growth in sales of SIM cards by 2.5 times).

At the same time the final amount of damage MegaFon asked to define court. What the parties as a result agreed about, it is unknown by September 11, 2020.

Claims of MegaFon concerned the UDR system (User Data Repository) — storage with the technical information about subscribers and the services available to each of them. As a result it was necessary to refuse it. Thus, the cost of the equipment and costs for its implementation are direct loss of the customer.

Because of failure of the IT system subscribers of MegaFon at several o'clock were left without communication. The American company then did not begin to give informative comments and were only limited to the following statement:

HPE gives a priority in the solution of such corporate questions and will continue to work closely with the clients for the purpose of ensuring the greatest possible service quality.[1]

2018: MegaFon filed a lawsuit against HPE for "catastrophic failures" in network

In November, 2018 it became known of submission by MegaFon of the claim to Hewlett-Packard Enterprise (HPE). The American corporation was accused that because of its software in network of the Russian operator there was a repeated failure. the statement is placed in base of the American justice PACER. 

It is about the project of virtualization of UDR (User Data Repository) - it is the so-called house register which connects technology network and billing. In this storage data on subscribers and services available to each of them are placed. MegaFon paid for implementation of this system to HPE about $28 million.

MegaFon submitted a multimillion claim to HPE because of failures of communication in Russia

The operator asks court of the northern District of State of California to collect this amount and also so-called "presumptive damages" which can potentially reach the sevenfold extent of the stated damage or $196 million. Thus, the amount of required compensation is $224 million. 

However as explained to Reuters news agency in MegaFon the final amount will be defined by court therefore to call specific payments still early. The operator says that use of a low-quality IT system of HPE led to economic and reputation costs.

At the same time MegaFon is ready to go to the settlement agreement at any stage of process in case of the satisfaction of all the requirements. It is noted that before submission of the claim of the company conducted negotiations, however they were ineffectual — was not reached specific arrangements on settlement of a situation.

Follows from the text of the statement of claim that as a result of the poor performance and "catastrophic failures" in work of UDR there was a shutdown of networks of MegaFon in November, 2016 and also in April, May and June of the 2017th. Accident because of which subscribers could not use a voice communication and to come out in the mobile Internet, concerned inhabitants of capital and central regions and also the Volga region. 

In the claim it is said that as a result of some of these failures the network was almost completely disconnected for more than 80 million subscribers. After large-scale shutdowns in  MegaFon the director of infrastructure was replaced though in the company claimed that staff movements were not connected with failure.

MegaFon not only lost the initial investments into this IT system, but also faced perspective of unknown additional expenses on its replacement which are estimated at not less than $28 million, claims of subscribers, partners from among virtual mobile operators, and, perhaps, shareholders, said in the claim.

In the document it is also specified that the contract for upgrade of a wireless network of MegaFon with HPE was signed in 2013. The claim was taken to court on October 23 the 2018th. In addition to HPE, the companies applied for the contract with MegaFon Alcatel-Lucent Ericsson Huawei Nokia Oracle, Tekelec and ZTE.

A choice for benefit of HPE was made because the American company during the presentations, correspondence and telephone negotiations promised to create "the best solution". HPE promised to construct a stable system with fault tolerance at the level of 99.999% which had to allow MegaFon to integrate regional networks, however it was defective and non-working, noted in the Russian telecommunication company.

The interests of MegaFon in court are provided by Winston & Strawn law company. It stated the following:

Imagine a similar stop of any other cellular company in California and the corresponding influence on its business and reputation. HPE knew that it has no experience or necessary examination for successful project implementation in such area, such scale and with such tasks. Besides, the company knew that its offer was unreasonable and unchecked and also that case there were no bases for assurances that it will conform to the technical requirements imposed in the contract of MegaFon … If MegaFon knew the truth, it would not select HPE.

According to lawyers, HPE convinced MegaFon that the American corporation had similar projects, and at signing of the contract the Russian operator will receive a ready-made product. However in fact it turned out that HPE mostly began to build a system for MegaFon from scratch.

After the failure order of HPE MegaFon, as it appears from the claim, should recover, recustomize or replace still faulty system of UDP.

After emergence of failures in network functioning HPE provided to MegaFon the additional hardware, updated the software and set on it patches, but could not achieve performance improvement of UDR and fix problems with synchronization between nodes of the operator.

It is specified that fault of a system has an effect in the first year of its work. MegaFon required from HPE the plan problem solving with a performance of UDR, since August, 2013.

As notes Reuters with reference to the website of the operator, in September, 2017 the company announced the tender for installation of the updated UDR system. According to sources of the agency, the project was estimated at $32.7 million MegaFon refrained from comments about the choice of the supplier of new UDR and contract amount with it.

In response to a request of Reuters to comment on a situation in HPE reported that the policy of the company does not allow it to describe the current judicial proceedings.

2017: The largest failure in the history of MegaFon

On May 19 in Moscow and some other the cities of Russia there was a large failure in network functioning of MegaFon mobile operator. Subscribers lost an opportunity to make voice calls, at the same time remained data transmission therefore specialists of operator recommended to the clients to call via messengers is available.

MegaFon could not render voice communication services throughout the most part of day on May 19, 2017

Now in Moscow and several other cities time complexities with a voice communication are observed. Decrease in success of dialing - 30%. Technical specialists of MegaFon carry out recovery work and are going to finish them in the nearest future. We pay your attention that the mobile Internet continues to work in the normal mode, and you can make calls via messengers, - the press service of operator in the afternoon on May 19 reported

Later in some media there were assumptions of possible connection of technical failure with the virus attack of Wanacry and also with testing of the equipment for implementation of the Yarovaya package. However representatives of operator rejected these versions, having reported that failure occurred on one of elements of network equipment and specialists of the company are already engaged in elimination of effects.

About 21 hour Moscow time the press service of MegaFon distributed video in which Pyotr Lidov, the director of public relations of MegaFon, reported that the happened failure became for the operator "probably, the major accident in the history".

According to him, the problem arose in the software set in a data processing system. Failure led to the fact that there was a permanent peak load on a system therefore it did not maintain.

He also reported that developers of this software and the equipment – HP company – are engaged in creation of the system solution. In turn, engineers of MegaFon are engaged in redistribution of load from the idle systems of working.

A bit later the press service of operator added that there was a failure in the software - the database of production of Hewlett-Packard company - on the main and reserve nodes servicing a number of territorial subjects of the federation. Technical support is also performed by Hewlett-Packard.

Hewlett-Packard is going to set a software update in the nearest future, - reported in MegaFon

By the time of the publication of material (21.00 across Moscow 5/19/17) communication was reestablished in Samara, Kazan, Saratov. The success of voice calls across Moscow and area was 93%.

Software and the equipment of HP is widely used in MegaFon many years. About it read in the help below.

Compensation to subscribers

The MegaFon mobile operator intends to compensate to subscribers of inconvenience, the communications caused by hours-long large-scale failure on May 19, 2017. The CEO of the company Sergey Soldatenkov reported about it on May 23 in VKontakte social network. Compensation is offered to users since May 20, however earlier for its receiving it was necessary to address to contact center of the operator. There is no such need now.

According to Soldatenkov, injured subscribers can select one of three options of compensation: receive free of charge 1 GB of traffic and 50 minutes of a voice communication, or to receive free of charge 2 GB of traffic, or to browse free of charge one movie on Megafon.TV. Subscribers at whom during failure an opportunity to make voice calls, to receive and send the SMS disappeared have entitlement to compensation, or problems with the mobile Internet were observed. These persons will receive the SMS with the additional information about receiving compensation within the next week.

The director of infrastructure left

The director of infrastructure of MegaFon of Boots Alexander left the company in May, 2017, its place remains free so far. There was it in two weeks after large-scale failure in MegaFon network.

2013

Implementation of the module of HP of PPM Demand Generation for management of commercial initiatives

In 2013 MegaFon told about implementation of the module of HP of PPM Demand Generation for management of commercial initiatives of MegaFon.

Implementation of HP CloudSystem in the Caucasian branch

The Caucasian branch of MegaFon for the solution of the new big tasks connected with rapid development and operational implementation of new services aimed to virtualize IT infrastructure^[2]. Having considered offers of different vendors, the management of MegaFon came to a conclusion that the HP solution of CloudSystem and software of VMware quite conform to requirements of the growing business and will allow the company to achieve an effective objective – to provide a public cloud environment for support of all external services, including for automation of drawing of accounts, one of key workflows.

As the MegaFon company works in the different, deleted from each other cities, it was necessary to exclude the risks connected with possible delays in network. Four HP solutions of CloudSystem in four different branches were for this purpose implemented. Convergent infrastructure, in that is the cornerstone of the HP CloudSystem system number of software of Matrix Operating Environment 7.0, BL460c working for four a blade-HP servers and 19 HP BL685 from softwares of VMware vSphere 5.0 connected to XP12000 storage system.

The Complete company built all process - from pre-sale preparation before implementation of the CloudSystem system. Users received some skills of work with HP CloudSystem in training center of NR company in Moscow. Service of systems is made on the basis of the three-year agreement of partners providing technical support in mode 24х7.

The HP CloudSystem system is the integrated open platform. Using it, service providers can manage services in the field of cloud computing both in private, and in state sectors. The complete cloud solution which allows to support quickly growing sales volume is as a result created.

2012

Network of the data centers designed by HP

Until the end of 2013 MegaFon promised to construct the commercial DPCs intended for rendering services to third-party customers in each federal district. In addition to the DPCs existing at that time in Samara, Yekaterinburg, Novosibirsk and Khabarovsk from where such services were already on sale, the operator was going to construct data centers in Moscow area, St. Petersburg and Rostov. The general designer became Hewlett-Packard.

The HP BPM platform in the Far East branch

In 2012 the Far East branch "MegaFon" announced the choice of software solutions of HP for support of the subscribers in the Far East region. To support high-quality service of the subscriber base, the operator needed to optimize business processes and to guarantee a minimum of idle times. The company stopped the choice on HP Business Process Monitor (BPM), platform of monitoring from HP [1] (in more detail).

2011: Implementation of HP of Storage Essentials Storage Resource Management in the Volga region branch

In January, 2011 the Komplit company announced implementation in the Volga region branch of MegaFon of the complete solution for resource management and infrastructure of storage area networks.

The Storage Area Network (SAN), or Storage Area Network (SAN), are developed for consolidation of resources of storage systems and providing to access servers to these resources. Data exchange between storage systems and servers is performed at the level of blocks and requires high reliability and performance. In order to avoid data loss in SAN the probability of emergence of transmission errors or falling of performance should be minimized, noted in Komplita.

That potential threats did not become real, it was required to automate management of SAN. Specialists of Komplit company, having studied possible problems and requirements of the Volga region branch "MegaFon", offered for testing the software product of HP Storage Essentials Storage Resource Management which represents the complete solution for resource management and infrastructure of heterogeneous storage area networks of corporate scale

This system is based on the commonly accepted standards and protocols in the field of storage area networks – SMI-S, CIM, WBEM. Thus, it provides management of storage area networks with components from different producers.

Engineers of MegaFon jointly with specialists of Komplit unrolled a test environment which covered elements of the available storage area network. Thanks to it the customer could get acquainted with a management system during the two-month test period after which specialists and managers of the operator recognized implementation of HP of Storage Essentials SRM an optimal variant for their company.

In addition to functionality and standards on which a system is based they noted advantages of modular structure of the selected solution, the structure and a flexible license policy of HP company allows to increase functionality as required.

2009: A possibility of the centralized monitoring of physical and virtual infrastructure

The basic principles of business of MegaFon are the maximum satisfaction of customer needs and fair competition. In view of the high quality standards of services in the market of telecommunications, the company realized need of ensuring effective and smooth operation of corporate IT systems, reported in MegaFon in 2009.

Virtualization became the solution for creation of reliable and effective IT infrastructure in the financial plan ^[3]. Within the first stage of virtualization the company implemented several servers of VMware of ESX 3.5 and consolidated them in a cluster of DRS. The hardware of the solution was made 4th by the processor HP servers of ML570 G2 and HP ML570 G3 connected to the general storage system based on HP MSA2212 fc.

By the beginning of 2009 in the company it was operated already about 40 virtualized servers under control of VirtualCenter, DL360G4p unrolled on a physical HP server. MegaFon announced that it actively increases a share of the virtualized applications - by estimates of specialists of the company, the planned growth of virtualization is 10-15 virtual machines a quarter, and by the beginning of 2010 was going to transfer to virtual infrastructure of VMware to one hundred physical servers. As a hardware platform for expansion of virtual infrastructure of VMware MegaFon was going to use a blade-HP servers.

Services of sending technology Sms, Web servers for processing of subscriber requests and a management system for technology equipment work on virtual machines. Besides, we transferred services of monitoring, critical for us, to virtual infrastructure of VMware - the decision to use them in VMware was made in a type of high fault tolerance of a cluster on the basis of VMware of ESX, - Ivan Iotchenko, the engineer of group of management systems reported MegaFon

With implementation of virtual infrastructure of VMware the MegaFon company faced a new problem - a problem of the centralized monitoring of IT services. For tracking of a status of IT infrastructure the HP solution of Operations Manager for Unix was used: on the managed physical servers the agents providing collection of information from different sources were established.

However installation of agents of HP on the virtualized servers cannot employ complete pictures of all systems of virtual infrastructure, noted in MegaFon. Unavailable are major performance measures and stability, such as events of vCenter, metrics of ESX on waiting of resources of the virtual machine (CPU ready time), the redistributed memory (memory baloon), the swap file (memory swap rate) and so on.

Implementation of virtual infrastructure demanded from us search of a new solution for the centralized monitoring, - Ivan Iotchenko said. - At the same time we wanted to use HP Operations Manager, already usual for us, as a single platform, i.e. the product which could expand possibilities of HP of Operations Manager and provide the full centralized monitoring of both physical, and virtual servers was required for us

In March, 2009, within the project which is carried out by the platinum partner of HP - Complete company, MegaFon stopped the choice on nworks Smart Plug-in for VMware (Veeam Software) – the corporate solution of monitoring allowing to monitor work of virtual infrastructure of VMware in the standard Operations Manager HP console.

Providing a full integration with HP Operations Manager, the solution nworks does not require installation of agents on the ESX servers that is crucial for steady work of infrastructure.

Use of the agent always assumes additional load of the server. In a situation of the applications critical for business, it is simply inadmissible, - Ivan Iotchenko explained. - Unlike alternative solutions, for data acquisition about virtual environment of nworks Smart Plug-in for VMware uses the VI API, thereby reducing load of the ESX and VirtualCenter servers

nworks Smart Plug-in for VMware is delivered with set ready the politician for tracking events and metrics of virtual environment of VMware and the built-in knowledge base providing informative information messages of different priorities.

The group of management systems cannot just know all subtleties of functioning of virtual environment, - Ivan Iotchenko continued. - The preconfigured politicians of monitoring and the knowledge base entering the solution nworks significantly accelerated process of implementation and increased performance of control group. The solution nworks is ready to use right after its installation

In a project deliverable virtual environment of VMware provided to MegaFon company a number of serious advantages – effective use of hardware resources, cutting of costs for equipment procurement and its service, a possibility of fast deployment of servers, fault tolerance of IT systems.

Thanks to nworks Smart Plug-in the MegaFon company had an opportunity to consolidate data of the corporate IT systems working both at virtual and on physical servers, on the Operations Manager HP screen.

2007: HP Superdome in Ural "subsidiary"

In August, 2007 MegaFon and Hewlett-Packard completed the project on upgrade of IT infrastructure of "Ural Dzhi Es of Em" operator, OJSC MegaFon affiliated enterprise. The project included transition to the convergent billing solution of Petter Service company and updating of the current hardware platform taking into account the increased requirements of computing loading.

Use of the technology of HP allowing to combine processors of architecture PA-RISC and Intel Itanium 2 in one HP Superdome system allowed the company to put performance reserve when preserving investments into the current information system", - noted in MegaFon

2005: System implementation of management of HP OpenView

In 2005 in MegaFon for the purpose of increase in level of controllability of a corporate information system of the company HP OpenView management system was implemented.

Creation of uniform command center the I-Teco company became an information system of one of the largest Russian mobile operators.

The contractor created a single platform of management and monitoring which covers all main equipment rooms and software packages, consolidates all necessary information on failures and problems in a uniform point and provides convenient management tools IT infrastructure. The solution was implemented on the basis of software products of the OpenView HP family and the product I-Teco Business Observer  — the means of technology monitoring of automated information systems developed by I-Teco.

The guidance loop included the main information systems and all levels of IT infrastructure from network devices to the application software and technology processes are covered. Were unrolled a management subsystem and monitoring by network infrastructure on the basis of HP OpenView Network Node Manager, servers and applications  — on the basis of HP OpenView Operations, performance  — on the basis of HP solutions of OpenView Performance Manager and HP OpenView Reporter. Management subsystems were constructed by quality of IT services and monitoring of technology processes respectively on the basis of software products of HP OpenView Internet Services and I-Teco Business Observer. The management subsystem circuit by servers and applications included all main systems of infrastructure of the customer. Quality of IT services were covered by a management subsystem as the basic servicing information services  — such as postal services, Internet channels, system services and also information services of the top level on the basis of the specialized application software.

The subsystem of monitoring of technology processes provided monitoring of internal processes of a billing system on accomplishment of technology transactions of service of subscribers and interactions of a billing system with communication hardware-software frameworks. This subsystem provided the effective tool for assessment of passing of these processes in real time and information for the choice of the directions them to optimization.

At the final stage of the project all information services and resources covered by a management system and monitoring were united in uniform service and resource model which provided means of the fast analysis of problems and dependences.

The project deadline made about nine months.

Notes