DPC of DataLine Moscow Pine-forest, 7

Data center of OST on Borovaya St. — the very first DPC of DataLine. In 2009 in this DPC the company entered with opening of two machine halls on the market of services of commercial data centers.

2019

How to fight against the fire in DPC: bitter experience of Dataline

Incident details in data center on Borovaya St.

In October, 2019 in DataLine gave the details of circumstances of the fire which happened in the summer in their DPC on Borovaya St. and also the experience and lessons taken from this incident. The company decided to make it in connection with a large number of the questions arriving on this subject.

Kirill Shadsky, the head of department external DPCs of Dataline which was one of participants of recovery after the fire told that they at first tried to localize ignition manual fire extinguishers. But as it began under a metal roof of the building of DPC, to extinguish it in such a way it appeared it is impossible.

After evacuation of people from the building and an otgon of cars the company met fire-fighting crews.

15 fire-fighting crews which separated into three parts coming to the place of an incident from the different parties were used for suppression

Ignition began on June 5 at 2:10 p.m. In 15-20 minutes after the beginning of ignition cottons became heard: it freonoprovoda of the systems of cold supply blew up, Kirill Shadsky explained. In total completely 5 conditioners failed, and at a part of conditioners feed cables of external blocks only burned down, and one conditioner did not suffer.

The place where the fire began

In DPC on Borovoy of only 16 machine halls, from them one suffered, brought data into the companies. In it after explosions of freonoprovod temperature began to increase. Having understood that quickly to recover the system of cold supply it will not turn out, Dataline made the decision to disconnect this turbine hall on a power supply to save the client equipment from overheating.

In the affected turbine hall there were virtualization clusters, and 781 client virtual machines as a result of the fire were unavailable. Partially also the management system for vCloud Director virtual machines did not work and to other virtual machines there was no access in spite of the fact that they worked.

According to the representative of Dataline, their company, according to conditions of SLA, paid afterwards penalties for all idle services, problems and idle times, provided to customers the place for reservation and helped to be recovered.

In addition to shutdown the equipment in the turbine hall there were also other problems. It concerned also internal services Dataline: "it is possible to tell that the shoemaker was left without boots", Shadsky noticed. It concerned service ServiceDesk, partially – Internet access, management of telephony, etc. Strongly some offices suffered, including where the service of operation sat.

Fire effects

The company emphasizes that at the same time the equipment did not suffer neither from fire, nor from the large volume of water with which firefighters filled in the building.

All premises of machine halls and engineering infrastructure at us are in certain pressurized zones. The pressurized zone saved us. For data center it is one of the major systems, - Kirill Shadsky noted.

At 4:45 p.m. the fire was extinguished and the company started recovery. It was required to recover the system of cold supply, to transfer a part of the critical equipment from the affected turbine hall to next, to recover client and services and also to pump out water from corridors.

Cold supply recovery

At the same time concerning the forthcoming recovery of freonoprovod the company began to call the contractors even before completion of the fire. For recovery of conditioners in Dataline at first just cut the burned-down sections and on their place sealed the whole tubes since start of the turbine hall was a paramount task. Then the company purchased reserve compressors, being afraid that the cooling systems repaired thus can fail. In three months from the moment of the fire three compressors already burned down, the representative of the company provided data.

Chronology of an incident

The learned lessons

After the occurred fire in Dataline the huge abnormal report was made. Kirill Shadsky selected several main main moments. The first is the reason of ignition. Short circuit of a cable from the internal block to the external block of conditioners became it. After the incident added to the list of the infrastructure which is subject to regular survey according to the diagram also cables of external blocks. Earlier they were not included into this list.

To escape completely from short circuits it is unreal, but Dataline does everything that it did not repeat any more, the representative of the company said.

Part of problems was connected with the fact that the roof of data center contained wooden elements though itself both was metal, and had difficult construction. Now technically difficult project on creation of a new roof with nonflammable materials is started. For the period of the project the time roof on technology of preservation of objects and one more time roof over the pressurized zone was built.

Loss of communication channels because of the fire in the company and with clients aggravated a situation. After the fire of Dataline dubbed all communication channels and internal services in the data centers of Nord and Ost.

At the time of the fire the company also came up against a situation that surveillance cameras in DPC were used only for security, for protection against malefactors, and covered not all technology equipment. After the incident the company added additional cameras in affected data center and reviewed politicians concerning video surveillance and on other objects.

Also after the fire reserve jobs for employees since during the incident faced their shortage were added, and the zone of a system of early fire extinguishing was expanded. Earlier in space between the pressurized zone and a roof there were no smoke detection sensors.

Additional problems and the taken measures

Not only engineering changes, but also regulations were carried out, Kirill Shadsky says. One of them – regulations of visit of DPC. For example, earlier in the company there were no regulations of access to machine halls during state of emergency. Now it is and also the responsible persons responsible for authorization of contractors and customers in similar cases.

Separate work was carried out to areas of exercises on fire safety. At the time of the incident it turned out that not all knew how to work at the fire.

This situation is given us a lot of new experience and made us stronger. We coped with this situation, left it, - Shadsky noted.

Source of images - Kirill Shadsky's presentation.

The fire in data center

On June 5, 2019 in data center of DataLine in Moscow there was a fire. TAdviser received the photo from the place of an event from the person working at one of the next offices.

As reported in the press service of the Ministry of Emergency Situations of Moscow, the two-storeyed administrative process building located at   the address: Borovaya St., 7, building 10 lit up. In this building there is a data processing center of OST, it is specified on the official site of DataLine.

The photo from the place of the fire of data center of DataLine. Source: TAdviser

The message about the fire  across Moscow came to an operational duty shift of Crisis Management Centre of General Directorate of Ministry of Emergency Situations of Russia at 14 hours 29 minutes. By 3:22 p.m. ignition managed to be localized. The fire area was about 200 square meters. According to eyewitnesses, on site the event cottons were heard.

There something blows up — the interlocutor of TAdviser told.

By 4:30 p.m. on Moscow time information about victims is not present. Also the cause of the fire is not stated. According to the preliminary information the wooden furring of a roof of the building lit up. On site rescue and fire fighting divisions work.

Due to the fire the websites and services serviced by data center can be unavailable. Because of the fire in data center of DataLine there was a failure in work of all services Qiwi. Problems were observed approximately from 3:50 p.m. Moscow time. At first on the homepage of the website of Qiwi error 403 appeared. After visitors began to redirect on  status.qiwi.ru where it is announced that problems are in all services.

TAdviser requested comments in DataLine, but at the time of a release of the publication (4:30 p.m. Moscow time) did not receive the answer.

In addition to data center, in the lit-up building there are different shops and travel agency, data from open sources demonstrate.

Because of ignition in the administrative building the traffic on Borovaya Street was blocked off, said in Twitter blog of GKU "Center of the Organization of Traffic" (COT).

The traffic is blocked completely off from house 6 to house 20. On site operational services work. Go round! — reported in TsODD.

On June 6, 2019 the message from DataLine in which the company shared pre-trial detention of experts about a cause of the fire came to TAdviser. By their estimates, ignition happened owing to short circuit in a cable of the external block of a conditioning system OST-2.

On June 5 around 14.00 on Borovaya Street there was an ignition of a roof of data center of OST. For fire extinguishing to the scene there arrived more than 10 fire-fighting crews as the fire area was about 200 sq.m. The warning system worked in the normal mode. All staff of the company and clients working for the moment of the beginning of the fire in data center were timely evacuated. There are no victims, said in the statement of DataLine.

As told TAdviser in DataLine, during the fire some freonoprovoda which are on a roof near the ignition center were damaged. As a result the conditioning system of one of OST-2 halls suffered.

In order to avoid losses of the client equipment on overheating by the management of DataLine the decision on partial shutdown of the equipment in the hall was made. All clients, whose equipment was placed in this hall, were quickly notified and whenever possible transferred to reserve platforms. The equipment of some clients was transferred to other halls of data center of OST. The machine hall was not touched with fire and did not suffer from water at suppression — assured of the company.

Recovery work of a conditioning system started right after fire liquidation at 4:45 p.m. By 4:45 a.m. in the morning the hall earned in the normal mode.

As of June 6 DataLine works in the normal mode, except for service of capital construction which already started recovery work on a building roof.

2014: Start of two new halls

In June, 2014 from the platform OST on Borovaya St. earned two new machine halls. In the Golf and Nadazero halls of 204 and 80 sq.m 100 and 34 racks respectively will be placed.

Halls are located in a new part of data center of OST - OST 3 which is designed as independent data center. It has autonomous systems of power supply (two inputs of power supply, certain premises of the UPS, battery rooms, DGU) and conditioning. In the near future in DPC of OST 3 some more halls and Meet-Me-Room Elba will be started.

2010: the 2nd serious accident in Data center of DataLine for the last 3 months

8:03 p.m., there was an accident in Data center of DataLine. Owing to squally wind the chillers placed on a roof of DPC were filled up with metal constructions from the adjoining building and put out of action. As a result of accident, some clients could vosstavit work only in the morning^[1].

Situation for Data center, positioning itself as TIER-3, softly to tell not pleasant. Why the chillers installed on an extension roof were not protected by special trellised casings, on condition of existence of explicit risks from the old factory constructions located in blazhashchy availability. One more interesting fact — the indicators of statuses of climatic conditions in DPC halls on Borovoy published on the homepage of the website remained throughout all accident within regulation that was obviously not true. This mismatch directs at reflection that digits are taken not from a monitoring system of Data center, and are entered content-mendzherom of the website and can be far from real and now. It is necessary to pay tribute to the representatives of DataLine who very in detail lit the occurred accident and published the photo report on results of the raged storm given below. Such detailed the description of accident and terms of its elimination representatives of Data centers of Russia indulge the clients extremely seldom though on the other hand, dry lines about incidents moreover and not from an official source, authors could add a last straw of oil in the inflaming fire of discontent with service quality of placement of the equipment to DPC of DataLine long ago.

HURRICANE NA BOROVOY I OF ITS EFFECT

7/20/2010 at 6:18 p.m. o'clock owing to the heavy wind caused by a hurricane, external blocks of a conditioning system (chillers) were filled up with the heavy metal constructions (by our estimates more than 10 tons of the steel prof sheet and two-T-shaped beams) which failed on a roof of DPC from roofs of adjoining buildings. As a result all three chillers sustained physical damage: in particular, radiators with ethylene glycol are punched and depressurized that led to sharp pressure decrease in the cooling system. Besides, two-T-shaped beams damage 10 fans on two of three refrigerators. All these factors led to fast temperature increase in machine halls.

For pressure recovery in a system the ethylene glycol reserves which are available for us were used. We could solder one of the damaged sections of the glycoleft circuit and at 7:50 p.m. to bring the first chiller into operation that led to decrease in temperature in halls approximately by 5 degrees.

Further works on resuscitation of other chillers were carried out, as it was reached at 11:30 p.m. in 4 hours of recovery work. Then the DPC turned into work in the normal mode. Temperature in halls was stabilized to 4 o'clock in the morning of the next day (7/21/2010).

Now 2 chillers work, temperature in halls is stabilized, the DPC works without reserve chiller. Necessary purchases of spare parts are made for its recovery, repair work of its freon circuit is conducted. System recovery of a reserve happens by forces of service company.

According to our forecasts within 24 hours the DPC will pass into the normal mode of work with existence of a reserve.

Notes