"Yandex": Transition from Oracle to the open source software allows ourselves to solve the arising problems

"Yandex.Mail" was started in 2000. As of the middle of 2016, according to "Yandex", the service is visited by more than 10 million users a day.

All data in service were initially stored and processed in Oracle DBMS, and in 2016 "Yandex.Mail" moved to open PostgreSQL DBMS.

Oracle DBMS shortcomings

"Yandex" was dissatisfied that with Oracle DBMS it was necessary to perform a set of manual operations (for example, "fruit liqueur" of new base, data transfers), there were problems with development environments, "not really sympathetic support", etc. It is said in one of prezentvtion of the system administrator of "Yandex.Mail" Vladimir Borodin devoted to the project of migration on PostgreSQL.

First, Oracle costs as the pig-iron bridge. Secondly, quite often we encounter some internal oraklovy problems. Something to make the only way is to address in support. They promise us that they will sometime repair, but "sometime" - it is bad. It is necessary to us right now. For these two reasons we decided to refuse Oracle, - Vladimir Borodin spoke about it, speaking at the PostgreSQL Meetup conference in September, 2014[1].

At the PGConf conference in February, 2015 he noted that Oracle ceased suit "Yandex" long before sanctions and import substitution^[2].

In 2012 "Yandex" made the "political" decision to transfer "Yandex.Mail" from Oracle DBMS on PostgreSQL

2012-2013: Start of the project of migration on PostgreSQL and "pilot"

In 2012 the company made the "political" decision to get rid of Oracle in three years.

Since July, 2013 June, 2014 of "Yandex" made an experiment based on PostgreSQL with collectors of mail. The collector is "such piece which goes to other mail systems on IMAP or POP3, collects letters and puts to you in a box. It is necessary to store information where and how to go, and information on what is already collected. All metadata lay in Oracle, exactly as well as all metadata of mail", Vladimir Borodin told.

As a result of migration of collectors 2.5 TB data (about 17 billion lines) were transferred to Postgres. System loading - 40 thousand requests per second. On time the project took about half a year.

As a result of failure from Oracle in this project, according to Borodin, saved so much on licenses, "what will be enough for people who are engaged in it for Bentley".

2015: Cooperation with Postgres Professional

According to Oleg Bartunov, the CEO of company "Postgres Professional", "Yandex" Postgres became seriously interested. "Lately I gave to their employees three lectures", - he reported TAdviser in July, 2015.

In the project of migration of the Yandex mail "itself showed an initiative, saved a lot of money, provided loading, fault tolerance", the Deputy CEO "Postgres Professional" Ivan Panchenko adds.

But the interest of "Yandex" concerning Postgres, according to Bartunov, is much wider, than the mail project (in "Yandex" it is not confirmed).

They appealed to us to create such tools that it was easier for them to carry out migration of other systems in Postgres, - the CEO "Postgres Professional" told TAdviser. - We made a possibility of profiling of the database for work in multinuclear servers - it is a thing which is necessary to all owners of the large systems. Its existence will allow to make Postgres even more suitable for big customers.

In "Yandex" of TAdviser explained that "Postgres Professional" develops the additional functionality connected with diagnostics and optimization of performance for the company: "After a while it will be available in free open assembly of PostgreSQL that could use it and other developers is one of conditions of our agreement".

In "Postgres Professional" call "Yandex" one of the major clients: "For many other companies it is the serious example showing possibilities of Postgres".

2016: Completion of migration

By December, 2015 collectors of mail were completely transferred to new DBMS, and in January, 2016 the main transfer on it a mail service began, follows from Vladimir Borodin's materials. According to the results of the project in "Yandex" realized that it is not the last project with Postgres: "Very much it was pleasant to us. Other projects were started".

In July, 2016 "Yandex" completed transfer of the mail service to PostgreSQL. Rewriting of all code for work with Oracle and PostgreSQL in total took 10 cheloveko-years, provide data in "Yandex".

Vladimir Borodin connects the problems which arose in the course of migration, generally not with PostgreSQL, and with data: for more than 10 years of existence of their "Yandex.Mail" collected much. Besides, there were errors in the code of transfer. In "Yandex" told TAdviser that generally worked on errors in logic of data transfer.

In general difficulties during operation of mail on PostgreSQL appeared less, than expected "Yandex", but they, of course, were, representatives of the company noted in a conversation with TAdviser.

Deployment of a mail service based on PostgreSQL demanded use three times more hardware. It is connected with fundamental differences of used "iron", with the changed load of DBMS and its performance, explained TAdviser in "Yandex".

We passed from vertical scaling of a small amount of expensive servers to horizontal when it is more servers, but they are cheaper, - told TAdviser in "Yandex", having added that at the same time only the structure changed, producers did not change.

Speaking about advantages of PostgreSQL, in "Yandex" noted:

PostgreSQL is open system therefore we can quickly solve the arising problems. It is important for the service working 24/7. Besides, in PostgreSQL the cache of libraries is not blocked, so deployment of changes (including on a test stage) it becomes simpler, - told TAdviser in the company.

Economic effect of migration in the company was preferred not to be commented, as well as plans for transfer from Oracle on PostgreSQL of any other systems and services. Earlier Vladimir Borodin told that PostgreSQL is successfully used in Yandex.Maps - "because of divine PostGIS'a". Free DBMS was selected from this project initially.

Notes