Citymobil uses an analytical DB of Exasol for support of solutions for the multiple growth of business

2020: Use of an analytical DB of Exasol for support of solutions for the multiple growth of business

On May 17, 2020 ATK consulting group reported that Citymobil, one of services for the order of the taxi, began to use analytical database Exasol. The company made such decision to centralize access to to analytics for business users and to support decision making in real time.

Initially for storage and data processing Citymobil used a linking of Open Source of technologies, including MySQL, ClickHouse, Apache Airflow and Kafka. However in 2019 the company began active expansion to regions and amount of data multiply increased, having exceeded 30 Tb. At that time the technology stack for work with data needed to be strengthened for ensuring efficiency and convenience of work.

The analytical database had to become an endpoint of data access both for business users, and for systems which will address for data. In selection process of the solution Citymobil company was guided by the following criteria: performance, compliance to the SQL standard, existence of window functions for analytics, compatibility with Tableau and Power BI, existence of the functions determined by the user (UDF) based on the languages Java, LUA, Python. The analytical Exasol database was the first in complex of parameters.

Within our own testing Exasol exceeded other solutions on the speed of work and scaling of analytics. Separately It should be noted autoindeces – they automatically form how data are used, and minimize labor costs on query tuning. Also use of Exasol within the data warehouse helped us to reduce time for preparation of the reporting, without watching us the growing information volume. In 2020 we are going to save the current growth rate of business and we expect increase in amount of data up to 100 Tb,

The ATC advisory Group provided access and helped to integrate Exasol into a company performance. For May, 2020 analytical IT infrastructure Citymobil includes the data bus Apache Kafka, a steydzhing a layer on the basis of ClickHouse in which data without preprocessing, and the main layer, the analytical Exasol database with the processed data and show-windows are stored. Also storage of a part of a steydzhingovy layer in Exasol for online access is planned. Such architecture allows to support the high speed of data acquisition from storage, completeness and reliability of the stored data and also high-quality process of replenishment of storage.

2018

What work with data began with:

• Legacy Stack
• Open Source
• MySQL,PHP

Dynamics of amounts of data:

• total growth by 350% quarter by a quarter
• Data Pipeline: MySQL (OLTP) to MySQL (Aggregated)
• ETL are PHP scripts on cron

MySQL ceased to cope with the saved-up volume

• Efficiency of data above all
• We select ClickHouse
• ETL - bash +cron

Why passed from krone to airflow

• dependences of processes
• monitoring
• autorestart of processes
• CI/CD organization
• separation of roles into development and support

Request from business on:

• compliance to the SQL standard
• window functions for analytics
• User Defined Functions
• BI is compatibility with Tableau including in the Live-mode for self-service

• support of high speed of data acquisition from storage;
• support of internal data consistency;
• possibility of receiving and comparison of so-called cuts of data (slice and dice);
• availability of convenient utilities of viewing data in storage;
• completeness and reliability of the stored data;
• support of high-quality process of replenishment of data.

• Organization / access isolation
• Team building of DWH/BI on roles
• Scaling of a cluster
• reporting timeto market

• in the ekzasol the storage of a part  of STG for online access is planned
• minimization of work on query tuning
• autoindeces