Research of TAdviser: Outsourcing market of services of testing of IT systems in Russia

Introduction

Subject and research purposes

This research of TAdviser is devoted rather narrow, in terms of financial assessment, area in a market segment of outsourcing IT services – to outsourcing of services of software testing. Despite rather modest – in comparison with other types of IT services - the size of this segment of the market, in the last 2-3 years it drew to itself(himself) attention of analysts and profile media because of several big competitions with budgets in hundreds of millions rubles held by key customers (large banks, state structures of federal level, retail networks).

Main objectives of the research conducted in September-November, 2018 were:

• Quality standard of the market of services of testing – who and what services orders who and what services provides;
• Assessment of the current size of the market of services of software testing in terms of money;
• Formation of rating of the main companies providers of services of testing in annual total revenue;
• Identification of top trends in the market, problems and risks for suppliers and customers of services of testing.

Research objects

Both parties of the market relations – both customers of services, and providers were objects of a research.

Concerning customers of services of testing the analysis consisted in quality and quantitative standard of their requirements; for providers of services assessment of a performing resource, assessment of annual revenues in the considered category of IT services, assessment of potential of further development were the main contents of the analysis.

The largest customers of services of software testing in the last several years are banks from Top-20 of rating of the Russian banks – Sberbank of Russia VTB VTB 24/ Alfa-Bank, "Gazprombank" FC Otkritie. The analysis of publicly available information on IT purchases of Sberbank allowed to create almost exhaustive list of the second party of the market – the companies providers of services of testing. Informally the criterion of belonging to group of the main providers can be formulated so: "If the company never took part in tenders of Sberbank on testing, so it is not a participant of this market".

The companies for which services of software testing and related services are the main or one of several key types of activity became the main objects of a research. Selection criteria of the companies in the list of investigated:

• The minimum volume of annual revenues from software testing services – not less than 20 million rubles;
• Existence in the state not less than 10 software testing specialists;
• Existence in the list of the provided services of testing not less than 2 services from the most demanded in the market;
• Existence in the list of clients of, at least, one big customer of services of testing from the profile industries.

Software testing – basic determination and classification

Basic determination

Classical determination of the concept "software testing" Software Engineering - Guide to the software engineering body of knowledge (SWEBOK)" is recorded in ISO/IEC TR standard 19759 "(the first version of the standard is released in 2005, the current version – 2016):

Testing (software testing) – the activity executed for assessment and improvement of quality of the software. This activity, generally, is based on detection of defects and problems in program systems.

Deserves to be quoted one more fragment from the same standard:

The general view on software testing actively evolved the last years, becoming more and more constructive, pragmatic and approximate to realities of modern projects of development of program systems. Testing is not considered as the activity beginning only after designing phase completion any more. Today testing is considered as activity which needs to be carried out throughout all development process and maintenance and is an important part of designing of software products. Really, planning of testing should begin at early stages of work with requirements, it is necessary to develop and specify systematically and constantly plans of tests and the corresponding test procedures.

Classifications of types and methods of testing

The main characteristics of quality of the software are for the first time described in the ISO/IEC 25010:2011 standard "by Systems and software engineering. Systems and software Quality Requirements and Evaluation (SQuaRE). System and software quality models":

• functional suitability,
• performance level,
• compatibility,
• convenience of use,
• reliability,
• security,
• soprovozhdayemost,
• portability (mobility).

One of options of the classification of types and methods of testing tied to model of quality of software and the purposes of testing is given in Fig. 1.

The potential range of services in the software testing market as it is visible from Fig. 1, is rather wide, however, the customers needs existing today concentrate, generally around a limited set of services, actually and creating the market in its current form and volume.

These demanded services include:

• the functional testing (FT) - manual (RFT) and automated (AFT) options;
• the stress testing (ST) - the main subspecies of nonfunctional testing;

Software testing and ensuring quality of software – in what a difference?

Rather often in publications on software testing subjects (including, and on the websites of the companies providers of IT services) along with the term "software testing" other term – "ensuring quality of software" and also the abbreviation of SQA (is frequent – just QA) coming from the English option "software quality assurance" is used as a synonym. Such mixing and mutual substitution of terms is normal occur in marketing publications, in names of thematic actions, in communication within professional community.

Procedures of quality assurance cover all software development cycle, including such processes as determination of requirements, design, coding, control of the source code, the analysis of the code, configuration management, testing, management of releases and integration of products. Procedures of quality assurance include the purposes, opportunities, procedures, measurements and checks. If to formulate shortly and intentionally simply, software testing provides quality check of a ready-made product, and the procedure of quality assurance, in a full compliance with the name, will organize all production process of a product for the purpose of obtaining qualitative result.

It is indicative that there is a concept integrating in itself development and testing – "The development managed by testing (Test-driven development)". In fact, it is not so much technology of testing, how many style of the organization of development process, lifecycle when tests are an integral part of requirements (and the corresponding specifications).

Why testing outsourcing is necessary?

Even in the presence of strong divisions of testing in customer companies, it is possible to avoid the order of outsourcing services not always. Own staff of testers (even very good) is limited in the opportunities, first of all quantitatively – infinitely it is impossible to increase this staff, proceeding from structure of the company, KPI of business, etc.

Pavel Eydeland, head of the Center of testing of Aplana company^[2]:

Advantages of outsourcing in the field of testing by and large differ in nothing from advantages of use of such model in any other area. The main plus of outsourcing – an opportunity to quickly receive competence which the company does not locate to avoid need to employ additional specialists to the state. Search of specialists in the market are long and sometimes very difficult process, especially if it is about rare competences. Secondly, it is obvious that the companies it is inexpedient to employ people on projects which last only several months or are executed discretely. Besides, the outsourcing model gives the chance to scoop labor resources in regions where usually cost is significantly lower than them, than in Moscow.

Often there is a situation when the number of the business and critical systems which are subject to testing multiply exceeds performing opportunities of the staff of internal testers.

Sergey Lunev, X5 Retail Group, head of department of quality assurance:

Today in X5 Retail Group from all volume of costs for testing up to 80% it is the share of testing outsourcing / outstaffing.

Here it is necessary to make small refining concerning the term "outstaffing" which appeared in the above-stated quote. Officially rendering services of loan work (the Russian name of outstaffing) in the Russian Federation is authorized only for the accredited recruitment agencies and the organizations which are in uniform holding structures. Providers and customers of services of testing apply this term as a working slang – because externally it is really very similar to outstaffing. In the projects on functional testing which usually have duration from 6 to 12 months and longer the teams of testers created by the company provider work at platforms of customers and are in operational management of the customer. Such operation mode developed historically: it is caused not only by convenience to the customer, but also reasons of information security since often the tested systems contain the confidential information and/or information which is a trade secret of the customer, personal data of clients.

But upon (and legally) it is outsourcing on conditions T&M (Time and Materials) with long term of rendering services and formation of actual volume of services on the basis of the specifications arriving from the customer. The impossibility of exact determination of labor input of the project before acquaintance with a system is a basic reason of such form of work and began testings because of what rendering services in the fixed cost of the outsourcing contract is not possible.

Sergey Lunev, X5 Retail Group:

Personally I do not perceive austaff as the external resource is just people who get paid not in our company, and in other place. If from provider at the same time still there is an additional service for development of these people and their motivation, for consulting regarding execution of works if they<провайдер> constantly watch a situation, then it very well works.

The market of software testing in Russia – development stages

The starting point of counting of history of the market should be considered 1998 when the first company representing only outsourcing services of software testing was created. Thus, this year at the testing market anniversary – 20 years.

Last two decades can be separated into 4 periods conditionally:

• 1998-2008 – the market grouping period on the party of contractors and fading in of a noticeable and permanent pool of customers. The first customers come from the industries where performance and stability of information systems – banks, a telecom are crucial. "Old residents" – the system integrators and IT outsourcers formed at the beginning and the middle of the 90th years look at emerging market with interest, but, generally in the observation mode; some of these companies will really come to the market only in the second phase of its development;
• 2008-2012 – the market maturing period. Emergence of the specialized companies providers of services of testing, arrival to potentially attractive segment of the market of the known integrators. Stabilization of the main set of services of testing, accumulation of number of skilled staff, mastering of work benches. Growth of culture on the party of customers;
• 2013 - 2017 – gradual stable growth with sharp splash in 2015-2017. The main driver of the market – Sberbank (and his IT - "daughter" - "Sberbank of Technology", SberTech). The market considerably grows in the volume of total revenue, the second wave of providers (again formed companies – comes including, and created by natives of the companies of the first wave). The competition amplifies, deficit of qualified personnel is considerably designated enough;
• 2018 – the beginning of a new historical stage in the market. The main milestone noting the beginning of this stage is the reorganization of SberTech happening within a year (and the actual abolition of the Department of Quality (DQ) which was for the previous 3-4 years a peculiar "point of force" happened within this reorganization – is more detailed about it see in Section 6 "Sberbank of Russia – the strategist in the software testing market").

Economy of projects of testing

Who orders testing outsourcing?

In the interview which took place 7 years ago, but did not lose relevance on many aspects of the considered subject, the deputy CEO for business development of EPAM Systems company Artak Oganesyan gave a portrait of the typical customer of services of testing^[3]:

The typical customer of outsourcing of software testing is the company which has a sufficient maturity to watch closely reliability of information systems, but which does not want to spend money for creation in itself in fact the full-fledged IT organization. Such clients become more and more, and the demand of outsourcing of testing grows. Generally the service is relevant for the financial, insurance and telecommunication industries where there is a critical dependence of business on a large number of information systems, at the same time their permanent development and implementation of new applications are required.

For last 7 years (though, practically did not extend) the list of the industries in which there are main customers today changed a little:

• The bank financial industry unconditionally is in the first place (first of all, banks from Top-10 – Sberbank, VTB) to which it makes sense to carry also insurance companies;
• The second place is rather surely taken today by retail – large retail chain stores (X5 Retail Group, Eldorado, the Tape) on equipment means of IT and complexity of the IT infrastructure can already quite compete to banks;
• The telecom companies, saving high IT armament and also having an extensive range of the business and critical systems, nevertheless, gradually lose the status of top customers of services of testing. It does not mean that the need for control of quality of software for a telecom decreased – simply, most large telecom operators, several years ago being some of main customers of outsourcing QA services, gradually pass to model of an insors and create competences on software testing in themselves;
• Still in the status of potentially promising customers there are state structures (generally federal level and the largest regional - the Pension Fund of the Russian Federation, FTS, FCS, DIT of Moscow). In spite of the fact that large QA providers call "state exams" and among the today's customers, it looks more likely as advance payment to itself on the future. Several not really large-scale projects on testing which took place 3-4 years ago did not create a steady trend yet.

Economy of projects of testing

Project cost of testing forms initially on the party of the customer in the form of the starting maximum price of the contract (SMPC) announced on competitive biddings. Calculation formulas of NMTsK are rather simple – they are based on the maximum sentries or day rates of specialists in testing approved as the customer (taking into account types of testing and professional categories) and the approved labor input of works on testing calculated according to standards of the customer and taking into account the saved-up statistics on last projects.

On the basis of publicly available data from tenders of Sberbank (see Tab. 1) it is possible to draw a conclusion on what wage level of specialists is greatest possible, in terms of the salaries of the involved specialists the project of testing remained profitable for the contractor.

Tab. 1. Market rates and the salaries of specialists in software testing including VAT (in the region Moscow)

Role in a testing command	Skill level	Rate, rub / persons put	Rate, rub / persons month	Gross salary *, rub
the Specialist in the field of functional (including automated) testings	Junior	9 737	202 529,60	81 000
	Middle	12 106	251 804,80	100 700
	Senior	12 366	257 212,80	102 900
	Test manager	15 368	319 654,40	127,900
the Specialist in the field of stress testing	Junior	12 464	259 251,20	103 700
	Middle	12 677	263 681,60	105 500
	Senior	15 183	315 806,40	126 300
	Test manager	15 183	315 806,40	126,300

Source – the tender documentation of Sberbank

* Values of a gross salary are received by division of a monthly rate of specialists into the coefficient 2.5 considering the social assignments, taxes, overheads, profit margin calculated as the corresponding derivative values from the wage fund (WF). Market values of this coefficient usually are in range of 2.3-2.7.

The method of calculation of cost which is based not on rates of contractors, and on the cost of the standard transactions executed during the project of testing is rather often used. In this case the customer sets the maximum value assessments of each transaction, and NMTsK is calculated by the number of transactions of each type which should be executed in the project (preparation of a test case, a test run, preparation of the report, etc.). The competition at biddings happens concerning reduction in cost of standard transactions.

At the same time it is only necessary to remember that during the competitive procedures the cost of the contract "is rather seriously stamped" - quite often for 25-30%. Besides, practice of big customers provides additional negotiations with winners of tenders regarding additional reduction of price (quite often for 5-10%).

One more aspect of cost of contracts which it is necessary to know and remember is the system of the penalties provided for the contractor of the contract in case of not achievement by it of the set criteria of quality.

Penalties against contractors register in contracts and are applied practically by all big customers. However, penalties are not a panacea. Failure of the project of testing of a business and critical system is usually fraught for the customer with serious losses, not paying off penalties.

S. Lunev, X5 Retail Group:

For the management (X5, a comment of TAdviser) does not matter on whose specifically to fault there is a failure of the project. Explain to the Chief information officer that terms were not sustained by outsourcers – pathetic justification. Effects of similar failure for X5 with its volumes – short-reception of profit, nonserviceability of service. At the same time it is frivolous to console itself in the fact that the careless contractor is fined – losses of profit on not project implementation do not become covered by any penalties exposed to the outsourcer. It is possible to expose to the outsourcer such penalties that he will just stop existence, but it will not help us.

The current state of the market of software testing in Russia

Today's market situation of software testing is characterized as follows:

1. The market was created is informative.

• The main set of services in the market is clear, methods of rendering these services are clear.
• Dynamics of demand of different services is clear.
• There are organizational forms and work benches of rendering services

2. The market was quite created in terms of demand and supply.

• There are customers with clear requirements, clear forecasts for solvency, clear mechanisms of communications.
• There are providers of services capable to provide execution of all basic needs of customers.
• Rather stable pool of the providers creating the market qualitatively also quantitatively contains 10+ of the companies. The general pool of market participants on the party of providers – about 20 + the companies.
• The pool of customers is in number limited, but these are very big customers, the requirements loading all existing providers.

3. The market on key parameters corresponds to universal trends and practicians – the same methodologies, the same tools, the same problems and risks.

The current market condition of software testing in Russia is unambiguously characterized as "the market of customers", i.e., all basic rules of a game in the market, a development trend, dynamics, market size in terms of money are defined by customers of services. In figurative sense it is possible to tell even more strictly: the market of software testing is the market reformatted by one customer – Sberbank of Russia.

In an industry section, despite declarations of providers that they have in the portfolios customers from 4-5 industries the main total revenue in the market forms at the expense of representatives of the banking and financial sector and retail.

Is still, of course, the market of small developers and small providers of testing (is conditional – "the market of handicraftsmen" as opposed to "the industrial market") consisting of tens (if not hundreds) subjects on each party, but this market (despite its potentially considerable volume) is almost immeasurable – concerning its total characteristics it is possible to build guesses only. However, and the testing types of service demanded in this market strongly differ from "the industrial market" – as a rule, this testing of web settings and web interfaces, setup of performance of web applications, UX/UI testing, etc.

Providers of services of testing

Now at the market there are about 20 + the companies providing services of software testing (see Tab. 2). Among them:

• Specialized providers: Aplana, Performance Lab, number of the small companies;
• Large developers of the custom software with the centers of testing: EPAM Systems, Luxoft Professional
• System integrators with specialized divisions of testing of AT Consulting, Bell Integrator, "I-Teco" / ITs I-Teco.

The main companies providers of services, attendees in the market are presented today in the table. It is visible that on an experience in the market of the company rather accurately are divided into groups of "veterans" and "beginners" (the companies created after 2011 treat "beginners"). But at the same time it is necessary to remember that some "beginners", actually, are child structures of "veterans" (for example, ITs "I-Teco") and by experience of the personnel concede nothing to the last. On the other hand, at a number of "veterans" the direction of testing long enough could be in the "monitoring" mode, and its activation happened relatively recently, together with market growth.

Data on total revenue and number of staff of testers are collected from different sources; some digits are estimates of TAdviser (about it there is a mark); where confidence in correctness of assessment was not, it is specified "by N / d" (there are no data). The companies are provided to the sequences, corresponding to total revenue for services of testing in 2017.

Tab. 2. The main providers of services of testing (assessment by 2017)

Name of the company	Year obr.	Staff of testers	Total sales, one thousand rub	Revenue on QA, thousand rubles.
Aplana	2001	650	1 477 000	1 373 000
Exactpro **	2009	560+ **	981 330	883 197 *
EPAM EPAM Systems	2001	800+ *	3,570,000	820,000 *
Luxoft Professional of	2006	N /	7 624 033	762 403 *
ITs "I-Teco" / "I-Teco"	1997	-	19 880 000	538 916
Performance Lab	2008	435	460 000	460 000
Philosophy. IT	2011	150	2,500,000	375,000
Bell Integrator of	2003	N / d	N / d	305,447 *
Jet Infosystems	1993	86	20 432 332	302 900
Technologies of quality (A1QA) of	2013	N /	201 900	201 900
AT Consulting of	2009	N /	6 296 404	111 364
Laboratory of quality	2009	50	97 490	97 490
Technoserv of AS / Technoserv Consulting	1992	-	N / d	134,724
Logrokon	2012	50	N / d	82,010 *
Lanit	1989	N / d	N / d	58,888
Kvoliti Ekspert (Quality Expert)	2013	20	45 300	45 300
Neoflex Consulting of	2005	N /	1 147 223	35 500 *
Premium IT Solution of	2015	N /	115 876	34 763
Kvoliti Servisez (Quality Services)	2015	10 *	20,552	20,552
Desk service	N / d	N / d	N / d	20,385
Cinimex Information Science of	1997	N / D	1 077 366	3 875

* Assessment of TAdviser

** Exactpro (till 2015 was called ITS Expert) specializes in development and software testing for the exchanges, investment banks, brokers. In view of rather narrow specialization and orientation to the western markets, the company "was not lit" neither at big customers of services of testing, nor in contracts with state structures.

Quantitative indices of the market

The total score of size of the market of services of testing was created from the following components received or by search and processing of available information, or calculations for special model:

1. Direct sales of services of testing. It is the scope baseline of the market calculated as the amount of annual total revenue of the companies providers, received from performance of contracts which main subject was a rendering services of software testing. Calculations for this component join revenue received from project implementation at customers of any field of activity – banks, business companies, government institutions.
2. Indirect sales of services of testing in the commercial sector. Total annual total revenue from the services of testing rendered within complex projects of custom software development for commercial customers. This component (at its correct determination) is not crossed with other components of overall assessment.
3. Indirect sales of services of testing in public sector. Total annual volumes of services of testing within projects of creation/development/upgrade federal GIS. This component is also not crossed with other components of overall assessment.

Tab. 3. Components of assessment of size of the market of testing, thousand rubles.

'	2015	2016	2017
Direct sales of services of testing	3 376 763	4 379 054	6,648,787
Indirect sales of services of testing – the commercial sector	2,500,000 *	2,700,000 *	3,000,000 *
Indirect sales of services of testing – public sector	2 800 000	3 000 000	3,250,000
Total:	8 676 763	10 079 054	12,898,787

* Assessment of TAdviser

From Tab. 3 it is visible that the principal component of market size – direct sales of services of testing – rather steadily increased year by year. This stability, of course, was provided, first of all, with Sberbank, and rather sharp accumulation of volumes of testing in integrated VTB / VTB 24 (see Tab. 4).

However, in the current year, by a preliminary estimate, direct sales of services of testing came to "plateau" – their growth in comparison with 2017 is insignificant (though, of course. year did not end yet, but the remained month will hardly seriously change something).

Tab. 4. Purchases of services of testing by Sberbank and VTB / VTB 24, thousand rubles.

Customer	2015	2016	2017	2018
Sberbank of Russia	649 448	2 090 658	1 782 643	1 297 287
VTB / VTB 24	61 704	185 063	423 609	737 718

Invisible players of the market

Public sector – "the sleeping customer"

At assessment of total amount of the market which is stated above the "Indirect Sales of Services of Testing — Public Sector" component was added not in order that the final digit on the market looked more impressively. The public sector of federal level is the group player of the market having high potential, but being now in an inactive status. In 2014-2016 some providers executed contracts on testing (mainly, load and integration) for a number of the federal agencies developing and operating big of GIS – the Pension fund (AIS RPF-2)], the Federal Tax Service (AIS Tax-3), however, similar contracts did not become regular. Generally such situation is caused by the standard reason – presence of the confidant to department of insourcer or not less close development team which closes on itself all services within lifecycle of systems.

As it was already mentioned above, the summary annual budget on creation and development federal GIS is at about in 15 billion rubles; about 15 billion more are annually spent for exploitation of all these by GIS. Business criticality of the state systems, in general, strongly concedes to criticality of financial and banking systems, but from several hundred of existing GIS it is possible to count about 15-20 to which functionality and high availability rather high requirements are imposed. Therefore will not be big error to consider that totally for testing of all of GIS about 15% leave the 15-billion budget of development, i.e., about 2.25 billion rubles.

To assessment received for federal agencies it was decided to add assessment for the largest regional IT department – DIT of Moscow. The annual budget of DIT in 2016 made more than 30 billion rubles, and in 2017 exceeded 45 billion rubles. On structure the budget of DIT is almost similar to the summary budget of federal agencies therefore it is possible to accept that regional GIS is spent for creation and development – in percentage terms – the same share of the budget, as at the federal level (15%). The testing share in this budget of development can be also up is accepted to standard 15%. Thus, estimated costs of DIT of Moscow for testing annually make about 0.7-1 billion rubles.

Among purchases of DIT of Moscow it is possible to detect easily the annual contracts entirely relating to software testing. The subject of these contracts – "Rendering the Services in Technical Expertise of the Information Systems Developed, Upgraded and Operated within the State Program of the City of Moscow " Information City "", annual cost is at the level of 65-70 million rubles. At the same time selection check of several large contracts for development of city ICs (with budgets in 100-300 million rubles) showed that as a part of works costs for testing at the level of 15-25% of total amount are provided in these contracts. Thus, the above-stated calculation of volume of costs for services of testing in DIT of Moscow is conservative and statistically reliable.

"Black hole" of the market – "pocket" insourcers

The image of an astronomical "black hole" is not casually applied to "pocket" insourcers of state corporations. It is clear, that state corporations for the normal functioning need large and difficult IT infrastructure. This infrastructure exists, develops and operated, receiving nearly 100% of necessary volume and a range of IT services from the approximate companies insourcers. As "the black hole" in astrophysics does not produce light for the horizon of events, and insourcers practically do not issue IT budgets of state corporations out of limits of the "horizon of finance". The only exception among insourcers – SberTech which is a unique example of the "pocket" insourcer which is widely using outsourcing of IT services in the activity.

Try to estimate virtual (to be exact – imaginary) the volume of services of testing falling on the largest insourcers it is senseless – access to this quasi-market for strangers is reliably closed. But this market as gravitational field of "a black hole", quite noticeably affects the real market of services of testing – specifically, its human resources. Testing specialists leave from providers not only to customers within the observed market, but also on other side of "the horizon of finance" - to insourcers. So far it occurs not in such scales which in due time (2014-2016) were shown by SberTech, but the threat is quite real – especially in the context of the approaching trend of digital transformation (see the section "Trends and market prospects").

It is possible to estimate the potential amount of this threat, having compared, for example, the main characteristics of two insourcers – Sibinteka and SberTech. Annual total revenue at both companies is approximately identical, so, it is possible to assume, as profiles of the serviced IT infrastructure can be compared with a certain coefficient of similarity. Of course, IT infrastructure of Sberbank, owing to higher business criticality and loading, is significantly more difficult than infrastructure of Rosneft serviced Sibinteky. Probably, will not be big error to consider that the complexity factor "SberTech to Sibintek" matters in the range of 3-5. It means that requirement of Sibintek for specific IT resources (software developers, testers) concedes to requirement of SberTech by 3-5 times. At the same time internal and external requirement of SberTech for testers (before reorganization of 2018) is estimated approximately at 3 thousand people. Thus, the current demand of Sibintek in testers can be estimated approximately at 600-1000 people. How many from this number today really works in insourcer, it is unknown. But if in the near future Sibintek comes to public market for testers, then it without effort will absorb the company of the size of Aplana or Performance Lab.

Sberbank of Russia – the strategist in the software testing market

The role of Sberbank of Russia in formation of current market condition of software testing (both in financial, and in all other aspects of the market – competitive, qualification, resource) is so significant that by preparation of the overview the decision to devote to detailed consideration of a phenomenon of Sberbank the separate section was made. Of course, it is necessary to specify that in many respects this phenomenon was implemented on the considered interval of time (2015-2017) through IT "subsidiary" of Sberbank – SberTech company

Everything that occurs today in the testing market, is anyway connected with requirements of Sberbank:

• For 2015-2017 the volume of the services of testing which are purchased by Sberbank was, by different estimates (including, the providers proceeding from representatives being "in the market"), 20-25% of total amount of the market. Assessment executed by preparation of this overview showed that the share of Sberbank in the considered market in 2015-2017 almost reached 50% (see Tab. 5)!
• The demand in specific types of service of testing (and, respectively, demand and number of specialists of certain qualifications) was defined in the market in many respects by requirements of Sberbank;
• In the competitive materials Sberbank set the maximum admissible rates of specialists in testing (and during tenders also tried to obtain their considerable decrease) that outright affected the wage level which the companies providers of services are ready to pay to specialists;
• Need to optimize staff costs forced practically all companies providers to the movement to regions;
• In addition to influence on the salaries of testers in the companies providers, Sberbank (through IT "subsidiary" – SberTech) actually "vacuumed" the personnel market of specialists in testing (as earlier it arrived also with the market of developers, and with other markets of IT specialists). During the poll by preparation of the overview practically all companies noted the facts of buying up of specialists by Sberbank with the offer significantly bigger (approximately twice higher market) the level of salaries.

Tab. 5. Assessment of a share of Sberbank in the software testing market

'	2015	2016	2017
Total amount of the market, thousand rubles *	3 376 763	4 379 054	6,628,402
Purchases of Sberbank, thousand rubles	649 448	2 090 658	1,782,643
Share of Sberbank, %%	19%	48%	27%

* B of Tab. 5 is understood as total amount only a component of direct sales

Department of quality of SberTech – as the player of the market of testing

Speaking about a key role of Sberbank in formation (and reformatting) the market of software testing in Russia, it must be kept in mind that this role through one of structural divisions of SberTech – the Department of Quality (DQ) was implemented.

As well as all SberTech, in 2014-2016 recreation center very quickly developed in quantitative and structural aspects. In the FB-account of the chief of one of managements of recreation center, there is a photo dated 2014 with the signature "… and these are only 1/5 my new commands" (on a photo – 15 people in the course of team building); in 2 years the same photo had a new comment – "... and today it is only 1/20 most professional commands", i.e., for 2 years (2014-2016) one of managements of recreation center grew from 75 to 300 people. The general number of recreation center approached the end of 2016 to 1500 people – in two years from the market (i.e., actually, from other specialized companies) more than 1000 people were taken.

The top manager of one of the companies representing testing services remembers:

When they employed this one thousand people, it was 2014 - 2016, in the market the horror was created. Salaries flew up, enticed our people, just multiplying salary by 2. At us many testers left there therefore there strong practice.

Such aggressive growth was caused by the development of the new technology platform of Sberbank including 3 components which began in 2015 – the Single Frontal System (SFS), the Platform of Support of Business Development (PSBD) and Factory of data. Full translation on a new technology platform of clients, products and data is going to be completed in 2020.

Anton Romanov, the assistant department head of testing of corporate credit systems of recreation center (from an interview of 2017):

In Department of quality about 1,500 people work, these are 38 departments. About five managements which are divided by functional competences turn out: Management of stress testing (and automation, a comment of TAdviser), Management of testing of the automated banking systems, Management of testing of the frontal systems, Management of administration. And in each management somewhere on five departments.

Comment of TAdviser: The above-stated quote is given according to^[4]. It is obvious that with digits on structure of recreation center there is a certain discrepancy – the stated total number of departments and managements gives a median number of departments in each management at the level of 7-8 (and the average number of one department about 40 people).

And even at such own staff of testers exceeding the number of the largest specialized providers – Aplana and EPAM Systems companies – more, than twice, the recreation center could not provide independently with a performing resource all that volume of testing which was caused by development of the new systems. For this reason in 2015 the first megatender of Sberbank by request of testing services, and a year later – the second took place.

Now the fate of recreation center in the reorganized SberTech is unclear. On the basis of sketchy publications and insider news it is possible to draw a conclusion that the bulk of testers from staff of recreation center (practically all who were engaged with testing in the projects using methodologies Agile/DevOps) is transferred to IT and block Sberbank, and, not in the status of testers, and in the status of the developers working in a paradigm of CD/CI (Continuous Delivery/Continuous Integration). But, as development of platform solutions EFS and PPRB remained in SberTech, it is possible to assume that some part of the state of recreation center (most likely, from managements of testing of the frontal systems and testing of the core banking system) also remained in SberTech (see an interview of TAdviser "CIO Sberbank - about transfer of thousands IT shnikov in bank and the fate of SberTech").

The general statistics on purchases of Sberbank in the field of software testing

During preparation of the overview from several open sources (EIS in the field of state procurements, Sberbank ETP, the project Goszatrata) collected data on the majority of the purchases of Sberbank relating to software testing, for the period 2015-2018. At the same time selection of data was made only on purchases in which the subject of purchase in an explicit form announced one or several types of software testing. For the analysis were selected as complete purchases (in such cases the contracts signed according to the results of procurement procedures were analyzed), and the purchases which are at stages of collecting of requests, works of a tender committee and summing up.

The purchases relating to creation new or development/upgrade of the existing information systems in an array of the analyzed data did not join though, clear, as as a part of such contracts surely there is a testing as one of work types.

It is reasonable to carry out the analysis of purchases of Sberbank in the field of testing not by dates of purchases, and on completion dates of the relevant contracts. In 2015-2017 (and partially in 2018) the rule of thumb worked: the necessary volume of services in FT was bought for the year ahead, and services in NT, depending on the planned duration of their carrying out were distributed approximately equally between a year of purchase and next year.

In 2016-2017 the lion share of annual purchasing amounts fell on megatenders. About price and quantitative indices of these tenders wrote all profile media in due time (including and TAdviser – see. "Saving more than 1 billion rubles at tenders on testing of automated systems"). Though, and in addition to them, every year there were several tens of purchases to small volumes of services in each of them.

As in 2017 megatenders were not, all purchases of 2017-2018 were made with significantly smaller volumes of each single purchase, but at significantly the increased number of purchases.

Statistics of purchases of Sberbank by providers and by years is presented in Tab. 6.

Tab. 6. Distribution of purchases of Sberbank by the main providers, one thousand rub

Provider	2015	2016	2017	2018	All 2015-2018
Aplana	384 547	1 131 462	430 921	439 469	2,386,399
ITs I-Teco / I-Teco	207 946	736 900	460 100	257 184	1,662,130
Performance Lab	-	-	302 325	129 379	431,704
Bell Integrator	35 811	104 230	133 706	108 408	382,155
Jet Infosystems	-	34 984	146 278	192 400	373,662
AT Consulting	3 070	822	111 364	60 322	175,578
Luxoft Professional	12 954	73 898	60 412	-	147,264
Technoserv EXPERT / Technoserv Consulting	5 120	3 371	90 276	34 706	133,473
Desk service	-	-	20 385	20 385	40,770
Premium IT Solution	-	-	-	12,941	12,941
OCK	-	-	2 616	7 848	10,464
Kvoliti Servisez	-	-	-	9,166	9,166
Cinimex of the Information scientist	-	4 991	3 875	-	8,866
Kvoliti Ekspert	-	-	-	3,764	3,764
INNO Consulting	-	-	-	930	930
Total by years:	649 448	2 090 658	1 762 258	1 276 902	5,779,266

From the table it is well visible that the main purchasing amount of services of testing by Sberbank fell on 2016-2017. The current year shows certain "shrinkage" of purchases, but this results from the fact that a part of services of testing went from under direct procurements to the purchases connected with software development.

Total amount of procurements by Sberbank of the services of testing conducted in 2018 is about 1.13 billion; from them about 450 million rubles fall on contracts which will be performed in 2019 (and it is all known budget of Sberbank for services of testing for 2019 so far).

Deficit of qualified personnel – the main problem of the market

Problems and risks of the market are connected with its structure, dynamics and an external environment. Pacing factors of problems and risks:

• dependence on a small amount of big customers;
• tendency to formation of insors-resources of testing at today's large customers creating the market;
• existence of "a black hole" in the form of external insourcers at large government and commercial structures (The Gazprom, Rosneft, Rosatom, etc.)

The main factor of risk and the main problem for several years is permanent deficit of qualified personnel.

Now the market, in terms of a current demand in direct contractors, specialists testers, is in conditions of permanent staff deficit – it is noted by both providers, and customers who also regularly look for testing specialists to themselves to the state. Orientation to building of insors-resources of testing in the companies creating the market is, according to almost all providers, one of the main threats for the market.

The number of staff of testers at the main providers increases by tens and hundreds of units year by year, but they all the same are not enough. At the same time often it is necessary to take to the state enough "crude" candidates and to bring them to standard the accelerated training with the subsequent quick entry in "fighting" projects.

Yury Kovalyov, CEO of Performance Lab company:

Since 2016 in the market the strong staff deficit is observed. Sharply there are not enough testing specialists, especially initial level. Salaries flew up for 40-60%. For training of the interns ready to work, we spend 3-4 weeks of training. The personnel flows away both to competitors, and to clients.

Customers also became more exacting to qualification of contractors. If earlier only the minimum experience of practical work (normally – of 6 months) was the standard requirement, then now at competitive documents even more often there are requirements for qualitative line-up. So, Sberbank allows existence in teams of testers only of 10% of specialists with work experience in projects of testing less than 1 year now; the condition about existence not less than 10% of specialists with practical experience of 3 years and more is at the same time laid down.

By preparation of the overview one of interlocutors of TAdviser who is at the moment on the party of the customer, but having and work experience in provider, noted that a situation in the market such is that if today the new customer with requirements of scale of Sberbank or VTB suddenly appears, then any provider from existing will not be able to pick up this customer – due to the lack of free resources of contractors. Moreover, the customer of such scale also several providers who integrated in a pool – because of the same staff shortage will not be able "to digest". Of course, emergence of "the second Sberbank" a situation more likely fantastic, but where a guarantee that the customer of type of Gazprom or Rosatom suddenly will not change the IT strategy and will not begin to attract resources from public market? Eventually, possibilities of their "home" insourcers are also not infinite.

Experience of the Aplana company which systemically approached the solution of this problem deserves attention. In 2016, suffering a shortage of specialists in the automated testing (at this time only performance of the contracts in Sberbank won the previous year on the first of mega competitions demanded from the company of ensuring permanent presence "in the field" not less than 30 specialists in AFT), Aplana made the decision on the organization of the open corporate university based on the existing training center. In this idea the fact that the applicant "from the street", successfully passed entrance test could become the student of the university free of charge was essentially new.

For 2 years which passed from the moment of creation of the corporate university of Aplana, applications for training were submitted by nearly 1000 people. After consideration of requests for entrance testing about a half of candidates was invited. In only two years 20 groups are let out; 196 people were trained. At the moment in the company 102 employees recruited after training at the corporate university work.

Trends and market prospects

1. Stable demand for testing services

In the near future (2019-2021) demand for testing services, at least, will remain at the level reached in 2016-2017.

Alla Prokhorov, CEO of Aplana company:

One of the biggest segments is RFT. With it there begin testing outsourcing most of new customers. As development processes and testings in the company grow ripe, more demanded is AFT. Sberbank together with SberTech which is today one of the innovation centers in the field of IT in general one of the first approves new methodologies and new tools, including, and in the field of software testing. In Sberbank the emphasis is now placed on increase in a share of the automated testing, and in the next 2-3 years we will see explicit growth in this direction there. But it does not mean that similar growth will be in the companies which are at earlier maturities of processes of testing. Stress testing in itself is in demand and its volume will increase as the volume of the developed applications increases.

2. Changes in structure of demanded services

Changes happen already now:

• In tenders of Sberbank remaining by the main fashion-maker in the market requirements about presence at specialists in testing of experience of execution of works in the Agile/DevOps models even more often appear
• Specialization of the project teams involved in testing accomplishment also changes – the need for specialists in AFT increases. If earlier the share of such specialists made about 20% in the announced competitions, then in 2018 the quota for specialists in AFT already increased to 30%. It completely corresponds to the trends mentioned in the report of "World Quality Report 2017-2018" so, in this direction the Russian market of QA is synchronized with world.

3. Regional vector of development of providers

For the period of the "low" market what was a software testing segment in two years prior to Big Bang generated by tenders of Sberbank, the movement of providers of services to regions was quite justified tactics. Suppliers of any IT services for which the location of the specific contractor does not matter for the receiver of result of rendering service acted with a similar image then (and continue to work now).

Today practically all main players of the market have regional offices. In addition to solution of the problem of cost of resources of contractors, it also theoretically gives sales opportunity of projects of testing in regions of presence of local offices. However, about regional projects on testing of still significant news did not appear.

4. Foreign markets as reserve airfield

Though the negative scenario (stagnation or reduction in demand) for the Russian market of testing looks as improbable, absolutely you should not exclude it from consideration.

In the conditions of stabilization of the Russian market of software testing practically all main players of the market will inevitably face a problem of search of new orders and new customers. The western markets for such search are represented by almost only possible option.

Many providers of services considered in this overview have not only regional offices in Russia, but also foreign representations. Such giants as Luxoft Professional and EPAM Systems, in general are the open public companies with the headquarters in the West, and for them Russia only one of regions of presence (and not the biggest in terms of revenue.

The Performance Lab company in 2014-2015 also made efforts for entry into the western market. It was announced opening of office Performance Lab US, LLC in the city of Santa Clara (California, the USA) and signing of the contracts for testing with the innovative companies of Splunk and Illumina.

Situation in the market in 2018 and the forecast for 2019-2020.

The market of services of testing in 2018

The main events which had an impact on the market of services of testing in 2018 became:

1. Reorganization of SberTech (actually – liquidation of SberTech recreation Center in its usual to the market a type). As it was already mentioned, externally in the market it was expressed that in purchases of services of testing by Sberbank there were much more purchases in quantitative expression with simultaneous decrease in average volumes of single purchases, i.e., the trend of decentralization of purchases of services of testing was shown. The reason of it is clear – now these purchases are initiated in divisions of Sberbank at the level of which Agile/DevOps-development of the new systems and where, respectively, a considerable part of testers – now in the status of DevOps-developer migrates (or already migrated) concentrates.

Estimating requirement of Sberbank for testing services, it is necessary to understand that formally (if to consider only direct procurements) this volume decreased in the current year in comparison with 2016-2017, but actually it increased – though, for demonstration of this growth it should analyze routinely all contracts of Sberbank relating to creation and development of software. In that part of program developments of Sberbank which is executed on Agile/DevOps methodology testing can even obviously and not be present at specifications of the performed works, however, in practice it, of course, takes place and its volumes did not decrease.

2. Consolidation VTB and VTB 24. In the current year the volume of direct procurements of services of testing in VTB united bank got significantly closer to the purchasing amount of Sberbank – 737.7 million rubles against 1,276.9 million rubles at Sberbank. And, purchases of VTB in 2018 showed significant growth (more, than for 300 million rubles), and purchases of Sberbank in comparison with 2017 were reduced more, than by 500 million rubles. But here it is necessary to consider (about it it was already told above) that a part of requirement of Sberbank for testing services "is disguised" in services in software development now. By the way, and in VTB the same trend - in addition to direct procurements of services of testing is observed, in considerable volumes services in creation and development of information systems in which also testing services are placed are bought. Sampling analysis of large contracts of VTB on software development showed that in budgets of these projects the considerable volume of services of testing – 20-25% of the amounts of contracts is put.

3. Start in operation and expansion of scales of use of a row large GIS. In 2018 operation of several state and quasi-state ICs of federal scale – GIS of housing and public utilities, EIS of housing construction (ДОМ.РФ), the Single Biometric System (SBS) began; the functionality extended and the number of users in the systems of the Pension fund (AIS RPF-2), the Ministry of Finance and Treasury (GIIS "Electronic Budget"), FTS (AIS Tax-3), the Ministry of Telecom and Mass Communications (The portal of state services and Infrastructure of the electronic government) increased.

Present load of these systems is incomparable to load of the business and critical systems of banks, but effort of regulators on giving to these systems of the status infrastructure within digital economy certainly will lead to increase in requirements for functionality, reliability and availability. So far the closed "club" of the developers who are regularly winning competitions within state procurements more or less copes with the current development and testing of these systems. However, the cases of failures and incorrect functioning which are rather often becoming publicly known big GIS, drawn attention even of such control instances as Accounts Chamber and Control management of Presidential Administration, in the near future will inevitably lead to toughening of requirements for testing and acceptance tests that, in turn, will force customers to address from a public sector for services of professional providers of services of testing.

In 2018 to speak about quantitative assessment of the market for the present early – a considerable part of this assessment is made by non-public revenue of the companies which volumes will become available to assessment and the analysis only at the beginning of 2019, but the general opinion of analysts and providers with whom TAdviser discussed this issue, are inclined to consider that the market saved, let and small, but positive dynamics. As indirect confirmation of this assessment serves that at the competitions held by the largest customers regularly appear (though, and without noticeable progress) the new companies from category of medium and small business (no more than 10 people, some with staff of testers).

The forecast for 2019-2020.

1. Market size

Market size in the next two years should not change considerably towards reduction (such pessimistic version of the forecast exists among providers to whom TAdviser - because of reorganization of SberTech talked), towards increase.

Sberbank and VTB will continue to support the market by the need for services of testing at the level of not lower than 1.2-1.5 billion rubles on each of banks. Other banks from Top-20 will also remain among the largest customers though, and will concede on volumes of the purchases to Big Two (so, for example, the FC Otkritie bank in the published purchase plan for 2018 designated the need for services of testing at the level of 300 million rubles – and, all volume of these services is referred to 2019).

Purchasing amount from large retail networks (the second most important industry segment for the testing market), most likely will also remain at the current level.

Appearance of new big customers is theoretically possible only from spheres of a telecom and/or state structures, but neither that, nor another (conditional) segments of the market give the grounds on succession of events according to the similar scenario yet.

As the strategy of Sberbank on creation of a new technology platform which is one of the main drivers of the market of services of testing (and not only it), is calculated till 2020, the providers working with Sberbank can regard the business perspectives for the next two years as rather stable.

The predicted stability of the market (regarding direct sales of services of testing) is caused by the fact that in all listed industry segments (banks, retail, a telecom) the list of players (both outsourcers, and insourcers), the model and volume of consumption of services directly connected with volumes of the executed software development settled.

From the industry obvious signals to the sharp consumption growth of services of development are not visible yet (and, respectively, testing services).

There is also one more segment which was not considered in the overview – the Internet company. The largest player of this Yandex market conducts software developments in large volumes, but mainly all this volume of development also tests; a similar situation – with Mail.Ru Group. It is possible to expect possible entry into the market of services of testing as providers from these Internet giant more likely, but not as customers.

The Internet focused IT companies of the smaller size (Internet trade, services on the basis of IT platforms, mobile applications) consume outsourcing services of testing already today and, in principle, could become new drivers of the market, but their specific weight in the general structure of economy is not so high. Therefore even significant growth within a potential segment hardly somehow will cardinally affect the size of the market of development and testing.

2. Increase in need for "Agile-ready" specialists and changes in a range of services

The increasing number of customers – generally the large banks following the example of Sberbank, but also not only they – pass to development into Agile/DevOps methodologies. The initial inclusiveness of the SQA procedures in development processes based on Agile/DevOps means that the customers practicing such style of creation of software impose essentially new requirements to test procedures (testing becomes an integral part of development process, the frequency of its accomplishment increases, requirements to automation increase) and, respectively, in requirements to testers points on knowledge and practical experience of work in the context of these methodologies even more often appear.

Lev Shelkovoy, development director of information technologies, Bell Integrator:

Adjusting development of these or those directions in our company, we accurately realize what specialists will be demanded in 3-4 years. Among them there are autotesters, testers such the practician as CI/CD, testers in the DevOps model, testers with a bias in robotization. And here the trend of functional testing, in our opinion, in the next 2-3 years will decline.

For providers of services of testing it is both a new call, and a new opportunity. With one, the parties, it will be required to provide additional training of testers and to increase their competences of the new directions, but, on the other hand, charge rates of specialists in the automated types of testing are 1.2-1.3 times higher, than rates for testers with skills only of manual testing.

Besides, expansion of volumes of the automated testing can reduce terms of implementation of long projects and promote increase in revenue of providers due to accomplishment of bigger number of projects.

3. Influence of import substitution policy and western sanctions

Since 2015, in Russia the policy on import substitution of the western software is pursued. It is clear, that any similar replacements in architecture of complex information systems shall be followed by considerable volumes of testing. For the main pool of today's clients of the companies providers of testing the import substitution problem will hardly arise in 2019, however, in one and a half-two years "coercion to the register of domestic software" can already reach also them.

In general, the probability of succession of events according to such scenario for giants of type of Sberbank is small, but, for example, for state corporations it is quite actual – in the market already enough examples when in corporate systems replacement of western IT - "bases", on the Russian analogs is made.

4. Influence of the Digital Economy program and trend on digital transformation

The national project "Digital Economy" about which spoke much in 2017-2018, at last, found financial outlines and, since 2019, its implementation will go in six main directions (see. "Digital economy of Russia"), three of which – "Information infrastructure", Information security and "Digital public administration" - can become generators of a significant amount of new program developments in a public sector in the next 2-3 years. And it can be systems with higher functional requirements and characteristics of performance, i.e., and potential volumes of necessary testing at these systems will be significantly higher, and, therefore, the state component in a total amount of the market of testing will also have to increase.

Now (at least, declare it) state corporations most actively are engaged in digital transformation of the activity. Practically all corporations have "pocket" insourcers, but the volume of developments (and, therefore, and testings) at digital transformation can exceed possibilities of insourcers (a sheaf Sberbank SberTech which it is possible to watch owing to its openness – the real proof of this postulate). Therefore "pocket" insourcers can go marketing – and behind services (it will be chance for the market), and behind resources for themselves (it will be direct and explicit threat to the market).

Research methodology

The main metrics and hypotheses used for market assessment

Within this overview it was necessary to give repeatedly the quantitative estimates which are based on certain assumptions and the assumptions. The main of the used assumptions and short justifications of their acceptability are given below:

1. One of the main estimated metrics used in the overview are the average annual revenues provided with one specialist tester. The analysis of rather big array of objective data (open publications of the tender documentation of the largest customers, authentic data on the number of staff and annual revenues of key providers) showed that this value is in range from 1 to 4 million rubles a year. It should be noted that digit of annual revenues in 1 million rubles rather modest and corresponds to salary at about 35 thousand rubles a month – such salary is acceptable for IT specialists of the initial and average level in regions, but, at least, the salaries of specialists of the same qualification in Moscow and St. Petersburg are twice less. Therefore in estimated calculations (where, for example, through value of single revenue and a hypothesis of number of the involved specialists annual revenues of the company provider or the budget spent for testing by customer company were calculated) as value of a metrics the value of 1.5-2 million rubles was used. Range of value is designed to consider features of activity of the estimated company – types of the provided services, existence of regional offices, market history.

2. At assessment of total revenue received for testing services for the companies representing full range of services certain assumptions of revenue share for testing in the known gross revenue of the company became:

• For small (10-20 specialists) and averages (20-50 specialists) of the companies providing only services of custom software development and testing revenue ratio was accepted to testing and revenues on other services at the level of 1:1;
• For the large companies of a "softovy" profile (i.e., not selling IT - "iron" and related services) representing services of development, testing and maintenance of information systems, revenue share for testing was determined by more difficult algorithm. A half of revenue of the company belonged on services of maintenance of software in which usually there are no testing services, and the second half - on software development services, a certain share in which was occupied by testing services. The specific size of a share of services of testing (settlement coefficient multiplier) was defined by purpose of the developed systems, their scale, complexity and business criticality. So, for example, the share of services of testing as a part of the cost of the business and critical high-loaded systems was accepted in the range from 20 to 30% of the total cost (it is rather conservative assessment), and for less critical systems cost portion of testing was accepted in the range from 10 to 20%.
• For the companies which are engaged in addition to development/maintenance of software, also supplies of equipment carried out in the beginning separation of total sales into "iron" and "softovy" parts (such division could be executed in the range from 50:50 to 70:30 for benefit of "iron" - depending on the known profile of activity of the company), and then the above described procedures were applied to the remained "softovy" part of revenue.

3. In some cases assessment for customer companies was executed at the enlarged level – proceeding from the total value of the IT budget of the customer (or homogeneous group of customers – for example, federal agencies) and the known profile of expenses as a part of such budget. So, the profile of expenses of large "pocket" IT insourcers at state corporations was accepted identical to the profile of expenses of SberTech which is also "pocket" insourcer (the truth, with a considerable volume of consumption of outsourcing services).

More detailed description of a technique of estimated calculations is beyond the overview and will not be provided here. In all places of the overview where the estimates which are based on the assumptions and metrics described above will form references to this subsection will be given.

Assessment methodology

Receiving assessment of size of the market of services of software testing in terms of money was one of main goals of a research of TAdviser. In a work progress the decision on accomplishment of assessment on the period of 2015-2017 and the analysis of a situation in 2018 with formation of the forecast for the current year was made.

Assessment formed, proceeding from the following facts and assumptions:

1. There is rather compact group of about 20 + the companies providers of services of testing whose revenue substantially (from 70 to 100% of volume) forms due to project implementation, the main (or) which purpose accomplishment of one or several types of testing (the manual / automated functional testing, stress testing and their modifications) commercially is only (with costs of contracts from 1-3 million rubles to hundreds of millions of rubles). As main customers for this group of providers the largest banks, some federal and regional state customers perform with the developed IT infrastructure, large telecom operators and retail retail chain stores.

The most part of information on contracts of the specified customers is available publicly (or in EIS of state procurements, either on electronic marketplaces, or on the corporate websites in sections Purchases/tenders). Thus, a considerable part of total amount of the market from customers - with dynamics by years, with breakdown by types of service, etc. – can be analyzed and used in other parts of a design model.

On the other hand, financial performance of activity of the companies providers rendering not only service of testing, is available in services like " circuit.focus " and SPARK-Interfax. On condition of knowledge (or plausible hypotheses) about what share of revenue of providers forms from contracts on software testing, assessment of a part of market size from providers of services can be received – by use of the corresponding correction factors.

Correction factors for the total revenue of each of the companies providers were established in the range from 0.2 to 1 by expert evaluation, taking into account all available information on activity of the companies. At expert evaluation conservative approach was used – at insufficiency of the proving information on the specific company the minimum value was appropriated to coefficient multiplier.

In some cases, in the absence of reliable information about total sales of the company and/or its distribution by types of activity, as value of revenue the doubled value of revenue received in projects, information on which is available publicly, was accepted to services of testing (i.e., the assumption became that the company provider received equal shares of revenue from the public and non-public projects).

2. There is a compact group of the specialized IT companies (EPAM Systems, Luxoft Professional, Neoflex Consulting, I-Teco) rendering services of full-scale custom development and deployment of software with project budgets from several tens millions of rubles to several billion. These companies have at the order big development teams of software with a full range of specializations, including also testers. At some of such companies (for example, in EPAM Systems) all testers treat the uniform corporate Center of testing and ensuring quality of software and, actually, execute internal projects of testing within complex projects of custom development.

On condition of knowledge (or plausible hypotheses) about what share from the general project budgets of custom development is the share of testing, assessment of a part of market size from software developers can be received.

Also that circumstance that the companies of this group practically never participate in "blank" projects on software testing is important. Thus, the total revenue of developer companies supplements the total revenue of the companies providers, practically without being crossed with them on the general projects and customers.

For formation of values of the correction factors used in calculations for this group the analysis of several tens publicly available large contracts on custom software development containing in the structure considerable amounts of works on testing was carried out. It was as a result established that works on testing make from 6-7% to 25% of total amount of the contract.

On the basis of the carried-out analysis, for assessment on the companies of this group the value of the correction factor 0.15 was selected that will well be approved with statistical data on a share of costs for testing in large projects of custom software development.

3. There is also a segment of federal state customers (about 80 ministries, the agencies and services) having several hundred state information systems (SIS) at the order. About 15 billion rubles annually are spent for creation new and development/upgrade existing GIS at the federal level. It is natural that within these volumes are financed as well works on software testing – practically always only within government contracts on creation/development of GIS; purposeful projects on testing are rare exceptions of the general rule. Nevertheless, assessment of volume of services of software testing is important for this segment because these services also introduce the share in the general amount of the market.

The list of the companies making the Russian market of services of software testing included 21 companies (see Tab. 2).

Reliable information about annual volumes of sales returns of services of software testing was obtained from 3 companies entering in Top-10 of the market (Aplana, Performance Lab, Jet Infosystems). The approaches described above were applied to estimates on other companies.

So, for example, assessment for EPAM Systems company was based on the following initial assumptions:

• It is known the total number of the company on a global scale (is openly published); it is also known (total number of the Center of testing of the company almost annually occurs in press releases);
• Total number of the staff of the company in Russia (about 5 thousand people) is known. Proceeding from the assumption that a proportion of number of testers in Russia same as in other regions of activity of the company, the number of testers was accepted at the level of 800 + the person.
• Average annual revenues are in projects of testing for one tester ranging from 1 to 4 million rubles – it corresponds to the calculations made on the basis of rates of the testers announced in tenders and on the basis of the actual data on annual total revenue and number of staff on "net" testers – such as Aplana and Performance Lab. Conservative assessment – 1 million rubles a year on the tester was applied to EPAM Systems that corresponds to the strategy of the company forming teams of specialists both from Moscow and St. Petersburg and from the regional centers where the cost level is lower than capital.

Notes