High-speed performance of EDMS – how to compare two platforms by this parameter

The attention to high-speed performance leads to the fact that today on the websites of producers of systems among other utilization properties also high-speed performance, as a rule in the "time of opening of a card" format, or even more generally – "a system response time" is specified. One vendor writes "3 sec.", and at other product "2 sec." is written - leaves, by this parameter it is one and a half times better. But "2 sec." is a response time on what? what opening of a card? Under what loading how loading was measured and how time is measured? And what it should be? If it is about platforms – in what application with what business logic measurements were taken?

In this article we will clear a situation that such high-speed performance of ESM/SED systems that you could estimate reliability of information provided by developers of systems and learned to compare competently by this parameter the platform which you select for implementation.

What does the developer pay in high-speed performance for?

Designing any ESM/SED platform, the developer aims "squeeze" in it as much as possible ready applied functionality to create solutions on the platform was quicker, more simply and more reliably and that the solution designer could focus on the customer's business challenge entirely. The ultimate goal - to increase system value for the customer.

Reverse side of rich functionality is the large volume of the code and its complexity. For example, as soon as you added a role model of access control to the platform, you should calculate constantly availability of an object before transaction with this object. What will slow down an output of search results and navigation in any solution

On the other hand, the developer aims to implement as much as possible high-level business model and to provide the speed of creation and modification of solutions on the platform. Eventually, it allows to reduce costs in an implementation project. A reverse side of the high level of a business model is the large volume of the calculations connected with interpretation of setup data. Sometimes from this it is possible to leave by preliminary compilation of settings and use of the ready code in the operation mode, however it cuts off a possibility of visualization of a current status of model. In general it also adds the number of calculations.

There is nothing new in these reasons. It is necessary to pay for any complexity. In case of business platforms it is necessary to pay in high-speed performance.

Does not happen, and the developed platforms of the business applications showing ultrahigh high-speed performance cannot be functional and "model". Such high-speed performance can demonstrate that "money" was not paid for development of the platform, and she is "underdeveloped". There are, of course, also just advertizing gimmicks when real high-speed performance is much worse declared.

How high-speed performance of the platform is measured?

It is necessary to pay for complexity, but the you "will spend" less high-speed performance for it, the more will remain to the user other things being equal. To "other equal" the functionality of the solution on the platform belongs, first of all.

To release a competitive product, the developer needs to be able to measure precisely final high-speed performance and to control it in relation to any universally recognized norm.

Standards of high-speed performance

For example, the order of the Ministry of Telecom and Mass Communications 221 of 2011 describes requirements of access to an electronic document management system. In them standard time of a response of a system is determined no more than 3 seconds. In different situations it is necessary to deal with regulations of response time of the program, realizing at the same time that ideal response time there will be no 0 seconds at all. Who does not trust – remember that on smartphones animation is very widely used though to achieve there issue of result in 0 seconds not a problem. I specially tried to turn off animation in the application for reading Google Play Books – the page is replaced instantly, but from such reading the head quickly begins to hurt if the finger trembles it is possible to thumb through several pages at once and then bewildered to get a grasp of the text.

With an accuracy of microsecond to tell what time is ideal, will not be able nobody and never, however to estimate time as acceptable, simply. It "does not have to be worse than others".

Simple example. I work with Microsoft Outlook long ago and I consider response time of this program acceptable. Measured it a stop watch – on the home computer the letter opens from the list entering approximately in 2 seconds, on the worker – for 3. Personally for me 3 seconds – acceptable time, as well as 2 seconds, and differences between these digits I do not feel. Also I will not consider "brake" other application for work with a set of documents with similar high-speed performance

Now it is necessary to deal with what needs to be measured for high-speed performance determination. The answer, apparently, is obvious – that response time of the program and it is necessary to measure. But if we speak about the EDMS/ECM platform, then the user practically never works with the platform, he deals with the application solution on its base, and response time of the program with which it works concerns him. The simple example – is the platform of development a. net Framework. With high probability many programs on your computer are developed on this platform. How you how end user, high-speed performance of the platform can concern?

Therefore the first - needs to be measured response time in the specific application. If we want to compare two applications on response time, then they at the same time should solve one business challenge. For example, support of organizational document flow according to the Russian standards.

At what loading to measure?

The following important aspect – to the user response time of a system, of course, is important when opening the document, but also other transactions in the application – search time and an output of results, time of preserving of the document, time of its delivery in an approval chain to the following addressee, etc. are not less important for it. Daily work of the user is modelled in the form of scenarios – the sequences of transactions. For example, the scenario of the head – received the document for consideration, read it, issued necessary instructions.

Here an example of actions of the user in the real project and time of execution of transactions measured in stress testing (in seconds). Realities in the scenario of stress testing include hundreds of transactions.

And response time on each transaction matters. Let's assume, we measured time of each such transaction. All? It is already possible to estimate on compliance to the standard? Compare one program to another? No, still it is impossible. Because the user at us not one, is a lot of them, they different and all of them work in a system at the same time.

In more detail about formation of scenarios of stress testing it is possible to look in article "at High-speed performance of EDMS: we sort assessment technique"

"And users?"

We measure high-speed performance in a multi-user system. If at first to measure response time according to scenarios of one user, and then to put one more nearby and to ask them to execute at the same time the same scenario, the result will be practically same, as in case of one user. However in process of growth of number of at the same time working users response time will begin to grow. At first slowly, and then quicker and quicker.

Here so the diagram of growth of runtime of transactions depending on the number of simultaneous users in an electronic document management system can look.

Figure 1. Curve degradations in overload tests (example)

Curves on the diagram are called degradation curves, they show how response time in process of growth of number of users worsens.

Now it is clear that for assessment of high-speed performance of a system it is necessary to measure it in standard scenarios at different loading, to construct curve degradations, to find the number of simultaneous users on a horizontal axis and, having drawn a vertical straight line up, in crosspoints with curves to see the predicted response time of a system at your loading.

If you have curve degradations of two systems from which you want to select the best (selecting on functionality, but controlling compliance of high-speed performance to the standard), then you can make this choice "other things being equal". To trust data on results of stress testing, you need to have access both to a measurement technique, and to specific results

In practice everything is much more difficult – high-speed performance depends not only on the number of users, but also on volume and structure of information which is stored in a system. There are also other nuances, for example, about automation of stress testing. For persons interested to go deep into a subject, we can recommend our article on Habré'Practice of automation of measurement of indicators of high-speed performance of EDMS'

Good practice of the publication of results

Responsible developers of platforms provide these documents with a technique and results of measurements or publicly via the website, or on demand. It is important that you made the choice, being guided by reliable data, but not by advertizing slogans.

Here an example of information about high-speed performance of one recently appeared Russian EDMS platform: "Opening of difficult cards and transitions between stages of processes less than 1 sec." (a month ago on the same place 0.1 seconds were written).

How less? At what loading? What is a difficult card? What scenario was measured? Whether the measurement technique is available? Whether the array of results of measurements is available? Anything there is no it.

Of course, it is only open information from the website, but it is unlikely that behind it there is a serious work on achievement of balance between functionality and high-speed performance.

Optimization of high-speed performance at stabilization of the new version of the platform

Of course, work on reduction of losses of high-speed performance when adding functionality begins developers even in design process of the new version. Also it begins with testing of high-speed performance of the leaving version and the publication of its results.

How does fight for high-speed performance without decrease in functionality happen? We can tell about it on an example EDMS/ ECM- platforms Docsvision.

The new version of Docsvision 5 was the biggest functional jump for all history of a product - it concerned, mainly, reorganization of a business model of the platform with the purpose to lower time and costs for development and modification of solutions (the second way listed at the beginning of article). Results of functional development were good, but high-speed performance it was insufficient. It was possible to remove some of innovations, and it was possible to begin deep optimization. We selected the second way, understanding that the functionality is necessary for the customer, and high-speed performance should be acceptable (see above). By no means not on the contrary.

At end of each cycle of optimization we published results of testing. As in reality the result is a one hundred indicators, we on the basis of this array calculate a certain conditional index by which it is possible to estimate as far as high-speed performance in the next updating improved. All indexes were published and published.

Figure 2. Decrease in response time of a system according to versions. The latest version of Docsvision 5.4 left in January, 2015, and in January, 2017 we summed up the results of optimization.

Stress testing was made on the Management of Documents built-in application implementing standard scenarios of work with documents and tasks and which is widely used at many customers.

Today we can note with confidence that high-speed performance of version 5.4 exceeded the best indicators of the previous version of Docsvision 4.5 and as we for years published both techniques, and results of testing – nobody will tell that it is groundless statements

At the same time process of optimization is not finished further to develop functionality of the platform, we need to provide a stock on high-speed performance.

Communication of high-speed performance and scalability

On the diagram with curves of degradation (figure 1) it is visible that in process of growth of number of simultaneous users response time of a system grows, will not overstep the bounds of acceptable yet. It is important that the number of your users appeared more to the left of critical point. And, as a rule, increase in computing power does not save the situation – in practice when power ceases to be enough, a system not just brakes – it stops, hangs. There is not enough memory in line input-output, the organization of session connections, etc.

Because of growth of a maturity of the large Russian organizations and enterprises there is an increase in demand for the solutions ECM. ECM is uniform storage of all documents of the organization irrespective of in what systems (EDMS ERP CRM etc.) they were created.

In the recent past it seemed to most of customers normal when documents of office-work are stored in one system, accounting supporting documents - in another, personnel - in the third, etc. Now demand for consolidation of all applied storages and for centralization of document storage of different branches and subsidiaries grows. It requires essentially other scalability

If at head office even of very large Russian enterprises normal no more than 10 thousand simultaneous users of EDMS, then the number of simultaneous users of ECM in all branches and subsidiaries reaches 100 thousand and more. A widespread trend is centralization of management systems in the distributed organizations, before such implementations there were units. Some Russian ECM implementations were implemented on the western software platforms (Documentum Lotus Domino FileNet OpenText Alfresco). The request for scalability in addition amplified in the conditions of import substitution and depreciation of ruble, the replacement adequate on scalability and high-speed performance is necessary today.

It creates a window of opportunities for developers of the ECM platform. For example, the DocsVision company in 2015 began the project of creation of essentially new architecture of the platform which had to provide in the centralized solution when preserving functionality and high-speed performance scalability to 100 thousand simultaneous users with the corresponding flows of new documents and the saved-up volume of storage (terabytes of metadata without the stored files). During 2016 this task was solved by creation of new edition of the Docsvision 5 ECM platform, and transfer of the system of documentary management on it[1]Priority is a development of Digital Design company.

During overload tests a system showed normative high-speed performance at loading in 100,000 at the same time working users within the centralized installation. In more detail about it it is possible to read here.

Outputs

For the organizations and the enterprises selecting the platform of implementation of EDMS we recommend:

• Formulate own requirements to standards of high-speed performance. Being guided, for example, according to programs of the corporate standard of the workstation to which your users got used. It is the best of all to have requirements to standard transactions of users, for example, the first opening of the client software program, opening of a card of the document, document retrieval, etc.
  

• Request results of stress testing of the current version and a technique of their measurement from vendors.
  

• Estimate "freshness" of the version of a product. Recently released versions, as a rule, undergo optimization, and high-speed performance can be improved.
  

• Begin after all with the choice of systems in the short list on functionality, but not on high-speed performance.
  

• Save the economic horizons of estimates – the project cost, ownership cost (taking into account need of permanent modifications) and payback.
  

Successful projects!

Author: Sergey Putsin, product area director of DocsVision, June, 2017

¹ In more detail about response time regulations see in the excellent book by Stephen Seov "We project time. Psychology of perception of time in the software". Transfer not brilliant, for persons interested – the original "Designing and Engineering Time: The Psychology of Time Perception in SoftwareApr 29, 2008 by Steven C. Seow Ph.D».

² The DocsVision company in 2010 of the first among the Russian EDMS producers published a technique and results of measurement of high-speed performance of Docsvision 4.5 on the website of CNews. The testing methodology of the new version of Docsvision 5 is published and is constantly updated on the system website. On demand it is possible to receive detailed results of testing.

173