Software Product Testing

Even if you are so tolerant that you can restart the program 18 times within half an hour after a failure and only then throw the monitor exactly into the window, you will agree that working with this program would be more comfortable if it did not "fall."

How to make it so that cases of falling, hanging, non-fulfillment of the necessary actions of the program developed by you became very rare?

There is no exact answer to this question. But for centuries, the wisest scientists have been thinking about this topic for years and, nevertheless, were able to find a means that, if not eliminate all the mistakes of the program, then at least creates the illusion of activities to eliminate them.

And this tool is called SOFTWARE TESTING.

According to wise people, Testing is one of the most established ways to ensure the quality of software development and is included in the set of effective tools of the modern software product quality assurance system [1].

The quality of the software product is characterized by a set of properties that determine how good the product is in terms of stakeholders, such as product customer, sponsor, end user, product developers and testers, support engineers, marketing, training and sales staff. Each of the participants can have a different idea of ​ ​ the product and how good or bad it is, that is, how high the quality of the product is. Thus, setting the task of product quality assurance results in the task of determining the stakeholders, their quality criteria and then finding the optimal solution that meets these criteria.

When and who?

According to experienced developers, testing of a software product should be carried out directly from the very beginning of its creation. But at the same time, the experienced developers themselves should not take part in testing, since this is not a royal matter. Specially trained employees called testers should test the software product, because even the most experienced developer will not be able to see his mistake, even using the latest optical devices.

However, all developers agree that testing a software product in terms of classification by purpose should be divided into two classes:

• Functional testing
• Non-functional testing

Functional testing

Functional testing means checking the compliance of the software product with the functional requirements specified in the terms of reference for the creation of this product. To put it simply, functional testing checks whether the software product performs all the functions that it should.

So, you decided to conduct functional testing. You look into the terms of reference, read the functional requirements and understand that at least they are not located in the order in which testing can be carried out. You will be surprised that for a long time others have already noticed this discrepancy and figured out how to overcome it.

For functional testing, the personnel of the Technical Control Department shall develop a document and procedure for testing the functionality of the Appendix. The PMI document contains a list of test cases with a detailed description of the steps. Each step of the test scenario is characterized by the actions of the user (test specialist) and the expected results - the program's response to these actions. The test program and methodology must simulate the operation of the software product in real mode. This means that the test scenario should be built on the basis of an analysis of the operations that future users of the system will perform, and not be an artificially composed sequence of manipulations understandable only to the developer. [2]

Typically, functional testing is performed at two levels:

• Component (unit) testing. Testing of individual components of the software product, focused on their specifics, purpose and functional features.
• Integration testing. This type of testing is carried out after component testing and is aimed at identifying defects in the interaction of various subsystems at the level of control flows and data exchange.

Non-functional testing

Non-functional testing assesses the quality of a software product, such as ergonomics or performance.

I think the importance of this type of testing is understandable and does not require justification. After all, everyone understands that if, for example, the system performance is not sufficient, then users will have to wait half a day for a response to their actions, which can lead to their mass hibernation.

As the name suggests, during non-functional testing, the compliance of the software product with non-functional requirements from the terms of reference for its creation is checked. And, as in the case of functional testing, a test program and methodology is developed for non-functional testing.

Embedded software Testing and Standards Compliance in the Agile Era

Meeting industry standards is not something you can neglect or do later; is an integral part of the inline development process software (). ON For some industries - such as avionics, automotive, and - health care strict adherence to quality standards in the development of complex and fail-safe embedded systems becomes a vital prerequisite for launching the product on the market. Traditionally, testing plays an important role in the development of embedded systems for standard-regulated industries. However, in recent years, established testing practices and processes, their place and role in such projects have changed significantly. It's dramatically changed all the rules of the game, and when the rules of the game change, you have to change with them to win.

With new, state-of-the-art technologies constantly evolving, companies need to quickly offer the market reliable, secure, easy to use, and compatible with other systems - just to avoid getting lost in a rapidly changing technology world. In such a situation, the traditional cascade model, where the software development process is strictly consistent and testing is carried out at the very end of it, is a thing of the past. DevOps and Agile methods are gaining great popularity because they allow engineers to perform tasks that used to follow each other at the same time.

A study supported Auriga by the independent research company LTM Research shows that this evolution of the role of testing in the software development cycle is of great importance. With constant time constraints, manufacturers are still unable to sacrifice the quality, reliability and safety of their product. For example, unmanned vehicles widely discussed today are a source of increased danger, which means they require strict compliance with standards. You cannot do without testing the embedded software, since almost all solutions in the field IoT and Connectivity are based on built-in technologies.

All industries strive for innovation, rapid development and parallelization of processes, and this makes testing embedded software even more important. Healthcare, where standards are traditionally very high, is distinguished by a huge demand for complex and ultra-accurate algorithms - such as, for example, an automatic heart rhythm recognition algorithm for an innovative defibrillator, which Auriga engineers are now working on. New smart hospital systems, smart medical equipment and wearable devices that appear almost every day must be safe and reliable.

Speaking of security, one cannot fail to mention the sphere of finance and the growing interest in biometrics. Fingerprint and retinal scans, voice and face recognition are what will be used to identify users instead of the usual passwords we are so used to. But before allowing embedded software to scan your retina, manufacturers need to make sure it meets all standards and is resistant to cyber threats, which today are getting^[1].

Performance testing

During the performance test stage, first of all, load testing is carried out, the purpose of which is to check whether the system will adequately respond to external influences in a mode close to the real operation mode.

In addition to load testing, tests are carried out in conditions of minimal hardware and maximum load - stress testing, as well as tests in conditions of maximum volumes of processed information - volumetric testing.

Another type of testing is distinguished: stability and reliability testing, which includes not only a long-term test of the software product under normal conditions, but also the ability to return to normal operation after short periods of stress loads.

Documentation for testing

As mentioned above, testing is carried out in accordance with the test program and procedure, which is developed in accordance with GOST 34.603-92.

For testing, a test case is developed that should contain enough data to check all modes of operation of the software product. Typically, a test case is created jointly by the customer and the executor based on real data.

To conduct all types of performance tests, the so-called data generator is most often created, which allows you to automatically create enough data to achieve an objective result when evaluating performance.

During testing, a test protocol is drawn up, which contains information on the passage of all stages and steps of testing and comments received during tests.

If the test result is negative, troubleshooting and retesting are performed.

Research testing

Exploratory testing (ad hoc testing - subspecies of functional testing. It is used in fast-growing projects with flexible development methods, where there is no clear documentation and requirements. Research testing - aerobatics in software testing. Quality testing is available to specialists with the highest qualifications and almost completely depends on the performer, his experience, knowledge (both in the subject area and in testing methods), the ability to quickly penetrate the essence.

Load testing

Load testing - the process of analyzing the performance of the tested system under the influence of loads. The purpose of load testing is to determine the application's ability to handle external loads. Usually tests are carried out in several stages.

1. Generating Test Scripts

For efficient analysis, scenarios must be closest to actual use cases. It is important to understand that exceptions are always possible, and even the most detailed testing plan may not cover an individual case.

2. Test Configuration Development

With testing scenarios, it is important to distribute the order in which the load increases. Performance evaluation criteria (response rate, query processing time, etc.) must be highlighted for successful analysis.

3. Test Run

When conducting tests, it is important to monitor the execution of scenarios and the response of the tested system in a timely manner. High-load emulation requires a serious hardware and software infrastructure. In some cases, mathematical modeling methods are used to reduce the cost of work. The data obtained at low loads are taken as a basis and approximated. The higher the level of the simulated load, the lower the accuracy of the estimate. However, such a method significantly reduces costs.

Test automation

The main feature of automated testing is the ability to quickly conduct regression tests. The main advantages of automation (according to Worksoft) are increased personnel efficiency, earlier detection of defects and higher quality of business processes. These advantages are offset by a significant drawback: the high cost is due to the high price of introducing and supporting testing automation, about 50% of companies still use mainly manual testing.

Usability testing

Any application is created to be used. Usability is an important qualitative indicator of the program. The IT industry knows many examples when projects took off after a successful usability fix. The wider the audience, the more important the usability factor. Usability testing involves detailed analysis of user behavior. To assess ergonomics, it is important to have data not only on the speed of the business task, but also on the user's emotions, facial expressions, voice timbre.

Configuration testing

Configuration testing gives confidence that the application will work on different platforms, which means that the maximum number of users. For WEB applications, cross-browser testing is usually chosen. For Windows applications - testing on various operating systems and bits (x86, x64). An important component of configuration testing is the test infrastructure: to conduct tests, you need to constantly maintain a fleet of test machines. Their number varies from 5 to several dozen.

Integration testing

If your project has more than one component, it needs integration testing. With a complex application architecture, a prerequisite for quality assurance is to check for interaction between parts of the program. Testing is achieved by developing and conducting "end-to-end" cases. Integration testing is performed after component testing. Therefore, it is very important to take into account the experience of component testing, while observing the business orientation of test cases.

Stress testing

Any system has a limit of normal functioning. When the limit is exceeded, the system falls into a state of stress and significantly changes its behavior. Stress testing checks the operation of the application in conditions of exceeding the limits of normal operation. This is especially important for "critical" programs: banking software, programs of the aviation industry, medicine. Stress testing is carried out not only at the stage of software development, but also throughout the entire cycle of operation in order to obtain and process data on system behavior over a long period of time.

Software Testing Market Research

2022: How tools and approaches to software testing in large Russian business are changing. TAdviser Overview

Software testing services in Russia are in great demand, and competition in this segment is high. Having sifted several times during the pandemic, in 2021 this market showed dynamic growth. And since February 2022, demand has fueled the increased need for organizations to switch from Western IT solutions to alternative ones, because this also implies a large layer of work related to software testing. TAdviser spoke with representatives of large businesses from various industries with deep competencies in the field of testing, as well as with representatives of players providing such services, and prepared an overview of the state of this area in Russia. The partners of the material were LANIT Expertise (part of the LANIT group) and T1 Group. Read more here.

2020: Only 13% of Russian companies attract QA specialists at all stages of software development

On September 3, 2020, Performance Lab released the annual RQR 2020 (Russia Quality Report), reflecting the state of the market for IT product testing and quality assurance services in 2020. RQR 2020 describes trends and changes in the field of testing and quality assurance of IT products in Russia in 2020 compared to previous years based on feedback from respondents from different areas.

In particular, surveys of respondents showed that requests for testing IT products are growing every year in the domestic market. Business understands how important it is to use testing methods for the release of a high-quality product, but the widespread use, in general, testing of IT products is not yet observed in all areas, due to ignorance of processes, the high cost of using testing tools and other reasons. Although there are many industries where testing is in demand today and has a fairly high demand.

As you know, creating high-quality software solutions is impossible without testing them. It is necessary to involve the QA team (Quality Assurance) in the work on the product in the early stages. This approach also helps to identify critical defects in a timely manner and subsequently launch a high-quality IT solution on the market.

At the time of the survey, only 13% of Russian organizations and companies involve testing specialists at all stages of the software life cycle. Unfortunately, the domestic market has not yet reached maturity in this regard. However, the indicator, although slowly, is steadily increasing: compared to the results of the previous survey, its growth was 5%.

Compared to the 2017-2018 survey, organizations began to engage the testing department more often in the early stages of the software life cycle. QA specialists are connected to work at the stages of operation and support (in 50% of cases), description of requirements (50%), design (46%).

In addition to the testing phase, in which QAs take part in 91% of cases, these employees are also often brought in during implementation (61%) and product development (57%).

The main goal of the business in 2020 was to reduce the time for launching products on the market - this trend is observed in all industries, except for the sphere of system integration. In previous years, the reduction in time to market was lower than the IT product quality metric. Performance Lab specialists believe that the speed of product launch on the market will remain a priority criterion for evaluating testing and one of the main goals of QA throughout 2020 and 2021.

In comparison with 2019, given the situation of the crisis in many industries, the business is acutely aware and resolutely acts in the direction of market entry leadership: whoever entered the market first with a high-quality product is getting more demand in the market.

A fairly constant indicator remains a large number of surveyed companies, which, first of all, involve their QA department in the task of improving the quality of IT products. This was stated by 80% of the study participants. 69% of respondents choose to increase user satisfaction as the goal of QA specialists.

The role of automated testing in IT processes has become multifaceted in recent years. Service customers strive to gain full control over the quality of the product being developed and reduce testing time by eliminating the human factor.

According to Performance Lab experts, automation allows you to provide maximum test coverage and efficiency in the use of test cases, leads to cost optimization, increased labor productivity and reduced time to market. Thus, automated testing becomes the best way to achieve a number of QA goals. The majority of survey participants (82%) use autotest products to check the quality of the developed products. The same number of respondents are engaged in the development of competencies in automated testing among their specialists in the field of functional testing.

The majority of respondents (71%) understand the importance of the mobile application testing procedure and conduct it.

Respondents began to use Agile more: the number of study participants who complain about the lack of experience in applying flexible methodologies has decreased over the past two years. Flexible methodologies (Agile) continue to win the sympathy of Russian companies and organizations. Four years ago, this approach was used by only 43% of the surveyed market players. In 2020, their number increased to 80%. Another 17% of respondents pointed to a lack of understanding of Agile's approaches to testing.

Flexible development methodologies (Agile) are used by the majority (91%) of respondents from banking organizations, and in telecom such methodologies are used by slightly fewer respondents - 80%. As a survey of trading companies showed, Agile in retail is gaining momentum from year to year - most of such organizations 60% use this approach. Here over the past two years, this figure has grown by 7%.

According to researchers, Russian companies have some inertia in the issue of changing testing tools, rather this is a gradual process. Respondents noted that most of them use at least three different tools at the same time. The difficulties of switching to new products are often associated for someone with the high cost of switching, ignorance of the QA products market. According to the results of the report, telecom and eCommerce industries look the most flexible and are ready to consider domestic products in comparison, for example, with the banking sector. Only 14% of respondents limit the use of foreign tools for organizing testing within the framework of the import substitution program. Despite the fact that most of the players in the Russian market still prefer foreign testing tools, the number of companies and organizations that restrict their use increased by 8% compared to the previous report.

During the data collection period for the 2020 report, the threat of the spread of coronavirus Covid-19 has not yet been so acute. However, it is already clear that the pandemic has made its changes in life and in business. Budgets allocated for the sphere, in IT general, and for testing, in particular, are reduced. At the same time, service providers and QA specialists will switch to. remote work format It is also clear that the current crisis will not cause serious damage to industries related, first of all, to. digital content

This is primarily the telecommunications industry. The business partially left offices, and users began to use the Internet much more often and communicate through mobile and other online tools. Innovative technology solutions in the telecommunications environment take root more easily than other areas, according to the report. For example, Big Data products are tested by the majority of surveyed representatives of this industry. Telecommunications companies more often use QA services - during periods of design, development and testing, system integrators - at the stages of testing and operation.

Retail for the most part held its position, it turned out to be easier for those who work through online sales channels. There was also a wave of crisis on eCommerce, but not as much as in other areas.

All banking organizations that participated in the survey test their IT products. Of these banking companies, 81% have relevant specialists, 63% contain a specialized department and another 63% use outsourcing services in this area.

In general, according to RQR reports, 67% of respondents have their own software testing department. In the state, 65% of the companies and organizations surveyed have relevant specialists. Finally, 39% of market players who participated in the study use outsourcing services for testing on a regular basis.

Public sector representatives more often noted that they have their own testing department or use outsourcing services as a last resort. System integrators prefer to keep the necessary employees on projects. Outsourcing services are more often used in the financial and public sectors. The popularity of outsourcing in the fields of telecommunications, e-commerce and system integration is lower: only a quarter of the surveyed players in these markets use this format of work for software testing.

In addition, more than a third of organizations that took part in the survey (38%) have a specialist in their staff responsible for digital transformation. For example, such employees help 58% of respondents working in the banking sector develop their business. A similar approach is practiced by 50% of our interlocutors from the state and telecommunications sectors. At the same time, none of the surveyed system integrators resort to this practice.

The number of Russian companies and organizations using the capabilities of artificial intelligence is growing. So, in 2017-2018. these technologies were not used by 77% of respondents. According to RQR 2020, their share has decreased to 68%. Works with AI 32% of survey participants.

Artificial intelligence technologies are also in the field of quality assurance (QA), with just under half (44%) of market players surveyed saying they already use AI and analytics to optimize QA. The most popular reason for using artificial intelligence is automation: 62% of respondents answered this way. These technologies are also actively used for the purposes of predictive analytics (in practice, 31% of the respondent).

In banks, 60% of cases, artificial intelligence is used for automation, 40% for predictive analytics.

Machine learning is one of the fields of artificial intelligence, it uses methods to "train" analytical systems by solving a large number of similar problems. Only 19% of survey participants said they had experience testing machine learning products. This technology was quickly adopted in the banking sector: 33% of industry representatives surveyed announced its use.

As for blockchain technology, only 3% of survey participants use its capabilities.

2018: TAdviser Study: IT System Testing Services Outsourcing Market in Russia

This study TAdviser is devoted to a rather narrow, from the point of view of financial assessment, area in the market segment outsourcing IT services -. outsourcing software testing services Despite the relatively modest size of this market segment compared to other types of IT services, in the past 2-3 years it has attracted the attention of analysts and specialized media due to several large contests with budgets of hundreds of millions rubles conducted by key customers (large, banks state structures federal level, retail chains). Read more about the results of the study. here

See also

• Application Development Tools - Product and Project Catalog
• Programmer
• Agile software development
• DevOps Methodology
• Programmer's Day
• Programmer cemeteries. Why not all IT people want to work for corporations
• Site developers (contacts of programmers)
• HackerRank Programmer Rating
• Cost of hour of programmer's work
• Machine Learning
• Artificial intelligence
• Machine intelligence
• Cognitive computing
• Data Science
• Neuronets

• [1] ]Testing - A Way to Ensure Quality
• [2] ]Testing during the product creation phase
• [3] ]The ideal process of creating information systems. Testing. Why is it needed?

Notes