Computer vision: technology, market, prospects

Importance of research

Computer vision and artificial intelligence are one of the most sought-after areas in the modern IT world, "said Vladimir Ufnarovsky, co-owner of Computer Vision Systems. - Very little is known about the achievements of Russia in these areas, but at the same time a huge number of developments are based, in particular, on Russian research. In addition, there are many different "myths" around KZ and AI - some believe that computers can already easily "see" and "realize" something, surpassing a person, while others, on the contrary, do not understand how advanced technology has already been. We (the team "LANIT-TERKOM," and then "Computer Vision Systems") have been engaged in PC since 2006 and are very glad that at last comprehensive material has appeared describing the current state of affairs

Basics of Computer Vision

Computer Vision (CV), including machine vision (MV), is the automatic fixation and processing of images, both still and moving objects using computer tools^[1]. In Russia, the term "technical vision" is also used.

The first attempts to make the computer "see" date back to the early 60s of the 20th century. However only in recent years in connection with increase in computing power and speed processors, memory sizes, increase in resolution and other parameters of cameras, development of bandwidth of channels communications and also with the advent of such technologies, machine as well as deep training (Machine/Deep Learning), artificial intelligence AI (Artificial Intelligence) of CV/MV technology began to find more and more applications in various branches of the industry and everyday life of people.

In recent years, CV has become actively used in industry, including in such industries as automotive, food industry, pharmaceuticals, the production of microelectronic products and many others.

For example, in the automotive industry, CV systems are used to read component markings when assembled on a conveyor. Computer vision is also used to improve quality, in particular, for inspection, calibration, verification of dimensions, gaps, distances, as well as for leveling parts on car assembly lines.

In the production of food products, CV systems can check whether all ingredients are indicated on the packaging of the product, especially those that may contain allergic substances.

Pharmaceutics implies a high responsibility for ensuring safety, so it is necessary to reliably monitor all components of the composition and the quality of the finished products.

When manufacturing chips and electronic components, CVs are used in clean rooms to control the placement of silicon plates, marking and position of the chip of integrated circuits and other elements.

Today, computer vision is widely used for many components of the digital economy:

• Smart City,
• Intelligent transport Transportation System
• Autonomous cars (Driverless Car) and ADAS driver assistance systems (Advanced driver-assistance systems),
• Unmanned aerial vehicles (including drones),
• High-tech agriculture (Smart Agriculture),
• Electronic Medicine (eHealth)
• Military applications systems,
• Additive manufacturing (3D-printing)

and in many others. Moreover, more and more new areas and scenarios for the use of CV are constantly appearing.

Today's development of CV systems is still far from realizing all its capabilities. However, this industry is rapidly developing and its range of applications is rapidly expanding.

"Computer vision" ("machine vision," technical vision ") is often confused with video analytics. However, these concepts are unequal. We can say that video analytics is an integral part of computer vision in the part of image analysis.

Computer Vision is a technology (as well as a field of research) to automate the understanding of what we see in the world around us.

Video Content Analysis (VCA) is a private computer vision application that extracts information and knowledge from video content, that is, answers to questions:

• Who: recognition and identification of people;
• What: objects, actions, events, behavior, relationships;
• Where: geolocation, spatial (3D) and planar (2D) location;
• When: marking date and time, season.

Three main types of video analytics applications:

• Retrospective: what has already happened, i.e. managing video archives, searching, sorting, obtaining legal evidence;
• The present point: what is happening now, i.e. monitoring the situation, receiving warnings in real time, coding, compression of the video stream;
• Looking to the future: what can or will most likely happen, i.e. predictions based on the events of the past and present, forecasting events or activity, detecting emerging anomalies.

Read more about the market and video analytics technologies in a separate review by TAdviser.

CV Tasks

The tasks of the CV are mainly to obtain useful information (insight) from photo or video images. The most common CV tasks may be:

1. The tasks of calibrating cameras and optical systems, both consisting of one camera and a set of cameras.
2. Image Motion Definition Tasks.
3. The tasks of identifying obstacles in the course of movement.
   1. In a 3D cloud by stereo camera or camera set
   2. One camera by motion
4. Onstage object recognition tasks.
5. Spatial Scene Reconstruction Tasks.
6. Image localization tasks in a pre-known scene.
7. Image Set Difference Analysis Tasks.

Technologies

In general, CV systems consist of a photographic or video camera, as well as a computer on which image processing and analysis programs operate.

If the image processing software is located directly in the camera, such a camera is called a "smart camera." The software can also work on a remote computer or computers, or run in the cloud according to the SaaS model (Software as a Service).

Computer vision systems include the following main components:

• object illumination (not always required) and optics (lenses and lenses)
• sensor matrix for image projection
• image processing system obtained from a matrix.

If necessary, for example, indoors, when light can be monitored, the part of the object to be inspected can be highlighted so that the desired characteristics of the object are visible to the camera.

The optical system projects the resulting image in the form of a visible or invisible human eye spectrum onto the sensor matrix. The camera sensor array converts the image into a digital image, which is then sent to the processor for analysis.

In most cases, CV systems are designed to operate in natural light. In addition, CV systems can operate in ranges invisible to the human eye.

For operation in conditions of insufficient lighting, illuminated cameras can be used, in which an annular light source provides bright uniform illumination of the object when it is necessary to highlight the texture of the material, small parts, etc. Also, lighting helps to get rid of highlights, illumination of the object, is used in difficult conditions, for example, in fog.

Such an integrated source does not shade and provides even lighting of matte surfaces. The sensor array is located in the camera and is designed to capture the image of an appropriately illuminated object. Usually touch matrixes are under construction on the basis of semiconductor devices with charging communication of the CCD, CCD (charge coupled device), or the complementary metal-oxide-semiconductor technology of KMOP, or CMOS (complementary metal oxide semiconductor) can be used.

The image is a set of elements - pixels, the color of which depends on the illumination. Pixel density (sensor matrix resolution) is very important for the correct operation of the computer vision application. The higher the resolution, the more detail in the image, the more accurate the measurements will be. The required pixel density depends on the size of the object, the working distance of the camera, and other parameters.

Types of CV systems and image processing methods

There are three main types of CV^[2]

• one-dimensional (1D),
• 2D (2D)
• volume (3D) CV systems.

Separately, there are panoramic multi-chamber systems and fisheye systems, which are usually classified as a special type, and sometimes, depending on the number of cameras, their design and location, one of the above types.

Stereovision

Stereo viewing is one of the methods of extracting information about the depth of the scene using images from two cameras (stereo pairs). The method is based on the principle of human vision, when the human brain receives information about the volume of a picture from two eyes. In the same way, the difference in the location of pixels in the image from two cameras gives information about the depth.

By adjusting the distance between the cameras of the stereo pair (baseline), you can adjust the desired scene recognition depth.

Spherical and panoramic systems

Spherical (panoramic) fisheye systems are used to emulate panoramic PTZ cameras for video surveillance and to integrate broadcast webcams into 2D and 3D applications of geographic information systems (GIS), such as Google Earth and Google MapsAbrams^[3]

The panoramic fisheye-systems working with applications of processing of images of cloud providers are applied, for example, in the systems of the help to the driver (ADAS), self-driving cars, during the monitoring of big spaces and calculation of quantity ^[4]

Multi-chamber systems (arrays)

Camera arrays (networks) are used to track the movement of individual people indoors or in places with limited visibility (warehouses in seaports, factory territories, etc.), as well as to control traffic in intelligent transport systems (ITS).

Systems of a small number (2-6) of cameras are used for areas such as:

• Automation of production,
• Video surveillance with UAV,
• 3D movies,
• Interactive Games AR/,VR
• Facial recognition, movement, identification, etc.

For example, the use of a multi-chamber system of five cameras on the conveyor in mass production greatly facilitates quality control of^[5].

Computer Vision Software Libraries

• OpenCV (Open Source Computer Vision Library) - a library of computer vision algorithms, image processing and numerical algorithms for general purposes. Implemented in C/C + +, also developed for Python, Java, Ruby, Matlab, Lua and other languages.
• PCL (Point Cloud Library) is a large-scale open project for processing 2D/3D images and point clouds. The PCL platform contains many algorithms, including filtering, performance assessment, surface reconstruction, registration, model matching, and segmentation.
• ROS (Robot Operating System) is a software development platform for robots. It is a set of tools, libraries and agreements that simplify the development of complex and efficient programs for managing many types of robots.
• MATLAB is a high-level language and interactive environment for programming, numerical calculations and visualization of results. Using MATLAB, you can analyze data, develop algorithms, create models and applications.
• CUDA (Compute Unified Device Architecture) is a parallel software and hardware architecture that allows you to significantly increase computing performance thanks to the use of Nvidia graphics processors.

Image processing systems and methods

In simple CV processing systems, it is usually necessary to obtain quantitative and qualitative information from visual data (images): parameters such as size, color, number, direction and nature of movement, as well as contrast transitions in the vicinity of the image pixel, from which characteristic features are obtained, CH (the so-called "fichi," from English Features). Based on these, an image is analyzed to extract useful information.

CV image processing systems use techniques such as Machine Learning, Deep Learning, and Neural Networks. These methods mimic the recognition and analysis process that takes place in the human brain.

Main approaches to solving CV problems

The main approaches to solving CV problems:

• Contour Analysis
• Template matching
• Out-of-pattern search, feature detection, description matching
• Data Fusion

Computer vision is not limited to these basic methods, for example, so-called genetic algorithms used, in particular, for face recognition can be distinguished.

Contour Analysis

An object boundary is a curve that corresponds to the boundary of an object in an image. This method does not analyze the full image of the object, but only its outline, which significantly reduces the complexity of algorithms and calculations during processing. Boundary analysis method constraints:

• with the same brightness with the background, the object may not have a clear border on the image or it may be "noisy" with interference, which leads to the impossibility of highlighting the contour;
• overlapping or grouping objects causes the boundary to not be highlighted correctly and does not match the boundary of the object.
• poor resistance to interference, which leads to the fact that any violation of the integrity of the loop or poor visibility of the object leads either to the impossibility of detection or to false positives.

Template matching

The most common method of recognizing objects in CV is to search for compliance with template matching ^[6] to determine if there is a predetermined object in the image, and, if there is, where it is in the image. Applications of the method: recognition of vehicles, laying routes for mobile robots, production and applications in medicine, etc. Main types of search by template:

• Simple compliance

Simple matching is one of the main methods of finding the desired object in an image when searching by template. The method consists in step-by-step scanning with the template of the original image, at each step of which the degree of correspondence of the image area to the template is measured or calculated. At the end of the scan, the area most corresponding to the pattern is highlighted in the image.

• Feature-based matching

The feature matching method, CH, is applicable when both the image and template contain more CH and checkpoint matches ^[7] than the whole image. In this case, the CH can include points, curves, or surface models that are checked against the pattern. The purpose of such a test is to find paired connections between a target (the so-called "reference") and a part of the image using spatial relations or HF.

• Area-based matching

Area-based matching methods, also called correlation methods, are based on a combined feature detection and template matching algorithm. Such a method works well if the patterns do not have noticeable common RF with the image, since the comparison occurs at the pixel level. Correspondences are measured by the intensity of the template and image ^[8]

In some cases, it is not possible to find a direct match between the template and the image (see figure below). Therefore, eigenvalue and eigenspace are used when matching. These values ​ ​ contain information necessary to compare images under different conditions of illumination, contrast of contours or coincidence in the position of objects.

• Image Correlation Matching

This method measures similarity metric between the original image and the pattern. Unlike the simple fit method, the original image and pattern may have different image intensities or noise levels. In this case, the comparison is made using a similarity metric based on correlations between the template and the original.

Computer vision significantly expands the ability to control product quality (actually moving control to a new level) directly in the production process, and not after the manufacture of a part or product^[9].

Automatic visual inspection (detection of defects) using computer vision today significantly exceeds manual inspection methods in terms of accuracy, speed, ease of execution and cost.

• Neural networks

The term "neural networks" was very popular in the late 1980s and early 1990s. Neural networks consist of layers, the so-called "neurons," which are computational nodes that mimic the work of neural cells of a living organism. These networks can transmit information in only one direction and can learn from examples (for object classification or regression analysis).

• Deep CV Training

Deep learning can be useful in tasks where the base element (single image pixel, single signal frequency, single word or letter) does not carry a large meaning, however, the combination of such elements has a useful meaning.

Deep learning systems can extract such useful combinations without human intervention (Unsupervised Feature Learning).

Deep neural networks, with more than 1-2 layers, previously seemed either unrealizable or impractical to use. Until 2006, the outer layers of the neural network were incapable of extracting CH (features) of the input images, since the neural network learning algorithms remained imperfect.

The figure shows an example of a machine-trained CV system using the example of the task of segmenting a scene into three types: "horizontal," "vertical" and "sky"^[10].

Pixel data from the original color-corrected image is supplied to the deep learning neural network, where the image is pre-processed and it is recognized to which type of relief each pixel belongs with a certain degree of probability.

An example of handwritten digit recognition using a simple single-level neural network is shown in the figure below^[11].

• Calibration of cameras by template

Camera calibration technologies can be divided into 2 types^[12]:

• Photogrammetric calibration (by template).
• Calibrating by scene.

Calibration of cameras according to a template is usually carried out by observing a calibration object (template), the geometry of which in space is known with greater accuracy. The calibration object usually consists of 2 or 3 planes arranged at different angles to each other. These approaches require expensive calibrators and their careful installation.

Calibration of cameras on stage does not use calibration objects, but is carried out only by the movement of the camera in a static scene. If the images are taken from the same cameras with fixed internal parameters, the correspondence between the three pictures will be enough to obtain both internal and external parameters that will allow reconstructing the volumetric structure of the object.

Computer vision outside templates

The RF image processing sets in computer vision may, for example, be image elements such as dots, edges, lines, or boundaries of objects. Other examples of HF relate to motion in a sequence of images, to shapes represented as curves between regions of an image, or to properties of those regions.

• Object detection and recognition

Object detection is the finding of instances of objects in an image. Object recognition not only establishes the presence of an object in the image, but also determines its location in the^[13] image]. The figure below shows examples of detection (left) and object recognition (right).

Object detection involves comparing two or more images when searching for images of unique objects, for example, architectural structures, sculptures, paintings, etc., detecting on images of classes of objects of different degrees of commonality (cars, animals, furniture, faces of people, etc., as well as their subclasses), categorizing scenes (city, forest, mountains, coast, etc.)^[14].

Applications for detecting objects are also very diverse: sorting images in home digital photo albums, searching for goods from their images in online stores, extracting images in geographic information systems, biometric identity identification, targeted search for images on social networks and much more.

Recognizing this variety of objects and applications requires the use of machine and deep learning techniques.

Some other examples of using the recognition method outside the templates: photogrammetry, obstacle detection, simultaneous localization of objects and map building in an unknown space (SLAM), flaw detection.

• Photogrammetry

Photogrammetry is the process of creating 3D models from several images of a single object, photographed from different angles.

This method has long been used in cartography and geodesy and has become more popular due to the availability due to the increased power of computers. This allowed the use of photogrammetry in other areas:

• creation of geographic information systems (GIS);
• environmental protection (study of glaciers and snow cover, soil bonitisation and erosion processes, observation of vegetation cover changes, study of sea currents);
• design and construction of buildings and structures;
• film industry (combining the game of live actors with computer animation);
• automated construction of spatial models of the object by images;
• computer games (creating three-dimensional models of game objects, creating realistic terrain landscapes, etc.).

• Detection of obstacles

Obstacle detection is used, for example, in ADAS driver assistance systems (Adnvanced Driver Assistance System), in unmanned aircraft control systems, etc.

ADAS algorithms include the following:

• Lane control,
• Detection of objects on the way and on the sides,
• Recognition of road objects,
• Adaptive cruise control,
• Circular overview.

• SLAM

SLAM (Simultane Localization And Mapping) - a method of simultaneously localizing objects and building a map in an unknown space or to update a map in a previously known space with simultaneous control of the current location and the path traveled. It is used in autonomous vehicles to orient them in space.

This method is used for spatial reconstruction (Stereo-SLAM) during the movement of vehicles to create volumetric maps of objects from images from one or more^[15] CV cameras^[16]

• Flaw detection

Non-template CV systems are often used to identify various defects in materials and products.

• Object recognition and localization in a pre-shot scene

In addition to the terms "detection" and "recognition" in computer vision technologies, the terms "classification" and "localization" are also used, as well as "segmentation" of objects^[17].

• Classification of an object - recognition in an image of one category of object, usually the most noticeable. This type of recognition is most often used in smartphones equipped with "artificial intelligence."
• Object localization - the object is not only recognized, but also localized on the original image.
• Object detection - The image can have objects of different classes that are recognized and localized on the original image.
• Segmentation of objects - for each object, not only its class and its location are recognized, but also the boundaries of the object in the image are highlighted.

• Observer localization and measurement control

Localization algorithms allow you to determine the position of the camera relative to the scene (localization) and detect differences in the scene in the historical perspective (the presence of new objects in the scene and a change in the coverage of the scene) at the level of the cloud^[18].

During localization the following tasks are performed:

• Localization in the image sequence: find the position of the new image in the previously captured image sequence;
• Point cloud localization (3D models):

• • finding the position of a new image in an existing point cloud
  • * with existing images, sources for this point cloud
  • * find the position of a new image with a textured cloud of points, with additional data from GPS (data fusion);
  • Find the position of the new point cloud in the existing cloud through the source images

• Detect changes in images and point clouds.

• Color and exposure correction

Color recognition in CV systems for some tasks helps to determine the properties of materials: what is an object made of and in what state is it? For example, on a black and white photo it is impossible to determine which berries are ripe. On a color photo it is possible ^[19]

Color CV can much more accurately determine the color tint, which is often required in various industries, for example, in car repair, in medicine, etc.

Color CV is actively used in the following applications:

• Games;
• Inspection of medicines and medical diagnostics;
• Identification of parts and spare parts;
• Inspection of colored material (fabric, film...) for compliance with the specified color;
• Inspection of labels, stickers and ave.;
• Sorting of spent materials;
• Remote sensor, tracking;
• Biometrics, traffic monitoring;
• Testing of paints and pigments, etc.

Data Fusion

Data Fusion - combine data from different sources with images from CV cameras in order to obtain more accurate and useful information. In CV, you may encounter the following^[20] problems]:

• Different HCs can be isolated from the same image;
• Different instances of the same object type (e.g. "people," "machines") may look very different;
• Different instances of the same class of objects can "behave" differently, at least at times;
• The same object from different viewing points (i.e. from different cameras) may look different;
• Various combinations of all of the above.

Combined analysis of data from the CV system and the sensor complex helps significantly increase the value of information received from the CV system and significantly improve the operation of the application using it. For example, in addition to CV cameras, ADAS systems can be equipped with many different sensors: LIDAR, Radar, odometer, ultrasonic sensors (see figure below).

A comprehensive analysis of data (Fusion Algorithm) from all sensors and additional sensors (Auxiliary Sensor) and the CV (Vision System) system will make it possible to make an unambiguous conclusion: "There is a pedestrian in the course of the car at 11.6 m."

Applications

Today, computer vision is widely used in many sectors of the digital economy, such as Smart City, autonomous cars and driver assistance systems (ADAS), unmanned aerial vehicles, high-tech agriculture, healthcare and many others.

Video surveillance and security

Video surveillance is an important part of physical security. Video surveillance with the participation of a person, for the most part, comes down to long periods of waiting for something unusual on the video monitor. This is a very important job, but very tedious. According to psychologists, the average time to keep a person's attention on one object does not exceed 14 minutes^[21]

Therefore, the so-called intelligent video surveillance systems IVS (intelligent video surveillance) were created based on deep ^[22]., whose task is to recognize unusual events or objects on video surveillance frames (see figure below).

The Image processing platform performs Face detection, Motion detection, Static object detection, Privacy protection, Human tracking, Anomaly detection, and Human pose identification. If any unusual phenomena are detected, warnings are issued, the object is highlighted on the screen by a frame, etc. (Warning, Emphasizing, Retrieving, Counting, и т.д.).

In some cities, such as Las Vegas and Dubai, deep training in video surveillance systems has become practical in Smart City systems. For example, such systems can inform the relevant services when and where it is necessary to collect garbage, maintain street lighting or control traffic lights, for example, switch light from red to green, if there are no cars in the transverse direction^[23].

Machine vision for robots

Industrial robot manipulators usually perform repetitive routine tasks well. However, they are practically helpless when the task changes, for example, when the object of manipulation is of a different size or configuration. Machine vision gives the robot the ability to automatically adapt to changes in size or inaccuracies of objects and their arbitrary location. Thus, the use of machine vision for robots allows you to produce different products, without changing anything in the robotic complex itself and without its complete reprogramming.

Automotive industry

The share of deaths due to car accidents accounts for 2.2% of the total number of deaths in the world. This is about 1.3 million a year, or almost 3,300 people a day, except that between 20 and 50 million people a year are seriously injured by accidents. The cause of such high mortality is most often the "human factor"^[24].

The^[25] systems also help prevent many accidents when the driver does not notice the transport moving in the transverse direction. Such systems are usually built on the basis of radars operating at a high frequency (20 GHz and above). However, they are quite expensive and can be installed in high-end cars as an additional option^[26].

Computer vision can greatly simplify such systems and make them widely available.

Use of computer vision for military purposes

The main applications of CV for military purposes are the following^[27]:

• Video surveillance,
• Autonomous vehicles,
• Means of clearance of minefields,
• Quality control during ammunition production.

Consumer market

Drone with computer vision that recognizes obstacles

DJI has released the latest Phantom 4 drone, which can recognize obstacles with built-in CV and machine learning. He is able to independently choose the flight route to the target specified by the operator^[28]. The GPU drone processor was developed by Movidius.

Movidius announced a collaboration with Google in a project to introduce deep training in smartphones, which allows you to develop images on your smartphone locally, and not send a large amount of graphics data to the cloud. The drone DJI uses exactly this technology.

Medicine and Health Care

The use of computer vision for processing medical images is often used in computer diagnostics to plan personal therapy, medical care, and improve the decision-making of^[29].

Machine learning systems based on computer vision images help the doctor make a diagnosis, since the image may contain small details that the doctor may not notice, but such details can be recognized by the CV system with a high degree of reliability.

In addition, the image can be compared to thousands of other similar images in the medical system database, and the comparison result is used to make a more accurate diagnosis by a medical professional.

3D imaging of cancer by computed tomography

Microsoft has developed a CV InnerEye system that can visually identify and display possible tumors and other abnormal formations on a doctor's monitor from ^[30] computed tomography]. The attending physician can then identify them more accurately. To develop the InnerEye, a deep learning algorithm was applied on millions of scans of computed tomography of different patients.

Despite the fact that there are many breakthroughs and technological advances in healthcare, due to the peculiarities of the work of medicine, it will probably take many more years before CV technologies in healthcare become widespread^[31].

Agriculture

Agricultural output must nearly double to meet the demand for food for 9.7 billion people by 2050, according to UN^[32] agriculture The use of CV technologies together with global positioning systems allows accurate agriculture (precision agriculture)^[33]which can significantly improve agricultural productivity and efficiency.

The use of unmanned aerial vehicles allows you to obtain topographic maps of the area, and the use of image processing technologies allows you to obtain 3D models of sections of the earth's surface with the ability to determine any geometric dimensions. The error of geometric measurements does not exceed tens of centimeters.

3.7.1. Determination of cotton maturity In large agricultural enterprises, such as cotton or corn fields, the determination of crop maturity is usually done manually. Such calculations, as a rule, allow you to get only an approximate estimate and take a long time. Therefore, developers from the University of Tennessee (USA) developed a CV system with a quadcopter equipped with cameras to monitor the maturity of^[34] cotton^[35]

The photos obtained from the quadcopter were processed using an image recognition algorithm, while it was possible to calculate the crop with an accuracy of 85% to 93% using various methods and analysis tools.

Determining the weight of pigs

Weighing pigs is usually done only twice during their entire life: at the beginning and at the end of fattening. It is not very difficult to drive animals to the scales, but this is a huge stress for the animal, and pigs lose weight from stress. If livestock farmers knew more precisely how the fattening process of each piglet is going on, then it would be possible to draw up an individual fattening program and determine the individual composition of food additives, which would significantly improve the overall yield of products.

Therefore, a new, non-invasive method of weighing animals based on a computer vision system was developed that estimates the weight of pigs from photo and video data using machine learning. Based on the obtained data, the fattening process is corrected.

CV for milking cows

GEA Farm Technologies has developed a cow milking robot CV system. The CV system solves the problem of accurately pointing the cups of pumps on the nipples of the udder (objects), using the object tracking system and the structural illumination system, which is necessary to determine the range from the camera to all objects.

A system was also developed for automatic detection of objects on video and their tracking in inter-frame space in real time: assessing the position, shape parameters and motion dynamics at each moment of time.

In the CV system, important parameters of object tracking are the time of capturing an object by the CV system and its further tracking. In the developed CV, it was possible to achieve a time of 3-10 ms for capturing all 4 objects, and 0.6 ms for installing tracking of all found objects. The CV system can also determine the range to objects in the range of 200-700 mm with an accuracy of less than 2.5 mm over the entire range of distances.

Precision farming

Currently, solutions for Precision Agriculture are widespread around the world, which, due to the accurate positioning of agricultural machinery on the cultivated field and, therefore, more accurate processing of arable land, allow raising yields by 10% or more.

Computer Vision Systems has developed an accurate farming system, which, due to the use of the CV system, allows you to achieve an accuracy of positioning the machining tool (implant) 2 cm at a distance of 6 m (see figure below).

Retail

CV, combined with artificial intelligence algorithms, allows retailers to automate processes that previously needed to be performed manually. You can automatically receive notifications of finished goods or non-compliance with other buyer requirements by creating an analytics system directly at the point of sale.

For example, X5 plans to introduce computer vision systems in its stores. to monitor the presence of a complete assortment of goods on the shelves and to post sold-out goods on time, and control the length of the queue at the cash desk and, if more customers appear in the store halls, immediately increase the number of working cash desks^[36].

In 2018, Amazon opened the Amazon Go store, with the Just Walk Out Shopping^[37] solution^[38]which allows you to pay for goods automatically when leaving the store without approaching the box office. CV cameras are able to recognize not only the actions of the buyer when he takes goods from the shelf and puts them in the basket, but also vice versa when he puts goods back on the shelf. In this case, the item is removed from the buyer's virtual cart. The cameras keep track of the customer all the time he's inside the store, without facial recognition.

Logistics, delivery of goods

Inventory Analysis

Computer Vision Systems has developed a new technology for determining the volumes of wood logs using image analysis. To obtain accurate data, it is enough to photograph a stack of logs from two sides. Then the image processing program will independently determine the number of logs, the density of laying and introduce the necessary corrections. Additional options include bark quantity, wood quality (rot detection), and some other parameters^[39].

This system is capable of providing an error in calculating the volume of wood of no more than 3%. Measurement accuracy is 97-98%. For comparison, with the manual method, the measurement accuracy is 85-95%, and when passing a forest truck through an expensive laser frame - 90-95%.

Other Pattern Recognition Applications

Computer Vision Systems has developed a system for controlling the population and movement of Amur tigers using computer vision based on the recognition of individuals from photographs from camera traps (more details). The system automatically determines whether the tiger belongs to a specific unique number or name based on images obtained by the system from the camera traps. The system allows you to enter information about each tiger: a unique identifier, name, gender, age, how many times was photographed and a map with his photos, have links to related tigers and the ability to place these related ties. Tiger identification is carried out using computer vision algorithms using convolutional neural networks.

Production

Modern high-tech production requires special approaches to controlling the quality of products. Computer vision (CV) made a real technological breakthrough and significantly expanded the possibilities of flaw detection in industry, moved it to a new, higher level. Technology now allows you to track quality not only after the manufacture of a product or product, but also directly during the production process. In addition, CV systems can significantly simplify and accelerate the inspection of production equipment, units and utilities in service (more details).

The Future of Computer Vision

CV is a rapidly growing area of ​ ​ digital technology that affects many aspects of everyday life.

Apple introduced facial recognition into new iPhone models, acquiring companies such as PrimeSense, RealFace and Faceshift. The American AngelList portal, combining startups and investors, compiled a list of 529 new companies that work in the field of computer vision^[40]. The average capitalization of such startups is $5.2 million. Many startups attract capital from 5 to 10 million dollars. The portal notes that the flow of investment in computer vision is increasing. Replacing human vision with computer vision in many areas is a very profitable investment of capital.

The accuracy of computer analysis of video information is growing all the time and the use of CV can give great savings along with improved quality.

There are five main trends in the development of CV^[41]:

1. Growth of industrial computer vision systems. CV for medical devices, pharmaceuticals, food production, automotive industry provides a higher level of quality control, and CV for industry is expected to be the main trend in the field of computer vision in 2019.
2. Cloud-based deep learning systems. Deep learning algorithms and neural network classifiers will allow faster and more accurate classification and recognition of images from CV systems. In the coming years, the number of such developments will increase significantly.
3. Robotics. The use of industrial robots is rapidly increasing. Therefore, the demand for CV systems for robots will grow.
4. The increase in requirements for optics parameters for CV, which is caused by the increase in requirements for clarity and resolution of CV images. Sensors for CV cameras with higher resolution and with a large number of pixels are being developed and manufactured, however, without high-quality optics, these improvements will be of little use. Therefore, innovative solutions are being developed, such as microlenses for each pixel, etc., which can radically increase the performance of optical systems that have already approached their technological limits in traditional solutions.
5. Use thermal images to monitor manufacturing processes. Usually, thermal cameras were used mainly for military purposes, in security video surveillance. Thermal imaging in combination with CV can detect such anomalies in the manufacturing process that are not visible to the eye or conventional CV systems.

General Use Artificial Intelligence (AGI)

The term (AGI) Artificial General Intelligence, which appeared relatively recently, means the ability of a computer to make abstracted conclusions or at least imitate this process, thereby bringing it closer to the thinking of a person^[42]. However, AGI is still at the earliest stage of development. Abstract thinking remains an unsolvable problem for artificial intelligence.

It is for this reason that AGI technologies are on the Gartner curve at the very beginning of the rise of the "innovation trigger."

CV is one of the important components of AI artificial intelligence technologies (see figure below).

Market

The range of CV applications has expanded considerably in the last 10-15 years. Tractica in its report on the computer vision market in 2014 indicates six areas of CV. In 2016, the new version of the Tractica report already indicates eight areas of application of computer vision: Retail (retail) trade and Agriculture (agriculture) were added.

Global CV Market

Estimates of the volume of the world market for CV systems by various analytical companies vary quite a lot depending on the research methodology, taxonomy and classification of computer vision technologies. Various analytical companies evaluate the market according to their own methods, including or not including certain technologies and areas in the scope of substantive assessment. For example, some companies may include X-ray installations or MRI in CV technologies, others believe that these technologies are not related to CV. Some companies distinguish the pattern recognition market separately from the CV market, and according to their estimates, it exceeds the CV market (or what they consider CV).

Some companies can evaluate the CV market along with the accompanying artificial intelligence (AI) technology, others distinguish AI into a separate market segment. On the other hand, not the entire volume of AI belongs to CV.

All this makes it difficult to objectively evaluate the CV market in the world, individual regions and countries.

For example, the most authoritative analytical company McKinsey in its study on artificial intelligence indicates that the boundaries between many CV technologies are not clearly defined, so market volumes cannot be accurately determined.

So, for 2016, McKinsey estimates the Computer Vision market with a large "spread": from $2.5 to $3.5 billion. Moreover, the largest share of investments among related technologies where CV can be used is machine learning (Machine Learning) with an investment level of $5-7 billion.

Below are estimates of the CV market, its segments and related technologies from various global analytics companies.

Market Research Future

Market Research Future estimates the global CV market in 2017 at 9.2 billion dollars USA and expects it to exceed $48.3 billion by 2023 with steady growth of^[43]which increases after 2020 (see figure below).

Marketsandmarkets

According to the company MarketsandMarkets the leaders of the CV market in 2023 will be the following regions of^[44]:

• Asia Pacific (about 32%),
• North America (about 27%),
• Europe (about 22%).

In the same order, leaders in terms of growth are located, and the Asia-Pacific region is ahead by a large margin (more than 8% of average annual growth).

Marketsandmarkets also estimates the AI artificial intelligence solutions market for CV at $3.62 billion in 2017, rising to $25.32 billion in 2023.

Tractica

Tractica is more conservative in its estimates due to the stricter segmentation of CV technologies. Analysts divided the CV market into three main segments: Software, Services and Hardware^[45].

Tractica uses a method of estimating the market by the income generated by each segment. If in 2016 the revenue on it amounted to $1.1 billion, then in 2017 the figure was already close to $2 billion. By 2025, revenue in the market under consideration, according to Tractica, will reach $26.2 billion.

Maximize Market Research

Maximize Market Research is not as optimistic about the growth rate of the CV market as Market Research Future, although it begins its forecast even with a slightly higher level - $10.06 billion in 2016. However, for 2024, it predicts a market volume of almost half as much - only $18.07 billion.

Component Markets for CV Systems

Sensor Matrix Market

According to the French company Yole Développement^[46], the market for sensor arrays for computer vision cameras will grow from $2 billion in 2017 to about $4 billion in 2023 with an average annual growth rate of CAGR of 12%. Market shares of companies producing sensor matrices (not only for computer vision), according to data for 2015-2016. are shown in the figure below.

The absolute leader in this market is Sony, followed by Samsung and Omnivision. This market, like the camera market as a whole, is very dynamic and there are a large number of mergers and acquisitions in it.

Camera Market

In 2018, the volume of deliveries of video cameras for video surveillance in the world amounted to about 130 million pieces ^[47].

In the UK in 2013 there were about 6 million IVS cameras (approximately 1 camera per 11 people)^[48].

In the Chinese city of Tianjin (a large industrial center near Beijing) in 2015, there were more than 600 thousand high-resolution cameras that produced 50 petabytes of video daily.

Although many video surveillance systems still require staff supervision, development in the field of automated computer vision for security purposes is one of the most noticeable trends in^[49].

Consumer market

The consumer market (Consumer), according to many analysts, remains the largest vertical market for computer vision and one of its fastest growing segments^[50].

According to the portal Statista.com the consumer market for artificial intelligence with CV grew from $2 billion in 2015 to $17.7 billion in 2019 with a CAGR of 40%^[51]

CV market in Russia

Main article: Video analytics (Russian market)

As mentioned above, the computer vision market is very complex to estimate its volume and forecast its growth for a number of reasons:

• Uncertainty of taxonomy: what exactly should be attributed to computer vision? Some companies evaluate, for example, only the market for video cameras, not including software. Others refer to computer vision only as smart cameras with embedded image processing software, and individual image processing platforms and artificial intelligence are considered another segment, etc. On the other hand, it is obvious that not all artificial intelligence solutions are used for CV purposes.
• Despite the fact that CV technologies have been used for a long time, in Russia the CV systems and solutions market cannot yet be considered finally formed.
• Computer vision is used in a wide range of tasks and often it is evaluated only for certain segments, for example, facial recognition, which, in turn, can also be used in many sectors of the market: security video surveillance, retail, search activities, etc.
• Solutions using CV technologies are in most cases an organic part of other, broader solutions, such as Smart City, and sometimes it is difficult to estimate its share in these solutions.

During the study, a survey of more than 50 CV market participants was conducted. Many of them found it difficult to assess the volume of the CV market in Russia and give a forecast of its development for the next 3-5 years. The responses of the survey participants, who estimated the volume of the CV market in Russia, were characterized by a large "spread" - from 1 to 30 billion rubles. currently and from 5 to 100 billion by 2025.

Forecasts of the average annual growth rate for the period from 2019 to 2025 also had quite wide deviations: from 5% to 50% of CAGR.

According to TAdviser analysts, this is due to the fact that survey participants usually work in certain segments of a wide and multifaceted market, so their estimates for determining the total volume can be subjective.

Evaluation methodology

Based on these prerequisites, a multi-factor comparative evaluation technique was chosen to evaluate the CV market in Russia, which is used to process data on insufficiently well-defined and structured markets and new technologies.

This method involves comparing the volumes of other well-known markets (most often the world, as well as the market of developed countries and regions of the world) and evaluating regional markets by other known parameters, for example, by the share in world GDP. The obtained result is subjected to multifactorial verification according to other indirect data, which shows the degree of objectivity of the initial assessment and makes it possible to correct it. In this study, this evaluation method showed a fairly good convergence of results.

The assessment of the computer vision market in Russia included three of its main segments: software, services and equipment. Valuation factors: income generated by each segment, as well as the volume of sales of equipment and software related to CV technologies.

As a result of the analysis of trends in the development of the computer vision market in Russia by the method and comparing them with world trends, the estimate of the CV market in Russia in 2018 amounted to about 8 billion rubles.

By the end of 2023, the volume of the CV market in Russia may reach and exceed 38 billion rubles. while maintaining the ruble exchange rate as of the study date, as well as maintaining current trends in the development of the digital economy.

The CAGR growth rate of the CV market in Russia after 2021 may increase to 40% per year, with the planned results of the implementation of the national digital economy program.

Factors contributing to the growth of the CV market in Russia

The growth factors in the use of CV technologies in Russia include, first of all, the following:

• Development of the national digital economy program, in which computer vision is prescribed by a separate paragraph in the section "Neurotechnologies and Artificial Intelligence"^[52];
• Potential of Russian innovations in the automation and robotics market^[53];
• Development of CV applications in retail to trade^[54]
• Development of Smart City solutions^[55], Safe City^[56][57] and intelligent transport systems^[58]
• Automation of industrial production^[59];
• Development of the Internet of Prophetic (IoT) and industrial internetaIIoT Internet^[60]

Factors impeding CV growth in Russia

Some experts believe that the impact of computer vision on the economy, social sphere and our daily life will be significant, however, Russia the conditions for innovation, the creation of new technology companies and the launch of large projects are not yet enough. For the success of Russian companies in the field of computer vision, high-class specialists are needed, however, according to these experts, the Russian system of the highest special one formations is not yet coping with the requirements of the market for training professional personnel in the field of CV "^[61]

According to other market participants, one of the main constraints may be the lack of necessary skills and knowledge among Russian developers of CV systems and related software^[62].

A deterrent to the development of CV in Russia may also be the insufficient development of related technologies of a wide range - from actuators to integrated circuits. This prevents the development of related industries in which CV technologies are needed (for example, robotics).

Among other obstacles to the development of CV in Russia that CV market participants indicate are the following:

• Imperfection of legislation in the field of artificial intelligence in the Russian Federation;
• Security and privacy issues;
• Improper pricing and business models;
• Insufficient accuracy and reliability of systems;
• High cost of development and implementation;
• Regulatory issues;
• Long payback time;
• High customer expectations for CV system performance and cost.

Status and prospects of CV development in Russia

The results of the survey conducted during the study show the state of the market for 2018 and the most promising industries for the development of computer vision for the next 2-3 years (see figures below).

• Video surveillance and security 32%
• Cars and transport 5%
• Production sector 17%
• Defense sector 8%
• Consumer market 5%
• Medicine and health care 14%
• Agriculture 3%
• Retail and wholesale 10%
• Mail, logistics, delivery of goods 3%
• Other 3%

The most promising industries for the application of CV in Russia according to the results of the survey during the study:

• Video surveillance and security 28%
• Cars and vehicles 16%
• Production sector 16%
• Defense sector 12%
• Consumer market 8%
• Medicine and health care 28%
• Agriculture 4%
• Retail and wholesale 8%

Economic impact of computer vision systems

• Lower costs

Many enterprises use manual product quality checks, which leads to high labor costs. CV systems, with their proper integration, can perform quality control tasks faster and more accurately than a person. The calculation of efficiency is carried out on a case-by-case basis, some common templates do not exist here.

• Operational benefits

Reducing the time for quality control or sorting of products during mass production allows you to significantly increase the production speed and, therefore, the overall labor productivity. In addition, the non-compliance of the product with the specified parameters not detected in advance can cause the production line to stop and, therefore, the cost of time and labor for its maintenance and restarting. Computer vision can eliminate these losses.

• Data collection and parameter tracking

The collection of data from CV systems and their integration into the software solutions of MES (Manufacturing Execution System) systems provides a deep understanding of the production process, facilitates analysis and finding ways to improve it. Such improvements can be monitored and then taken into account in the overall assessment of the effectiveness of CV implementation.

• Quality improvement

CV systems ensure that the output of production lines will only appear exactly corresponding to the product quality parameters. It is often this aspect that is crucial when implementing CV systems. The degree of satisfaction of the end user and public opinion about the quality and reliability of the products of the enterprise is difficult to assess in terms of economic efficiency, however, from the point of view of marketing this is a very important aspect.

• Reduce unproductive costs

Manufacturers often put some level of unproductive costs (losses) in the cost of production. For example, when releasing packaged liquids, manufacturers lay down some volumes in excess of the rated ones, since underfilling by several percent will lead to a complete rejection of the tank at the output control stage, but on the contrary, an excessive volume of liquid in most cases will not lead to losses of the nominal volume. The use of CV systems can significantly reduce the allowable tolerance for excess packaged products and reduce unproductive costs.

• Increased security

This is one of the main motivations in the implementation of CV systems. Eliminating operators from harmful environments and replacing them with CV systems will reduce labor costs and increase employee safety, that is, reduce possible losses for the payment of insurance and compensation.

CV Implementation Performance Assessment Example

Consider an online marketplace where users sell used clothing. To do this, users of the portal need to download several photos of clothes and give a short description of the product^[63].

Marketplace rules prescribe that you can offer only new or little-worn clothes for sale, and it is clothes, not gadgets, for example. In addition, preferences may be given to known clothing brands with proven quality. Insufficient high-quality photos, even if other requirements are met, are not subject to publication. To do this, the marketplace staff has several moderators who evaluate the suitability of a particular announcement for publication.

Suppose that there are four moderators in the marketplace staff to perform this work, which must view 150,000 unique images per day uploaded by users to the site. Suppose that 15% of them will be rejected due to poor content quality (i.e. 22,500 pictures). Each moderator takes about 10 seconds to evaluate the usability of the image for publication.

The salary of the moderator is 12 dollars per hour, each moderator works 8 hours a day.

A simple calculation shows that under ideal conditions, moderators can look at no more than 11,520 pictures (i.e. 8%) out of 150 thousand uploaded to the site daily.

What can the marketplace do in this situation? There are three possible options:

1. Hire more moderators, thereby significantly increasing overhead costs, which will inevitably lead to an increase in the size of the marketplace commission and, therefore, a decrease in the number of users.
2. Develop methods to reduce the number of images to be moderated.
3. Apply pattern recognition technology in the CV system, which will take on most routine work (for example, 90%), leaving the person (moderator) only the most difficult cases (10%).

Calculations show that when choosing the third option, 73% of manual work is saved, which gives a saving of $6850 per month. Cognex cites several of its own cost reduction cases when using CV systems in various industries.

• A well-known automaker introduced a CV system to control production equipment, which made it possible to reduce several service items with a fee of about $5,000 per month. The implementation of the CV system paid off in 6 months. The total efficiency estimate was about 100 thousand dollars per year.
• The introduction of computer vision robots at the production plant made it possible to reduce the cost of an hour of working time by $15 per worker. Savings per year amounted to about 160 thousand dollars.
• The use of computer vision has enabled a manufacturing facility in the United States to eliminate the use of expensive production line tooling for accurate positioning of processed products. This made it possible for the company to save about 120 thousand dollars a year.

Advanced Study Version

The expanded version of the study contains more complete information on computer vision technologies, on the latest trends in its convergence with artificial intelligence, as well as on new areas of application of SC. In addition, the expanded version provides an overview of Russian companies operating in various areas of KZ. For a report, request the mr@tadviser.ru address.

Notes