Video analytics terms, scopes of application, Video Content Analysis technologies

Determinations

Video analytics (VCA, Video Content Analysis) – the computerized processing and the automatic analysis of video content which arrives on the video server from video cameras, wearable devices and devices of Internet of Things of IoT equipped with webcams.

• The video analytics is the technology using methods of computer vision for the automated obtaining different data on the basis of the analysis of the sequence of the images arriving from video cameras in real time or from historical records.
• The video analytics represents the software (S) for work with video content. The complex of the algorithms of machine vision allowing a message video monitoring is the cornerstone of the software and to make data analysis without direct participation of the person.
• The traditional solution including functions of any video analytics is under construction according to the scheme: camera + analyst's back-end. I.e. the camera just drives video flow on the server, and special software on the server does all video analysis.

During video surveillance in industries, city and housing and also in different social media, a huge number of video data for which it is required systems data storage (DWH) with a high capacity is generated. Resolution capability of video images all the time increases, and the quantity of the stored content grows exponential.

The video analytics gains the increasing popularity for many reasons in recent years. It allows to manage flexibly video flows in the analysis of their content "on the fly", at automation of analytic functions. It allows personnel to concentrate on certain incidents on videos, but not to spend time for viewing long uniform video flows that allows to reduce costs and number of staff. Intelligent systems of security with video analytics can begin record, for example, only at the beginning of some movement in a camera viewing field. At the same time load of network decreases and the space in storage system is saved.

By means of the systems of video analytics, it is possible to obtain valuable information on quality of work of personnel of the enterprise (for example, selling assistants in a trading floor), thus, it is possible to make more adequate estimates of its work.

The systems of video analytics do not require excessively bulky infrastructure and even the small enterprises, shops and so forth quite are able to afford its use. The intensity of use of functions of video analytics can be regulated flexibly in process of business needs, selecting those functions which are necessary in a specific case. It allows to create the customized solutions.

The standard system architecture of VCA is shown in the drawing below.

The video analytics automates video surveillance process, does it convenient in use and considerably reduces costs for actions in which video surveillance is used. The need for video analytics grows in various sectors of the economy, such as financial sector and services, retail, transport, production and transportation of minerals, production, etc. Besides, growth of requirements to IP security systems and their infrastructure and also increase in importance of security in everyday life, also leads to growth of the market of video analytics.

There is also a term "computer vision" ("machine vision", technical sight"). This technology is often confused with video analytics. However, they are inadequate. It is possible to tell that the video analytics is a component of computer vision regarding the analysis of the image.

Computer vision (Computer Vision) is a technology (and also area of researches) on automation of understanding of what we see in the world around.

Video analytics (VCA, Video Content Analysis) are private applications of computer vision which take information and knowledge from video content, i.e. give answers to questions:

• Who: recognition and identification of people;
• That: objects, actions, events, behavior, relationship;
• Where: geolocation, space (3D) and planar (2D) location;
• When: marking of date and time, season.

Three main types of applications of video analytics:

• Retrospective: what already happened, i.e. management of videos of archives, search, sorting, obtaining legal proofs;
• This moment: that there is now, i.e. a control of a situation, receiving warnings in real time, coding, a video flow compression;
• Prospection: what can or most likely will occur, i.e. predictions on the basis of events of the past and the present, forecasting of events or activity, detecting of the outlined anomalies.

Types of platforms of video analytics

Video analytics on a dedicated server

For example, it can be the server of intellectual video surveillance of IVS (Intelligent Video Surveillance) and the server of automatic recognition of numbers of the cars ALPR (Automatic License Plate Recognition). Such server is well scaled at increase in number of cameras and input of a new feature of the analysis of video images allows. Video data in this case are stored on the server and can be taken via the remote client program.

Video analytics on a network video recorder of NVR

The network video recorder of NVR (Network Video Recorder) can have some built in by functions of video analytics. However, input of new analytic functions in this case either is impossible, or is difficult. It is profitable to use such solution if the number of cameras is small and functions are fixed. Data in this case are stored on a video recorder and can be taken via the remote client program.

Video analytics on cameras

Surveillance cameras can have the built-in functions of video analytics also. Advantage is here that such opportunities of analytics in such cameras do not depend on the bandwidth of network and response time of the server. Such solution is profitable where the high efficiency and an immediate response is required, for example when tracking via the PTZ dome cameras. Video data in this case are stored on video cameras and can be taken via the remote client program.

Development History

There is a legend that by means of the big mirrors set on the upper platform of the Lighthouse of Alexandria, ancient Greeks could observe the ships far in the sea.

With the advent of the first casinos, their security services used complex systems of mirrors to observe game rooms. It is possible to tell that they were prototypes of video surveillance systems. However, development of these video surveillance systems began with emergence of the iconoscope – the electronic device for transfer of images.

The father of modern video systems and the inventor of the iconoscope, device for capture of video images, Victor Kuzmich Zvorykin, the Russian engineer, the graduate with honors of Saint Petersburg State Institute of Technology of 1911, the veteran of World War I and the officer of White Army is considered. However, working in Russia, he managed to conduct only basic researches in the field of remote transfer of images, and the invention of the iconoscope was made in the USA where Zvorykin emigrated after the victory of Bolsheviks (to return to Russia more precisely, did not return from another business trip to the USA where it was sent by command of White army for purchase of radio stations, without seeing sense).

During scientific work in Saint Petersburg State Institute of Technology, it conducted researches together with professor Boris Rozing who created not electronic option of a kinescope to which at that time it was possible to transfer only the simplest images. Professor Rozing died in the thirties, being in exile in Arkhangelsk, not having an opportunity to continue scientific developments.

The 103-storey skyscraper of the Empire State Building in New York in 1932 became the first point of the telecast of the image. Video signal from the iconoscope was transferred by the 2.5 kW transmitter and was accepted on the kinescope of construction of Rozinga which is at distance of 100 km in the building of RCA (Radio Corporation of America).

Thus, the beginning of an era of television is considered 1932, however, it belongs also to the beginning to development of video surveillance systems.

The first practical use of the so-called "the closed television system" of CCTV (closed curcuit television), was performed by the German engineer Walter Bruch in 1941 in Penemyunda, during tests of the Fau-2 rocket. It is the first a case of use of video surveillance known in the history in practice. The operator had to sit permanently in front of the monitor, watching the events on the launch pad since the video then was not implemented yet. So proceeded till 1951, the first VTR videorecorders (Video Tape Recorder) did not appear yet.

Since then, video surveillance systems were improved practically each 10 years.

• Beginning of the 1950th years: emergence of the devices allowing to transfer the image on the magnetic tape;
• The end of 1950 – the beginning of the 1960th: use of video cameras for observation on roads, important objects and in places of mass accumulation of people;
• The 1970th years: availability of home videorecorders and video cameras;
• The 1990th years: emergence of digital video systems (DVR);
• The 2000th years: emergence of network systems of video surveillance;
• The 2010th years: development and use of cloud video cameras which can work without peripheral equipment (servers of video analytics, recorders, IP systems) at the enterprise platform, sending video data to a cloud.

Technologies continue to develop, and during 2020-2025 there can be algorithms and systems which will be capable to distinguish objects and even events directly in a video flow. Cameras will be capable to distinguish unusual situations and to take the corresponding actions – to inform the operator, to independently cause intelligence agencies and so forth.

Standards

2020: In Russia the AI standard for situational video analytics is developed

On July 20, 2020 it became known of creation of the Russia's first national standard in the field of artificial intelligence for situational video analytics. The document prepared by Videointellect LLC (develops the systems of computer vision for use in difficult conditions, public places with big accumulation of people, on objects of the industry), provided technical committee on standardization of shopping mall 164th Artificial intelligence created based on RVC.

GOST P "Information technologies. Artificial intelligence. Situational video analytics. Terms and determinations" is the first in group of the standards setting regulatory requirements in the field of situational video analytics. They will regulate utilization properties, test procedures and quality evaluations and requirements to placement of the equipment of the technical systems of intellectual video surveillance.

It is supposed that adoption of the standard as national will allow to arrange normative regulation in the field of situational video analytics and, in the subsequent, to eliminate technical barriers at application of similar "smart" information systems.

Modern video surveillance systems are inconceivable without use of the intellectual technologies of data processing allowing to analyze in real time not only separate images, but also the whole sequences of dynamic events and scenes — the chairman of technical committee on standardization of shopping mall 164th Artificial intelligence  Sergey Garbuk says. — Domestic and foreign developers propose the whole range of solutions of this sort. However the lack of terminological unity in this area often puts customers and integrators of systems in a difficult situation, complicating the choice of the solution optimal in each case.

According to him, introduction of the standard setting uniform terms and determinations in the field of situational video analytics will promote growth of efficiency of application of the similar systems and, eventually, – to increase in interest of the market in use of artificial intelligence technologies.^[1]

Functionality

In the drawing basic functions of video analytics are shown below. On the basis of these basic functions and their combination, various services and new features of analytics can be created.

Improvement of images

In computer vision and in computer graphics different methods and algorithms of recovery and improvement of images, such as noise reduction (denoising) and elimination of blurring (deblurring) are applied. Besides, methods of increase in clearness of images by means of neuronets are used: i.e. "super-permission" of SR (Super Resolution) based on several images of an object and also super-permission based on the only image of SISR (Single Image Super Resolution) ^[2]

Detecting of the movement

Detecting of the movement – process of detection of change of provision of an object concerning its environment or change of an environment concerning an object. When comparing several consecutive images of a scene, the VCA system can recognize a start of motion of any object in a scene.

Face recognition

Face recognition – the practical application of the theory of image identification which task includes automatic localization of the person on a still or moving image and, in case of need, identification of the personality by characteristic parameters of the person. Face recognition of people and determination of the identity of the person – one of the most common VCA functions which is used practically in all modern security systems based on intellectual video surveillance.

Recognition of aimless behavior (Loitering)

"The aimless behavior", loitering (Loitering) is a stay on one place or within one scene in public space for an appreciable length of time at large. In a number of the countries such behavior is forbidden legislatively. Anyway, it can indirectly demonstrate illegal intentions therefore the persons showing signs of such behavior happens it is necessary to reveal at video surveillance. The VCA systems have flexibly configured algorithms defining Loitering-behavior of subjects. In the drawing the example of recognition of Loitering with tracking of movement of the subject (a white dotted line) is shown below.

Loss recognition, or the objects left unguarded

In the drawing the example of the suspicious object left unguarded (Abandoned object) is also shown above. Such objects in the VCA systems are usually selected with a framework with the corresponding explanation. It can be sign of the preparing terrorist attack therefore on the basis of data of video analytics it is necessary to delay as soon as possible the suspicious subject which left a subject and to find out that it left in it.

Similarly loss (disappearance) of an object, for example, of museum piece can be distinguished. In this case the VCA system immediately issues warning in one way or another.

The closed zone

Examples of differentiation of the closed zones are shown in the drawing below. At penetration of people into the closed zone a system issues warning, for example selects the violator with a frame.

Penetration detecting

Penetration detecting – a part of The Closed Zone service, an example it is shown in the drawing above)

Recognition of car numbers

Automatic license plate recognition is a VCA technology which uses optical character recognition on images for reading of registration signs of vehicles for obtaining information on location of vehicles.

In the drawing the process of recognition of number of the car consisting three stages is shown below: detection of number (License Plate Detection), Detection of characters on number (Character Detection) and character recognition (Character Recognition) at which methods of machine learning of a system of video analytics are used.

Tracking objects

Tracking objects – supporting service for recognition service of "aimless behavior" (Loitering), however, it can be used also for other purposes. In the drawing the example of such recognition is shown below. The image in the middle shows suspicious behavior, the person going behind. Usually people so ("a trace in a trace") do not go, and the system of video analytics is trained to make recognition of such behavior of subjects, with issue of warning of suspicious behavior.

Integration of functions

Many functions of video analytics often represent integration of several basic functions. For example, the analytics of the parking of cars can include the following functions:

• Penetration into the closed zone;
• Leaving of objects without supervision during certain time;
• Recognition of the movement of objects;
• Recognition of numbers.

Spheres of use of video analytics

Let's consider some practical applications of above-mentioned basic functions of video analytics. Let's notice that many more composite functions described below actually are integration of basic functions.

• The systems of video analytics - the Directory of systems and projects of video analytics

Systems of the Smart City

The systems of the Smart City – one of the most perspective scopes of the systems of video analytics.

Calculation of people and transport

Function of calculation of the people crossing the set line provides valuable information for acceptance of business solutions, in such spheres as:

• Trade: information on the number of visitors of shops, shopping centers and also separate zones of shops and shopping centers;
• Banks: obtaining information on the number of visitors of departments;
• Hotels and tourism: obtaining information on the number of visitors of restaurants, movie theaters, travel agencies and so forth.

Having this information, the management of the enterprise can:

• estimate overall effectiveness of a company performance;
• estimate efficiency of the carried-out marketing actions;
• estimate load of the areas;
• improve service by regulation of working schedules of personnel according to data on attendance.

It is separately necessary to note benefits of use of a system of calculation of visitors for lessors of floor spaces:

• assessment of popularity and forecasting of development of shopping center;
• assessment of attractiveness of certain squares and correction of rental rates.

The systems of calculation can also analyze routes and behavior of buyers in shopping centers. For example, by calculation of buyers in a zone of outdoor advertizing it is possible to estimate its efficiency and also the type of goods can estimate the sales demand on different.

Similarly, for vehicles it is possible to obtain the following valuable information:

• The number of the machines passing down the street for a certain period depending on time of day, a day of the week and a season;
• The number of the machines accumulating at the traffic light and the average time of waiting of journey of the intersection;
• The number of the machines which are passing through the check point in the closed zone and leaving it;
• Fillability of street parkings and its dependence on time;
• And also other information necessary for development planning of the transport system of the city.

The functionality of calculation of number of people and vehicles is important for work of the automated intelligent transport systems (ITS) which can improve a transport situation in the city, increase capacity of roads, optimize operation of traffic lights and so forth.

According to collected information it is possible to calculate macroscopic characteristics of traffic flow, namely such indicators as:

• average speed of a flow;
• flow volume (number of vehicles per hour);
• flux density (the number of vehicles on km);
• average employment of a band;
• length of vehicles (for the solution of a problem of classification of vehicles);
• queue length before the intersection;
• detecting of driving into the oncoming lane.

Analysis of video surveillance of a limited zone and perimeter

The analytics of video surveillance systems for protection of the closed zones and perimeters is intended for identification of attempts of unauthorized penetration into the closed zone, even for lack of physical barrier. The main services of analytics of systems for protection of the closed zones the following:

• identification of potential threats to an object in the closed zone;
• determination of probabilities of implementation of potential threats;
• determination of vulnerable zones of an object in the closed zone;
• detection of the fact of intersection of perimeter of the closed zone;
• informing relevant services on existence of potential threats or facts of penetration;
• sending of notifications and images of an incident to personnel of security on duty of an object, including wearable devices.

Standard problems of video analytics of vulnerable zones of the protected objects are:

• search, detection and recognition of suspicious objects and people;
• identification and recognition of changes of video images of certain zones in time.

For observation of perimeter of the closed zone the directed all-weather video cameras, including with function of infrared vision, with protection against weather influences (a rain, snow, a sleet, fog) are used. For observation in the closed zone PTZ dome cameras, with a possibility of turn of a lens in the necessary direction most often are used.

Face recognition

Now any commercial camera with permission of not less Full-HD can be suitable for face recognition. Therefore practically any shop, shopping center or office where there are people, is able to afford to mount the camera for face recognition, detection of queue and other^[3].

Many cameras for house video surveillance contain the built-in functions of face recognition that allows their owner to create databases of family members and friends who regularly visit him. The system of protection of the house can be customized on opening of a door for the permitted persons from the database and also issue of warnings, at a visit of unknown or undesirable persons^[4]. At the same time a system can consider a set of factors: such as existence or lack of points, make-up, and many other things.

In face recognition different technologies, but the main steps of process following^[5] can be used^[6]:

1. Из photo pictures or videos the image of the person (detection of the person) is taken. The person can be as lonely, and is in an environment of many persons. The turn of the head has no decisive impact on this step.
2. Приложение face recognition reads out geometrical parameters of the person: such as distance between eyes, distance from a forehead to a chin, etc. In total about 100 and more similar geometrical parameters can be considered. On the basis of these data the digital signature of the person (facial signature) is formed.
3. Сигнатура persons it is compared to other signatures from the database of the famous persons. As of May, 2018 the Federal Bureau of Investigation of the USA (FBI) has access to 412 million images of persons. Images of persons at least of 117 million Americans are available in different databases of police of the USA.
4. Определение the identity of the person with rather high accuracy exceeding 90%.

Some airports of the USA (New York, Atlanta, Minneapolis, Salt Lake City, etc.) use function of face recognition at registration for run instead of the boarding pass (with the consent of the passenger)^[7].

The similar systems are available in Russia in many institutions of club type (fitness clubs and so forth) with permanent clients^[8].

In marketing and advertizing campaigns it is used so-called anonymous (without identification) face recognition as for marketing efforts information on that is very useful, what is the time the person looks at this or that advertizing and what thus expresses to emotion his face. At the same time the following metrics can be used:

• Visibility (how many people paid attention to advertizing);
• Demography (age and a floor paid attention);
• Viewing time (how many on average time look at advertizing);
• Time of day (in what hours most of all attention is paid to advertizing)^[9].

At the same time costs and time of studying and market research, in comparison with manual methods in the past are considerably reduced: polls, manual calculation of visitors, and so forth.

There are many practical applications of face recognition by means of video analytics, some of them are listed below:

• Security at the airports. The department of internal security of the USA (The Department of Homeland Security) uses video analytics for face recognition of the people who are entering and leaving buildings of the airports to define those, people with the expired visa or wanted or under investigation.
• Face recognition for access to mobile devices. The company Apple for the first time used face recognition for an unblocking smartphones iPhone X (Face ID). According to the statement of Apple, chances of an incorrect unblocking are one on one million, however, MEDIA announced cases of an unblocking smartphones of parents their children of century China.
• Control at examinations in educational institutions. It is an effective remedy against attempts of examination by figureheads instead of poor students.
• Social web media. Facebook uses an algorithm for finding of persons when loading a photo on the platform, at the same time the inquiry is submitted whether you want to mark out friends on a photo. At the affirmative answer to a question, the link on pages of the mentioned friends is created. Face recognition accuracy on Facebook is 98%.
• Control on an input of the organizations. Some companies replace scanners of office badges with recognizers of persons.
• Religious communities. In churches face recognition for control of those who regularly go to services to keep track of activity of believers and also entering donations is used.
• Retail sellers in shopping centers. The video analytics can be used for recognition of suspects to identify potential thieves.

Industrial application

Production

On production of the video analyst it is used for the following main objectives:

• Quality control of products;
• Help in management of technology processes;
• Security of working;
• Prevention of plunders or other malicious actions.

Several decades of the video analyst ("machine vision", "technical sight") it is used in production processes for detection of defects, pollution, and other deviations in the made products. In the drawing the simplest system of video analytics for sorting of products on the conveyer belt is shown below.

Example of use of VCA in chemical production

The server of the VCA system perceives warning signals from the program of video analytics which works with a set of the video cameras installed in the territory of the enterprise of chemical production^[10]. Possible actions of reaction to the warning signals:

• Control of cameras (movement, record and so forth);
• Providing new video and audioinformation for operators and personnel of the enterprise, for example, change of a point of the overview, turning on of additional microphones;
• Commands for other attached devices or programs through the HTTP protocol
• Commands through a user interface (Windows) for start and setup of other devices or software;
• Start of SNMP traps (SNMP traps) for indication a monitoring software status under control of the SNMP protocol;
• Logging of warning messages and preserving them in databases for the subsequent analysis.

Use of video on difficult production sites often means many hours of hard work on viewing and classification of events on videos from many hundreds of cameras for the analysis of events. Nevertheless, at the same time there is no complete guarantee that the problem will be correctly identified. However, use of IP video cameras with good resolution capability, infrared vision and protection from weather conditions, working together with the platform of video analytics, gives the chance of the adequate analysis of events and reaction to them in real time.

The growing requirements to security on chemical production, require more sensitive and exact methods of video analytics.

Power

The energy sector is one of crucial for ensuring life activity of modern society and therefore it should provide reliable and stable power supply of the enterprises and housing. Security risks, unforeseen accidents, malicious actions and vandalism, theft of materials lead to growth of costs for power supply and increase risks of shutdowns and total accidents ("blackouts"). These problems are aggravated with the fact that many objects of power supply are in a zone of public reach and are not always provided with reliable protection or protection.

Therefore electricity providers are very interested in high-quality and effective video surveillance behind numerous distributed by objects and also in solutions of video analytics which allow to increase efficiency of video surveillance considerably. Features of use of solutions of video surveillance and analytics in the power industry, following:

• need of adaptation to severe environmental conditions;
• high cost of service of power objects;
• compatibility with the existing equipment;
• compliance to numerous regulatory requirements of the industry.

Severe conditions of the environment on distribution power substations are especially problematic and require specialized solutions. The high level of electromagnetic interferences, broad range of temperature changes, vibration and shaking and also existence of corrosion pollution promote increase in a possibility of degradation or failure of electric equipment.

The systems of video analytics in the power industry are applied to the following main objectives:

• Security
  Video surveillance – a fixed asset of prevention and investigation of cases of theft, unauthorized penetration, vandalism, terrorism and other undesirable actions concerning power objects. The precious metals which are a part of components of electric equipment are the desired purpose of criminals. However, seldom serviced remote objects of power supply usually cannot brag of good protection against thieves. Besides, such objects sometimes are exposed to the attacks of terrorists and normal vandals. All this can lead to not scheduled repairs, growth of housekeeping overheads of objects, to blackouts and rolling blackouts.
  Therefore, very important part of a system of video analytics for power objects is capability proactively to inform personnel on invasions on their territory quickly to direct the relevant employees for this purpose, for prevention of crimes and emergence of damage. Videos of the occurred incidents also help at investigation of crimes.
• Monitoring of the equipment
  The VCA systems allow to implement warning of wear, degradation, the threatening faults earlier, thus to provide efficiency and reliability of operation of electrical systems without involvement of additional personnel. Probability of hardware failures at the same time significantly decreases and service life of components can be increased significantly if to make preventive service of components which wear will be revealed through VCA. At the same time the cost of the systems of video analytics and observation will be only a small part in comparison with the asset cost of the equipment and its repair.
• Automation of power supply systems
  In the VCA systems cameras with determination of temperature of components of the equipment of substation can be used. By means of enough difficult algorithms of determination of anomalies, all thermal characteristics and their trends can be analyzed and, using "thermal rules", the warning signals of possible problems of overheating can be automatically started.
  Integration of video analytics into the SCADA systems which are used for control of electric equipment allows to bring extent of automation to new level.

Logistics

In the transport and logistic industry the greatest application was received by the following functions of video analytics^[11]:

Recognition of numbers of motor transport

Main Functions:

• Adding of numbers in black and white lists;
• Fast registration and the admission of motor transport for the territory of the logistic center, with record of an episode of journey through gate and across the territory and fixing of time;
• Data loading in the XLS or CSV format.

Advantages:

• Prevention of journey of not authorized vehicles on the territory of the logistic center;
• Automatic raising of a barrier at entrance and departure.

Search and tracking of suspects

When choosing the suspicious character on record from the camera, the platform of video analytics can perform the following operations:

• Make a freeze frame and create the video clip with images of similar people on records from other cameras in a chronological order;
• Construct a trajectory of the movement of an object on the plan of the premises.

Object search in a videoarchive using the loaded images according to the following parameters is possible:

• Form;
• Color;
• Size;
• Provision in a frame.

Using function of search of suspects (Suspect Search) it is possible to reconstruct an object route within a minute. It allows to find quickly the suspect and to give a command to security forces on detention of the violator.

Control of the cameras PTZ

Main Functions:

• Turn of the cameras PTZ in the desirable direction by means of the joystick or the keyboard;
• Scaling of the image by means of an optical zoom;
• Focus control of the camera in the automatic or manual modes;
• Task of the scenario of automatic operation of the cameras PTZ.

Advantages:

• Possibility of replacement of several stationary cameras with one camera PTZ with expansion of opportunities of the overview;
• Registration of the smallest parts on the image;
• Focusing of the camera on a desirable object and tracking it.

Tracking of objects

Main Functions:

• Installation of the minimum size of an object which movements should be monitored;
• Obtaining the immediate notification of alarm on the monitor, phone or e-mail;
• Intersection by an object of the set line (invasion on the territory and so forth);
• Movement of an object on the set zone;
• Long finding of an object on one place (Loitering).

Advantages:

• From personnel 24 hours a day are not required attention on monitors;
• Protection of property, loads and infrastructure of the logistic center;
• Security of the logistic center and its personnel;
• Prevention of possible terrorist attacks.

• Sabotage recognition

Function allows to prevent the following types of sabotage:

• Video camera defocusing;
• Turn of the camera aside from the direction of shooting set for it;
• Long dazzle of the camera;
• Peregorazhivaniye of a type of the camera.

Function provides issue of the immediate notification of ALARM on all listed actions on the monitor, phone or e-mail.

Face recognition

Functions:

• Integration with a control system of access to the check point of the logistic center or a warehouse;
• Creation of databases of "authorized representatives" and blacklisted;
• Obtaining automatic notifications of ALARM on the monitor, phone or e-mail about attempts of unauthorized penetration;
• Search of fragments on the person registered in archive, search of people in a videoarchive on their photo.

Advantages:

• It is not required to have personnel of protection at all check-points;
• Automatic admission to the territory and in limited zones of warehouses only the authorized personnel and control of time spent by them in this or that zone;
• High safety of personnel, the stored values and infrastructure of warehouses.

Deployment of the image from the panoramic camera like "fish eye"

Obtaining the "flat" image from the panoramic camera like "fish eye" which usually strongly distorts image perspective is possible. At the same time becomes possible to replace several normal cameras with one panoramic with more wide functionality, and to control several zones by means of one camera.

"Thermal card"

Function allows:

• trace the frequency of movement of personnel and vehicles across the territory of the logistic center or a warehouse;
• impose "the thermal card" on the image from the camera;
• create "the thermal map" of the premises, a warehouse, or all logistic center;
• generate reports on traffic density in certain time frames.

Advantages:

• Optimization of routes of movement of personnel;
• Tracking time, the necessary product which is carried out by the employee behind observation or the device.

It allows to optimize efficiency of personnel of the logistic center or a warehouse, to reduce costs.

Monitoring of personnel

It is possible to set several zones of monitoring of activity in a viewing field of one camera. A system otsluzhivat the movement or lack of activity in a monitoring zone in real time. If in a zone there is no movement during the set time frame, automatic notifications of ALARM STAFF MEMBER ABSEND on the monitor, phone or e-mail about absence of the employee in a workplace.

It allows to increase efficiency of the logistic center or a warehouse and also to trace validly time and quality of work of employees, to reduce risks of the undesirable situations connected with absence of the employee in a workplace.

Banks

The video analytics in banks is used first of all for calculation of visitors, issue of coupons early to the recognizable clients, identification of new clients, and also for:

• safety in the operational hall and meeting rooms
• safety in zones of self-service and ATMs
• safety in the work area of bank employees and clerks
• prevention of penetration on the territory of bank of robbers and malefactors
• notifications of security service specialists about appearance of swindlers from "the chrny list" and other unwanted persons
• prevention of terrorist attacks (detection of the left ownerless objects)
• prevention of emergence and a skaplivaniye of different marginal persons in self-service zones
• prevention of the acts of vandalism leading to damage of ATMs

Retail

Growth of amounts of retail requires permanent expansion of floor spaces and expansion of outlets. Besides, requirements of business consist in growth of efficiency and overhead reduction.

To estimate such important parameters for trade process optimization as number of visitors, conversion (the address of attention to these or those goods), the average size of a basket, and also to increase effective management of personnel, inventory managements and speed of calculations on point-of-sale terminals, the video analytics is one of the main tools. On the other hand, the video analytics helps to increase security, to minimize number of thefts, and frauds. The analysis of behavior of buyers by means of tools of video analytics can give valuable information (insight) for increase in satisfaction and growth of number of buyers.

Based on researches of Einfochips company, application of video analytics in retail trade can give the following benefits for trade enterprises:^[12]

In retail trade the following basic functions of video analytics are used:

Detection of the movement

The algorithm of detection of the movement will recognize the movement or movement of an object or the person in sight of the camera. The camera begins record at recognition of the movement against the background of a motionless environment. Algorithms and recognizers of the movement can be both very simple, and quite difficult, depending on an effective objective.

Management of queues

Queues to cash desks can be managed by means of special algorithms. For example, at achievement of a threshold of queue length, the warning signal can be sent to command center to open new cash desk. It helps to increase not only satisfaction of buyers, but also attendance of shop.

Face recognition

The algorithm of face recognition compares parameters of the person recognized by the camera to parameters of the person stored in the database. It gives the chance to identify frequent buyers, to trace an order of their purchases and the average time spent in shop. Face recognition can be also used for the purpose of prevention of theft and vandalism. Persons of the characters making illegal actions can be also stored in the special database and at emergence them in shop, the signal to protection can be issued more carefully to trace their behavior.

Thermal card

The Thermal Card function can help to keep track of advertizing efficiency in the trade premises. On the card the integrated intensity of finding of buyers about trade or displays and the average time of viewing is displayed.

It helps to identify the goods escaping buyers from attention and also to take measures on increases in amounts of their sales.

Integration of POS terminals

Long time retail suffered from undesirable activity of employees who bought discount cards of shop (or through friends) and did not authorized discounts to friends and other buyers. Then, using bonuses on the card which were quickly enough saved they received the different not earned benefita. Integration of video surveillance systems into POS terminals allows to reveal such suspicious actions, for example, when the cashier carries too often out the discount card which lies at near it on the terminal. Video proofs with a sale date are included in the special database according to which afterwards it is possible to carry out analysis of legitimacy of actions of staff of shop. It not only allows to record quickly fraudulent activity, but also at once to reveal their source.

"Blackening" of areas

Sometimes in shops there is a need to maintain privacy for VIP clients. Private data, such as credit card number which can be visible from the video camera on a settlement counter also should be protected. In such situations the video in such perimeters can be "blackened" that on record confidential parts did not get. It helps to prevent stealing of personal data, such as information of the credit card and other data.

Calculation of buyers

This function allows to count the number of the people who are entering and leaving the trade premises. In most cases traffic of buyers varies depending on time of day and days of the week. Calculation of buyers allows to understand better when to expect more clients and to take adequate measures.

Advantages of use of video analytics in retail trade:

• Analysis of behavior of consumer behavior and its trends;
• Optimization of work of personnel and its structure;
• Decrease in the total costs of ownership of trade infrastructure;
• Pro-active service of buyers;
• Prevention of losses.

Scientific research in the field of video analytics

Intelligent systems of video analytics which can take valuable information from video content flow gain ground in different areas, including retail, transport, municipal economy, the vital infrastructures, the enterprises, etc.

Since then, when the first CCTV system of Siemens company was installed on test rocket airfield in Penemyunda in Nazi Germany in 1942 far off to watch missile launches of "Fau-2", technologies of video surveillance and video analytics made significant progress. However, the high cost, insufficient image quality and limited opportunities of dissemination of monitoring information, caused the necessity of improvement of technologies.

In the modern systems of video analytics intelligent cameras with the built-in processing video or special analytical software platforms working at a remote server can be used. In such platforms algorithms of machine learning are even more often used to facilitate interpretation and data analysis in more and more increasing amounts of flows of video content.

Before application video analysts mentioned, generally the fields of protection and security, however, of technology of video analytics began to be diversified now, including wide applications of a business intelligence of BI (business intelligence) and also situation analysis (situational awareness)^[13].

In the field of video analytics which combines analytics of data (data science) and computer vision a set of vendors actively work, including both startups, and the big known companies. The activity and the competition in this area are very high that gives rise to life to a set of innovations in technologies and business models. However, the market is rather fragmented so far, i.e. many solutions on video analytics remain proprietary and transition from one vendor to another can cost very much, and it frightens off investors so far.

Use of neuronets and deep learning

Use of high-precision neuronets in video analytics allowed to expand functionality of security systems of the enterprises considerably. Neuronets became widely known since 2012. From now on there are more and more companies, both known, and beginners, began to use widely technology of neuronets for exact and reliable image understanding.

Neuronets use such Internet companies as Microsoft, Facebook, Google, Amazon, Instagram, Yandex and others, for example:

• Yandex provides function of recognition of make of the car for the Auto.ru portal;
• The CaptionBot application of Microsoft company automatically offers the signature for the image;
• The WhatDog application distinguishes breeds of dog.

For these purposes neuronets with deep learning of DNN (Deep Neural Network), or just deep neuronets are used now.

Deep neuronets are used for creation of systems which can distinguish objects and their properties from volume arrays of not marked data. Recently for the purposes of deep learning of neuronets the increasing application is found by the graphic processors GPU which allow to train huge data arrays for rather short time. Modern algorithms of recognition exceed on accuracy, existing 20-25 years ago approximately on two orders.

Models on the basis of DNN are used for image identification "on the fly", in those cases where the speed of recognition is very important quickly to perform some operations. However, time of training can occupy big ^[14] Therefore, standard DNN not always meet requirements of a delay for some real-time applications.

However, well "trained" DNN can have the high accuracy of image identification that is very important for video surveillance development.

In the drawing the structure of a system of video analytics with a neuronet of DNN is shown below.

Some number n of cameras monitor a certain area for the purpose of tracking of trajectories of the movement of people and objects. The neuronet of DNN is previously trained in object recognition, direction finding and speed of their movement. On the basis of this information the analysis of characteristics of an object is performed (for example, type and the brand of the vehicle, face recognition of people and so forth).

It can be a difficult task, especially in the conditions of limitation of cash computing resources. The technology of data scrubbing on the basis of relationship of RelDC (Relationship-Based Data Cleaning) can increase quality of recognition, even in the conditions of not really high-quality video.

Normal neuronets consist of the mutually connected computing nodes called by neurons, each of which activates nodes of the next layer with the set weight (value) of a signal. Activation begins on entrance neurons, and then internal "layers" of neurons are activated under the influence of the neurons attached to them according to signal transmission ratios. Normal neuronets work using the simple mechanism of distribution of a signal from an input for an exit and have no more than 2-3 inside layers of neurons.

Depending on number of the buried layers of neurons for training, neuronets are classified as "small" (shallow) and "deep" (deep), DNN.

Small neuronets usually contain 1-3 buried layers while number of layers in deep networks DNN – from three and more. Increase in number of layers increases learning efficiency of a neuronet and accuracy of image identification.

DNN can have a different network architecture, "model" (model) which also significantly influences learning process.

Deep learning of DL (Deep Learning) as kind of machine learning of ML (Machine Learning), uses different algorithms for data processing and simulation of thinking process to do the different conclusions consisting in object recognition and their behavior. At the same time becomes possible to recognize the handwritten text (even if DNN never "saw" handwriting of this person earlier), to understand the live speech (without the need for preliminary biometrics of a voice), and to distinguish different objects, for example, the class "animals", and in it – subclasses: "dog", "cat", "cow" and so forth. There are neuronets which "are able" to determine breeds of dog, cats and other animals by certain signs^[15].

Information in DNN is transferred and processed consistently from a layer on a layer when the signal output after processing serves as an input signal in a neuron of the previous layer for all, or a part of neurons of the subsequent layer, and force the value (amplitude) of a signal is defined by "weight" of this link from a neuron of the previous layer to a neuron of the following layer.

Depending on the received result at the exit of a layer of output neurons, consecutive correction of scales of separate links between neurons of the next layers can be made. This iterative process of correction of scales of links is called neuronet Learning.

Short history of development of neuronets

In 1943 American scientists: the neuropsychologist, the neurophysiologist, one of founders of cybernetics Warren McCulloch and neurolinguistics, the logician and the mathematician Walter Pitts invented the first device which it was possible to call a neuronet, Threshold Logic working by the principle for simulation of elementary transactions of neurons of a human brain^[16].

In the early sixties, Henry Kelly (Henry J. Kelley), professor of Polytechnical institute of the State of Virginia, developed model of the return distribution (Back Propagation Model) for training of a neuronet. Approximately at the same time the Japanese scientist Kunihiko Fukushima developed the concept of a convolution neuronet of CNN (Convolutional Neural Network), a kind of DNN. In the late seventies Fukushima developed the first hierarchical multilayer neuronet, under ^[17] which could distinguish visual images.

In development of scientists from Institute of cognitive science (Institute for Cognitive Science) of the university of California in San Diego, David Rumelkhart (David E. Rumelhart) and Ronald Williams (Ronald J. Williams) and also Jeffrey Hinton (Geoffrey E. Hinton) from Carnegie-Mellon's University from Philadelphia in 1989 for the first time in practice was used the algorithm of the return distribution (Back Propagation) which is theoretically offered at the beginning of the 60th^[18].

In 1997 Sepp Hochreiter and Jürgen Schmidhuber from Johann Kepler's University in Austria developed a so-called "long short-term memory" of LSTM (Long Short-Term Memory) for recursive neuronets of RNN (Recurrent Neural Network)^[19].

Now the set of inventions and improvements in architectural models of neuronets, activation functions of neurons and so forth is made that led to the explosive growth of development of deep neuronets. Played a role and the accompanying technologies, concepts and a deposit of numerous scientists and developers that led to synergy development of the region of neuronets in relation to video analytics.

Analysis of Big Data, artificial intelligence

Artificial intelligence technologies of AI (Artificial intelligence, AI) quickly extend worldwide. Possibilities of artificial intelligence, in particular, are widely applied in video analytics: for example, for monitoring of traffic of traffic in the cities (Smart City), or in intelligent systems of electric power distribution (Smart Grid).

AI technologies (AI) – in fact are other name of neuronets with a possibility of training. There are three main training methods of neuronets: with the teacher, without teacher, with a reinforcement^[20].

At supervised learning the neural network studies at previously marked data set for obtaining answers which are used for algorithm accuracy assessment on training data. At unsupervised learning the model uses not marked data from which the algorithm independently tries to take signs and dependences.

Semi-supervised learning represents something. It uses a small amount of the marked data and a big set of not marked data. And training with a reinforcement trains an algorithm by means of the system of encouragement.

Therefore when we speak about use of AI in video surveillance, we actually mean use of neuronets with a possibility of unsupervised learning.

Use of AI in video surveillance

At Carnegie's (USA) university in 2019 the research of use of AI for video surveillance was conducted and the Global Index of use of AI was developed for video surveillance of AIGS (AI Global Surveillance) which shows extent of use of AI for video surveillance in 176 countries of the world (without distinction of legitimacy of such use)^[21].

The research showed that now AI technologies for video surveillance extend quicker and in bigger number of the countries, than it is represented to many experts working as in the field of AI, and video analytics. At least, 75 of 176 countries in the world actively use AI for the purposes of video surveillance and video analytics. Most often AI is used in such applications of video analytics as platforms of the Smart or Safe City (56 countries), the systems of face recognition (64 countries) and also in the systems of Smart protection of law and order, Smart Police (52 countries).

Most violently AI technologies for video surveillance develop in China, thanks to developments of such companies as Huawei, Hikvision, Dahua and ZTE which deliver AI technologies to 63 countries of the world. More than thirty of them are members of the initiative "One belt and one way" which is put forward by China as offers on consolidation of projects of "An economic belt of Shelkovy of a way" and "Sea Shelkovy ways of the 21st century" (Belt and Road Initiative, BRI).

Only one company Huawei delivers AI technologies for video surveillance at least to 50 countries of the world. With a big separation over number of the countries the Japanese NEC Corporation (14 countries) follows.

The companies of the USA also actively work in this area. The American AI technologies for video surveillance are delivered to 32 countries of the world.

The largest American players in this area are IBM companies (11 countries), Palantir (9 countries) and Cisco (6 countries). An important role is also played by developments of the companies from France, Germany, Israel and Japan.

The card of origin of the used AI technologies for video surveillance is given in a research (see the drawing below).

The countries where the American AI technologies for video surveillance prevail are shown on the map by blue color, red – the Chinese, red-blue bands show the countries where both the Chinese, and American technologies are used, and black – the countries where other technologies prevail.

On the card red and blue colors and also blue-red bands prevail. And it is interesting that both in the USA, and in China, both the Chinese, and American technologies are used. It means that China in this area. at least. does not cede the USA,

In the drawing the chart where distribution of AI technologies for video surveillance in different regions of the world is shown is shown below. From the chart it is visible that leaders in a scope of "smart video surveillance systems" are the countries of Southeast Asia (about 65% of the countries of the region), the Middle East and North Africa (more than 60% of the countries) and also the Southern and Central Asia (about 60% of the countries of the region).

In Europe (including the CIS countries) this indicator falls short of 50% of the countries, and in America – of 40%.

Also the chart of a contribution of the different companies to distribution of AI technologies is of a certain interest to video surveillance in the countries of the world. On the chart it is visible that from 75 countries where AI is applied to video surveillance, in 50 countries technologies of the Chinese company Huawei are used.

Use of Big Data in video analytics

"Big video data" are made by the escalating number of the cameras located in public places. In the world a large number of the network IP cameras making huge arrays of video data is already installed. More and more progressive tense, according to regulation on security of the different countries is required to store these data.

In the drawing growth of amount of data from 1995 to 2020 is shown below. Only one camera of high resolution produces 10-50 Gbytes of data in ^[22]. It is visible that in the last five years amount of data will increase approximately five times and they will be a source of valuable information (insight) which can be taken from Big Data.

From these Big Data it is possible to take a lot of useful information for marketing, for the organization of road traffic, for electric power distribution optimization and so forth.

For example, Japanese the operator NTT DoCoMo in 2018 implemented the solution of Internet of Things (IoT) which gives the chance to interpret and analyze data from cameras video surveillances directly on network edge (Edge computing) together with information from sensors and^[23]. DoCoMo implements this project together with the company Cloudian from California (USA) which developed the compact and high-speed device for the analysis Big Data of Cloudian AI Box. This device is equipped with the interface for IP- the cameras working in networks LTE and Wi-Fi.

The analysis of Big Data from surveillance cameras can be applied in different scenarios, such as:

• protection of public order;
• monitoring of quality on production;
• detection of presence of people;
• marketing programs in retail trade.

Transfer of large volumes of data to the central cloud is quite long process at which data are transferred with delays, at the same time there is quite heavy load of traffic on infrastructure of network. The solution at which processing of Big Data happens in close proximity to their generation and use (Edge computing) allows to accelerate considerably image identification and receiving useful information.

Group of scientists of department of computer sciences of the Kyyung He university, BB. Korea (Department of Computer Science and Engineering, Kyung Hee University) developed system architecture of video analytics for the distributed analysis of the Big^[24]shown in the drawing below. In it not only these video surveillances, but also strimingovy videos from the Internet and also video from the video hosting websites are used (Youtube and so forth)

Technology trends

Now, evolution of the systems of video analytics is defined by the following technology trends.

Intellectual and context-dependent data collection

In intelligent systems of video surveillance data collection happens on the basis of recognition of the taking place events getting to camera lenses. Depending on a context, collecting and fixing of data can happen more or less intensively. Therefore, it is possible to spend resources of video surveillance systems and analytics more effectively and also to increase the accuracy and reliability of collected data.

Infrastructures of Big Data

Technologies of Big Data give many opportunities for video analytics. Collecting of stream data from a set of video cameras and data processing directly by transfer considerably facilitate the analysis of Big Data. Architecture of Big Data facilitate scaling of the systems of intellectual video surveillance and input of new features of video analytics.

The systems of analytics in stream data transmission (streaming)

In recent years there were many strimingovy systems which allow to take data from video flows directly in the course of transfer to facilitate load of network and to accelerate process data analysis.

Predictive video analytics

In 2016-17 many algorithms of deep learning, for example, of Alpha AI in Google were developed. Deep neuronets receive most of all are used in video analytics as they allow to improve considerably process of investigation of incidents using video cameras and also in many cases allow to prevent the outlined incidents, or to reduce their negative effects.

The predictive video analytics can predict influence of an interference of signals, traffic overloads on networks and the quality of video of QoE (Quality of Experience) perceived by the user. These data then can be combined with indicators of KPI for business of service provider (for example, the telecom operator), including outflow of users, NPS and profitability of services. The automatic recommendations issued by the system of predictive video analytics help service providers to hold subscribers, to take pro-active actions for quality improvement of service and to quickly resolve the arising problems. Besides, the predictive video analytics can reduce buffering of video traffic to 40% and also raise QoE for the most important subscribers^[25].

Drones and Internet of Things (IoT).

Use of IoT devices and smart devices considerably expands opportunities and functionality of security systems and video surveillance. In the last time to expand a scope and functionality of video surveillance, the UAV unmanned aerial vehicles (drones) begin to be used more and more.

In the drawing examples of the video analytics received from cameras of Turn Drone Data into Business Outcomes drones are shown below^[26].

Integration of physical security and cyber security

Digital transformation of industrial assets and processes gradually leads to convergence of measures of physical and cyber security. Many enterprises still consider information technology security (IT) and operational technologies (OT) as certain areas. For this reason, malefactors often have an opportunity to find "holes" in physical protection thanks to different priorities and practicians of cyber security of IT and^[27].

The world of physical security passes to IP platforms more and more. The analytical agency IMS Research estimates that in 2020 about 22 billion devices will be connected to the Internet. Many of them treat video surveillance and video analytics. Therefore these two areas need to be developed in synergy. It not only will lead to increased security both in OT, and in IT, but also will allow to save many means.

According to Ponemon Institute, the effective plan of cyber security will help to reduce on average for the enterprises of the USA to 28% of losses of violation of normal transactions and damage from the IT attacks^[28].

New architecture of video surveillance systems and video analytics

All listed technology trends open many new opportunities for the systems of video analytics. However, system architects and developers should implement functionality of these technologies in specific developments and systems. Therefore, first of all, it is very important to develop and implement the corresponding architecture of video surveillance systems and video analytics in order that these opportunities could be implemented and used in synergy interaction.

In modern system architecture for video surveillance cloud computing and also the concept of boundary calculations (edge/fog computing) are actively used to process video given in close proximity to the place of their generation and use. It allows to receive significant savings on the bandwidth of network and to reach high efficiency of monitoring systems of security due to decrease in delays by transfer of video flows on network.

The cameras stopped at network edge are a part of nodes of video analytics which are capable to process video frames in real time, without their transfer to a remote central cloud. Boundary nodes are also capable to intellectualize data collection due to flexible configuration of picture frequency depending on a context of events in front of the video camera. If anything special does not happen on the stage, then picture frequency can be reduced. If in a frame the movement begins, the video camera increases picture frequency and if an incident is recognized – includes shooting with a high speed and in high resolution. It allows not only to save the bandwidth, but also computing resources and also to reduce the required amount of storage systems.

Open standards to which the main vendors of video surveillance systems and analytics follow also help to simplify considerably architecture of systems and to make them independent of solutions of specific vendors. The customer at the same time receives benefit that he can select the best vendor solution on each element of a system at a guarantee that all elements will be guaranteed compatible during the work.

Video surveillance systems and analysts

• The systems of video analytics - the Directory of systems and projects of video analytics
• Video surveillance systems Product catalog and projects
• Video surveillance (market of Russia)
• Video surveillance (world market)
• Trends in development of video surveillance systems
• Video surveillance as service - Video Surveilance as a Service, VSaaS
• Video surveillance systems: Design features of cameras
• How competently to select surveillance cameras
• Video analytics
• 10 methods raise income using the covert video surveillance
• IP video surveillance by the retail organizations for security and prevention of plunders
• ACS of the Control and management system for access
• Biometric identification

You See Also

• Video analytics (projects)
• Video surveillance systems - the directory of systems and projects
• Cybersecurity - Biometric identification - the directory of systems and projects
• VisionLabs - VisionLabs Luna
• Synes Tsefey Kompleks of video analytics - Synes
• Synes: The video analytical device - Synes
• Kipod Cluster - Synes
• CasRetail - Synes
• VisorJet with video analytics for the rotary camera - ElVIS-NeoTech
• ELISE Semantic processor (ELVEES Image Semantic Engine) - ElVIS-NeoTech
• Videointellect: The system of video analytics is Videointellect
• Video Synopsis - BriefCam
• R-Style: PRIME of the Video analyst - R-Style Computers
• The analytics put: The video analytics for shopping centers - Analytics put
• As the systems of computer vision change logistics and medicine
• Vocord
• Yandex Data Factory
• Cognitive Technologies
• Macroscop (Satellite Innovation)
• Aggregator
• Integra-with
• 3iTech (earlier 3i Technologies)
• LogistiX (Logistics-that)
• Intelligence Retail
• Wireless analytics (WiFi-analytics)
• Wireless analytics in the Russian retail
• Technologies of calculation of visitors
• Retailbase: WiFi-analytics - NPO Analitika

Robotics

Notes