Harmful machine learning: what it is dangerous by and how to be protected

The contingency can appear one of the most serious obstacles on the way to the predicted mass implementation of the smart systems supplied with elements of the artificial intelligence (AI) – these systems are yet not so smart to have capability to opposition to malicious intent or fraud from the person. The scale of effects of such deception is unpredictable, it depend only on that, the functions transferred to the system supplied with AI are how responsible.

In technology history emergence of such obstacles not an isolated case – while this or that innovation stays in a germinal status, do not think of its possible negative effects. But sooner or later there comes the moment when unexpected acts into the forefront earlier. At the beginning of an avtomobilizm the security issue did not stand, and now it is the major. In the field of programming nobody could provide a possibility of existence of the malware (malware) up to creation of ARPANET network (the predecessor of the Internet), but from the middle of the eightieth malware in all its forms became serious threat for personal, and later and other computers. In the same way, since the end of the fiftieth years of the 20th century and almost until quite recently creators of machine learning technologies (ML) did not take a possibility of any threat into account. However and here it arose, in this case from Adversarial Machine Learning (AML) - the side branch of ML which became a theoretical basis for development of the tools capable to create noises in work of systems on the basis of ML. The term Adversarial Machine Learning still seldom occurs in Russian-language texts, it is translated as "competitive machine learning", but more precisely values from a row antagonistic, confrontational or contradictory therefore by analogy with malware we will translate it "harmful machine learning" are suitable for the word adversarial. Opening of a theoretical possibility of existence of AML and the first publications on this subject carry to 2004. The history AML and the analysis of current situation can be found in article "Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning" two Italian researchers of Battista Biggioa and Fabio Rolia published in 2018 ^[1]

Adversarial Machine Learning is the side branch of Machine Learning which became a theoretical basis for development of the tools capable to create noises in work of systems on the basis of machine learning

For the first ten years of existence of AML, i.e. till that time until systems on the basis of ML gained noticeable distribution, the relation to AML had theoretical character. However the situation sharply changed after a few years ago Ian Gudfellou, the child prodigy from Stanford University, one of the first publicly started talking about reality of the attacks on ML using AML. The lecture given by it devoted to the theory of this question is laid out in^[2], there is an article "Explaining and harnessing adversarial examples"^[3] written to them with coauthors].

For the translation of its name it is necessary to specify the key term of adversarial example. Adversarial example (AE) is that other as actually the instrument of the attack, it makes harmful impact on neural network with the purpose to cause errors in her behavior. Harmful influence of AE is an analog influence of malware, but in the application not to the code, and to ML.

What is AE simplest to explain on the most often used example of systems from computer vision where means of AML possible to cause optical illusions. It means that the attacked system "sees not that is actually". The classical laboratory case when "admixture" of specially created hostile noise (Adversarial Noise) to the source image of a panda recognized with the probability of 57.7% leads to recognition it, but already as a gibbon and with higher probability of 99.3% is given in article Gudfellou. The scope of AML is not limited to machine vision, she can mention the most different scopes of ML, everything where it is necessary to solve problems of recognition (the text, a sound …). Such means it is possible to bypass function Face ID in smartphone iPhone X or other biometric means of protecting.

The AML system attacked by means "sees not that is actually"

Existence of AML as antipode, helps to understand the nature of ML better. It is obvious that technologies ML are any not "artificial intelligence", but only the approach which is an alternative to programming at all it is more flexible regarding knowledge transfer to the computer. No more than that. And, if to the program as simple (if not trivial) to the carrier of knowledge, other program in which the programmer puts own knowledge can counteract, then trained the ML harm only another can do, but the technology too trained. In other words, against the program other program acts, and against a system other system on the basis of ML called by AML acts on the basis of ML.

For what to attack an AML system should have data on the victim, so-called "harmful knowledge" (Adversarial Knowledge, AK), i.e. to have an idea of how prepare and from what sources data for training, what these data, what basic functions of the attacked system, in what algorithms come it works what results, etc. Knowledge of subject to attack of AK defines the possible strategy of the attack. All AML attacks manage to be separated into two types – poisoning (poisoning) and distorting (evasion). The poisoning attacks are aimed at learning process of model, and the distorting attacks break functioning of earlier trained model which is already built in this or that system. In that and other case for creation of the corresponding weapon in the form of AE it is required to receive AK and to oppose them to knowledge gained in the course of normal by ML. So some knowledge opposes other knowledge. If to use the terminology accepted by information security specialists, then the strategy of "a white box", and distorting – "a black box" is suitable for the poisoning attacks more though one and does not exclude another.

The persons performing the poisoning attack aim to get access to data and to learning process of model to poison her and that in the subsequent it proved inadequately. For this purpose they can use distinction network means of penetration and deformation. The purpose of the distorting attacks is the inadequate behavior of already ready-made product with the model which is built in it created by means of ML. If the malefactor gets access to such product, then he can consider it to a large extent as a black box, finding out its characteristics and not penetrating into its device.

By efforts of marketing specialists as subject to the distorting attack usually consider cars UAVs, though all other devices equally can become the victims. The attack begins with the moment when the car becomes available to the malefactors using AML. Their actions carry to so-called return development (reverse engineering). In this case under a reverse engineering the research of the trained model and detection of its vulnerabilities is understood. For the analysis of behavior of a subsystem of machine vision of the car on its input slightly modified images of road signs can move in a huge number. The model is not ideal, sooner or later it is possible to find weak points and to use them in the evil. As show the experiments made in Princeton University when properties of model are studied, it is enough to apply simple distortions on the restrictive sign of speed that a system apprehended it as the sign of an obligatory stop. It is easy to imagine what will lead sudden braking on the highway to. These procedures are in details described in^[4].

The fact of existence of such threats caused immediate reaction of many vendors. The Nvidia company working in cooperation with Mercedes-Benz published the report^[5] in which the infrastructure solutions for protection of cars enclosed by it are described. The companies specializing in safety of airplanes suggest to extend to cars the technologies developed by them, for example Communication Lockdown (Blocking of communications) which completes the fighters of Israel of F-35I and F-16I which are under construction by request.

However, at the current level of development there are no ready theoretical approaches for creation of means of counteraction to the distorting attacks therefore potential threats from their party remain the major brake on the way of further distribution of autonomous cars. Professor of the Massachusetts Institute of Technology (MIT) Dan Song in the presentation at the conference of EmTech Digital which passed in San Francisco said on March 25, 2019 about it ^[6]

Robotics

Notes