GPipe (library for training of neuronets)

2019: Disclosure of source codes

In March, 2019 Google opened source codes of the GPipe library used for training of deep neural networks. The library is intended for the Lingvo platform which is constructed on TensorFlow technology and is applied to modeling of the sequences.

GPipe can be applied to any neuronet consisting of a set of levels of the sequences and allows researchers to scale "easily" performance of artificial intelligence, the Yanping Huang Google AI programmer (Yanping Huang) says.

Google opened source codes of library for training of big neuronets

Deep neural networks (DNN) carry out a set of tasks of machine learning, including speech recognition, visual recognition and language processing. Larger DNN models lead to improvement of quality of accomplishment of tasks. Past experience shows that there is a direct dependence between the size of model and accuracy of classification. In GPipe we show use of pipeline parallelism to increase learning efficiency of DNN and to bypass these restrictions, - Juan in the blog of Google wrote.

Considerable part of a gain of performance of GPipe is caused with distribution of memory for AI models. On tensor processors of the second generation Google (TPU), each of which contains eight computing cores and 64 Gbytes of memory (8 Gbytes on a core), GPipe reduces intermediate use of memory from 6.26 Gbytes to 3.46 Gbytes that allowed to use 318 million parameters on one core of the accelerator. Huang says that without GPipe one core ​​ can train up to 82 million model parameters.

It not only advantage of GPipe. It also breaks models on different accelerators and automatically breaks "mini-batches" of the training examples into smaller "microbatches" and also conveyorizes accomplishment on "microbatches".^[1]

Robotics

Notes