MISiS: Language model for manipulator robot control

The main articles are:

• Robots (robotics)
• Service robots

2023: Integration with Robot Manipulator Control System

Specialists of the Department of Engineering Cybernetics NUST MISIS and the Center robotics Sber integrated are a large open language AI model in the control system - robot a manipulator. Young engineers have demonstrated the ability of the language model to draw up a plan for solving various tasks related to object manipulation without participation programmer and writing code. The university announced this TAdviser on June 28, 2023.

Service robots will soon not need a programmer and complex code

It is enough for a person to formulate in a natural language what exactly needs to be obtained as a result of the actions of a robot with objects in front of him. This approach opens up additional possibilities in robotics, allowing you to solve a whole range of problems that can now be set in the form of a text request in natural language.

Service robots have already become an attribute of everyday life, but they are still not able to reason. For example, a robo can be asked: "Bring orange juice." Our language model already has knowledge of the world and considers different options acceptable: go to the kitchen, go to the store, go to the juices machine, etc. She assesses them on the principle: this will positively or negatively affect the solution of the global task of "bringing orange juice." Thus, the option to "go to the bedroom" is discarded. Further, using sorting algorithms, the robot evaluates the possibility of performing this action. As a result, we get an action plan to achieve a global goal, consisting of many simple actions of the robot.

As practice has shown, our development makes it possible to apply knowledge about the world of a large language model (LLM), already trained on a huge amount of data, to a physical robot without its preliminary further training. In addition, we are not limited to choosing only language models for code generation. The first experiments we conducted with a local open source model - LLaMA-7B. However, the more powerful the language model, the more logically complex application problems it can solve, noted the author of the study Yaroslav Savelyev, 3rd year student of the Department of Engineering Cybernetics of NUST MISIS, expert of the Sberbank Robotics Laboratory.

Work on the project began nine months ago with the initiative of Albert Efimov, Ph.D., Head of the Department of Engineering Cybernetics, NUST MISIS, Vice President of Sberbank. The supervisor of studies was Alexander Semochkin, Ph.D. (F.-M.N.), chief engineer of the Center for Robotics of Sberbank. The goal was to use the language model's ability to reason to help robots adapt to different tasks and environments.

This AI capability holds very great promise for practical applications in industrial and service robotics. It will significantly reduce the cost in those areas where it is planned to regularly change or adjust tasks by programmers in production, for example: the automotive industry, electrical engineering, electronics; construction, chemical industry, medicine and pharmaceuticals, etc. In addition, the integration of large language models and robots will come in handy where a person lacks reaction speed or dangerous conditions exist, as well as in the event that people do not have the necessary skills and knowledge to interact with the robot.

The developed ability of robots will help engineers avoid developing complex algorithms for setting sequences of actions, as well as setting tasks through commands in a user-friendly form. Research in this direction is carried out by many scientific centers in the world.