The new system allows you to create robots of any shape and complexity.
Scientists from the Laboratory of Computer Science and Artificial Intelligence at the Massachusetts Institute of Technology (MIT CSAIL) have developed a new robot control system called "Neural Jacobian Fields" (NJF). This system allows robots to learn how to control their own bodies using only visual data from the camera, without the need for complex sensors or preset models. As noted by Sizhe's lead researcher Lester Lee, a graduate student at MIT CSAIL.:
Instead of writing code, it's enough for the robot to show what needs to be done, and it will learn to achieve the goal on its own.
The main idea of NJF is to change the approach to creating robots. Traditionally, robots are designed to be rigid and equipped with multiple sensors to create an accurate digital replica used for control. However, this approach is not suitable for soft, deformable or non-standard robots. NJF allows the robot to independently create a model of its body based on observations. This allows developers to focus on creating new, non-standard forms of robots without worrying about the complexities of modeling and control. The learning process is similar to how a human learns to control his fingers: the robot makes random movements, observes the result and adapts.
The NJF system has been tested on various robots: a pneumatic soft robotic arm, an Allegro rigid arm, a 3D-printed robotic arm, and a rotating platform without built-in sensors. In all cases, the system has successfully learned to determine the shape of the robot and its response to control signals using only video from the camera and random movements. NJF is based on a neural network that captures the relationship between the three-dimensional geometry of the robot and its response to control signals. The system uses the principle of neural radiation fields (NeRF), extending it to teach not only the shape of the robot, but also the Jacobian field, which predicts the movement of any point of the robot's body in response to motor commands.
The model is trained without human intervention and prior knowledge of the robot's structure. The robot makes random movements, which are recorded by several cameras. The system independently determines the relationship between control signals and movement. After training, the robot only needs one camera for real-time control. This approach and learning rate makes NJF more efficient than many physical simulations for soft robots, which are often too resource-intensive for real-time use. Even simple 2D models of fingers and sliders were able to learn this mapping using just a few examples.
The NJF system has great potential for practical application. Robots with NJF will be able to perform agricultural work with centimeter accuracy, work on construction sites without complex sensors, and navigate dynamic environments where traditional methods don't work. Although NJF training now requires multiple cameras, and the training needs to be repeated for each robot, researchers are already working on a more accessible version. In the future, users will be able to record the robot's movements using their phone and use this video to create a control model without special equipment and prior knowledge.
