Yunzhu Li, a PhD student at MIT CSAIL who led the project.
MIT CSAIL
We have moved one step closer towards human-level intelligence. Image recognition technology and tactile sensors are being joined together and are using each other to improve their abilities. A team at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have created a database of tactile and visual information and are using it to train an AI system to infer the look and feel of an object.
This system is still in the early research stage, as are most projects coming out of MIT CSAIL, but by connecting these two “senses: digitally the team may have given AI a new way of experiencing the world. This breakthrough could lead to far more sensitive and practical robotic arms that could improve any number of delicate or mission-critical operations. It also promises more to come in the advancement of AI systems that can understand, or at least imbibe, the world as we do.
Sense and sensibility
With a simple webcam, the team headed by Yunzhu Li, CSAIL PhD student and lead author of the paper on the system, built up a dataset of over 200 everyday objects being touched more than 12,000 times. They then reduced those 12,000 video clips into static frames, and used those to compile “VisGel,” a dataset of more than 3 million visual/tactile-paired images. Using that dataset, the team trained an AI model to predict what an object would feel like based on visual data of the surrounding area, and used a KUKA robotic arm paired with a GelSight tactile sensor to acquire the corresponding tactile information. For instance, the team would feed the system images of a certain point on a computer mouse, and the AI would use a generative adversarial network (GAN) to build a tactile map of the area.
GANs use a pair of networks to play off each other and improve their outputs: a generator network that compiles an image (or in this case a tactile map) for a discriminator network to test and compare against real (or ground truth) data. The team would then compare the model produced by the GAN to the tactile data picked up by the KUKA robotic arm, to check once more against the measurable “ground truth.” The system can also work the other way round, using tactile sensor data to create an image prediction of what a certain point on the object might look like. These images would also be run through a GAN, and compared a final time against a ground truth image to test the validity of the model’s outputs.
Dexterous digits
Connecting an image feed and a tactile sensor within an AI model represents a fascinating step in the progress of AI systems and robotic arms that experience the world more like us. Giving a dual-insight to a digital intelligence effectively compounds the information and the “knowledge” that this system has access to. This would theoretically allow an AI system to learn things about its environment and process information much faster and more effectively than a single-input system. In surgery, for example, robotic arms can currently handle incredibly delicate procedures such as prostatectomies using minimally invasive, or keyhole, surgery, which make up around 86% of all robotic surgeries in the U.S.
If a robotic arm could learn what an area should feel like only by using images of that area—for example, X-ray or MRI images of someone’s internal bones and organs—then these keyhole surgeries, often only requiring an incision of less than 2cm, could be performed in many more cases perhaps even where a robotic arm could not have performed surgery before. Robotic arms do not usually hold scalpels themselves, for example, and take much longer to sew wounds together than a human surgeon even if the results are better overall. More capable robotic arms could change this, and further improve the work of human surgeons as well.
In more industrial situations, an AI system that can recognize different materials and grasp things more effectively without having to repeatedly try to pick up an object could bring new capabilities to a wide range of different processes and sectors. Handling extremely hazardous materials such as nuclear waste, for example, could be made far safer if a human were not required to control a robotic arm and a system could use image inputs to learn how best to pick up a container or even raw radioactive waste with a significantly reduced chance of dropping and spilling toxic material. In construction, autonomous lifting arms or those attached to vehicles could calculate the weight of an object based on its material and 3D images of, say, a steel girder. When digging or drilling to lay foundations, prepare a site, or laying underwater pipelines, ultrasonic images could be fed into the system and paired with tactile probe data to determine exactly where to drill in real-time without damaging existing infrastructure or delicate ecosystems.
This project from MIT CSAIL is another step towards more autonomous robots, the development of which has gathered significant pace in recent months. Robots that can connect senses together and infer much more about their context and their environment could lead to groundbreaking advances in any number of industries that currently utilize robotic limbs. This project is still in the research phase, but by showing that this kind of intelligence is possible, the MIT CSAIL team have proved that the field of robotics is still only in its infancy, and there is far more to come before we reach the limit of what robots are capable of.