Gemini Robotics On-Device is intended to make powerful robotics models more accessible and adaptable. | Source: Google DeepMind
Google DeepMind this week introduced an on-device Gemini Robotics model for general-purpose dexterity and fast task adaptation. DeepMind said this vision language action, or VLA, model will bring Gemini 2.0’s multimodal reasoning and real-world understanding into the physical world.
Gemini Robotics On-Device is a robotics foundation model for two-armed robots, engineered to require minimal computational resources. Since the model is optimized locally and operates independently of a data network, DeepMind said it’s helpful for latency-sensitive applications. It can also ensure robustness in environments with intermittent or zero connectivity.
In addition to Gemini Robotics On-Device, DeepMind introduced the Gemini Robotics software development kit (SDK). Developers can use it to evaluate the VLA model for their tasks and environments, test it in DeepMind’s MuJoCo physics simulator, and quickly adapt it to new domains, with as few as 50 to 100 demonstrations. Developers can access the SDK by signing up to DeepMind’s trusted tester program.
DeepMind builds on Gemini 2.0 momentum
It has been only a few months since DeepMind introduced Gemini Robotics, and it is already building on its task generalization and dexterity capabilities capabilities. The Google unit said the on-device model is:
Designed for rapid experimentation with dexterous manipulation
Adaptable to new tasks through fine-tuning to improve performance
Optimized to run locally with low-latency inference
Gemini Robotics On-Device achieves strong visual, semantic, and behavioral generalization across a wide range of testing scenarios, the company claimed. The platform also enables robots to follow natural language instructions and complete highly dexterous tasks, such as unzipping bags or folding clothes. DeepMind will still offer the Gemini Robotics model for those seeking similar results without on-device limitations.
This system isn’t limited to tasks that will work out of the box. DeepMind said developers can adapt the model to achieve better performance for specific applications. The company tested the model on seven dexterous manipulation tasks of varying degrees of difficulty, including zipping a lunchbox, drawing a card, and pouring salad dressing.
DeepMind expands Gemini to more robot embodiments
While DeepMind trained its on-device model only for ALOHA robots, it was able to further adapt the model to a bi-arm Franka FR3 robot and the Apollo humanoid robot by Apptronik.
On the FR3 robot, DeepMind said the AI model followed general-purpose instructions. It could handle previously unseen objects and scenes, complete dexterous tasks like folding a dress, or execute industrial belt-assembly tasks that required precision and dexterity.
On the Apollo humanoid, DeepMind adapted the model to a significantly different embodiment. The same generalist model can follow natural language instructions and manipulate different objects, including previously unseen objects, in a general manner, said the company.
DeepMind asserted that it is developing all of its models in alignment with its AI principles and applying a holistic safety approach spanning semantic and physical safety. In practice, this means capturing semantic and content safety using the Live API and interfacing the models with low-level safety-critical controllers to execute the actions.
The company recommends evaluating the end-to-end system on its recently developed semantic safety benchmark and performing red-teaming exercises at all levels to expose the model’s safety vulnerabilities.
DeepMind added that its Responsible Development & Innovation (ReDI) team continues to analyze and advise on the real-world impact of all Gemini Robotics models, finding ways to maximize their societal impact and minimize risk. Its Responsibility & Safety Council (RSC) then reviews the assessments, providing feedback to help further maximize benefits and minimize risk.
To gain a deeper understanding of Gemini Robotics On-Device’s usage and safety profile and to gather feedback, the company is initially releasing it to a select group of trusted testers.