(Source: Shutterstock)
It’s obvious when a dog has been poorly trained. It doesn’t respond properly to commands, pushes boundaries, and behaves unpredictably. The same is true with a poorly trained artificial intelligence (AI) model. Only with AI, its not always easy to identify what went wrong with the training.
Research scientists globally are working with a variety of AI models trained on experimental and theoretical data. The goal is to predict a material’s properties before creating and testing it. They are using AI to design better medicines and industrial chemicals in a fraction of the time it takes for experimental trial and error.
But how can they trust the answers that AI models provide? It’s not just an academic question. Millions of investment dollars can ride on whether AI model predictions are reliable.
A research team from the Department of Energy’s Pacific Northwest National Laboratory has developed a method to determine how well a class of AI models called neural network potentials has been trained. Further, it can identify when a prediction is outside the boundaries of its training and where it needs more training to improve—a process called active learning. The research team, led by PNNL data scientists Jenna Bilbrey Pope and Sutanay Choudhury, describes how the new uncertainty quantification method works in a research article published in NPJ Computational Materials.
A dog that has been poorly trained is like an AI model that has been poorly trained. It doesn’t know its boundaries. (Source: Jaromir Chalabala/Shutterstock) [Editor’s Note: “But he’s a good boy.”]
The research team, led by PNNL data scientists Jenna Bilbrey Pope and Sutanay Choudhury, describes how the new uncertainty quantification method works in a research article published in NPJ Computational Materials. The team is also making the method publicly available on GitHub as part of its larger repository, Scalable Neural Network Atomic Potentials (SNAP), to anyone who wants to apply it to their own work.
“We noticed that some uncertainty models tend to be overconfident, even when the actual error in prediction is high,” said Bilbrey Pope. “This is common for most deep neural networks. However, a model trained with SNAP gives a metric that mitigates this overconfidence. Ideally, you’d want to look at both prediction uncertainty and training data uncertainty to assess your overall model performance.”
Instilling trust in AI model training to speed discovery
Research scientists want to take advantage of AI’s speed of predictions, but right now, there’s a tradeoff between speed and accuracy. An AI model can make predictions in seconds that might take a supercomputer 12 hours to compute using traditional computationally intensive methods. However, chemists and materials scientists still see AI as a black box.
The PNNL data science team’s uncertainty measurement provides a way to understand how much they should trust an AI prediction.
“AI should be able to accurately detect its knowledge boundaries,” said Choudhury. “We want our AI models to come with a confidence guarantee. We want to be able to make statements such as ‘This prediction provides 85% confidence that catalyst A is better than catalyst B, based on your requirements.’”
In their published study, the researchers chose to benchmark their uncertainty method with one of the most advanced foundation models for atomistic materials chemistry, called MACE. The researchers calculated how well the model is trained to calculate the energy of specific families of materials. These calculations are important to understanding how well the AI model can approximate the more time- and energy-intensive methods that run on supercomputers. The results show what kinds of simulations can be calculated with confidence that the answers are accurate.
This kind of trust and confidence in predictions is crucial to realizing the potential of incorporating AI workflows into everyday laboratory work and the creation of autonomous laboratories where AI becomes a trusted lab assistant, the researchers added.
“We have worked to make it possible to ‘wrap’ any neural network potentials for chemistry into our framework,” said Choudhury. “Then in a SNAP, they suddenly have the power of being uncertainty aware.” Now, if only puppies could be trained in a snap.
In addition to Bilbrey and Choudhury, PNNL data scientists Jesun S. Firoz and Mal-Soon Lee contributed to the study. This work was supported by the “Transferring exascale computational chemistry to cloud computing environment and emerging hardware technologies” (TEC4) project, which is funded by the DOE Office of Science, Office of Basic Energy Sciences.
About PNNL
Pacific Northwest National Laboratory draws on its distinguishing strengths in chemistry, Earth sciences, biology and data science to advance scientific knowledge and address challenges in energy resiliency and national security. Founded in 1965, PNNL is operated by Battelle and supported by the Office of Science of the U.S. Department of Energy. The Office of Science is the single largest supporter of basic research in the physical sciences in the United States and is working to address some of the most pressing challenges of our time. For more information, visit the DOE Office of Science website. For more information on PNNL, visit PNNL’s News Center. Follow us on Twitter, Facebook, LinkedIn and Instagram.
Note: This article was initially posted on the PNNL News Site and is reproduced here with permission.
Karyn Hede is a Senior Science Communicator and Media Relations Advisor at PNNL.
