IBM’s open-source TerraMind AI uses 9 data modalities to transform Earth observation

IBM Corp. is partnering with the European Space Agency to create an artificial intelligence system that will keep tabs on climate change and other global issues, such as water scarcity, in real time using space-based data.

IBM says the new system, called TerraMind, is the most advanced “Earth observation” model ever conceived, and it’s being made available to researchers through the Hugging Face platform today.

The open-source model was trained on TerraMesh, which is the largest set of freely available geospatial data, created by researchers as part of the TerraMind project.

IBM says TerraMind is a multimodal model based on a novel “symmetric transformer-based encoder-decoder architecture.” It can handle pixel-based, token-based and sequence-based inputs and learn correlations across modalities. Although it was trained on more than 500 billion tokens of data, it’s a relatively small and lightweight model that uses about 10 times fewer computing resources than existing models for each modality it covers. As a result, it’s possible to scale TerraMind at a much lower cost, IBM says.

TerraMind’s multimodal capabilities are essential for Earth observation tasks, IBM says, because there are so many factors that need to be considered when weighing up the impact of climate change.

For instance, when it comes to predicting the risk of water scarcity, it’s necessary to take into account factors such as climate, temperature, rainfall, land use, vegetation and agricultural activities. Previously, researchers would have to use multiple models to study each of these factors separately. TerraMind brings all of this information together in a single model to create a more informed picture of what’s happening in a given location, allowing it to predict water scarcity risks with unprecedented accuracy

“What sets TerraMind apart is its ability to go beyond simply processing Earth observations with computer vision algorithms,” said IBM Research UK and Ireland Director Juan Bernabé-Moreno.

To build the TerraMesh dataset that underpins TerraMind, IBM’s researchers compiled data on everything from biomes to land use, land cover types and regions, to ensure that the model can be used to make predictions in any part of the planet, with minimal bias.

TerraMesh contains more than nine million globally distributed and spatiotemporally aligned data samples across nine core modalities. These include observations from the ESA’s satellites, information on surface characteristics, such as vegetation and land use, and geometry used to describe locations and features, such as latitude, longitude, altitude and simple text descriptions of different regions.

IBM says tests demonstrate that TerraMind is by far and away the best-performing foundational model for Earth observation ever made. The ESA led these evaluations, pitting TerraMind against 12 other models in its class on the PANGEA benchmark, which is used to measure AI performance on real-world tasks such as land cover classification, change detection, environmental monitoring and multi-temporal analysis.

According to the ESA, TerraMind outperformed competing models by 8% or more on every task within the PANGEA benchmark tests.

ESA Earth Observation Programmes Director Simonetta Cheli said TerraMind’s advantage stems from the insights provided by so many different modalities. “The ability to intuitively bring in contextual information and generate unseen scenarios is a critical step in unlocking the value of ESA data,” she said. “Compared to competitive models, it can uncover a deeper understanding of the Earth for researchers and businesses alike.”

Going further, TerraMind also employs a new technique called “Thinking-in-Modalties” tuning, which is similar to the “Chain-of-Thought” method used in reasoning-focused large language models such as OpenAI’s GPT-4o and DeepSeek Ltd.’s R1. TiM, as it’s known, involves self-generating additional synthetic training data from other modalities to enhance TerraMind’s performance beyond what regular fine-tuning could achieve.

IBM Research scientist Johannes Jakubik, who helped develop the TiM technique, said it works by generating synthetic data related to the problem the model is trying to address. So it might be programmed to generate additional detailed maps when asked to “think” about land cover when mapping bodies of water.

“This breakthrough can unlock unprecedented accuracy when specializing TerraMind for particular use cases,” Jakubik said.

IBM said TerraMind can be applied to solve long-term issues such as natural disaster management, environment monitoring, high-precision agriculture, urban and regional planning, critical infrastructure monitoring, forestry and biodiversity monitoring, as well as short-term problems like wildfire and flood monitoring.

TerraMind is yet another example of how AI is impacting almost every aspect of technology, said Holger Mueller of Constellation Research Inc.

“AI models that can make sense of our planet will be extremely useful when it comes to predicting, simulating and explaining natural and man-made events,” the analyst said. “IBM is advancing this with its new Thinking-in-Modalities approach, with nine core modalities based on more than nine million data points. It’s by far and away the most complete AI model of our planet, and it’s great that it’s being made open source.”

IBM has already published TerraMind on Hugging Face and the IBM Geospatial Studio, and it is planning to release fine-tuned version for disaster response and other high-impact use cases in the next few weeks.

“We are unlocking the full potential of space-based data to protect our planet,” said ESA Earth Observation Data Scientist Nicolas Longepe. “This project is a perfect example where the scientific community, big tech companies, and experts have collaborated to leverage this technology for the benefit of Earth sciences. The magic happens when earth observation data experts, machine learning experts, data scientists, and HPC engineers come together.”

Images: IBM/ESA