Falcon-H1, a new series of large language models with a hybrid architecture combining Transformer-based attention and State Space Models, achieves state-of-the-art performance and efficiency across various tasks and sizes.
In this report, we introduce Falcon-H1, a new series of large language models
(LLMs) featuring hybrid architecture designs optimized for both high
performance and efficiency across diverse use cases. Unlike earlier Falcon
models built solely on Transformer or Mamba architectures, Falcon-H1 adopts a
parallel hybrid approach that combines Transformer-based attention with State
Space Models (SSMs), known for superior long-context memory and computational
efficiency. We systematically revisited model design, data strategy, and
training dynamics, challenging conventional practices in the field. Falcon-H1
is released in multiple configurations, including base and instruction-tuned
variants at 0.5B, 1.5B, 1.5B-deep, 3B, 7B, and 34B parameters. Quantized
instruction-tuned models are also available, totaling over 30 checkpoints on
Hugging Face Hub. Falcon-H1 models demonstrate state-of-the-art performance and
exceptional parameter and training efficiency. The flagship Falcon-H1-34B
matches or outperforms models up to 70B scale, such as Qwen3-32B, Qwen2.5-72B,
and Llama3.3-70B, while using fewer parameters and less data. Smaller models
show similar trends: the Falcon-H1-1.5B-Deep rivals current leading 7B-10B
models, and Falcon-H1-0.5B performs comparably to typical 7B models from 2024.
These models excel across reasoning, mathematics, multilingual tasks,
instruction following, and scientific knowledge. With support for up to 256K
context tokens and 18 languages, Falcon-H1 is suitable for a wide range of
applications. All models are released under a permissive open-source license,
underscoring our commitment to accessible and impactful AI research.