Multimodal Large Language Models (MLLMs) excel at simple vision-language
tasks but struggle when faced with complex tasks that require multiple
capabilities, such as simultaneously recognizing objects, counting them, and
understanding their spatial relationships. This might be partially the result
of the fact that Visual Instruction Tuning (VIT), a critical training step for
MLLMs, has traditionally focused on scaling data volume, but not the
compositional complexity of training examples. We propose COMPACT
(COMPositional Atomic-to-complex visual Capability Tuning), which generates a
training dataset explicitly controlling for the compositional complexity of the
training examples. The data from COMPACT allows MLLMs to train on combinations
of atomic capabilities to learn complex capabilities more efficiently. Across
all benchmarks, COMPACT achieves comparable performance to the LLaVA-665k VIT
while using less than 10% of its data budget, and even outperforms it on
several, especially those involving complex multi-capability tasks. For
example, COMPACT achieves substantial 83.3% improvement on MMStar and 94.0%
improvement on MM-Vet compared to the full-scale VIT on particularly complex
questions that require four or more atomic capabilities. COMPACT offers a
scalable, data-efficient, visual compositional tuning recipe to improve on
complex visual-language tasks.
