Diverse instruction data is vital for effective instruction tuning of large
language models, as it enables the model to generalize across different types
of inputs . Building such diversified instruction dataset is an essential step
in this process. Existing approaches often leverage large language models to
automatically explore and generate diverse instructions, ensuring both data
diversity and quality. However, they tend to overlook an important factor in
real-world applications: on-task relevance. In practice, only a few real-world
applications require a truly general-purpose model; most benefit from
task-specific knowledge tailored to their particular use case. Therefore, it is
vital to develop instruction augmentation methods that not only maintain
diversity but are also optimized for specific, real-world scenarios.
We thus introduce Task Centric Instruction Augmentation (TCIA), a framework
that systematically expands instructions while preserving both diversity and
task alignment. By representing instructions in a discrete query-constraints
space, TCIA creates a rich set of task-relevant instructions and enables models
to generalize to these task-specific instructions without sacrificing overall
performance. Experiments show that TCIA improves open-source LLMs’ performance
by an average of 8.7% across four real-world, task-specific applications, and
in some cases outperforming leading closed-source models. These improvements do
not compromise general instruction-following ability, making TCIA a scalable
and efficient solution for adapting LLMs to real-world, task-focused
applications.