LLM agents are an emerging form of AI systems where large language models
(LLMs) serve as the central component, utilizing a diverse set of tools to
complete user-assigned tasks. Despite their great potential, LLM agents pose
significant security risks. When interacting with the external world, they may
encounter malicious commands from attackers, leading to the execution of
dangerous actions. A promising way to address this is by enforcing the
principle of least privilege: allowing only essential actions for task
completion while blocking unnecessary ones. However, achieving this is
challenging, as it requires covering diverse agent scenarios while preserving
both security and utility.
We introduce Progent, the first privilege control mechanism for LLM agents.
At its core is a domain-specific language for flexibly expressing privilege
control policies applied during agent execution. These policies provide
fine-grained constraints over tool calls, deciding when tool calls are
permissible and specifying fallbacks if they are not. This enables agent
developers and users to craft suitable policies for their specific use cases
and enforce them deterministically to guarantee security. Thanks to its modular
design, integrating Progent does not alter agent internals and requires only
minimal changes to agent implementation, enhancing its practicality and
potential for widespread adoption. To automate policy writing, we leverage LLMs
to generate policies based on user queries, which are then updated dynamically
for improved security and utility. Our extensive evaluation shows that it
enables strong security while preserving high utility across three distinct
scenarios or benchmarks: AgentDojo, ASB, and AgentPoison. Furthermore, we
perform an in-depth analysis, showcasing the effectiveness of its core
components and the resilience of its automated policy generation against
adaptive attacks.