Developing reliable AI systems to assist human clinicians in multi-modal
medical diagnosis has long been a key objective for researchers. Recently,
Multi-modal Large Language Models (MLLMs) have gained significant attention and
achieved success across various domains. With strong reasoning capabilities and
the ability to perform diverse tasks based on user instructions, they hold
great potential for enhancing medical diagnosis. However, directly applying
MLLMs to the medical domain still presents challenges. They lack detailed
perception of visual inputs, limiting their ability to perform quantitative
image analysis, which is crucial for medical diagnostics. Additionally, MLLMs
often exhibit hallucinations and inconsistencies in reasoning, whereas clinical
diagnoses must adhere strictly to established criteria. To address these
challenges, we propose MedAgent-Pro, an evidence-based reasoning agentic system
designed to achieve reliable, explainable, and precise medical diagnoses. This
is accomplished through a hierarchical workflow: at the task level,
knowledge-based reasoning generate reliable diagnostic plans for specific
diseases following retrieved clinical criteria. While at the case level,
multiple tool agents process multi-modal inputs, analyze different indicators
according to the plan, and provide a final diagnosis based on both quantitative
and qualitative evidence. Comprehensive experiments on both 2D and 3D medical
diagnosis tasks demonstrate the superiority and effectiveness of MedAgent-Pro,
while case studies further highlight its reliability and interpretability. The
code is available at https://github.com/jinlab-imvr/MedAgent-Pro.
