PRvL presents the first comprehensive, open-source benchmark and toolkit for evaluating and deploying LLM-based PII redaction, systematically comparing architectures, training paradigms, and inference strategies to optimize accuracy, efficiency, and privacy leakage control across domains and languages.
➡️ 𝐊𝐞𝐲 𝐇𝐢𝐠𝐡𝐥𝐢𝐠𝐡𝐭𝐬 𝐨𝐟 𝐏𝐑𝐯𝐋:
🧪 𝐒𝐲𝐬𝐭𝐞𝐦𝐚𝐭𝐢𝐜 𝐌𝐮𝐥𝐭𝐢-𝐀𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞 𝐁𝐞𝐧𝐜𝐡𝐦𝐚𝐫𝐤𝐢𝐧𝐠: Evaluates six LLM families (Dense, Small, MoE, LRM, SSM, and NER baselines) across fine-tuning, instruction tuning, and RAG, measuring span accuracy, label fidelity, semantic preservation, and privacy leakage (SPriV) under both in-domain and cross-domain/language shifts.
🧩 𝐎𝐩𝐞𝐧-𝐒𝐨𝐮𝐫𝐜𝐞 𝐏𝐈𝐈 𝐑𝐞𝐝𝐚𝐜𝐭𝐢𝐨𝐧 𝐒𝐭𝐚𝐜𝐤: Releases PRvL — a reproducible, domain-customizable suite of fine-tuned and instruction-tuned models, retrieval pipelines, and evaluation scripts, supporting secure, self-hosted deployment without third-party dependencies.
🧠 𝐀𝐝𝐚𝐩𝐭𝐚𝐭𝐢𝐨𝐧 𝐏𝐚𝐫𝐚𝐝𝐢𝐠𝐦 𝐈𝐧𝐬𝐢𝐠𝐡𝐭𝐬: Demonstrates that instruction tuning consistently outperforms fine-tuning and RAG for PII redaction by reducing mislabels and leakage while preserving span accuracy, with smaller models like DeepSeek-Q1 matching or surpassing large-scale LLMs in efficiency–accuracy trade-offs.