Beyond F1: Evaluating Coverage and Failure Recovery in AI Model Security Scanners
[Kryptografia i bezpieczeństwo] Autorzy: Qianlong Lan, Vinothini Pandurangan, Anuj Kaul i 1 więcej
Static scanners are increasingly used to identify executable or otherwise unsafe content in machine- learning artifacts, yet conventional evaluation metrics characterize only cases where a scanner yields a usable security judgment. We evaluate ModelScan, ModelAudit, and Fickling using a controlled, artifact-backed benchmark on a synthetic corpus of 170 Pickle and PyTorch focused artifacts across 145 specimen families, 135 of which have binary security ground truth and 10 of which are intentionally malformed without labels. We explicitly distinguish non-N/A coverage, analysis completion, definitive security decisions, non-security findings, and unsupported outcomes. On labeled families, ModelAudit produced definitive security decisions for all 135 families (100%), Fickling for 110 (81.5%), and ModelScan for 67 (49.6%). Conditional on making a definitive judgment, ModelScan achieved 100% …
- Kategoria arxiv
- cs.CR
- Kategoria
- Kryptografia i bezpieczeństwo