HardEval: An Execution-Verified Benchmark for AI-Driven Security Hardening and Compliance Remediation
[Kryptografia i bezpieczeństwo] Autorzy: Manohar Vellala, Soumya Banerjee, Sandip Roy i 2 więcej
Implementing Security Technical Implementation Guide (STIG) controls remains a demanding operating-system administration task. Each natural-language security requirement must be translated into a precise and persistent configuration change. These changes span Secure Shell (SSH) configuration, authentication policy, audit rules, and kernel parameters. Existing benchmarks do not measure whether a model can perform this task on a live system. Code benchmarks evaluate isolated programs without persistent system state, while agent benchmarks largely focus on applications rather than operating-system configuration. In this paper, we introduce \textbf{HardEval}, an execution-verified benchmark for evaluating language-model-assisted security hardening. HardEval is built from Red Hat Enterprise Linux 8 STIG controls that fail on an unhardened baseline system. Expert-written reference remediation…
- Kategoria arxiv
- cs.CR
- Kategoria
- Kryptografia i bezpieczeństwo