This paper proposes R2Vul, a novel method for detecting software vulnerabilities. R2Vul combines Reinforcement Learning-Based AI Feedback (RLAIF) and Structured Inference Distillation to train small-code LLMs to detect vulnerabilities and generate security-conscious explanations. Unlike existing thought processes and directive tuning methods, R2Vul rewards well-founded explanations over plausible but unfounded ones through RLAIF, resulting in more accurate detections and higher-quality inferences. To support RLAIF, we built the first multilingual vulnerability detection preference dataset, consisting of 18,000 high-quality samples from C#, JavaScript, Java, Python, and C. Across five programming languages, we compared the performance of four static analysis tools, eight state-of-the-art LLM-based baseline models, and various fine-tuning methods. We demonstrate that the 1.5 billion-parameter R2Vul model outperforms a 32 billion-parameter teacher model and leading commercial LLMs such as Claude-4-Opus. Additionally, we introduced a lightweight correction step to reduce the false positive rate under various imbalanced data distributions. Finally, qualitative analysis shows that both LLM and human raters consistently rated the R2Vul model's inferences higher than other inference-based baseline models.
