This paper presents a learning-based algorithm for improving the safety and efficiency of roundabout driving in environments where autonomous vehicles (AVs) and human drivers coexist. A deep Q-learning network is used to learn optimal strategies in complex multi-vehicle roundabout scenarios, and the Kolmogorov-Arnold Network (KAN) enhances the AV's understanding of the environment. To further enhance safety, an action inspector is used to filter out unsafe behaviors, a path planner optimizes driving efficiency, and model predictive control ensures stability and accuracy. Experimental results demonstrate that the proposed system outperforms existing state-of-the-art methods in terms of collision reduction, travel time reduction, stable training, and smooth reward convergence.