This paper presents a novel method to accurately predict the concentrated solar flux distribution on the receiver, which is essential for the safe and efficient operation of a concentrating solar power (CSP) plant. Conventionally, the control system was operated assuming an ideal heliostat surface, but the imperfections of the actual heliostat surface caused performance degradation and safety hazards. To address this issue, this study presents an inverse deep learning ray tracing (iDLR) technique that infers the heliostat surface profile from target images captured during a standard calibration procedure, and successfully performs a sim-to-real transfer in a real environment. When evaluated under real operating conditions for 63 heliostats, iDLR achieved a mean absolute error (MAE) of 0.17 mm, and agreed well with the deflectometry-based ground truth in 84% of cases. When applied to a ray tracing simulation, the flux density prediction accuracy reached 90%, which is 26% better than the method assuming an ideal heliostat surface. It has demonstrated generalization performance by maintaining high prediction accuracy even in double extrapolation scenarios involving unknown solar positions and receiver projections. iDLR is a scalable, automated, and cost-effective solution that is expected to contribute to improved flux control, accurate performance modeling, and improved efficiency and safety of CSP plants by integrating realistic heliostat surface models into digital twins.