SecInfer is a prompt injection attack defense system built on \emph{inference time scaling}, a novel paradigm that improves the performance of LLM by allocating additional computational resources during inference. SecInfer consists of two main steps: system-prompt-guided sampling, which generates multiple responses to a given input using various system prompts, and target-task-guided aggregation, which selects the response most likely to perform the intended task. Extensive experiments demonstrate that SecInfer effectively mitigates both conventional and adaptive prompt injection attacks by leveraging additional computation during inference, outperforming state-of-the-art defense systems and existing inference time scaling approaches.