This paper focuses on Internet of Things (IoT) systems, which must respond in real time while managing fluctuating resource constraints such as energy and bandwidth. We note that existing methods struggle to handle operational constraints that change over time, and propose a novel Budgeted Multi-Armed Bandit framework tailored for IoT applications with dynamic operational limits. This model introduces a decaying violation budget, which restrictively allows constraint violations in the early stages of learning and gradually enforces stricter compliance over time. We present the Budgeted Upper Confidence Bound (UCB) algorithm, which adaptively balances performance optimization and compliance with time-varying constraints, and provide theoretical guarantees that Budgeted UCB achieves sublinear regret and logarithmic constraint violations during the learning period. Extensive simulations in a wireless communication environment demonstrate that the proposed method achieves faster adaptation and better constraint satisfaction than standard online learning methods, highlighting the framework's potential for building adaptive and resource-aware IoT systems.
