This paper addresses the limitations of evaluating existing event stream-based trackers on short-term tracking datasets and presents FELT, a novel, large-scale long-term tracking dataset that considers long-term tracking in real-world scenarios. FELT comprises 1,044 long-term videos, 1.9 million RGB frame and event stream pairs, 60 different target objects, and 14 challenging attributes. Furthermore, we retrain and evaluate 21 baseline trackers on the FELT dataset to establish a benchmark. Furthermore, we propose AMTTrack, an RGB-event long-term visual tracker based on the Associative Memory Transformer (AMT). AMTTrack follows a single-stream tracking framework, efficiently aggregates multi-scale RGB/event templates and search tokens via a Hopfield search layer, and maintains dynamic template representations via an associative memory update method to address the problem of appearance changes in long-term tracking. We validate the effectiveness of the proposed tracker through extensive experiments on the FELT, FE108, VisEvent, and COESOT datasets. The datasets and source code will be made publicly available.