This paper proposes a Learning and Graph Optimization (LEGO) modular tracker to improve data association performance in online multi-object tracking (MOT). The LEGO tracker efficiently generates an association score map that facilitates accurate and efficient matching between objects by integrating graph optimization and a self-attention mechanism. By incorporating a Kalman filter to incorporate temporal consistency of object states, the state update process is enhanced, ensuring consistent tracking. The proposed method, which uses only LiDAR, achieves top performance among online vehicle trackers on the KITTI MOT benchmark.