To address the challenges of multispectral object detection for unmanned aerial vehicles (UAVs), we propose the DEPFusion framework, which includes the Dual-Domain Enhancement (DDE) and Priority-Guided Mamba Fusion (PGMF) modules. DDE addresses detail loss caused by low-light RGB images, while PGMF reduces interference information to improve local target modeling. With the DDE module, which utilizes the Cross-Scale Wavelet Mamba (CSWM) block and the Fourier Details Recovery (FDR) block, and the PGMF module, which utilizes priority-based serialization, we achieve state-of-the-art performance on the DroneVehicle and VEDAI datasets.