In this paper, we propose SimAD (Simple dissimilarity-based approach for time series Anomaly Detection) to overcome the limitations of existing reconstruction-based deep learning methods in time series anomaly detection (limited temporal context, insufficient representation of normal patterns, incorrect evaluation metrics). SimAD integrates normal behavior patterns using patch-based feature extractor and EmbedPatch encoder that handle extended time windows, and enhances the robustness of anomaly detection by highlighting the distributional difference between normal and anomalous data through ContrastFusion module. In addition, we propose two enhanced evaluation metrics, Unbiased Affiliation (UAff) and Normalized Affiliation (NAff), to overcome the limitations of existing metrics. Experimental results on seven diverse temporal time series datasets show that SimAD outperforms existing state-of-the-art methods, achieving relative performance improvements of 19.85% in F1, 4.44% in Aff-F1, 77.79% in NAff-F1, and 9.69% in AUC on six multivariate datasets. The code and pre-trained models are available on Github.