This paper introduces VarCoNet, an improved self-supervised learning framework for robust functional connectome (FC) extraction from resting-state fMRI (rs-fMRI) data, which considers inter-individual variability in brain function as meaningful data. VarCoNet leverages self-supervised contrastive learning to exploit inherent functional inter-individual variability and acts as a brain feature encoder that generates FC embeddings that can be directly applied to downstream tasks without labeled data. VarCoNet facilitates contrastive learning through a novel augmentation strategy based on rs-fMRI signal segmentation, integrates a 1D-CNN-Transformer encoder for improved time-series processing, and employs robust Bayesian hyperparameter optimization. VarCoNet is evaluated on two downstream tasks: subject fingerprinting using rs-fMRI data from the Human Connectome Project and autism spectrum disorder (ASD) classification using rs-fMRI data from the ABIDE I and ABIDE II datasets. Extensive testing against state-of-the-art methodologies, including 13 deep learning methods, using various brain parcellations demonstrated the superiority, robustness, interpretability, and generalizability of VarCoNet.