This paper points out the shortcomings of modern CNNs and proposes a novel neural network architecture, the Visual Cortex Network (VCNet), that mimics the structure and computational principles of the primate visual cortex. VCNet geometrically interprets key biological mechanisms, such as hierarchical processing, dual-stream information separation, and bottom-up predictive feedback, to induce the learning of a structured, low-dimensional neural manifold. We evaluate the performance of VCNet on the Spots-10 animal pattern dataset and the optical field image classification task, and find that it achieves accuracy superior to existing models.