This pilot study investigated the impact of generative AI (GAI), a rapidly expanding tool in educational settings, on developing critical thinking skills. Unlike previous studies that have examined GAI as a tool for tutoring specific subjects or completing assignments, this study focuses on students' ability to critically evaluate the accuracy and appropriateness of GAI-generated answers. Targeting students in an introductory computer and data science course, we designed learning activities that required them to analyze, critique, and revise AI-generated solutions, given that GAI often generates contextual errors or factual errors. The findings provide initial insights into students' ability to critically engage with GAI content and lay the foundation for more comprehensive research in future semesters.