This pilot study explores the impact of the rapidly expanding use of generative AI (GAI) tools in educational settings on developing critical thinking skills. Unlike previous studies that have examined GAI as a subject-specific tutor or task completion tool, this study focuses on students' ability to critically evaluate the accuracy and appropriateness of GAI-generated responses. We designed structured critical thinking activities in an introductory computer and data science course that require students to analyze, critique, and revise GAI output, taking into account the fact that GAI often generates responses with contextual or factual errors. The findings provide initial insights into students' ability to critically engage with GAI content and lay the foundation for more comprehensive research in future semesters.