This paper presents a new benchmark dataset, KatFish, and a detection model, KatFishNet, for detecting Korean texts generated by large-scale language models (LLMs). Unlike previous studies that mainly focused on English, we propose a text generation detection method that is suitable for Korean characteristics by considering Korean's unique morphological analysis, word order, and punctuation patterns. The KatFish dataset consists of human-written texts and four LLM-generated texts in three genres, and KatFishNet achieves an average of 19.78% higher AUROC performance than the previous best-performing models. We expect that the open code and data will contribute to the research on Korean LLM-generated text detection.