Wyckoff Diffusion (WyckoffDiff) is a novel generative model that takes into account the symmetry of crystal structures. Unlike existing generative models that model each atom without constraints on the position or element type of the atom, WyckoffDiff generates a symmetry-based crystal description using a representation that encodes all symmetries of the crystal structure. This is made possible within the framework of a discrete generative model by a newly designed neural network structure. The discrete nature of the model allows for fast generation, and it has the advantage of considering symmetry in the generation. We present a new evaluation metric, the Fr echet Wrenformer Distance, to capture the symmetry aspect of the generated material and perform a comparative evaluation with existing crystal generation models. As a proof-of-concept study, we find a new material below the convex hull of thermodynamic stability using WyckoffDiff.