This paper discusses recent advances in the application of Transformer-based language models, which have achieved remarkable results in natural language processing (NLP), to the field of bioinformatics. In particular, it focuses on protein sequence analysis and design, and analyzes and presents strengths and weaknesses of studies in various application areas such as gene ontology, functional and structural protein identification, novel protein production, and protein binding. It points out the shortcomings of existing studies and suggests future research directions, and aims to help researchers in the field to grasp the latest research trends and design future research.