This paper addresses knowledge editing, which corrects outdated or inaccurate knowledge in neural networks. Instead of manually labeled factual triples used in previous studies, we explore knowledge editing using easily accessible documents. To this end, we construct DocTER, the first evaluation benchmark consisting of documents containing semi-real knowledge. We perform a comprehensive evaluation in four aspects: editing success rate, locality, inference, and cross-language transfer. To apply existing triple-based knowledge editing methods to this task, we develop an Extract-then-Edit pipeline that extracts triples from documents and then applies existing methods. Experiments on several knowledge editing methods show that editing using documents is significantly more difficult than using triples. In document-based scenarios, even the best-performing in-context editing method lags behind the gold triples in editing success rate by 10 points. This observation also holds true for inference and cross-language test sets. We analyze the key factors that affect the performance of the task, including the quality of the extracted triples, the frequency and location of the edited knowledge in the documents, various methods to improve inference, and the performance differences according to different directions of cross-language knowledge editing, providing valuable insights for future research.
