Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

IPBench: Benchmarking the Knowledge of Large Language Models in Intellectual Property

Created by
  • Haebom

Author

Qiyao Wang, Guhong Chen, Hongbo Wang, Huaren Liu, Minghui Zhu, Zhifei Qin, Linwei Li, Yilin Yue, Shiqiang Wang, Jiayan Li, Yihang Wu, Ziqiang Liu, Longze Chen, Run Luo, Liyang Fan, Jiaming Li, Lei Zhang, Kan Xu, Hongfei Lin, Hamid Alinejad-Rokny, Shiwen Ni, Yuan Lin, Min Yang

Outline

Considering the complexity and knowledge intensity of the intellectual property (IP) domain, this paper presents the possibility of improving the efficiency of IP-related task processing using large-scale language models (LLMs). To overcome the limitations of existing datasets and benchmarks that focus only on patents or cover only limited aspects of the IP domain, we introduce a comprehensive IP task taxonomy including eight IP mechanisms and 20 tasks and a large-scale bilingual benchmark, IPBench. IPBench is designed to evaluate the understanding and generation capabilities of LLMs in real-world IP applications. The benchmark results of 16 LLMs show that even the best-performing model only achieves 75.8% accuracy, indicating significant room for improvement. In particular, open-source IP and legal-related models are found to underperform closed-source general-purpose models. In this paper, we will release all data and code of IPBench and continuously update additional IP-related tasks to better reflect real-world IP challenges.

Takeaways, Limitations

•
Takeaways:
◦
Provides a comprehensive IP task classification system and benchmark (IPBench) reflecting real-world IP applications.
◦
Assessing and comparing the performance of various LLMs in performing IP-related tasks.
◦
Establishing standards for improving open source models and suggesting future research directions.
◦
Presenting the possibilities and limitations of utilizing LLM in the IP field.
•
Limitations:
◦
The current benchmark accuracy is 75.8%, leaving significant room for improvement.
◦
Open source IP and legal models perform worse than closed source general-purpose models.
◦
The types of IP mechanisms and operations included in the benchmark may not fully cover all real-world IP-related problems.
👍
Made with Slashpage