Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

Automatic Prompt Optimization for Knowledge Graph Construction: Insights from an Empirical Study

Created by
  • Haebom

Author

Nandana Mihindukulasooriya, Niharika S. D'Souza, Faisal Chowdhury, Horst Samulowitz

Outline

This paper presents an experimental study that applies an automatic prompt optimization technique instead of manually writing prompts in knowledge graph (KG) construction using a large-scale language model (LLM). We focus on the fundamental task of extracting 3-tuples (subject-relation-object) from text, and compare the performance of three automatic prompt optimization techniques (DSPy, APE, and TextGrad) under various settings (prompting strategy, LLM model, schema complexity, input text length and diversity, optimization index, and dataset) using two datasets, SynthIE and REBEL. The experimental results show that the automatic prompt optimization technique achieves performance similar to that of human-written prompts, and that the performance improvement becomes more pronounced as schema complexity and text length increase.

Takeaways, Limitations

Takeaways:
We empirically demonstrate that the efficiency of knowledge graph construction using LLM can be improved through an automatic prompt optimization technique.
The effect of automatic prompt optimization becomes more significant as schema complexity and text length increase.
Presents the possibility of reducing the effort and cost of manually writing prompts.
Limitations:
These experimental results are limited to a specific automated prompt optimization technique and LLM dataset. Further research is needed to determine generalizability across diverse environments.
Additional research may be needed to improve the performance of the automatic prompt optimization technique itself.
Further validation is needed for applicability to various relationship types and complex sentence structures.
👍