Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

AutoPDL: Automatic Prompt Optimization for LLM Agents

Created by
  • Haebom

Author

Claudio Spiess, Mandana Vaziri, Louis Mandel, Martin Hirzel

Outline

This paper proposes AutoPDL, an automated prompt optimization technique for improving the performance of large-scale language models (LLMs). AutoPDL defines the problem of combining various prompting patterns (e.g., Zero-Shot, CoT, ReAct, ReWOO) with prompt content, including several examples, as a structured AutoML problem. It efficiently finds optimal prompt configurations using the Successive Halving technique. Leveraging a library of prompting patterns implemented using the PDL prompt programming language, AutoPDL generates human-readable, editable, and executable PDL programs. Evaluation results on three tasks and seven LLMs (ranging from 3 billion to 70 billion parameters) demonstrate an average accuracy improvement of 9.21±15.46 percentage points (up to 67.5 percentage points). The selected prompting strategy varies across models and tasks.

Takeaways, Limitations

•
Takeaways:
◦
LLM's prompt engineering automation demonstrates how to save time and effort and achieve better performance than manual methods.
◦
Prompt programming using PDL allows for the creation of reusable prompts that are easy for humans to understand and modify.
◦
We present a general prompt optimization methodology applicable to a variety of LLMs and tasks.
◦
We experimentally demonstrate that the optimal prompting strategy varies depending on the model and task.
•
Limitations:
◦
The effectiveness of the Successive Halving technique can vary depending on the size of the search space. For very large search spaces, it's possible that the optimal prompt may not be found.
◦
Because the types of LLMs and tasks used in the evaluation were limited, further research is needed to determine generalizability to other LLMs or tasks.
◦
Learning about the PDL prompt programming language may be required.
👍
Made with Slashpage