Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

LLM Agents for Interactive Workflow Provenance: Reference Architecture and Evaluation Methodology

Created by
  • Haebom

Author

Renan Souza, Timothy Poteet, Brian Etz, Daniel Rosendo, Amal Gueroudji, Woong Shin, Prasanna Balaprakash, Rafael Ferreira da Silva

Outline

This paper presents an interactive data analysis methodology, reference architecture, and open-source implementation based on large-scale language model (LLM) agents to address the challenges of analyzing massive provenance data generated from scientific data processing workflows spanning edge, cloud, and HPC environments. A lightweight, metadata-driven design transforms natural language questions into structured provenance queries, and the methodology is evaluated against a real-world chemistry workflow using various LLMs, including LLaMA, GPT, Gemini, and Claude. We demonstrate that the modular design, prompt tuning, and Retrieval-Augmented Generation (RAG) approach enable accurate and insightful responses beyond the historical provenance data.

Takeaways, Limitations

Takeaways:
A new methodology is presented to effectively analyze complex provenance data using LLM agents.
Providing a natural language-based interactive data analysis environment that allows even non-expert users to easily utilize provenance data.
Improved LLM performance and accuracy through modular design, prompt tuning, and RAG techniques.
Open source implementation allows other researchers to utilize and develop it.
Limitations:
As these results are based on a specific LLM and workflow, further research is needed to determine generalizability.
Depends on the performance of LLM, and limitations of LLM (hallucination, etc.) may affect the results.
Scalability verification is needed for various types of provenance data and workflows.
Consideration needs to be given to the processing speed and resource consumption of LLM.
👍