Daily Arxiv

This page organizes papers related to artificial intelligence published around the world.
This page is summarized using Google Gemini and is operated on a non-profit basis.
The copyright of the paper belongs to the author and the relevant institution. When sharing, simply cite the source.

LLM Agents for Interactive Workflow Provenance: Reference Architecture and Evaluation Methodology

Created by
  • Haebom

Author

Renan Souza, Timothy Poteet, Brian Etz, Daniel Rosendo, Amal Gueroudji, Woong Shin, Prasanna Balaprakash, Rafael Ferreira da Silva

Outline

This paper presents a novel approach for analyzing massive amounts of provenance data generated from scientific data processing workflows spanning edge, cloud, and HPC environments. To overcome the limitations of existing approaches, which rely on custom scripts, structured queries, and static dashboards for data interaction, we propose a runtime data analysis methodology, reference architecture, and open-source implementation leveraging an interactive Large Language Model (LLM) agent. A lightweight, metadata-driven design transforms natural language into structured provenance queries. Evaluations on real-world chemistry workflows using various LLMs, including LLaMA, GPT, Gemini, and Claude, demonstrate that the modular design, prompt tuning, and Retrieval-Augmented Generation (RAG) enable accurate and insightful responses beyond recorded provenance.

Takeaways, Limitations

Takeaways:
We present a novel methodology that simplifies complex provenance data analysis by leveraging interactive LLM agents.
Enables non-expert users to easily analyze provenance data through natural language-based queries.
We demonstrate that modular design, prompt tuning, and RAG can improve the accuracy and efficiency of LLM.
Accessibility and reproducibility achieved through open source implementation.
Limitations:
Because these are evaluation results for a specific LLM and workflow, generalizability to other LLMs or workflows may be limited.
Since it depends on the performance of LLM, limitations of LLM (hallucination, etc.) may affect the accuracy of the results.
Because the appropriateness of metadata design affects the quality of analysis results, further research on metadata design may be necessary.
👍