Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

MAFA: A multi-agent framework for annotation

Created by
  • Haebom

Author

Mahmood Hegazy, Aaron Rodrigues, Azzam Naeem

Outline

This paper presents a FAQ annotation method using a multi-agent framework to capture the nuances of diverse user questions. It consists of multiple expert agents using a structured inference approach inspired by Attentive Reasoning Queries (ARQs) and a decision agent that re-ranks candidates. Each agent receives a different set of examples, employing a strategy that enhances ensemble diversity and query space coverage. We demonstrate significant performance improvements over existing single-agent approaches on real-world banking datasets and public benchmark datasets (LCQMC and FiQA), demonstrating its effectiveness in handling ambiguous questions.

Takeaways, Limitations

Takeaways:
We achieved improved FAQ annotation performance compared to single-model approaches using a multi-agent framework (up 14% in Top-1 accuracy, 18% in Top-5 accuracy, and 12% in average reverse ranking).
Its excellent ability to handle ambiguous questions makes it suitable for application in real-world banking applications.
It has excellent generalization ability to other domains and languages.
We demonstrate the utility of a structured inference approach based on ARQs.
Limitations:
There may be a lack of concrete implementations of the proposed multi-agent framework and detailed algorithmic descriptions of each agent.
Information on the specifics and size of the banking dataset used is limited.
There may be a lack of detailed information on assessing generalizability across languages.
Additional evaluation of performance and scalability in a real-world deployment environment may be required.
👍