Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

That's So FETCH: Fashioning Ensemble Techniques for LLM Classification in Civil Legal Intake and Referral

Created by
  • Haebom

Author

Quinten Steenhuis

Outline

Every year, millions of people seek legal help through legal aid program hotlines, legal aid offices, or lawyer referral services. Identifying the legal issues an applicant is experiencing is the first step in connecting them with the right help. Misdirection can result in missed deadlines, physical abuse, loss of housing, or loss of child custody. This paper introduces and evaluates the FETCH classifier for legal issue classification and describes two methods to improve its accuracy: a hybrid LLM/ML ensemble classification method and automatic generation of follow-up questions that enrich the initial problem description. We use a novel dataset consisting of 419 real-world questions from nonprofit lawyer referral services. We demonstrate that using a combination of inexpensive models, we achieve a classification accuracy of 97.37% (hits@2), outperforming the current state-of-the-art GPT-5 model. Our approach demonstrates the potential to significantly reduce the cost of directing legal system users to appropriate resources for their issues while achieving high accuracy.

Takeaways, Limitations

•
Takeaways:
◦
We demonstrate that the accuracy of legal issue classification can be significantly improved by using a hybrid LLM/ML ensemble classification method and automatic generation of follow-up questions.
◦
We demonstrate the feasibility of building a cost-effective legal support system by achieving high accuracy (97.37% hits@2) using an inexpensive model.
◦
Validate the model's practicality and generalization performance using real-world datasets.
•
Limitations:
◦
The dataset used may be relatively small (419 cases).
◦
Because we used data from a specific nonprofit lawyer referral service, further research is needed to determine generalizability to other services or regions.
◦
The need for long-term performance maintenance and continuous updates of the model.
◦
Further validation of the comprehensive handling capabilities for various types of legal issues is needed.
👍
Made with Slashpage