Daily Arxiv

This page organizes papers related to artificial intelligence published around the world.
This page is summarized using Google Gemini and is operated on a non-profit basis.
The copyright of the paper belongs to the author and the relevant institution. When sharing, simply cite the source.

LPI-RIT at LeWiDi-2025: Improving Distributional Predictions via Metadata and Loss Reweighting with DisCo

Created by
  • Haebom

Author

Mandira Sawkar, Samay U. Shetty, Deepak Pandita, Tharindu Cyril Weerasooriya, Christopher M. Homan

Outline

This paper presents improvements and analysis of the Distribution from Context (DisCo) architecture, developed as part of the "Learning With Disagreements (LeWiDi) 2025" collaborative effort, which leverages soft label distribution prediction and perspective-based evaluation to model inter-annotator disagreements. We extend DisCo to better capture discordant patterns by introducing annotator metadata embeddings, improved input representations, and a multi-objective training loss. Extensive experiments demonstrate significant improvements in both soft and perspective-based evaluation metrics, and we conduct in-depth calibration and error analysis to demonstrate when and why the improved discordance-aware modeling occurs. We also demonstrate that directly optimizing the distribution metric, conditioned on annotator demographics, can better capture disagreements.

Takeaways, Limitations

Takeaways:
Modeling using annotator metadata improved mismatch prediction performance.
Direct optimization of distribution metrics contributed to consistent performance improvements.
We provide an in-depth analysis to identify the strengths and weaknesses of mismatch awareness modeling.
Limitations:
This paper does not specifically mention Limitations. (It is difficult to identify Limitations based on the paper summary alone.)
👍