Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

Geo-Registration of Terrestrial LiDAR Point Clouds with Satellite Images without GNSS

Empowering Bridge Digital Twins by Bridging the Data Gap with a Unified Synthesis Framework

AI Agent Smart Contract Exploit Generation

ModelCitizens: Representing Community Voices in Online Safety

Sequential Attention-based Sampling for Histopathological Analysis

Classification of autoimmune diseases from Peripheral blood TCR repertoires by multimodal multi-instance learning

From Video to EEG: Adapting Joint Embedding Predictive Architecture to Uncover Visual Concepts in Brain Signal Analysis

Reinforcement Learning-based Feature Generation Algorithm for Scientific Data

RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programs

On Jailbreaking Quantized Language Models Through Fault Injection Attacks

PBa-LLM: Privacy- and Bias-aware NLP using Named-Entity Recognition (NER)

DriveMRP: Enhancing Vision-Language Models with Synthetic Motion Data for Motion Risk Prediction

Probing and Steering Evaluation Awareness of Language Models

Generating Heterogeneous Multi-dimensional Data: A Comparative Study

Autonomy by Design: Preserving Human Autonomy in AI Decision-Support

PBCAT: Patch-based composite adversarial training against physically realizable attacks on object detection

Federated Breast Cancer Detection Enhanced by Synthetic Ultrasound Image Augmentation

XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecs

DeepTalk: Towards Seamless and Smart Speech Interaction with Adaptive Modality-Specific MoE

SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications

LLM Agent for Hyper-Parameter Optimization

MedSyn: Enhancing Diagnostics with Human-AI Collaboration

Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directions

ConTextTab: A Semantics-Aware Tabular In-Context Learner

QUITE: A Query Rewrite System Beyond Rules with LLM Agents

Saffron-1: Safety Inference Scaling

Hdl2v: A Code Translation Dataset for Enhanced LLM Verilog Generation

Knockout LLM Assessment: Using Large Language Models for Evaluations through Iterative Pairwise Comparisons

Humanoid World Models: Open World Foundation Models for Humanoid Robotics

EMORL: Ensemble Multi-Objective Reinforcement Learning for Efficient and Flexible LLM Fine-Tuning

Adaptive Elicitation of Latent Information Using Natural Language

Substance over Style: Evaluating Proactive Conversational Coaching Agents

Real AI Agents with Fake Memories: Fatal Context Manipulation Attacks on Web3 Agents

Sparse Autoencoder as a Zero-Shot Classifier for Concept Erasing in Text-to-Image Diffusion Models

Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifacts

UniF$^2$ace: Fine-grained Face Understanding and Generation with Unified Multimodal Models

DynamicID: Zero-Shot Multi-ID Image Personalization with Flexible Facial Editability

GMLM: Bridging Graph Neural Networks and Language Models for Heterophilic Node Classification

Towards Enterprise-Ready Computer Using Generalist Agent

Oscillation-Reduced MXFP4 Training for Vision Transformers

Understanding Fixed Predictions via Confined Regions

Multi-Attribute Steering of Language Models via Targeted Intervention

Teaching LLMs According to Their Aptitude: Adaptive Reasoning for Mathematical Problem Solving

Theme-Explanation Structure for Table Summarization using Large Language Models: A Case Study on Korean Tabular Data

Scaling 4D Representations

AutoPrep: Natural Language Question-Aware Data Preparation with a Multi-Agent Framework

CHAI for LLMs: Improving Code-Mixed Translation in Large Language Models through Reinforcement Learning with AI Feedback

Vital Insight: Assisting Experts' Context-Driven Sensemaking of Multi-modal Personal Tracking Data Using Visualization and Human-In-The-Loop LLM

Hespi: A pipeline for automatically detecting information from hebarium specimen sheets

Diversifying Robot Locomotion Behaviors with Extrinsic Behavioral Curiosity

Pullback Flow Matching on Data Manifolds

The Impact of Generative AI on Collaborative Open-Source Software Development: Evidence from GitHub Copilot

Breaking PEFT Limitations: Leveraging Weak-to-Strong Knowledge Transfer for Backdoor Attacks in LLMs

Tokenization for Molecular Foundation Models

DilateQuant: Accurate and Efficient Diffusion Quantization via Weight Dilation

PersonaFlow: Designing LLM - Simulated Expert Perspectives for Enhanced Research Ideation

Disentangling Uncertainty for Safe Social Navigation using Deep Reinforcement Learning

CodeMirage: Hallucinations in Code Generated by Large Language Models

A Policy-Gradient Approach to Solving Imperfect-Information Games with Best-Iterate Convergence

Diffusion-Driven Semantic Communication for Generative Models with Bandwidth Constraints

Semantic Augmentation in Images using Language

Geometric Constraints in Deep Learning Frameworks: A Survey

Enhancing Plasticity for First Session Adaptation Continual Learning

Efficient Transfer Learning via Causal Bounds

FEVO: Financial Knowledge Expansion and Reasoning Evolution for Large Language Models

A Wireless Foundation Model for Multi-Task Prediction

GTA1: GUI Test-time Scaling Agent

Modeling (Deontic) Modal Operators With the s(CASP) Goal-directed Predicate Answer Set Programming System

Fuzzy Classification Aggregation for a Continuum of Agents

MedGellan: LLM-Generated Medical Guidance to Support Physicians

Animation Needs Attention: A Holistic Approach to Slides Animation Comprehension with Visual-Language Models

Establishing Best Practices for Building Rigorous Agentic Benchmarks

SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents

TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systems

The end of radical concept nativism

AI-Driven Scholarly Peer Review via Persistent Workflow Prompting, Meta-Prompting, and Meta-Reasoning

Do Larger Language Models Imply Better Generalization? A Pretraining Scaling Law for Implicit Reasoning

SagaLLM: Context Management, Validation, and Transaction Guarantees for Multi-Agent LLM Planning

Towards Reasoning Era: A Survey of Long Chain-of-Thoughts for Reasoning Large Language Models

ROCKET-2: Steering Visuomotor Policy via Cross-View Goal Alignment

Hybrid Quantum-Classical Multi-Agent Pathfinding

FinSphere, a Real-Time Stock Analysis Agent Powered by Instruction-Tuned LLMs and Domain Tools

Multi-Agent Pathfinding Under Team-Connected Communication Constraint via Adaptive Path Expansion and Dynamic Leading

Can adversarial attacks by large language models be attributed?

Planning Anything with Rigor: General-Purpose Zero-Shot Planning with LLM-based Formalized Programming

OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework

Stepwise functional refoundation of relational concept analysis

A Survey on Event Prediction Methods from a Systems Perspective: Bringing Together Disparate Research Areas

An AI Approach for Learning the Spectrum of the Laplace-Beltrami Operator

Latent Acoustic Mapping for Direction of Arrival Estimation: A Self-Supervised Approach

DeepRetro: Retrosynthetic Pathway Discovery using Iterative LLM Reasoning

Comparative Analysis of CNN and Transformer Architectures with Heart Cycle Normalization for Automated Phonocardiogram Classification

A Novel Hybrid Deep Learning Technique for Speech Emotion Detection using Feature Engineering

Advances in Intelligent Hearing Aids: Deep Learning Approaches to Selective Noise Cancellation

Modeling Heterogeneity across Varying Spatial Extents: Discovering Linkages between Sea Ice Retreat and Ice Shelve Melt in the Antarctic

Surrogate Model for Heat Transfer Prediction in Impinging Jet Arrays using Dynamic Inlet/Outlet and Flow Rate Control

PLAME: Leveraging Pretrained Language Models to Generate Enhanced Protein Multiple Sequence Alignments

Design and Implementation of an OCR-Powered Pipeline for Table Extraction from Invoices

FlexOlmo: Open Language Models for Flexible Data Use

Generating Multi-Table Time Series EHR from Latent Space with Minimal Preprocessing

Establishing Best Practices for Building Rigorous Agentic Benchmarks

Created by

Haebom

Author

Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Weiss, Fazl Barez, Rahul Gupta, Jwala Dhamala, Jacob Merizian, Mario Giulianelli, Harry Coppock, Cozmin Ududec, Jasjeet Sekhon, Jacob Steinhardt, Antony Kellerman, Sarah Schwettmann, Matei Zaharia, Ion Stoica, Percy Liang, Daniel Kang

Outline

This paper points out the problems of agent benchmarks for evaluating the performance of AI agents and presents the Agentic Benchmark Checklist (ABC), a guideline to solve them. It shows that many existing agent benchmarks can under- or overestimate agent performance by up to 100% due to problems in task setting or reward design. For example, SWE-bench Verified uses insufficient test cases, and TAU-bench considers empty answers as successful. ABC is created by synthesizing benchmark building experiences, best practice investigations, and previously reported problems, and is applied to CVE-Bench, which has a complex evaluation design, and shows the effect of reducing performance overestimation by 33%.

Takeaways, Limitations

•

Takeaways:

◦

We provide a checklist (ABC) that can contribute to improving the reliability of AI agent benchmarks.

◦

It clearly presents the problems with the design and evaluation methods of existing benchmarks.

◦

We present methods to minimize errors that may occur during the benchmark design and evaluation process.

◦

ABC can be used to improve the accuracy of AI agent performance evaluation.

•

Limitations:

◦

Further validation is needed to determine whether ABC is applicable to all types of agent benchmarks.

◦

Consideration should be given to any additional costs or time required to implement ABC.

◦

Further research is needed to determine the comprehensiveness of the ABC checklist and its effectiveness in practical application.

View PDF

Made with Slashpage