Daily Arxiv

전 세계에서 발간되는 인공지능 관련 논문을 정리하는 페이지 입니다.
본 페이지는 Google Gemini를 활용해 요약 정리하며, 비영리로 운영 됩니다.
논문에 대한 저작권은 저자 및 해당 기관에 있으며, 공유 시 출처만 명기하면 됩니다.

Maximizing Confidence Alone Improves Reasoning

Pre-training for Recommendation Unlearning

FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control

On the performance of machine-learning-assisted Monte Carlo in sampling from simple statistical physics models

Agent-UniRAG: A Trainable Open-Source LLM Agent Framework for Unified Retrieval-Augmented Generation Systems

Topological Structure Learning Should Be A Research Priority for LLM-Based Multi-Agent Systems

Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning

SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splitting

Skywork Open Reasoner 1 Technical Report

Speculative Decoding Meets Quantization: Compatibility Evaluation and Hierarchical Framework Design

DORAEMON: Decentralized Ontology-aware Reliable Agent with Enhanced Memory Oriented Navigation

Cross-modal RAG: Sub-dimensional Retrieval-Augmented Text-to-Image Generation

CAST: Contrastive Adaptation and Distillation for Semi-Supervised Instance Segmentation

Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modeling

Is Attention Required for Transformer Inference? Explore Function-preserving Attention Replacement

VietASR: Achieving Industry-level Vietnamese ASR with 50-hour labeled data and Large-Scale Speech Pretraining

Hume: Introducing System-2 Thinking in Visual-Language-Action Model

DeSocial: Blockchain-based Decentralized Social Networks

Subgroups Matter for Robust Bias Mitigation

Hybrid Disagreement-Diversity Active Learning for Bioacoustic Sound Event Detection

Automatic Transmission for LLM Tiers: Optimizing Cost and Accuracy in Large Language Models

The challenge of hidden gifts in multi-agent reinforcement learning

Retrieval Visual Contrastive Decoding to Mitigate Object Hallucinations in Large Vision-Language Models

Risk-aware Direct Preference Optimization under Nested Risk Measure

Surrogate-Assisted Evolutionary Reinforcement Learning Based on Autoencoder and Hyperbolic Neural Network

A Novel Zero-Trust Identity Framework for Agentic AI: Decentralized Authentication and Fine-Grained Access Control

BroadGen: A Framework for Generating Effective and Efficient Advertiser Broad Match Keyphrase Recommendations

Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effects

How We Won the ISLES'24 Challenge by Preprocessing

SP2RINT: Spatially-Decoupled Physics-Inspired Progressive Inverse Optimization for Scalable, PDE-Constrained Meta-Optical Neural Network Training

Seek-CAD: A Self-refined Generative Modeling for 3D Parametric CAD Using Local Inference via DeepSeek

Rethinking the Sampling Criteria in Reinforcement Learning for LLM Reasoning: A Competence-Difficulty Alignment Perspective

OrionBench: A Benchmark for Chart and Human-Recognizable Object Detection in Infographics

EarthSE: A Benchmark Evaluating Earth Scientific Exploration Capability for Large Language Models

CASS: Nvidia to AMD Transpilation with Data, Models, and Benchmark

Beyond Face Swapping: A Diffusion-Based Digital Human Benchmark for Multimodal Deepfake Detection

Edge-First Language Model Inference: Models, Metrics, and Tradeoffs

Smaller, Smarter, Closer: The Edge of Collaborative Generative AI

YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering

DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models

Articulatory Feature Prediction from Surface EMG during Speech Production

Exploring Spatiotemporal Emotional Synchrony in Dyadic Interactions: The Role of Speech Conditions in Facial and Vocal Affective Alignment

LEXam: Benchmarking Legal Reasoning on 340 Law Exams

Neural Networks as Universal Finite-State Machines: A Constructive Deterministic Finite Automaton Theory

The Geometry of ReLU Networks through the ReLU Transition Graph

RepCali: High Efficient Fine-tuning Via Representation Calibration in Latent Space for Pre-trained Language Models

Fusing Bidirectional Chains of Thought and Reward Mechanisms A Method for Enhancing Question-Answering Capabilities of Large Language Models for Chinese Intangible Cultural Heritage

Multimodal Survival Modeling in the Age of Foundation Models

Burger: Robust Graph Denoising-augmentation Fusion and Multi-semantic Modeling in Social Recommendation

The Aloe Family Recipe for Open and Specialized Healthcare LLMs

To Judge or not to Judge: Using LLM Judgements for Advertiser Keyphrase Relevance at eBay

LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection

CoordField: Coordination Field for Agentic UAV Task Allocation In Low-altitude Urban Scenarios

Learning to Reason under Off-Policy Guidance

A Combinatorial Theory of Dropout: Subnetworks, Graph Geometry, and Generalization

Error Broadcast and Decorrelation as a Potential Artificial and Natural Learning Mechanism

Carbon-Efficient 3D DNN Acceleration: Optimizing Performance and Sustainability

Agentic Knowledgeable Self-awareness

Are Domain Generalization Benchmarks with Accuracy on the Line Misspecified?

MiZero: The Shadowy Defender Against Text Style Infringements

Firm or Fickle? Evaluating Large Language Models Consistency in Sequential Interactions

LoTUS: Large-Scale Machine Unlearning with a Taste of Uncertainty

Temporal Relation Extraction in Clinical Texts: A Span-based Graph Transformer Approach

LEAVS: An LLM-based Labeler for Abdominal CT Supervision

From Head to Tail: Towards Balanced Representation in Large Vision-Language Models through Adaptive Data Calibration

Enhancing Retrieval for ESGLLM via ESG-CID -- A Disclosure Content Index Finetuning Dataset for Mapping GRI and ESRS

DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generation

Aligning Text to Image in Diffusion Models is Easier Than You Think

Can LLMs Reason About Program Semantics? A Comprehensive Evaluation of LLMs on Formal Specification Inference

BatteryLife: A Comprehensive Dataset and Benchmark for Battery Life Prediction

Bridging Critical Gaps in Convergent Learning: How Representational Alignment Evolves Across Layers, Training, and Distribution Shifts

ExpandR: Teaching Dense Retrievers Beyond Queries with LLM Guidance

Audio Visual Segmentation Through Text Embeddings

Privacy-Aware Joint DNN Model Deployment and Partitioning Optimization for Collaborative Edge Inference Services

Learning to Reason from Feedback at Test-Time

ParamMute: Suppressing Knowledge-Critical FFNs for Faithful Retrieval-Augmented Generation

STeCa: Step-level Trajectory Calibration for LLM Agent Learning

GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuning

Enhancing Semi-supervised Learning with Zero-shot Pseudolabels

DELMAN: Dynamic Defense Against Large Language Model Jailbreaking with Model Editing

Multilingual Encoder Knows more than You Realize: Shared Weights Pretraining for Extremely Low-Resource Languages

Jailbreaking to Jailbreak

Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning

Position: Scaling LLM Agents Requires Asymptotic Analysis with LLM Primitives

CodeSteer: Symbolic-Augmented Language Models via Code/Text Guidance

Toward universal steering and monitoring of AI models

SPRI: Aligning Large Language Models with Context-Situated Principles

MedRAX: Medical Reasoning Agent for Chest X-ray

Adaptive Exploration for Multi-Reward Multi-Policy Evaluation

Wake-Informed 3D Path Planning for Autonomous Underwater Vehicles Using A* and Neural Network Approximations

Fast Large Language Model Collaborative Decoding via Speculation

A Statistical Learning Perspective on Semi-dual Adversarial Neural Optimal Transport Solvers

Joint Localization and Activation Editing for Low-Resource Fine-Tuning

KBQA-o1: Agentic Knowledge Base Question Answering with Monte Carlo Tree Search

Towards Unified Attribution in Explainable AI, Data-Centric AI, and Mechanistic Interpretability

Chain of Grounded Objectives: Bridging Process and Goal-oriented Prompting for Code Generation

Re-ranking Using Large Language Models for Mitigating Exposure to Harmful Content on Social Media Platforms

Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment

Tensor Product Attention Is All You Need

VideoRAG: Retrieval-Augmented Generation over Video Corpus

MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections

Created by

Haebom

저자

Da Xiao, Qingye Meng, Shengping Li, Xingyuan Yuan

개요

본 논문은 Transformer의 잔차 연결(residual connection) 한계를 극복하고 계층 간 정보 흐름을 향상시키는 간단하면서도 효과적인 방법인 MUltiway Dynamic Dense (MUDD) 연결을 제안합니다. 기존의 정적이고 공유된 연결 가중치를 사용하는 밀집 연결 방식과 달리, MUDD는 각 시퀀스 위치와 Transformer 블록의 각 분리된 입력 스트림(쿼리, 키, 값 또는 잔차)에 따라 동적으로 연결 가중치를 생성합니다. MUDD 연결은 어떤 Transformer 아키텍처에도 통합되어 MUDDFormer를 만들 수 있습니다. 광범위한 실험 결과, MUDDFormer는 다양한 모델 아키텍처와 규모에서 언어 모델링에서 Transformer를 크게 능가하며, 1.8배에서 2.4배의 계산량으로 훈련된 Transformer의 성능을 달성합니다. 특히, MUDDPythia-2.8B는 사전 훈련 ppl 및 다운스트림 작업에서 Pythia-6.9B와 일치하며, 5-shot 설정에서는 Pythia-12B와 경쟁하며, 매개변수는 0.23%만, 계산량은 0.4%만 증가시킵니다. JAX와 PyTorch 코드 및 사전 훈련된 모델은 https://github.com/Caiyun-AI/MUDDFormer 에서 이용 가능합니다.

GitHub - Caiyun-AI/MUDDFormer

Contribute to Caiyun-AI/MUDDFormer development by creating an account on GitHub.

시사점, 한계점

•

시사점:

◦

Transformer의 성능을 향상시키는 새로운 방법인 MUDD 연결 제안.

◦

기존 Transformer보다 적은 계산량으로 동등하거나 더 나은 성능 달성. (1.8X-2.4X compute 감소)

◦

소량의 매개변수 추가로 큰 성능 향상 (MUDDPythia-2.8B vs Pythia-6.9B, 12B)

◦

다양한 Transformer 아키텍처와 규모에서 적용 가능.

◦

JAX와 PyTorch 코드 및 사전 훈련된 모델 공개.

•

한계점:

◦

MUDD 연결의 동적 가중치 생성이 계산 비용에 미치는 영향에 대한 자세한 분석 부족.

◦

다른 언어 모델링 작업이나 다른 유형의 데이터에 대한 일반화 성능에 대한 추가 연구 필요.

◦

극도로 큰 모델에 대한 확장성에 대한 추가적인 평가 필요.

Made with Slashpage