Daily Arxiv

전 세계에서 발간되는 인공지능 관련 논문을 정리하는 페이지 입니다.
본 페이지는 Google Gemini를 활용해 요약 정리하며, 비영리로 운영 됩니다.
논문에 대한 저작권은 저자 및 해당 기관에 있으며, 공유 시 출처만 명기하면 됩니다.

Exploring Diffusion Transformer Designs via Grafting

Teaming in the AI Era: AI-Augmented Frameworks for Forming, Simulating, and Optimizing Human Teams

ECoRAG: Evidentiality-guided Compression for Long Context RAG

Dissecting Bias in LLMs: A Mechanistic Interpretability Perspective

Does It Make Sense to Speak of Introspection in Large Language Models?

Sparse Autoencoders, Again?

Feature-Based Lie Group Transformer for Real-World Applications

TracLLM: A Generic Framework for Attributing Long Context LLMs

SemiOccam: A Robust Semi-Supervised Image Recognition Network Using Sparse Labels

Labelling Data with Unknown References

Tug-of-war between idiom's figurative and literal meanings in LLMs

State-Covering Trajectory Stitching for Diffusion Planners

Deep Learning Weather Models for Subregional Ocean Forecasting: A Case Study on the Canary Current Upwelling System

DORAEMON: Decentralized Ontology-aware Reliable Agent with Enhanced Memory Oriented Navigation

Subspecialty-Specific Foundation Model for Intelligent Gastrointestinal Pathology

RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving

An Uncertainty-Aware ED-LSTM for Probabilistic Suffix Prediction

SageAttention2++: A More Efficient Implementation of SageAttention2

Autocomp: LLM-Driven Code Optimization for Tensor Accelerators

IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysis

Decoupling Representation and Learning in Genetic Programming: the LaSER Approach

Common Data Format (CDF): A Standardized Format for Match-Data in Football (Soccer)

Web Intellectual Property at Risk: Preventing Unauthorized Real-Time Retrieval by Large Language Models

How can Diffusion Models Evolve into Continual Generators?

Open Your Eyes: Vision Enhances Message Passing Neural Networks in Link Prediction

m-KAILIN: Knowledge-Driven Agentic Scientific Corpus Distillation Framework for Biomedical Large Language Models Training

FinSage: A Multi-aspect RAG System for Financial Filings Question Answering

Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning

LauraTSE: Target Speaker Extraction using Auto-Regressive Decoder-Only Language Models

Reasoning Towards Fairness: Mitigating Bias in Language Models through Reasoning-Guided Fine-Tuning

Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models

Multivariate Temporal Regression at Scale: A Three-Pillar Framework Combining ML, XAI, and NLP

GENIUS: A Generative Framework for Universal Multimodal Search

TinySQL: A Progressive Text-to-SQL Dataset for Mechanistic Interpretability Research

ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability

A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models

Knowledge Retention for Continual Model-Based Reinforcement Learning

Adversarial Tokenization

UDora: A Unified Red Teaming Framework against LLM Agents by Dynamically Hijacking Their Own Reasoning

SAGE: A Framework of Precise Retrieval for RAG

Graph Attention Networks Unleashed: A Fast and Explainable Vulnerability Assessment Framework for Microgrids

SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models

Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Models

Improving Customer Service with Automatic Topic Detection in User Emails

DualSpec: Text-to-spatial-audio Generation via Dual-Spectrogram Guided Diffusion Model

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations

SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference

MimeQA: Towards Socially-Intelligent Nonverbal Foundation Models

A Comprehensive Survey on Concept Erasure in Text-to-Image Diffusion Models

LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws

Maximum Entropy Reinforcement Learning with Diffusion Policy

TituLLMs: A Family of Bangla LLMs with Comprehensive Benchmarking

Relational Conformal Prediction for Correlated Time Series

RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models

UniDB: A Unified Diffusion Bridge Framework via Stochastic Optimal Control

The Complexity of Learning Sparse Superposed Features with Feedback

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture

Rollout Roulette: A Probabilistic Inference Approach to Inference-Time Scaling of LLMs using Particle-Based Monte Carlo Methods

An Optimal Cascade Feature-Level Spatiotemporal Fusion Strategy for Anomaly Detection in CAN Bus

ProofAug: Efficient Neural Theorem Proving via Fine-grained Proof Structure Analysis

FDLLM: A Dedicated Detector for Black-Box LLMs Fingerprinting

The Bakers and Millers Game with Restricted Locations

Diving into Self-Evolving Training for Multimodal Reasoning

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation

A Riemannian Optimization Perspective of the Gauss-Newton Method for Feedforward Neural Networks

CoopetitiveV: Leveraging LLM-powered Coopetitive Multi-Agent Prompting for High-quality Verilog Generation

TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies

The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Data

Understanding Memorization in Generative Models via Sharpness in Probability Landscapes

A Cognac shot to forget bad memories: Corrective Unlearning in GNNs

Fundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiency

Who Can Withstand Chat-Audio Attacks? An Evaluation Benchmark for Large Audio-Language Models

CLIPErase: Efficient Unlearning of Visual-Textual Associations in CLIP

The Impact of Inference Acceleration on Bias of LLMs

pLDDT-Predictor: High-speed Protein Screening Using Transformer and ESM2

Simmering: Sufficient is better than optimal for training neural networks

PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoning

Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation

AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML

VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasters

Deconfounding Multi-Cause Latent Confounders: A Factor-Model Approach to Climate Model Bias Correction

DyGMamba: Efficiently Modeling Long-Term Temporal Dependency on Continuous-Time Dynamic Graphs with State Space Models

Proximal Policy Distillation

BoA: Attention-aware Post-training Quantization without Backpropagation

Certification for Differentially Private Prediction in Gradient-Based Training

Multi-Agent Collaboration via Cross-Team Orchestration

LlavaGuard: An Open VLM-based Framework for Safeguarding Vision Datasets and Models

Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Models

Boolean matrix logic programming for active learning of gene functions in genome-scale metabolic network models

Mirage: A Multi-Level Superoptimizer for Tensor Programs

Multidimensional Adaptive Coefficient for Inference Trajectory Optimization in Flow and Diffusion

Longitudinal Targeted Minimum Loss-based Estimation with Temporal-Difference Heterogeneous Transformer

TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languages

Structure Guided Large Language Model for SQL Generation

GraphGPT: Generative Pre-trained Graph Eulerian Transformer

Graph Deep Learning for Time Series Forecasting

Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning

Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning

Rethinking Machine Unlearning in Image Generation Models

The Coming Crisis of Multi-Agent Misalignment: AI Alignment Must Be a Dynamic and Social Process

Fundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiency

Created by

Haebom

저자

Jerry Yao-Chieh Hu, Wei-Po Wang, Ammar Gilani, Chenyang Li, Zhao Song, Han Liu

개요

본 논문은 Transformer 기반 기초 모델에서 프롬프트 튜닝의 통계적 및 계산적 한계를 조사합니다. 단일 헤드, 단일 셀프 어텐션 레이어를 가진 Transformer에서의 프롬프트 튜닝이 (i) 보편적이며 (ii) 강한 지수 시간 가설(SETH) 하에서 효율적인 (거의 선형 시간) 알고리즘을 지원함을 보여줍니다. 통계적으로, 이러한 가장 단순한 Transformer에서의 프롬프트 튜닝이 시퀀스-투-시퀀스 Lipschitz 함수에 대한 보편적 근사기임을 증명합니다. 또한, 1-레이어, 1-헤드 Transformer로 어떤 데이터셋이라도 암기하기 위해 필요한 소프트 프롬프트 토큰에 대한 지수적($dL$ 및 $(1/\epsilon)$) 하한을 제공합니다. 계산적으로, 소프트 프롬프트에 의해 유도된 키와 쿼리의 노름에 의해 결정되는 프롬프트 튜닝 효율성의 상전이를 확인하고, 상한 기준을 제공합니다. 이 기준을 넘어서는 경우, SETH 하에서 프롬프트 튜닝을 위한 하위 2차(효율적인) 알고리즘은 존재하지 않습니다. 이 기준 내에서, 거의 선형 시간 프롬프트 튜닝 추론 알고리즘의 존재를 증명함으로써 이론을 보여줍니다. 이러한 기본적인 한계는 실무자를 위한 표현력 있고 효율적인 프롬프트 튜닝 방법을 설계하기 위한 중요한 필요 조건을 제공합니다.

시사점, 한계점

•

시사점:

◦

단일 헤드, 단일 셀프 어텐션 레이어 Transformer에서의 프롬프트 튜닝이 보편적이고 효율적인 알고리즘을 지원할 수 있음을 밝힘.

◦

소프트 프롬프트 토큰 수의 하한과 프롬프트 튜닝 효율성의 상전이 기준을 제시하여 효율적인 프롬프트 튜닝 방법 설계에 필요한 조건을 제시함.

◦

거의 선형 시간 프롬프트 튜닝 추론 알고리즘의 존재를 증명.

•

한계점:

◦

제시된 하한과 상한 기준은 단일 헤드, 단일 셀프 어텐션 레이어 Transformer에 국한됨. 실제 응용에 사용되는 복잡한 Transformer에 대한 일반화는 추가 연구가 필요.

◦

SETH 가정에 기반한 계산적 한계 분석은 실제 환경에서의 성능을 완벽하게 반영하지 못할 수 있음.

◦

증명된 거의 선형 시간 알고리즘의 실제 구현 및 성능 평가는 추가 연구가 필요.

Made with Slashpage