Daily Arxiv

전 세계에서 발간되는 인공지능 관련 논문을 정리하는 페이지 입니다.
본 페이지는 Google Gemini를 활용해 요약 정리하며, 비영리로 운영 됩니다.
논문에 대한 저작권은 저자 및 해당 기관에 있으며, 공유 시 출처만 명기하면 됩니다.

Aggregating Concepts of Accuracy and Fairness in Prediction Algorithms

Self-cross Feature based Spiking Neural Networks for Efficient Few-shot Learning

Graph neural networks and MSO

Generative Pre-trained Autoregressive Diffusion Transformer

SAS-Bench: A Fine-Grained Benchmark for Evaluating Short Answer Scoring with Large Language Models

Towards user-centered interactive medical image segmentation in VR with an assistive AI agent

Towards Scalable IoT Deployment for Visual Anomaly Detection via Efficient Compression

X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real

UniVLA: Learning to Act Anywhere with Task-centric Latent Actions

Assessing Tenstorrent's RISC-V MatMul Acceleration Capabilities

Examining the Source of Defects from a Mechanical Perspective for 3D Anomaly Detection

Understanding In-context Learning of Addition via Activation Subspaces

Risk-sensitive Reinforcement Learning Based on Convex Scoring Functions

Intelligently Augmented Contrastive Tensor Factorization: Empowering Multi-dimensional Time Series Classification in Low-Data Environments

Modeling Human Behavior in a Strategic Network Game with Complex Group Dynamics

GPU Performance Portability needs Autotuning

RM-R1: Reward Modeling as Reasoning

Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environments

Data-Driven Calibration of Prediction Sets in Large Vision-Language Models Based on Inductive Conformal Prediction

WildFireCan-MMD: A Multimodal Dataset for Classification of User-Generated Content During Wildfires in Canada

CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectives

Optimizing Power Grid Topologies with Reinforcement Learning: A Survey of Methods and Challenges

Flexible Graph Similarity Computation With A Proactive Optimization Strategy

CryoSAMU: Enhancing 3D Cryo-EM Density Maps of Protein Structures at Intermediate Resolution with Structure-Aware Multimodal U-Nets

PromptMobile: Efficient Promptus for Low Bandwidth Mobile Video Streaming

Implicit Bias-Like Patterns in Reasoning Models

EMMOE: A Comprehensive Benchmark for Embodied Mobile Manipulation in Open Environments

Temporal Triplane Transformers as Occupancy World Models

The Lazy Student's Dream: ChatGPT Passing an Engineering Course on Its Own

Atlas: A Framework for ML Lifecycle Provenance & Transparency

Centrally Coordinated Multi-Agent Reinforcement Learning for Power Grid Topology Control

Illegal Waste Detection in Remote Sensing Images: A Case Study

ARR: Question Answering with Large Language Models via Analyzing, Retrieving, and Reasoning

ImprovNet -- Generating Controllable Musical Improvisations with Iterative Corruption Refinement

Lightspeed Geometric Dataset Distance via Sliced Optimal Transport

Benchmarking Generative AI for Scoring Medical Student Interviews in Objective Structured Clinical Examinations (OSCEs)

Representation Convergence: Mutual Distillation is Secretly a Form of Regularization

Goal-Conditioned Supervised Learning for Multi-Objective Recommendation

Not All Adapters Matter: Selective Adapter Freezing for Memory-Efficient Fine-Tuning of Language Models

Simple and Provable Scaling Laws for the Test-Time Compute of Large Language Models

KBAlign: Efficient Self Adaptation on Specific Knowledge Bases

Natural Language Reinforcement Learning

Translating Electrocardiograms to Cardiac Magnetic Resonance Imaging Useful for Cardiac Assessment and Disease Screening: A Multi-Center Study AI for ECG to CMR Translation Study

TSINR: Capturing Temporal Continuity via Implicit Neural Representations for Time Series Anomaly Detection

Towards Graph Foundation Models: Training on Knowledge Graphs Enables Transferability to General Graphs

Temporal-Difference Variational Continual Learning

FAMMA: A Benchmark for Financial Domain Multilingual Multimodal Question Answering

Time Awareness in Large Language Models: Benchmarking Fact Recall Across Time

Self-supervised Learning for Acoustic Few-Shot Classification

Double Successive Over-Relaxation Q-Learning with an Extension to Deep Reinforcement Learning

SAKR: Enhancing Retrieval-Augmented Generation via Streaming Algorithm and K-Means Clustering

Compensate Quantization Errors+: Quantized Models Are Inquisitive Learners

Beyond Next Token Prediction: Patch-Level Training for Large Language Models

PersLLM: A Personified Training Approach for Large Language Models

PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling

Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models

RoBERTa-BiLSTM: A Context-Aware Hybrid Model for Sentiment Analysis

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization

Aligning Transformers with Continuous Feedback via Energy Rank Alignment

Large Language Models for Cyber Security: A Systematic Literature Review

Construction and Application of Materials Knowledge Graph in Multidisciplinary Materials Science via Large Language Model

CreativeSynth: Cross-Art-Attention for Artistic Image Synthesis with Multimodal Diffusion

LLM A*: Human in the Loop Large Language Models Enabled A* Search for Robotics

Neurodivergent Influenceability as a Contingent Solution to the AI Alignment Problem

AssertionForge: Enhancing Formal Verification Assertion Generation with Structured Representation of Specifications and RTL

Demonstrating specification gaming in reasoning models

SensorChat: Answering Qualitative and Quantitative Questions during Long-Term Multimodal Sensor Interactions

MapExplorer: New Content Generation from Low-Dimensional Visualizations

Addressing and Visualizing Misalignments in Human Task-Solving Trajectories

AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agents

Learning Progress Driven Multi-Agent Curriculum

Neural Thermodynamic Laws for Large Language Model Training

MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoning

Does Feasibility Matter? Understanding the Impact of Feasibility on Synthetic Training Data

Real-Time Out-of-Distribution Failure Prevention via Multi-Modal Reasoning

LibIQ: Toward Real-Time Spectrum Classification in O-RAN dApps

Knowledge capture, adaptation and composition (KCAC): A framework for cross-task curriculum learning in robotic manipulation

Multi-Token Prediction Needs Registers

PnPXAI: A Universal XAI Framework Providing Automatic Explanations Across Diverse Modalities and Models

UniEval: Unified Holistic Evaluation for Unified Multimodal Understanding and Generation

Fine-tuning Diffusion Policies with Backpropagation Through Diffusion Timesteps

Superposition Yields Robust Neural Scaling

SEAL: Searching Expandable Architectures for Incremental Learning

Vision language models have difficulty recognizing virtual objects

Are Large Language Models Robust in Understanding Code Against Semantics-Preserving Mutations?

IN-RIL: Interleaved Reinforcement and Imitation Learning for Policy Fine-Tuning

PIF: Anomaly detection via preference embedding

Learned Lightweight Smartphone ISP with Unpaired Data

Visual Fidelity Index for Generative Semantic Communications with Critical Information Embedding

Rethinking Repetition Problems of LLMs in Code Generation

Inconsistency Handling in DatalogMTL

Uncovering Magnetic Phases with Synthetic Data and Physics-Informed Training

Schreier-Coset Graph Propagation

Multi-Agent Path Finding For Large Agents Is Intractable

Are Sparse Autoencoders Useful for Java Function Bug Detection?

ILIF: Temporal Inhibitory Leaky Integrate-and-Fire Neuron for Overactivation in Spiking Neural Networks

FactsR: A Safer Method for Producing High Quality Healthcare Documentation

SpikeVideoFormer: An Efficient Spike-Driven Video Transformer with Hamming Attention and $\mathcal{O}(T)$ Complexity

Uniform Loss vs. Specialized Optimization: A Comparative Analysis in Multi-Task Learning

Emergence of Structure in Ensembles of Random Neural Networks

Beyond Next Token Prediction: Patch-Level Training for Large Language Models

Created by

Haebom

저자

Chenze Shao, Fandong Meng, Jie Zhou

개요

본 논문은 대규모 언어 모델(LLM)의 막대한 훈련 비용 문제를 해결하기 위해 패치 단위 훈련(patch-level training) 기법을 제시합니다. 여러 토큰을 하나의 정보 밀도가 높은 단위인 '패치'로 묶어 훈련의 기본 단위로 사용하는 방식입니다. 이를 통해 LLM은 더 짧은 패치 시퀀스를 처리하여 훈련 비용을 크게 줄일 수 있습니다. 일부 데이터는 토큰 단위 훈련을 통해 추론 모드와의 정합성을 유지합니다. 다양한 크기(3억 7천만~27억 파라미터)의 모델 실험 결과, 패치 단위 훈련은 훈련 비용을 최대 0.5배까지 절감하면서 성능 저하 없이 LLM을 훈련할 수 있음을 보여줍니다. 소스 코드는 공개되어 있습니다.

시사점, 한계점

•

시사점:

◦

LLM 훈련 비용을 획기적으로 절감할 수 있는 새로운 방법 제시.

◦

패치 단위 훈련을 통해 성능 저하 없이 훈련 비용을 50%까지 줄일 수 있음을 실험적으로 증명.

◦

다양한 크기의 LLM에 적용 가능성을 확인.

◦

소스 코드 공개를 통한 재현성 및 확장성 확보.

•

한계점:

◦

제시된 방법이 모든 종류의 LLM 또는 모든 데이터셋에 동일한 효과를 보이는지는 추가 연구가 필요.

◦

패치 크기 및 토큰-패치 전환 방식 등 하이퍼파라미터 최적화에 대한 추가 연구 필요.

◦

다양한 데이터셋에 대한 실험 결과가 제한적일 수 있음.

◦

토큰 수준의 미세 조정 단계가 여전히 필요하다는 점이 비용 절감의 절대적인 효율성을 제한할 수 있음.

Made with Slashpage