Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

Sophisticated Learning: A novel algorithm for active learning during model-based planning

Created by
  • Haebom

Author

Rowan Hodson, Bruce Bassett, Charel van Hoof, Benjamin Rosman, Mark Solms, Jonathan P. Shock, Ryan Smith

Outline

Sophisticated Learning (SL) is a plan-and-learn algorithm that integrates active parameter learning within the Sophisticated Inference (SI) tree search framework of Active Inference. Unlike SI, which optimizes beliefs about hidden states, SL also updates beliefs about model parameters within each simulated branch, enabling counterfactual inference about how future observations might improve subsequent plans. We designed experiments using a biologically inspired seasonal foraging task, forcing agents to balance probabilistic reward harvesting with information gathering. SL survived an average of 8.2% longer than SI and 35% longer than Bayes-adaptive Reinforcement Learning (BARL) in early trials, when rapid learning is crucial. While SL and SI showed identical convergence performance, SL converged 40% faster than SI. Furthermore, SL showed robust performance improvements over other algorithms under changing environmental configurations. These results demonstrate that incorporating active learning into multi-stage planning substantially improves decision-making under radical uncertainty and enhances the broad utility of active inference for modeling biologically relevant behaviors.

Takeaways, Limitations

Takeaways:
Integrating active learning into multi-step planning significantly improves decision-making under radical uncertainty.
SL outperforms existing algorithms (SI, BARL) in situations where fast learning in the early stages is important.
SL shows convergence performance equivalent to existing algorithms while achieving faster convergence speed.
SL maintains solid performance across a variety of configurations.
We highlight the utility of Active Inference in modeling biologically relevant behaviors.
Limitations:
Further research is needed to determine the generalizability of the presented biologically inspired tasks.
The performance of SL needs to be evaluated in more complex and diverse environments.
An analysis of the computational complexity of the SL algorithm is required.
Optimization possibilities for specific tasks should be considered.
👍