We present a physics-information-based deep learning framework called PhISM. This framework explicitly separates hyperspectral observations and models them using continuous basis functions without supervised learning. It outperforms existing methods on several classification and regression benchmarks, requires limited labeled data, and provides additional insights thanks to interpretable latent representations.