Hecto is a lightweight MoE architecture proposed to overcome the limitations of existing MoE models, such as identical inductive bias and static computational path. It exploits the heterogeneity of the architecture by combining GRU experts (temporal inference) and FFNN experts (static abstraction) under a Top-1 gating mechanism. It is evaluated on three inference benchmarks (AG News, SST-2, HotpotQA) and a regression task (STS-B), and shows similar or only a small performance difference from the homogeneous baseline model despite the separated input representation. Each expert is shown to be specialized for a distinct type of inference: temporal inference and static inference. In large batch sizes, the computational constraints are relaxed, which leads to better optimization of the heterogeneous architecture, resulting in improved performance. Experimental results demonstrate that the stability and interpretability of Hecto across a variety of inference tasks stem from the diversity of the architecture. In conclusion, Hecto is positioned as a new conditional computational benchmark that provides a principled framework for specialized inference in resource-constrained environments.
