NVIDIA has unveiled a groundbreaking approach to enhancing small language model performance with the introduction of Hymba, a novel family of models combining transformer attention and state space models. Traditional transformer-based models excel in natural language processing due to their ability to retain long-term context and parallel processing capacity; however, these models demand significant computational and memory resources, which poses efficiency challenges. State space models, while more memory efficient, struggle with memory recall. NVIDIA’s Hymba was designed to overcome these issues.
By introducing a hybrid-head parallel architecture, Hymba amalgamates the attention mechanisms of transformers with the constant complexity of state space models. This blend results in superior performance and efficiency, as demonstrated by outperforming the Llama-3.2-3B model. Hymba achieved a 1.32% higher average accuracy, reduced cache size by a factor of 11.67, and increased throughput by 3.49 times. This innovative design integrates attention heads and state space model heads within the same layer, allowing for simultaneous high-resolution recall and efficient context summarization.
Further enhancing the model’s capabilities, NVIDIA introduced learnable meta tokens that optimize performance across a variety of tasks, particularly those requiring memory recall. By sharing key-value cache between layers, inspired by layer correlation, and utilizing sliding window attention, the Hymba models minimize resources while maximizing output. Comprehensive evaluations have shown Hymba to set new state-of-the-art performance benchmarks, paving the way for future advancements in efficient language models.
