Hunyuan Large outperforms leading models such as LLama 3.1 70B and LLama 3.1 – 405B which support 128,000 tokens.
Large language models (LLMs), in recent years, have become the cornerstone of many AI systems, significantly advancing NLP, computer vision, and scientific research. Taking a step ahead, Chinese giant Tencent has released Hunyuan Large, the largest open source Transformer-based mixture of experts model.
With a total of 389 billion parameters and 52 billion activation parameters, the Hunyuan Large model is capable of handling up to 256,000 tokens. Hunyuan Large outperforms leading models such as LLama 3.1 70B and LLama 3.1 – 405B which support 128,000 tokens.
Key practices of Hunyuan-Large include large-scale synthetic data that is orders larger than in previous literature, a mixed expert routing strategy, a key-value cache compression technique, and an expert-specific learning rate strategy.
Tencent has conducted extensive experiments on diverse types of benchmarks in both English and Chinese to demonstrate the power of Hunyuan-Large. The company compared the best-performing dense and MoE models with similar parameter sizes.
“We find that Hunyuan-Large is capable of handling various tasks including commonsense understanding, question answering, mathematics reasoning, coding, and aggregated tasks, achieving the overall best performance among existing open-source similar-scale LLMs,” the company said in an announcement.
Hunyuan-Large is pre-trained on 7 trillion tokens, containing nearly 1.5 trillion tokens of high-quality and diverse synthetic data that improve learning across diverse fields like mathematics, coding, and multilinguality.
The pre-trained and post-trained Hunyuan-Large models are publicly released to facilitate the LLM community.
The code and checkpoints of Hunyuan-Large are released to facilitate future innovations and applications.
Also Read:
- IBM Introduces High Performing AI Models ‘Granite 3.0’ for Businesses
- Stability AI Introduces Stable Diffusion 3.5 Text-to-Image AI Model Suite
Stay Tuned to The Future Talk for more AI news and insights!