This AI Research from China Proposes YAYI2-30B: A Multilingual Open-Source Large Language Model with 30 Billion Parameters

Researchers have identified a critical need for models tailored specifically for Chinese applications in large language models. The YAYI2-30B model addresses this imperative by refining the existing paradigms, aiming to overcome limitations encountered in models like MPT-30B, Falcon-40B, and LLaMA 2-34B. The central challenge revolves around developing a model capable of comprehending knowledge across diverse domains and excelling in mathematical reasoning and programming tasks.

Existing models such as MPT-30B, Falcon-40B, and LLaMA 2-34B represent the state of the art in large language models. However, a team of researchers from Beijing Wenge Technology Co., Ltd. and the Institute of Automation, Chinese Academy of Sciences, introduced a pioneering solution in YAYI2-30B, a multilingual model meticulously crafted for Chinese applications. Departing from conventional architectures, YAYI2-30B adopts a decoder-only approach, differentiating itself by incorporating FlashAttention 2 and MQA to accelerate training and inference processes. This innovative methodology lays the foundation for a model designed to surpass its predecessors in efficiency and performance.

The intricacies of YAYI2-30B’s architecture unfold as researchers delve into the unique features that set it apart. The decoder-only design, enriched by FlashAttention 2 and MQA, stands out as a testament to the model’s commitment to efficiency. Through the strategic use of distributed training, employing the Zero Redundancy Optimizer (ZeRO) stage 3, gradient checkpointing, and the AdamW optimizer, YAYI2-30B showcases increased efficiency and superior performance.

The meticulous alignment processes of Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) contribute to the model’s adaptability and proficiency across various benchmarks. Evaluations on MMLU, AGIEval, CMMLU, GSM8K, HumanEval, and MBPP underscore YAYI2-30B’s versatility, highlighting its prowess in knowledge understanding, mathematical reasoning, and programming tasks.

The model’s real-world applicability is a testament to the successful fusion of FlashAttention 2, MQA, and alignment processes. YAYI2-30B emerges as an incremental improvement and a leap forward in large language models. Its strategic design and superior performance attest to the researchers’ dedication to overcoming existing challenges.

In conclusion, the research team’s tireless efforts materialize through YAYI2-30B. The strategic alignment processes and innovative architecture position YAYI2-30B as a frontrunner in large language models, particularly tailored for Chinese applications. The researchers’ commitment to refining large language models is evident in YAYI2-30B’s capacity to understand and reason across domains and execute complex programming tasks. The journey to address the challenges of language understanding in Chinese applications takes a remarkable leap forward with the advent of YAYI2-30B, showcasing the potential for groundbreaking advancements in the field. However, users are urged to approach its implementation responsibly, given the potential impact on safety-critical scenarios.

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 35k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, LinkedIn Group, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..