Researchers from Microsoft and Hong Kong Baptist University Introduce WizardCoder: A Code Evol-Instruct Fine-Tuned Code LLM

Large Language Models (LLMs) have recently attracted much interest and achieved remarkable success. OpenAI’s ChatGPT, in particular, stands out as a notable example. These models have achieved state-of-the-art (SOTA) zero-shot performance across various tasks by utilizing significant pre-training on massive quantities of internet data and further fine-tuning with precise instruction data. This pattern is also seen in the comprehension and creation of code. Many Code LLMs have been proposed to address the difficulties inherent to activities using code. These Code LLMs go through pre-training utilizing a large quantity of code data, allowing them to perform admirably in various activities linked to code. 

There needs to be more investigation into fine-grained instruction tailoring in the Code domain, in contrast to most prior Code LLMs that largely emphasize the pre-training phase. To improve the generalization skills of LMs across various activities, instruction tweaking was first used. For instance, OpenAI’s InstructGPT asked human annotators to offer specific instructions to verify conformity with users’ objectives. Like Alpaca, a recent effort used ChatGPT to produce the instruction data using the self-instruct approach. Vicuna took advantage of chats that users had posted on ShareGPT.com. The Evol-Instruct approach was established by WizardLM and entailed modifying the current instruction data to produce more intricate and varied datasets. 

However, it is important to note that these techniques should have specifically taken the code domain into account when designing instead of focusing mostly on the general domain. Inspired by the Evol-Instruct approach, researchers from Microsoft and Hong Kong Baptist University in this project intend to improve the StarCoder open-source Code LLM’s capabilities by producing detailed code instruction data using code-specific Evol-Instruct. They have modified the evolutionary prompt process in several ways designed particularly for activities involving coding to achieve this. The evolutionary prompts have been simplified, the evolutionary instructions have been improved, and code debugging and time-space complexity limitations have been included. Their approach is initially used to develop the fundamental Code Alpaca instruction data. 

They next use their freshly developed code instruction-following training set to fine-tune StarCoder and get their WizardCoder. Their WizardCoder beats all other open-source Code LLMs, attaining state-of-the-art (SOTA) performance, according to experimental findings from four code-generating benchmarks, including HumanEval, HumanEval+, MBPP, and DS-100. They notice a significant rise in pass@1 scores, namely a +22.3 (57.3 vs. 35.0) increase in HumanEval and a +8.2 (51.8 vs. 43.6) increase in MBPP. Surprisingly, their WizardCoder even outperforms Anthropic’s Claude and Google’s Bard in terms of pass rates on HumanEval and HumanEval+ despite being considerably smaller. 

The following is a summary of the contributions made by this work:

• We provide WizardCoder, which applies Code Evol-Instruct to improve the functionality of the open-source Code LLM, StarCoder.

• WizardCoder significantly outperforms all other open-source Code LLMs, including StarCoder, CodeGen, CodeGee, CodeT5+, InstructCodeT5+, StarCoder-GPTeacher, and Instruct-Codegen-16B, in terms of code creation.

• Despite being significantly lower in size, WizardCoder outperforms the major closed-source LLMs, including Claude, Bard, PaLM, PaLM-2, and LaMDA, in terms of code creation.

Check out the Paper and Github link. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 29k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, please follow us on Twitter