Meet CipherChat: An AI Framework to Systematically Examine the Generalizability of Safety Alignment to Non-Natural Languages-Specifically Ciphers

Artificial intelligence (AI) systems have advanced significantly as a result of the introduction of Large Language Models (LLMs). Leading LLMs such as ChatGPT released by OpenAI, Bard by Google, and Llama-2 have demonstrated their remarkable abilities in carrying out innovative applications, ranging from assisting in tool utilization and enhancing human evaluations to simulating human interactive behaviors. The extensive deployment of these LLMs has been made possible by their extraordinary competencies, but it comes with a significant challenge of assuring the security and dependability of their responses.

In relation to non-natural languages, specifically ciphers, recent research by a team has introduced several important contributions that advance the understanding and application of LLMs. These innovations have been proposed with the aim of improving the dependability and safety of LLM interactions in this particular linguistic setting.

The team has introduced CipherChat, which is a framework created expressly to evaluate the applicability of safety alignment methods from the domain of natural languages to that of non-natural languages. In CipherChat, humans interact with LLMs through cipher-based prompts, detailed system role assignments, and succinct enciphered demonstrations. This architecture ensures that the LLMs’ understanding of ciphers, participation in the conversation, and sensitivity to inappropriate content are thoroughly examined.

This study highlights the critical need for the creation of safety alignment methods when working with non-natural languages, such as ciphers, in order to successfully match the capabilities of the underlying LLMs. While LLMs have shown extraordinary skill in understanding and producing human languages, the research says that they also demonstrate unexpected prowess in comprehending non-natural languages. This information highlights the significance of developing safety regulations that cover these non-traditional forms of communication as well as those that fall within the purview of traditional linguistics.

A number of experiments have been done using a variety of realistic human ciphers on modern LLMs, such as ChatGPT and GPT-4, to assess how well CipherChat performs. These evaluations cover 11 different safety topics and are available in both Chinese and English. The findings point to a startling pattern which is that certain ciphers are able to successfully get around GPT-4’s safety alignment procedures, with virtually 100% success rates in a number of safety domains. This empirical result emphasizes the urgent necessity for creating customized safety alignment mechanisms for non-natural languages, like ciphers, to guarantee the robustness and dependability of LLMs’ answers in various linguistic circumstances.

The team has shared that the research uncovers the phenomenon of the presence of a secret cipher within LLMs. Drawing parallels to the concept of secret languages observed in other language models, the team has hypothesized that LLMs might possess a latent ability to decipher certain encoded inputs, thereby suggesting the existence of a unique cipher-related capability. 

Building on this observation, a unique and effective framework known as SelfCipher has been introduced, which relies solely on role-play scenarios and a limited number of demonstrations in natural language to tap into and activate the latent secret cipher capability within LLMs. The efficacy of SelfCipher demonstrates the potential of harnessing these hidden abilities to enhance LLM performance in deciphering encoded inputs and generating meaningful responses.

Check out the Paper, Project, and GitHub. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 28k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.