Meet JEN-1: A Universal AI Framework that Combines Bi-Directional and Uni-Directional Modes to Generate High-Quality Music Conditioned on Either Text or Music Representations

Music, often hailed as the universal language of humanity by Henry Wadsworth Longfellow, carries within it the essence of harmony, melody, and rhythm, weaving a tapestry of cultural significance that profoundly resonates with people across the globe. Recent advancements in deep generative models have driven progress in music generation. However, the challenge of generating high-quality, realistic music that captures its complexity and nuances, particularly when conditioned on textual descriptions, remains formidable.

 Existing methods for generating music have made significant strides, but developing intricate and lifelike music that aligns with free-form textual prompts still needs to be improved. Music’s multifaceted nature, spanning various instruments and harmonies, requires addressing specific challenges:

1. The music encompasses a wide frequency spectrum, necessitating high sampling rates like 44.1KHz stereo for capturing intricate details. This contrasts with speech, which operates at lower sampling rates.
2. The intricate interplay of instruments and the arrangement of melodies and harmonies result in complicated and complex musical structures. Precision is crucial, as music is highly sensitive to dissonance.
3. Maintaining control over attributes like key, genre, and melody is pivotal to realizing the intended artistic vision.

To address these challenges of the text-to-music generation, the Futureverse research team designed JEN-1. JEN-1 leverages a unique omnidirectional diffusion model that combines autoregressive (AR) and non-autoregressive (NAR) paradigms, allowing it to capture sequential dependencies while accelerating generation. Unlike prior methods that often convert audio data to mel-spectrograms, JEN-1 directly models raw audio waveforms, maintaining higher fidelity and quality. This is possible through a noise-robust masked autoencoder that compresses original audio into latent representations, preserving high-frequency details.Researchers introduce a normalization step that reduces anisotropy in the latent embeddings to enhance the model’s performance further.

JEN-1’s core architecture is an omnidirectional 1D diffusion model combining bid and unidirectional modes. The model leverages a temporal 1D efficient U-Net inspired by the Efficient U-Net architecture. This architecture is designed to model waveforms effectively, and it includes both convolutional and self-attention layers to capture sequential dependencies and contextual information. The unidirectional mode, crucial for music generation due to its time-series nature, is incorporated through causal padding and masked self-attention, ensuring the generated latent embeddings on the right depend on their left counterparts.

One of JEN-1’s unique strengths lies in its unified music multi-task training approach. It supports three main music generation tasks:

• Bidirectional text-guided music generation
• Bidirectional music inpainting (restoring missing segments)
• Unidirectional music continuation (extrapolation)

Through multi-task training, JEN-1 shares parameters across tasks, allowing it to generalize better and handle sequential dependencies more effectively. This flexibility makes JEN-1 a versatile tool that can be applied to diverse music generation scenarios.

The experiment setup involves training JEN-1 on 5,000 hours of high-quality music data. The model uses a masked music autoencoder and FLAN-T5 for text embeddings. During training, multi-task objectives are balanced, and classifier-free guidance is employed. JEN-1 is trained for 200k steps using the AdamW optimizer on 8 A100 GPUs.

JEN-1’s performance is compared against several state-of-the-art methods using objective and subjective metrics. It outperforms other methods in terms of plausibility (FAD), audio-text alignment (CLAP), and human-rated text-to-music quality (T2M-QLT), and alignment (T2M-ALI). Despite its computational efficiency, JEN-1 surpasses competing models in text-to-music synthesis.

Ablation studies demonstrate the effectiveness of different components in JEN-1. Incorporating the auto-regressive mode and employing multi-tasking objectives enhance music quality and generalization. The proposed method consistently achieves high-fidelity music generation without increasing training complexity.

Overall, JEN-1 presents a powerful solution for text-to-music generation, significantly advancing the field. It generates high-quality music by directly modeling waveforms and combining auto-regressive and non-autoregressive training. The integrated diffusion models and masked auto-encoders enhance sequence modeling. JEN-1 demonstrates superiority in subjective quality, diversity, and controllability compared to strong baselines, highlighting its effectiveness for music synthesis.

Check out the Paper and Project. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 28k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.