Meet Magika: A Novel AI-Powered File Type Detection Tool that Relies on the Recent Advances of Deep Learning to Provide Accurate Detection

In the digital realm, identifying the type of files we encounter is crucial for ensuring safety and security. However, with the increasing complexity and diversity of file formats, accurately detecting the content of files becomes a challenge. Existing solutions often face limitations in precision and recall, leaving room for improvement in file type detection.

Magika steps in as a novel AI-powered solution to address the need for a more accurate and efficient file type detection tool. Magika tackles the common problem of misidentifying file types using deep learning technology. Unlike existing tools that may struggle with accuracy, Magika relies on a custom, highly optimized Keras model that weighs only about 1MB. This allows for rapid and precise file identification, even when running on a single CPU.

Magika’s performance is truly noteworthy, especially when compared to existing approaches. In an evaluation involving over 1 million files and spanning more than 100 content types, including both binary and textual formats, Magika achieves a remarkable 99% or more in both precision and recall. This means it correctly identifies files and minimizes false positives or negatives.

The tool offers multiple modes of accessibility, available as a Python command line, a Python API, and even an experimental TFJS version. Trained on a substantial dataset of over 25 million files across diverse content types, Magika exhibits near-constant inference time, taking only about five milliseconds per file after the model is loaded. Its ability to process batches of files simultaneously further enhances its efficiency.

One unique feature of Magika lies in its per-content-type threshold system. This system helps determine the level of trust in the model’s prediction for each file type, allowing for more nuanced and accurate results. Additionally, Magika supports three prediction modes – high-confidence, medium-confidence, and best-guess – catering to varying error tolerance levels.

In conclusion, Magika emerges as a powerful and efficient solution to the challenge of file type detection. Its impressive metrics and versatile accessibility make it a valuable tool for enhancing safety and security, especially in large-scale applications like Gmail, Drive, and Safe Browsing. With an open invitation for community collaboration, Magika represents a positive stride towards improving the accuracy and reliability of file type detection in the digital landscape.

Installation

Magika is available as magika on PyPI:

$ pip install magika