Skip to content
Search

Magika

lastmod 2025-02-27created 2024-02-192 edits

Detect file content types with deep learning

Open-source AI-powered fast and efficient file type identification tool

About

  • Google open-sourced Magika, an AI-based file type identification system
  • Magika uses a custom, optimized deep learning model that enables accurate file identification within milliseconds, even on a CPU
  • You can try the Magika web demo, and install it as a Python library and standalone command-line tool with the pip install magika command

Why file type identification is hard

  • Since the early days of computing, accurately detecting file types has been crucial in determining how to process files.
  • Linux has the libmagic and file utilities, which have served as the de facto standard for file type identification for over 50 years.
  • File type detection is essential for many kinds of software, such as web browsers and code editors, to render files correctly.
  • Because each file format has a different structure, or no structure at all, accurate file type detection is a notoriously difficult problem.
  • libmagic and most file type identification software rely on handcrafted heuristics and rules to detect each file format.
  • This manual approach is time-consuming and error-prone.

Magika performance

  • Thanks to its AI model and large training dataset, Magika performs about 20% better than existing tools on a 1-million-file benchmark covering more than 100 file types.
  • It shows even larger performance gains on file types that other tools struggle with, such as text files, code files, and configuration files.

Magika's use within Google

  • Within Google, Magika is used to improve user safety by routing Gmail, Drive, and Safe Browsing files to the appropriate security and content policy scanners.
  • Averaged weekly over hundreds of billions of files, Magika improved file type identification accuracy by 50% compared to the previous system that relied on handcrafted rules.
  • Through integration with VirusTotal, Magika will complement the existing Code Insight feature and contribute to the global cybersecurity ecosystem, helping create a safer digital environment.

Open-sourcing Magika

  • By open-sourcing Magika, Google aims to improve file identification accuracy in other software and give researchers a reliable way to identify file types at scale.
  • Magika's code and model are freely available on Github under the Apache2 license and can be easily installed via the pypi package manager.
  • For details on how to use Magika, see the Magika documentation site.

GN⁺'s opinion

  • Open-sourcing Magika looks likely to help greatly in improving the accuracy of file type identification.
  • Accurate file identification is especially important in the security field, and Magika could be a powerful tool for it.
  • Google's technical capability and contributions to the open source community are expected to play an important role in strengthening the global cybersecurity ecosystem.

Hacker News comments

  • Magika: AI powered fast and efficient file type identification | Hacker News
  • Tried the new file type detection tool on web crawl data.
    • Some simple HTML files were misdetected.
    • Some WOFF and WOFF2 files were also misdetected.
    • Judged it not yet reliable enough for automation.
    • Also complained that it doesn't strip shell color escapes when the output is a pipe.
  • Shared an experience with spreadsheet file type detection 10 years ago.
    • Proposed a patch to detect file types using "magic", but it was rejected.
    • Commented on file type detection now using deep learning.
    • Hoped Google would publish performance benchmarks.
  • Shared detection test results for 100 files.
    • Most were detected correctly, but some were misdetected or detected as an unclear type.
    • Errors occurred mainly on file types Magika does not support.
    • Showed accuracy similar to the existing file utility.
  • Positive reaction to a new tool appearing in the file type detection space.
    • Questioned why the Node module was released.
    • The documentation mentions it is slow, and the model is loaded at runtime.
    • Noted that it is marked experimental and supports a limited set of file types.
  • Confusion over why Google put resources into a neural-network-based file type detection tool instead of improving libmagic.
    • Pointed out that the neural network is more accurate but supports fewer file types and is less effective in adversarial situations.
  • Curious how polyglot files that are valid in multiple formats are detected.
    • In an actual test, only the ZIP layer was detected.
  • Questioned the precision or recall of other tools for accurately detecting APK or JAR files.
    • Asked for an explanation of cases where specific files affected precision or recall.
  • Shared experience implementing libmagic in Racket.
    • Noted that libmagic detects more file types, but Magika could be useful for detecting text files.
  • Argued that file type detection is inherently deterministic.
    • Questioned the need for heuristics or probabilistic inference, since "magic bytes" should be consistent.

See also

Favorite site

author Zer0 sitemap