LiteRT-LM

lastmod 2026-04-22created 2026-04-222 edits

Google's high-performance LLM inference framework for edge devices

About

• A production-grade on-device LLM inference engine built by Google; runs large language models across edge environments including Android, iOS, web, desktop, and IoT (Raspberry Pi)
• Support for the latest Gemma 4 models added; delivers optimal inference performance even on edge devices through GPU and NPU hardware acceleration
• Multimodal support allows vision (image) and audio input; the CLI also supports inference with attached images via the --attachment option
• Built-in Function Calling (Tool Use) for agentic workflows
• Compatible with various LLMs including Gemma, Llama, Phi-4, and Qwen; a single CLI command downloads a model from Hugging Face and runs inference immediately
  • Get started right away with uv tool install litert-lm → litert-lm run
• Actually shipped in Google products such as Chrome, Chromebook Plus, and Pixel Watch, powering on-device GenAI
• Models can be run instantly on mobile through the Google AI Edge Gallery app (available on Google Play and the App Store)
• Language APIs: stable support for Kotlin (Android/JVM), Python (prototyping), and C++ (high-performance native); Swift (iOS/macOS) in development
• Release status: v0.10.2 is the latest; v0.10.1 introduced Gemma 4 and the CLI, v0.8.0 added desktop GPU and multimodal, v0.7.0 added NPU acceleration
• Apache-2.0 license

See also

• LLM

Favorite site

• Github - google-ai-edge/LiteRT-LM
• Google LiteRT-LM - High-performance LLM inference framework for edge devices | GeekNews