LiteLLM

lastmod 2026-03-25created 2023-08-164 edits

Python SDK, Proxy Server (AI Gateway) to call 100+ LLM APIs in OpenAI (or native) format, with cost tracking, guardrails, loadbalancing and logging. [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, VLLM, NVIDIA NIM]

A proxy server supporting more than 50 LLMs

Features

• Can call the /chat/completion API for various LLMs such as Azure, OpenAI, Replicate, Anthropic, and Hugging Face
• Unifies the Input/Ouput format into the OpenAI format
• Supports model Fallback (e.g. call llama2 when GPT-4 fails)
• Logging support: Supabase, Posthog, Mixpanel, Sentry, Helicone
• Token usage tracking
• Implements Semantic Caching
• Streaming and asynchronous support

See also

• LLM

Favorite site

• LiteLLM web site
• Github - BerriAI/litellm
• litellm/cookbook/proxy-server/readme.md at main · BerriAI/litellm
• LiteLLM, the JDBC driver for LLM models

Articles

• LiteLLM was hacked in a supply chain attack. | GeekNews