OpenLLM

<div align="center">

<h1>🦾 OpenLLM: Self-Hosting LLMs Made Easy</h1>

[![License: Apache-2.0](https://img.shields.io/badge/License-Apache%202-green.svg)](https://github.com/bentoml/OpenLLM/blob/main/LICENSE)
[![Releases](https://img.shields.io/pypi/v/openllm.svg?logo=pypi&label=PyPI&logoColor=gold)](https://pypi.org/project/openllm)
[![CI](https://results.pre-commit.ci/badge/github/bentoml/OpenLLM/main.svg)](https://results.pre-commit.ci/latest/github/bentoml/OpenLLM/main)
[![X](https://badgen.net/badge/icon/@bentomlai/000000?icon=twitter&label=Follow)](https://twitter.com/bentomlai)
[![Community](https://badgen.net/badge/icon/Community/562f5d?icon=slack&label=Join)](https://l.bentoml.com/join-slack)

</div>

OpenLLM allows developers to run **any open-source LLMs** (Llama 3.3, Qwen2.5, Phi3 and [more](#supported-models)) or **custom models** as **OpenAI-compatible APIs** with a single command. It features a [built-in chat UI](#chat-ui), state-of-the-art inference backends, and a simplified workflow for creating enterprise-grade cloud deployment with Docker and Kubernetes.

Understand the [design philosophy of OpenLLM](https://www.bentoml.com/blog/from-ollama-to-openllm-running-llms-in-the-cloud).

## Get Started

Run the following commands to install OpenLLM and explore it interactively.

```bash
pip install openllm  # or pip3 install openllm
openllm hello
```

![hello](https://github.com/user-attachments/assets/5af19f23-1b34-4c45-b1e0-a6798b4586d1)

## Supported models

OpenLLM supports a wide range of state-of-the-art open-source LLMs. You can also add a [model repository to run custom models](#set-up-a-custom-repository) with OpenLLM.

<table>
  <tr>
    <th>Model</th>
    <th>Parameters</th>
    <th>Required GPU</th>
    <th>Start a Server</th>
  </tr>
  <tr>
    <td>deepseek</td>
    <td>r1-671b</td>
    <td>80Gx16</td>
    <td><code>openllm serve deepseek:r1-671b</code></td>
  </tr>
  <tr>
    <td>gemma2</td>
    <td>2b</td>
    <td>12G</td>
    <td><code>openllm serve gemma2:2b</code></td>
  </tr>
  <tr>
    <td>gemma3</td>
    <td>3b</td>
    <td>12G</td>
    <td><code>openllm serve gemma3:3b</code></td>
  </tr>
  <tr>
    <td>jamba1.5</td>
    <td>mini-ff0a</td>
    <td>80Gx2</td>
    <td><code>openllm serve jamba1.5:mini-ff0a</code></td>
  </tr>
  <tr>
    <td>llama3.1</td>
    <td>8b</td>
    <td>24G</td>
    <td><code>openllm serve llama3.1:8b</code></td>
  </tr>
  <tr>
    <td>llama3.2</td>
    <td>1b</td>
    <td>24G</td>
    <td><code>openllm serve llama3.2:1b</code></td>
  </tr>
  <tr>
    <td>llama3.3</td>
    <td>70b</td>
    <td>80Gx2</td>
    <td><code>openllm serve llama3.3:70b</code></td>
  </tr>
  <tr>
    <td>llama4</td>
    <td>17b16e</td>
    <td>80Gx8</td>
    <td><code>openllm serve llama4:17b16e</code></td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>8b-2410</td>
    <td>24G</td>
    <td><code>openllm serve mistral:8b-2410</code></td>
  </tr>
  <tr>
    <td>mistral-large</td>
    <td>123b-2407</td>
    <td>80Gx4</td