open_clip

# OpenCLIP

[[Paper]](https://arxiv.org/abs/2212.07143) [[Citations]](#citing) [[Clip Colab]](https://colab.research.google.com/github/mlfoundations/open_clip/blob/main/docs/Interacting_with_open_clip.ipynb) [[Coca Colab]](https://colab.research.google.com/github/mlfoundations/open_clip/blob/main/docs/Interacting_with_open_coca.ipynb)
[![pypi](https://img.shields.io/pypi/v/open_clip_torch.svg)](https://pypi.python.org/pypi/open_clip_torch)

> **Versions**
>
> - **4.x** is the current release line, developed on `main`. 4.2.1 is the first 4.x release on PyPI; earlier 4.x version numbers were development versions only.
> - **3.x** remains available on the [`v3` branch](https://github.com/mlfoundations/open_clip/tree/v3). To stay on it, run `pip install 'open_clip_torch<4'`. Further 3.3.x releases will only be made if needed.
> - **Upgrading from 3.x:** pretrained image/text inference is intended to remain compatible, but training scripts and downstream integrations need review. See the [migration guide](https://github.com/mlfoundations/open_clip/blob/main/docs/MIGRATING_3_TO_4.md).

Welcome to an open source implementation of OpenAI's [CLIP](https://arxiv.org/abs/2103.00020) (Contrastive Language-Image Pre-training). OpenCLIP also includes language-audio ([CLAP](https://arxiv.org/abs/2211.06687)) models and related generative models.

Using this codebase, we have trained several models on a variety of data sources and compute budgets, ranging from [small-scale experiments](docs/LOW_ACC.md) to larger runs including models trained on datasets such as [LAION-400M](https://arxiv.org/abs/2111.02114), [LAION-2B](https://arxiv.org/abs/2210.08402) and [DataComp-1B](https://arxiv.org/abs/2304.14108).
Many of our models and their scaling properties are studied in detail in the paper [reproducible scaling laws for contrastive language-image learning](https://arxiv.org/abs/2212.07143).
Some of the best models we've trained and their zero-shot ImageNet-1k accuracy are shown below, along with the ViT-L model trained by OpenAI and other state-of-the-art open source alternatives (all can be loaded via OpenCLIP).
We provide more details about our full collection of pretrained models [here](docs/PRETRAINED.md), and zero-shot results for 38 datasets [here](docs/openclip_results.csv).
Experimental helper scripts for [multilabel zero-shot evaluation](docs/multilabel_zeroshot.md) (`scripts/multilabel_zeroshot.py`) and [linear probes](docs/linear_probe.md) (`scripts/linear_probe.py`) are also available.



| Model    | Training data | Resolution | # of samples seen | ImageNet zero-shot acc. | 
| -------- | ------- |  ------- |  ------- |  ------- |  
| ConvNext-Base | LAION-2B  | 256px | 13B | 71.5% |
| ConvNext-Large | LAION-2B  | 320px | 29B | 76.9% |
| ConvNext-XXLarge | LAION-2B | 256px | 34B | 79.5% |
| ViT-B-32-256  | DataComp-1B  | 256px | 34B | 72.8% |
| ViT-B-16  | DataComp-1B  | 224px | 13B | 73.5% |
| ViT-L-14  | LAION-2B  | 224px | 32B | 75.3% |
| ViT-H-14  |