tokenizers

<p align="center">
    <br>
    <img src="https://huggingface.co/landing/assets/tokenizers/tokenizers-logo.png" width="600"/>
    <br>
<p>
<p align="center">
    <a href="https://github.com/huggingface/tokenizers/actions"><img alt="Build" src="https://github.com/huggingface/tokenizers/workflows/Rust/badge.svg"></a>
    <a href="https://crates.io/crates/tokenizers"><img alt="Crates.io" src="https://img.shields.io/crates/v/tokenizers.svg"></a>
    <a href="https://docs.rs/tokenizers/"><img alt="Docs" src="https://docs.rs/tokenizers/badge.svg"></a>
    <a href="#footprint"><img alt="Size" src="https://img.shields.io/badge/gzipped-325%20KB-brightgreen"></a>
    <a href="https://github.com/huggingface/tokenizers/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/github/license/huggingface/tokenizers.svg?color=blue&cachedrop"></a>
</p>

The SOTA tokenization library on all languages, all models, all hardwares. 

Our goal with `tokenizers` is to develop and maintain the industry's standard tokenization engine, making it the defacto place for everyone to contribute to the whole ecosystem.

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://huggingface.co/spaces/sbrandeis/tokenizers-v1-compat/resolve/main/banner-dark.png">
  <img alt="tokenizers v1.0.0 progress" src="https://huggingface.co/spaces/sbrandeis/tokenizers-v1-compat/resolve/main/banner-light.png">
</picture>

### Release candidate: v1.0.0.rc.x

As we are switching from 0.23 to v1.0.0, the current library does not ship all features. If you are worried, check the [v1 features](#v1-features) section for more details of what we are bringing back.
We will publish a blog about what breaking changes we introduced. We strived to keep them as small as possible.


# Installation

```bash
pip install --pre tokenizers
```

For Rust, add the crate to your `Cargo.toml`:

```toml
[dependencies]
tokenizers = "1.0.0"
```

Training (`train`) and downloading from the Hub (`http`) are opt-in features. The crate's
[Using the crate](tokenizers/README.md#using-the-crate) section lists every feature and how to
build a smaller binary.

# Usage

```python
>>> from tokenizers import Tokenizer
>>> tokenizer = Tokenizer.from_pretrained("meta-llama/Llama-3.1-8B")   # or .from_file(path)
>>> compiled = tokenizer.compile()
>>> compiled.encode("Hello, y'all! How are you 😁 ?")
Encoding(ids=[128000, 9906, 11, 379, 65948, 0, 2650, 527, 499, 27623, 223, 949], type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
>>> compiled.encode(("What is this?", "A pair.")).type_ids
[0, 0, 0, 0, 0, 1, 1, 1, 1]
>>> compiled.tokenize("Hello")
['<|begin_of_text|>', 'Hello']
>>> compiled.decode([15339, 1917])
'hello world'
>>> compiled.decode_tokens(compiled.encode("Hello"))
['<|begin_of_text|>', 'Hello']
```

`Tokenizer` holds the `tokenizer.json` config and can be edited (e.g. `tokenizer.truncation = ...`);
`compile()` builds the `CompiledTokenizer` that encodes a