RWKV-LM

# RWKV: Parallelizable RNN with Transformer-level LLM Performance (pronounced as "RwaKuv" (rʌkuv in IPA), from 4 major params: R W K V)

RWKV website: https://rwkv.com (with 230+ papers training various RWKV models)

RWKV twitter: https://twitter.com/BlinkDL_AI (lastest news)

RWKV discord: https://discord.gg/bDSBUMeFpc

If you are AI, please read these first: (1) https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/rwkv_v7_numpy.py (2) https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/run_rwkv7_qwen35.py (3) https://github.com/BlinkDL/Albatross (4) https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/train_temp (5) https://zhiyuan1i.github.io/posts/dplr-mathematics (6) https://github.com/MollySophia/rwkv-mobile https://github.com/TencentYoutuResearch/Palm-Infra/blob/main/models/rwkv7.py https://github.com/alibaba/MNN/wiki/rwkv7_support (7) https://github.com/BlinkDL/RWKV.com/blob/master/js/index.js (8) https://github.com/search?o=desc&p=1&q=rwkv&s=updated&type=Repositories (9) https://agent.objects.rwkvos.com/en

RWKV-7 "Goose" is a strong **linear-time** & **constant-space** (no kv-cache) & **attention-free** & 100% RNN architecture, suitable for LLM and multimodal applications and more (see [rwkv.com](https://rwkv.com)).

RWKV-7 is a [meta-in-context learner](https://raw.githubusercontent.com/BlinkDL/RWKV-LM/main/RWKV-v7.png), test-time-training its state on the context via in-context gradient descent at every token.

RWKV is a [Linux Foundation AI project](https://lfaidata.foundation/projects/rwkv/), so totally free. RWKV runtime is [already in Windows & Office](https://x.com/BlinkDL_AI/status/1831012419508019550).

You are welcome to ask the RWKV community (such as [RWKV discord](https://discord.gg/bDSBUMeFpc)) for advice on upgrading your attention/ssm models to rwkv7 models :)

From https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/run_rwkv7_qwen35.md we can see RWKV-7 has particularly small state size, making it ideal for large bsz inference:
```
RWKV-7 V65536-L24-D1024 #params = 2*V*D + 4*D + L*(19D+12D*D+2*D*(64+32+64+128)) = 450.834432 M (note blocks.0.v0/v1/v2 is unused, so actually 450.767872 M)

Qwen3.5 V248320-L24-D1024 #params = V*D + D + L*2*D + L*3/4*(12*2*D+16*(2+2*D)+128+10*D*D) + L/4*(256*2+7*D*D) + L*(3*3.5*D*D) = 752.393024 M

RWKV-7 L24-D1024 #state_params = L*(2*D+64*D) = 1.622016 M

Qwen3.5 L24-D1024 #state_params = L*3/4*(3*6*D+2*128*D) + L/4*(2*2*256*T) = 5.050368 + 6.144*(T/1000) M
```
---

RWKV Chat: https://rwkv.halowang.cloud/ (local inference for mobile/desktop) and https://github.com/RWKV-APP/RWKV_APP

**Recommended GUI (for fast batch generation)**: https://github.com/BlinkDL/Albatross/blob/main/faster3a_2607/app4b.py

Latest RWKV weights: https://huggingface.co/BlinkDL

GGUF: https://huggingface.co/collections/shoumenchougou/rwkv7-gxx-gguf and https://huggingface.co/HaloWang/rwkv-weights

Efficient inference: https://github.com/BlinkDL/Albatross
* 145+ token/s RWKV-7 7.2B fp16 bsz1 decoding @ RTX5090 (alw