AirLLM: run a 70B AI model on a 4GB graphics card, free and open source

Big open AI models usually need big, expensive GPUs. AirLLM takes a different route. It never loads the whole model at once. It loads one layer, runs it, and moves on to the next. That's how its README says a 70B model can run on a single 4GB card. This is the full version of the AIRLLM guide from my Reel: what it is, how to set it up, and the catch you should know about before you try.

01What AirLLM is

AirLLM is a free, open source Python library by Gavin Li. The code is under the Apache-2.0 license. You give it a Hugging Face model, and it runs that model on your own machine. No API, no per-message bill.

The README says it lets 70B models run on a single 4GB GPU without quantization, distillation or pruning. So you get the original model, not a squeezed-down copy.

02How it works: one layer at a time

A language model is a stack of layers. Normally all of them sit in GPU memory together. AirLLM keeps only the layer it's working on in memory. The rest wait on your disk. So the memory you need depends on one layer, not the whole model.

The first time you load a model, AirLLM splits it into those layers and saves them to disk. After that it streams them in as it goes.

The README's numbers are wild. It lists Kimi K3 (2.8T parameters) running in 3.72 GB of VRAM, DeepSeek-V3 (671B) in about 12 GB, and Qwen3-235B in about 3 GB. Those are the project's own measurements, on specific cards.

03How to install it

Make sure you have Python and PyTorch installed.
Install AirLLM with pip (command below).
Pick a model on Hugging Face and copy its repo ID, for example Qwen/Qwen3-32B.
Check you have plenty of free disk space. The model gets split and saved layer by layer.
Run the quickstart code below. The first run is slow because of the split.

Install command

pip install airllm

Quickstart code (from the README)

from airllm import AutoModel

MAX_LENGTH = 128
model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

input_text = ["What is the capital of United States?"]
input_tokens = model.tokenizer(input_text, return_tensors="pt", return_attention_mask=False, truncation=True, max_length=MAX_LENGTH, padding=False)

generation_output = model.generate(input_tokens["input_ids"].cuda(), max_new_tokens=20, use_cache=True, return_dict_in_generate=True)

print(model.tokenizer.decode(generation_output.sequences[0]))

04What you need

Linux with an NVIDIA GPU. The quickstart uses CUDA.
Or a Mac with Apple silicon. The README says to install mlx and torch first, and that only Apple silicon is supported on macOS.
Lots of free disk space. The README warns that splitting a model uses a lot of it, and that running out of space is the most likely cause of errors.
Some very new or very large models have extra requirements. Kimi K3, for example, needs flash-attn and specific versions of torch and transformers. Check the README's updates list for your model.

05The catch: it's slower

Nothing is free. Because layers stream in from disk, AirLLM is much slower than running a model that fits fully in memory. The README itself says disk loading is the main bottleneck. A fast SSD helps.

So I see it as a way to try a huge model on hardware you already own, or to run batch jobs where you don't mind waiting. It's not a replacement for a snappy chat app.

06Tip: save disk space and speed things up

07Good to know

Free and open source under Apache-2.0. You only pay for your own hardware and electricity.
Your prompts stay on your machine. Nothing goes to a cloud API.
The README lists supported model families: Llama, Qwen, DeepSeek, Mistral and Mixtral, Phi, Gemma, ChatGLM, Kimi K3 and more.
The big numbers in the README were measured on specific GPUs. Your speed and memory use may differ.
If something breaks, read the FAQ at the bottom of the README first. Disk space is the usual suspect.

Sources

AirLLM on GitHub (README)