A 24GB GPU can run a ~30b parameter model at 4bit quantization at about 8k-12k c... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		hnuser123456 9 months ago \| parent \| context \| favorite \| on: Qwen3-4B-Thinking-2507 A 24GB GPU can run a ~30b parameter model at 4bit quantization at about 8k-12k context length before every GB of VRAM is occupied.

iamnotagenius 9 months ago [–]

Not quite true. Depends on number of KV heads. GLM4 32b at IQ4 quant and Q8 context can run full context with only 20GiB VRAM.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact