https://jackliusr.github.io/tags/llmcompressor/ Recent content in llmcompressor on My learning and diary Hugo -- gohugo.io en-us Mon, 20 Jul 2026 20:00:00 +0800 https://jackliusr.github.io/posts/2026/07/optimize-llms-for-vllm-deployment/ Mon, 20 Jul 2026 20:00:00 +0800 https://jackliusr.github.io/posts/2026/07/optimize-llms-for-vllm-deployment/ Quantization techniques were mentioned in huggingface and unsloth, and I used those quantized models in ollama and llama.cpp. I always wonder how to implement it for vllm. Today I learnt to use llmcompressor to optimized models for vllm. import warnings warnings.filterwarnings("ignore") import os, gc, math, pathlib import torch from transformers import AutoTokenizer, AutoModelForCausalLM import warnings os.environ['TOKENIZERS_PARALLELISM'] = 'false' MODEL_DIR = "Qwen3-0.6B" OUTPUT_DIR = "Qwen3-0.6B-W4A16" print(f"Base model: {MODEL_DIR}") print(f"Quantized model: {OUTPUT_DIR}") from llmcompressor.