Serving LLMs with Dev Containers: A New Rabbit Hole One of the interesting things about working with AI is that you often discover new ideas when you’re exploring something completely different. I never imagined I would write about serving Large Language Models (LLMs) using Dev Containers. Although I’ve been using the vLLM container for quite some time, I always thought of it simply as another containerized application running in Kubernetes or Docker.

Quantization techniques were mentioned in huggingface and unsloth, and I used those quantized models in ollama and llama.cpp. I always wonder how to implement it for vllm. Today I learnt to use llmcompressor to optimized models for vllm. import warnings warnings.filterwarnings("ignore") import os, gc, math, pathlib import torch from transformers import AutoTokenizer, AutoModelForCausalLM import warnings os.environ['TOKENIZERS_PARALLELISM'] = 'false' MODEL_DIR = "Qwen3-0.6B" OUTPUT_DIR = "Qwen3-0.6B-W4A16" print(f"Base model: {MODEL_DIR}") print(f"Quantized model: {OUTPUT_DIR}") from llmcompressor.

Finally, I managed to get vLLM running on my AMD AI MAX+ 395 GPU on Ubuntu 24.04. It was not straightforward — ROCm support on Ryzen AI (gfx1151) is still evolving, and I ran into multiple low-level GPU faults before finding a stable setup. This post documents: - What didn’t work - The errors I encountered - The working configuration Hopefully this saves you a few hours (or days).

Introduction Several days ago, I setup ollama on my MacBook M4, and it works pretty well. At that time, I tried to use it with copilt with local models codegemma:7b and qwen3:8b. My expectation was not so high as the hardware configuration of my macbook pro m4 is just a entry level, just want to see how it works. I also learned there are other options such as vllm. After comparing the two, I found vllm is more flexible, powerful, product-ready, used widely in enterprises.