Serving LLMs with Dev Containers: A New Rabbit Hole One of the interesting things about working with AI is that you often discover new ideas when you’re exploring something completely different. I never imagined I would write about serving Large Language Models (LLMs) using Dev Containers. Although I’ve been using the vLLM container for quite some time, I always thought of it simply as another containerized application running in Kubernetes or Docker.

Quantization techniques were mentioned in huggingface and unsloth, and I used those quantized models in ollama and llama.cpp. I always wonder how to implement it for vllm. Today I learnt to use llmcompressor to optimized models for vllm. import warnings warnings.filterwarnings("ignore") import os, gc, math, pathlib import torch from transformers import AutoTokenizer, AutoModelForCausalLM import warnings os.environ['TOKENIZERS_PARALLELISM'] = 'false' MODEL_DIR = "Qwen3-0.6B" OUTPUT_DIR = "Qwen3-0.6B-W4A16" print(f"Base model: {MODEL_DIR}") print(f"Quantized model: {OUTPUT_DIR}") from llmcompressor.

OpenShift enhances container security by assigning a random, non-root User ID (UID) to workloads by default. This helps isolate workloads running in different namespaces and prevents containers from running with predictable user IDs. While this security model works well for cloud-native applications, some third-party or legacy container images expect to run with a specific UID. A common example is the nginxinc/nginx-unprivileged image, which expects to run as UID 101.