Model serving is the process of making a trained model available for inference — accepting input data and returning predictions. This sounds simple, but production serving must handle: low latency (respond in milliseconds), high throughput (thousands of requests per second), reliability (99.9%+ uptime), scalability (auto-scale with traffic), and efficiency (maximize GPU utilization). There are two main patterns: online serving (real-time, request-response via REST/gRPC) and batch serving (process large datasets offline, results stored for later use). Most production systems use online serving; batch is for non-time-sensitive predictions like recommendations or risk scores.

NVIDIA Triton Inference Server is the most widely used production inference server for GPU workloads. Key capabilities: multi-framework support (PyTorch, TensorFlow, ONNX, TensorRT, vLLM — all in one server), dynamic batching (automatically groups incoming requests into batches for GPU efficiency), model ensembles (chain multiple models in a pipeline), concurrent model execution (run multiple models on the same GPU), and metrics (Prometheus-compatible latency, throughput, and GPU utilization metrics). Triton is free and open-source, deployed via Docker containers. It’s the default choice for teams serving models on NVIDIA GPUs.

vLLM is the dominant open-source inference engine for large language models. Its key innovation is PagedAttention — inspired by virtual memory in operating systems, it manages the KV cache (key-value pairs stored during autoregressive generation) in non-contiguous memory blocks (“pages”). This eliminates memory fragmentation and waste, achieving near-zero KV cache waste compared to ~60–80% waste in naive implementations. vLLM also implements continuous batching — instead of waiting for all requests in a batch to finish, new requests are added as old ones complete, keeping the GPU busy at all times. Result: 2–4x higher throughput than HuggingFace Transformers.

ONNX Runtime (by Microsoft) is a cross-platform inference engine that runs ONNX models with hardware-specific optimizations. Convert your PyTorch/TensorFlow model to ONNX format, and ONNX Runtime applies: graph optimizations (operator fusion, constant folding), hardware acceleration (CUDA, TensorRT, DirectML, OpenVINO), and quantization (reduce precision from FP32 to INT8 for 2–4x speedup). Other optimization techniques: TensorRT (NVIDIA’s optimizer for maximum GPU performance), distillation (train a smaller model to mimic a larger one), and pruning (remove unimportant weights). For LLMs, quantization (GPTQ, AWQ, GGUF) is the primary optimization — reducing from FP16 to INT4 cuts memory by 4x with minimal quality loss.

GPUs are massively parallel processors — they’re designed to process many inputs simultaneously. Processing one request at a time wastes 90%+ of GPU capacity. Batching groups multiple requests together for parallel processing. Three strategies: Static batching (fixed batch size, wait until full — simple but adds latency). Dynamic batching (collect requests for a short window, batch whatever arrived — Triton’s approach). Continuous batching (for LLMs — don’t wait for all sequences to finish; add new requests as old ones complete). The trade-off is always latency vs. throughput: larger batches = higher throughput but higher latency per request.

Latency is how long one request takes (measured in p50, p95, p99 percentiles). Throughput is how many requests per second the system handles. They’re inversely related: optimizing for one hurts the other. Latency-sensitive applications (fraud detection, search autocomplete): minimize batch size, use faster hardware, apply model optimization. Throughput-sensitive applications (batch scoring, content moderation): maximize batch size, use dynamic batching, scale horizontally. For LLMs, two metrics matter: Time to First Token (TTFT) (how fast the first word appears) and tokens per second (TPS) (generation speed). Users perceive TTFT as responsiveness and TPS as fluency.

Vertical scaling: bigger GPU (A100 → H100). Simple but has a ceiling. Horizontal scaling: more replicas behind a load balancer. The standard approach for production. Auto-scaling: automatically add/remove replicas based on metrics (GPU utilization, queue depth, latency). Kubernetes is the standard orchestrator — use KEDA (Kubernetes Event-Driven Autoscaling) or custom HPA (Horizontal Pod Autoscaler) metrics. For LLMs, scaling is more complex: large models may span multiple GPUs (tensor parallelism), so you scale in units of “model replicas” rather than individual pods. KV cache memory is often the bottleneck, not compute.

Choose vLLM if: you’re serving LLMs and need maximum throughput with PagedAttention and continuous batching. Choose Triton if: you serve multiple model types (CV, NLP, tabular) on GPUs and need dynamic batching, ensembles, and multi-model support. Choose ONNX Runtime if: you need cross-platform inference (CPU, GPU, edge) with minimal dependencies. Choose BentoML if: you want a Python-first framework that packages models as Docker containers with minimal boilerplate. Choose a managed service (SageMaker Endpoints, Vertex AI Prediction, Azure ML) if: you want zero infrastructure management and are on that cloud platform.