Self-hosting an open model with vLLM

vLLM is a popular open-source inference server for LLMs. It exposes an OpenAI-compatible API, so most client code works unchanged.

Before you start

1. Pick a model whose weights fit in your GPU memory, with headroom for the KV cache.
2. Choose a host: a GPU cloud pod, a serverless GPU platform or your own machine.
3. Decide how you will store weights so restarts do not re-download them.

First deployment

1. Start from an official vLLM container image.
2. Launch the server with your model name and a sensible maximum context length.
3. Send a test request with an OpenAI-compatible client.
4. Load test with realistic prompts before sending real users.

Going to production

• Put an authenticating proxy or gateway in front of the server.
• Track latency, tokens per second and error rates.
• Plan for model upgrades and rollbacks.