You can run DeepSeek with vLLM by loading a supported model checkpoint and serving it through a local API. Your application sends a request to that server; vLLM runs the model on your hardware and returns the answer. This is useful when you want to choose the exact model files, keep inference on infrastructure you control, or share one GPU-backed endpoint between applications.
For your first setup, start with DeepSeek-R1-Distill-Qwen-1.5B. It is small enough to make installation problems easier to separate from the demands of a large deployment. The walkthrough below uses that same model from installation to the first Python request. Once the connection works, you can move to a larger distilled model or a hardware-specific recipe for full R1, V3.2, or V4.
If you mainly want a desktop chat interface, the LM Studio guide is a more direct route. If you want an application endpoint without running GPUs, use the hosted DeepSeek API. vLLM makes sense when managing the model server is part of what you need.
On this page: Requirements · Installation · Start the server · Python and streaming · Larger models · Thinking and tools · Memory and performance · Shared deployments · Troubleshooting · FAQ
What you need before installing vLLM
The main walkthrough uses Linux, an NVIDIA GPU, and Python 3.12. vLLM’s current GPU installation documentation lists Linux, Python 3.11–3.14, and NVIDIA compute capability 7.5 or higher. The commands belong in a Linux terminal. They are not Windows PowerShell commands. Check the official GPU installation requirements against your card and driver before downloading a model.
On Windows, use a compatible WSL environment rather than expecting native Windows installation. AMD GPUs need the ROCm installation path. Apple Silicon has a separate vLLM-Metal route; installing the NVIDIA package on a Mac does not make CUDA available. These are different backend choices, so settle the platform before debugging the model.
You also need space for the checkpoint download and a working GPU driver. A model’s file size is only part of the memory calculation: the running server also needs space for its working data, attention cache, and other runtime allocations. Context length and simultaneous requests affect how much room remains.
There is no single VRAM requirement for “DeepSeek with vLLM.” The 1.5B distilled checkpoint and a full mixture-of-experts model are very different workloads. In particular, an MoE model’s active parameter count does not tell you how much total weight storage it needs. For a broader explanation of hardware sizing, see DeepSeek system requirements.
Before continuing on NVIDIA, run:
nvidia-smiYou should see the GPU and its driver information. If that command cannot communicate with the driver, fix the driver or GPU access first. Reinstalling a Python package will not make a GPU appear inside an environment that cannot see it.
1. Install vLLM in a clean environment
Install uv using its official installation instructions if it is not already available. Then create a working directory and an isolated Python environment:
mkdir -p deepseek-vllm
cd deepseek-vllm
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
uv pip install openaiThis follows the NVIDIA installation approach in the vLLM quickstart. The automatic backend selection helps choose a PyTorch package for the detected driver; it does not install or repair the system’s GPU driver. A fresh environment also avoids replacing packages that another project depends on.
The openai package is the Python client used later in the guide. Installing it does not route your prompts to OpenAI: the client’s base_url determines which server receives the request.
Check the installed vLLM version and whether PyTorch can see CUDA:
vllm --version
python -c "import torch; print('CUDA available:', torch.cuda.is_available()); print('GPU count:', torch.cuda.device_count())"For this NVIDIA path, CUDA availability should be True and the GPU count should be greater than zero. Keep the version number with your launch command. It becomes useful if a later upgrade changes a parser or introduces a backend problem.
2. Start a DeepSeek-R1-Distill server
Run this command in the activated environment:
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
--host 127.0.0.1 \
--port 8000 \
--served-model-name deepseek-local \
--reasoning-parser deepseek_r1 \
--max-model-len 4096 \
--max-num-seqs 4The checkpoint and reasoning parser follow vLLM’s documented R1-Distill example. The local address, alias, and smaller serving limits make this walkthrough easier to follow; they are starting settings, not a claim about the model’s maximum capability or a guarantee that it fits every GPU.
| Setting | What it does here |
|---|---|
127.0.0.1:8000 | Keeps the HTTP endpoint on the same machine for the initial setup. |
deepseek-local | Gives the loaded checkpoint a short API name. It does not change the underlying model. |
deepseek_r1 | Parses the R1 model’s reasoning separately from its final answer. |
4096 context limit | Bounds the combined prompt and generated output accepted for a request. |
4 sequences | Limits how many sequences the server processes together. |
The first start may spend time downloading weights, loading them, and preparing execution. Wait until the logs show that the server has started successfully before sending requests. A download progress bar finishing is not the same as the server being ready.
Leave this terminal running. Open a second terminal for the client commands below. Pressing Ctrl+C in the server terminal stops this local service; it does not delete the downloaded checkpoint.
3. Confirm the model name and send a request
From the second terminal, ask the server which model it exposes:
curl http://127.0.0.1:8000/v1/modelsThe returned list should include deepseek-local, because that is the alias in the launch command. If you omit --served-model-name, vLLM normally uses the checkpoint name instead. The engine argument reference explains this mapping. Use the returned name in your requests rather than borrowing a model ID from a hosted service.
Now send a short chat request:
curl http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "deepseek-local",
"messages": [
{"role": "user", "content": "What is 17 multiplied by 19? Give the answer briefly."}
],
"temperature": 0.6,
"top_p": 0.95,
"max_tokens": 1024
}'Look for a response containing choices, a message, and a finish_reason. The arithmetic answer is 323; the exact wording and reasoning can vary. Receiving JSON from /v1/models confirms discovery, while completing this request checks that the loaded model can generate a response.
If finish_reason is length and the final answer is empty, the model may have used the output allowance on reasoning. Give the request more output room while keeping the prompt and output within the server’s context limit. The 1,024-token allowance above is a modest first request, not a suitable budget for every reasoning task.
4. Connect a Python application
In the second terminal, return to your working directory and activate the same environment:
cd deepseek-vllm
source .venv/bin/activateIf the terminal is already in that directory, only the activation command is needed. Save the following as client.py:
from openai import OpenAI
client = OpenAI(
base_url="http://127.0.0.1:8000/v1",
api_key="EMPTY",
timeout=300.0,
)
response = client.chat.completions.create(
model="deepseek-local",
messages=[
{"role": "user", "content": "What is 17 multiplied by 19? Give the answer briefly."}
],
temperature=0.6,
top_p=0.95,
max_tokens=1024,
)
choice = response.choices[0]
message = choice.message
print("Reasoning:", getattr(message, "reasoning", None) or "")
print("Answer:", message.content or "")
print("Finish reason:", choice.finish_reason)Run it with python client.py. This calls the model already loaded by your local server; it does not start a second model process. The placeholder key works here because the walkthrough has not enabled authentication. If you later configure a real server key, the client must send that key.
Current vLLM uses reasoning for the separate reasoning field. Older vLLM examples may call it reasoning_content. Check the actual response from your installed version when adapting an existing application. Keeping reasoning and the final answer separate also lets your interface display the answer without manually cutting text out of <think> tags.
The official R1 model card recommends a temperature between 0.5 and 0.7, with 0.6 as its suggested choice, and putting instructions in the user prompt instead of a system message. That is why the example is written this way. Revisit those settings when changing model families; they are not universal DeepSeek defaults.
Stream the answer without losing reasoning chunks
For an interactive application, replace the request and printing section of client.py with the following. Keep the client initialization above it:
stream = client.chat.completions.create(
model="deepseek-local",
messages=[{"role": "user", "content": "Explain why a longer prompt can slow down an AI reply."}],
temperature=0.6,
top_p=0.95,
max_tokens=1024,
stream=True,
)
reasoning_parts = []
finish_reason = None
for chunk in stream:
if not chunk.choices:
continue
choice = chunk.choices[0]
delta = choice.delta
reasoning = getattr(delta, "reasoning", None)
if reasoning:
reasoning_parts.append(reasoning)
if delta.content:
print(delta.content, end="", flush=True)
if choice.finish_reason is not None:
finish_reason = choice.finish_reason
print("\nFinish reason:", finish_reason)
# Inspect reasoning_parts separately if your application needs them.Some chunks can carry reasoning before any answer text arrives. Others may have no choices or no content. Handling those cases avoids treating a normal stream as a failed request. This example prints answer text as it arrives and keeps reasoning separate in memory.
Moving from a small model to full DeepSeek releases
A successful 1.5B request establishes that your installation and client can communicate. It does not establish the quality, memory use, or speed of another checkpoint. Choose the next model for the work you need it to do, then check that release’s deployment instructions.
| Model route | Why you might choose it | What changes from the walkthrough |
|---|---|---|
| R1-Distill 7B, 14B, or 32B | Explore a larger distilled reasoning model on suitable hardware. | Weight memory grows; context, precision, and GPU count need another sizing pass. |
| Full R1 / R1-0528 | Serve the full reasoning model rather than a smaller distilled derivative. | Use the R1 deployment recipe for the selected checkpoint and hardware. |
| DeepSeek-V3.2 | Run an application built around V3.2’s thinking and tool-use behavior. | Use its own tokenizer and parsers, plus a suitable multi-GPU layout. |
| DeepSeek-V4.1-Flash | Self-host this text-and-image checkpoint. | Use the dedicated V4.1 image and deployment recipe, including its memory requirements. |
| DeepSeek-V4-Pro-0813 | Run the Pro release on substantial infrastructure. | Follow its model card and hardware-specific V4-Pro recipe. |
For example, DeepSeek’s R1 documentation includes a 32B distilled-model launch using two-way tensor parallelism:
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
--host 127.0.0.1 \
--port 8000 \
--served-model-name deepseek-local \
--tensor-parallel-size 2 \
--max-model-len 32768 \
--enforce-eager \
--reasoning-parser deepseek_r1This adapts the official 32B serving example to the local address and client name used here, and adds reasoning parsing. Stop the first server before reusing port 8000. Two GPUs alone are not a memory guarantee: their capacity, the checkpoint precision, and your serving limits still matter. The 32,768-token limit is part of that example, not a requirement to reserve that much context for every workload.
V4.1-Flash and V4-Pro need their own recipes
The V4.1-Flash checkpoint accepts image and text input and generates text. Its small active parameter counts should not be mistaken for the size of the model you must load: the release includes a much larger backbone and Engram memory. Plan for the actual stored weights and the deployment layout.
The dedicated vLLM V4.1-Flash recipe currently specifies vLLM 0.30.0 or later and a suitable container image. Its prerequisites say the architecture requires the image route rather than the ordinary pip-wheel setup. It also uses V4.1-specific tokenizer and parser settings. A bare command copied from a generic model-hosting widget leaves out those requirements.
Select your GPU platform and deployment layout in that recipe, then retain its matching image and flags as one configuration. The older V4-Flash recipe is not a substitute. If you enable image handling, confirm it with an image request as well as a text request; a successful text answer does not exercise the vision path.
For V4-Pro-0813, DeepSeek publishes a reference configuration for a single node with four GB300 GPUs, including expert parallelism and DSpark speculative decoding. That explains the scale of the example, but does not establish a universal minimum. Use the V4-Pro recipe for your hardware instead of applying that profile to an unrelated set of GPUs.
Configure thinking and tool calling for the model you serve
A reasoning parser controls how the server separates reasoning from answer text. A tool parser interprets a supported model’s tool-call format. Neither flag teaches an arbitrary checkpoint a new capability. The small R1-Distill setup above is a reasoning example, so do not turn it into an agent server merely by adding a tool-call flag.
For standard V3.2, the vLLM V3.2 recipe uses the following tokenizer and parser options as part of its larger launch configuration:
--tokenizer-mode deepseek_v32
--tool-call-parser deepseek_v32
--enable-auto-tool-choice
--reasoning-parser deepseek_v3These are options to add to the complete V3.2 recipe, not a standalone shell command or a replacement for its GPU and backend settings. The recipe enables thinking in requests through extra_body={"chat_template_kwargs": {"thinking": True}}. V4.1 uses different model-specific handling; keep the request configuration paired with the model’s recipe.
Tool use also needs an application loop. Your program sends tool definitions, receives a requested tool and its arguments, validates what is allowed, executes the operation, and returns the result to the model. vLLM serving a tool-capable checkpoint does not automatically grant the model access to your shell, database, or files. The vLLM tool-calling documentation explains the supported modes and parsers.
Handle an absent tool call as well as a returned list. A check such as if message.tool_calls: covers both an empty list and None; then validate each actual call before execution. For a multi-step tool task, confirm that your application preserves the required conversation messages, rather than sending each turn as an unrelated request.
Standard V3.2 and Speciale are different choices here. The Speciale model card explicitly excludes tool calling. Choose it for an appropriate reasoning workload, not for an agent workflow that depends on tool invocation.
Fit the model first, then tune performance
Start with one short request and a modest context limit. Once that works, increase the prompt length and request volume toward what your application actually needs. Changing model size, context, concurrency, precision, and execution mode at once makes a failure much harder to explain.
If memory is tight, the vLLM memory guide describes reducing --max-model-len and --max-num-seqs, using supported quantization, or splitting a model across GPUs. Disabling CUDA graph execution with --enforce-eager can also reduce graph-related memory demands, with a possible performance tradeoff. None of these turns an oversized checkpoint into a small one.
Quantization is a choice of compatible model and runtime support, not just a filename change. A lower-precision checkpoint may need particular kernels or hardware. If you are choosing between downloads, the GGUF and Safetensors guide explains the distinction between file format and weight precision.
On several GPUs, tensor parallelism splits model computation between devices; data parallelism increases request-serving capacity with parallel workers; and expert parallelism distributes an MoE model’s experts. These options address different bottlenecks. More GPUs can introduce more communication, so increasing a parallelism number does not guarantee a faster response. Use the model recipe’s layout as a starting point and evaluate it on the target hardware.
Keep the model’s generation settings in view as well. vLLM can load a repository’s generation_config.json by default; --generation-config vllm selects vLLM defaults instead. Two deployments using different inherited sampling settings are not equivalent merely because their model names match.
Measure the experience your application needs
The vLLM serving benchmark CLI supports controlled request workloads. Begin at low load on your own endpoint, then increase it gradually. For an application used by several people, record both throughput and response latency: a server can produce more tokens overall while making each user wait longer.
Time to first token tells you how long a user waits before output begins. Time per output token helps explain the pace after that. Also record failed requests, simultaneous requests, input and output lengths, and whether the server was freshly started or already warm. A coding assistant with long prompts needs a different workload from a short-answer support bot.
Keep a record of the model revision, vLLM version or container image, launch flags, precision, GPU layout, and sampling parameters with each result. A change in any of them can explain a performance difference. Treat published benchmark results as evidence about their reported configurations, not a speed guarantee for your server.
Before sharing the endpoint with other applications
The walkthrough binds the HTTP server to 127.0.0.1 for a local first run. When another machine needs access, choose the intended private network and access controls before changing that binding. A browser app should normally call your application backend, which can enforce user access and request limits, rather than hold a shared server credential in frontend code.
vLLM supports API-key authentication, but its security documentation makes an important distinction: --api-key protects specified API routes, not every endpoint the service can expose. Use a firewall and an authenticated reverse proxy with a restricted route allowlist for shared deployments. Add TLS where traffic crosses an untrusted network, and keep distributed-worker and administrative ports private.
Decide what the service records. Prompt text, generated answers, uploaded material, and tool arguments can end up in application logs even when inference is local. Give tool execution only the permissions it needs, validate arguments, and keep model-serving credentials separate from access to other systems. Local hosting gives you control over these choices; it does not make them automatic.
Common DeepSeek vLLM problems
The installation succeeds, but the GPU is unavailable
Compare the two checks at the start of the guide. If nvidia-smi works but PyTorch reports no CUDA device, focus on the Python environment and its installed backend. If neither sees the GPU, focus on the driver, WSL or container access, and device visibility. Use the installation instructions for the actual GPU platform rather than repeatedly reinstalling the same wheel.
If the error specifically says BF16 is unsupported, check the GPU and selected dtype. The engine reference includes --dtype half for FP16. That can address a precision mismatch on suitable hardware, but it cannot fix an unsupported architecture or an insufficient-memory error.
The server runs out of GPU memory
First identify when it happens. Failure while loading weights points toward model capacity or the loading setup. Failure while preparing graphs or allocating cache points toward runtime settings. Failure only after you increase traffic suggests the serving workload also matters.
Close unrelated GPU workloads you control, lower the context and sequence limits, and retry with the 1.5B checkpoint. If the weights themselves cannot fit, select a smaller or supported quantized model, or use an appropriate multi-GPU layout. Lowering the requested output length alone will not make the model’s weights smaller.
The client gets connection refused or model not found
For connection refused, check that the server process is still running and listening on the same host and port as the client. From a different computer or container, 127.0.0.1 refers to that client’s own environment, not automatically to the GPU server.
For model not found, repeat GET /v1/models against the same base URL your application uses. Copy its served ID into the request. In this walkthrough the ID is deepseek-local; changing it to the name of a hosted DeepSeek model does not load a different checkpoint.
A tokenizer, chat template, or parser is missing
Check the exact checkpoint and vLLM version together. Newer DeepSeek releases can need a dedicated prompt encoder, parser, or container build. Use the matching model recipe instead of attaching an unrelated model’s template just to clear the error. If a recipe calls for --trust-remote-code, understand that it permits repository-supplied code to run, and review the repository and revision you trust.
There is reasoning but no final answer
Inspect the finish reason before changing the parser. A length stop can mean the model ran out of output budget before finishing its reasoning. Allow more output within the context limit, or select an appropriate thinking setting where that specific model supports it. When the answer exists but your app does not display it, inspect how the client handles content and streamed chunks.
If the separate reasoning field is missing entirely, check the launch parser and installed vLLM response format. A parser mismatch, an older field name, and a model that did not generate reasoning are different problems; printing the raw response during development helps distinguish them.
The response is slow or differs from another service
Separate startup time from request latency. Downloading and preparing a checkpoint happen before normal serving. Once the server is ready, compare a short prompt at low load with your real workload. Long inputs, long generated reasoning, and queueing under load can each account for a slow reply.
For differences in answers, compare the checkpoint revision, quantization, prompt formatting, inherited generation settings, and request parameters. An OpenAI-compatible endpoint is an integration interface, not a promise that a self-hosted model reproduces a provider’s service exactly.
Frequently asked questions
Do I need a DeepSeek API key to run vLLM?
No. Serving downloaded model weights does not require a DeepSeek hosted-service key. The local example uses EMPTY to satisfy the client configuration. A key that you configure on your own server is a separate credential, and restricted model repositories may separately require download authorization.
Can I use this guide on Windows or a Mac?
The commands here target Linux with an NVIDIA GPU. Windows users need a compatible Linux environment such as WSL. Mac users should follow the separate Apple Silicon backend instructions or choose a desktop runtime suited to their machine. The NVIDIA installation commands are not a cross-platform installer.
Can I run full DeepSeek V4 on a normal PC?
The full V4 deployments discussed here are infrastructure-scale workloads. A successful small R1-Distill setup does not show that your PC can load V4. Choose from the model’s documented hardware profiles before committing to a large download or a rented GPU server.
Can I use Docker instead of a Python environment?
Yes. vLLM provides a Docker deployment path, and some new-model recipes require a particular image. Use an image that supports your checkpoint and GPU platform, and configure GPU access, storage, and networking deliberately. For repeatable deployments, record the image version instead of relying only on a moving tag.
Is self-hosting DeepSeek with vLLM free?
You are not paying DeepSeek’s hosted API for these local requests, but the hardware, electricity or rented GPU time, storage, and maintenance still cost something. Check the checkpoint’s license for your intended use. Whether self-hosting is economical depends on your workload and operations, not simply on the availability of downloadable weights.
Can my vLLM server accept images?
Only when the selected checkpoint and runtime support image input. V4.1-Flash has a vision path; the R1-Distill checkpoint used in the first-run example does not gain vision by sharing the same server software. A file ID issued by a hosted service is also not automatically meaningful to your local server.
Does API compatibility include every completion feature?
No. Check the endpoint you intend to use. For example, vLLM’s online-serving documentation says its Completions API does not support suffix. Do not move a fill-in-the-middle workflow to that endpoint merely because ordinary chat requests already work.
Build on a working first request
Get the small server and client working together, then try a few representative prompts from your application. Keep the configuration that produced those answers. From there, you can decide whether the next useful change is a larger model, a longer context, tool support, or more serving capacity—and check that change on its own. That gives you a usable starting point for a DeepSeek service instead of a collection of launch commands that belong to different models.