To run DeepSeek locally with Docker, use Ollama to serve a model and Open WebUI for the chat interface. The Compose setup below keeps model downloads and conversations in separate volumes and exposes the interface only on your computer. You can start with a small DeepSeek R1 distilled model, then enable GPU access or choose a larger model when your hardware allows it.
If you want DeepSeek in a containerized application without hosting model weights, use the API-backed Docker example instead. In that setup, Docker runs your application and DeepSeek processes its requests remotely. The distinction matters for hardware, costs, and where your prompts go.
Table of Contents
Choose what Docker will run
| Your goal | Starting point | What you operate |
|---|---|---|
| Chat with a model on your own machine | Ollama + Open WebUI | Model runtime, downloaded weights, and chat interface |
| Add DeepSeek to an application without managing GPUs | An application container calling the hosted API | Your application; DeepSeek operates inference |
| Use Docker’s built-in model tooling | Docker Model Runner | A supported local model package and inference engine |
| Serve a model to several applications on GPU infrastructure | vLLM or a model-specific SGLang recipe | An inference service, capacity, and access controls |
For a first local installation, follow the Ollama path from beginning to end. For an existing web application, jump to the API example. These are alternatives: you do not need to install every runtime in this guide.
The names belong to different systems. ollama/ollama is a container image; deepseek-r1:1.5b is a model tag that Ollama downloads; deepseek-flash is a hosted API model ID. Pulling the Ollama image does not also download a DeepSeek model. Our local versus API comparison explains the wider trade-offs if you have not chosen where inference should happen.
What you need before starting
Install Docker with Compose v2, and confirm that the daemon is running. The commands below use a Bash-compatible terminal on Linux, macOS, or WSL on Windows. They assume Linux containers; shell quoting and line continuations differ in PowerShell.
docker version
docker compose versionBoth commands should report version information; docker version should also reach the server. If it only shows the client and a connection error, start Docker Desktop or the Docker service before continuing. You also need internet access for the initial images and model download, free disk space for both, and enough memory for the model plus the rest of your system.
The walkthrough uses deepseek-r1:1.5b, which Ollama lists as a roughly 1.1 GB Q4_K_M package. That is a download size, not a RAM requirement. It is a small distilled model for getting the deployment working, not a replacement for the full DeepSeek R1 or the current hosted models. DeepSeek’s R1 repository distinguishes the full model from its distilled releases.
Allow extra memory for the runtime, conversation context, and Open WebUI. There is no single “DeepSeek Docker RAM requirement”: changing the model, quantization, context, or number of simultaneous requests changes the budget. See the DeepSeek hardware guide before committing to a larger download.
| Host | What to expect |
|---|---|
| Linux | The basic stack runs on CPU. NVIDIA acceleration needs a supported GPU, driver, and Container Toolkit; AMD follows a different device and image configuration. |
| Windows with Docker Desktop | Use Linux containers and a WSL 2 terminal for these commands. NVIDIA container acceleration requires the WSL 2 backend and compatible Windows drivers. |
| macOS | The Ollama container can run on CPU. Ordinary Docker Desktop containers do not get Apple GPU passthrough. Native Ollama or Docker Model Runner is a separate route to Apple Silicon acceleration. |
Docker documents the Windows GPU prerequisites; Ollama explains the macOS container limitation. Docker Model Runner has its own platform requirements, so do not apply a container GPU limitation to every Docker AI feature.
Run DeepSeek with Docker Compose and Open WebUI
1. Create a project and keep one WebUI secret
Create a new directory for this installation and work inside it. Keep this directory when updating: Compose uses the project name to identify its resources.
mkdir deepseek-docker
cd deepseek-dockerIn this new directory, create a .gitignore containing the following entries. These files can contain secrets or private chat data and should stay out of source control.
.env
secrets/
backups/Generate a random value with openssl rand -hex 32, then create a .env file and paste it after WEBUI_SECRET_KEY=. Do this once, and keep the same value across restarts and updates. This is Open WebUI’s application secret, not a DeepSeek API key; the local model does not need a DeepSeek account.
WEBUI_SECRET_KEY=replace_with_your_generated_random_valueOn Linux or macOS, restrict the file with chmod 600 .env. On Windows, restrict access using the file’s permissions. Open WebUI recommends a persistent secret key so recreating the container does not invalidate everyone’s sessions. Do not regenerate the key each time you start Compose.
2. Save the Compose configuration
Create compose.yaml beside .env. This uses the separate Ollama and WebUI services shown in the Open WebUI project’s Compose configuration, with the interface bound to localhost and a required secret value.
services:
ollama:
image: ollama/ollama:latest
volumes:
- ollama:/root/.ollama
restart: unless-stopped
open-webui:
image: ghcr.io/open-webui/open-webui:main
ports:
- "127.0.0.1:3000:8080"
environment:
OLLAMA_BASE_URL: "http://ollama:11434"
WEBUI_SECRET_KEY: "${WEBUI_SECRET_KEY:?Set WEBUI_SECRET_KEY in .env}"
volumes:
- open-webui:/app/backend/data
depends_on:
- ollama
restart: unless-stopped
volumes:
ollama:
open-webui:The two containers communicate over Compose’s default network using the service name ollama. The Ollama API does not need a published host port for WebUI to reach it. Inside the WebUI container, localhost would mean WebUI itself, so http://localhost:11434 would be the wrong address for this arrangement. This follows Docker’s service discovery rules.
The latest and main tags make this a convenient starting configuration, but they can change. Before relying on the installation for other people, select reviewed release tags or image digests. Open WebUI’s main tag follows its main branch; it is not a guarantee of a fixed stable release.
3. Start the services and download the model
docker compose config --quiet
docker compose up -d
docker compose ps
docker compose logs --tail=50 ollama open-webuiThe first command checks the Compose configuration without printing its expanded secret value. The remaining commands start the containers and show their state. Allow startup to finish before opening the interface. depends_on sets startup order here; it does not prove that the model service is ready to answer.
docker compose exec ollama ollama pull deepseek-r1:1.5b
docker compose exec ollama ollama list
docker compose exec ollama ollama run deepseek-r1:1.5bWait for the download to complete, then enter a short prompt in the interactive session. An answer confirms that the selected local model can load and generate text. Exit with /bye. This check is separate from the WebUI connection: it helps you tell a model-loading problem from an interface problem. Ollama documents this container and model workflow.
4. Open the chat interface
Open http://127.0.0.1:3000 in your browser, create the first administrator account, and select deepseek-r1:1.5b for a new chat. The model you pulled should be available once the Ollama connection is active. For connection settings, use your avatar → Settings → Admin → Connections, as described in Open WebUI’s provider guide.
If the model list is empty, check that the Ollama connection points to http://ollama:11434 and that ollama list shows the model. An existing WebUI data volume may retain earlier connection settings, so check the saved setting as well as the Compose environment. Do not delete the volume just to reset a connection; it also holds your conversations and account data.
Keep the first installation local while creating the administrator account. If Docker runs on a remote server, localhost refers to that server. Use an SSH tunnel, for example ssh -L 3000:127.0.0.1:3000 user@server, then open the same localhost URL on your own computer. Replace the user and server with your actual SSH destination.
5. Use the local model from another application
You can stop at the browser interface. If a program on the host also needs the Ollama API, add this ports entry under the existing ollama service, keeping its image and volume entries. Then run docker compose up -d again.
ports:
- "127.0.0.1:11434:11434"From the host terminal, send a request to Ollama’s chat endpoint:
curl -sS http://127.0.0.1:11434/api/chat \
-H "Content-Type: application/json" \
-d '{
"model": "deepseek-r1:1.5b",
"messages": [{"role": "user", "content": "Explain a Docker volume in two sentences."}],
"stream": false
}'With streaming disabled, the response is one JSON object; read the answer from message.content. A different container in this Compose project should call http://ollama:11434, not the published host address. Keep the exact model tag consistent in the download command, interface, and API request.
Give Ollama access to a GPU
CPU execution is enough to establish the workflow, but a larger model may be too slow for your needs. On Linux with NVIDIA hardware, first check nvidia-smi on the host and install the NVIDIA Container Toolkit for your distribution. The documented rootful Docker configuration includes the commands below; restarting Docker can interrupt other containers on that host.
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart dockerThen add the following under your existing ollama service at the same indentation level as image and volumes. This is Docker’s Compose GPU reservation syntax.
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]docker compose up -d
docker compose exec ollama ollama run deepseek-r1:1.5b
docker compose exec ollama ollama psExit the interactive chat before the last command, or run it from another terminal while the model is loaded. The PROCESSOR column distinguishes CPU, GPU, and mixed placement. A GPU visible to Docker does not prove that the whole model fits in VRAM.
Check the exact card against Ollama’s hardware support list. It currently lists NVIDIA compute capability 5.0+ with driver 550+, with driver 570+ needed for compute capability 5.0–6.2. These are Ollama requirements, not universal requirements for vLLM or every container image.
For supported AMD hardware on Linux, Ollama documents the ollama/ollama:rocm image with /dev/kfd and /dev/dri passed through. Follow that hardware-specific route instead of adding an NVIDIA reservation. Changing only Open WebUI to a CUDA image accelerates WebUI’s own supported local processing; Ollama still needs GPU access in its separate container.
Keep models and chats through restarts and updates
The ollama volume stores downloaded models under /root/.ollama. The open-webui volume stores this installation’s database and files under /app/backend/data. Docker volumes outlive a container, which lets you replace an image without downloading every model or starting your chat history again.
To pause the stack, use docker compose stop; use docker compose start to bring the existing containers back. docker compose down removes the containers and network but normally keeps these named volumes. Do not add -v unless you intend to delete the project’s volumes and their data.
Back up before upgrading
A persistent volume is not a backup: accidental deletion, disk failure, and database migrations can still cost you data. For the default local WebUI database in this example, stop WebUI while copying its data directory. From your project folder, the following creates a new timestamped directory using Compose’s copy command.
backup_dir="backups/$(date +%Y%m%d-%H%M%S)"
mkdir -p "$backup_dir"
docker compose stop open-webui
docker compose cp open-webui:/app/backend/data "$backup_dir/webui-data"
docker compose start open-webuiCheck that the copy completed and that the backup contains the database and files before updating. Protect it because it can include chats and credentials. Keep a protected copy of .env and the Compose configuration too. If you later switch to an external database or storage service, back that service up separately; copying this directory alone will no longer cover the installation.
Review the release notes, choose the image versions, and then update:
docker compose pull
docker compose up -d
docker compose ps
docker compose logs --tail=50 ollama open-webuiSign in again and open an existing conversation to verify that the expected data remains. Also send one short request to the selected model. Open WebUI’s update and rollback guide explains why an older image may need the matching pre-update database backup after a migration. Merely changing the image tag back is not always enough.
Updating the Ollama container and updating model weights are separate operations. To refresh the chosen model package, pull the same explicit tag again. To try another size, pull that size and select it in WebUI; do not assume the running model changed because the server image did.
Dockerize an app that calls the hosted DeepSeek API
Choose this path when you want to deploy your own service and let DeepSeek handle inference. It needs a DeepSeek platform key and API balance, but no model download or local GPU. Prompts leave the container for DeepSeek’s servers. The example provides a small text endpoint and a local health check; authentication and user quotas belong in your application before you share it.
Use a separate project directory from the Ollama stack. Create app.py, requirements.txt, Dockerfile, compose.yaml, and a secrets directory. Create both .gitignore and .dockerignore with these exclusions before building:
secrets/
.env
.env.*
.git
__pycache__/
*.pycGit exclusions keep files out of commits; Docker exclusions keep them out of the build context. One does not replace the other. Runtime mounting of a secret still works when its host file is excluded from the build.
The application and image
Put the following in requirements.txt. These unpinned dependencies are suitable for working through the example; lock reviewed versions before a repeatable deployment.
fastapi
uvicorn[standard]
openaiSave this as app.py. The model is selected in server configuration, not supplied by the caller. The request format and base URL follow DeepSeek’s API quick-start.
import logging
import os
from pathlib import Path
from fastapi import FastAPI, HTTPException
from openai import OpenAI, OpenAIError
from pydantic import BaseModel, Field
app = FastAPI()
logger = logging.getLogger(__name__)
key_file = Path(os.environ["DEEPSEEK_API_KEY_FILE"])
api_key = key_file.read_text(encoding="utf-8").strip()
if not api_key:
raise RuntimeError("The mounted DeepSeek key file is empty.")
model = os.getenv("DEEPSEEK_MODEL", "deepseek-flash")
client = OpenAI(
api_key=api_key,
base_url="https://api.deepseek.com",
timeout=60.0,
max_retries=0,
)
class AskRequest(BaseModel):
prompt: str = Field(min_length=1, max_length=20000)
@app.get("/healthz")
def healthz():
return {"status": "ok"}
@app.post("/ask")
def ask(payload: AskRequest):
prompt = payload.prompt.strip()
if not prompt:
raise HTTPException(400, "The prompt cannot be empty.")
try:
response = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
max_tokens=1000,
stream=False,
extra_body={"thinking": {"type": "disabled"}},
)
except OpenAIError as exc:
logger.warning("Provider request failed: %s", type(exc).__name__)
raise HTTPException(502, "Model provider request failed.") from None
if not response.choices:
raise HTTPException(502, "The model returned no choice.")
choice = response.choices[0]
answer = choice.message.content
if not answer:
raise HTTPException(502, "The model returned no answer.")
return {"answer": answer, "finish_reason": choice.finish_reason}The thinking-mode setting is disabled explicitly for this short text example. The response includes finish_reason so your client can identify an answer cut off by the output limit. The health endpoint only checks the local application; it does not contact DeepSeek or prove that the key has usable balance.
Save this Dockerfile. It follows the FastAPI container pattern of installing dependencies before copying application code, then runs the service as numeric user 10001.
FROM python:3.12-slim
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
USER 10001
EXPOSE 8080
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080"]Mount the API key at runtime
Save this separate compose.yaml. Compose secrets give the application a file at runtime. The application code above reads that file; Docker does not automatically interpret a _FILE environment variable for arbitrary applications.
services:
deepseek-app:
build: .
ports:
- "127.0.0.1:8080:8080"
environment:
DEEPSEEK_MODEL: "deepseek-flash"
DEEPSEEK_API_KEY_FILE: "/run/secrets/deepseek_api_key"
secrets:
- deepseek_api_key
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/healthz', timeout=3).read()"]
interval: 30s
timeout: 5s
retries: 3
start_period: 10s
restart: unless-stopped
secrets:
deepseek_api_key:
file: ./secrets/deepseek_api_key.txtFor a rootful Linux Docker daemon without user-namespace remapping, create the key file with the Bash commands below. They accept the key without displaying it or putting its value directly in a command. Run them once for a new file; replacing an existing protected key should be a deliberate rotation.
mkdir -p secrets
chmod 700 secrets
umask 077
read -rsp 'DeepSeek API key: ' deepseek_key_input
printf '\n'
printf '%s' "$deepseek_key_input" > secrets/deepseek_api_key.txt
unset deepseek_key_input
sudo chown 10001:10001 secrets/deepseek_api_key.txt
sudo chmod 600 secrets/deepseek_api_key.txtThe file must be readable by the container’s UID 10001. Compose’s file-backed secret rules do not remap ownership through YAML uid, gid, or mode. Rootless Docker, Docker Desktop, and remapped daemons can need different host permissions. Use the read check below before starting; do not solve a permissions error by making the key world-readable.
docker compose build
docker compose run --rm --no-deps --entrypoint sh deepseek-app \
-c 'test -r /run/secrets/deepseek_api_key && test -s /run/secrets/deepseek_api_key && echo "Key file is readable and non-empty"'Continue only if the build succeeds and the read check prints its success message. Neither command sends a model request. File-based Compose secrets still exist on the host, so protect that directory and use your platform’s managed secret facility when deploying to a shared environment.
Start the app and make one request
docker compose up -d
curl -sS http://127.0.0.1:8080/healthz
curl -sS http://127.0.0.1:8080/ask \
-H "Content-Type: application/json" \
-d '{"prompt":"Explain why Docker volumes are useful in two sentences."}'The health request should return {"status":"ok"}. The /ask request uses your paid API balance and should return an answer and finish_reason. A healthy container with a failed /ask request means you need to inspect the provider connection or account state, not necessarily rebuild the image.
This example intentionally has one text route. Add image handling, files, conversation storage, or streaming only when your application needs them. Keep the provider key on the server. If several applications need centrally controlled provider access, a gateway can be useful; LiteLLM documents DeepSeek integration, but it adds another service to secure and maintain.
When to use Docker Model Runner or vLLM
Docker Model Runner: a separate local route
Model Runner manages model packages through docker model. On supported Docker Desktop installations, enable it in Settings → AI. On Linux Docker Engine, install the model plugin using Docker’s platform-specific instructions. It is not another service you must add to the Ollama Compose file.
Docker’s DeepSeek R1 Distill Llama package includes an 8B Q4_K_M variant. If that model fits your hardware, use its exact package name:
docker model version
docker model pull ai/deepseek-r1-distill-llama:8B-Q4_K_M
docker model run ai/deepseek-r1-distill-llama:8B-Q4_K_MThis is a different distilled checkpoint from the Qwen-based 1.5B Ollama example. On macOS and Windows, Model Runner’s engines run in a sandbox rather than ordinary containers; this is why its hardware support differs from container GPU passthrough.
For host applications, enable host-side TCP support and use http://localhost:12434/engines/v1 as the OpenAI-compatible base URL. Containers on Docker Desktop use http://model-runner.docker.internal/engines/v1. Docker Engine networking differs; follow the Model Runner API reference for its host-gateway configuration.
curl -sS http://localhost:12434/engines/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "ai/deepseek-r1-distill-llama:8B-Q4_K_M",
"messages": [{"role": "user", "content": "What does a Docker volume preserve?"}],
"max_tokens": 1024,
"temperature": 0.6,
"stream": false
}'The response uses the chat-completions structure. Keep the namespace and tag in the request consistent with the downloaded package. Model Runner’s API is unauthenticated by default; making its port reachable also exposes model operations. Keep it restricted to trusted access.
vLLM: a containerized inference server
For a GPU server that will serve applications, vLLM gives you an OpenAI-compatible endpoint. With a supported NVIDIA host and Container Toolkit configured, the following adapts vLLM’s official Docker image to the small R1 distilled checkpoint. The 4,096-token context is a conservative example setting, not the model’s maximum.
docker run --rm --name deepseek-vllm --gpus all \
-v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
-p 127.0.0.1:8000:8000 \
--ipc=host \
vllm/vllm-openai:latest \
--model deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
--served-model-name deepseek-local \
--max-model-len 4096 \
--reasoning-parser deepseek_r1Wait for the server to finish loading. The cache mount preserves downloaded weights; --ipc=host supplies shared-memory access used by the serving stack. Use this on a trusted host, and choose a reviewed image version or digest before a repeatable deployment. The R1 reasoning parser separates reasoning output from the final answer.
From a second terminal, check the exposed model and then request a completion:
curl -sS http://127.0.0.1:8000/v1/models
curl -sS http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "deepseek-local",
"messages": [{"role": "user", "content": "What is 17 multiplied by 19?"}],
"max_tokens": 1024,
"temperature": 0.6,
"stream": false
}'Use deepseek-local in the client because the command explicitly sets that served name. Check the final answer against 323; a successful HTTP response alone does not establish answer quality. A response that ends at its token limit may contain incomplete reasoning or no final answer. Stop this foreground example with Ctrl+C when finished; the weights remain in the mounted cache.
Full-size DeepSeek deployments need a recipe for the exact checkpoint and accelerator configuration. Do not replace the model name in this small example and assume the same memory or launch flags will work. DeepSeek’s V4 Pro model card and SGLang’s V4 recipes cover substantially different deployments. Our DeepSeek with vLLM guide explains model-specific setup and client behavior in more depth.
Before sharing the deployment
The examples publish ports on 127.0.0.1 because they start as local installations. Docker warns that published ports can otherwise be reachable outside the host. Keep Docker current and check access from the intended network before sharing a service.
For a team, put the user-facing application behind authenticated access and TLS, and keep the model endpoint private. Apply per-user limits and timeouts so one long request cannot monopolize the service. The API-backed example has no user authentication, and its local health check is not a production readiness guarantee.
Record the image versions, model tag or revision, context settings, and rollback path. Monitor errors, response time, restarts, memory pressure, and—where applicable—provider usage. Keep prompts and credentials out of routine logs, and decide who can read chat backups. Local inference controls where the model runs; optional search, cloud providers, plugins, and external backups can still move data elsewhere.
Troubleshoot the layer that is failing
The interface does not open or the port is occupied
Run docker compose ps and docker compose logs --tail=100 open-webui. Allow initial startup to finish. If port 3000 is already occupied, change only the host side to 127.0.0.1:3001:8080, run docker compose up -d, and open port 3001. The container still listens on 8080.
WebUI opens, but there are no models
Run docker compose exec ollama ollama list. If it is empty, complete the model pull. If the model exists, inspect the WebUI Ollama connection and saved URL. The WebUI configuration reference documents the backend URL settings and the multi-URL setting that can take precedence. Fix that connection before changing model files.
The GPU is unavailable or the model is slow
Check the host driver first, then the Docker GPU configuration, then Ollama’s ps output while a model is loaded. If Docker cannot use the GPU, changing a prompt will not fix it. If the model is partly on CPU, it may exceed available VRAM. Try a smaller model or lower context before buying more hardware; also check the memory allocated to Docker Desktop.
The container exits or reports insufficient memory
Read the Ollama or vLLM startup logs before repeatedly restarting. Model weights, context cache, and concurrent requests all consume memory. Reduce the requested context or model size and leave room for the runtime. A file fitting on disk does not establish that it will fit in RAM or VRAM, and containerization does not shrink it.
The API app is healthy, but requests fail
Run docker compose logs --tail=100 deepseek-app. A missing or unreadable secret prevents startup; repeat the non-printing read check after fixing its permissions. A provider authentication error points to the key, while a network error calls for checking DNS, outbound HTTPS, proxy configuration, and the base URL. Do not print the key to diagnose either case.
The example returns a generic 502 to the caller and logs the error class without the prompt or raw provider response. For production diagnostics, add controlled status and request-ID logging, then handle authentication, balance, rate limits, and transient failures separately. Avoid blind retries: repeated paid requests can create cost without fixing the original problem.
An update appears to have lost chats or models
Check that you are in the same Compose project and using the same volumes. Renaming the project directory can produce a different default project name and new empty volumes. Inspect with docker volume ls and compare your saved configuration before deleting or restoring anything. If the correct volume is present but an upgrade failed, use the matching backup and the application’s rollback instructions.
Questions before deploying
Do I need a DeepSeek API key for the local Docker setup?
No. The Ollama example downloads a local distilled model and runs inference on your hardware. The separate API-backed application does require a DeepSeek key and usable API balance. A WebUI login or WebUI secret is a different credential.
Can this run on a server without a GPU?
The small Ollama model can run on CPU, with speed determined by your processor and available memory. The hosted API application also needs no local GPU. The NVIDIA vLLM example is a different configuration and should not be used unchanged on a CPU-only server.
Will it work without an internet connection?
Local inference can run after the required images and model files are available. Check optional WebUI features and external providers separately, and prepare any extra dependencies before disconnecting. A containerized application calling the hosted DeepSeek API still needs a network connection to that service.
Can I use a larger DeepSeek R1 model in the same Compose stack?
Yes, if Ollama offers the tag and your hardware can run it. Pull the explicit tag, select it in WebUI, and update any client requests that name the model. Keep the smaller working model until the replacement loads and answers satisfactorily; there is no need to rebuild WebUI for a different model size.
Is the local model the same as DeepSeek’s website or hosted API?
No. This walkthrough uses a specific R1 distilled checkpoint. The official services can expose different models, capabilities, and limits. Use the local model for tasks it can handle; do not assume a small distilled download reproduces the hosted service.
Can I make this available to my team?
Yes, after configuring access control, TLS, backups, and capacity for the people using it. Share the application interface rather than opening every model port. Confirm that another user cannot change administration settings or exhaust your model resources or API budget.
Should I use Kubernetes instead of Compose?
Compose is a reasonable starting point for this single-host setup. Kubernetes becomes useful when you already need multi-host scheduling, managed rollout, and shared infrastructure. It does not remove the work of sizing a model, protecting an API key, or backing up application data.
Finish one working deployment before expanding it
For local chat, the useful milestone is a model that answers through WebUI and a conversation that survives a container restart. For the hosted API path, it is a protected key, a healthy application, and one successful request whose usage you can account for. Reach that point first, then improve the model choice, GPU configuration, or team access to match the work you actually need to do.
Before your next upgrade, keep the Compose configuration and a verified data backup together. That makes the installation something you can maintain, not just a command that happened to start once.