I have been using Ollama to run local LLM for a long time and I don’t have much to complain about. It does the job and I don’t have to modify many things. Recently, Docker announced its own runner modelwhich offers many of the same features you get with Ollama. You can pull, run, manage, and serve local AI models through Docker Desktop or Docker Engine. Supports GGUF models via llama.cpp, Safetensors models via vLLM, hardware acceleration on supported systems, and models from Docker Hub, Hugging Face, or other OCI registries. I’ve been using it instead of Ollama for almost a week now, but I can’t say I keep it up anymore.
Docker Model Runner fits perfectly into Docker
And it’s very easy to set up
Docker Model Runner works differently than running a model inside a normal container. Docker handles the model as its own OCI artifact, while a dedicated inference engine loads and runs it locally. The model remains cached in your storage after the first download, is loaded into memory when it receives a request, and is unloaded when it is no longer used. This keeps the process familiar if you already use Docker, as the same CLI now handles local containers, images, and models.
For most people, the default llama.cpp backend is the one that matters. Run quantized GGUF models on Apple Silicon, Windows, and Linux, with automatic Metal acceleration on supported Macs. Docker also offers vLLM for higher-performance inference with Safetensors models, although that backend requires a supported NVIDIA GPU and currently has much more limited platform support. You can access any of the backends via OpenAI and Ollama-compatible APIs, meaning many apps already designed for Ollama can connect to Docker Model Runner with little more than a base URL change.
Setting it up through Docker Desktop is ridiculously easy. You need Docker Desktop 4.40 or later on macOS or version 4.41 or later on Windows. Open Docker Desktop, go to Settings -> AIand enable Docker Model Runner. You can also enable host-side TCP access here if you want to connect to Open WebUI, a coding tool, or another local application. Once enabled, the new Models section allows you to discover, download, run and chat with models without opening the terminal. I still prefer the command line because the workflow is very similar to Ollama.
Using Docker Model Runner for everyday tasks
The experience is similar to Ollama.
Once the model is downloaded, using Docker Model Runner is not much different from using Ollama. Docker Model Runner opens an interactive chat inside the terminal, where you can send messages and continue the conversation without restarting the model after each response. Docker Desktop also provides a basic chat interface in the Models tab, which is useful when I want to test a model without setting up another application.
The built-in interfaces are sufficient for quick directions, but I wouldn’t use them as my primary way of interacting with local LLMs. Docker Model Runner becomes much more useful when you connect it to something like Open WebUI. Since it provides an Ollama-compatible API, Open WebUI can treat it almost exactly like an Ollama server. You just need to point OLLAMA_BASE_URL to Docker Model Runner.
This gives you a proper ChatGPT-like interface with conversation history, Markdown rendering, model switching, system prompts, and configurable spawn settings. Models are still run through Docker Model Runner, while Open WebUI handles the interface.
The same approach works with AI coding tools like Cline, Continue, Aider, Cursor, and OpenCode.
You can use any placeholder as an API key because local requests do not require it. After that, select the model using the same name shown in the Docker model list, such as ai/qwen2.5-coder. I also like that Docker allows me to configure models individually. If an encoding model starts with an unnecessarily small context window, I can easily increase it.
Docker Model Runner does not make Ollama obsolete
Ollama is so good
After using both, I don’t think Docker Model Runner produces fundamentally better results than Ollama. When both tools run the same GGUF model via llama.cpp with similar configurations, the model, quantization level, context size, and hardware have a much larger effect than the application serving it. Docker allows you to use vLLM on supported NVIDIA systems, but that won’t matter to most people running quantized models on a regular laptop.
Docker Model Runner makes more sense if Docker Desktop is already part of your workflow. It allows you to manage models and containers through the same interface, store models as OCI artifacts, use them within Compose projects, and inspect requests through Docker Desktop. I also prefer their graphical model manager to remember every CLI command. However, installing and keeping Docker Desktop running just to use local LLMs adds unnecessary weight.
Ollama also has the advantage of a more established ecosystem. Many on-premise AI applications offer a dedicated Ollama option, and most configuration guides assume their API is available on port 11434. Docker Model Runner offers an Ollama-compatible API, so many of these applications still work, but some require manual changes to the base URL or model name. Docker’s support for different inference engines and logging is more flexible, although it also introduces additional configurations and platform-specific limitations.
I’m better with Ollama
If you already use Ollama and everything works, there is no many practical reasons to change. You will probably use the same models at very similar speeds and receive the same results. Docker Model Runner is most appealing to someone who already uses Docker Desktop or wants to integrate local models into containerized applications.
