I tested my local LLM with 4 different tools and only one unlocked its potential

If you’re like me and aren’t particularly comfortable with getting CLI tools up and running, or perhaps new to self-hosted LLMs, then chances are you The default runner is LM Studio.. It’s been my default runner for about six months and it’s still just down to muscle memory. But the local LLM space moves fast and I’ve been keeping an eye on it because I’m already so deep, so I figured I should at least see what else is out there. I’ve come across a few other brokers (and a tool that’s not exactly a broker), and they’re all pretty decent in their own way.

So if you’re undecided on which one to land on, or maybe you’re just getting started, here’s how the same model went through four of them. The model in question is the Gemma 4 E4B, a multi-modal edge variant that I have subjected all the tools here to at one time or another.

LM studio

It’s just a habit at this point.

LM Studio is free but closed source, GUI-first, and has the HuggingFace hub connected directly to the model browser. You get one-click installs, there’s no terminal or configuration files unless you really want to dig into them. It’s been the easiest on-ramp to local AI for a while, and that’s exactly why it’s still on my machine. I learned it when I first started self-hosting and it pretty much taught me how this whole space works. And it’s still my default for creating a new model that I want to test.

But Gemma 4 E4B specifically does this weird thing in LM Studio where reasoning merges with actual answer. This way you will get the trace of the thought and the response together in a single block. I think it’s a known bug with Gemma models in LM Studio, I’ve seen others mention it and it still happens to me. Add to that that LM Studio doesn’t have audio input support yet, which means that the core feature of E4B (the native audio thing) simply goes unused. So for this particular model, LM Studio is the worst option.

Call.cpp

It allows me to use Gemma to her full potential.

llama.cpp is the open source C++ runtime that almost all the other runners on this list are built on (including LM Studio). So when you use the others, you’re already using llama.cpp, just with a layer wrapped around it, and going straight you skip that wrapper. It’s primarily a CLI tool, which is usually all I need to hear to say nothing, but there is a browser-based GUI that you can launch with a command.

This one was really recommended to me. Specifically for Gemma 4 E4B because it has audio input.. Setup was much less intimidating than I had thought, the pre-built binaries on GitHub got me up and running in a few minutes, and once the GUI is up it looks like a normal chat window. So now I could use the audio side of the E4B. The thing is, there is no live recording: you have to upload an audio file. And you can’t just upload any audio file, you need to convert it to WAV format first. So yes, llama.cpp allows you to use E4B to its full potential (it also has vision), but it’s a bit annoying if I’m honest.

january ai

It has audio, but it also has something else that sold me.

Jan is open source, free, GUI-first, and based on a file-over-app philosophy so your chats and settings live on disk in simple formats you can open yourself. It’s a desktop app, ships with built-in HuggingFace model downloads, and also allows you to connect cloud providers like Claude or OpenAI via API keys if you want one interface for everything. This makes it a hybrid LLM broker, which I have found is quite rare.

I went after llama.cpp because I wanted the opening without the terminal step, and Jan basically falls right in that middle ground. Carrying my own Gemma 4 E4B GGUF was easy. The UI is also very clean, nothing developer-y and it also has audio and vision, so E4B can get the use it deserves here.

My love for Jan. However, it extends beyond Gemma; What really sold me on this broker was the MCP setup. I intended to play more with the MCP servers, but it’s not a particularly quick process through the other runners. However, with Jan, several MCP servers are pre-configured, just activate them (and install half of the Chrome extension for the MCP browser). I love when tools make these integrations more accessible.

AnythingLLM

He’s not a runner, but he gives Gemma memory.

AnythingLLM is not actually a broker, it is a Mintplex Labs interface that adapts to the broker of your choice. So you still need llama.cpp or LM studio or something else that does the actual model in the background. It’s free, open source, and more of a RAG-centric tool with workspace management.

I tried because I was searching for locally persistent memory and AnythingLLM kept showing up. And yes, it delivers on that, there’s an automatic memory system that pulls data from your chats every few hours in the background, plus you can add memories manually if you want to be precise. I tested it with Gemma 4 E4B running in LM Studio and the memory side worked great, the model started referencing things from old chats without me repeating anything. There is no audio input on the AnythingLLM side, so the E4B’s audio feature cannot be accessed through it. However, it did not convey the LM Studio response bleeding issue.

I’d rate it as a general local AI tool, but since it doesn’t run the model itself, I’m not sure how much of what I was experiencing was AnythingLLM versus just Gemma 4 doing Gemma 4 stuff through a different window. However, it’s worth mentioning if you’re looking for memory.

Audio plus MCP plus GUI

LM Studio would have been my choice if it had audio and if the reasoning bleed wasn’t there, because otherwise it does everything I need. llama.cpp is great and audio is its biggest plus for this model, but Jan also has audio, plus it’s easier to use than llama.cpp and the MCP setup is what threw me for a loop. So, to run the Gemma 4 E4B specifically, Jan ended up being the one that allowed me to use the model to its fullest without ever getting frustrated.