There is no denying that next-generation graphics cards are ideal for hosting large language models, especially for productivity tasks where high precision is desired without compromising performance. That said, you don’t necessarily need a powerful server for simple inference tasks, and that’s something I realized only after building a Raspberry Pi-powered LLM hosting workstation.
Heck, I’ve hosted simple text-to-speech and speech-to-text models on simple virtual machines without any GPU pass-through provision, and gotten some pretty fantastic inference results. So one shower thought led to another, and I tried to create a completely local voice assistant workflow that works with none other than my Raspberry Pi 5.
I decided to use llama.cpp for the central LLM that powers my voice assistant.
Had to settle for Qwen3.5-2B for best performance
Considering the weak specs of the ARM-based Raspberry Pi SBCs, I wanted to minimize the performance overhead as much as I could. So, I opted for the Lite version of the Raspberry Pi OS, which removes the GUI elements and non-essential packages that this distro is known for without causing compatibility issues like certain light versions of Linux (cough, alpine cough). Ollama would have been my first choice back then, but since it introduces an additional performance penalty, I decided to compile call.cpp manually in the SBC.
Truth be told, I was tempted to rely on llama-server’s RPC functionality to connect another SBC to my Raspberry Pi, thus creating a cluster for my LLM hosting duties. Having worked in this damn setup In the past, I gave up after realizing that the slow LAN connectivity of these SBCs would end up slowing down inference speeds. But that meant I couldn’t load anything larger than a 5B (or at most a 6B) on my Raspberry Pi 5 (8GB).
My first choice was Gemma 4 E4B, the same model that saved my Pi in the Raspberry Pi project. Unfortunately, even if you could use this surprisingly capable edge LLM on the Raspberry Pi, its token generation speed is in the 2-3 t/s range, which is too slow for a voice assistant. So, I went with the Qwen3.5-2B as it is much faster than the Gemma 4 E4B but doesn’t blow out as much as the typical 2B models. But we’ll talk more about that in a moment…
Speeches provides the STT and TTS models for this project.
Implementing it was also quite simple
If I wanted to add voice control to my Qwen3.5-2B instance driven by a llama server, I needed two more AI models. The first is a speech-to-text model that converts everything I say into text for Qwen3.5, since the clanker cannot directly interpret voice commands. You would then need a text-to-speech model at the other end of the voice assistant process, as it would convert all the text generated by the LLM into audio-based responses.
Since I was planning to use Open WebUI as the control center for this voice assistant, I needed my STT and TTS models to support the OpenAI API. So, I thought I could use Speakes as a server for these audio processing models, since it’s the same application I connected to my Open Notebook instance to generate podcasts from notes. Anyway, once I implemented Speakes, I chose faster distil-whisper-small.en as my speech-to-text model, with the old Kokoro-82M-v1.0-ONNX as its TTS counterpart. But I still had to configure the last missing piece for my AI assistant.
Open WebUI provides the interface for my voice assistant process
It connects with all the models I have implemented so far.
On their own, llama-server and Speakes can’t talk to each other, which means I needed another utility to tie them together. That’s where the Swiss army knife of local LLM apps, Open WebUI, comes into the picture, as it not only has microphone input but also supports automatic audio playback once an LLM finishes its inference tasks.
Pairing Open WebUI with llama-server was as simple as adding the latter’s IP address (and port number) as a local OpenAPI model within the Connections tab of Admin Settings. As for Speakes, I had to add /v1 after the URLs for the TTS and STT models within the Audio section. But even then, I still couldn’t use my Open WebUI channel without HTTPS, since my browser blocks microphone access for HTTP web pages. Since I didn’t want to waste any more time generating self-signed certificates, I temporarily enabled the allow-localhost-insecure flag and restarted Brave.
Sure enough, I was able to grant microphone access to the Open WebUI interface. With all the models connected to Open WebUI, I started asking my Raspberry Pi-powered AI servers for random things. During my testing, fast-distil-whisper-small.en would typically take 10-15 seconds to convert my voice commands to text, while Qwen3.5-2B would take 15-45 seconds to generate the response. Combine that with the extra 10 seconds the Kokoro-82M-v1.0-ONNX takes, and simple prompts will be answered within a minute.
While I wouldn’t go so far as to say that this setup can replace the GPT-OSS-20B pipeline running on my RTX 3080 Ti (via MoE offload), it’s pretty impressive that a Raspberry Pi can generate perfectly good responses to voice commands without forcing me to wait 2-3 minutes. I can technically cut the total processing time in half if I switched to the 0.7B and 1.5B models, but I haven’t had good luck with anything other than Qwen3.5 1.5B and LFM2 1.2B. And even then, these noisemakers wouldn’t be able to process demanding cues with the same precision as my current setup.
