Running your own LLM is surprisingly easy. With software like Ollama or LM Studio, you can install and run local AI models that work completely offline. The size and speed of the models will depend on your hardware, but you can run small models even on modest hardware; The real challenge is deciding what to do with your local LLM once you have it up and running.
Work with documents and private data
Keep confidential data in your own home
One of the best reasons to use a local LLM is to keep your data private. Everything you write or upload to a cloud-based LLM is sent to third-party servers where it can be used for training purposes or even read by a human reviewer. This could include sensitive data that you would prefer not to share with others, such as medical or financial information, personally identifiable information, or legal documents.
If there is data you would prefer not to share, a local LLM is a great way to be able to summarize, analyze or edit that data without it leaving your computer. Even smaller LLMs can be useful for working with text and data, although they are likely to be slower and less capable. Privacy is one of the key things that a cloud-based AI simply cannot match.
For example, I used an on-premise LLM to remove all personally identifiable information from a financial tracking spreadsheet before uploading it to a cloud-based LLM for analysis. This way, I get the best of both worlds: privacy and a powerful cloud-based model.
Get help with code instead of having it written for you
Autocomplete and explanations that never leave your machine
AI has made coding accessible to millions of people in a way that was not possible before. You don’t need to know or understand a single line of any programming language to be able to use AI to write functional and usable code. The best cloud-based AI models can do amazing things, like create a full online video game in operation in the style of world of warcraft.
If you have powerful hardware, you can install large local models that have impressive coding skills, but even if your AI equipment doesn’t cost the same as your car, you can still use a local LLM to help with coding.
Instead of writing all the code from scratch, you can use a local LLM as a tool to help you code. You can use one to autocomplete code, explain unknown functions, debugging errorswrite documentation or translate code between languages. You can often connect local LLMs with code editors. using extensions like Continue for VS Code.
The quality and speed of response will depend on the model you are running and the hardware it is running on. You may not be able to create a full MMObut it can help make coding easier and keep everything private.
Build a second brain
A local source for your ideas
One problem with an on-premises LLM is that if you want to keep it truly local, you only have access to the information on your computer or home network. You can give a local LLM access to web search, but it won’t work completely offline.
An alternative is to create your own local data source for your LLM. Instead of relying on your training data, you can give your local LLM access to files and documents on your local hardware so they can answer questions based on that information.
For example, you could give your local LLM access to notes, PDF Documentsmeeting transcripts, saved web pages, exported emails and more. Wearing increased recovery generation (RAG), relevant information from this data store can be retrieved and provided to the LLM so that it can respond to your requests using that context.
The quality of the answers will often be affected by the quality and organization of the original documents. Using a organized system of specific documents will work much better than just pointing the LLM at the entire hard drive.
smart home automation
Keep your smart home local
I use home assistant to control and automate my smart home. There are many benefits of using AI with Home Assistant, such as creating your own voice assistant which uses natural language and can understand the intent of commands like “it’s a little dark in the living room.” You can also use a Home Assistant MCP Server to allow an AI to interact with Home Assistant using natural language, allowing it to create automations, create dashboards, or control your smart home, depending on the permissions you grant it.
The problem is that if you use a cloud-based AI service, information about your smart home ends up on third-party servers. This may include sensitive data such as API keys, real-time presence information, your home address, and more.
Using an on-premises LLM is unlikely to give you the same performance as the best cloud-based models, but there are still much you can do without having to risk your privacy. I use small local LLMs in many of my automations, including a morning briefing that extracts weather and calendar information, converts it into a written summary, and then converts that summary to speech using a text-to-speech (TTS) engine. This summary is then played through a smart speaker when we walk into the kitchen in the morning, and everything works completely locally without any information leaving my home network.
My mini PC does not have a dedicated GPU, so generating the final report takes a while, but this is not a problem. I use a n8n automation generate the report early in the morning every day, so it’s ready to play when we come down to breakfast.
Uncensored writing and roleplaying.
Escape from overzealous barriers
Another major benefit of running your own on-premises LLM is that you can choose models that don’t have the same restrictive barriers as most cloud-based LLMs. These chatbots often flatly refuse to respond to requests that ask for medical advice or relate to polarizing topics.
While many of these barriers exist for good reason, they can often interfere with legitimate conversations. An example is role-playing games; An AI may be great for text-based RPGs, but if you ask it to draw your sword to take down an enemy, it may refuse to continue for safety reasons.
There is uncensored and annihilated models that have some of these railings removed or reduced. With a local LLM running one of these models, you might be able to kill that orc after all.
You can do a lot with a local LLM
Unless you’ve spent a small fortune setting it up, an on-premises model won’t be able to match the performance of the best closed models running in the cloud. A local LLM can still be very useful, even on modest hardware. You just need to decide how to use it.
