Two years ago this section of the blog would not have existed. Now AI tools are just part of how we build things, alongside the compilers and the text editors, and it felt dishonest to write a homelab blog and pretend otherwise.
The tools we actually reach for
For day to day coding and writing, we lean on Claude and ChatGPT. Not because they are magic, but because typing out a rough idea and getting a reasonable first draft back, whether that is a config file, a paragraph of documentation, or a bash script we would have otherwise copy-pasted from three different Stack Overflow answers, saves real time. We still read every line before it runs on anything that matters. That has not changed and should not change for you either.
Running AI locally
For anything private, or anything we just want to run without an internet connection or a monthly bill, Ollama has become the default answer. Pull a model with one command the same way you would pull a Docker image, and you have a local LLM running on your own hardware. No account, no API key, no data leaving the building.
Pair it with Open WebUI if you want something that looks and feels like a chat app instead of a terminal prompt. It sits on top of Ollama and gives you a proper interface without much setup work.
If you want an all-in-one option instead of stitching pieces together, LocalAI wraps model management, inference, and an OpenAI-compatible API into a single Docker container. That compatibility matters more than it sounds like it should, since a lot of tools already expect to talk to something shaped like the OpenAI API, and LocalAI lets you point them at your own hardware instead.
Where the line actually sits for us
Cloud tools like Claude and ChatGPT for anything where the quality of the model matters more than where it runs. Local models for anything private, experimental, or where we just want to see what a tiny homelab server can actually handle. Neither approach is right for everything, and anyone telling you their setup is the only correct one is selling something.
The honest caveat
Local models are genuinely good now, but they are not the same as the frontier cloud models, and pretending otherwise wastes your time. Try both. Figure out which one earns a permanent spot on your machine for which job. That is the whole exercise.
Talk shop with us
Running something interesting locally, or have a workflow that puts these tools to shame? We would rather hear about it than guess. Drop by The Bench and tell us what is actually running on your machine.