Two years ago this section of the blog wouldn’t have existed. Now AI tools are just part of how we build things, sitting alongside the compilers and the text editors, and it felt dishonest to write a homelab blog and pretend otherwise.
The tools we actually reach for
For day to day coding and writing, we lean on Claude and ChatGPT. Not because they’re magic. Because typing out a rough idea and getting a reasonable first draft back saves real time, whether that’s a config file, a paragraph of documentation, or a bash script we’d otherwise have stitched together from three different Stack Overflow answers.
We still read every line before it runs on anything that matters. That hasn’t changed and shouldn’t change for you either.
Running AI locally
For anything private, or anything we want running without an internet connection or a monthly bill, local models do the job.
Ollama is the easiest on-ramp. Pull a model with one command the same way you’d pull a Docker image and you’ve got a local LLM running on your own hardware. No account, no API key, nothing leaving the building. Pair it with Open WebUI if you want a chat interface instead of a terminal prompt.
Once you’re past the on-ramp, though, we’d point you at llama.cpp or vLLM instead. You give up the one-command convenience and you get real control over quantization, context size, and how much of the model actually lands on the GPU. On constrained hardware that’s the difference between a model that runs and a model that runs well.
If you want an all-in-one instead of stitching pieces together, LocalAI wraps model management, inference, and an OpenAI-compatible API into a single Docker container. That compatibility matters more than it sounds like it should, because a lot of tools already expect to talk to something shaped like the OpenAI API, and LocalAI lets you point them at your own hardware.
Where the line sits for us
Cloud models when the quality of the model matters more than where it runs. Local models for anything private, experimental, or when we want to see what a small server can actually handle.
Neither is right for everything, and anyone telling you their setup is the only correct one is selling something.
The honest caveat
Local models are genuinely good now. They’re still not the frontier cloud models, and pretending otherwise wastes your time. Try both, figure out which one earns a permanent spot on your machine for which job.
Talk shop with us
Running something interesting locally, or have a workflow that puts these to shame? We’d rather hear about it than guess. Drop by The Bench and tell us what’s actually running on your machine.