This is the first in a series of posts where I’ll share my deep dive down the rabbit hole in search of a local, private AI that can come close to current ChatGPT-4o levels, all within the safety and control of your own home lab.
My Backstory with AI
If you had asked me what I thought of artificial intelligence when OpenAI released ChatGPT to the world, I would have told you I was unimpressed. There was all the hype, all the hoopla, and endless announcements about how AI would elevate the world and humanity to the next level. It was being called the greatest invention ever made. Maybe it was just me, but my first experience with AI was, well, more than underwhelming.
I tried the free version after waiting for my invitation. After asking some general questions and doing a bit of light coding, I figured the paid version had to be better because it used more powerful language models. So, I plopped down my $20 per month. It was a little better, but not by much. Some of the code I asked it to review had significant errors, and I had to send multiple follow-up prompts just to get a somewhat usable result.
Fast Forward — Things Started to Improve, a Lot
By the start of summer 2024, ChatGPT suddenly got smarter. The newer model was significantly better, more friendly, and becoming genuinely useful. I found myself using it for a wide range of home lab projects: setting up and configuring servers, assisting with home automation tasks, and the big one—it was beginning to give solid advice on all kinds of topics, like I was talking to a friend who happened to be extremely knowledgeable across a massive number of domains.
Why I Wanted Local AI
After seeing how well ChatGPT was working, I thought, there has to be an open-source version of AI that I can run privately in my home lab—secure and disconnected from the outside world. I had projects involving data that I didn’t want exposed or used to train public models.
My first exposure to local AI was a lot like my early experience with ChatGPT. It technically worked, but just barely. It was painfully slow, even though I ran it on a decent laptop with a powerful processor, 32GB of RAM, and an Nvidia 2070 GPU. It was what I’d call a neat proof of concept and a fun challenge to get running. I tried many different language models and compared the results, but I didn’t find anything I could use for more advanced topics.
And Then the Real Quest Began
Near the end of October, I found a sale on an Nvidia 3060. I originally wanted a “good” GPU for gaming and video encoding, but somewhere in the back of my mind, I knew I was going to take another shot at local AI—without spending an arm and a leg.
With the new GPU, an Oculink dock, and a powerful AMD Ryzen 7 mini PC with Oculink support, I started hunting for the next level of local AI performance. That’s when I discovered an open-source application called LM Studio. It ran well on my hardware, with decent performance and a fairly simple interface.
A few days later, my next goal became clear: I wanted to access my local AI—what I nicknamed my Ironman AI server—from any device on my home network, through a web browser. I needed a clean, user-friendly interface, the ability to search the web using the Brave Search API, and a way to upload local documents for reference using Retrieval-Augmented Generation (RAG).
Enter Open WebUI
That’s when I found Open WebUI, a simple but powerful open-source frontend that supports multiple LLM backends. It connects easily to LM Studio and even public models like OpenAI through an API.
Using Open WebUI was an important milestone in my journey to running AI at home. The interface is clean, with a definite ChatGPT feel, and it offers many options to customize how it interacts with the backend models. It’s actively updated, with new features added regularly.
One of my goals was to find something my non-technical wife could use, without losing the flexibility I needed to push the AI envelope. Open WebUI helps me reach what I call the “WAF”—the Wife Approval Factor—while still letting me experiment with new features and integrations with my home lab servers and automation systems.
In the next post, I’ll walk you through how I set up LM Studio on my hardware and why I chose it over other options, with all the juicy details to help you get your own local language model running at home. Future posts will cover every piece of the puzzle and how it all comes together to bring my little local AI to life.