Open source self-hosted AI – when you want to run AI offline, in secret on your own computer

by Stélio Inácio, Founder at Jon AI and AI Specialist

Your Own Private AI: The World of Self-Hosting

So far, we've talked about using AI models like ChatGPT or Gemini that run on massive computer servers owned by big tech companies. When you use them, your data travels over the internet to their computers, gets processed, and a response is sent back. But what if you could run a powerful AI model entirely on your own computer, with no internet required and with absolute privacy?

Welcome to the world of open-source, self-hosted AI. Think of it like cooking at home versus eating at a restaurant. When you use a public AI service, you're eating at their restaurant; it's convenient, and they have a massive kitchen, but you don't know the exact recipe, and you're in a public space. Self-hosting is like having your own private kitchen. You choose the ingredients (the AI models), you control the entire process, and no one else sees what you're making. Your data never leaves your device.

Why Run AI on Your Own Computer?

Opting for a self-hosted setup involves a few extra steps, but the trade-offs are compelling for many users.

The Advantages (Your Private AI Lab)

Absolute Privacy: This is the number one reason. Your conversations and the data you use are never sent to a third party. They stay on your machine, period.
Offline Capability: Once you've downloaded the models, you can use your AI assistant even without an internet connection. It's perfect for a flight or a remote cabin.
No Subscription Fees: The software and the open-source models are free. You are only limited by the power of your own computer.
Ultimate Customization: You can download and switch between hundreds of different models, from small, speedy ones to large, powerful ones, each with unique strengths.

The Considerations (What's the Catch?)

Hardware Demands: Running these models requires a reasonably modern computer with plenty of RAM (16GB is a good start). A powerful graphics card (GPU) makes a huge difference in speed.
Slightly More Technical: Tools have made it much easier, but it still requires some comfort with installing software and occasionally using a command line.
Model Performance: While open-source models are incredibly powerful and getting better every day, the absolute state-of-the-art models (like the very latest from OpenAI or Google) are typically not available for self-hosting.

Step-by-Step Guide: Setting Up Your Personal AI

Ready to try it? We'll use a fantastic tool called Ollama, which does all the heavy lifting of managing and running the models. Then we'll add a user-friendly chat interface so you don't have to live in a terminal.

Install the "Engine" (Ollama):
Ollama is the core program that runs in the background on your computer, serving the AI models. It's the engine of our setup. Go to ollama.com and download the installer for your operating system (Mac, Windows, or Linux). Follow the simple installation instructions.
Download Your First AI Model:
Once Ollama is installed, open your computer's command line tool (Terminal on Mac/Linux, Command Prompt or PowerShell on Windows). To download and chat with Meta's popular Llama 3 model, type the following command and press Enter:
ollama run llama3
The first time you run this, it will download the model (a few gigabytes). After that, it will start a chat session right in your terminal. This confirms your engine is working! You can type /bye to exit.
Install a User Interface (Open WebUI):
Now for the user-friendly part. We need a "front end" or a "dashboard" to chat with our models. A great option is Open WebUI, which gives you a ChatGPT-like experience. The easiest way to install it is with Docker. If you don't have Docker, you can get it from the official Docker website.

With Docker running, open your command line tool and run this single command:
docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main
This command downloads the Open WebUI, sets it up, and connects it to Ollama running on your computer.
Start Chatting Privately:
Open your web browser and go to http://localhost:3000. You'll see the Open WebUI interface. Create a local account, and you can start chatting! You can select which model to use (like the `llama3` model you downloaded) and even pull new models directly from the web interface. Congratulations, you are now running your own private AI!

A Note on Computer Resources

Running AI models locally can be demanding on your computer. The larger the model (e.g., a 70-billion parameter model vs. a 7-billion one), the more RAM it will require and the slower it might run, especially without a powerful GPU. Don't be discouraged if it's a bit slow; experimenting with smaller, faster models is part of the fun!

Quick Check

What is the primary and most important reason for someone to choose a self-hosted AI setup?

A) To ensure data privacy and security by keeping all conversations on their own computer.

B) To get access to the most powerful models like GPT-4o for free.

C) Because it is the easiest and least technical way to use an AI.

Recap: Open Source Self-Hosted AI

What we covered:

The concept of self-hosted AI, where you run open-source models on your own computer.
The key benefits: absolute privacy, offline access, no fees, and total customization.
The main considerations: hardware requirements and the technical nature of the setup.
A step-by-step guide to getting started using Ollama as the "engine" and a Web UI as the "dashboard."

Why it matters:

Self-hosting puts the ultimate control back in your hands. It's a powerful way to use AI on your own terms, ensuring your data remains truly yours while you explore the vast world of open-source models.

Next up:

We'll look at the other side of the open-source coin: running open-source AI models in the cloud when you don't want to use your own hardware but still want to avoid sharing data with Big Tech.

Jon AI Services