Setting up a local LLM

Hello Friend

I was sitting in a coffee shop with a bit of a spotty connection. Writing articles, creating strategy - perfect tasks for AI to assist with.

It reminded me how valuable a local AI setup can be. It is what I want to talk about today.

Welcome to this week’s issue of the AI-Enhanced Creator. Thank you for being a subscriber. 💛

Let’s get to it.

Resources

AI development I’m most excited about this week - Photoshop plugin in ComfyUI. Combining different tools is incredibly powerful, It does get a bit bothersome when we have to change workspace too often. I’ve quite enjoyed seeing this plugin in action. It does create a much more fluid and smooth experience.

Live event - Comfy Coffee with Chris Vespaziani. Chris is absolutely incredible. Take a look at his Instagram and you will quickly see why I am so excited about this session. You can join our session on LinkedIn or Youtube.

This Weeks Lesson - Running a local LLM

With LLMs, you can have a powerful tool at your fingertips to help with tasks such as writing, research, ideation...

But did you know that you don’t need to rely on cloud-based services to access these capabilities? With a little setup, you can run LLMs locally on your machine.

Running LLMs locally offers several benefits:

• Improved Security: By storing your data and models locally, nobody else, but you, can access it.

• Offline Capability: With a local setup, you can still use LLMs even when you don’t have an internet connection.

• Performance: Local access means faster response times and reduced latency. It does depend on the machine you have though. The better it is, the faster it will run.

• It’s free: Open source LLMs are free to use.

So, how do we get started?

Ollama + Anything LLM

We will interact with our LLM inside AnythingLLM. It has a clean interface with a lot of power and flexibility.

Before we get into AnythingLLM, we need to install Ollama, which will allow us to load custom models. This will run in the terminal in the background while we interact through AnythingLLM.

You could use LM studio as well, which has a more fleshed out UI than Ollama, but I found the combination of Ollama and Anything LLM the smoothest.

Download and install Ollama.

It’s pretty straight forward. You click a few buttons. 😁

When you have it installed, Open up your terminal. Type in ollama and press enter. This will show you a list of commands you can use with ollama.

Next step is downloading a model. You can go to this site and choose any of the models.

I recommend Llama 3.1 8B parameters Q8. It runs well on my macbook pro.

Copy the code in the corner.

Paste it into the terminal and it will start your download. Once it’s finished type in ollam serve which will start a serve with the loaded LLM model.

We will access this through anythingLLM.

We can go into settings and choose our LLM provider, which is Ollama

We choose our model and max tokens. You can google this if you’re not sure.

We paste the url from our terminal usually it something like this: http://127.0.0.1:11434

And that is it!

Now we have Workspaces where we can set up different assistants with different system prompts.

You can click on settings for each workspace.

This will open up a few options.

If you go into chat settings and scroll down a bit, you will find a prompt window. In here you can put a prompt which will act as a system prompt.

You can give very specific instructions to each of you workspaces, using them for variety of use cases. I wrote a post on how you can write a prompt for anything if you want to dive deeper into that.

With this you have set up your own local LLM with some powerful features - you can upload documents and knowledge base, you can set up agents, you can search the web, etc.

If you want to learn more about any of these features, reply to this mail and I will do a lesson on that. 😁

Homework

I would like you to block 30 minutes in your calendar. In those 30 minutes, sit down, follow this guide and set up AI on your computer.

Setting up your own AI structure is so incredibly valuable. In a world where data is becoming more and more valuable, having ownership of it and autonomy on how to use it and apply AI with it…

Take a moment to think about it. All of the experience you have, everything you have done in your (digital) life. It’s all out there. It’s all available. It is being used. It will continue being used.

Would you prefer to have a say on how it is used, even if only a tiny bit?

Potential is high. For us. And what we can do with this technology. If we choose to stand up and manage it.

Talk to you next week. ☀️

Whenever you are ready, here are 4 ways I can help you:

• FREE Beginner’s Guide: Take your first steps with a 6-lesson course designed to introduce core mindset, skills and software to start working with AI. It's the foundation on which you can build your expertise.

• We have a free comunity of creators just like you. A place to connect, collaborate and learn together. It’s always easier to grow with others.

• AI-Enhanced Creator: Power of a Production House: Building on top of the core concepts, we dive deep into workflows and techniques. With step by step walkthrough and process breakdown.

• Coaching: For personal and hands-on learning experience. Focusing on your goals.