Running Ollama without a GPU
tl;dr You can run Ollama on an older device, but the response will be slow and/or low quality.
I have successfully run Ollama with a new Macbook M2 and a mid-range gaming PC, but I wanted to experiment using an older computer. My personal laptop is a 2017 Lenovo Yoga with Ubuntu and no graphics card. Here are some specs:
- CPU: Intel i5-7200U CPU @ 2.50GHz
- RAM: 4GB
- Memory: 128GB SSD
Following the setup instructions for Linux, Ollama installed fine but printed the following:
WARNING: No NVIDIA GPU detected. Ollama will run in CPU-only mode.
This was foreshadowing for everything to follow.
I decided to run mistrel and sent the model a prompt by the terminal. The
response was streaming in at about one character every four seconds, and my
computer was obviously struggling with the task. I held on for around three
minutes but then had to quit when the response was less than on sentence.
That seemed conclusive enough, but I decided to try with a smaller model. I ran
tinyllama and was able to stream responses at a normal rate. However, the
quality of the responses was terrible. I don’t have experience with this model
on a more powerful machine, so I can’t blame the model. But the responses were
so bad that I wouldn’t use it for anything on this device.
These results were disappointing, but the reality is that running LLMs is resource intensive. So if you want to try running LLMs locally with Ollama, make sure you have a modern CPU or a dedicated GPU.