Ollama

October 31, 2026

—

ollama

Run a LLM locally on Linux with Ollama. Everything runs on your machine: once the model is downloaded, nothing leaves it.

Contents

Deployment

$ curl -fsSL https://ollama.com/install.sh | sh

The installer sets up ollama as a systemd service and uses an NVIDIA GPU automatically if one is present (no CUDA toolkit needed).

Models

$ ollama run hf.co/<user>/<repo>

With 6 GB of VRAM, models up to ~8B (Q4) fit fully on the GPU. Bigger models such as qwen3.6:35b still run, split between GPU and RAM (slower).

The Ollama library only sorts by popularity or date, not quality. To compare models:

Then check the model’s size on https://ollama.com/library/<model>/tags to see if it fits your GPU.

Qwen3.5:4B

qwen
$ ollama run qwen3.5:4b

The first run downloads the model (~3.3 GB). Exit with /bye or Ctrl+D, and run the same command again to start a new chat.

Qwen3.5 “thinks” before answering. For quick replies start it with --think=false, or type /set nothink inside the chat:

$ ollama run qwen3.5:4b --think=false

Check where it is running:

$ ollama ps        # PROCESSOR column, e.g. "100% GPU"
NAME          ID              SIZE      PROCESSOR    CONTEXT    RUNNER      UNTIL
qwen3.5:4b    d8b0f5e9760c    3.1 GB    100% GPU     4096       llamacpp    4 minutes from now

$ ollama list      # models on disk
NAME          ID              SIZE      MODIFIED
qwen3.5:4b    d8b0f5e9760c    3.3 GB    26 seconds ago

Example with the GPU idle (no model loaded), only gnome-shell on it:

$ nvidia-smi
Sat Oct 31 00:21:13 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.178.04             Driver Version: 580.178.04     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX PRO 500 Black...    Off |   00000000:01:00.0 Off |                  N/A |
| N/A   35C    P4             10W /   60W |      11MiB /   6113MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A           23922      G   /usr/bin/gnome-shell                      1MiB |
+-----------------------------------------------------------------------------------------+

With qwen3.5:4b loaded, llama-server shows up in Processes with the model’s VRAM:

$ nvidia-smi
Sat Oct 31 00:48:53 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.178.04             Driver Version: 580.178.04     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX PRO 500 Black...    Off |   00000000:01:00.0 Off |                  N/A |
| N/A   39C    P8              3W /   50W |    3747MiB /   6113MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A           23922      G   /usr/bin/gnome-shell                      1MiB |
|    0   N/A  N/A         1330762      C   ...local/lib/ollama/llama-server       3728MiB |
+-----------------------------------------------------------------------------------------+

Remove a model you no longer need:

$ ollama rm qwen3:4b
deleted 'qwen3:4b'

Don’t trust what the model says about itself (e.g. “I run in the cloud”): it can’t see your system, ollama ps can.

Sessions and history

Each ollama run starts from scratch: the model doesn’t remember previous chats.

Save and restore a chat from inside a session:

>>> /save my-chat
>>> /load my-chat

or reopen it directly:

$ ollama run my-chat

~/.ollama/history only keeps the lines you typed (up/down arrows in the prompt), not the model’s answers, and the model never sees it.

For a chat history UI like ChatGPT’s, put Open WebUI in front of Ollama: it runs locally in the browser, keeps your chats on your own disk and talks to Ollama.

Open WebUI

open-webui

Ollama only listens on 127.0.0.1:11434, so run Open WebUI with the host network and keep it local too (HOST/PORT):

$ docker run -d --name open-webui --restart unless-stopped \
    --network=host \
    -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
    -e HOST=127.0.0.1 -e PORT=3000 \
    -v open-webui:/app/backend/data \
    ghcr.io/open-webui/open-webui:0.12.5

Open http://localhost:3000 and create the first (admin) account, stored only on your machine. Models from ollama list show up in the model picker.

Chats and settings live in the open-webui Docker volume, so they survive restarts and upgrades:

$ docker logs -f open-webui     # first start takes a minute or two
$ docker stop open-webui
$ docker start open-webui
$ docker rm -f open-webui       # upgrade: remove, then run again with the new tag

Check how much disk Docker and the open-webui volume are using:

$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          132       1         83.2GB    66.44GB (79%)
Containers      3         1         58.73MB   3.953MB (6%)
Local Volumes   1         1         1.118GB   0B (0%)
Build Cache     95        0         5.609GB   5.495GB

$ sudo du -sh /var/lib/docker/volumes/open-webui
1,1G    /var/lib/docker/volumes/open-webui