Local LLM Deployment Guide: Running open-source models with Ollama

Local LLM Deployment Guide: Running open-source models with Ollama

Running a large language model on your own machine used to be a GPU-cluster privilege. Today, an installer a few hundred MB in size lets an ordinary laptop run a 3B-parameter model — that's the change open-source models brought. Ollama packages model download, inference, and API exposure into three commands. No GPU needed for small models; with a GPU you can run larger ones. For teams that value data privacy, want to cut API costs, or need full control over inference, this is a very practical path.

Installing Ollama

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Docker
# docker run -d --name ollama -p 11434:11434 ollama/ollama

The Linux one-liner sets up a systemd service with auto-start. Docker is a good fit for servers, but note that GPU passthrough requires the NVIDIA Container Toolkit inside the container. Verify with ollama --version after installing.

Pulling and Running Models

# Pull and run a model (auto-download)
ollama run llama3.2:3b    # Llama 3.2 3B, CPU-friendly
ollama run mistral         # Mistral 7B, needs 8GB RAM
ollama run qwen2.5:7b      # Qwen 2.5 7B, excellent Chinese
ollama run llama3.2:11b    # Llama 3.2 11B, needs 16GB+ RAM

ollama run downloads the model to ~/.ollama and drops you into an interactive chat. Handy management commands: ollama list shows downloaded models, ollama pull <model> downloads without running, ollama rm <model> deletes a model to free disk space.

Model Comparison

Two criteria matter most. First, Chinese quality: the Qwen series is trained extensively on Chinese corpora, so it often beats larger English-centric models on Chinese tasks. Second, memory: weights load into RAM during inference — a 7B quantized build needs roughly 6-8GB, 11B needs 16GB+. On small-memory machines, pick 3B or 7B quantized builds. If you have a discrete GPU, run the larger model on it and leave the CPU for other services.

Calling via API

Ollama exposes a built-in REST API you can call from your app:

# API request
curl http://localhost:11434/api/generate -d '{
  "model": "qwen2.5:7b",
  "prompt": "Why choose a cloud VPS?",
  "stream": false
}'

// Call from Node.js
const response = await fetch('http://localhost:11434/api/generate', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    model: 'qwen2.5:7b',
    prompt: 'Introduce cloud hosting in three sentences',
    stream: false
  })
});
const data = await response.json();
console.log(data.response);

Default port is 11434. /api/generate handles one-shot generation, /api/chat supports multi-turn conversation; stream: true returns SSE streaming for typewriter-style chat UIs. The OpenAI-compatible endpoint /v1/chat/completions lets you point existing OpenAI code at your local model by changing a single base URL.

Docker Compose Deployment (Server)

version: '3.8'
services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama-models:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped

volumes:
  ollama-models:

Note ports binds to 127.0.0.1:11434 — localhost only, so unauthorized API calls can't burn your compute. The ollama-models named volume persists model files across container rebuilds. The GPU section requires the NVIDIA driver and nvidia-container-toolkit on the host.

Quantization and memory choice

The same 7B model behaves very differently depending on whether it runs in FP16 or Q4 quantization. Quantization compresses weights from 16 bits down to 4-8 bits, slashing memory usage at the cost of a tiny quality loss. Common suffixes in Ollama model tags: q4_K_M (~4.7GB) is the value pick for 7B models; q8_0 (~7.5GB) is higher quality; f16 is the original precision, for machines with plenty of RAM. The rule is simple: satisfy the memory constraint first, then pick the highest precision tier that fits. Specify the tier directly in the tag when pulling:

ollama pull qwen2.5:7b-q4_K_M

In practice, the Q4 build of a 7B model is nearly indistinguishable from the unquantized version on most tasks, yet runs smoothly on 8GB machines — the standard way to run 7B on CPU.

A Real-World Scenario

An online-education team wanted an "AI teaching assistant" that answers course questions from lecture notes. They deployed Qwen 2.5 7B on a 16GB cloud VPS, retrieved relevant notes from a vector store, and fed them into the prompt for generation. Compared with cloud APIs, per-request cost is near zero, and — more importantly — the lecture content never leaves the server. Before launch they validated answer quality with the Q4 quantized build on a dev machine, then moved to the 16GB VPS, bound Ollama to localhost, and put a gateway in front for auth. At peak, with streaming output and request queuing, the experience rivals cloud models — but the cost structure is completely different: no per-token billing, just one fixed monthly server.

FAQ

• Can I run it without a GPU? Yes. A 3B model chats smoothly on CPU alone; 7B is usable but slower, so prefer quantized builds like qwen2.5:7b-q4_K_M.
• Where are models stored? ~/.ollama/models by default; change it with the OLLAMA_MODELS env var.
• Concurrency? Ollama queues requests by default; for high concurrency use Open WebUI, vLLM, or a front-end gateway.
• How do I control memory usage? Use the OLLAMA_NUM_PARALLEL and OLLAMA_MAX_LOADED_MODELS environment variables to cap concurrent loading so several models don't sit in RAM at once.

16IDC Takeaway

Ollama makes local LLM deployment easier than ever. For privacy-conscious websites and SaaS products, self-hosted LLMs are worth considering — data stays on your server, no API fees, fully controllable. Start with Qwen 2.5 7B or Llama 3.2 3B.

References: Ollama docs https://docs.ollama.com/; model library https://ollama.com/library