Ollama
Practical reference for serving large language models locally with Ollama: installation, embedding models, REST/Python/JS APIs, and Docker Compose with GPU support.
What this covers
Running large language models locally with Ollama for privacy and control: installation, pulling and serving models, generating embeddings via REST/Python/JavaScript, and deploying Ollama with Docker Compose (including GPU support). For broader AI development concepts (RAG, embeddings theory, ML fundamentals), see AI Development.
When to use it
- Setting up or configuring Ollama to serve LLMs locally, including with Docker Compose
- Choosing an embedding model for local, offline semantic search
- Deciding between local (Ollama) and cloud-based model deployment for privacy or cost reasons
- Pre-loading a fixed set of models into a container at startup
Installation & Basic Commands
# Pull models
ollama pull llama3.2:1b
ollama run deepseek-r1:8b
# Start Ollama server
ollama serve
Embedding Models
Embeddings convert text into vectors for semantic search.
| Model | Size | Description |
|---|---|---|
mxbai-embed-large | 334M | High-quality general embeddings |
nomic-embed-text | 137M | Balanced performance and efficiency |
all-minilm | 23M | Lightweight, low-resource model |
Generating Embeddings:
REST API:
curl http://localhost:11434/api/embed -d '{
"model": "mxbai-embed-large",
"input": "Llamas are members of the camelid family"
}'
Python:
import ollama
result = ollama.embed(model='mxbai-embed-large', input='Llamas are members of the camelid family')
JavaScript:
import ollama from 'ollama';
const result = await ollama.embed({model: 'mxbai-embed-large', input: 'Llamas are members of the camelid family'});
Running Ollama with Docker Compose
Basic Setup:
version: '3.8'
services:
ollama:
image: ollama/ollama:latest
ports:
- "11434:11434"
volumes:
- ollama_data:/root/.ollama
restart: unless-stopped
environment:
- OLLAMA_HOST=0.0.0.0
volumes:
ollama_data:
GPU Support Example:
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
Usage Commands:
docker-compose up -d # Start Ollama
docker-compose exec ollama bash # Access container
docker-compose down # Stop Ollama
Pre-load Models:
#!/bin/bash
ollama pull llama3.2:1b
ollama pull deepseek-r1:8b
ollama pull mxbai-embed-large