NVIDIA GPU Requirements

Not all NVIDIA GPUs support Ollama's CUDA-based acceleration. Ollama requires NVIDIA GPUs with CUDA Compute Capability 5.0 or higher, which includes:

Verify your GPU's compute capability by checking your GPU model against NVIDIA's official CUDA GPUs list. You can identify your GPU on Ubuntu with:

lspci | grep -i nvidia

Example output: 01:00.0 VGA compatible controller: NVIDIA Corporation TU117 [GeForce GTX 1650]

Memory and Storage Requirements

Minimum requirements are 4GB RAM and 10GB disk space, but this only supports tiny models. Recommended specs are 16GB RAM, 50GB disk space, and an NVIDIA GPU with 8GB+ VRAM. For production deployments running multiple concurrent models, consider:

Storage considerations extend beyond model files. Each LLM download consumes 4-40GB depending on parameter count and quantization. These models are not small. Make sure you have at least 30-50GB of free space on a fast drive (like an SSD) to be comfortable. Some models alone can take up over 20GB. Plan for 100GB+ storage on NVMe SSDs for production systems managing multiple model variants.

GPU Memory Management

Ollama automatically manages GPU memory allocation, but custom limits prevent out-of-memory errors in shared GPU environments or when running multiple applications.

Set GPU memory fraction limit:

sudo systemctl edit ollama

Add memory fraction configuration:

This limits Ollama to 80% of available VRAM, reserving 20% for other applications or system overhead. Recommended values:

• • Dedicated AI workstation: 0.9 (90% allocation)
• • Shared development machine: 0.7 (70% allocation)
• • Multi-tenant server: 0.5 (50% allocation)

For models exceeding available VRAM, Ollama automatically offloads layers to CPU. Configure layer distribution manually for optimal hybrid performance.

Model Keep-Alive Configuration

Ollama unloads models from VRAM after inactivity to free resources. The default keep-alive duration is 5 minutes. You can adjust this system-wide by configuring the OLLAMA_KEEP_ALIVE environment variable in the systemd service:

sudo systemctl edit ollama

Add the keep-alive configuration:

Restart Ollama to apply changes:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Common keep-alive settings:

• • OLLAMA_KEEP_ALIVE=60m - Keep model loaded for 1 hour
• • OLLAMA_KEEP_ALIVE=-1 - Keep model loaded indefinitely
• • OLLAMA_KEEP_ALIVE=0 - Unload immediately after inference

Choose settings based on your usage patterns—longer keep-alive reduces model loading latency for frequent queries but consumes VRAM continuously.

Docker-Based Open WebUI Deployment

Open WebUI runs as a containerized application, simplifying deployment and isolating dependencies. First, ensure Docker is installed:

# Install Docker if not present
sudo apt install docker.io -y
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
newgrp docker

Deploy Open WebUI container:

docker run -d \
  --network=host \
  -v open-webui:/app/backend/data \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

Parameters explained:

• • --network=host: Simplifies Ollama connectivity by sharing host networking
• • -v open-webui:/app/backend/data: Persists chat history and settings
• • -e OLLAMA_BASE_URL: Configures Ollama API endpoint
• • --restart always: Ensures automatic startup after reboots

Access Open WebUI at http://your-server-ip:8080. First login automatically creates an admin account—secure this immediately in production deployments.

Verifying End-to-End Functionality

After Open WebUI deployment, perform comprehensive testing to validate the entire stack:

1. 1. Access Open WebUI and create admin account
2. 2. Select a model from the dropdown (should show models pulled via CLI)
3. 3. Send a test prompt and monitor nvidia-smi for GPU activity
4. 4. Verify response quality and generation speed match CLI performance

Connection issues usually stem from incorrect OLLAMA_BASE_URL settings or Docker networking problems. Verify Ollama is running with systemctl status ollama and test the API manually:

curl http://127.0.0.1:11434/api/tags

This should return JSON listing available models. No response indicates Ollama service issues or firewall blocking.

Simple Performance Comparison

Benchmark GPU vs CPU inference with identical prompts:

# GPU-accelerated inference (default)
time ollama run gemma:2b "Write a Python function to calculate fibonacci numbers"

# Force CPU-only inference
OLLAMA_NO_GPU=1 time ollama run gemma:2b "Write a Python function to calculate fibonacci numbers"

Expected performance characteristics on RTX 3060-class hardware:

The 6-10x performance improvement justifies GPU infrastructure investment for any serious local LLM deployment.

Monitoring Long-Term Performance

Implement continuous monitoring to detect performance degradation from driver updates, thermal throttling, or resource contention:

# Real-time GPU monitoring with 1-second refresh
watch -n 1 nvidia-smi

# Log GPU utilization to file for analysis
nvidia-smi --query-gpu=timestamp,utilization.gpu,utilization.memory,memory.used,temperature.gpu --format=csv --loop=10 > gpu_metrics.csv

Review logs periodically to identify thermal throttling (temperatures exceeding 80°C) or memory pressure indicating need for model optimization or hardware upgrades.

Enterprise Ollama Deployment Services

Our MSP ELITE package now includes professional Ollama infrastructure deployment and management, delivering turnkey AI capabilities backed by 24/7 expert support. ITECS eliminates the trial-and-error of driver configuration and provides enterprise-grade reliability from day one.

Why Choose ITECS for AI Infrastructure

Building internal expertise for GPU-accelerated AI infrastructure diverts resources from core business objectives. ITECS brings proven methodologies developed across hundreds of enterprise deployments, eliminating the costly learning curve that derails internal projects.

One-Command Installation

curl -fsSL https://ollama.com/install.sh | sh

This command performs multiple operations automatically:

• • Downloads the latest Ollama binary optimized for Linux AMD64 architecture
• • Installs the binary to /usr/local/bin/ollama with executable permissions
• • Creates dedicated ollama system user and group for service isolation
• • Installs and enables systemd service for automatic startup
• • Configures model storage directory at /usr/share/ollama/.ollama/models

The installation script includes error handling and typically completes in 10-30 seconds on modern systems with good internet connectivity. Installation output confirms successful service creation and startup.

Verifying Ollama Installation and GPU Detection

Immediately after installation, verify Ollama correctly detects your NVIDIA GPU. Ollama shows clear error messages when GPU detection fails, so thorough verification prevents frustrating debugging sessions later.

Step 1: Check Ollama service status

systemctl status ollama

Output should show active (running) status. If the service failed to start, examine logs with journalctl -xeu ollama.

Step 2: Test GPU detection with a model pull

ollama pull gemma:2b

This downloads Gemma 2B, a compact model perfect for verification testing. The 2B variant requires only ~1.7GB VRAM, ensuring compatibility with entry-level GPUs while providing meaningful performance comparison between CPU and GPU inference.

Step 3: Run inference and monitor GPU utilization

Open a second terminal and start real-time GPU monitoring:

watch -n 1 nvidia-smi

In your primary terminal, run an inference test:

ollama run gemma:2b "Explain quantum computing in simple terms"

While the model generates output, observe your monitoring terminal. Successful GPU acceleration shows:

• Memory-Usage increase: VRAM allocation grows by 2-3GB as the model loads into GPU memory
• GPU-Util spikes: Utilization reaches 60-100% during active inference
• Processes section lists: /usr/local/bin/ollama as an active GPU process
• Rapid token generation: Text appears quickly, indicating GPU-accelerated performance

Pitfall #1: Nouveau Driver Conflicts

Ubuntu ships with the open-source Nouveau driver for NVIDIA GPUs, which conflicts with proprietary NVIDIA drivers. The open-source NVIDIA kernel driver nouveau can conflict with the proprietary NVIDIA driver when both are loaded. While ubuntu-drivers typically handles this automatically, manual installations or upgrade scenarios sometimes leave Nouveau active.

Detection: Check for Nouveau module loading

lsmod | grep nouveau

Any output indicates Nouveau is loaded. Proceed with blacklisting.

Solution: Blacklist Nouveau driver permanently

sudo bash -c "echo blacklist nouveau > /etc/modprobe.d/blacklist-nvidia-nouveau.conf"
sudo bash -c "echo options nouveau modeset=0 >> /etc/modprobe.d/blacklist-nvidia-nouveau.conf"
sudo update-initramfs -u
sudo reboot

After reboot, verify Nouveau is no longer loaded and NVIDIA driver functions correctly.

Pitfall #2: Driver Version Mismatches

API mismatch errors occur when the userspace driver packages were upgraded while the kernel module is still on the older version. This situation often occurs after a system upgrade. Ollama may refuse to detect GPUs when driver component versions don't align.

Detection: Check for version mismatches

# Check userspace driver version
nvidia-smi

# Check kernel module version
cat /proc/driver/nvidia/version

Mismatched version numbers indicate the problem. Kernel logs may show: NVRM: API mismatch: the client has the version 570.172.08, but this kernel module has the version 570.158.01

Solution: Synchronize driver versions through reboot

sudo reboot

Simple reboot typically resolves version mismatches by loading updated kernel modules. If issues persist, reinstall the driver package:

sudo apt install --reinstall nvidia-driver-550

Pitfall #3: Secure Boot Complications

Modern Ubuntu systems enable Secure Boot by default, requiring kernel module signatures for loading. Unsigned NVIDIA modules trigger boot failures or prevent driver initialization. Ubuntu's ubuntu-drivers tool automatically handles Secure Boot by installing signed drivers from repositories, but manual installations or custom driver versions cause problems.

Detection: Check Secure Boot status

mokutil --sb-state

Output showing SecureBoot enabled requires signed modules.

Solution Option 1: Use signed drivers from Ubuntu repositories (recommended)

Install drivers via ubuntu-drivers or official Ubuntu packages, which include proper Secure Boot signatures.

Solution Option 2: Disable Secure Boot (less secure)

Access UEFI/BIOS settings during boot (typically F2, F10, or Del key) and disable Secure Boot. This enables unsigned module loading but reduces system security posture—not recommended for production deployments.

Pitfall #4: Suspend/Resume GPU Loss

On Linux, after a suspend/resume cycle, sometimes Ollama will fail to discover your NVIDIA GPU and fallback to running on the CPU due to a driver bug. This affects laptops and workstations using system suspend, causing mysterious performance degradation after resume.

Detection: GPU detected at boot but not after suspend/resume cycle

Solution: Reload NVIDIA UVM driver

sudo rmmod nvidia_uvm && sudo modprobe nvidia_uvm

For persistent solution, create a systemd service to reload UVM after resume:

sudo nano /etc/systemd/system/nvidia-resume.service

Add configuration:

Enable the service:

sudo systemctl enable nvidia-resume.service

Pitfall #5: Specific Driver Version Incompatibilities

After updating to NVIDIA driver 555.85, Ollama can no longer use the GPU. The issue stems from incompatibility between driver 555.85 and Ollama. Downgrading the driver to version 552.44 resolves the problem. Not all driver versions maintain equal compatibility with Ollama's bundled CUDA libraries.

Detection: Ollama suddenly stops detecting GPU after driver update, despite nvidia-smi functioning correctly

Solution: Downgrade to known-good driver version

# Remove problematic driver
sudo apt remove --purge nvidia-driver-555
sudo apt autoremove

# Install known-compatible version
sudo apt install nvidia-driver-550
sudo reboot

Hold the driver package to prevent automatic upgrades:

sudo apt-mark hold nvidia-driver-550

Monitor Ollama GitHub issues and release notes before upgrading drivers in production environments.

Configuring Network Access

By default, Ollama binds to localhost only. To make it reachable from other machines or containers, you must configure the OLLAMA_HOST environment variable. This enables web UI integrations, API access from other machines, and containerized client connections.

Edit the systemd service configuration:

sudo systemctl edit ollama

Add override configuration:

Restart Ollama to apply changes:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Multi-GPU Configuration

Systems with multiple NVIDIA GPUs require explicit configuration to control which GPUs Ollama utilizes. Set CUDA_VISIBLE_DEVICES to a comma-separated list of GPUs. UUIDs are more reliable than numeric IDs for consistent GPU identification.

Identify GPU UUIDs:

nvidia-smi -L

Example output:

Configure Ollama to use specific GPUs:

sudo systemctl edit ollama

Add environment variable:

For multiple GPUs, use comma-separated UUID list. Restart Ollama after configuration changes.