Running Local LLMs on Linux (lubuntu)

Check System OS

  • Kernel version
adminping@adminping-localai:~$ uname -r
7.0.0-30-generic
  • CPU features (AVX2 needed for efficient inference)
adminping@adminping-localai:~$ lscpu | grep -i avx
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpuid_fault cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx_vnni avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid bus_lock_detect movdiri movdir64b overflow_recov succor smca fsrm avx512_vp2intersect flush_l1d amd_lbr_pmc_freeze
  • Total RAM
adminping@adminping-localai:~$ free -h
               total        used        free      shared  buff/cache   available
Mem:            30Gi       2.2Gi        23Gi        49Mi       5.1Gi        28Gi
Swap:          511Mi          0B       511Mi
  • List GPU
adminping@adminping-localai:~$ lspci | grep -i 'vga\|3d\|display'
01:00.0 VGA compatible controller: NVIDIA Corporation GB203 [GeForce RTX 5070 Ti] (rev a1)
73:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Granite Ridge [Radeon Graphics] (rev c5)

Install NVIDIA 5070Ti (Driver)

Update Package & Linux Kernel with build tools (For Compile Driver)

sudo apt update && sudo apt upgrade -y
sudo apt install linux-headers-$(uname -r) build-essential dkms -y

Let ubuntu Recommend Avaliable Driver Device - nvidia-driver-595-open - distro non-free recommended

adminping@adminping-localai:~$ ubuntu-drivers devices
ERROR:root:aplay command not found
== /sys/devices/pci0000:00/0000:00:01.1/0000:01:00.0 ==
modalias : pci:v000010DEd00002C05sv00001569sd0000F332bc03sc00i00
vendor   : NVIDIA Corporation
model    : GB203 [GeForce RTX 5070 Ti]
driver   : nvidia-driver-615-open - distro non-free
driver   : nvidia-driver-580-server-open - distro non-free
driver   : nvidia-driver-595 - distro non-free
driver   : nvidia-driver-595-server-open - distro non-free
driver   : nvidia-driver-580-server - distro non-free
driver   : nvidia-driver-580 - distro non-free
driver   : nvidia-driver-595-open - distro non-free recommended
driver   : nvidia-driver-595-server - distro non-free
driver   : nvidia-driver-615-server-open - distro non-free
driver   : nvidia-driver-580-open - distro non-free
driver   : xserver-xorg-video-nouveau - distro free builtin

You can choose 2 options

  • Ubuntu’s packaged driver
# List available drivers
ubuntu-drivers list

# Install the recommended driver
sudo ubuntu-drivers install

# Reboot
sudo reboot
  • NVIDIA’s official driver
# Add NVIDIA's repository for ubuntu 20-24 
sudo apt install -y software-properties-common
sudo add-apt-repository -y ppa:graphics-drivers/ppa
sudo apt update

# Install the latest driver 595 from recommended 
sudo apt install nvidia-driver-595-open

# Reboot
sudo reboot

check command

adminping@adminping-localai:~$ nvidia-smi
Sun Nov  1 17:57:36 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.99.02              Driver Version: 595.99.02      CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5070 Ti     Off |   00000000:01:00.0 Off |                  N/A |
|  0%   47C    P8             23W /  300W |      15MiB /  16303MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            1524      G   /usr/lib/xorg/Xorg                        4MiB |
+-----------------------------------------------------------------------------------------+

if no device found

Install Container

Docker + Docker Compose

1. Install dependency packages:

sudo apt install apt-transport-https ca-certificates curl -y

2. Add Docker's GPG key + Register the Docker repository

sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

3 Update Repo

sudo apt update

4. Install Docker Engine and plugins:

sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y

Verify the install

Configure Docker as a System Service

  • Enable and start the service
sudo systemctl enable docker
sudo systemctl start docker
  • Check Service Status
sudo systemctl status docker
  • If you want to restart / stop
sudo systemctl stop docker
sudo systemctl restart docker

Add Your User to the Docker Group

sudo usermod -aG docker $USER
newgrp docker

# Test 
docker ps 

NVIDIA Container Toolkit

sudo apt-get update && sudo apt-get install -y --no-install-recommends \
  ca-certificates curl gnupg2

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update

export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.1-1
sudo apt-get install -y \
  nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

unsloth

/home/adminping/unsloth/
├── compose.yaml
├── .env
├── work/                 # notebook, dataset, script
├── outputs/              # LoRA adapter, GGUF, checkpoint
└── cache/
    └── huggingface/      # model ที่ดาวน์โหลด

create path

mkdir -p /home/adminping/unsloth/{work,outputs,cache/huggingface,studio}
cd /home/adminping/unsloth

full compose.yaml

services:
  services:
  unsloth:
    image: unsloth/unsloth
    container_name: unsloth
    restart: unless-stopped
    ipc: host
    ulimits:
      memlock: { soft: -1, hard: -1 }
      stack: 67108864
    ports:
      - "18880:8000"   # Unsloth Studio (UI)
      - "18888:8888"   # JupyterLab
    environment:
      - JUPYTER_PASSWORD=${JUPYTER_PASSWORD:?set JUPYTER_PASSWORD in .env}
      - UNSLOTH_STUDIO_PASSWORD=${UNSLOTH_STUDIO_PASSWORD:?set UNSLOTH_STUDIO_PASSWORD in .env}
      - HF_TOKEN=${HF_TOKEN:-}
    volumes:
      - ${BASE_DIR}/work:/workspace/work
      - ${BASE_DIR}/outputs:/workspace/outputs
      - ${BASE_DIR}/cache/huggingface:/workspace/.cache/huggingface
      - ${BASE_DIR}/studio:/opt/unsloth-studio
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

env

BASE_DIR=/home/adminping/unsloth
JUPYTER_PASSWORD=รหัสผ่าน-jupyter
UNSLOTH_STUDIO_PASSWORD=รหัสผ่าน-studio
HF_TOKEN=

wsewe

sudo ufw allow 18880/tcp
sudo ufw allow 18888/tcp

Test Accces via Web UI http://192.168.1.191:18880/

Reference