Setting Up an AI Server on Proxmox 9.x Using LXC and Docker

This guide walks through setting up a local AI server on Proxmox VE 9.x. The finished setup uses a Debian 13 LXC container to run Docker, Dockge, and llama.cpp, with an NVIDIA GPU available to the llama.cpp server.

The example uses a Qwen3.5 4B GGUF model, but the same setup can be adapted for other models and supported hardware. This guide starts with an existing Proxmox VE installation; it does not cover installing Proxmox itself.

To make it clear where each command belongs, the steps identify whether you should run it on the Proxmox host, inside the Debian LXC, or in the Dockge web interface.

1. Prepare the Proxmox Host

The first step is to prepare the Proxmox host for the NVIDIA driver and create the shared directory that will hold model files and presets.

Run the Proxmox post-install script

I recommend running the Proxmox post-install script from the Community Scripts project. It takes care of several common post-install tasks.

Proxmox VE Post Install

Run on the Proxmox host:

bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/tools/pve/post-pve-install.sh)"

Install the required packages

Install the packages needed to build and load the NVIDIA driver. The package list includes the headers for the currently running Proxmox kernel.

Run on the Proxmox host:

apt install g++ freeglut3-dev build-essential libx11-dev libxmu-dev libxi-dev libglu1-mesa-dev libfreeimage-dev libglfw3-dev wget htop btop nvtop nano glances git pciutils cmake curl libcurl4-openssl-dev pve-headers-$(uname -r) dkms make -y

Install the NVIDIA driver

This guide uses NVIDIA driver version 580.196.07. Copy the driver installer to the Proxmox host. The example command uses 192.168.150.49 as the host address; change it to match your system.

Run from the machine that has the installer:

scp NVIDIA-Linux-x86_64-580.196.07-no-compat32.run root@192.168.150.49:/root/

Connect to the Proxmox host, make the installer executable, and install the driver with DKMS enabled.

Run on the Proxmox host:

chmod +x NVIDIA-Linux-x86_64-580.196.07-no-compat32.run
./NVIDIA-Linux-x86_64-580.196.07-no-compat32.run --dkms

Update the initramfs and reboot:

update-initramfs -u && reboot now

After the host comes back up, run the installer a second time:

./NVIDIA-Linux-x86_64-580.196.07-no-compat32.run --dkms

Use nvtop to verify that the NVIDIA driver is loaded and the GPU is visible:

nvtop

Configure NVIDIA persistence and memory locking

Enable NVIDIA persistence mode and set the power limit at boot. Edit root’s crontab:

crontab -e

Add this line:

@reboot nvidia-smi -pm 1 && nvidia-smi -pl 250

Next, configure memory locking limits:

nano /etc/security/limits.conf

Add these lines:

* soft memlock unlimited
* hard memlock unlimited
root soft memlock unlimited
root hard memlock unlimited

Save the file and reboot the host for the memory-locking changes to take effect.

Create the shared models directory

The model files and preset configuration will be stored in /opt/models. This directory will later be mounted inside the LXC container.

Run on the Proxmox host:

mkdir /opt/models

2. Create and Configure the Debian LXC

The AI services will run inside a Debian 13 LXC container. This setup uses a privileged container and enables nesting so Docker can run inside it.

Download the Debian 13 template

In the Proxmox web interface, select your node and then select the storage where you want to keep the template. Open CT Templates, click Templates, and download the latest Debian 13 template.

The template is stored on the selected Proxmox storage. You only need to download it once for each storage location where you plan to create containers.

Create the container

Create a new container and select the Debian 13 template. Make sure Unprivileged container is unchecked. This guide uses a privileged container.

When assigning CPU cores, count physical cores rather than hyperthreading threads. For example, an 8-core/16-thread processor has 8 physical cores, not 16. You can assign up to 8 cores to the container, depending on how many you want to reserve for Proxmox and other workloads.

Allocate the amount of memory you want the container to use. Complete the remaining settings and create the container, but do not start it yet.

Enable nesting

There is a Proxmox quirk worth knowing: after you select a privileged container, the Nesting option cannot be enabled in the creation wizard. Create the container first, then open Options → Features and enable Nesting.

Add GPU device access

Before starting the container, add the GPU device entries. The exact device names and numbers depend on your hardware. This is an example for a system with one NVIDIA GPU:

/dev/nvidia0
/dev/nvidiactl
/dev/nvidia-uvm
/dev/nvidia-uvm-tools
/dev/nvidia-caps/nvidia-cap1
/dev/nvidia-caps/nvidia-cap2
/dev/dri/card0
/dev/dri/renderD128

Set the device permissions to 0666. Systems with multiple GPUs will have additional NVIDIA device entries. The DRI devices are also needed for Vulkan access.

Configure the LXC

Open the container configuration file on the Proxmox host. Replace <CTID> with the container ID:

Run on the Proxmox host:

nano /etc/pve/lxc/<CTID>.conf

Add the following configuration:

# allow lxc to pin large memory blocks
lxc.prlimit.memlock: unlimited

# mount shared models directory
mp0: /opt/models,mp=/opt/models

# cpu pinning
lxc.cgroup2.cpuset.cpus: 0-5

The CPU pinning example assigns logical CPU IDs 0 through 5 to the container. Adjust this range to match your system and the CPUs you want to reserve for the container.

Start the container and copy the driver installer

Start the LXC container from the Proxmox web interface. Then copy the NVIDIA installer from the Proxmox host into the running container.

Run on the Proxmox host:

pct push <CTID> /root/NVIDIA-Linux-x86_64-580.196.07-no-compat32.run /root/NVIDIA-Linux-x86_64-580.196.07-no-compat32.run

3. Prepare Debian and Install the NVIDIA Driver

The next steps are performed inside the Debian LXC container.

Update Debian and install dependencies

Open a shell in the Debian container and update the system:

Run inside the Debian LXC:

apt update && apt upgrade -y

Install the packages required for NVIDIA tools, Vulkan, and the container setup:

apt update && apt install nvtop libvulkan1 mesa-vulkan-drivers vulkan-tools curl gpg gnupg2 sudo libxext6 libegl1 -y

The libegl1 package is important for this setup. It must be installed in the LXC container; it is not added to the llama.cpp Docker image.

Install the NVIDIA driver in the LXC

The NVIDIA driver version inside the LXC should match the driver installed on the Proxmox host. Run the installer without kernel modules because the host provides the kernel driver.

chmod +x NVIDIA-Linux-x86_64-580.196.07-no-compat32.run

Run the installer:

./NVIDIA-Linux-x86_64-580.196.07-no-compat32.run --no-kernel-modules

Verify NVIDIA and Vulkan access

Check that the NVIDIA driver is visible inside the container:

nvidia-smi

Then check Vulkan:

vulkaninfo --summary

4. Install NVIDIA Container Toolkit

The NVIDIA Container Toolkit allows Docker containers to access the NVIDIA GPU.

Add the NVIDIA package repository

Create the APT keyrings directory:

Run inside the Debian LXC:

mkdir -p /etc/apt/keyrings

Download and install the NVIDIA repository key:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /etc/apt/keyrings/nvidia-container-toolkit-keyring.gpg

Add the NVIDIA Container Toolkit repository:

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list -o /etc/apt/sources.list.d/nvidia-container-toolkit.list

Configure the repository to use the keyring:

sed -i 's#deb https://#deb [signed-by=/etc/apt/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' /etc/apt/sources.list.d/nvidia-container-toolkit.list

Update APT and install the toolkit:

apt-get update && apt-get install -y nvidia-container-toolkit

5. Install Docker

Docker will run inside the Debian LXC container. The following steps add Docker’s official Debian repository and install Docker Engine with the Compose plugin.

Add the Docker package repository

Run inside the Debian LXC:

apt update
apt install -y ca-certificates curl
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc

Create the Docker repository definition:

cat > /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/debian
Suites: $(. /etc/os-release && echo "$VERSION_CODENAME")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF

Install and configure Docker

Install Docker Engine and the Compose plugin:

apt update
apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Configure Docker to use the NVIDIA runtime and restart Docker:

nvidia-ctk runtime configure --runtime=docker
systemctl restart docker

Generate the NVIDIA CDI device specification:

nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

This creates the device specification Docker can use to expose NVIDIA GPUs to containers.

6. Install Dockge

Dockge provides a web interface for managing Docker Compose stacks.

Deploy Dockge

Run inside the Debian LXC:

mkdir -p /opt/stacks /opt/dockge
cd /opt/dockge

Download the Dockge Compose file:

curl https://raw.githubusercontent.com/louislam/dockge/master/compose.yaml --output compose.yaml

Start Dockge:

docker compose up -d

Open Dockge in a browser:

http://<LXC-IP-ADDRESS>:5001

Create the administrator account when prompted.

7. Download the Qwen Model

The model files are stored in /opt/models, which is shared between the Proxmox host and the LXC container. You can download the model from either the Proxmox host shell or the LXC shell.

This example uses the Qwen3.5 4B GGUF model from Unsloth:

Unsloth Qwen3.5 4B GGUF

Download the Q4_K_M file:

cd /opt/models
wget https://huggingface.co/unsloth/Qwen3.5-4B-GGUF/resolve/main/Qwen3.5-4B-Q4_K_M.gguf

Verify that the file is present:

ls -lh /opt/models/Qwen3.5-4B-Q4_K_M.gguf

8. Configure Model Presets

llama.cpp model presets let you define different configurations for CPU and GPU testing. The preset file is stored in the shared /opt/models directory, so you can create or edit it from either the Proxmox host shell or the LXC shell.

Create the preset file

nano /opt/models/mini-server.ini

Add the following configuration:

#
# llama-server --models-preset config.ini --host 127.0.0.1 --port 8080 --models-max 1
#
# This is a master file of different models that have been tested.
# Comment and uncomment as needed.
#

# =========================================================
# Global default fallback settings for your system
# =========================================================
[*]
b = 64
ub = 64
cache-prompt = true
cache-type-k = q4_0
cache-type-v = q4_0
context-shift = true
cont-batching = true
cram = 1024
ctx-checkpoints = 32
ctx-size = 8192
flash-attn = on
kv-unified = true
np = 1
prio = 2
t = 4
tb = 4

# =========================================================
# Test models - Comment out models when done.
# =========================================================
# Pure CPU Test
[test-qwen3.5_4B-CPU]
model = /models/Qwen3.5-4B-Q4_K_M.gguf 
gpu-layers = 999
ctx-size = 16384
load-mode = none
split-mode = none
device = none
t = 6
tb = 6

# GPU 0 Test
[test-qwen3.5_4B-GPU0]
model = /models/Qwen3.5-4B-Q4_K_M.gguf 
gpu-layers = 999
ctx-size = 16384
no-host = 1
load-mode = mlock
split-mode = none
device = Vulkan0

# GPU 1 Test
[test-qwen3.5_4B-GPU1]
model = /models/Qwen3.5-4B-Q4_K_M.gguf 
gpu-layers = 999
ctx-size = 16384
no-host = 1
load-mode = mlock
split-mode = none
device = Vulkan1

# GPU 2 Test
#[test-qwen3.5_4B-GPU1]
#model = /models/Qwen3.5-4B-Q4_K_M.gguf 
#gpu-layers = 999
#ctx-size = 16384
#no-host = 1
#load-mode = mlock
#split-mode = none
#device = Vulkan2

# GPU 0 + 1 Test
[test-qwen3.5_4B-dual_GPUs]
model = /models/Qwen3.5-4B-Q4_K_M.gguf 
gpu-layers = 999
ctx-size = 32768
no-host = 1
load-mode = mlock
split-mode = layer
device = Vulkan0,Vulkan1

# GPU 0 + 1 + 2 Test
#[test-qwen3.5_4B-GPU1]
#model = /models/Qwen3.5-4B-Q4_K_M.gguf 
#gpu-layers = 999
#ctx-size = 16384
#no-host = 1
#load-mode = mlock
#split-mode = none
#device = Vulkan0,Vulkan1,Vulkan2

The [*] section defines global defaults. Each model section inherits those settings and can override them. Uncomment the preset you want to test and comment out the other test presets. The preset file is passed to llama.cpp when the Docker container starts.

9. Deploy the llama.cpp Server

Use Dockge to create a new stack for llama.cpp. The Compose configuration below uses the Vulkan server image. To use the CUDA image instead, uncomment that image and comment out the Vulkan image. Keep only one image line active.

The NVIDIA runtime is enabled for NVIDIA GPUs. The optional cpuset setting can be enabled if the LXC container has not already been pinned to specific physical cores. Adjust the CPU range for your system.

Create the llama.cpp stack

In the Dockge web interface: create a new stack and use this Compose configuration:

services:
  ai-stack:
    container_name: ai-stack
    # uncomment only one of the image lines below.
    # image: ghcr.io/ggml-org/llama.cpp:server-cuda
    image: ghcr.io/ggml-org/llama.cpp:server-vulkan
    # uncomment only one of the runtimes below. Nvidia GPUs need
    # the nvidia runtime.
    runtime: nvidia
    #runtime: runc
    restart: unless-stopped
    # uncomment below if your lxc isn't pinned to the physical cores.
    # You will need to adjust to the number of cores you have.
    # cpuset: 0-5
    ports:
      - 8080:8080
    volumes:
      - /opt/models:/models
    devices:
      - /dev/dri/card0:/dev/dri/card0
      - /dev/dri/renderD128:/dev/dri/renderD128
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=all
      - GGML_VK_FORCE_MAX_ALLOCATION=1
      # - __GL_THREADED_OPTIMIZATIONS=0
    ulimits:
      memlock:
        soft: -1
        hard: -1
    command: >
      --host 0.0.0.0 --port 8080 --models-preset /models/mini-server.ini -lv 3 --models-max 1
networks: {}

What the Compose configuration does

Selects the llama.cpp Vulkan server image. The CUDA image is available as an alternative.
Uses the NVIDIA runtime for NVIDIA GPUs.
Publishes the server on port 8080.
Mounts the shared /opt/models directory inside the container as /models.
Passes the DRI devices through for Vulkan access.
Makes NVIDIA devices and driver capabilities available to the container.
Sets unlimited memory locking for the container.
Loads model presets from /models/mini-server.ini and limits the server to one loaded model with --models-max 1.

Save and deploy the stack in Dockge.

10. Test the llama.cpp Installation

Once the stack has started, check the logs in Dockge and confirm that llama.cpp starts without errors.

From the Debian LXC shell, you can also check GPU visibility:

nvidia-smi

Open the llama.cpp web interface in a browser:

http://<LXC-IP-ADDRESS>:8080

The web interface should load and allow you to select one of the configured model presets. Select the Qwen3.5 4B preset and start a conversation. The first request loads the model, so allow time for the model to finish loading.

While the model is running, use nvtop in the LXC shell to monitor GPU memory use and activity.

Conclusion

You now have a working foundation for running local AI models on a Proxmox server. Proxmox manages the hardware, while a dedicated Debian LXC container provides an environment for Docker, NVIDIA Container Toolkit, Dockge, and llama.cpp.

The shared /opt/models directory makes it easy to manage model files and presets without copying them into individual containers. Dockge provides a convenient web interface for managing the Docker Compose stacks.

From here, you can add more models, create presets for different GPU configurations, and experiment with model sizes and quantization levels as your hardware allows. This gives you a flexible platform for running and testing local language models without depending on cloud-based AI services.