How Much Should Live Together? Learning to Isolate Services the Hard Way

Also Published On

• and how much time and pain they can save you. Its hundreds of community-built and maintained scripts make managing and deploying services in your homelab a breeze. Applications pre-configured, sensible defaults applied, and in the case of Plex, GPU detection and passthrough handled automatically. It's a big part of why "one LXC container per service" is realistic to live with instead of just a nice idea on paper.
  
  
  

  The instinct when you hear "isolated container" is to assume everything about that service, including its data, lives inside it. I've landed on the opposite approach for anything with real state. Keep the container itself as small as possible, and put the data somewhere that isn't tied to any one container or node.

  
  
  

  Plex is a good example of why this matters most. Plex's metadata directory (thumbnails, artwork cache, SQLite database, watch history, etc.) consumes a lot of storage that can grow to hundreds of GBs in larger setups. If that lived inside the container's own disk, moving that container between nodes would mean moving 100GB+ with it every time. Instead, my data now lives on NVMe-backed shared storage, mounted identically on every node in the Proxmox cluster. The LXC itself stays under 10GB. When it needs to move for maintenance, load balancing, or hardware failure, it's not carrying its data across the network, it's just reconnecting to storage that was already sitting there waiting for it.

  
  
  

  
  
  
  The Actual Migration
  

  
  
  

  Plex was one of the last holdouts. Nearly everything else in the stack had already made the jump to LXC over the previous several months. Plex stuck around in its VM the longest because it was a frequently used service with GPU constraints and years of accumulated metadata to carefully handle.

  
  
  

  The GPU part turned out to be an interesting problem. The install script handled basic detection and passthrough automatically by pointing the container at the same device files the host driver already created, things like /dev/nvidia0, instead of handing over the whole card the way a VM does. That worked immediately. It just didn't survive a reboot, and it hard-failed on any node without a GPU at all. The fix was to edit the raw LXC config, mark each device mount as optional, and add cgroup rules that grant access without requiring the device to exist. Example:
  
  

  
  
  
  
  

  CODE Kopieren

  # /etc/pve/lxc/<vmid>.conf
  
  lxc.cgroup2.devices.allow: c 195:* rwm
  
  lxc.cgroup2.devices.allow: c 236:* rwm
  
  lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
  
  lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
  
  lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
  
  

  
  
  
  
  
  
  
  

  The optional flag on each mount means Proxmox won't refuse to start the container just because that device file doesn't exist on a given node, and the cgroup2.devices.allow rules grant access without requiring every device to be present up front. Now the container starts fine either way and picks up whatever GPU is available.

  
  
  

  There was also a kernel compatibility snag along the way. Newer Proxmox kernels had shipped API changes the current NVIDIA driver didn't support yet, and the driver branch that did support them had dropped older GPU architectures. Pinning both nodes to a slightly older, still-supported kernel and driver combo fixed it for now, and is an easy update once those are available.

  
  
  

  There are a few services I run that don't and might not ever have a Proxmox helper script. For those, the plan is a small LXC container running Docker, pointed at the same shared storage pattern as everything else. Just as portable and all the benefits of both without the issues of a large VM.

  
  
  

  
  
  
  The Outcome
  

  
  
  

  Once all wired up, testing confirmed migrations to other nodes with different GPUs worked flawlessly. Same identity, same library, same users, and hardware transcoding picked up automatically on whatever card happened to be there. No manual reconfiguration, no re-pairing devices, no rebuilding anything.

  
  
  

  A service that used to be pinned to one bulky VM and GPU is now a lightweight container that can land anywhere in the cluster and pick up local hardware acceleration if available. The data that actually matters never has to move and makes recovery and failover instant and painless. The performance and efficiency gains have been unmatched in previous configurations.

  
  
  
  
  
  

  If you've made it this far and know of a better version of this, I'd love to discuss it in the comments. The iterative process has brought me here and I couldn't be happier where it's at today. I also understand that I don't know what I don't know until I know and there's many ways to solve the same problem I might be unaware of. I'm not here to tell you what's the "right" way, just what has worked for me and the knowledge gained thus far. Every headache encountered taught me something I didn't know I needed, and I'm better off for having gone through it. Enjoy and appreciate the struggle, it's what gets you where you need to be far more than answers do.

  
  
  

  If you're curious where the homelab is at today, check it out on my portfolio site at https://trever.cloud/projects/homelab/.

  Vollständiger Original-Bericht

  Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.

  ↗ Original-Artikel auf dev.to lesen