How Ubuntu 24.04's PEP 668 Broke Our Month-End Automation — SSH Multi-Node Workaround

Background

We run automation across multiple nodes in our home lab. At month-end, a script called monthly-endofmonth.sh handles time-tracking, invoice submission, and form filling automatically.

This month, it exploded on first run.

What Happened

The error surfaced like this:

error: externally-managed-environment



× This environment is externally managed

╰─> To install Python packages system-wide, either use a

    distribution package (apt install python3-xxx) or create

    a virtual environment (python3 -m venv PATH)...

Running pip install playwright on Ubuntu 24.04 got blocked by PEP 668.

PEP 668, adopted in 2022, changes how Python package management works. On Ubuntu 24.04 (Noble), direct pip install to the system Python is forbidden. The goal is to prevent apt-managed Python and pip from conflicting and breaking the system.

Previously, pip install --break-system-packages could force through. Ubuntu 24.04 blocks even that in many cases.

The Real Root Cause: Phase 0 and Production Ran on Different Nodes

PEP 668 itself is well-known. The real problem was something else.

Phase 0 (pre-flight check) ran on our infra server — a node with playwright, openpyxl, and google-auth already set up. Everything passed.

But the actual month-end job was designed to run on a different Ubuntu 24.04 node that had none of those libraries.

Result: Phase 0 all green → production blew up immediately.

This environment mismatch is a classic multi-node trap. The infra equivalent of "works on my machine."

The Fix: Route Python Jobs via SSH to the Prepared Node

We changed the approach: delegate Python execution to the infra server via SSH.

The key change in monthly-endofmonth.sh:

# Before: local execution on Ubuntu 24.04 node (blocked by PEP 668)

python3 /mnt/shared/scripts/make_timesheet.py



# After: run on infra server via SSH

ssh user@infra-server "python3 /mnt/shared/scripts/make_timesheet.py"

The infra server has a fixed LAN IP, and all scripts are on an NFS mount (/mnt/shared/), so both nodes access the same files regardless of which machine runs the command.

Post-fix results:

Lessons Learned

1. Run Phase 0 Checks on the Production Node

"It worked somewhere" doesn't mean "it works everywhere." Dependencies, Python environments, and file paths vary per node. Phase 0 should run on the actual node that will execute production jobs.

2. Move Work to the Prepared Environment, Not the Other Way Around

Fighting PEP 668 is the wrong battle. Moving the job to an already-configured environment is faster and safer.

Using venv is also an option, but in a multi-node NFS setup, managing venv paths across machines gets messy. SSH-delegating to a single trusted node is simpler.

3. Test Month-End Scripts Mid-Month

Testing one day before month-end is too late. Next month, we plan to add a --dry-run pass around the 25th to catch environment issues early.

Bonus: The SameFileError Bug

We found a bonus bug during debugging. Running make_timesheet.py twice on the same day produces:

shutil.SameFileError: '/mnt/shared/.../timesheet.xlsx' and '/mnt/shared/.../timesheet.xlsx' are the same file

shutil.copy() was passed the same path as both source and destination. A simple existence check before copying will fix it — planned for next month's run.

Summary

• Ubuntu 24.04 enforces PEP 668, blocking system-wide pip install
  


• Multi-node setups need explicit awareness of which node runs which job
  


• SSH-delegating to a prepared environment is a clean, pragmatic solution


• Month-end automation needs mid-month dry-run testing, not last-minute manual checks