Skip to content

🖥️ Host Administration Runbooks

Audience: System Administrators / DevOps Leads Purpose: Standard Operating Procedures (SOPs) for managing users, permissions, and core packages on the bare-metal Debian host.

Root Access Warning

Avoid executing commands as the root user directly. Always log in with your personal admin account and use sudo.

1. Create a New Developer Account

We grant developers server access without giving them full root privileges. Developers are permitted to log in and switch to the app-runner service account to perform deployments.

Step 1: Create the System User

sudo adduser <username>

Set a secure temporary password and store it in the company password manager to share with the new hire.

Step 2: Grant Deployment Permissions (Limited Sudo)

We do not add developers to the sudo group. We allow them to switch only to the app-runner user.

  1. Open the sudo configuration safely: 

    sudo visudo

  2. Scroll to the bottom and append this rule (replace <username>): 

    # Allow <username> to switch to app-runner without a password
<username> ALL=(app-runner) NOPASSWD: ALL

Step 3: Inject SSH Key

Do not rely on the temporary password. Once the user provides their ed25519.pub key:

sudo su - <username>
mkdir -p ~/.ssh
chmod 700 ~/.ssh
nano ~/.ssh/authorized_keys # Paste the public key here
chmod 600 ~/.ssh/authorized_keys
exit

2. The app-runner Context Switch

When an engineer needs to restart a container or pull code, they must switch to the service account. Never run docker compose up as your personal user.

# Switch from personal user to the service account
sudo -u app-runner -H bash

Verify the prompt changes to app-runner@server: before touching the /opt/ directories.

3. GitLab Runner Storage Management

The GitLab Runner docker executor leaks disk space in two ways that it never cleans up on its own:

  • Dangling images, stopped containers, and build cache left behind by every job.
  • Per-project named volumes (runner-<runner-id-hash>-cache-<path-hash>) holding each project's /cache (GitLab's cache: archives) and /builds (the persistent git checkout). One pair is created per project, and — critically — the <runner-id-hash> portion changes whenever the runner's registration token is rotated or a new project first runs a pipeline on it. Each change strands the old volume's data and creates a fresh one, and Docker always creates a fresh named volume under /var/lib/docker/volumes, i.e. on the root filesystem.

Left unmanaged, both of these have repeatedly filled the root SSD. Two scripts, run via /etc/cron.d/, keep this under control:

gitlab-ci-cleanup.sh — image, container, and cache-key cleanup

Runs daily. Evicts dangling images, runner-managed stopped containers, build cache older than 7 days, and — inside each /cache volume — superseded cache archives (keeping only the newest per dependency key).

Never docker image prune -a on this host

-a removes any image with no running container, which also catches a service that simply happens to be stopped at that moment. This is exactly how a production service's image was deleted by an earlier ad-hoc cleanup attempt. The script deliberately uses the conservative docker image prune -f (dangling only) instead — do not "optimize" this without re-reading why.

# Manual run
sudo /usr/local/bin/gitlab-ci-cleanup.sh
tail -60 /var/log/gitlab-ci-cleanup.log

# Preview only, changes nothing
sudo DRY_RUN=1 /usr/local/bin/gitlab-ci-cleanup.sh

runner-volume-janitor.sh — keeps runner volumes off the root disk

Runs hourly. Finds any runner-* Docker volume currently backed by the root filesystem, relocates its data onto /mnt/newhdd/gitlab-runner/volumes/, and re-creates the volume under the same name pointed at that path — so whatever references the volume by name keeps working with no reconfiguration. It also removes backing directories on /mnt/newhdd whose volume no longer exists (e.g. after a token rotation), once they've been orphaned for a few days. Relocation is skipped for the hour if a CI job is currently running, and retried on the next tick.

# Manual run
sudo /usr/local/bin/runner-volume-janitor.sh
tail -60 /var/log/runner-volume-janitor.log

# Preview only, changes nothing
sudo DRY_RUN=1 /usr/local/bin/runner-volume-janitor.sh

This is a size-based exception to the Enterprise Storage Standard

Server Topology states that persistent volumes must map to /mnt/newhdd/. In practice, only relocate a volume if it will realistically grow past ~2 GB — the janitor above exists because runner cache/build volumes routinely do. A small, short-lived volume for a lightweight service is fine left on the default root-backed Docker volume store; moving everything regardless of size adds operational complexity (an extra relocation step, an extra thing that can be left mismatched after a restore) for no real disk-space benefit.

Large CI images

Some projects' jobs use big images (for example the Playwright image, about 2 GB, which ships browsers for end-to-end tests). The runner keeps them like any other image, so they count against the root disk until gitlab-ci-cleanup.sh evicts them once unused. Keep an eye on df -h / after adding a project with heavy images.

4. Letting a Deploy Restart Its Own Service

Projects that run under systemd (Deployment Patterns) need app-runner to restart their own unit after a deploy, and nothing else. Give it one exact command per service, never ALL:

echo 'app-runner ALL=(root) NOPASSWD: /usr/bin/systemctl restart <service>' |
  sudo tee /etc/sudoers.d/<service>
sudo chmod 440 /etc/sudoers.d/<service>
sudo visudo -c           # must say "parsed OK"

The deploy script then calls sudo -n systemctl restart <service> (-n fails at once instead of waiting for a password). The unit file itself is installed by an admin (sudo cp deploy/<service>.service /etc/systemd/system/ && sudo systemctl daemon-reload), so a merged change to it is applied only when an admin copies it again.

Appendix: Bare-Metal Docker Installation

If the server is ever rebuilt, do not use apt install docker.io. We require the official Docker repository to support modern Compose V2 features.

1. Clean the System:

sudo apt-get remove docker docker-engine docker.io containerd runc

2. Add the Official GPG Key:

sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

3. Setup the Repository & Install:

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian $(. /etc/os-release && echo \"$VERSION_CODENAME\") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

4. Permission Configuration:

sudo groupadd docker
sudo usermod -aG docker app-runner