Skip to content

🚀 Manual Deployments & Project Provisioning

Audience: DevOps Leads / Backend Engineers Purpose: Standard Operating Procedures (SOPs) for provisioning new GitLab repositories on the bare-metal host and executing manual deployments when the CI/CD pipeline is unavailable.

The Service Account Rule

All application code, services and Docker containers must be owned and executed by the app-runner service account (home /opt/app-runner, login shell /bin/bash, member of the docker group). Never clone repositories, run docker compose or start services as your personal user.

Choose the pattern first

Static site, Python service under systemd, or Docker Compose: see Deployment Patterns. The steps below are common to all three.

1. Initializing a New Project

When a new microservice or frontend (like a new Korvy backend module) is ready for its first staging or production deployment, follow this sequence to securely bridge GitLab to the host server.

Step 1: Switch Context

Log into the server with your personal credentials, then immediately switch to the service account. 

# From your local machine
ssh <your-username>@<server-ip>

# Switch to the service account
sudo -u app-runner -H bash

Step 2: Generate a GitLab Deploy Key

We use a unique, read-only SSH key for each repository. This ensures that if a container is compromised, the attacker only gains read access to that specific codebase, not the entire Apollyon GitLab instance.

  1. Generate the key (replace <project-name>):
ssh-keygen -t ed25519 -C "deploy-key-<project-name>" -f ~/.ssh/id_ed25519_<project-name>

(Leave the passphrase empty by pressing Enter twice so automated pulls can run without human intervention).

  1. Tell the SSH daemon to use this specific key for this specific repository by editing ~/.ssh/config:
nano ~/.ssh/config

Add this block:

# Key for <project-name>
Host gitlab-<project-name>
    HostName gitlab.apollyon.lat
    Port 2222
    User git
    IdentityFile ~/.ssh/id_ed25519_<project-name>
    IdentitiesOnly yes

GitLab's SSH is on port 2222

Port 22 on the host is the server's own SSH daemon; GitLab's container listens on 2222. Without the Port 2222 line the alias connects to the host's SSH and the clone fails. Remotes written out in full use ssh://git@gitlab.apollyon.lat:2222/<group>/<repo>.git.

Step 3: Register Key & Clone

  1. Copy the public key output from cat ~/.ssh/id_ed25519_<project-name>.pub.
  2. Navigate to the project in GitLab -> Settings -> Repository -> Deploy Keys.
  3. Paste the key, name it Production Server (app-runner), and leave "Write access" unchecked.
  4. Clone the repository into the standard /opt/ directory using your new alias:
# Syntax: git clone git@<Host-Alias>:<Group>/<Subgroup>/<Repo>.git /opt/<project-name>
git clone git@gitlab-<project-name>:apollyon/teapoyoapp/teapoyo-backend.git /opt/<project-name>

Step 4: Environment & Permissions

Copy the .env template and securely inject the production secrets (retrieved from our Secret Manager, never Nextcloud).

cd /opt/<project-name>
cp .env.example .env
nano .env

Permissions. Project folders belong to app-runner:app-runner with the setgid bit (2775 on directories), so new files keep the group. Caddy (user caddy) is not in that group: it reads static files through the "others" read bit, which 2775/664 keep. There is no webapps group on the host. The .env file holds production secrets and must be readable by app-runner only:

exit # Drops you back to your personal user
sudo chown -R app-runner:app-runner /opt/<project-name>
sudo find /opt/<project-name> -type d -exec chmod 2775 {} +
sudo chmod 600 /opt/<project-name>/.env

Never chmod -R 775 a project folder

A recursive 775 makes every file, including .env with the database password, readable by every user on the server.

2. Routine Manual Deployment

Projects that ship a deploy/deploy.sh (Deployment Patterns) deploy by hand exactly as CI does:

sudo -u app-runner -H bash -c 'cd /opt/<project-name> && deploy/deploy.sh'

For Docker Compose projects without a script, the workflow is:

  1. Switch to the service account:
sudo -u app-runner -H bash
  1. Navigate to the project and pull the latest main branch:
cd /opt/<project-name>
git pull origin main
  1. Rebuild and recreate the containers that changed:
docker compose up -d --build

Do not use down

Running docker compose down followed by up tears down the container network and causes a full outage. up -d --build builds first and only recreates the containers that changed, which keeps the gap short (a few seconds per container); it is not zero downtime.

  1. Verify the health of the newly deployed containers:
docker compose logs -f --tail=50

3. Emergency Rollbacks

If a new deployment immediately crashes (e.g., throwing 500 errors in the logs), execute a rapid rollback.

With a deploy script, go back to the last good commit on purpose (the script otherwise refuses to move backwards). If the bad commit added a database migration, undo it first, from the newer code (.venv/bin/alembic downgrade -1):

sudo -u app-runner -H bash -c 'cd /opt/<project-name> && FORCE=1 deploy/deploy.sh <stable-commit-hash>'

Without a script:

  1. Identify the last stable commit hash from GitLab.
  2. Hard reset the local repository and rebuild:
cd /opt/<project-name>
git reset --hard <stable-commit-hash>
docker compose up -d --build
  1. Once the service is restored, alert the #engineering channel in Mattermost so the team can investigate the bad code on their local machines.