Skip to content

🚀 Manual Deployments & Project Provisioning

Audience: DevOps Leads / Backend Engineers Purpose: Standard Operating Procedures (SOPs) for provisioning new GitLab repositories on the bare-metal host and executing manual deployments when the CI/CD pipeline is unavailable.

The Service Account Rule

All application code and Docker containers must be owned and executed by the app-runner service account. Never clone repositories or run docker compose as your personal user.

1. Initializing a New Project

When a new microservice or frontend (like a new Korvy backend module) is ready for its first staging or production deployment, follow this sequence to securely bridge GitLab to the host server.

Step 1: Switch Context

Log into the server with your personal credentials, then immediately switch to the service account. 

# From your local machine
ssh <your-username>@<server-ip>

# Switch to the service account
sudo -u app-runner -H bash

Step 2: Generate a GitLab Deploy Key

We use a unique, read-only SSH key for each repository. This ensures that if a container is compromised, the attacker only gains read access to that specific codebase, not the entire Apollyon GitLab instance.

  1. Generate the key (replace <project-name>):
ssh-keygen -t ed25519 -C "deploy-key-<project-name>" -f ~/.ssh/id_ed25519_<project-name>

(Leave the passphrase empty by pressing Enter twice so automated pulls can run without human intervention).

  1. Tell the SSH daemon to use this specific key for this specific repository by editing ~/.ssh/config:
nano ~/.ssh/config

Add this block:

# Key for <project-name>
Host gitlab-<project-name>
    HostName gitlab.apollyon.lat
    User git
    IdentityFile ~/.ssh/id_ed25519_<project-name>
    IdentitiesOnly yes

Step 3: Register Key & Clone

  1. Copy the public key output from cat ~/.ssh/id_ed25519_<project-name>.pub.
  2. Navigate to the project in GitLab -> Settings -> Repository -> Deploy Keys.
  3. Paste the key, name it Production Server (app-runner), and leave "Write access" unchecked.
  4. Clone the repository into the standard /opt/ directory using your new alias:
# Syntax: git clone git@<Host-Alias>:<Group>/<Repo>.git /opt/<project-name>
git clone git@gitlab-<project-name>:apollyon/korvy-backend.git /opt/<project-name>

Step 4: Environment & Permissions

Copy the .env template and securely inject the production secrets (retrieved from our Secret Manager, never Nextcloud).

cd /opt/<project-name>
cp .env.example .env
nano .env

Critical Permission Fix: Ensure the webapps group has read access so Caddy can serve static files if necessary. Exit back to your personal user to run the sudo fix:

exit # Drops you back to your personal user
sudo chown -R app-runner:webapps /opt/<project-name>
sudo chmod -R 775 /opt/<project-name>

2. Routine Manual Deployment

For routine updates to existing services, the workflow is much simpler.

  1. Switch to the service account:
sudo -u app-runner -H bash
  1. Navigate to the project and pull the latest main branch:
cd /opt/<project-name>
git pull origin main
  1. Execute a Zero-Downtime Restart:
docker compose up -d --build

Do not use down

Running docker compose down followed by up will completely destroy the container network and cause a hard outage. Always use up -d --build to let Docker gracefully recreate the containers in the background and hot-swap them.

  1. Verify the health of the newly deployed containers:
docker compose logs -f --tail=50

3. Emergency Rollbacks

If a new deployment immediately crashes (e.g., throwing 500 errors in the logs), execute a rapid rollback.

  1. Identify the last stable commit hash from GitLab.
  2. Hard reset the local repository and rebuild:
cd /opt/<project-name>
git reset --hard <stable-commit-hash>
docker compose up -d --build
  1. Once the service is restored, alert the #engineering channel in Mattermost so the team can investigate the bad code on their local machines.