⚙️ CI/CD Pipeline Standards

Audience: DevOps Leads / Backend Engineers Purpose: Standard Operating Procedure (SOP) for configuring automated continuous deployment pipelines for new repositories.

At Apollyon, we utilize a Push-to-Deploy model via GitLab CI/CD. To protect the bare-metal host, we strictly use the Docker + SSH methodology. The GitLab Runner spins up an ephemeral Alpine Linux container, injects a masked SSH key, logs into the server as the app-runner service account, executes the update, and destroys itself.

The 1-to-1 Key Rule

Never reuse CI/CD SSH keys across different repositories. Every project must generate its own unique SSH key pair. This limits the blast radius; if a single project's pipeline is compromised, the attacker only gains access to that specific project folder in /opt/, not the entire server.

---

1. Server-Side: Key Provisioning

When setting up a new repository (e.g., a new client API or frontend), you must first create its dedicated access key on the host.

1. Generate the Key: Log in with your personal admin user and create a passphrase-less key named after the project:

   ssh-keygen -t ed25519 -C "gitlab-ci-<project-name>" -f ~/.ssh/id_ci_<project-name>
   

2. Authorize the Key: Copy the public key and append it to the service account's authorized list:

   cat ~/.ssh/id_ci_<project-name>.pub
   
   # Switch context
   sudo -u app-runner -H bash
   nano ~/.ssh/authorized_keys
   

3. Encode and Destroy: To safely store the private key in GitLab, it must be Base64 encoded to remove line breaks.

   # Copy the output of this command:
   cat ~/.ssh/id_ci_<project-name> | base64 -w 0
   
   # Destroy the unencrypted file
   rm ~/.ssh/id_ci_<project-name>
   

---

2. GitLab-Side: Variable Configuration

Navigate to the new repository in GitLab ➡️ Settings ➡️ CI/CD ➡️ Variables.

• SSH_PRIVATE_KEY
• Value: The Base64 string you just copied.

• Flags: Check Mask variable and Protect variable.

• SERVER_IP

• Value: The direct IP address of the deployment server.
• Flags: Leave unprotected (unless required by strict branch rules), but do not mask.

---

3. The Standard .gitlab-ci.yml Template

Create this file in the root of the new repository. Modify the folder paths and deployment commands as needed.

stages:
  - deploy

production_deployment:
  stage: deploy
  image: alpine:latest

  # Execute only on protected branches
  only:
    - main

  before_script:
    - apk add --no-cache openssh-client bash
    - eval $(ssh-agent -s)
    # Decode the Base64 key
    - echo "$SSH_PRIVATE_KEY" | base64 -d | ssh-add -
    # Secure Host Verification
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - ssh-keyscan -H $SERVER_IP >> ~/.ssh/known_hosts
    - chmod 644 ~/.ssh/known_hosts

  script:
    - >
      ssh app-runner@$SERVER_IP
      "cd /opt/<project-name> &&
      git pull origin main &&
      docker compose up -d --build"

  environment:
    name: production
    url: https://<project-url>

---

4. Troubleshooting Silent Pipeline Failures

SSH pipelines can be notoriously difficult to debug because they often swallow standard error outputs. If your pipeline returns ERROR: Job failed: exit code 1 but provides no text context, check the following:

The "Invisible Shell" Trap

If ssh connects but instantly dies without running your script, the app-runner account might be missing a valid login shell. Debian sometimes defaults service accounts to /bin/false. The Fix: Assign a proper bash shell to the account from your admin user:

sudo usermod -s /bin/bash app-runner

Absolute Paths for Package Managers

Automated SSH sessions often fail to load user-specific $PATH variables (like ~/.local/bin). If a command like poetry, npm, or yarn is failing silently, use the absolute path to the executable in your YAML script block:

# BAD:
git pull origin main && poetry install

# GOOD:
git pull origin main && ~/.local/bin/poetry install

The "Nuclear" Debug Method

To force the pipeline to print exactly why a command is failing, modify your script block to redirect all output to a server-side log, then read it back:

  script:
    - >
      ssh app-runner@$SERVER_IP
      "bash -l -c 'set -x && cd /opt/<project-name> && git pull origin main' > /tmp/ci-deploy.log 2>&1" || true
    - ssh app-runner@$SERVER_IP "cat /tmp/ci-deploy.log"
    - ssh app-runner@$SERVER_IP "grep -i 'error\|failed\|fatal' /tmp/ci-deploy.log && exit 1 || exit 0"