⚙️ CI/CD Pipeline Standards

Audience: DevOps Leads / Backend Engineers Purpose: Standard Operating Procedure (SOP) for configuring automated continuous deployment pipelines for new repositories.

At Apollyon, we utilize a Push-to-Deploy model via GitLab CI/CD. To protect the bare-metal host, we strictly use the Docker + SSH methodology. The GitLab Runner spins up an ephemeral Alpine Linux container, injects a masked SSH key, logs into the server as the app-runner service account, executes the update, and destroys itself.

The 1-to-1 Key Rule

Never reuse CI/CD SSH keys across different repositories. Every project must generate its own unique SSH key pair, so one project's access can be revoked (delete its line from app-runner's authorized_keys) without touching the others, and a leak is traced to one repository. Note that every key logs in as app-runner, which owns every folder in /opt/: a leaked key is a server-wide incident, so rotate it at once.

---

1. Server-Side: Key Provisioning

When setting up a new repository (e.g., a new client API or frontend), you must first create its dedicated access key on the host.

1. Generate the Key: Log in with your personal admin user and create a passphrase-less key named after the project:

   ssh-keygen -t ed25519 -C "gitlab-ci-<project-name>" -f ~/.ssh/id_ci_<project-name>
   

2. Authorize the Key: Copy the public key and append it to the service account's authorized list:

   cat ~/.ssh/id_ci_<project-name>.pub
   
   # Switch context
   sudo -u app-runner -H bash
   nano ~/.ssh/authorized_keys
   

3. Encode and Destroy: To safely store the private key in GitLab, it must be Base64 encoded to remove line breaks.

   # Copy the output of this command:
   cat ~/.ssh/id_ci_<project-name> | base64 -w 0
   
   # Destroy the unencrypted file
   rm ~/.ssh/id_ci_<project-name>
   

---

2. GitLab-Side: Variable Configuration

Navigate to the new repository in GitLab ➡️ Settings ➡️ CI/CD ➡️ Variables. Variables belong to the project: a new repository needs its own, even when another project already has variables with the same names (unless they are set on a parent group, where the project's page lists them as inherited).

Variable	Value	Flags
SSH_PRIVATE_KEY	The Base64 string from step 1 (a key made for this repository only)	Masked and Protected
SERVER_IP	The server's address (an IP or a host name)	Visible, not protected
SSH_KNOWN_HOSTS	The server's host key lines (see below)	Protected, visible (it is a public key; GitLab cannot mask multi-line values)

Protected variables only reach pipelines on protected branches and tags, so protect main (Settings ➡️ Repository ➡️ Protected branches) and any v* release tags.

Getting SSH_KNOWN_HOSTS

The three variables answer three different questions: SERVER_IP says where CI connects, SSH_PRIVATE_KEY proves who CI is to the server, and SSH_KNOWN_HOSTS proves who the server is to CI. Without it, a job that runs ssh-keyscan at deploy time trusts whatever machine answers at that address.

1. From your laptop (not the server), scan exactly the value of SERVER_IP: each line is tied to that host string.

   ssh-keyscan -H <SERVER_IP> 2>/dev/null > known_hosts.txt
   ssh-keygen -lf known_hosts.txt        # the fingerprints that answered
   

2. On the server, list the real fingerprints:

   for f in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf "$f"; done
   

3. The SHA256:... values must match. If they do not, stop.

4. Paste the |1|... lines of known_hosts.txt (not the # comment lines) as the variable's value, then delete the file.

The same value works for every project deploying to the same server. If the host keys ever change (a reinstall), deploys fail with Host key verification failed until the variable is updated: that failure is the protection working.

---

3. The Standard .gitlab-ci.yml Deploy Job

Every deploy job connects the same way; only the command it runs on the server differs with the project's pattern (Deployment Patterns):

• Docker Compose (most services): cd /opt/<project> && git pull origin main && docker compose up -d --build.
• Static site or systemd service: cd /opt/<project> && deploy/deploy.sh $CI_COMMIT_SHA, the project's own script, which an admin also runs by hand (the deploy script).

The template below shows the script variant; for a Docker Compose project, replace the last line of script: with the Compose command.

deploy_main:
  stage: deploy
  image: alpine:3.22                 # pinned, never :latest
  needs: [lint, test]                # deploy only what passed
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE == "push"
  # One deploy at a time. A resource group does not keep them in order, so the
  # script also refuses to go back to an older commit.
  resource_group: production
  interruptible: false
  timeout: 30m
  environment:
    name: production
    url: https://<project-url>
  variables:
    GIT_STRATEGY: none               # the server has its own checkout
    DEPLOY_DIR: /opt/<project>
  before_script:
    - apk add --no-cache openssh-client
    - eval $(ssh-agent -s)
    - echo "$SSH_PRIVATE_KEY" | base64 -d | ssh-add -
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - |
      if [ -n "${SSH_KNOWN_HOSTS:-}" ]; then
        printf '%s\n' "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
      else
        echo "SSH_KNOWN_HOSTS is not set; trusting the host key the server presents" >&2
        ssh-keyscan -H "$SERVER_IP" >> ~/.ssh/known_hosts
      fi
  script:
    - ssh "app-runner@$SERVER_IP" "cd $DEPLOY_DIR && deploy/deploy.sh $CI_COMMIT_SHA"
    # Docker Compose project instead:
    # - ssh "app-runner@$SERVER_IP" "cd $DEPLOY_DIR && git pull origin main && docker compose up -d --build"

Notes:

• Use rules:, not only:/except: (the older syntax, which cannot combine conditions).
• Also turn on Settings ➡️ CI/CD ➡️ General pipelines ➡️ Prevent outdated deployment jobs.
• With merge requests, every merge to main deploys. Docs-only changes can be skipped with changes: on the rule.

Configuration Pitfalls

• Never name a job after a global keyword. At the top level of .gitlab-ci.yml, image, services, cache, variables, default, stages, include, workflow, before_script and after_script are keywords. A job called image: makes GitLab reject the whole file (image name should be a string), so no job runs at all. Name it docker_build or similar. Check new files in Build ➡️ Pipeline editor ➡️ Validate.
• Slim images lack tools. python:*-slim has no git; install what the tests need in the job (apt-get install -y --no-install-recommends git).
• Hidden templates. Share the SSH setup between deploy jobs with a hidden job (.deploy:) and extends:, or reuse one step with !reference [.job, before_script].

---

4. Troubleshooting Silent Pipeline Failures

SSH pipelines can be notoriously difficult to debug because they often swallow standard error outputs. If your pipeline returns ERROR: Job failed: exit code 1 but provides no text context, check the following:

The "Invisible Shell" Trap

If ssh connects but instantly dies without running your script, the app-runner account might be missing a valid login shell. Debian sometimes defaults service accounts to /bin/false. The Fix: Assign a proper bash shell to the account from your admin user:

sudo usermod -s /bin/bash app-runner

Absolute Paths for Package Managers

Automated SSH sessions often fail to load user-specific $PATH variables (like ~/.local/bin). On this host Poetry is installed with pipx and linked at /usr/local/bin/poetry, and Node and npm are in /usr/bin, so both are on the default PATH. For anything installed per user, use the absolute path to the executable:

# BAD:
git pull origin main && poetry install

# GOOD:
git pull origin main && ~/.local/bin/poetry install

The "Nuclear" Debug Method

To force the pipeline to print exactly why a command is failing, modify your script block to redirect all output to a server-side log, then read it back:

  script:
    - >
      ssh app-runner@$SERVER_IP
      "bash -l -c 'set -x && cd /opt/<project-name> && git pull origin main' > /tmp/ci-deploy.log 2>&1" || true
    - ssh app-runner@$SERVER_IP "cat /tmp/ci-deploy.log"
    - ssh app-runner@$SERVER_IP "grep -i 'error\|failed\|fatal' /tmp/ci-deploy.log && exit 1 || exit 0"