Git Devops

Motivation

After we have built a web service, the following actions quickly become necesary:

How to deploy it?

Common approaches are to simply dump it into heroku / vercel / any other cloud service or introduce a heavyweight like Ansible / Kubernetes.

A lot of my projects run on cheap VMs on linode. None of these approaches are viable for me without spending a lot of effort maintaining the infrastructure around these.

How to re-deploy on git push / tag?

Integrating tightly with your git host is the common approach. To have netlify / vercel / render.com auto-deploy from github or gitlab.

I can't use that when I don't have an app that fits into the way they want to deploy apps. For example, I want to use a rust and python web server in the same project. Nope. None of the existing system will allow for that directly without having to fiddle around with their settings, at which point I'm investing into their infrastructure.

How to manage separate environments / services?

Besides running on dev machines projects need to run on test/prod/staging/uat and many more environments. Sometimes even on-premises. Managing that is not a trivial task.

Managing secrets / configuration

Cross language projects need to pick up things from configuration. That causes another headache since now you need to manage who has access to what config secrets.

Solution

This is how I manage my projects now. A git dev ops workflow.

Machine setup

1. All machines need to have git and docker compose installed.
2. We need to add the following snippet to ~/.bashrc

dk (){
    REPOROOT=$(git rev-parse --show-toplevel)
    ([ ! -z "$REPOROOT" ] && source $REPOROOT/cicd/bashrc 2> /dev/null && $@)
}

Project setup

1. Single docker-compose.yml file per project in the root of the project
2. cicd folder to contain scripts we can use. This contains bashrc script that provides management commands for the project.
3. secrets folder to contain secrets using SOPS. Each environment is in a separate file. We never commit <env>.key files and only commit <env>.enc files.

Example

Base layout

.
├── cicd
│   └── bashrc
├── docker-compose.yml
└── secrets
    ├── ci.enc
    └── ci.key

Environment vars

secrets/bin/set_env.sh is used to set environment variables in CI / prod / dev etc.

#!/usr/bin/env bash

export BIN=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
export SECRETS=$(echo "$BIN/..")
export NAME=$1
export SOPS_AGE_KEY_FILE=$SECRETS/$NAME.key 
export $($BIN/sops --decrypt --input-type dotenv --output-type dotenv $SECRETS/$NAME.enc | xargs)

secrets/bin/edit_env.sh is used to edit env vars using some text editor.

#!/usr/bin/env bash

set -o errexit
set -o pipefail

main (){
    NAME=$1
    BIN=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
    SECRETS=$(echo "$BIN/..")
    KEY_FILE=$(echo "$SECRETS/$NAME.key")
    ENC_FILE=$(echo "$SECRETS/$NAME.enc")
    PLAINTEXT_FILE=$(echo "$SECRETS/$NAME.plaintext")
    echo $BIN
    echo $SECRETS
    echo $KEY_FILE
    echo $ENC_FILE
    echo $PLAINTEXT_FILE
    if [[ -f "$SECRETS/$NAME.enc" ]]; then
        SOPS_AGE_KEY_FILE=$KEY_FILE $BIN/sops --decrypt --input-type dotenv --output-type dotenv $ENC_FILE > $PLAINTEXT_FILE
    fi
    vim $PLAINTEXT_FILE
    $BIN/sops --input-type dotenv --output-type dotenv --encrypt --age $($BIN/age-keygen -y $KEY_FILE) $PLAINTEXT_FILE > $ENC_FILE
    rm $PLAINTEXT_FILE
}
(main $1)

docker-compose.yml

• A single docker-compose file is used to define the services for the project.
• I use docker compose profiles to define when a service should be selected / deselected for operations.
  • For example, I will have the profiles ["infra", "redis_prod", "prod-dbs"] on my redis service.
  • For my web server I will have ["prod", "api_prod"].
• Common profiles used in my compose are:
  • prod : For prod api services
  • infra: Anything that does not need to start / redeploy when I do a prod release. These are databases / load balancers etc.
  • dev : For local dev testing / development.
  • <service>_<env>: To specifically target a single service. Useful when I want to restart only redis_uat.

Services

• I define multiple api services for my api servers. One per environment. I use YAML anchors to avoid duplication of values.
• Databases are also defined in this compose file itself. One per environment. These can be redis / postgres etc.
• Observation infrastructure like glitchtip / victoria / grafana are also part of the compose file.
• To tie it all together I use caddy as a gateway and use that to expose the entire project to the world.

Running project

cicd/bashrc has a few command to allow me to run operations on the project.

reporoot(){
    git rev-parse --show-toplevel
}
set_env_vars() {
    USER_ID=$(id -u)
    echo "
        touch .$ENV.env \
        && source secrets/bin/set_env.sh $ENV \
        && bash secrets/bin/create_envfile.sh $ENV \
        && export USER_ID='$USER_ID' "

}
up(){
    ENV=$1
    PROFILE="${2:-$ENV}"
    cd $(reporoot)
    echo "
    (
        $(set_env_vars) \
        && docker compose --profile $PROFILE \
        up --build --force-recreate -d
    )"
}
down(){
    ENV=$1
    PROFILE="${2:-$ENV}"
    cd $(reporoot)
    echo "
    (
        $(set_env_vars) \
        && docker compose --profile $PROFILE \
        down
    )"
}

This allows me to run commands like:

• dk up dev | bash to compose up dev environment.
• dk down dev | bash to compose down dev environment.
• dk up prod infra | bash will compose up all of the infrastructure required for my services.