Git Devops
Motivation
After we have built a web service, the following actions quickly become necesary:
How to deploy it?
Common approaches are to simply dump it into heroku / vercel / any other cloud service or introduce a heavyweight like Ansible / Kubernetes.
A lot of my projects run on cheap VMs on linode. None of these approaches are viable for me without spending a lot of effort maintaining the infrastructure around these.
How to re-deploy on git push / tag?
Integrating tightly with your git host is the common approach. To have netlify / vercel / render.com auto-deploy from github or gitlab.
I can't use that when I don't have an app that fits into the way they want to deploy apps. For example, I want to use a rust and python web server in the same project. Nope. None of the existing system will allow for that directly without having to fiddle around with their settings, at which point I'm investing into their infrastructure.
How to manage separate environments / services?
Besides running on dev machines projects need to run on test/prod/staging/uat and many more environments. Sometimes even on-premises. Managing that is not a trivial task.
Managing secrets / configuration
Cross language projects need to pick up things from configuration. That causes another headache since now you need to manage who has access to what config secrets.
Solution
This is how I manage my projects now. A git dev ops
workflow.
Machine setup
- All machines need to have git and docker compose installed.
- We need to add the following snippet to
~/.bashrc
dk (){
REPOROOT=$(git rev-parse --show-toplevel)
([ ! -z "$REPOROOT" ] && source $REPOROOT/cicd/bashrc 2> /dev/null && $@)
}
Project setup
- Single
docker-compose.yml
file per project in the root of the project cicd
folder to contain scripts we can use. This containsbashrc
script that provides management commands for the project.secrets
folder to contain secrets using SOPS. Each environment is in a separate file. We never commit<env>.key
files and only commit<env>.enc
files.
Example
Base layout
.
├── cicd
│ └── bashrc
├── docker-compose.yml
└── secrets
├── ci.enc
└── ci.key
Environment vars
secrets/bin/set_env.sh
is used to set environment variables in CI / prod / dev etc.
#!/usr/bin/env bash
export BIN=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
export SECRETS=$(echo "$BIN/..")
export NAME=$1
export SOPS_AGE_KEY_FILE=$SECRETS/$NAME.key
export $($BIN/sops --decrypt --input-type dotenv --output-type dotenv $SECRETS/$NAME.enc | xargs)
secrets/bin/edit_env.sh
is used to edit env vars using some text editor.
#!/usr/bin/env bash
set -o errexit
set -o pipefail
main (){
NAME=$1
BIN=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
SECRETS=$(echo "$BIN/..")
KEY_FILE=$(echo "$SECRETS/$NAME.key")
ENC_FILE=$(echo "$SECRETS/$NAME.enc")
PLAINTEXT_FILE=$(echo "$SECRETS/$NAME.plaintext")
echo $BIN
echo $SECRETS
echo $KEY_FILE
echo $ENC_FILE
echo $PLAINTEXT_FILE
if [[ -f "$SECRETS/$NAME.enc" ]]; then
SOPS_AGE_KEY_FILE=$KEY_FILE $BIN/sops --decrypt --input-type dotenv --output-type dotenv $ENC_FILE > $PLAINTEXT_FILE
fi
vim $PLAINTEXT_FILE
$BIN/sops --input-type dotenv --output-type dotenv --encrypt --age $($BIN/age-keygen -y $KEY_FILE) $PLAINTEXT_FILE > $ENC_FILE
rm $PLAINTEXT_FILE
}
(main $1)
docker-compose.yml
- A single docker-compose file is used to define the services for the project.
- I use docker compose profiles to define when a service should be selected / deselected for operations.
- For example, I will have the profiles
["infra", "redis_prod", "prod-dbs"]
on my redis service. - For my web server I will have
["prod", "api_prod"]
.
- For example, I will have the profiles
- Common profiles used in my compose are:
prod
: For prod api servicesinfra
: Anything that does not need to start / redeploy when I do a prod release. These are databases / load balancers etc.dev
: For local dev testing / development.<service>_<env>
: To specifically target a single service. Useful when I want to restart onlyredis_uat
.
Services
- I define multiple
api
services for my api servers. One per environment. I use YAML anchors to avoid duplication of values. - Databases are also defined in this compose file itself. One per environment. These can be redis / postgres etc.
- Observation infrastructure like glitchtip / victoria / grafana are also part of the compose file.
- To tie it all together I use caddy as a gateway and use that to expose the entire project to the world.
Running project
cicd/bashrc
has a few command to allow me to run operations on the project.
reporoot(){
git rev-parse --show-toplevel
}
set_env_vars() {
USER_ID=$(id -u)
echo "
touch .$ENV.env \
&& source secrets/bin/set_env.sh $ENV \
&& bash secrets/bin/create_envfile.sh $ENV \
&& export USER_ID='$USER_ID' "
}
up(){
ENV=$1
PROFILE="${2:-$ENV}"
cd $(reporoot)
echo "
(
$(set_env_vars) \
&& docker compose --profile $PROFILE \
up --build --force-recreate -d
)"
}
down(){
ENV=$1
PROFILE="${2:-$ENV}"
cd $(reporoot)
echo "
(
$(set_env_vars) \
&& docker compose --profile $PROFILE \
down
)"
}
This allows me to run commands like:
dk up dev | bash
to compose up dev environment.dk down dev | bash
to compose down dev environment.dk up prod infra | bash
will compose up all of the infrastructure required for my services.