/home /blog 28 Feb 2023 | Get ipynb

Jaypore CI


A long time back, I used to create git projects for small ideas that I used to have and push them to github. Eventually I used Travis CI for the automated testing of my code. Then came a time when I discovered that gitlab offered infinite private repositories! I moved a lot of my projects there and was delighted by the easy to use CI that they offered. One yaml file and things would get tested quickly.

Eventually CI became a pain point. I found myself copy pasting common configs from one project to another. It seemed that a cookie cutter tool was needed. As my projects grew in quantity, and in individual size it I started needing more complex behavior from the CI system itself. I wanted to build docker images, publish them, build binaries, publish those, combine binaries / test reports / changelogs and build documentation out of them. Config variables, deployment keys had to be managed / rolled / deprecated. Certain jobs only had to be run on specific branches, specifit commit messages, when certain files changed, when certain jobs failed and so on.

As my projects started being used by others, other issues arose. I had to keep on sharing/removing access for CI as well as the projects, worry about secrets being leaked, help people to debug code since a lot of the times the bugs were found in the CI system so they had to learn the CI system's way of debugging things. Projects were scattered across github / gitlab / bitbucket and consequently CI started being scattered across github Actions / gitlab CI / Jenkins / Drone CI / Travis CI and a lot others. This was obviously a reflection of how people work in teams. Every team had a different thing going on and sometimes within the same git repo you needed to use different CI systems since different parts were handled by different teams. When releases were done stakeholders had to be emailed sometimes. At other times, jobs had to be run when people requested them. Pull requests (PRs) had to be updated with CI reports and so we had to integrate them with github/gitlab/bitbucket, then issues had to be re-labelled / marked as done. Sometimes based on a PR's labels we had to cherry pick to other branches and so on.

Commonly people's answer to this was to live inside one of the gardens. Either pick github or gitlab and stick to whatever they provide. The nature of my projects meant that I was not the one dictating where the projects lived.

Premature clarity

Eventually it got to a point where I just stopped using fancy stuff and did everything in git as much as possible and those practicies gave rise to the Jaypore CI project.

Some of these practices were enjoyable, some brought immense productivity to junior devs at a small cost, some became footguns, some were outright questionable but since the load of maintenance was ultimately on me I was happy. Eventually I thought of putting all of this ritual into a template of sorts, or a package, and eventually decided on this blog post.

This was also the time I started to prefer self hosting things. It started with gitlab changing their pricing for one of out clients. When they looked at self hosting I realized that it does not make sense to shell out so much money just to have a single repo on that instance. I started to self host my code on a linode using gitea.

A lack of CI systems that could compete with gitea in terms of weightlessness drove me first to agola CI, then drone, eventually leading me to write my own.

Choosing the name itself took some time. Simple CI? No, CI should be powerful and minimal, not necessarily simple. Writing yaml is simple, but debugging it is horrific. No CI? Well, we're not exactly against CI itself are we? Power CI? It is powerful, but I don't want people to mistakenly think that it's something in the family of Power BI/PowerShell. Cross CI? It does work across gitlab/github/gitea but that's not the main point of it. It could work anywhere! The final name I decided on was JayporeCI. I live in the city of Jaipur. It's an ancient city, powerful enough to repell invaders for centuries. The people live a simple life, a happy life. It has adopted to modern times well enough to change it's name from Jaypore to Jaipur. It has all the charm of monster cities like Delhi but none of the size. Yes; small, powerful, and very flexible. That's what our CI system was. Jaypore CI.

Jaypore CI: growing slowly with needs

The first cut had a very simple flow.

This was amazing! Nothing ever broke down and I could use it wherever I wanted without having to integrate CI with my git provider. The script itself was in git so I could simply clone on another machine and use it there as well without any extra config/setup effort. However I missed some of the things that years of CI usage had drilled into me. I did not want to merge my PRs unless CI passed. I wanted a nice graph, a way to see job dependencies, which ones took too long, and which ones ran suspiciously fast.

JayporeCI has matured to some extent now. Over time and repeated usage in different projects it has acquired a list of features that enable very powerful workflows for small teams / individual developers. Each of these features was added as a direct requirement for some project / team structure / automation need.

Local and offline first

Config in python

Secrets are in SOPS

Sharing logs / reports